Table of Contents
1. Introduction
It might happen one day that you wake up in the morning and realize that .security index that you deleted last night on production cluster is still not there and you need to face reality. Not much to worry if you have working backup that was done with SUCCESS status not PARTIAL (because nobody check it). Worse if you do not have a backup because then you have to recreate roles and users. If you use LDAP for authentication you still need to recreate mapping between AD groups and roles. Let’s start.
2. Start Elasticsearch
Because I want to show you also how to restore it from backup please configure working elasticsearch instance as it was described in my article
Elasticsearch backup & restore to S3 IPFS using docker swarm with secrets
3. Populate .security-7 index
If you check .security-7 index it will only have information about elastic built-in superuser. I want to show you case with more data.
To populate security index
# execute commands on container Elasticsearch
dockerContainerID=`docker ps --filter name=elastic -q`;
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto --batch --url https://localhost:9200/';
# setup password again to previous
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';
curl -k -XGET -u elastic "https://localhost:9200/.security-7/_search"
4. Create role with rights to delete system index
superuser elastic does not have permission to delete system indexes started their names with ‘.'(dot). Therefore it is necessary to create new role and new user that will have that role. Notice line 9 and 11 – I set minimal privilege just to delete index and I am giving permission to restricted indices.
curl -k -XPOST -u elastic "https://localhost:9201/_security/role/deleter" -H 'Content-Type: application/json' -d'
{
"indices": [
{
"names": [
"*"
],
"privileges": [
"delete_index"
],
"allow_restricted_indices": true
}
]
}'
5. Create user who will delete
Once role is there, time for user who will be happy to delete .security index.
Note line 4 where assignment to role deleter is declared.
curl -k -XPOST -u elastic "https://localhost:9201/_security/user/del" -H 'Content-Type: application/json' -d'
{
"password":"123456",
"roles": ["deleter"]
}'
Please check how to create repository called
my_s3_filebase_repository
in the tutorial I created before
Elasticsearch backup & restore to S3 IPFS using docker swarm with secrets
This is required to run below command. Otherwise you will get exception that repository does not exist.
Notice that I am using feature_states – this is only way to backup system indices.
curl -k -XPUT -u elastic "https://localhost:9200/_snapshot/my_s3_filebase_repository/snapshot_20230722?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
"indices":".security-7*",
"include_global_state":"true",
"feature_states" :["security"]
}'
6. Delete .security index
Now the best part 🙂 Notice that I am using ‘del’ user for that action not ‘elastic’ anymore.
curl -k -XDELETE -u del "https://localhost:9201/.security-7"
To confirm that index is deleted you can try to run
curl -k -XGET -u elastic "https://localhost:9200/.security-7/_search"
And I said try intentionally because it maybe after Elasticsearch loose credentials info from memory you won’t be able to login. Time to recover.
7. Restore with data loss
This step will simply recreate index .security-7 and populate it with elastic user password. We did that already in previous steps but since we lost index completely let’s do this one more time.
# provide some password when asked
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';
Once it’s done you can login and use a cluster but all users created later on like ‘del’ are lost.
What about restore from backup? Yes
8. Restore from snapshot
This is good scenario. You are a hero, after few hours of outage that you created yourself now you can rescue whole system back to normal condition within seconds. Just one command
curl -k -XPOST -u elastic "https://localhost:9200/_snapshot/my_s3_filebase_repository/snapshot_20230722/_restore" -H 'Content-Type: application/json' -d'
{
"indices":"*",
"feature_states" :["security"]
}'
9. Final thoughts
It’s good to have a backup