Search
Close this search box.

Deleted .security Elasticsearch index? Cannot login? Watch and recover!

Table of Contents

1. Introduction

It might happen one day that you wake up in the morning and realize that .security index that you deleted last night on production cluster is still not there and you need to face reality. Not much to worry if you have working backup that was done with SUCCESS status not PARTIAL (because nobody check it). Worse if you do not have a backup because then you have to recreate roles and users. If you use LDAP for authentication you still need to recreate mapping between AD groups and roles. Let’s start.

2. Start Elasticsearch

Because I want to show you also how to restore it from backup please configure working elasticsearch instance as it was described in my article
Elasticsearch backup & restore to S3 IPFS using docker swarm with secrets

3. Populate .security-7 index

If you check .security-7 index it will only have information about elastic built-in superuser. I want to show you case with more data.

To populate security index

				
					# execute commands on container Elasticsearch
dockerContainerID=`docker ps --filter name=elastic -q`;

docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto --batch --url https://localhost:9200/';

# setup password again to previous
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';

curl -k -XGET -u elastic "https://localhost:9200/.security-7/_search"
				
			

4. Create role with rights to delete system index

superuser elastic does not have permission to delete system indexes started their names with ‘.'(dot). Therefore it is necessary to create new role and new user that will have that role. Notice line 9 and 11 – I set minimal privilege just to delete index and I am giving permission to restricted indices.

				
					curl -k -XPOST -u elastic "https://localhost:9201/_security/role/deleter" -H 'Content-Type: application/json' -d'
{
        "indices": [
            {
                "names": [
                    "*"
                ],
                "privileges": [
                    "delete_index"
                ],
                "allow_restricted_indices": true
            }
        ]
}'

				
			

5. Create user who will delete

Once role is there, time for user who will be happy to delete .security index.

Note line 4 where assignment to role deleter is declared.

				
					curl -k -XPOST -u elastic "https://localhost:9201/_security/user/del" -H 'Content-Type: application/json' -d'
{
    "password":"123456",
    "roles": ["deleter"]
}'
				
			

Please check how to create repository called
my_s3_filebase_repository
in the tutorial I created before
Elasticsearch backup & restore to S3 IPFS using docker swarm with secrets

This is required to run below command. Otherwise you will get exception that repository does not exist.

Notice that I am using feature_states – this is only way to backup system indices.

				
					curl -k -XPUT -u elastic "https://localhost:9200/_snapshot/my_s3_filebase_repository/snapshot_20230722?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
    "indices":".security-7*",
    "include_global_state":"true",
    "feature_states" :["security"]
}'
				
			

6. Delete .security index

Now the best part 🙂 Notice that I am using ‘del’ user for that action not ‘elastic’ anymore.

				
					curl -k -XDELETE -u del "https://localhost:9201/.security-7"
				
			

To confirm that index is deleted you can try to run

				
					curl -k -XGET -u elastic "https://localhost:9200/.security-7/_search"
				
			

And I said try intentionally because it maybe after Elasticsearch loose credentials info from memory you won’t be able to login. Time to recover.

7. Restore with data loss

This step will simply recreate index .security-7 and populate it with elastic user password. We did that already in previous steps but since we lost index completely let’s do this one more time.

				
					# provide some password when asked
docker exec -it $dockerContainerID sh -c '/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic -i --batch --url https://localhost:9200';
				
			

Once it’s done you can login and use a cluster but all users created later on like ‘del’ are lost.

What about restore from backup? Yes

8. Restore from snapshot

This is good scenario. You are a hero, after few hours of outage that you created yourself now you can rescue whole system back to normal condition within seconds. Just one command

				
					curl -k -XPOST -u elastic "https://localhost:9200/_snapshot/my_s3_filebase_repository/snapshot_20230722/_restore" -H 'Content-Type: application/json' -d'
{
    "indices":"*",
    "feature_states" :["security"]
}'

				
			

9. Final thoughts

It’s good to have a backup

Follow me on LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *

Share the Post:

Enjoy Free Useful Amazing Content

Related Posts