Table of Contents
1. Introduction
One of the most important index setting is number of replicas that will be stored in Elasticsearch. If it is about your own index then no problem, you can easily control this parameter. Challenging is when you have to edit .security index settings, that moment you realize that superuser neither user with allow_restricted_indices are able to do it.
Solution is to use special API that will allow you to update these settings, although it is limited only to number of replicas and other settings are not available at the moment.
Three security indexes are editable with their parameter auto_expand_replicas:
- .security
- .security-tokens
- .security-profile
2. Start Elasticsearch cluster
Start first node
docker run --rm \
--name elk01 \
-e node.name="elk01" \
-e xpack.license.self_generated.type=trial \
-d \
-p 9200:9200 \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
And set password for elastic user
docker exec -it elk01 bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
This will create first index .security so from now you can change it’s replica number. But to see changes better to start additional nodes, because you can adjust higher than 0 number of replica.
token=`docker exec -it elk01 elasticsearch-create-enrollment-token -s node | tr -d '\r\n'`
docker run --rm \
--name elk02 \
-e node.name="elk02" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker run --rm \
--name elk03 \
-e node.name="elk03" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker run --rm \
--name elk04 \
-e node.name="elk04" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
Make sure all nodes formed the cluster
curl -k -u elastic:123456 -XGET "https://localhost:9200/_cat/nodes?pretty&v"
3. View current settings values
To see current settings run query
curl -k -u elastic:123456 -XGET "https://localhost:9200/_security/settings?pretty"
Right now you should see something like that
{
"security" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
},
"security-tokens" : { },
"security-profile" : { }
}
4. Populate data
To see entries for tokens and profile indexes you should call related APIs like creating token and profile for user.
4.1. Creating security-tokens by getting token
curl -k -u elastic:123456 -XPOST "https://localhost:9200/_security/oauth2/token" \
-H 'Content-Type: application/json' -d'
{
"grant_type" : "client_credentials"
}'
After that new section is present in settings
{
"security" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
},
"security-tokens" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
},
"security-profile" : { }
}
4.2. Creating security-profile by activating user profile
curl -k -u elastic:123456 -XPOST "https://localhost:9200/_security/profile/_activate" \
-H 'Content-Type: application/json' -d'
{
"grant_type": "password",
"username" : "elastic",
"password" : "123456"
}'
Finally last missing index is also created.
{
"security" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
},
"security-tokens" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
},
"security-profile" : {
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
}
}
4.3. Verify presence of indexes in Elasticsearch
curl -k -u elastic:123456 -XGET "https://localhost:9200/_cat/indices/.*?v"
Should give you response
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green open .security-tokens-7 EUA8sGvzTWeTjmc-JvDftA 1 1 1 0 14.5kb 7.2kb 7.2kb
green open .security-7 HgrgaTyPQ02PniZaAK_gaA 1 1 2 0 22.3kb 13kb 13kb
green open .security-profile-8 zbi2i59vRNmCWjPx8kriAw 1 1 1 0 16.8kb 8.4kb 8.4kb
5. Update settings of security indexes
You can update all of them at once or separately. You can effectively update only auto_expand_replicas value as updating number_of_replicas will not make any change although request will not throw any error.
curl -k -u elastic:123456 -XPUT "https://localhost:9200/_security/settings" \
-H 'Content-Type: application/json' -d'
{
"security" : {
"index.auto_expand_replicas" : "0-3"
},
"security-tokens" : {
"index.auto_expand_replicas" : "0-3"
},
"security-profile" : {
"index.auto_expand_replicas" : "0-3"
}
}'
When you will list shards you will noticed that right now there are 3 replicas and all assigned.
curl -k -u elastic:123456 -XGET "https://localhost:9200/_cat/shards?v&s=state:asc&index=.*"
example response
index shard prirep state docs store dataset ip node
.security-tokens-7 0 r STARTED 1 7.3kb 7.3kb 172.17.0.2 elk01
.security-tokens-7 0 r STARTED 1 7.3kb 7.3kb 172.17.0.4 elk03
.security-tokens-7 0 p STARTED 1 7.3kb 7.3kb 172.17.0.3 elk02
.security-tokens-7 0 r STARTED 1 7.3kb 7.3kb 172.17.0.5 elk04
.security-profile-8 0 r STARTED 1 8.5kb 8.5kb 172.17.0.2 elk01
.security-profile-8 0 r STARTED 1 8.5kb 8.5kb 172.17.0.4 elk03
.security-profile-8 0 p STARTED 1 8.5kb 8.5kb 172.17.0.3 elk02
.security-profile-8 0 r STARTED 1 8.5kb 8.5kb 172.17.0.5 elk04
.security-7 0 p STARTED 2 13kb 13kb 172.17.0.2 elk01
.security-7 0 r STARTED 2 9.3kb 9.3kb 172.17.0.4 elk03
.security-7 0 r STARTED 2 13kb 13kb 172.17.0.3 elk02
.security-7 0 r STARTED 2 13kb 13kb 172.17.0.5 elk04
Of course you can see right values when you query security settings with GET request.
curl -k -u elastic:123456 -XGET "https://localhost:9200/_security/settings?pretty"
response
{
"security" : {
"index" : {
"auto_expand_replicas" : "0-3",
"number_of_replicas" : "3"
}
},
"security-tokens" : {
"index" : {
"auto_expand_replicas" : "0-3",
"number_of_replicas" : "3"
}
},
"security-profile" : {
"index" : {
"auto_expand_replicas" : "0-3",
"number_of_replicas" : "3"
}
}
}
number_of_replicas is showing current value of replicas.
6. Final thoughts
In this tutorial you have learned how to effectively change number of replicas for three security settings that appear in Elasticsearch. You run code to start 4 nodes Elasticsearch cluster, then create indexes and edit settings for them.
I am sure it will be useful for you so enjoy.
Have a nice coding!