Table of Contents
1. Introduction
Watcher is Elasticsearch build-in scheduler for executing tasks frequently. When you create first watch, in the system index called .watches is automatically created. This index stores definition of all created watches and therefore needs to be properly protected. Additionally watches are executed on shards so more shards then more places for watch to be run over there.
These are all features for fault tolerant behavior. In this article you will learn how to increase replica number by editing .watches index settings.
2. Start Elasticsearch
Watcher is included in licensed version of Elasticsearch so you have to include line with xpack.license and trial value.
2.1. Start 3 nodes
docker run --rm \
--name elk01 \
-e node.name="elk01" \
-e xpack.license.self_generated.type=trial \
-d \
-p 9200:9200 \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
Once you start your first node, for better understanding of how replica setting is working please bootstrap 2 more nodes that will join the cluster
token=`docker exec -it elk01 elasticsearch-create-enrollment-token -s node | tr -d '\r\n'`
docker run --rm \
--name elk02 \
-e node.name="elk02" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
docker run --rm \
--name elk03 \
-e node.name="elk03" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
2.2. Change password for elastic user
Then set password for elastic user so you can login to Elasticsearch
docker exec -it elk01 bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
2.3. Check if everything is started properly
Confirm that cluster is green
curl -k -u elastic:123456 -XGET "https://localhost:9200/_cluster/health?pretty"
and all nodes are up and running
curl -k -u elastic:123456 -XGET "https://localhost:9200/_cat/nodes?pretty&v"
3. Create dedicated user
Of course you can use elastic user to create and edit watcher but to try out segregation of duties you can have separate user that will be allowed only to perform action related to watcher.
3.1. Create role
Cluster role manage_watcher permission is required
curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/role/wat" \
-H 'Content-Type: application/json' -d'
{
"cluster": [ "manage_watcher" ]
}'
3.2 Create user
You will assign role ‘wat’ to user ‘wat’
curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/user/wat" \
-H 'Content-Type: application/json' -d'
{
"password":"123456",
"roles": ["wat"]
}'
3.3. Test user
Below command should return empty JSON document as currently .watches index does not exist.
curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
4. Create .watches index
Creating watch using watcher API will automatically create .watches index
curl -k -u wat:123456 \
-XPUT "https://localhost:9200/_watcher/watch/testingwatcher?active=false" \
-H 'Content-Type: application/json' -d'
{
"trigger": {
"schedule": {"interval":"5m"}
},
"condition": {"always": {}}
}'
5. View current settings of .watches index
Now you can display current settings of .watches index
curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
response:
{
"index" : {
"auto_expand_replicas" : "0-1",
"number_of_replicas" : "1"
}
}
6. Update settings
6.1. Change number of replicas
Run below command to increase replicas to 2.
curl -k -u wat:123456 -XPUT "https://localhost:9200/_watcher/settings" \
-H 'Content-Type: application/json' -d'
{
"index.auto_expand_replicas": "0-2"
}'
6.2. Check new settings
curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
response:
{
"index" : {
"auto_expand_replicas" : "0-2",
"number_of_replicas" : "2"
}
}
6.3. Verify if new settings affects system
For that call API to list shards of .watches index. Notice that you have to use ‘elastic’ user again because ‘wat’ user does not have permission.
curl -k -u elastic:123456 \
-XGET "https://localhost:9200/_cat/shards?v&s=state:asc&index=.watches"
should return list of 3 shards, 1 primary and 2 replicas allocated on separate nodes
index shard prirep state docs store dataset ip node
.watches 0 r STARTED 1 3.5kb 3.5kb 172.17.0.2 elk01
.watches 0 r STARTED 1 3.5kb 3.5kb 172.17.0.3 elk02
.watches 0 p STARTED 1 3.5kb 3.5kb 172.17.0.4 elk03
7. Conclusion
In this knowledge article you have learned how to start 3 node cluster, create user for watcher maintenance then you created .watches index and check it’s settings. Finally you have practice how to change number of replicas and verify it on the system.
