Search
Close this search box.

Update Watcher internal index settings in Elasticsearch

Table of Contents

1. Introduction

Watcher is Elasticsearch build-in scheduler for executing tasks frequently. When you create first watch, in the system index called .watches is automatically created. This index stores definition of all created watches and therefore needs to be properly protected. Additionally watches are executed on shards so more shards then more places for watch to be run over there.

These are all features for fault tolerant behavior. In this article you will learn how to increase replica number by editing .watches index settings.

2. Start Elasticsearch

Watcher is included in licensed version of Elasticsearch so you have to include line with xpack.license and trial value.

2.1. Start 3 nodes

				
					docker run --rm \
--name elk01 \
-e node.name="elk01" \
-e xpack.license.self_generated.type=trial \
-d \
-p 9200:9200 \
-e ES_JAVA_OPTS="-Xms1g -Xmx1g" \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1
				
			

Once you start your first node, for better understanding of how replica setting is working please bootstrap 2 more nodes that will join the cluster

				
					token=`docker exec -it elk01 elasticsearch-create-enrollment-token -s node | tr -d '\r\n'`

docker run --rm \
--name elk02 \
-e node.name="elk02" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1

docker run --rm \
--name elk03 \
-e node.name="elk03" \
-e ENROLLMENT_TOKEN=$token \
-e xpack.license.self_generated.type=trial \
-d \
-m 1GB \
docker.elastic.co/elasticsearch/elasticsearch:8.11.1

				
			

2.2. Change password for elastic user

Then set password for elastic user so you can login to Elasticsearch

				
					docker exec -it elk01 bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
				
			

2.3. Check if everything is started properly

Confirm that cluster is green

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/_cluster/health?pretty"
				
			

 and all nodes are up and running

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/_cat/nodes?pretty&v"
				
			

3. Create dedicated user

Of course you can use elastic user to create and edit watcher but to try out segregation of duties you can have separate user that will be allowed only to perform action related to watcher.

3.1. Create role

Cluster role manage_watcher permission is required

				
					curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/role/wat" \
-H 'Content-Type: application/json' -d'
{
  "cluster": [ "manage_watcher" ]
}'
				
			

3.2 Create user

You will assign role ‘wat’ to user ‘wat’

				
					curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/user/wat" \
-H 'Content-Type: application/json' -d'
{
    "password":"123456",
    "roles": ["wat"]
}'
				
			

3.3. Test user

Below command should return empty JSON document as currently .watches index does not exist.

				
					curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
				
			

4. Create .watches index

Creating watch using watcher API will automatically create .watches index

				
					curl -k -u wat:123456 \
-XPUT "https://localhost:9200/_watcher/watch/testingwatcher?active=false" \
-H 'Content-Type: application/json' -d'
{
  "trigger": {
    "schedule": {"interval":"5m"} 
  },
    "condition": {"always": {}}
}'
				
			

5. View current settings of .watches index

Now you can display current settings of .watches index

				
					curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
				
			

response:

				
					{
  "index" : {
    "auto_expand_replicas" : "0-1",
    "number_of_replicas" : "1"
  }
}
				
			

6. Update settings

6.1. Change number of replicas

Run below command to increase replicas to 2.

				
					curl -k -u wat:123456 -XPUT "https://localhost:9200/_watcher/settings" \
-H 'Content-Type: application/json' -d'
{
   "index.auto_expand_replicas": "0-2"
}'
				
			

6.2. Check new settings

				
					curl -k -u wat:123456 -XGET "https://localhost:9200/_watcher/settings?pretty"
				
			

response:

				
					{
  "index" : {
    "auto_expand_replicas" : "0-2",
    "number_of_replicas" : "2"
  }
}
				
			

6.3. Verify if new settings affects system

For that call API to list shards of .watches index. Notice that you have to use ‘elastic’ user again because ‘wat’ user does not have permission.

				
					curl -k -u elastic:123456 \
-XGET "https://localhost:9200/_cat/shards?v&s=state:asc&index=.watches"
				
			

should return list of 3 shards, 1 primary and 2 replicas allocated on separate nodes

				
					index    shard prirep state   docs store dataset ip         node
.watches 0     r      STARTED    1 3.5kb   3.5kb 172.17.0.2 elk01
.watches 0     r      STARTED    1 3.5kb   3.5kb 172.17.0.3 elk02
.watches 0     p      STARTED    1 3.5kb   3.5kb 172.17.0.4 elk03
				
			

7. Conclusion

In this knowledge article you have learned how to start 3 node cluster, create user for watcher maintenance then you created .watches index and check it’s settings. Finally you have practice how to change number of replicas and verify it on the system.

Follow me on LinkedIn

Leave a Reply

Your email address will not be published. Required fields are marked *

Share the Post:

Enjoy Free Useful Amazing Content

Related Posts