Table of Contents
1. Introduction
Docker in swarm mode brings secrets that allow to store sensitive information like passwords. Elasticsearch on another hand has keystore as storage for sensitive information. These two guys can cooperate together.
When starting Elasticsearch container user need to provide somehow password for elastic user, additionally it will be more secure to protect keystore from access with password. Providing these values with open text form as environment variable is not secure for sure as simple docker inspect command can reveal secret, instead use docker secrets to mount values inside container and read them when needed.
2. Official Docker Elasticsearch Image
Official Docker Image for Elasticsearch comes with two environmental variables that controls initial Elasticearch password named bootstrap.password and variable that allow Elastic process to read from secured keystore. Their names are respectively ELASTIC_PASSWORD and KEYSTORE_PASSWORD. Moreover there are versions for files with _FILE suffix ELASTIC_PASSWORD_FILE and KEYSTORE_PASSWORD_FILE serving same purpose but reading password candidates from files. This giving opportunity to utilize docker secrets as these are mounted in container as regular file.
3. Limitations of Official Image
As of now version docker.elastic.co/elasticsearch/elasticsearch:8.9.1 contain bug that makes bootstrap.password as empty string if you start Elasticsearch container with password protected keystore and ELASTIC_PASSWORD variable set. Reason for that is line of code responsible for insertion into keystore.
(echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
You can check this source yourself. Despite of this bug, current functionality do not allow to create password protected keystore using mentioned environmental variable KEYSTORE_PASSWORD_FILE as input. Everytime keystore will be only obfuscated which does not matter because you can open it using elasticsearch-keystore command. That limitation can be fixed by editing docker-entrypoint.sh script.
In this first part of tutorial I will focus on workaround how to start Elasticsearch with bootstrap password and later on run command that will take password from docker secret location and secure keystore.
4. Secure keystore manually
To password protect keystore simply run
elasticsearch-keystore passwd
and you will be asked for password. This require human interaction with terminal and that’s not what I aim for.
5. Secure keystore with scripting
When running setup with scripts human can sleep and machine to the work.
I am using Named pipe as FIFO messaging bus. Whatever is cast from secret file is waiting for pickup by bash process and therefore I can script it.
Look at below one-liner:
(mkfifo pipe1); ( (elasticsearch-keystore passwd < pipe1) & ( cat /run/secrets/keystore_password > pipe1) );sleep 5;rm pipe1
Now if you want to change key value in such storage you can use same technique.
6. Prepare docker secrets
Time for action. Start docker swarm node and create secrets. When you typing values for secrets you finish work by Ctrl+D combination or ^D
For bootstrap password you type value once, for keystore_password you type twice separated by new line character, this is because keystore password will be provided twice to elasticsearch-keystore shell program due to password confirmation needed.
docker swarm init
docker secret create bootstrap_password -
docker secret create keystore_password -
7. Start Elasticsearch container with non-secured keystore
Elasticsearch will start now with non-secured keystore with bootstrap.password equal to contant of bootstrap_password docker secret.
docker service create \
--name elastic \
--publish published=9200,target=9200 \
--secret source=bootstrap_password,target=bootstrap_password,uid=1000,gid=1001,mode=0400 \
--secret source=keystore_password,target=keystore_password,uid=1000,gid=1001,mode=0600 \
--env ELASTIC_PASSWORD_FILE=/run/secrets/bootstrap_password \
--env KEYSTORE_PASSWORD_FILE=/run/secrets/keystore_password \
--mount type=volume,src=elkdata,dst=/usr/share/elasticsearch/data \
--mount type=volume,src=elkconf,dst=/usr/share/elasticsearch/config \
docker.elastic.co/elasticsearch/elasticsearch:8.9.1
8. Secure Elasticsearch Keystore
For security you want keystore to be secured. By running below command one-liner is executed inside Elasticsearch container and makes keystore be password protected with password equal to keystore_password docker secret.
docker exec -it `docker ps --filter name=elastic -q` bash -c "(mkfifo pipe1); ( (elasticsearch-keystore passwd < pipe1) & ( cat /run/secrets/keystore_password > pipe1) );sleep 5;rm pipe1"
9. Restart Elasticsearch
To confirm that now Elasticsearch node can work with password protected keystore you will restart service. Elasticsearch process should get password via ‘<<<‘ known as here-string (check sources to know). To restart trigger update of docker service:
docker service update --force elastic
Service will start successfully and Elasticsearch will have green status. This is working!!!
10. Conclusions and next steps
You may noticed already that instead of running these commands I could prepare my own docker image with slightly different docker entrypoint script that will handle keystore password variable properly to let keystore be protected from the begining. This I will do in second part of this tutorial.
Stay tuned and have a nice coding!
