Table of Contents
1. Introduction
Maintenance of cluster with hundreds of nodes can be challenging. Imagine some nodes stopped working, shards automatically re-balance and you do not noticed even that something happen as data is still available. Imagine your cluster is under performance and you cannot track it on time. To solve that automation comes into picture. Alerting implemented in Kibana helps you setup automated checks over system health, queries to data and whatever you can imagine. There is 3rd party software integration in place so you can send alerts to email addresses, Jira and and webhook with custom service behind. In this tutorial I will show you how to integrate it with Microsoft Teams so you can send alerts into channel and all admins being channel members can see it and react. That’s powerful, especially in corporate environment.
2. Start Microsoft Teams
To follow tutorial you need to have configured Teams with channel capabilities. Based on table this feature is available in Microsoft 365 Business Basic license so you can start your trial with that. After setup process start window with teams and you can create your channel.
later on edit connectors for your channel to create webhook
search for Incoming Webhook connector, add it and configure by setting up name and logo. After that copy webhook URL to be later used in Kibana and close window. You will see notification in your channel that webhook is added.
3. Start Elasticsearch and Kibana
To use Teams connector you need to start Elasticsearch with license or trial. In this tutorial I am using trial option. Environment variable xpack.license.self_generated.type set as trial did the job.
docker network create kibana
docker run --rm \
--name elk \
--net kibana \
-p 9200:9200 \
-e xpack.license.self_generated.type=trial \
docker.elastic.co/elasticsearch/elasticsearch:8.10.1
For kibana you have to setup environment variable XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY
To generate value you can use built-in kibana-encryption-keys or any hash generator. Required length is over 32 bytes(chars).
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=`echo "toughcoding" | shasum -a256`
# or use
# docker exec -it kibana /usr/share/kibana/bin/kibana-encryption-keys generate
docker run --rm \
--name kibana \
--net kibana \
-e XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=$XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY \
-p 5601:5601 \
docker.elastic.co/kibana/kibana:8.10.1
reset password for elastic user and generate enrollment token to connect Elasticsearch and Kibana
# reset password
docker exec -it elk /usr/share/elasticsearch/bin/elasticsearch-reset-password -i -u elastic
# apply for new enrollment token
docker exec -it elk /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
open web browser at localhost:5601?code=123456, where code will be displayed in stdout of kibana process, and paste enrollment token.
Kibana is started.
4. Create alert
Login into Kibana with elastic username. Then in main menu choose Stack Monitoring and enable self-monitoring.
Next in right upper corner click Alerts and rules and create default rules. After that click on manage rules.
Choose Cluster health and edit it. Next scroll down to the list of connectors and choose Microsoft Teams.
Choose create connector and configure it by placing your webhook url from Teams that you saved before.
save changes. Now time to test it.
5. Test alert
Alert will be active when Elasticsearch cluster health will change to yellow or red. To simulate that create index with replicas. Because replica cannot be on the same node as primary shard therefore it cannot be assigned to any node on one-node cluster like in our case. Unassigned shards will make cluster yellow.
curl -k -u elastic -XPUT "https://localhost:9200/testing" -H 'content-type: application/json' -d'
{
"index" : {
"number_of_replicas" : 100
},
"mappings": {
"properties": {
"name": {
"type": "text"
}
}
}
}'
Now check your teams channel. There will be new message
[12:49 PM] elastic-toughcoding
Cluster health alert is firing for docker-cluster. Current health is yellow. Allocate missing replica shards.
This means everything is working as expected and configured alert about cluster health is coming into channel on teams. You can edit other monitoring alerts to send messages into teams using same webhook.
6. Conclusion
In this tutorial you have learned how to setup alerting system to send messages into Teams. This will be useful especially in corporate environment. Please share your thoughts in comment section under video or here.
Have a nice coding!
