Table of Contents
1. Introduction
You may seen data streams backing indexes in your Elasticsearch system index list. They are commonly used in Observability of logs when your data is growing fast, it’s time series and relevant during certain period so you need elegant system to discard old data.
Latest iteration of backing indices is writable index, rest are readonly.
If you will try to remove write index of data stream in Elasticsearch then you will encounter an error because it is not straightforward . Let me explain you by real example how to do it.
2. Start Elasticsearch cluster
Execute below to have one node ELK
docker network create elkai
docker run --rm \
--name elk \
--net elkai \
-e ES_JAVA_OPTS="-Xms4g -Xmx4g" \
-e cluster.deprecation_indexing.enabled=true \
-d \
-p 9200:9200 \
docker.elastic.co/elasticsearch/elasticsearch:8.15.2
Parameter ‘cluster.deprecation_indexing.enabled=true’ is key in this tutorial so please do not omit it.
2.1. Set Password for elastic User
docker exec -it elk bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
3. Create new user
To delete index user must have role with right:
"allow_restricted_indices" :true
Because built-in user called ‘elastic’ has only read-only access to system indices that name starting with dot ‘.’, therefore you have to create new user. New user need role that has right to access restricted system indices so you have to create one.
Please create role with rights to delete index:
curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/role/deleter" -H 'Content-Type: application/json' -d'
{
"indices": [
{
"names": [
"*"
],
"privileges": [
"delete_index"
],
"allow_restricted_indices": true
}
]
}'
create user del with role deleter
curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/user/del" -H 'Content-Type: application/json' -d'
{
"password":"123456",
"roles": ["deleter"]
}'
You will use that one to authorize delete operation over .ds* indices.
4. Creating Data Stream
4.1. Using deprecated functionality
You can create your own data stream but instead of that this time I want to show you how to obtain sample data stream by using deprecated functionality. After following my steps you will create .ds-.logs-deprecation data stream.
Execute below to create sample index and populate it with one document.
curl -k -XPUT -u elastic:123456 -H "Content-Type: application/json" "https://localhost:9200/my_index/_doc/1" -d '{"message": "test"}'
Now you will use Deprecated, since 7.16.0 version, parameter “ignore_throttled” which will trigger entry in both deprecation log on filesystem and additionally create data stream to store the same.
This works thanks to parameter ‘cluster.deprecation_indexing.enabled=true’ that you set when starting Elasticsearch node.
curl -k -u elastic:123456 -X GET "https://localhost:9200/_resolve/index/my_index?ignore_throttled=true&expand_wildcards=all"
4.2. Confirm data stream is present
New index should be present which you can confirm by running below query:
curl -k -XGET -u elastic:123456 "https://localhost:9200/_cat/indices/.ds-.logs-deprecation*"
example response, your will be different although with same naming convention
.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001
5. Delete index
5.1. Rollover
Like I said before you cannot delete writable index linked to data stream without any preparation. Before deleting index, you need to do manual rollover. Please run below:
curl -k -XPOST -u elastic:123456 "https://localhost:9200/.logs-deprecation.elasticsearch-default/_rollover/?pretty"
Notice that you are referring to data stream(.logs-deprecation.elasticsearch-default) in API call, not index.
example response:
{
"acknowledged" : true,
"shards_acknowledged" : true,
"old_index" : ".ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001",
"new_index" : ".ds-.logs-deprecation.elasticsearch-default-2025.01.21-000002",
"rolled_over" : true,
"dry_run" : false,
"lazy" : false,
"conditions" : { }
}
As you can see despite previously available index with suffix 000001, now there is one more with 000002. This is because old write index got replaced with new one and now read-only index *000001 can be safely deleted.
5.2. Delete call
Run below to delete *000001 index
curl -k -XDELETE -u del:123456 "https://localhost:9200/.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001"
response should be:
{"acknowledged":true}
5.3. Confirm deletion
You can do one more check to be sure index is not there anymore
curl -k -XGET -u elastic:123456 "https://localhost:9200/_cat/indices/.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001"
response:
...
index_not_found_exception
...
Index not found message means that index was successfully deleted.
6. Summary
In this knowledge article you have learned how to create special user who can delete system indexes, then you realize depracation logs functionality has its own data stream, finally you were able to delete write index of that stream.
Have a nice coding!
