Search
Close this search box.

How to delete data stream write index

Table of Contents

1. Introduction

You may seen data streams backing indexes in your Elasticsearch system index list. They are commonly used in Observability of logs when your data is growing fast, it’s time series and relevant during certain period so you need elegant system to discard old data.

Latest iteration of backing indices is writable index, rest are readonly.

If you will try to remove write index of data stream in Elasticsearch then you will encounter an error because it is not straightforward . Let me explain you by real example how to do it.

2. Start Elasticsearch cluster

Execute below to have one node ELK

				
					docker network create elkai

docker run --rm \
--name elk \
--net elkai \
-e ES_JAVA_OPTS="-Xms4g -Xmx4g" \
-e cluster.deprecation_indexing.enabled=true \
-d \
-p 9200:9200 \
docker.elastic.co/elasticsearch/elasticsearch:8.15.2
				
			

Parameter ‘cluster.deprecation_indexing.enabled=true’ is key in this tutorial so please do not omit it.

2.1. Set Password for elastic User

				
					docker exec -it elk bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
				
			

3. Create new user

To delete index user must have role with right:

				
					"allow_restricted_indices" :true
				
			

Because built-in user called ‘elastic’ has only read-only access to system indices that name starting with dot ‘.’, therefore you have to create new user. New user need role that has right to access restricted system indices so you have to create one.

Please create role with rights to delete index:

				
					curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/role/deleter" -H 'Content-Type: application/json' -d'
{
        "indices": [
            {
                "names": [
                    "*"
                ],
                "privileges": [
                    "delete_index"
                ],
                "allow_restricted_indices": true
            }
        ]
}'
				
			

create user del with role deleter

				
					curl -k -XPOST -u elastic:123456 "https://localhost:9200/_security/user/del" -H 'Content-Type: application/json' -d'
{
    "password":"123456",
    "roles": ["deleter"]
}'
				
			

You will use that one to authorize delete operation over .ds* indices.

4. Creating Data Stream

4.1. Using deprecated functionality

You can create your own data stream but instead of that this time I want to show you how to obtain sample  data stream by using deprecated functionality. After following my steps you will create .ds-.logs-deprecation data stream.

Execute below to create sample index and populate it with one document.

				
					curl -k -XPUT -u elastic:123456 -H "Content-Type: application/json" "https://localhost:9200/my_index/_doc/1" -d '{"message": "test"}'
				
			

Now you will use Deprecated, since 7.16.0 version, parameter “ignore_throttled” which will trigger entry in both deprecation log on filesystem and additionally create data stream to store the same.

This works thanks to parameter ‘cluster.deprecation_indexing.enabled=true’ that you set when starting Elasticsearch node.

				
					curl -k -u elastic:123456 -X GET "https://localhost:9200/_resolve/index/my_index?ignore_throttled=true&expand_wildcards=all"
				
			

4.2. Confirm data stream is present

New index should be present which you can confirm by running below query:

				
					curl -k -XGET -u elastic:123456 "https://localhost:9200/_cat/indices/.ds-.logs-deprecation*"
				
			

example response, your will be different although with same naming convention

				
					.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001
				
			

5. Delete index

5.1. Rollover

Like I said before you cannot delete writable index linked to data stream without any preparation. Before deleting index, you need to do manual rollover. Please run below:

				
					curl -k -XPOST -u elastic:123456 "https://localhost:9200/.logs-deprecation.elasticsearch-default/_rollover/?pretty"
				
			

Notice that you are referring to data stream(.logs-deprecation.elasticsearch-default) in API call, not index.

example response:

				
					{
 "acknowledged" : true,
 "shards_acknowledged" : true,
 "old_index" : ".ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001",
 "new_index" : ".ds-.logs-deprecation.elasticsearch-default-2025.01.21-000002",
 "rolled_over" : true,
 "dry_run" : false,
 "lazy" : false,
 "conditions" : { }
}
				
			

As you can see despite previously available index with suffix 000001, now there is one more with 000002. This is because old write index got replaced with new one and now read-only index *000001 can be safely deleted.

5.2. Delete call

Run below to delete *000001 index

				
					curl -k -XDELETE -u del:123456 "https://localhost:9200/.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001"
				
			

response should be:

				
					{"acknowledged":true}
				
			

5.3. Confirm deletion

You can do one more check to be sure index is not there anymore

				
					curl -k -XGET -u elastic:123456 "https://localhost:9200/_cat/indices/.ds-.logs-deprecation.elasticsearch-default-2025.01.21-000001"
				
			

response:

				
					... 
index_not_found_exception 
...
				
			

Index not found message means that index was successfully deleted.

6. Summary

In this knowledge article you have learned how to create special user who can delete system indexes, then you realize depracation logs functionality has its own data stream, finally you were able to delete write index of that stream.

Have a nice coding!

Leave a Reply

Your email address will not be published. Required fields are marked *

Follow me on LinkedIn
Share the Post:

Enjoy Free Useful Amazing Content

Related Posts