Search
Close this search box.

Test painless scripts and pipelines – simulate evaluation

Test painless scripts and pipelines – simulate evaluation

Table of Contents

1. Introduction

Workflow is important for every job. How are you resolving tasks is even more important than final results in my opinion. Instead of trying your code in environment you can perform dry run and see results through simulation. One of obvious benefit is no need to delete created objects every time – cluster is clean. In this article I will show you how to execute painless scripts and how to do dry run of pipelines.

2. Start Elasticsearch

Using docker start one node cluster

				
					docker run --rm \
--name elk01 \
-d \
-e node.name="elk01" \
-p 9200:9200 \
docker.elastic.co/elasticsearch/elasticsearch:8.11.3
				
			

and set password for ‘elastic’ user

				
					docker exec -it elk01 bash -c "(mkfifo pipe1); ( (elasticsearch-reset-password -u elastic -i < pipe1) & ( echo $'y\n123456\n123456' > pipe1) );sleep 5;rm pipe1"
				
			

3. Execute painless script

Having script to decode base64 value you can evaluate it without storing script in Elasticsearch and without creating pipeline simulation.

3.1. Test context

In below example decode method is called directly over String literal:

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_scripts/painless/_execute" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "\"QXBwbGU=\".decodeBase64();"
    }
}'
				
			

+ assigning String to variable

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_scripts/painless/_execute" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "def x=\"QXBwbGU=\";x.decodeBase64();"
    }
}'
				
			

+ adding params

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_scripts/painless/_execute" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "params.name_base64.decodeBase64();",
        "params": {
            "name_base64": "QXBwbGU="
        }
    }
}'
				
			

+ adding NPE protection

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_scripts/painless/_execute" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "def src = params.name_base64;if(src == null){return '\'''\'';} def target = params.target_field; target = src.decodeBase64();",
        "params": {
            "name_base64": "QXBwbGU=",
            "target_field": "name_decoded"
        }
    }
}'
				
			

3.2. Filter context

When you using script to filter data in search then

3.2.1. Testing prototype

Define test mapping to run filter context

				
					curl -k -u elastic:123456 -XPUT "https://localhost:9200/testindex1" \
-H 'content-type: application/json' -d'
{
  "mappings": {
    "properties": {
        "name_base64": {
        "type": "keyword"
      }
    }
  }
}'
				
			

Define filtering script that answering true/false question

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_scripts/painless/_execute" \
-H 'content-type: application/json' -d'
{
  "script": {
    "source": "def src = doc[params.field];if(src == null){return false;} params.filteronvalue == src.value.decodeBase64();",
    "params": {
      "field": "name_base64",
      "filteronvalue" : "Apple"
    }
  },
  "context": "filter",
  "context_setup": {
    "index": "testindex1",
    "document": {
      "name_base64": "QXBwbGU="
    }
  }
}'
				
			

because document in context_setup has exactly value ‘Apple’ encoded in base64 format then you will get true in return

				
					{
    "result":true
}
				
			

3.2.2. Loading test data

Load test data with below command:

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/fruits/_bulk" \
-H 'content-type: application/json' -d'
{"index":{"_id":"1"}}
{"name_base64":"QXBwbGU=","color_hex":"477265656e"}
{"index":{"_id":"2"}}
{"name_base64":"QW5hbmFz","color_hex":"59656c6c6f77"}
{"index":{"_id":"3"}}
{"name_base64":"Q2hlcnJ5","color_hex":"526564"}
'
				
			

and check if data is loaded properly

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search"
				
			

3.2.3. Using tested script in query for filtering

Because script was tested and sample data loaded, now is time to query ‘fruits’ index to search for document that has value ‘Apple’ encoded in base64.

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search" \
-H 'content-type: application/json' -d'
{
    "query": {
        "bool": {
            "filter": {
                "script": {
                    "script": {
                        "source": "def src = doc[params.field];if(src == null){return false;} params.filteronvalue == src.value.decodeBase64();",
                        "params": {
                            "field": "name_base64.keyword",
                            "filteronvalue": "Apple"
                        }
                    }
                }
            }
        }
    }
}'
				
			

example response:

				
					{
    "took": 7,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.0,
        "hits": [
            {
                "_index": "fruits",
                "_id": "1",
                "_score": 0.0,
                "_source": {
                    "name_base64": "QXBwbGU=",
                    "color_hex": "477265656e"
                }
            }
        ]
    }
}
				
			

Only one document got returned that matching filtering criteria.

3.3. Field contexts

3.3.1 Testing

Before defining runtime_mapping you can test it with field_context like

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/_scripts/painless/_execute?pretty" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "def src = doc[params.field];if(src.size()==0){return ;} emit(src.value.decodeBase64())",
        "params": {
            "field": "name_base64"
        }
    },
    "context": "keyword_field",
    "context_setup": {
        "index": "testindex1",
        "document": {
            "name_base64": "QXBwbGU="
        }
    }
}'
				
			

will return:

				
					{
    "result": [
        "Apple"
    ]
}
				
			

3.3.2. Using in query

Once you are sure everything is correct with script, you can run it inside query to define runtime field

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "runtime_mappings": {
        "Fruit name": {
            "type": "keyword",
            "script": {
                "source": "def src = doc[params.field];if(src.size()==0){return ;} emit(src.value.decodeBase64())",
                "params": {
                    "field": "name_base64.keyword"
                }
            }
        }
    },
    "fields": [
        "Fruit name"
    ],
    "_source": false
}'

				
			

this will return all fruits with new field ‘Fruit name’

				
					{
    "took": 4,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 3,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "fruits",
                "_id": "1",
                "_score": 1.0,
                "fields": {
                    "Fruit name": [
                        "Apple"
                    ]
                }
            },
            {
                "_index": "fruits",
                "_id": "2",
                "_score": 1.0,
                "fields": {
                    "Fruit name": [
                        "Ananas"
                    ]
                }
            },
            {
                "_index": "fruits",
                "_id": "3",
                "_score": 1.0,
                "fields": {
                    "Fruit name": [
                        "Cherry"
                    ]
                }
            }
        ]
    }
}
				
			

3.4. Score context

3.4.1. Testing score script

You created script that will manipulate over score value and you want to test it using score context like below

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/_scripts/painless/_execute?pretty" \
-H 'content-type: application/json' -d'
{
  "script": {
    "source": "def src = doc[params.field];if(src == null){return 0.0;} if(params.filteronvalue == src.value.decodeBase64()){return 1.0}",
    "params": {
      "field": "name_base64",
      "filteronvalue" : "Apple"
    }
  },
  "context": "score",
  "context_setup": {
    "index": "testindex1",
    "document": {
      "name_base64": "QXBwbGU="
    }
  }
}'
				
			

which basically scores decoded ‘Apple’ as 1.0 and nulls as zero.

3.4.2. Using new scoring

Tested scoring script can be now used now in queries

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "query": {
        "script_score": {
            "query": {
                 "match_all": {}
            },
            "script": {
                "source": "def src = doc[params.field];if(src == null){return 0.0;} if(params.filteronvalue == src.value.decodeBase64()){return 1.0}",
                "params": {
                    "field": "name_base64.keyword",
                    "filteronvalue": "Apple"
                }
            }
        }
    }
}'
				
			

in return will give score 1.0 to document with ‘Apple’ (base64 as ‘QXBwbGU=’)

				
					{
    "took": 9,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 3,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "fruits",
                "_id": "1",
                "_score": 1.0,
                "_source": {
                    "name_base64": "QXBwbGU=",
                    "color_hex": "477265656e"
                }
            },
            {
                "_index": "fruits",
                "_id": "2",
                "_score": 0.0,
                "_source": {
                    "name_base64": "QW5hbmFz",
                    "color_hex": "59656c6c6f77"
                }
            },
            {
                "_index": "fruits",
                "_id": "3",
                "_score": 0.0,
                "_source": {
                    "name_base64": "Q2hlcnJ5",
                    "color_hex": "526564"
                }
            }
        ]
    }
}
				
			

4. Use Field API

Above scoring script and others that you see in article can be rewritten to utilize Field API. Look at below example and compare with previous – it looks simpler, isn’t it.

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "query": {
        "script_score": {
            "query": {
                 "match_all": {}
            },
            "script": {
                "source": "def src=$(params.field,null);if(params.filteronvalue == src.decodeBase64()){return 1.0} else {return 0.0}",
                "params": {
                    "field": "name_base64.keyword",
                    "filteronvalue": "Apple"
                }
            }
        }
    }
}'
				
			

no need to null checking or calling doc.containsKey(field) method.

other examples:

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "runtime_mappings": {
        "Fruit name": {
            "type": "keyword",
            "script": {
                "source": "def x=$(params.field,\"\"); emit(x.decodeBase64());",
                "params": {
                    "field": "name_base64.keyword"
                }
            }
        }
    },
    "fields": [
        "Fruit name"
    ],
    "_source": false
}'
				
			
				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "query": {
        "bool": {
            "filter": {
                "script": {
                    "script": {
                        "source": "def src = $(params.field,\"\"); params.filteronvalue == src.decodeBase64();",
                        "params": {
                            "field": "name_base64.keyword",
                            "filteronvalue": "Apple"
                        }
                    }
                }
            }
        }
    }
}'
				
			

5. Painless Debugging

For objects in Painless you can surround them with Debug.explain so you can throw exception and see details of data types

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/_scripts/painless/_execute?pretty" \
-H 'content-type: application/json' -d'
{
    "script": {
        "source": "Debug.explain(params.base64value.decodeBase64())",
        "params": {
            "base64value": "QXBwbGU="
        }
    }
}'
				
			

will return

				
					{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "painless_class" : "java.lang.String",
        "to_string" : "Apple",
        "java_class" : "java.lang.String",
        "script_stack" : [
          "Debug.explain(params.base64value.decodeBase64())",
          "                                ^---- HERE"
        ],
        "script" : "Debug.explain(params.base64value.decodeBase64())",
        "lang" : "painless",
        "position" : {
          "offset" : 32,
          "start" : 0,
          "end" : 48
        }
      }
    ],
    "type" : "script_exception",
    "reason" : "runtime error",
    "painless_class" : "java.lang.String",
    "to_string" : "Apple",
    "java_class" : "java.lang.String",
    "script_stack" : [
      "Debug.explain(params.base64value.decodeBase64())",
      "                                ^---- HERE"
    ],
    "script" : "Debug.explain(params.base64value.decodeBase64())",
    "lang" : "painless",
    "position" : {
      "offset" : 32,
      "start" : 0,
      "end" : 48
    },
    "caused_by" : {
      "type" : "painless_explain_error",
      "reason" : null
    }
  },
  "status" : 400
}
				
			

and this call

				
					curl -k -u elastic:123456 -XGET "https://localhost:9200/fruits/_search?pretty" \
-H 'content-type: application/json' -d'
{
    "query": {
        "bool": {
            "filter": {
                "script": {
                    "script": {
                        "source": "Debug.explain($(params.field,\"\"))",
                        "params": {
                            "field": "name_base64.keyword",
                            "filteronvalue": "Apple"
                        }
                    }
                }
            }
        }
    }
}'
				
			

will return

				
					{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "painless_class" : "java.lang.String",
        "to_string" : "QXBwbGU=",
        "java_class" : "java.lang.String",
        "script_stack" : [
          "Debug.explain($(params.field,\"\"))",
          "              ^---- HERE"
        ],
        "script" : "Debug.explain($(params.field,\"\"))",
        "lang" : "painless",
        "position" : {
          "offset" : 14,
          "start" : 0,
          "end" : 33
        }
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "fruits",
        "node" : "tYPOnT9aQGupk-TOpTfY3Q",
        "reason" : {
          "type" : "script_exception",
          "reason" : "runtime error",
          "painless_class" : "java.lang.String",
          "to_string" : "QXBwbGU=",
          "java_class" : "java.lang.String",
          "script_stack" : [
            "Debug.explain($(params.field,\"\"))",
            "              ^---- HERE"
          ],
          "script" : "Debug.explain($(params.field,\"\"))",
          "lang" : "painless",
          "position" : {
            "offset" : 14,
            "start" : 0,
            "end" : 33
          },
          "caused_by" : {
            "type" : "painless_explain_error",
            "reason" : null
          }
        }
      }
    ]
  },
  "status" : 400
}
				
			

6. Dry run of pipeline

Your pipeline you can check before use with ‘_simulate’. Through docs section your values are accessible in pipeline

				
					curl -k -u elastic:123456 -XPOST "https://localhost:9200/_ingest/pipeline/_simulate?pretty" \
-H 'content-type: application/json' -d'
{
    "pipeline": {
        "processors": [
            {
                "script": {
                    "description": "Decode base64",
                    "lang": "painless",
                    "source": "def src=ctx[params['\''field'\'']];if(src == null){return;}def target=params['\''target_field'\'']; ctx[target]=src.decodeBase64();",
                    "params": {
                        "field": "name_base64",
                        "target_field": "name"
                    }
                }
            },
            {
                "script": {
                    "description": "Decode hex",
                    "lang": "painless",
                    "source": "def src=ctx[params['\''field'\'']];if(src == null){return;}def target=params['\''target_field'\''];StringBuilder sb = new StringBuilder();for (int i = 0; i < src.length(); i+=2) { String byteStr = src.substring(i, i + 2);char byteChar = (char) Integer.parseInt(byteStr, 16);sb.append(byteChar) } ctx[target] = sb.toString();",
                    "params": {
                        "field": "color_hex",
                        "target_field": "color"
                    }
                }
            }
        ]
    },
    "docs": [
        {
            "_source": {
                "name_base64": "QXBwbGU=",
                "color_hex": "477265656e"
            }
        }
    ]
}'
				
			

as an output you will get decoded strings

				
					{
    "docs": [
        {
            "doc": {
                "_index": "_index",
                "_version": "-3",
                "_id": "_id",
                "_source": {
                    "color_hex": "477265656e",
                    "name": "Apple",
                    "name_base64": "QXBwbGU=",
                    "color": "Green"
                },
                "_ingest": {
                    "timestamp": "2023-12-31T13:33:33.698900588Z"
                }
            }
        }
    ]
}
				
			

This tested pipeline can be used like in article about stored scripts.

7. Summary

In this article you have discovered how to test painless script before using it as part of pipeline or as runtime script. This let you improve your workflow so you can avoid runtime exceptions and speed up your work.

Have a nice coding!

Follow me on LinkedIn

One Comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Share the Post:

Enjoy Free Useful Amazing Content

Related Posts