I use the demo Operate instance as a docker swarm service with ELK stack version OSS 7.8.0.
image: docker.elastic.co/elasticsearch/elasticsearch-oss:7.8.0
It works fine, except for a recurrent error from the Operate container rapidly filling up the logs (1200/minute!):
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/operate-operation-0.23.0_/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]
It seems related to Elasticsearch date-time format:
date_time or strict_date_time
A formatter that combines a full date and time, separated by a T : yyyy-MM-dd'T'HH:mm:ss.SSSZZ .
Kibana json error:
{
"index": "operate-operation-0.23.0_",
"reason": {
"reason": "failed to parse date field [2020-07-21T09:36:03.337+0000] with format [date_time]: [failed to parse date field [2020-07-21T09:36:03.337+0000] with format [date_time]]",
"type": "parse_exception",
"caused_by": {
"reason": "failed to parse date field [2020-07-21T09:36:03.337+0000] with format [date_time]",
"type": "illegal_argument_exception",
"caused_by": {
"reason": "Text '2020-07-21T09:36:03.337+0000' could not be parsed at index 23",
"type": "date_time_parse_exception"
}
}
},
"shard": 0,
"node": "CyOZxJJERUWNREpmaVX08g"
}
The complete container log error:
2020-07-20 08:54:26.363 ERROR 6 --- [ Thread-1] o.c.o.w.z.o.OperationExecutor : Something went wrong, while executing operations batch. Will be retried. Underlying exception:
2020-07-20 08:54:26.364 ERROR 6 --- [ Thread-1] o.c.o.w.z.o.OperationExecutor : Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2030) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.camunda.operate.webapp.es.reader.OperationReader.acquireOperations(OperationReader.java:90) ~[classes!/:?]
at org.camunda.operate.webapp.es.writer.BatchOperationWriter.lockBatch(BatchOperationWriter.java:104) ~[classes!/:?]
at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.executeOneBatch(OperationExecutor.java:95) ~[classes!/:?]
at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.run(OperationExecutor.java:71) [classes!/:?]
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/operate-operation-0.23.0_/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"operate-operation-0.23.0_","node":"CyOZxJJERUWNREpmaVX08g","reason":{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2020-07-20T08:54:26.351+0000' could not be parsed at index 23"}}}}]},"status":400}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.camunda.operate.webapp.es.reader.OperationReader.acquireOperations(OperationReader.java:90) ~[classes!/:?]
at org.camunda.operate.webapp.es.writer.BatchOperationWriter.lockBatch(BatchOperationWriter.java:104) ~[classes!/:?]
at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.executeOneBatch(OperationExecutor.java:95) ~[classes!/:?]
at org.camunda.operate.webapp.zeebe.operation.OperationExecutor.run(OperationExecutor.java:71) [classes!/:?]
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://elasticsearch:9200], URI [/operate-operation-0.23.0_/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=true&expand_wildcards=open&allow_no_indices=true&ignore_throttled=false&search_type=query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"operate-operation-0.23.0_","node":"CyOZxJJERUWNREpmaVX08g","reason":{"type":"parse_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]: [failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]]","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-07-20T08:54:26.351+0000] with format [date_time]","caused_by":{"type":"date_time_parse_exception","reason":"Text '2020-07-20T08:54:26.351+0000' could not be parsed at index 23"}}}}]},"status":400}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122) ~[httpcore-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[httpcore-nio-4.4.13.jar!/:4.4.13]
at java.lang.Thread.run(Unknown Source) ~[?:?]
