Operate stopped following zeebe's exported data to Elasticsearch

Camunda 8 Topics Discussion & Questions
1

We run Operate in a development environment, and from some point, it stopped showing instances that exist in elasticsearch exported data.

When it starts there are messages like:
Latest loaded position for alias [workflow-instance] and partitionId [3]: ImportPositionEntity{aliasName=‘workflow-instance’, partitionId=3, position=137441769728, indexName=‘zeebe-record_workflow-instance_0.23.4_2020-07-12’}

But there are newer indices in ES.

Also, the Operate indices in ES are not created

Any idea how to re-sync?

2

Hi,
Thanks for your message.

Can you give me more information? It would be useful if you can include logging messages from start until importing starts. It looks like:

    020-07-21 14:23:04.125  WARN 60362 --- [           main] o.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
    2020-07-21 14:23:04.208  INFO 60362 --- [           main] o.c.o.ImportModuleConfiguration          : Starting module: importer
    2020-07-21 14:23:04.209  INFO 60362 --- [           main] o.c.o.ArchiverModuleConfiguration        : Starting module: archiver
    2020-07-21 14:23:04.286  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService
    2020-07-21 14:23:04.287  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService 'operationsThreadPoolExecutor'
    2020-07-21 14:23:04.288  INFO 60362 --- [           main] o.c.o.w.StartupBean                      : Operate Version: 0.25.0-snapshot
    2020-07-21 14:23:04.288  INFO 60362 --- [           main] o.c.o.w.StartupBean                      : INIT: Create users in elasticsearch if not exists ...
    2020-07-21 14:23:04.318  INFO 60362 --- [           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for username demo
    2020-07-21 14:23:04.442  INFO 60362 --- [           main] .o.w.s.e.ElasticSearchUserDetailsService : Create user in ElasticSearch for username act
    2020-07-21 14:23:04.533  INFO 60362 --- [           main] o.c.o.w.StartupBean                      : INIT: Start operation executor...
    2020-07-21 14:23:04.533  INFO 60362 --- [           main] o.c.o.w.StartupBean                      : INIT: DONE
    2020-07-21 14:23:05.163  INFO 60362 --- [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 5 endpoint(s) beneath base path '/actuator'
    2020-07-21 14:23:05.291  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService
    2020-07-21 14:23:05.293  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService 'importThreadPoolExecutor'
    2020-07-21 14:23:05.295  INFO 60362 --- [     Thread-514] o.c.o.z.ZeebeImporter                    : INIT: Start importing data...
    2020-07-21 14:23:05.297  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService
    2020-07-21 14:23:05.299  INFO 60362 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'archiverThreadPoolExecutor'
    2020-07-21 14:23:05.299  INFO 60362 --- [           main] o.c.o.a.Archiver                         : INIT: Start archiving data...
    2020-07-21 14:23:05.413  INFO 60362 --- [     Thread-514] o.c.o.z.RecordsReaderHolder              : Starting import for partitions: [1, 2]
    2020-07-21 14:23:05.414  INFO 60362 --- [           main] o.c.o.a.Archiver                         : Starting archiver for partitions: [1, 2]
    2020-07-21 14:23:06.118  INFO 60362 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''
    2020-07-21 14:23:06.825  INFO 60362 --- [           main] o.c.o.Application                        : Started Application in 10.232 seconds (JVM running for 11.114)

But there are newer indices in ES.

Do you mean there are indices with a newer date suffix or without date suffix?

Also, the Operate indices in ES are not created

Do you mean there are no indices that start with operate- ?

These information might help.
Thank you.

3

Hi @ralfpuchert,

Thanks.
Regarding ES indices, there are new indices with new data suffix for every day but ONLY for the zeebe exporter data (zeebe-record-)
The Operate indices (operate-) are not created or updated anymore.

See below the Operate start log.
Thanks!

Yoav


2020-07-21T14:25:32.285921585Z 
2020-07-21T14:25:32.285971404Z This Operate Trial Version is for testing and non-production use only.
2020-07-21T14:25:32.285975518Z The General Terms and Conditions for the Operate Trial Version are available here: https://zeebe.io/legal/operate-evaluation-license/
2020-07-21T14:25:32.288237328Z 
2020-07-21T14:25:32.288254449Z Login with the credentials:
2020-07-21T14:25:32.288258926Z   User:     demo
2020-07-21T14:25:32.288262996Z   Password: demo
2020-07-21T14:25:32.288266878Z 
2020-07-21T14:25:35.752339757Z    ___   _   __  __ _   _ _  _ ___   _      ___  ___ ___ ___    _ _____ ___ 
2020-07-21T14:25:35.752363782Z   / __| /_\ |  \/  | | | | \| |   \ /_\    / _ \| _ \ __| _ \  /_\_   _| __|
2020-07-21T14:25:35.752367423Z  | (__ / _ \| |\/| | |_| | .` | |) / _ \  | (_) |  _/ _||   / / _ \| | | _| 
2020-07-21T14:25:35.752370298Z   \___/_/ \_\_|  |_|\___/|_|\_|___/_/ \_\  \___/|_| |___|_|_\/_/ \_\_| |___|
2020-07-21T14:25:35.752373304Z                                                                             
2020-07-21T14:25:35.752375643Z 							   0.23.0
2020-07-21T14:25:35.752378416Z 							  
2020-07-21T14:25:35.752380672Z 
2020-07-21T14:25:35.954657641Z 2020-07-21 14:25:35.949  INFO 7 --- [           main] o.c.o.Application                        : Starting Application on bald-beetle-zeebe-operate-786bb59b96-8j7d7 with PID 7 (/usr/local/operate/lib/camunda-operate-webapp-0.23.0-exec.jar started by root in /usr/local/operate)
2020-07-21T14:25:35.95777689Z 2020-07-21 14:25:35.957 DEBUG 7 --- [           main] o.c.o.Application                        : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-21T14:25:35.958339098Z 2020-07-21 14:25:35.958  INFO 7 --- [           main] o.c.o.Application                        : The following profiles are active: auth
2020-07-21T14:25:39.782822471Z 2020-07-21 14:25:39.782  INFO 7 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2020-07-21T14:25:39.844276917Z 2020-07-21 14:25:39.844  INFO 7 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
2020-07-21T14:25:39.845713209Z 2020-07-21 14:25:39.845  INFO 7 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2020-07-21T14:25:39.846920288Z 2020-07-21 14:25:39.846  INFO 7 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-21T14:25:39.974717738Z 2020-07-21 14:25:39.974  INFO 7 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-21T14:25:39.974826195Z 2020-07-21 14:25:39.974  INFO 7 --- [           main] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 3916 ms
2020-07-21T14:25:41.359561694Z 2020-07-21 14:25:41.358  INFO 7 --- [           main] o.c.o.WebappModuleConfiguration          : Starting module: webapp
2020-07-21T14:25:41.5427001Z 2020-07-21 14:25:41.542 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-21T14:25:42.28160525Z 2020-07-21 14:25:42.281  WARN 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
2020-07-21T14:25:42.374272189Z 2020-07-21 14:25:42.374  INFO 7 --- [           main] o.c.o.e.ElasticsearchSchemaManager       : Elasticsearch schema won't be created, it either already exist, or schema creation is disabled in configuration.
2020-07-21T14:25:42.777703888Z 2020-07-21 14:25:42.777  INFO 7 --- [           main] o.c.o.ImportModuleConfiguration          : Starting module: importer
2020-07-21T14:25:42.778111498Z 2020-07-21 14:25:42.777  INFO 7 --- [           main] o.c.o.ArchiverModuleConfiguration        : Starting module: archiver
2020-07-21T14:25:43.154169076Z 2020-07-21 14:25:43.153  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService
2020-07-21T14:25:43.155010807Z 2020-07-21 14:25:43.154  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService 'operationsThreadPoolExecutor'
2020-07-21T14:25:43.157125519Z 2020-07-21 14:25:43.156 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-21T14:25:43.389360072Z 2020-07-21 14:25:43.389  WARN 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Elasticsearch cluster is not accessible
2020-07-21T14:25:43.453511687Z 2020-07-21 14:25:43.453 DEBUG 7 --- [           main] o.c.o.Application                        : Create Data generator stub
2020-07-21T14:25:43.455792651Z 2020-07-21 14:25:43.455  INFO 7 --- [           main] o.c.o.w.StartupBean                      : Operate Version: 0.23.0
2020-07-21T14:25:43.455859537Z 2020-07-21 14:25:43.455  INFO 7 --- [           main] o.c.o.w.StartupBean                      : INIT: Create users in elasticsearch if not exists ...
2020-07-21T14:25:43.678632638Z 2020-07-21 14:25:43.678 DEBUG 7 --- [           main] o.c.o.w.StartupBean                      : INIT: Generate demo data...
2020-07-21T14:25:43.678664815Z 2020-07-21 14:25:43.678  INFO 7 --- [           main] o.c.o.w.StartupBean                      : INIT: Start operation executor...
2020-07-21T14:25:43.67894829Z 2020-07-21 14:25:43.678  INFO 7 --- [           main] o.c.o.w.StartupBean                      : INIT: DONE
2020-07-21T14:25:43.772342026Z 2020-07-21 14:25:43.771 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:43.876138971Z 2020-07-21 14:25:43.875  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService
2020-07-21T14:25:43.876625303Z 2020-07-21 14:25:43.876  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskExecutor           : Initializing ExecutorService 'importThreadPoolExecutor'
2020-07-21T14:25:43.9431617Z 2020-07-21 14:25:43.942  INFO 7 --- [       Thread-1] o.c.o.z.ZeebeImporter                    : INIT: Start importing data...
2020-07-21T14:25:43.943392819Z 2020-07-21 14:25:43.943 DEBUG 7 --- [       Thread-1] o.c.o.z.PartitionHolder                  : Requesting partition ids from Zeebe client
2020-07-21T14:25:44.155702802Z 2020-07-21 14:25:44.155  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService
2020-07-21T14:25:44.157101579Z 2020-07-21 14:25:44.156  INFO 7 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'archiverThreadPoolExecutor'
2020-07-21T14:25:44.157410681Z 2020-07-21 14:25:44.157  INFO 7 --- [           main] o.c.o.a.Archiver                         : INIT: Start archiving data...
2020-07-21T14:25:44.157489333Z 2020-07-21 14:25:44.157 DEBUG 7 --- [           main] o.c.o.z.PartitionHolder                  : Requesting partition ids from Zeebe client
2020-07-21T14:25:44.642537005Z 2020-07-21 14:25:44.642  INFO 7 --- [           main] o.c.o.a.Archiver                         : Starting archiver for partitions: [1, 2, 3]
2020-07-21T14:25:44.642561508Z 2020-07-21 14:25:44.642  INFO 7 --- [       Thread-1] o.c.o.z.RecordsReaderHolder              : Starting import for partitions: [1, 2, 3]
2020-07-21T14:25:44.751205158Z 2020-07-21 14:25:44.750 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Finished workflow instances for archiving request: 
2020-07-21T14:25:44.751236301Z {
2020-07-21T14:25:44.751239309Z   "constant_score" : {
2020-07-21T14:25:44.751241787Z     "filter" : {
2020-07-21T14:25:44.751244159Z       "bool" : {
2020-07-21T14:25:44.751246681Z         "must" : [
2020-07-21T14:25:44.751248966Z           {
2020-07-21T14:25:44.751251167Z             "range" : {
2020-07-21T14:25:44.751253566Z               "endDate" : {
2020-07-21T14:25:44.751255915Z                 "from" : null,
2020-07-21T14:25:44.75125819Z                 "to" : "now-1h",
2020-07-21T14:25:44.75126061Z                 "include_lower" : true,
2020-07-21T14:25:44.751262927Z                 "include_upper" : true,
2020-07-21T14:25:44.75126528Z                 "boost" : 1.0
2020-07-21T14:25:44.751267515Z               }
2020-07-21T14:25:44.751269629Z             }
2020-07-21T14:25:44.751271745Z           },
2020-07-21T14:25:44.751273927Z           {
2020-07-21T14:25:44.751276037Z             "term" : {
2020-07-21T14:25:44.751278275Z               "joinRelation" : {
2020-07-21T14:25:44.751280483Z                 "value" : "workflowInstance",
2020-07-21T14:25:44.75128288Z                 "boost" : 1.0
2020-07-21T14:25:44.75128505Z               }
2020-07-21T14:25:44.751287166Z             }
2020-07-21T14:25:44.751289252Z           },
2020-07-21T14:25:44.751291306Z           {
2020-07-21T14:25:44.751293368Z             "terms" : {
2020-07-21T14:25:44.75129554Z               "partitionId" : [
2020-07-21T14:25:44.751297724Z                 1,
2020-07-21T14:25:44.751310878Z                 2,
2020-07-21T14:25:44.751313189Z                 3
2020-07-21T14:25:44.751315347Z               ],
2020-07-21T14:25:44.751317473Z               "boost" : 1.0
2020-07-21T14:25:44.75131971Z             }
2020-07-21T14:25:44.751321835Z           }
2020-07-21T14:25:44.751323938Z         ],
2020-07-21T14:25:44.751326099Z         "adjust_pure_negative" : true,
2020-07-21T14:25:44.751328367Z         "boost" : 1.0
2020-07-21T14:25:44.751330523Z       }
2020-07-21T14:25:44.751332653Z     },
2020-07-21T14:25:44.75133473Z     "boost" : 1.0
2020-07-21T14:25:44.751336899Z   }
2020-07-21T14:25:44.751339056Z }
2020-07-21T14:25:44.751341145Z  and aggregation: 
2020-07-21T14:25:44.751343439Z {"datesAgg":{"date_histogram":{"field":"endDate","format":"date","interval":"1d","offset":0,"order":{"_key":"asc"},"keyed":true,"min_doc_count":0},"aggregations":{"instancesAgg":{"top_hits":{"from":0,"size":100,"version":false,"seq_no_primary_term":false,"explain":false,"_source":{"includes":["id"],"excludes":[]},"sort":[{"id":{"order":"asc"}}]}},"datesSortedAgg":{"bucket_sort":{"sort":[{"_key":{"order":"asc"}}],"from":0,"size":1,"gap_policy":"SKIP"}}}}}
2020-07-21T14:25:44.761693004Z 2020-07-21 14:25:44.761 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Nothing to archive
2020-07-21T14:25:44.7628133Z 2020-07-21 14:25:44.762 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Finished batch operations for archiving request: 
2020-07-21T14:25:44.762828085Z {
2020-07-21T14:25:44.762830756Z   "constant_score" : {
2020-07-21T14:25:44.762833588Z     "filter" : {
2020-07-21T14:25:44.762835934Z       "range" : {
2020-07-21T14:25:44.762838234Z         "endDate" : {
2020-07-21T14:25:44.762842104Z           "from" : null,
2020-07-21T14:25:44.762869205Z           "to" : "now-1h",
2020-07-21T14:25:44.762871994Z           "include_lower" : true,
2020-07-21T14:25:44.762874441Z           "include_upper" : true,
2020-07-21T14:25:44.762876714Z           "boost" : 1.0
2020-07-21T14:25:44.762879103Z         }
2020-07-21T14:25:44.762881332Z       }
2020-07-21T14:25:44.762883447Z     },
2020-07-21T14:25:44.762885584Z     "boost" : 1.0
2020-07-21T14:25:44.76288783Z   }
2020-07-21T14:25:44.76289023Z }
2020-07-21T14:25:44.762892355Z  and aggregation: 
2020-07-21T14:25:44.762894582Z {"datesAgg":{"date_histogram":{"field":"endDate","format":"date","interval":"1d","offset":0,"order":{"_key":"asc"},"keyed":true,"min_doc_count":0},"aggregations":{"instancesAgg":{"top_hits":{"from":0,"size":100,"version":false,"seq_no_primary_term":false,"explain":false,"_source":{"includes":["id"],"excludes":[]},"sort":[{"id":{"order":"asc"}}]}},"datesSortedAgg":{"bucket_sort":{"sort":[{"_key":{"order":"asc"}}],"from":0,"size":1,"gap_policy":"SKIP"}}}}}
2020-07-21T14:25:44.766501838Z 2020-07-21 14:25:44.766 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Nothing to archive
2020-07-21T14:25:44.867371325Z 2020-07-21 14:25:44.860 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [workflow-instance] and partitionId [2]: ImportPositionEntity{aliasName='workflow-instance', partitionId=2, position=176093677424, indexName='zeebe-record_workflow-instance_0.23.4_2020-07-14'}
2020-07-21T14:25:44.948155143Z 2020-07-21 14:25:44.947 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable-document] and partitionId [3]: ImportPositionEntity{aliasName='variable-document', partitionId=3, position=90197001600, indexName='zeebe-record_variable-document_0.23.2_2020-07-08'}
2020-07-21T14:25:44.954305658Z 2020-07-21 14:25:44.954 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable-document] and partitionId [2]: ImportPositionEntity{aliasName='variable-document', partitionId=2, position=4294977584, indexName='zeebe-record_variable-document_0.23.2_2020-06-28'}
2020-07-21T14:25:44.960605991Z 2020-07-21 14:25:44.960 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [job] and partitionId [1]: ImportPositionEntity{aliasName='job', partitionId=1, position=781684127512, indexName='zeebe232h-record_job_0.23.4_2020-07-12'}
2020-07-21T14:25:45.053376126Z 2020-07-21 14:25:45.053 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable-document] and partitionId [1]: ImportPositionEntity{aliasName='variable-document', partitionId=1, position=107374195824, indexName='zeebe232h-record_variable-document_0.23.2_2020-06-09'}
2020-07-21T14:25:45.059229446Z 2020-07-21 14:25:45.058 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [deployment] and partitionId [3]: ImportPositionEntity{aliasName='deployment', partitionId=3, position=103085707832, indexName='zeebe-record_deployment_0.23.4_2020-07-09'}
2020-07-21T14:25:45.065351261Z 2020-07-21 14:25:45.065 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [workflow-instance] and partitionId [1]: ImportPositionEntity{aliasName='workflow-instance', partitionId=1, position=781684130944, indexName='zeebe232h-record_workflow-instance_0.23.4_2020-07-12'}
2020-07-21T14:25:45.071798492Z 2020-07-21 14:25:45.071 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [incident] and partitionId [1]: ImportPositionEntity{aliasName='incident', partitionId=1, position=678609366032, indexName='zeebe232h-record_incident_0.23.2_2020-07-07'}
2020-07-21T14:25:45.077714517Z 2020-07-21 14:25:45.077 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [incident] and partitionId [3]: ImportPositionEntity{aliasName='incident', partitionId=3, position=90197034872, indexName='zeebe-record_incident_0.23.2_2020-07-08'}
2020-07-21T14:25:45.150501553Z 2020-07-21 14:25:45.150 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable] and partitionId [1]: ImportPositionEntity{aliasName='variable', partitionId=1, position=781684128496, indexName='zeebe232h-record_variable_0.23.4_2020-07-12'}
2020-07-21T14:25:45.157149464Z 2020-07-21 14:25:45.156 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [job] and partitionId [2]: ImportPositionEntity{aliasName='job', partitionId=2, position=176093673992, indexName='zeebe-record_job_0.23.4_2020-07-14'}
2020-07-21T14:25:45.163649195Z 2020-07-21 14:25:45.163 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [incident] and partitionId [2]: ImportPositionEntity{aliasName='incident', partitionId=2, position=111669521408, indexName='zeebe-record_incident_0.23.4_2020-07-20'}
2020-07-21T14:25:45.169125221Z 2020-07-21 14:25:45.168 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable] and partitionId [3]: ImportPositionEntity{aliasName='variable', partitionId=3, position=137441769536, indexName='zeebe-record_variable_0.23.4_2020-07-12'}
2020-07-21T14:25:45.175570836Z 2020-07-21 14:25:45.175  INFO 7 --- [           main] o.s.s.w.DefaultSecurityFilterChain       : Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@2f236de0, org.springframework.security.web.context.SecurityContextPersistenceFilter@6b2e46af, org.springframework.security.web.header.HeaderWriterFilter@6a094db2, org.springframework.security.web.csrf.CsrfFilter@2d9dff65, org.camunda.operate.webapp.security.WebSecurityConfig$1@4eba373c, org.springframework.security.web.authentication.logout.LogoutFilter@5c77ba8f, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@43045f9f, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@2f37f1f9, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@5e1d03d7, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@767a014e, org.springframework.security.web.session.SessionManagementFilter@1dba4e06, org.springframework.security.web.access.ExceptionTranslationFilter@277bf091, org.springframework.security.web.access.intercept.FilterSecurityInterceptor@252dc8c4]
2020-07-21T14:25:45.241792555Z 2020-07-21 14:25:45.241 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [deployment] and partitionId [1]: ImportPositionEntity{aliasName='deployment', partitionId=1, position=773096124032, indexName='zeebe232h-record_deployment_0.23.2_2020-07-12'}
2020-07-21T14:25:45.249033735Z 2020-07-21 14:25:45.248 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [workflow-instance] and partitionId [3]: ImportPositionEntity{aliasName='workflow-instance', partitionId=3, position=137441769728, indexName='zeebe-record_workflow-instance_0.23.4_2020-07-12'}
2020-07-21T14:25:45.25671052Z 2020-07-21 14:25:45.256 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [deployment] and partitionId [2]: ImportPositionEntity{aliasName='deployment', partitionId=2, position=124560423064, indexName='zeebe-record_deployment_0.23.4_2020-07-09'}
2020-07-21T14:25:45.262282409Z 2020-07-21 14:25:45.262 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [job] and partitionId [3]: ImportPositionEntity{aliasName='job', partitionId=3, position=137441770424, indexName='zeebe-record_job_0.23.4_2020-07-12'}
2020-07-21T14:25:45.344848439Z 2020-07-21 14:25:45.344 DEBUG 7 --- [       Thread-1] o.c.o.z.ImportPositionHolder             : Latest loaded position for alias [variable] and partitionId [2]: ImportPositionEntity{aliasName='variable', partitionId=2, position=176093674976, indexName='zeebe-record_variable_0.23.4_2020-07-14'}
2020-07-21T14:25:45.354176096Z 2020-07-21 14:25:45.353  INFO 7 --- [           main] o.h.v.i.u.Version                        : HV000001: Hibernate Validator 6.0.18.Final
2020-07-21T14:25:45.776590763Z 2020-07-21 14:25:45.776 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:46.144168433Z 2020-07-21 14:25:46.143  INFO 7 --- [           main] o.s.b.a.w.s.WelcomePageHandlerMapping    : Adding welcome page: class path resource [META-INF/resources/index.html]
2020-07-21T14:25:46.461063173Z 2020-07-21 14:25:46.460  INFO 7 --- [           main] o.s.b.a.e.w.EndpointLinksResolver        : Exposing 5 endpoint(s) beneath base path '/actuator'
2020-07-21T14:25:46.559742525Z 2020-07-21 14:25:46.559  INFO 7 --- [           main] o.a.c.h.Http11NioProtocol                : Starting ProtocolHandler ["http-nio-8080"]
2020-07-21T14:25:46.574974882Z 2020-07-21 14:25:46.574  INFO 7 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat started on port(s): 8080 (http) with context path ''
2020-07-21T14:25:46.578018472Z 2020-07-21 14:25:46.577  INFO 7 --- [           main] o.c.o.Application                        : Started Application in 11.692 seconds (JVM running for 14.287)
2020-07-21T14:25:47.780840012Z 2020-07-21 14:25:47.780 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:49.785032088Z 2020-07-21 14:25:49.784 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:51.789020661Z 2020-07-21 14:25:51.788 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:53.849432583Z 2020-07-21 14:25:53.849 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:55.85336389Z 2020-07-21 14:25:55.853 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:57.857036082Z 2020-07-21 14:25:57.856 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:25:59.860881353Z 2020-07-21 14:25:59.860 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:01.866512331Z 2020-07-21 14:26:01.866 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:03.870340157Z 2020-07-21 14:26:03.870 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:05.873967956Z 2020-07-21 14:26:05.873 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:07.878257111Z 2020-07-21 14:26:07.877 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:09.881538567Z 2020-07-21 14:26:09.881 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:11.885118806Z 2020-07-21 14:26:11.884 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:13.889159784Z 2020-07-21 14:26:13.888 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:15.891950016Z 2020-07-21 14:26:15.891 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:17.895866181Z 2020-07-21 14:26:17.895 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:19.899431189Z 2020-07-21 14:26:19.899 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:21.941981649Z 2020-07-21 14:26:21.941 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:23.946054056Z 2020-07-21 14:26:23.945 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:25.950202753Z 2020-07-21 14:26:25.949 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:27.954026696Z 2020-07-21 14:26:27.953 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:29.957718809Z 2020-07-21 14:26:29.957 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:31.963360625Z 2020-07-21 14:26:31.963 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:33.966793745Z 2020-07-21 14:26:33.966 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:35.970418601Z 2020-07-21 14:26:35.970 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:37.973889474Z 2020-07-21 14:26:37.973 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:39.977593834Z 2020-07-21 14:26:39.977 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:41.981139905Z 2020-07-21 14:26:41.980 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:43.98464434Z 2020-07-21 14:26:43.984 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:44.763004591Z 2020-07-21 14:26:44.762 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Finished workflow instances for archiving request: 
2020-07-21T14:26:44.763031275Z {
2020-07-21T14:26:44.763035371Z   "constant_score" : {
2020-07-21T14:26:44.76303879Z     "filter" : {
2020-07-21T14:26:44.763042331Z       "bool" : {
2020-07-21T14:26:44.763045642Z         "must" : [
2020-07-21T14:26:44.763048951Z           {
2020-07-21T14:26:44.763052207Z             "range" : {
2020-07-21T14:26:44.763055729Z               "endDate" : {
2020-07-21T14:26:44.763078554Z                 "from" : null,
2020-07-21T14:26:44.763082352Z                 "to" : "now-1h",
2020-07-21T14:26:44.763085765Z                 "include_lower" : true,
2020-07-21T14:26:44.763089337Z                 "include_upper" : true,
2020-07-21T14:26:44.763092908Z                 "boost" : 1.0
2020-07-21T14:26:44.763096357Z               }
2020-07-21T14:26:44.763099742Z             }
2020-07-21T14:26:44.763102667Z           },
2020-07-21T14:26:44.763105778Z           {
2020-07-21T14:26:44.763108779Z             "term" : {
2020-07-21T14:26:44.763111898Z               "joinRelation" : {
2020-07-21T14:26:44.763114947Z                 "value" : "workflowInstance",
2020-07-21T14:26:44.763118195Z                 "boost" : 1.0
2020-07-21T14:26:44.763121662Z               }
2020-07-21T14:26:44.763124671Z             }
2020-07-21T14:26:44.763127817Z           },
2020-07-21T14:26:44.763130899Z           {
2020-07-21T14:26:44.763133994Z             "terms" : {
2020-07-21T14:26:44.763137277Z               "partitionId" : [
2020-07-21T14:26:44.763140916Z                 1,
2020-07-21T14:26:44.763144227Z                 2,
2020-07-21T14:26:44.763147075Z                 3
2020-07-21T14:26:44.763150026Z               ],
2020-07-21T14:26:44.763153292Z               "boost" : 1.0
2020-07-21T14:26:44.763156737Z             }
2020-07-21T14:26:44.763160002Z           }
2020-07-21T14:26:44.763162827Z         ],
2020-07-21T14:26:44.763165731Z         "adjust_pure_negative" : true,
2020-07-21T14:26:44.763169115Z         "boost" : 1.0
2020-07-21T14:26:44.763172302Z       }
2020-07-21T14:26:44.763174946Z     },
2020-07-21T14:26:44.763177832Z     "boost" : 1.0
2020-07-21T14:26:44.763180891Z   }
2020-07-21T14:26:44.763183687Z }
2020-07-21T14:26:44.763186415Z  and aggregation: 
2020-07-21T14:26:44.763189368Z {"datesAgg":{"date_histogram":{"field":"endDate","format":"date","interval":"1d","offset":0,"order":{"_key":"asc"},"keyed":true,"min_doc_count":0},"aggregations":{"instancesAgg":{"top_hits":{"from":0,"size":100,"version":false,"seq_no_primary_term":false,"explain":false,"_source":{"includes":["id"],"excludes":[]},"sort":[{"id":{"order":"asc"}}]}},"datesSortedAgg":{"bucket_sort":{"sort":[{"_key":{"order":"asc"}}],"from":0,"size":1,"gap_policy":"SKIP"}}}}}
2020-07-21T14:26:44.766676339Z 2020-07-21 14:26:44.766 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Nothing to archive
2020-07-21T14:26:44.767112805Z 2020-07-21 14:26:44.766 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Finished batch operations for archiving request: 
2020-07-21T14:26:44.767135311Z {
2020-07-21T14:26:44.767139214Z   "constant_score" : {
2020-07-21T14:26:44.767142733Z     "filter" : {
2020-07-21T14:26:44.767146094Z       "range" : {
2020-07-21T14:26:44.767149163Z         "endDate" : {
2020-07-21T14:26:44.767152679Z           "from" : null,
2020-07-21T14:26:44.767155893Z           "to" : "now-1h",
2020-07-21T14:26:44.767159353Z           "include_lower" : true,
2020-07-21T14:26:44.767162782Z           "include_upper" : true,
2020-07-21T14:26:44.767166006Z           "boost" : 1.0
2020-07-21T14:26:44.767169242Z         }
2020-07-21T14:26:44.767172727Z       }
2020-07-21T14:26:44.767175938Z     },
2020-07-21T14:26:44.767179091Z     "boost" : 1.0
2020-07-21T14:26:44.767182629Z   }
2020-07-21T14:26:44.767185743Z }
2020-07-21T14:26:44.767188924Z  and aggregation: 
2020-07-21T14:26:44.767192112Z {"datesAgg":{"date_histogram":{"field":"endDate","format":"date","interval":"1d","offset":0,"order":{"_key":"asc"},"keyed":true,"min_doc_count":0},"aggregations":{"instancesAgg":{"top_hits":{"from":0,"size":100,"version":false,"seq_no_primary_term":false,"explain":false,"_source":{"includes":["id"],"excludes":[]},"sort":[{"id":{"order":"asc"}}]}},"datesSortedAgg":{"bucket_sort":{"sort":[{"_key":{"order":"asc"}}],"from":0,"size":1,"gap_policy":"SKIP"}}}}}
2020-07-21T14:26:44.769822347Z 2020-07-21 14:26:44.769 DEBUG 7 --- [     archiver_1] o.c.o.a.WorkflowInstancesArchiverJob     : Nothing to archive
2020-07-21T14:26:45.988243545Z 2020-07-21 14:26:45.988 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:47.991877176Z 2020-07-21 14:26:47.991 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:49.995395883Z 2020-07-21 14:26:49.995 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:51.998854708Z 2020-07-21 14:26:51.998 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:54.002358907Z 2020-07-21 14:26:54.002 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:56.006040773Z 2020-07-21 14:26:56.005 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:26:58.009275156Z 2020-07-21 14:26:58.009 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:27:00.012535145Z 2020-07-21 14:27:00.012 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:27:02.016071081Z 2020-07-21 14:27:02.015 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:27:04.019600249Z 2020-07-21 14:27:04.019 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
2020-07-21T14:27:06.022875802Z 2020-07-21 14:27:06.022 DEBUG 7 --- [       Thread-0] o.c.o.w.e.w.BatchOperationWriter         : 0 operations locked
4

Hi @yoavsal,

in logs I see that Operate was importing data from two Zeebe clusters (with two different prefixes): zeebe232h-record and zeebe-record. Do you have an idea how this happened? Do you have two Zeebe clusters running?

In general one instance of Operate must load data from strictly one Zeebe cluster. In case you want to “switch” Operate to another cluster, you need to clean all Operate indices (or remove them and they will be recreated on next startup).

Best regards,
Svetlana

5

Thanks!
Yes, we changed the cluster location and name and it probably caused the mess.
After deleting all operate indices and reloading is seems to work.

6