Hi @salaboy
I’m not sure if this is an issue with the latest helm chart, which is why I haven’t raised an issue on GitHub. If it turns out to be a deployment issue I’d be happy to raise an issue there.
Since deploying zeebe-cluster-helm chart version 0.0.88 the broker bootstrap process takes an excessive amount of time to complete, over 20 min.
Looking at the zeebe-cluster logs I can see that the Bootstrap Broker-0 [6/10]: cluster service
step took 1280707ms.
logs
2020-04-14 10:49:53.525 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [3/10]: command api transport started in 416 ms
2020-04-14 10:49:53.525 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [4/10]: command api handler
2020-04-14 10:49:53.617 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [4/10]: command api handler started in 91 ms
2020-04-14 10:49:53.618 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [5/10]: subscription api
2020-04-14 10:49:53.810 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [5/10]: subscription api started in 192 ms
2020-04-14 10:49:53.811 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services
2020-04-14 11:11:14.518 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services started in 1280707 ms
2020-04-14 11:11:14.519 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager
2020-04-14 11:11:14.520 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager started in 1 ms
2020-04-14 11:11:14.521 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: metric's server
2020-04-14 11:11:14.531 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: metric's server started in 10 ms
2020-04-14 11:11:14.532 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [9/10]: leader management request handler
2020-04-14 11:11:14.533 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [9/10]: leader management request handler started in 1 ms
2020-04-14 11:11:14.534 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [10/10]: zeebe partitions
2020-04-14 11:11:14.536 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 partitions [1/1]: partition 1
2020-04-14 11:11:14.881 [] [main] DEBUG io.zeebe.broker.exporter - Exporter configured with ElasticsearchExporterConfiguration{url='http://elasticsearch-master:9200', index=IndexConfiguration{indexPrefix='zeebe-rec
ord', createTemplate=true, command=false, event=true, rejection=false, error=true, deployment=true, incident=true, job=true, message=false, messageSubscription=false, variable=true, variableDocument=true, workflowI
nstance=true, workflowInstanceCreation=false, workflowInstanceSubscription=false}, bulk=BulkConfiguration{delay=5, size=1000}, authentication=AuthenticationConfiguration{username='null'}}
2020-04-14 11:11:15.051 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.broker.system - Removing follower partition service for partition PartitionId{id=1, group=raft-partition}
2020-04-14 11:11:15.115 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.broker.system - Partition role transitioning from null to LEADER
2020-04-14 11:11:15.115 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.broker.system - Installing leader partition service for partition PartitionId{id=1, group=raft-partition}
2020-04-14 11:11:15.532 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.logstreams.snapshot - Available snapshots: [SnapshotImpl{position=38655128624, path=/usr/local/zeebe/data/raft-partition/par
titions/1/snapshots/6054-38-1586267129817-38655128624}, SnapshotImpl{position=38655109208, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/5997-38-1586266229810-38655109208}, SnapshotImpl{position=
38655094760, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/5955-38-1586265329782-38655094760}]
2020-04-14 11:11:16.468 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.logstreams.snapshot - Opened database from '/usr/local/zeebe/data/raft-partition/partitions/1/runtime'.
2020-04-14 11:11:16.470 [Broker-0-ZeebePartition-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.logstreams.snapshot - Recovered state from snapshot 'SnapshotImpl{position=38655128624, path=/usr/local/zeebe/data/raft-part
ition/partitions/1/snapshots/6054-38-1586267129817-38655128624}'
2020-04-14 11:11:16.722 [Broker-0-LogStream-1] [Broker-0-zb-actors-0] DEBUG io.zeebe.logstreams - Configured log appender back pressure at partition 1 as AppenderVegasCfg{initialLimit=1024, maxConcurrency=32768, al
phaLimit=0.7, betaLimit=0.95}. Window limiting is disabled
2020-04-14 11:11:16.871 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] DEBUG io.zeebe.logstreams - Recovering state of partition 1 from snapshot
2020-04-14 11:11:17.058 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] INFO io.zeebe.logstreams - Recovered state of partition 1 from snapshot at position 38655128624
2020-04-14 11:11:17.852 [Broker-0-SnapshotDirector-1] [Broker-0-zb-actors-1] DEBUG io.zeebe.logstreams.snapshot - The position of the last valid snapshot is '38655128624'. Taking snapshots beyond this position.
2020-04-14 11:11:17.915 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] DEBUG io.zeebe.broker.exporter - Recovering exporter from snapshot
2020-04-14 11:11:17.920 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 partitions [1/1]: partition 1 started in 3383 ms
2020-04-14 11:11:17.920 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 partitions succeeded. Started 1 steps in 3384 ms.
2020-04-14 11:11:17.920 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [10/10]: zeebe partitions started in 3386 ms
2020-04-14 11:11:17.920 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 succeeded. Started 10 steps in 1292283 ms.
2020-04-14 11:11:17.924 [Broker-0-HealthCheckService] [Broker-0-zb-actors-1] DEBUG io.zeebe.broker.system - All partitions are installed. Broker is ready!
2020-04-14 11:11:18.211 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] DEBUG io.zeebe.broker.exporter - Recovered exporter 'Broker-0-Exporter-1' from snapshot at lastExportedPosition 38655128624
2020-04-14 11:11:18.212 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] DEBUG io.zeebe.broker.exporter - Configure exporter with id 'elasticsearch'
2020-04-14 11:11:18.212 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] DEBUG io.zeebe.broker.exporter.elasticsearch - Exporter configured with ElasticsearchExporterConfiguration{url='http://elasticsearch-master:9
200', index=IndexConfiguration{indexPrefix='zeebe-record', createTemplate=true, command=false, event=true, rejection=false, error=true, deployment=true, incident=true, job=true, message=false, messageSubscription=f
alse, variable=true, variableDocument=true, workflowInstance=true, workflowInstanceCreation=false, workflowInstanceSubscription=false}, bulk=BulkConfiguration{delay=5, size=1000}, authentication=AuthenticationConfi
guration{username='null'}}
I’m currently not deploying Zeebe with the hazelcast exporter too make the upgrade easier, could this be the issue?
dev config:
{
"brokers": [
{
"partitions": [
{
"partitionId": 1,
"role": "LEADER"
}
],
"nodeId": 0,
"host": "workflow-engine-zeebe-0.workflow-engine-zeebe.dev.svc.cluster.local",
"port": 26501
}
],
"clusterSize": 1,
"partitionsCount": 1,
"replicationFactor": 1
}
ElasticSearch:
replicas: 1