Zeebe on kubernetes: containers not ready

Zeebe has been running on my Kubernetes for a while, and it has been working very well. Until recently, I found that a Pod kept reporting errors and restarting, but I did not see warnings or errors in the log.

➜  ~ kubectl describe pods zeebe-zeebe-0
Name:         zeebe-zeebe-0
Namespace:    default
Priority:     0
Node:         server4/192.168.137.124
Start Time:   Tue, 20 Oct 2020 16:38:54 +0800
Labels:       app.kubernetes.io/component=broker
              app.kubernetes.io/instance=zeebe
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=zeebe
              controller-revision-hash=zeebe-zeebe-555c6877cd
              statefulset.kubernetes.io/pod-name=zeebe-zeebe-0
Annotations:  <none>
Status:       Running
IP:           10.244.1.108
IPs:
  IP:           10.244.1.108
Controlled By:  StatefulSet/zeebe-zeebe
Containers:
  zeebe:
    Container ID:   docker://8f3a116d0fa70298a240a6c92d3213025b235cdfe772a7660e47291cff228659
    Image:          camunda/zeebe:0.24.2
    Image ID:       docker-pullable://camunda/zeebe@sha256:795ace31c498ad4bc37b7b0fab612307c34852f4187766e3f777a509821c9fb3
    Ports:          9600/TCP, 26501/TCP, 26502/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 20 Oct 2020 16:47:55 +0800
      Finished:     Tue, 20 Oct 2020 16:48:01 +0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     1
      memory:  4Gi
    Requests:
      cpu:      500m
      memory:   2Gi
    Readiness:  http-get http://:9600/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                zeebe-zeebe
      ZEEBE_LOG_LEVEL:
      ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:            3
      ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                3
      ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:          3
      ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:             2
      ZEEBE_BROKER_THREADS_IOTHREADCOUNT:              2
      ZEEBE_BROKER_GATEWAY_ENABLE:                     false
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:  io.zeebe.exporter.ElasticsearchExporter
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:   http://elasticsearch-master:9200
      ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:            26501
      ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:           26502
      ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:         9600
      K8S_POD_NAME:                                    zeebe-zeebe-0 (v1:metadata.name)
      JAVA_TOOL_OPTIONS:                               -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
    Mounts:
      /exporters from exporters (rw)
      /usr/local/bin/startup.sh from config (rw,path="startup.sh")
      /usr/local/zeebe/config/application.yaml from config (rw,path="application.yaml")
      /usr/local/zeebe/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mt74h (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-zeebe-zeebe-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      zeebe
    Optional:  false
  exporters:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-mt74h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-mt74h
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned default/zeebe-zeebe-0 to server4
  Normal   Pulled     10m (x4 over 12m)     kubelet            Container image "camunda/zeebe:0.24.2" already present on machine
  Normal   Created    10m (x4 over 12m)     kubelet            Created container zeebe
  Normal   Started    10m (x4 over 12m)     kubelet            Started container zeebe
  Warning  Unhealthy  10m (x5 over 12m)     kubelet            Readiness probe failed: Get "http://10.244.1.108:9600/ready": dial tcp 10.244.1.108:9600: connect: connection refused
  Warning  Unhealthy  10m (x3 over 12m)     kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503
  Warning  BackOff    2m50s (x33 over 11m)  kubelet            Back-off restarting failed container

for logs:

➜  ~ kubectl logs zeebe-zeebe-0
++ hostname -f
+ export ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local
+ ZEEBE_BROKER_NETWORK_ADVERTISEDHOST=zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local
+ export ZEEBE_BROKER_CLUSTER_NODEID=0
+ ZEEBE_BROKER_CLUSTER_NODEID=0
+ export ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=3
+ contactPointPrefix=zeebe-zeebe
+ contactPoints=
+ [[ -z '' ]]
+ (( i=0 ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
++ hostname -d
+ contactPoints=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
+ (( i++ ))
+ (( i<3 ))
+ export ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
+ ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=,zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502,zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502
++ ls -A /exporters/
+ '[' '' ']'
No exporters available.
+ echo 'No exporters available.'
+ exec /usr/local/zeebe/bin/broker
Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=25.0 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
2020-10-20 08:53:11,231 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
        at org.apache.logging.log4j.Level.valueOf(Level.java:320)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
        at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
        at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
        at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:263)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
        at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
        at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
        at org.apache.commons.logging.LogAdapter$Log4jLog.<clinit>(LogAdapter.java:155)
        at org.apache.commons.logging.LogAdapter$Log4jAdapter.createLog(LogAdapter.java:122)
        at org.apache.commons.logging.LogAdapter.createLog(LogAdapter.java:89)
        at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:46)
        at org.apache.commons.logging.LogFactoryService.getInstance(LogFactoryService.java:41)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
        at org.springframework.boot.SpringApplication.<clinit>(SpringApplication.java:196)
        at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

2020-10-20 08:53:12,913 main WARN Error while converting string [] to type [class org.apache.logging.log4j.Level]. Using default value [null]. java.lang.IllegalArgumentException: Unknown level constant [].
        at org.apache.logging.log4j.Level.valueOf(Level.java:320)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:288)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters$LevelConverter.convert(TypeConverters.java:284)
        at org.apache.logging.log4j.core.config.plugins.convert.TypeConverters.convert(TypeConverters.java:419)
        at org.apache.logging.log4j.core.config.plugins.visitors.AbstractPluginVisitor.convert(AbstractPluginVisitor.java:149)
        at org.apache.logging.log4j.core.config.plugins.visitors.PluginAttributeVisitor.visit(PluginAttributeVisitor.java:45)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:258)
        at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:135)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
        at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
        at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:618)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:691)
        at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:708)
        at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.reinitialize(Log4J2LoggingSystem.java:204)
        at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:73)
        at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:60)
        at org.springframework.boot.logging.log4j2.Log4J2LoggingSystem.initialize(Log4J2LoggingSystem.java:160)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:306)
        at org.springframework.boot.context.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:281)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:239)
        at org.springframework.boot.context.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:216)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226)
        at io.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:52)

  ______  ______   ______   ____    ______     ____    _____     ____    _  __  ______   _____
 |___  / |  ____| |  ____| |  _ \  |  ____|   |  _ \  |  __ \   / __ \  | |/ / |  ____| |  __ \
    / /  | |__    | |__    | |_) | | |__      | |_) | | |__) | | |  | | | ' /  | |__    | |__) |
   / /   |  __|   |  __|   |  _ <  |  __|     |  _ <  |  _  /  | |  | | |  <   |  __|   |  _  /
  / /__  | |____  | |____  | |_) | | |____    | |_) | | | \ \  | |__| | | . \  | |____  | | \ \
 /_____| |______| |______| |____/  |______|   |____/  |_|  \_\  \____/  |_|\_\ |______| |_|  \_\

2020-10-20 08:53:13.232 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v0.24.2 on zeebe-zeebe-0 with PID 6 (/usr/local/zeebe/lib/zeebe-distribution-0.24.2.jar started by root in /usr/local/zeebe)
2020-10-20 08:53:13.300 [] [main] INFO  io.zeebe.broker.StandaloneBroker - No active profile set, falling back to default profiles: default
2020-10-20 08:53:18.335 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 9600 (http)
2020-10-20 08:53:18.405 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-10-20 08:53:18.406 [] [main] INFO  org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-10-20 08:53:18.407 [] [main] INFO  org.apache.catalina.core.StandardEngine - Starting Servlet engine: [Apache Tomcat/9.0.36]
2020-10-20 08:53:18.797 [] [main] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-10-20 08:53:18.798 [] [main] INFO  org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext - Root WebApplicationContext: initialization completed in 5368 ms
2020-10-20 08:53:20.219 [] [main] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Initializing ExecutorService 'applicationTaskExecutor'
2020-10-20 08:53:21.126 [] [main] INFO  org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 2 endpoint(s) beneath base path '/actuator'
2020-10-20 08:53:21.218 [] [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-0.0.0.0-9600"]
2020-10-20 08:53:21.394 [] [main] INFO  org.springframework.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 9600 (http) with context path ''
2020-10-20 08:53:21.415 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 9.509 seconds (JVM running for 13.396)
2020-10-20 08:53:21.536 [] [main] INFO  io.zeebe.broker.system - Version: 0.24.2
2020-10-20 08:53:21.706 [] [main] INFO  io.zeebe.broker.system - Starting broker 0 with configuration {
  "network" : {
    "host" : "0.0.0.0",
    "portOffset" : 0,
    "maxMessageSize" : "4MB",
    "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
    "commandApi" : {
      "host" : "0.0.0.0",
      "port" : 26501,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 26501,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26501",
      "address" : "0.0.0.0:26501"
    },
    "internalApi" : {
      "host" : "0.0.0.0",
      "port" : 26502,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 26502,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502",
      "address" : "0.0.0.0:26502"
    },
    "monitoringApi" : {
      "host" : "0.0.0.0",
      "port" : 9600,
      "advertisedHost" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local",
      "advertisedPort" : 9600,
      "advertisedAddress" : "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:9600",
      "address" : "0.0.0.0:9600"
    },
    "maxMessageSizeInBytes" : 4194304
  },
  "cluster" : {
    "initialContactPoints" : [ "zeebe-zeebe-0.zeebe-zeebe.default.svc.cluster.local:26502", "zeebe-zeebe-1.zeebe-zeebe.default.svc.cluster.local:26502", "zeebe-zeebe-2.zeebe-zeebe.default.svc.cluster.local:26502" ],
    "partitionIds" : [ 1, 2, 3 ],
    "nodeId" : 0,
    "partitionsCount" : 3,
    "replicationFactor" : 3,
    "clusterSize" : 3,
    "clusterName" : "zeebe-zeebe",
    "membership" : {
      "broadcastUpdates" : false,
      "broadcastDisputes" : true,
      "notifySuspect" : false,
      "gossipInterval" : "PT0.25S",
      "gossipFanout" : 2,
      "probeInterval" : "PT1S",
      "probeTimeout" : "PT2S",
      "suspectProbes" : 3,
      "failureTimeout" : "PT10S",
      "syncInterval" : "PT10S"
    }
  },
  "threads" : {
    "cpuThreadCount" : 2,
    "ioThreadCount" : 2
  },
  "data" : {
    "directories" : [ "/usr/local/zeebe/data" ],
    "logSegmentSize" : "512MB",
    "snapshotPeriod" : "PT15M",
    "logIndexDensity" : 100,
    "logSegmentSizeInBytes" : 536870912,
    "atomixStorageLevel" : "DISK"
  },
  "exporters" : {
    "elasticsearch" : {
      "jarPath" : null,
      "className" : "io.zeebe.exporter.ElasticsearchExporter",
      "args" : {
        "url" : "http://elasticsearch-master:9200"
      },
      "external" : false
    }
  },
  "gateway" : {
    "network" : {
      "host" : "0.0.0.0",
      "port" : 26500,
      "minKeepAliveInterval" : "PT30S"
    },
    "cluster" : {
      "contactPoint" : "0.0.0.0:26502",
      "requestTimeout" : "PT15S",
      "clusterName" : "zeebe-cluster",
      "memberId" : "gateway",
      "host" : "0.0.0.0",
      "port" : 26502,
      "membership" : {
        "broadcastUpdates" : false,
        "broadcastDisputes" : true,
        "notifySuspect" : false,
        "gossipInterval" : "PT0.25S",
        "gossipFanout" : 2,
        "probeInterval" : "PT1S",
        "probeTimeout" : "PT2S",
        "suspectProbes" : 3,
        "failureTimeout" : "PT10S",
        "syncInterval" : "PT10S"
      }
    },
    "threads" : {
      "managementThreads" : 1
    },
    "monitoring" : {
      "enabled" : false,
      "host" : "0.0.0.0",
      "port" : 9600
    },
    "security" : {
      "enabled" : false,
      "certificateChainPath" : null,
      "privateKeyPath" : null
    },
    "longPolling" : {
      "enabled" : true
    },
    "initialized" : true,
    "enable" : false
  },
  "backpressure" : {
    "enabled" : true,
    "algorithm" : "VEGAS",
    "aimd" : {
      "requestTimeout" : "PT1S",
      "initialLimit" : 100,
      "minLimit" : 1,
      "maxLimit" : 1000,
      "backoffRatio" : 0.9
    },
    "fixedLimit" : {
      "limit" : 20
    },
    "vegas" : {
      "alpha" : 3,
      "beta" : 6,
      "initialLimit" : 20
    },
    "gradient" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0
    },
    "gradient2" : {
      "minLimit" : 10,
      "initialLimit" : 20,
      "rttTolerance" : 2.0,
      "longWindow" : 600
    }
  },
  "stepTimeout" : "PT5M",
  "executionMetricsExporterEnabled" : false
}
2020-10-20 08:53:21.733 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [1/10]: actor scheduler
2020-10-20 08:53:21.810 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [2/10]: membership and replication protocol
2020-10-20 08:53:23.806 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring DispatcherServlet 'dispatcherServlet'
2020-10-20 08:53:23.814 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Initializing Servlet 'dispatcherServlet'
2020-10-20 08:53:24.003 [] [http-nio-0.0.0.0-9600-exec-1] INFO  org.springframework.web.servlet.DispatcherServlet - Completed initialization in 189 ms
2020-10-20 08:53:30.219 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [3/10]: command api transport
2020-10-20 08:53:30.725 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [4/10]: command api handler
2020-10-20 08:53:31.199 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [5/10]: subscription api
2020-10-20 08:53:31.239 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services
2020-10-20 08:53:36.339 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager
2020-10-20 08:53:36.341 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: monitoring services
2020-10-20 08:53:36.349 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [9/10]: leader management request handler
2020-10-20 08:53:36.357 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [10/10]: zeebe partitions
2020-10-20 08:53:36.360 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [1/3]: partition 3
2020-10-20 08:53:36.796 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [2/3]: partition 2
2020-10-20 08:53:36.816 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [3/3]: partition 1
2020-10-20 08:53:36.903 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions succeeded. Started 3 steps in 543 ms.
2020-10-20 08:53:36.904 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 succeeded. Started 10 steps in 15172 ms.

I use helm to install it, but even if I use helm delete zeebe and then reinstall it the problem still exist.

Ok, we found the problem for this one… can you please mark this as solved. Reference here: https://github.com/zeebe-io/zeebe-full-helm/issues/100