Getting ClientStatusException in Java client

Hi, I am getting some exceptions in the Zeebe broker.

What does the error means? is it java client issue…?

    zeebe :0.20.1
    spring boot : 2.1.9
    java  :11

Client Exception :

io.zeebe.client.api.command.ClientStatusException: Unexpected error occurred during the request processing
        at io.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:93)
        at io.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:50)
        at com.xxxxx.business.workflow.component.handler.service.impl.ComponentWorkflowHandlerServiceImpl.sendVendorRequest(ComponentWorkflowHandlerServiceImpl.java:578)
        at com.xxxxx.business.workflow.component.handler.service.impl.ComponentWorkflowHandlerServiceImpl$$FastClassBySpringCGLIB$$f9512a9d.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:750)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
        at com.xxxxx.business.workflow.component.handler.service.impl.ComponentWorkflowHandlerServiceImpl$$EnhancerBySpringCGLIB$$9fe6a5d2.sendVendorRequest(<generated>)
        at com.xxxxx.business.workflow.component.handler.workers.SendVendorRequestWorker.handle(SendVendorRequestWorker.java:42)
        at io.zeebe.client.impl.worker.JobRunnableFactory.executeJob(JobRunnableFactory.java:44)
        at io.zeebe.client.impl.worker.JobRunnableFactory.lambda$create$0(JobRunnableFactory.java:39)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: INTERNAL: Unexpected error occurred during the request processing
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
        at io.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:48)
        ... 19 common frames omitted
Caused by: io.grpc.StatusRuntimeException: INTERNAL: Unexpected error occurred during the request processing
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        ... 3 common frames omitted

2019-10-28 07:17:27.619  WARN [xxxxx-component-workflow-handler,,,] 10 --- [pool-8-thread-1] io.zeebe.client.job.worker               : Worker default failed to handle job with key 2251799845564283 of type sendVendorRequest, sending fail command to broker


broker exception:

10/28/2019 12:48:36 PM07:18:36.877 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type validateDate from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.877 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type createFulfillmentItem from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.877 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type checkRealFulfillmentItemTask from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.877 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type createSelectionTask from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.878 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type initiateOrderUpdate from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.878 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type validateScopeOfWork from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM07:18:36.878 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.6.193:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type checkFulfillmentStatus from partition 1
10/28/2019 12:48:36 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:475) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-1.0.3.jar:1.0.3]
10/28/2019 12:48:36 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:106) ~[zeebe-transport-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:80) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:43) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:169) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:134) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:118) [zeebe-util-0.20.1.jar:0.20.1]
10/28/2019 12:48:36 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:96) [zeebe-util-0.20.1.jar:0.20.1]

Hi @regojoyson , can you please elaborate when are you getting that expiation? Is this happening in your client? or in the broker? Can you help me to reproduce the problem?
Can you quickly try with 0.21.1 ?

Hey @salaboy,

Here is the scenario, Got some exception while processing one job, then suddenly the broker starts giving the above error. then I restarted the client it started working properly. but the older which got exception is paused.

I am using the java client in spring boot (not spring zeebe).

I will try with zeebe 0.21.1 now, Is there any issue with zeebe 0.20.1?

@regojoyson any reason to not use spring-zeebe? I am interested in your feedback about that.
There shouldn’t be any issue with 0.20.1, but if it works with 0.21.1 we might be able to track the problem easily.
Cheers

@salaboy Actually 3 months ago there was an issue in spring-zeebe, it’s not creating incidents so I migrated to java client. I think now that the issue is resolved.

I am going to migrate from java-client to spring-zeebe now, But is it spring-zeebe is available for zeebe:0.21.1? if yes I will migrate broker and spring-zeebe to the latest one else will use 0.20.1?

I am working on that migration… there is an open pull request (https://github.com/zeebe-io/spring-zeebe/pull/68)… but I’ve used it against 0.21.1 (spring Zeebe 0.6.0) and everything seems to be working fine. If you can give it a try and see if that resolves your problem it will be great.

I am happy to collaborate with you testing these problems and making sure that spring-zeebe is stable.

1 Like

@salaboy Sure, I am going to test now and let you know if any issue.

2 Likes

@salaboy @jwulf

I am using spring-zeebe 0.6.0 with zeebe 0.21.1 now as per your suggestion @salaboy .

now I am getting some exceptions in broker.

10/30/2019 4:55:40 PMjava.lang.IndexOutOfBoundsException: index=228 length=4 capacity=228
10/30/2019 4:55:40 PM	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:1716) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:40 PM	at org.agrona.concurrent.UnsafeBuffer.getInt(UnsafeBuffer.java:505) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:40 PM	at io.zeebe.engine.state.instance.ElementInstance.wrap(ElementInstance.java:154) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:66) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:132) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:58) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.state.instance.ElementInstanceState.getInstance(ElementInstanceState.java:123) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.prepareEventOccurredEvent(TriggerTimerProcessor.java:137) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processTimerTrigger(TriggerTimerProcessor.java:88) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processRecord(TriggerTimerProcessor.java:72) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:26) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:46) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.ProcessingStateMachine.lambda$processInTransaction$1(ProcessingStateMachine.java:240) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:69) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:231) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processEvent(ProcessingStateMachine.java:200) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.engine.processor.ProcessingStateMachine.readNextEvent(ProcessingStateMachine.java:176) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:40 PM2019-10-30 11:25:40.810 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] WARN  io.zeebe.broker.logstreams - Blacklist workflow instance 2251799835501378, due to previous errors.
10/30/2019 4:55:40 PM2019-10-30 11:25:40.810 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] INFO  io.zeebe.processor - Error record was written at 36155036554248, we will continue with processing if event was committed. Current commit position is 36155036553216.
10/30/2019 4:55:40 PM2019-10-30 11:25:40.815 [partition-1-processor] [10.42.254.121:26501-zb-actors-0] INFO  io.zeebe.processor - Error event was committed, we continue with processing.
10/30/2019 4:55:41 PM2019-10-30 11:25:41.648 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] ERROR io.zeebe.processor - Expected to successfully process event 'LoggedEvent [type=0, version=0, streamId=1, position=36155036557400, key=2251799869714100, timestamp=1572434741643, sourceEventPosition=-1]' with processor, but caught an exception. Skip this event.
10/30/2019 4:55:41 PMjava.lang.IndexOutOfBoundsException: index=228 length=4 capacity=228
10/30/2019 4:55:41 PM	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:1716) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:41 PM	at org.agrona.concurrent.UnsafeBuffer.getInt(UnsafeBuffer.java:505) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:41 PM	at io.zeebe.engine.state.instance.ElementInstance.wrap(ElementInstance.java:154) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:66) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:132) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:58) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.state.instance.ElementInstanceState.getInstance(ElementInstanceState.java:123) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.prepareEventOccurredEvent(TriggerTimerProcessor.java:137) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processTimerTrigger(TriggerTimerProcessor.java:88) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processRecord(TriggerTimerProcessor.java:72) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:26) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:46) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.lambda$processInTransaction$1(ProcessingStateMachine.java:240) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:69) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:231) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processEvent(ProcessingStateMachine.java:200) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.readNextEvent(ProcessingStateMachine.java:176) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM2019-10-30 11:25:41.649 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] ERROR io.zeebe.processor - Expected to process event 'TypedEventImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=TRIGGER, requestStreamId=-2147483648, requestId=-1, protocolVersion=1, valueType=TIMER, rejectionType=NULL_VAL, rejectionReason=}, value={"elementInstanceKey":2251799869714099,"workflowInstanceKey":2251799835501391,"dueDate":1572434741635,"targetElementId":"IntermediateCatchEvent_0p42zq5","repetitions":1,"workflowKey":2251799835501330}}' without errors, but exception occurred with message 'index=228 length=4 capacity=228' .
10/30/2019 4:55:41 PMjava.lang.IndexOutOfBoundsException: index=228 length=4 capacity=228
10/30/2019 4:55:41 PM	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:1716) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:41 PM	at org.agrona.concurrent.UnsafeBuffer.getInt(UnsafeBuffer.java:505) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:41 PM	at io.zeebe.engine.state.instance.ElementInstance.wrap(ElementInstance.java:154) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:66) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:132) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:58) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.state.instance.ElementInstanceState.getInstance(ElementInstanceState.java:123) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.prepareEventOccurredEvent(TriggerTimerProcessor.java:137) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processTimerTrigger(TriggerTimerProcessor.java:88) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processRecord(TriggerTimerProcessor.java:72) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:26) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:46) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.lambda$processInTransaction$1(ProcessingStateMachine.java:240) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:69) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:231) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processEvent(ProcessingStateMachine.java:200) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.engine.processor.ProcessingStateMachine.readNextEvent(ProcessingStateMachine.java:176) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:41 PM2019-10-30 11:25:41.650 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] WARN  io.zeebe.broker.logstreams - Blacklist workflow instance 2251799835501391, due to previous errors.
10/30/2019 4:55:41 PM2019-10-30 11:25:41.650 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] INFO  io.zeebe.processor - Error record was written at 36155036558432, we will continue with processing if event was committed. Current commit position is 36155036557400.
10/30/2019 4:55:41 PM2019-10-30 11:25:41.654 [partition-1-processor] [10.42.254.121:26501-zb-actors-0] INFO  io.zeebe.processor - Error event was committed, we continue with processing.
10/30/2019 4:55:42 PM2019-10-30 11:25:42.301 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] ERROR io.zeebe.processor - Expected to successfully process event 'LoggedEvent [type=0, version=0, streamId=1, position=36155036561840, key=2251799869714114, timestamp=1572434742296, sourceEventPosition=-1]' with processor, but caught an exception. Skip this event.
10/30/2019 4:55:42 PMjava.lang.IndexOutOfBoundsException: index=228 length=4 capacity=228
10/30/2019 4:55:42 PM	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:1716) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:42 PM	at org.agrona.concurrent.UnsafeBuffer.getInt(UnsafeBuffer.java:505) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:42 PM	at io.zeebe.engine.state.instance.ElementInstance.wrap(ElementInstance.java:154) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:66) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:132) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:58) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.state.instance.ElementInstanceState.getInstance(ElementInstanceState.java:123) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.prepareEventOccurredEvent(TriggerTimerProcessor.java:137) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processTimerTrigger(TriggerTimerProcessor.java:88) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processRecord(TriggerTimerProcessor.java:72) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:26) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:46) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.lambda$processInTransaction$1(ProcessingStateMachine.java:240) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:69) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:231) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processEvent(ProcessingStateMachine.java:200) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.readNextEvent(ProcessingStateMachine.java:176) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM2019-10-30 11:25:42.302 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] ERROR io.zeebe.processor - Expected to process event 'TypedEventImpl{metadata=RecordMetadata{recordType=COMMAND, intentValue=255, intent=TRIGGER, requestStreamId=-2147483648, requestId=-1, protocolVersion=1, valueType=TIMER, rejectionType=NULL_VAL, rejectionReason=}, value={"elementInstanceKey":2251799869714105,"workflowInstanceKey":2251799835501503,"dueDate":1572434742284,"targetElementId":"IntermediateCatchEvent_0p42zq5","repetitions":1,"workflowKey":2251799835501330}}' without errors, but exception occurred with message 'index=228 length=4 capacity=228' .
10/30/2019 4:55:42 PMjava.lang.IndexOutOfBoundsException: index=228 length=4 capacity=228
10/30/2019 4:55:42 PM	at org.agrona.concurrent.UnsafeBuffer.boundsCheck0(UnsafeBuffer.java:1716) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:42 PM	at org.agrona.concurrent.UnsafeBuffer.getInt(UnsafeBuffer.java:505) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 4:55:42 PM	at io.zeebe.engine.state.instance.ElementInstance.wrap(ElementInstance.java:154) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:66) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:132) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.TransactionalColumnFamily.get(TransactionalColumnFamily.java:58) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.state.instance.ElementInstanceState.getInstance(ElementInstanceState.java:123) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.prepareEventOccurredEvent(TriggerTimerProcessor.java:137) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processTimerTrigger(TriggerTimerProcessor.java:88) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.workflow.timer.TriggerTimerProcessor.processRecord(TriggerTimerProcessor.java:72) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:26) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.TypedRecordProcessor.processRecord(TypedRecordProcessor.java:46) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.lambda$processInTransaction$1(ProcessingStateMachine.java:240) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.db.impl.rocksdb.transaction.ZeebeTransaction.run(ZeebeTransaction.java:69) ~[zeebe-db-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processInTransaction(ProcessingStateMachine.java:231) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.processEvent(ProcessingStateMachine.java:200) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.engine.processor.ProcessingStateMachine.readNextEvent(ProcessingStateMachine.java:176) ~[zeebe-workflow-engine-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 4:55:42 PM2019-10-30 11:25:42.303 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] WARN  io.zeebe.broker.logstreams - Blacklist workflow instance 2251799835501503, due to previous errors.
10/30/2019 4:55:42 PM2019-10-30 11:25:42.303 [partition-1-processor] [10.42.254.121:26501-zb-actors-1] INFO  io.zeebe.processor - Error record was written at 36155036562872, we will continue with processing if event was committed. Current commit position is 36155036561840.
10/30/2019 4:55:42 PM2019-10-30 11:25:42.307 [partition-1-processor] [10.42.254.121:26501-zb-actors-0] INFO  io.zeebe.processor - Error event was committed, we continue with processing.

What does this error means…??

And also when i am shutdown the client the broker gives another exception :slight_smile:


10/30/2019 5:51:57 PM	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368) ~[grpc-stub-1.24.0.jar:1.24.0]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.complete(LongPollingActivateJobsRequest.java:72) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.timeout(LongPollingActivateJobsRequest.java:94) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler.lambda$addTimeOut$9(LongPollingActivateJobsHandler.java:176) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM2019-10-30 12:21:57.811 [io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler] [10.42.145.18:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to complete io.zeebe.gateway.impl.broker.request.BrokerActivateJobsRequest@71c231f3
10/30/2019 5:51:57 PMio.grpc.StatusRuntimeException: CANCELLED: call already cancelled
10/30/2019 5:51:57 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 5:51:57 PM	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368) ~[grpc-stub-1.24.0.jar:1.24.0]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.complete(LongPollingActivateJobsRequest.java:72) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.timeout(LongPollingActivateJobsRequest.java:94) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler.lambda$addTimeOut$9(LongPollingActivateJobsHandler.java:176) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:57 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM2019-10-30 12:21:58.510 [io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler] [10.42.145.18:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to complete io.zeebe.gateway.impl.broker.request.BrokerActivateJobsRequest@66f8aa99
10/30/2019 5:51:58 PMio.grpc.StatusRuntimeException: CANCELLED: call already cancelled
10/30/2019 5:51:58 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 5:51:58 PM	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368) ~[grpc-stub-1.24.0.jar:1.24.0]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.complete(LongPollingActivateJobsRequest.java:72) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.timeout(LongPollingActivateJobsRequest.java:94) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler.lambda$addTimeOut$9(LongPollingActivateJobsHandler.java:176) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM2019-10-30 12:21:58.804 [io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler] [10.42.145.18:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to complete io.zeebe.gateway.impl.broker.request.BrokerActivateJobsRequest@2eaca279
10/30/2019 5:51:58 PMio.grpc.StatusRuntimeException: CANCELLED: call already cancelled
10/30/2019 5:51:58 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 5:51:58 PM	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368) ~[grpc-stub-1.24.0.jar:1.24.0]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.complete(LongPollingActivateJobsRequest.java:72) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.timeout(LongPollingActivateJobsRequest.java:94) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler.lambda$addTimeOut$9(LongPollingActivateJobsHandler.java:176) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:58 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM2019-10-30 12:21:59.612 [io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler] [10.42.145.18:26501-zb-actors-1] WARN  io.zeebe.gateway - Failed to complete io.zeebe.gateway.impl.broker.request.BrokerActivateJobsRequest@13720c2
10/30/2019 5:51:59 PMio.grpc.StatusRuntimeException: CANCELLED: call already cancelled
10/30/2019 5:51:59 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 5:51:59 PM	at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:368) ~[grpc-stub-1.24.0.jar:1.24.0]
10/30/2019 5:51:59 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.complete(LongPollingActivateJobsRequest.java:72) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsRequest.timeout(LongPollingActivateJobsRequest.java:94) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler.lambda$addTimeOut$9(LongPollingActivateJobsHandler.java:176) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:51:59 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 5:52:03 PM2019-10-30 12:22:03.109 [io.zeebe.gateway.impl.job.LongPollingActivateJobsHandler] [10.42.145.18:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to complete io.zeebe.gateway.impl.broker.request.BrokerActivateJobsRequest@27ec728e
10/30/2019 5:52:03 PMio.grpc.StatusRuntimeException: CANCELLED: call already cancelled

Is it normal behaviour while stopping the client…?

Hi @regojoyson I haven’t seen those problem here… can you give me an idea on how to reproduce them? So I can help you to figure out what is wrong?

Can you give me also more information about how are you running things? Are you also using 0.21.1 in the brokers?

Hey @salaboy

  1. I was using zeebe broker 0.20.1 version in Rancher (internally uses docker) and the spring-boot with zeebe-java client in rancher as container.
  2. then after your suggestion i upgraded the zeebe to 0.21.1 (data still older one, only image is 0.21.1).
    here is the startup logs :slight_smile:
10/30/2019 7:46:56 PM  "threads": {
10/30/2019 7:46:56 PM    "managementThreads": 1
10/30/2019 7:46:56 PM  },
10/30/2019 7:46:56 PM  "monitoring": {
10/30/2019 7:46:56 PM    "enabled": false,
10/30/2019 7:46:56 PM    "host": "10.42.145.18",
10/30/2019 7:46:56 PM    "port": 9600
10/30/2019 7:46:56 PM  },
10/30/2019 7:46:56 PM  "security": {
10/30/2019 7:46:56 PM    "enabled": false
10/30/2019 7:46:56 PM  }
10/30/2019 7:46:56 PM}
10/30/2019 7:46:56 PM2019-10-30 14:16:56.386 [io.zeebe.gateway.impl.broker.cluster.BrokerTopologyManagerImpl] [10.42.145.18:26501-zb-actors-0] INFO  io.zeebe.transport.endpoint - Registering endpoint for node '0' with address '10.42.145.18:26501' on transport 'gateway-broker-client'
10/30/2019 7:46:57 PM2019-10-30 14:16:57.448 [] [raft-server-system-partition-1] WARN  io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$427/0x0000000800495040@2f86f84d not registered
10/30/2019 7:47:08 PM2019-10-30 14:17:08.763 [] [raft-server-raft-atomix-partition-1] WARN  io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$427/0x0000000800495040@5e69af0c not registered
10/30/2019 7:47:58 PM2019-10-30 14:17:58.143 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [10.42.145.18:26501-zb-actors-0] INFO  io.zeebe.broker.system - Partition '1' is installed.
10/30/2019 7:47:58 PM2019-10-30 14:17:58.146 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [10.42.145.18:26501-zb-actors-0] INFO  io.zeebe.broker.system - All partitions are installed. Broker is ready!
10/30/2019 7:47:58 PM2019-10-30 14:17:58.857 [partition-1-processor] [10.42.145.18:26501-zb-actors-0] INFO  io.zeebe.logstreams - Recovered state of partition 1 from snapshot at position 99260993904880
10/30/2019 7:47:59 PM2019-10-30 14:17:59.425 [exporter] [10.42.145.18:26501-zb-fs-workers-0] INFO  io.zeebe.broker.exporter.elasticsearch - Exporter opened

Its giving some warning also.

  1. Then I changed the client to spring-zeebe with spring boot. and I connected the spring-zeebe client to the broker.

  2. after few minutes I got one exception in zeebe logs (1st log in question)

  3. Then I simply turned off the client after 5 minutes. its started giving the error in broker (the second log in question).

  4. And also once suddenly some jobs are failed and its created incidents.

Just now got another issue in client


2019-10-30 15:19:54.898  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ault-executor-8] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type validateDate

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999868398ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

2019-10-30 15:19:54.899  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ault-executor-8] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type sendWorkflowNotification

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999845297ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

2019-10-30 15:19:54.899  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ult-executor-11] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type createFulfillmentItem

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999873098ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

2019-10-30 15:19:54.899  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ult-executor-11] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type CompleteValidationTask

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999724194ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

2019-10-30 15:19:54.899  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ault-executor-6] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type updateWorkflowAudit

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999868998ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

2019-10-30 15:19:54.900  WARN [xxxxx-component-workflow-handler,,,] 10 --- [ult-executor-11] io.zeebe.client.job.poller               : Failed to activated jobs for worker component-workflow-handler-1 and job type ValidationTask

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999874697ns
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

broker logs :

10/30/2019 8:49:43 PM2019-10-30 15:19:43.188 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.50.221:26501-zb-actors-0] ERROR io.zeebe.gateway - Error handling gRPC request
10/30/2019 8:49:43 PMio.grpc.StatusRuntimeException: INTERNAL: Unexpected error occurred during the request processing
10/30/2019 8:49:43 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:291) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.job.ActivateJobsHandler.logErrorResponse(ActivateJobsHandler.java:109) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.job.ActivateJobsHandler.lambda$activateJobs$1(ActivateJobsHandler.java:96) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:130) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$5(BrokerRequestManager.java:161) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:32) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PMCaused by: io.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:291) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:344) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:100) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	... 6 more
10/30/2019 8:49:43 PM2019-10-30 15:19:43.193 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.50.221:26501-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type checkRealFulfillmentItemTask from partition 1
10/30/2019 8:49:43 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:143) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:291) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:344) ~[agrona-1.0.7.jar:1.0.7]
10/30/2019 8:49:43 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:100) ~[zeebe-transport-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:127) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM2019-10-30 15:19:43.194 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [10.42.50.221:26501-zb-actors-0] ERROR io.zeebe.gateway - Error handling gRPC request
10/30/2019 8:49:43 PMio.grpc.StatusRuntimeException: INTERNAL: Unexpected error occurred during the request processing
10/30/2019 8:49:43 PM	at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.24.0.jar:1.24.0]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:291) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.job.ActivateJobsHandler.logErrorResponse(ActivateJobsHandler.java:109) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.job.ActivateJobsHandler.lambda$activateJobs$1(ActivateJobsHandler.java:96) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:130) ~[zeebe-gateway-0.21.1.jar:0.21.1]
10/30/2019 8:49:43 PM	at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$5(BrokerRequestManager.java:161) ~[zeebe-gateway-0.21.1.jar:0.21.1]
`` 

what does it mean ?
--------------------------

Few Doubts
   1. Is there any issue with spring-zeebe with the latest zeebe 0.21.1?
  2. what does the first log in question says (IndexOutOfBoundsException)?
 3. what does StatusRuntimeException: CANCELLED means? 
  4. when will be the new version of spring-zeebe is available?