Hi, I am facing some job failed exception in zeebe and also workers are not able to pick jobs. If I delete the data volume and rebuild then it works fine. the error comes after certain amount of data. it happens 3 times for me. so i deleted the data and rebuild-ed the container.
zeebe : 0.17.0
client : java client 0.17.0
Zeebe log :
6/23/2019 7:35:12 PM14:05:12.692 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN io.zeebe.gateway - Failed to activate jobs for type sendWorkflowNotification from partition 0
6/23/2019 7:35:12 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:12 PM at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:12 PM at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM14:05:14.361 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN io.zeebe.gateway - Failed to activate jobs for type existsFulfillmentItem from partition 0
6/23/2019 7:35:14 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:14 PM at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:14 PM at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM14:05:17.689 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN io.zeebe.gateway - Failed to activate jobs for type createSelectionTask from partition 0
6/23/2019 7:35:17 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:17 PM at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:17 PM at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0
Java client log:
2019-06-23 19:56:14.684 WARN [test-component-workflow-handler,,,] 16428 --- [ult-executor-17] io.zeebe.client.job.poller : Failed to activated jobs for worker default and job type createVerificationTask
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:500)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:65)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:592)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:508)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:632)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/0:0:0:0:0:0:0:1:26500
at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
... 11 common frames omitted