Failed to activate jobs

Hi, I am facing some job failed exception in zeebe and also workers are not able to pick jobs. If I delete the data volume and rebuild then it works fine. the error comes after certain amount of data. it happens 3 times for me. so i deleted the data and rebuild-ed the container.

zeebe : 0.17.0
client : java client 0.17.0

Zeebe log :

6/23/2019 7:35:12 PM14:05:12.692 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type sendWorkflowNotification from partition 0
6/23/2019 7:35:12 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:12 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:12 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:12 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM14:05:14.361 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type existsFulfillmentItem from partition 0
6/23/2019 7:35:14 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:14 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:14 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:14 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM14:05:17.689 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] WARN  io.zeebe.gateway - Failed to activate jobs for type createSelectionTask from partition 0
6/23/2019 7:35:17 PMio.zeebe.transport.RequestTimeoutException: Request timed out after PT15S
6/23/2019 7:35:17 PM	at io.zeebe.transport.impl.sender.OutgoingRequest.timeout(OutgoingRequest.java:151) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.transport.impl.sender.Sender.onTimerExpiry(Sender.java:483) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at org.agrona.DeadlineTimerWheel.poll(DeadlineTimerWheel.java:284) ~[agrona-0.9.35.jar:0.9.35]
6/23/2019 7:35:17 PM	at io.zeebe.transport.impl.sender.Sender.processTimeouts(Sender.java:114) ~[zeebe-transport-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]
6/23/2019 7:35:17 PM	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0

Java client log:

2019-06-23 19:56:14.684  WARN [test-component-workflow-handler,,,] 16428 --- [ult-executor-17] io.zeebe.client.job.poller               : Failed to activated jobs for worker default and job type createVerificationTask

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700)
	at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
	at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
	at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
	at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:500)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:65)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:592)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:508)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:632)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: localhost/0:0:0:0:0:0:0:1:26500
	at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:327)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:665)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:612)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:529)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:491)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 common frames omitted
Caused by: java.net.ConnectException: Connection refused: no further information
	... 11 common frames omitted

The broker has gone away. How much data, and how are you running Zeebe.

For questions like this, please provide your operating system, how you are running Zeebe (are you using Docker?) and as much contextual detail as possible - ideally a MRE (see here).

If you are running in Docker, please be sure to read the docs about the memory configuration.

1 Like

Thanks @jwulf.
I am using docker 18.09 in Linux Rhel-6.

I got the issue. actually the memory is full due to partitions data and snapshots. As of now i am using default properties for data.

I just deployed 10 workflows and 60 instances but it tooks 25 GB (22 GB for partition-0/segments and 3 GB partition-0/state/snapshots).

How to reduce the data and What is the best suitable config changes in cfg.toml file? suggestion Please!

Maybe you have an exporter loaded that is not advancing its record pointer. This will stop the event log truncation from taking place. See the Disk Usage section here.

Check your zeebe.cfg.toml file for exporters.

Are you using the docker-compose configs from zeebe-docker-compose?

Hi @regojoyson. Please check this issue. I had similar problems on my cluster.

1 Like