Resource Exhausted Issue

Hi, I am using Zeebe 0.24.5 and spring Zeebe 0.24.0. I am getting one exception in logs, I am not able to figure out why this error is coming.

2020-11-23 16:29:32.709  WARN 24728 --- [ault-executor-5] io.zeebe.client.job.poller               : Failed to activated jobs for worker ac_check_client_status and job type ac_check_client_status

io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Expected to activate jobs of type 'ac_check_client_status', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
	at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.30.2.jar:1.30.2]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:460) ~[grpc-stub-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.30.2.jar:1.30.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.30.2.jar:1.30.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]


is there any configuration problem…?

As I remember right it is warning of Zeebe backpressure.

@salaboy @Zelldon @MaximMonin

i am using zeebe 0.24.5 version and spring-zeebe 0.24, i am constantly getting this above warning for job workers

what could be the issue…? can we stop this logs by changing any config?

3 Likes

I’m having the same problem using 0.26.0 Does anyone have a fix for this?

There are lots of posts in the Forum about this message. What you should do depends on your scenario. What are you doing / what exactly does your distributed system look like (workers/brokers)? Under which conditions do you see it? What resources does your system have?

From what I’ve learned so far, it’s usually the unusually high disk footprint that causes the exception.

@LLLLimbo do you mean the open file handles?

Not sure , but here’s a screenshot when I used to stress test Zeebe. If you need any other information I will also try to provide it if possible.

It is not necessary disk issues.
I am getting this warnings in stress test, when spamming 10000 workflow in 10 sec and waiting while workers and engine doing job.
It can be gateway point loads.

Try tuning cputhreadcount and ioThreadCount.
Here is my setup:
data:
useMmap: true
threads:
cpuThreadCount: 12
ioThreadCount: 6