Getting "io.grpc.StatusRuntimeException: PERMISSION_DENIED: HTTP status code 403 - invalid token" error

Camunda 8 Topics Discussion & Questions
1

Deployed Spring boot application with Zeebe workers and tested the flow successfully.

After some time, started getting “PERMISSION_DENIED: HTTP status code 403 - invalid token”.

Zeebe Version : 0.25
Spring Cloud Version: 2020.0.0-M4
Spring boot: 2.3.3.RELEASE

Looks like the oauth token is not getting refreshed after expiring. Is there any way to fix this?

 io.zeebe.client.job.poller               : Failed to activated jobs for worker zeebe-worker and job type <>

io.grpc.StatusRuntimeException: PERMISSION_DENIED: HTTP status code 403
invalid content-type: text/html
headers: Metadata(:status=403,server=openresty/1.17.8.2,date=Mon, 04 Jan 2021 09:05:20 GMT,content-type=text/html)
DATA-----------------------------
invalid token

at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.33.0.jar:1.33.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:616) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:802) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:781) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.33.0.jar:1.33.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:832) ~[na:na]
2

How are you running the broker?

Have you configured OAuth on the broker?

3

Yes, using Camunda Operate for testing.

The issue starts to appear when workers are deployed for longer time. Restarting resolves the issue.

4

Is it related to this? https://github.com/zeebe-io/zeebe/issues/3768

5

Yes similar to this issue, just that we are using spring-zeebe and this issue is for Java Client.

6

Pretty sure the Spring client is just a wrapper over the Java client.

You probably want to inspect the Spring client source code, and the dependency versions.

7