Getting "io.grpc.StatusRuntimeException: PERMISSION_DENIED: HTTP status code 403 - invalid token" error

Deployed Spring boot application with Zeebe workers and tested the flow successfully.

After some time, started getting “PERMISSION_DENIED: HTTP status code 403 - invalid token”.

Zeebe Version : 0.25
Spring Cloud Version: 2020.0.0-M4
Spring boot: 2.3.3.RELEASE

Looks like the oauth token is not getting refreshed after expiring. Is there any way to fix this?

 io.zeebe.client.job.poller               : Failed to activated jobs for worker zeebe-worker and job type <>

io.grpc.StatusRuntimeException: PERMISSION_DENIED: HTTP status code 403
invalid content-type: text/html
headers: Metadata(:status=403,server=openresty/,date=Mon, 04 Jan 2021 09:05:20 GMT,content-type=text/html)
invalid token

at io.grpc.Status.asRuntimeException( ~[grpc-api-1.33.0.jar:1.33.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose( ~[grpc-stub-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl.closeObserver( ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl.access$300( ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal( ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext( ~[grpc-core-1.33.0.jar:1.33.0]
at ~[grpc-core-1.33.0.jar:1.33.0]
at ~[grpc-core-1.33.0.jar:1.33.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker( ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$ ~[na:na]
at java.base/ ~[na:na]

How are you running the broker?

Have you configured OAuth on the broker?

Yes, using Camunda Operate for testing.

The issue starts to appear when workers are deployed for longer time. Restarting resolves the issue.

Is it related to this?

Yes similar to this issue, just that we are using spring-zeebe and this issue is for Java Client.

Pretty sure the Spring client is just a wrapper over the Java client.

You probably want to inspect the Spring client source code, and the dependency versions.