Got io.grpc.StatusRuntimeException: NOT_FOUND exception in zeebe broker but the instance was working fine

Hi, I am getting some exceptions in zeebe broker but still everything is working fine in handlers and also instance was completed. i want to know why the exception was coming…?

6/2/2019 7:18:30 PMio.grpc.StatusRuntimeException: NOT_FOUND: Command rejected with code 'FAIL': Expected to fail activated job with key '799690', but it does not exist

6/2/2019 7:18:30 PM at io.grpc.Status.asRuntimeException(Status.java:523) ~[grpc-core-1.19.0.jar:1.19.0]

6/2/2019 7:18:30 PM at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:257) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.gateway.EndpointManager.lambda$sendRequest$2(EndpointManager.java:235) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$1(BrokerRequestManager.java:90) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:109) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$6(BrokerRequestManager.java:191) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:35) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:30 PMCaused by: io.zeebe.gateway.cmd.BrokerRejectionException: Command (FAIL) rejected (NOT_FOUND): Expected to fail activated job with key '799690', but it does not exist

6/2/2019 7:18:30 PM ... 10 more

6/2/2019 7:18:33 PM13:48:33.140 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] ERROR io.zeebe.gateway - Error handling gRPC request

6/2/2019 7:18:33 PMio.grpc.StatusRuntimeException: NOT_FOUND: Command rejected with code 'FAIL': Expected to fail activated job with key '799696', but it does not exist

6/2/2019 7:18:33 PM at io.grpc.Status.asRuntimeException(Status.java:523) ~[grpc-core-1.19.0.jar:1.19.0]

6/2/2019 7:18:33 PM at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:257) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.gateway.EndpointManager.lambda$sendRequest$2(EndpointManager.java:235) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$1(BrokerRequestManager.java:90) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:109) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$6(BrokerRequestManager.java:191) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:35) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:33 PMCaused by: io.zeebe.gateway.cmd.BrokerRejectionException: Command (FAIL) rejected (NOT_FOUND): Expected to fail activated job with key '799696', but it does not exist

6/2/2019 7:18:33 PM ... 10 more

6/2/2019 7:18:35 PM13:48:35.646 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-zb-actors-0] ERROR io.zeebe.gateway - Error handling gRPC request

6/2/2019 7:18:35 PMio.grpc.StatusRuntimeException: NOT_FOUND: Command rejected with code 'FAIL': Expected to fail activated job with key '799714', but it does not exist

6/2/2019 7:18:35 PM at io.grpc.Status.asRuntimeException(Status.java:523) ~[grpc-core-1.19.0.jar:1.19.0]

6/2/2019 7:18:35 PM at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:257) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.gateway.EndpointManager.lambda$sendRequest$2(EndpointManager.java:235) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$1(BrokerRequestManager.java:90) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:109) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$6(BrokerRequestManager.java:191) ~[zeebe-gateway-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:35) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:90) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:53) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:189) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:154) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:135) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PM at io.zeebe.util.sched.ActorThread.run(ActorThread.java:112) [zeebe-util-0.17.0.jar:0.17.0]

6/2/2019 7:18:35 PMCaused by: io.zeebe.gateway.cmd.BrokerRejectionException: Command (FAIL) rejected (NOT_FOUND): Expected to fail activated job with key '799714', but it does not exist

6/2/2019 7:18:35 PM ... 10 more

Hi @regojoyson, someone else might recognise this scenario, but I would need more details to diagnose this. Steps to reproduce, or a Minimal Complete Verifiable Example. What is your handler code actually doing?