Hope you are doing good.
Looks like I have posted a lot of messages thus things are getting messed up, Sorry for that.
So after changing configuration to 5 nodes and increasing management thread to 5 we were unable to connect to our zeebe broker but then when we reverted the management thread to 1 it was all fine.
You can refer to the comments above where I have listed all the Scenarios which we have uncovered with our load.
To be specific see
Zeebe Low Performance (Configuration Type 1)
Zeebe Low Performance (Configuration Type 2)
All the configuration details are within the comment section.
After you change the configuration to 5 nodes you mentioned dealine exceeded. How often do you see this? Do you use then a standalone gateway?
Yes till when we changed the management thread to 1 it was always there.
We are using standalone gateway with 3 replicas.
It would help to understand the setup better if you always share your complete configuration, e.g. the values file, the helm version etc.
- effect: NoExecute
- key: role
accessModes: [ "ReadWriteOnce" ]
esJavaOpts: "-Xmx4g -Xms4g"
- effect: NoExecute
I revisited the previous posts and saw that you mentioned you are starting workflow instances via messages is this still the case? Are you using always the same
correlationKey ? If you use always the same correlationKey then it will be published on the same partition.
Yes we are using message start event to start our workflow
correlationKey is unique for a particular instance been started (that is our use case and we can’t devoid)
As I said correlationKey is unique for a particular instance and correlationKey is different for different instance.
The job workers have 8 threads, but only
32 max jobs activate is this the case? Maybe you can increase the number as well.
The above configuration was default but we updated the configuration on the worker side as
@ZeebeWorker(`type` = "tasting", name = "tasting", maxJobsActive = 200)
with maxJobsActive as 200
In you scenario descriptions what do you mean with “request processed = 20454 (Camunda operate)”*
You see that many instances in operate?
Grafana provides a graph for metrics result for
total_number_of_requests been fired in Zeebe.
Ideally these request should be equal to the number of instance creation request I am sending from service but when we verified that the number was vague in a way it gives us more number of requests than actually fired.
Thus as a fallback we verified total number of instances completed on
Operate which was genuine and we have noted down instances completed from the
Operate data itself not the grafana metrics.
We wanted to achieve a result of 1000 instance created / completed per seconds. Please let us know whatever configuration is required to achieve this number. We can even share you with the benchmark results but if someone can actively support us then we would really appreciate that since we have limited time now.