How can I configure how many maximum workflow instances can concurrently run on a Zeebe cluster?

Camunda 8 Topics Discussion & Questions
1

How can I set a limit on number of concurrent running workflow instances in a Zeebe cluster?

Also, can I assign a relative priority to indicate execution priority of workflow instances on Zeebe cluster?

2

Hey,

I’m unsure if I get this question right.
You could simply not start so many workflow instances?

What is the use case for this? What do you want to achieve? Currently execution priority is not supported.

Greets
Chris

3

My use case is to use Zeebe to configure workflow to create document > store document > identify recipient > deliver E or Paper (based on recipient delivery preference)
This workflow can be used for a single request (an account is opened, send welcome letter) or batch (collect all trades for the day and deliver trade confirms).

Lifecycle for workflow instance can be few seconds to tens of minutes depending on document size and complexity.

So a single workflow has to be invoked with different payload values (for simplicity ignore input/output payload schema difference).

At any given time a workflow can have multiple running instances. My question is how many workflow instances can Zeebe run for a single workflow at same time?

4

@Zelldon - Can you review the use case outlined and opine on concurrency aspect?

5

Hi @vermaan, we are going to spend time this year building standard benchmarks so we have a consistent, ongoing measure of Zeebe performance on a handful of key metrics, but even these won’t be perfectly representative of any single use case. I don’t think we can put forth a valid estimate of how many concurrent instances Zeebe will be able to handle for your use case (and would strongly encourage that you test it). On a single node, you might see hundreds to low thousands of new workflow instances started per second without issue, but again, definitely not certain about this and it depends on a number of different characteristics of the machine, model, payloads etc.

It’s worth saying that Zeebe’s main responsibility is managing the state of workflow instances–you’re not deploying any worker code embedded in Zeebe (as you can do with e.g. Camunda BPM), and jobs are carried out by services that are separate from Zeebe. And so these are services that can be scaled up and down separately from Zeebe, too.

Based on conversations we’ve had so far, in a lot of cases, we expect bottlenecks to occur with worker services rather than within Zeebe. And ideally, these worker services will be pretty straightforward to scale up and down. Hope that helps.

Best,
Mike

6

Hi Mike,

Thanks for your response.

I agree that in most cases bottlenecks will occur with worker services rather than within Zeebe.

In some cases, there may be challenges linearly scaling worker services by creating more instances - specially when they are accessing a shared resource or a 3rd party product/service which may not linearly scale. As a workaround, we may want to ensure that there must never run be more than X instances of such a workflow.

  1. One way to achieve this is to put the logic in client program to manage the number of requests submitted to Zeebe for such a workflow.

  2. Other way to achieve this would be to have a configuration in Zeebe - akin to a thread pool size - so that Zeebe will never have more than X concurrently running workflows.

Netflix Conductor has a a setting to control number of worker threads at node level (not per workflow), which puts a limit on number of concurrent tasks that Conductor can run.

I am trying to understand which of the 2 outlined options can be used with Zeebe.

Anuraag

7

Thanks for the context, Anuraag–it’s helpful and I see what you’re getting at. There’s no way to define this “number of running instances” limit within Zeebe right now, so the logic would indeed need to be managed in the client program. I’ll ask around and see if anyone has an idea of the best way to implement this sort of logic (or you might already know how you’d approach it).

Best,
Mike

8

Thanks Mike. I have a fair idea on how to do this externally in the client program. It will be awesome if as part of your product roadmap, this can be considered a feature down the line to configure maximum number of workflow instances Zeebe will spawn for a workflow.

9

Thanks for the input, Anuraag. I will make sure it’s on our radar. This sort of feedback is really helpful for us.

I’ll mention that if you’d be interested in contributing to Zeebe at any point (on this feature or something totally different), we have a contributor’s guide here: https://github.com/zeebe-io/zeebe/blob/master/CONTRIBUTING.md and would be happy to help with the “nuts and bolts” of how to contribute. Otherwise, please do continue to mention ideas or suggestions.

1 Like
10

Hi @vermaan, after reading through your post again, I think Zeebe might already offer a solution for what you’re looking for. The client allows you to set a numJobWorkerExecutionThreads parameter (default value is 1), and in limiting the number of workers running at any given time + using this parameter, you should be able to control the load on any shared resources that the client relies on. You can read more about it in the Javadoc here: https://static.javadoc.io/io.zeebe/zeebe-client-java/0.14.0/io/zeebe/client/ZeebeClientBuilder.html#numJobWorkerExecutionThreads-int-.

The Zeebe broker itself isn’t executing the task in the workflow–it’s simply storing the state of active workflow instances–so the number of workflow instances running in parallel shouldn’t impact these shared systems so long as you have a way to limit external job worker capacity.

Let me know if this makes sense.

2 Likes
11

Yes this makes sense. Thanks for your response.

12

How can I scale up my workers? And
One worker can work on one job only?

13