How to deploy multiple instance for one job/worker

Camunda 8 Topics Discussion & Questions
1

I want to start multiple instance of one worker to meet my 2 scenarios:

  1. Load balance between these workers.
  2. Dynamic routing to selected worker base on rule.
  3. High availability.
1 Like
2

Hi @kitepad,

if you start multiple instances of a worker then the jobs are distributed between the instances. If a job is not complete within a given timeout then it is assigned again to a new worker. So this fulfill the goals 1) and 3).

A dynamic routing between the workers 2) is currently not possible. The routing is only based on the job type. What kind of routing do you need?

Best regards,
Philipp

3

For example, I start 3 workers to process received orders from 3 regions. Orders from region A will processed by worker 1, orders from region B will processed by worker 2, orders from region C will processed by worker 3.

And another complex scenario is orders from region D will processed by worker 1 and worker 2 or more.

What policy is used to distribute jobs between the instance? Round robin or others?

Thanks.

4

Currently, the jobs are polled from the broker. So the control is by the clients. In a cluster, the jobs are requested from the brokers round robin.

5

your case should be done via exclusive gateway ļ¼Œi think

6

hiļ¼Œwhen a job fail, how does broke choose the next worker for retry? does this algorithm can be customized like exporter?

and another question, how does broker decide retry time intervalļ¼Ÿ it can be customize in process definition? eg, i want to retry job in fibonacci time sequence

7

Hi @aximo, retry is done by requeuing the job for execution immediately. Whichever worker polls it off the queue will get it. Zeebe uses a worker-pull model.

There is no primitive to tune the retry interval. It is immediate - or ā€œas soon as possibleā€.

If you need back-off retry, you need to pass a failure state in the variables (or test for the presence/absence of a success state to deal with fatal exceptions in workers that as a result do not set any state), then branch to a timer or the next step based on that state fragment.

You should also handle retry count in the state as well, to prevent an infinite loop.

8

worker-pull model

very clear and thank you

1 Like
9