How to deploy multiple instance for one job/worker


#1

I want to start multiple instance of one worker to meet my 2 scenarios:

  1. Load balance between these workers.
  2. Dynamic routing to selected worker base on rule.
  3. High availability.

#2

Hi @kitepad,

if you start multiple instances of a worker then the jobs are distributed between the instances. If a job is not complete within a given timeout then it is assigned again to a new worker. So this fulfill the goals 1) and 3).

A dynamic routing between the workers 2) is currently not possible. The routing is only based on the job type. What kind of routing do you need?

Best regards,
Philipp


#3

For example, I start 3 workers to process received orders from 3 regions. Orders from region A will processed by worker 1, orders from region B will processed by worker 2, orders from region C will processed by worker 3.

And another complex scenario is orders from region D will processed by worker 1 and worker 2 or more.

What policy is used to distribute jobs between the instance? Round robin or others?

Thanks.


#4

Currently, the jobs are polled from the broker. So the control is by the clients. In a cluster, the jobs are requested from the brokers round robin.


#5

your case should be done via exclusive gateway ,i think


#6

hi,when a job fail, how does broke choose the next worker for retry? does this algorithm can be customized like exporter?

and another question, how does broker decide retry time interval? it can be customize in process definition? eg, i want to retry job in fibonacci time sequence


#7

Hi @aximo, retry is done by requeuing the job for execution immediately. Whichever worker polls it off the queue will get it. Zeebe uses a worker-pull model.

There is no primitive to tune the retry interval. It is immediate - or “as soon as possible”.

If you need back-off retry, you need to pass a failure state in the variables (or test for the presence/absence of a success state to deal with fatal exceptions in workers that as a result do not set any state), then branch to a timer or the next step based on that state fragment.

You should also handle retry count in the state as well, to prevent an infinite loop.


#8

worker-pull model

very clear and thank you