what’s the broker ratelimit strategy?
I am running a performence testing with single zeebe broker
The linux machine has 16CPU,36G MEM, 500SATA。
I changed the zeebe broker partions from 8 -12 case by case, but the TPS seems always could only reach 130, does zeebe has any ratelimit strategy? how could i improve this
1 Like
Hi @Young200808,
See here for the back pressure configuration for 0.23.2.
I ran a quick test that is useful only for comparing between Zeebe releases. The code is here.
Test Setup
This a BPMN model with a single service task, and no workers running.
The broker is configured as the default: 1 partition, replication factor 1.
I tested 0.23.2 with back pressure enabled and disabled, but it didn’t make much of a difference.
I tested 0.22.4 and 0.21.1 with their default back pressure configuration.
Results
0.21.1 is significantly faster (up to 10x), but you can see it slowing down over time. It also has a very high CPU usage.
0.22.4 and 0.23.2 are slower, but more consistent and predictable.
I can get the following TPS, using the Node client on my laptop:
On 0.23.2, back pressure enabled:
Time : Total | wf/s | average/s
5s : 77 | 15 | 15/sec
10s : 145 | 14 | 15/sec
15s : 217 | 14 | 14/sec
20s : 285 | 14 | 14/sec
25s : 355 | 14 | 14/sec
30s : 424 | 14 | 14/sec
35s : 481 | 11 | 14/sec
40s : 553 | 14 | 14/sec
45s : 625 | 14 | 14/sec
50s : 696 | 14 | 14/sec
55s : 763 | 13 | 14/sec
60s : 834 | 14 | 14/sec
65s : 904 | 14 | 14/sec
70s : 971 | 13 | 14/sec
75s : 1026 | 11 | 14/sec
80s : 1089 | 13 | 14/sec
85s : 1156 | 13 | 14/sec
90s : 1225 | 14 | 14/sec
95s : 1294 | 14 | 14/sec
100s : 1364 | 14 | 14/sec
105s : 1430 | 13 | 14/sec
110s : 1496 | 13 | 14/sec
115s : 1565 | 14 | 14/sec
120s : 1634 | 14 | 14/sec
With back pressure disabled on 0.23.2, I still only get 14-15/sec.
On 0.22.4:
Time : Total | wf/s | average/s
5s : 69 | 14 | 14/sec
10s : 141 | 14 | 14/sec
15s : 213 | 14 | 14/sec
20s : 286 | 15 | 14/sec
25s : 361 | 15 | 14/sec
30s : 436 | 15 | 15/sec
35s : 511 | 15 | 15/sec
40s : 587 | 15 | 15/sec
45s : 663 | 15 | 15/sec
50s : 740 | 15 | 15/sec
55s : 817 | 15 | 15/sec
60s : 872 | 11 | 15/sec
65s : 948 | 15 | 15/sec
70s : 1025 | 15 | 15/sec
75s : 1102 | 15 | 15/sec
80s : 1180 | 16 | 15/sec
85s : 1258 | 16 | 15/sec
90s : 1335 | 15 | 15/sec
95s : 1412 | 15 | 15/sec
100s : 1489 | 15 | 15/sec
105s : 1566 | 15 | 15/sec
110s : 1642 | 15 | 15/sec
115s : 1719 | 15 | 15/sec
120s : 1794 | 15 | 15/sec
On 0.21.1:
Time : Total | wf/s | average/s
5s : 418 | 84 | 84/sec
10s : 921 | 101 | 92/sec
15s : 1492 | 114 | 99/sec
20s : 2037 | 109 | 102/sec
25s : 2639 | 120 | 106/sec
30s : 3254 | 123 | 108/sec
35s : 3866 | 122 | 110/sec
40s : 4453 | 117 | 111/sec
45s : 4978 | 105 | 111/sec
50s : 5418 | 88 | 108/sec
55s : 5805 | 77 | 106/sec
60s : 6126 | 64 | 102/sec
65s : 6362 | 47 | 98/sec
70s : 6485 | 25 | 93/sec
75s : 6580 | 19 | 88/sec
80s : 6674 | 19 | 83/sec
85s : 6770 | 19 | 80/sec
90s : 6854 | 17 | 76/sec
95s : 6950 | 19 | 73/sec
100s : 7044 | 19 | 70/sec
105s : 7144 | 20 | 68/sec
110s : 7393 | 50 | 67/sec
115s : 7727 | 67 | 67/sec
120s : 8151 | 85 | 68/secHere is a longer run with 0.21.1, around 10 mins. It has a fairly predictable average TPS, but you can see that it ebbs and flows:
Time : Total | wf/s | average/s
5s : 680 | 136 | 136/sec
10s : 1517 | 167 | 152/sec
15s : 2379 | 172 | 159/sec
20s : 3235 | 171 | 162/sec
25s : 4080 | 169 | 163/sec
30s : 4846 | 153 | 162/sec
35s : 5572 | 145 | 159/sec
40s : 6198 | 125 | 155/sec
45s : 6744 | 109 | 150/sec
50s : 7127 | 77 | 143/sec
55s : 7418 | 58 | 135/sec
60s : 7566 | 30 | 126/sec
65s : 7668 | 20 | 118/sec
70s : 7762 | 19 | 111/sec
75s : 7862 | 20 | 105/sec
80s : 7955 | 19 | 99/sec
85s : 8050 | 19 | 95/sec
90s : 8148 | 20 | 91/sec
95s : 8242 | 19 | 87/sec
100s : 8340 | 20 | 83/sec
105s : 8432 | 18 | 80/sec
110s : 8524 | 18 | 77/sec
115s : 8797 | 55 | 76/sec
120s : 9215 | 84 | 77/sec
125s : 9552 | 67 | 76/sec
130s : 10060 | 102 | 77/sec
135s : 10744 | 137 | 80/sec
140s : 11492 | 150 | 82/sec
145s : 12291 | 160 | 85/sec
150s : 13129 | 168 | 88/sec
155s : 13966 | 167 | 90/sec
160s : 14833 | 173 | 93/sec
165s : 15691 | 172 | 95/sec
170s : 16469 | 156 | 97/sec
175s : 17281 | 162 | 99/sec
180s : 18016 | 147 | 100/sec
185s : 18737 | 144 | 101/sec
190s : 19399 | 132 | 102/sec
195s : 20000 | 120 | 103/sec
200s : 20558 | 112 | 103/sec
205s : 21028 | 94 | 103/sec
210s : 21386 | 72 | 102/sec
215s : 21606 | 44 | 100/sec
220s : 21707 | 20 | 99/sec
225s : 21791 | 17 | 97/sec
230s : 21875 | 17 | 95/sec
235s : 21977 | 20 | 94/sec
240s : 22079 | 20 | 92/sec
245s : 22183 | 21 | 91/sec
250s : 22283 | 20 | 89/sec
255s : 22378 | 19 | 88/sec
260s : 22580 | 40 | 87/sec
265s : 22962 | 76 | 87/sec
270s : 23462 | 100 | 87/sec
275s : 24065 | 121 | 88/sec
280s : 24749 | 137 | 88/sec
285s : 25530 | 156 | 90/sec
290s : 26330 | 160 | 91/sec
295s : 27169 | 168 | 92/sec
300s : 27965 | 159 | 93/sec
305s : 28791 | 165 | 94/sec
310s : 29623 | 166 | 96/sec
315s : 30452 | 166 | 97/sec
320s : 31245 | 159 | 98/sec
325s : 31995 | 150 | 98/sec
330s : 32675 | 136 | 99/sec
335s : 33338 | 133 | 100/sec
340s : 33957 | 124 | 100/sec
345s : 34501 | 109 | 100/sec
350s : 34977 | 95 | 100/sec
355s : 35389 | 82 | 100/sec
360s : 35738 | 70 | 99/sec
365s : 35971 | 47 | 99/sec
370s : 36080 | 22 | 98/sec
375s : 36164 | 17 | 96/sec
380s : 36234 | 14 | 95/sec
385s : 36321 | 17 | 94/sec
390s : 36407 | 17 | 93/sec
395s : 36491 | 17 | 92/sec
400s : 36566 | 15 | 91/sec
405s : 36653 | 17 | 91/sec
410s : 36728 | 15 | 90/sec
415s : 36816 | 18 | 89/sec
420s : 36947 | 26 | 88/sec
425s : 37243 | 59 | 88/sec
430s : 37660 | 83 | 88/sec
435s : 38164 | 101 | 88/sec
440s : 38761 | 119 | 88/sec
445s : 39420 | 132 | 89/sec
450s : 40099 | 136 | 89/sec
455s : 40823 | 145 | 90/sec
460s : 41525 | 140 | 90/sec
465s : 42272 | 149 | 91/sec
470s : 42954 | 136 | 91/sec
475s : 43649 | 139 | 92/sec
480s : 44356 | 141 | 92/sec
485s : 45011 | 131 | 93/sec
490s : 45651 | 128 | 93/sec
495s : 46253 | 120 | 93/sec
500s : 46721 | 94 | 93/sec
505s : 47160 | 88 | 93/sec
510s : 47504 | 69 | 93/sec
515s : 47782 | 56 | 93/sec
520s : 48024 | 48 | 92/sec
525s : 48186 | 32 | 92/sec
530s : 48282 | 19 | 91/sec
535s : 48366 | 17 | 90/sec
540s : 48449 | 17 | 90/sec
545s : 48531 | 16 | 89/sec
550s : 48596 | 13 | 88/sec
555s : 48680 | 17 | 88/sec
560s : 48758 | 16 | 87/sec
565s : 48846 | 18 | 86/sec
570s : 48996 | 30 | 86/sec
575s : 49309 | 63 | 86/sec
580s : 49706 | 79 | 86/sec
585s : 50197 | 98 | 86/sec
590s : 50731 | 107 | 86/sec
595s : 51345 | 123 | 86/sec
600s : 51992 | 129 | 87/sec
605s : 52668 | 135 | 87/sec
610s : 53338 | 134 | 87/sec
615s : 54010 | 134 | 88/secregarding to 0.23.2, why it has the same performance no matter back pressure is on or off?
Then what the aim of back pressure, seems it’s not taking effect as no matter on or off, it has same performance
Because in the specific configuration that I am running it, with the gateway, the broker, and a single requestor all on the same machine, the bottleneck is not in the stream processor.
I might get a different result with more CPUs and more requestors.
“Your mileage may vary”.
I got your point, thanks.
1 Like