Back in September last year, I posted about the Zeebe Garage Benchmark. It’s a rough and ready benchmarking tool that allows you to compare Zeebe releases to get a sense of their relative performance.
Last year, I published results for Zeebe versions from 0.22 to 0.25.
I just ran it on 0.26.1 and 1.0.0-alpha1.
An obvious caveat: YMMV (in fact: YMWV - Your Mileage Will Vary). The only reliable benchmark for how a system will perform on your exact hardware with the exact configuration that you use is the one that you run on your exact hardware with the exact configuration you use. Watch these two videos I did with Camunda Consultant Falko Menge to understand Zeebe benchmarking and performance tuning: Video 1 Video 2.
Bearing that in mind, here are the TPS results for Zeebe running on a single broker with embedded gateway and 3 partitions. Bear in mind that the absolute number is meaningless - it depends on the hardware. But because all tests are run on the same hardware, it is the relative performance that is of interest.
Benchmark Results
Partitions: 3
| Version | Average TPS |
|---|---|
| 0.22.5 | 56 |
| 0.23.5 | 27 |
| 0.24.1 | 22 |
| 0.25.4 | 24 |
| 0.26.1 | 85 |
| 1.0.0-alpha1 | 99 |
Interpretation of the results
Zeebe 0.22 was the fastest release for a long time. With Zeebe 0.23, the engineering team undertook a major push for stabilisation, and it came at the cost of performance. They had a clear picture of what Zeebe did, and a good understanding how they needed to refactor it to make it more maintainable and easier to address stability issues. So they did that, and performance took a back seat.
Well, it looks like performance is back!
Going from these results, Zeebe 0.26 and Zeebe 1.0.0-alpha1 have not only recovered the lost performance, but increased it - at least in some scenarios.
Your production workload is not doing what this TPS test does. You have workers, and your BPMN models probably have messages and gateways. You probably have your system configured for redundancy. The Garage benchmark doesn’t test how these perform at all (yet). It just tests how fast you can start process instances.
You want to build your own test benchmark that tests the relevant parts of your setup (or feel free to fork Zeebe Garage Benchmark and share your work via a PR).
But at first glance, it looks very promising. In at least this corner of the bigger multi-dimensional picture of performance, Zeebe 1.0.0 is looking like the fastest Zeebe release yet!