Hello, the project drew my attention. Seems to be WF engine done right.
Looking at the arch document I am not sure I understand why you had to implement brokers from ground up, why not using kafka itself? There should be some valid reasons, do you mind listing them?
Thanks,
Milan
Hi @magaton, welcome, and thank you for the question. Here are a couple thoughts:
• The scope and target use cases of Kafka and Zeebe are quite different, and generally speaking, Kafka simply wasn’t the right fit; Kafka, as a storage system and message bus, includes many features that aren’t necessary for Zeebe, whose primary responsibility is managing the state of running workflow instances. Kafka also comes with some of its own operational complexity, e.g. the ZooKeeper requirement. We think of Zeebe as a nice complement to Kafka for users with a certain type of problem, though, and we’ve written a bit about that.
• I will say that we try to use existing libraries and products in Zeebe whenever possible. For example, we started using gRPC with the 0.12 release. And we’re currently looking into replacing some other homegrown components with well-established libraries, too. Our exporter system works with Elasticsearch out-of-the-box and is intended to be easy to use with existing storage tools.
Let me know if you have any follow-up questions!
Best,
Mike
@wints, thanks very much for the response.
I had read Bernd’s blog about zeebe and kafka connector before along with a couple of other ones that got me intrigued. Finally WF engine that scales 
But still if you look at kafka and zeebe-broker, is there any feature in zeebe-broker that cannot / (is hard to) be developed / configured on top of kafka?
Replication, leader election, partitions, distributed data store are difficult problems and kafka is an example of how they can be successfully solved. I understand additional complexity that kafka adds up (like zookeeper), but is there any other reason you went with your own broker implementation?
Hi again, I would really love to hear your view if this is not too much trouble for you.
We seem to have a perfect use case for zeebe in our product, but before we start with proper evaluation, I’d really like to know why you made such a big decision to develop zeebe broker and basically reimplement a significant subset of kafka from scratch.
Or, looking from different perspective, if we are about to start developing distributed WF engine (with camunda, activiti or flowable, doesn’t matter) using vanilla kafka and producer/consumer api instead of grpc and zeebe broker where can we possibly end? I am sure you guys evaluated things very thoroughly before going such route.
Thanks in advance,
Milan
Hi @magaton, gladly! I hope this helps.
It’s very likely possible to build a workflow engine on top of e.g. Kafka + Kafka Streams, but there were some other considerations that made this approach a nonstarter for us:
• How can we build Zeebe so it’s lightweight and easy to adopt? Along with horizontal scalability, one of the reasons we’re building Zeebe the way we are (paritions + replications vs. relational database) is to limit the number of components required to use Zeebe. Requiring Kafka (and therefore ZooKeeper) to run the engine would have been a lot to ask from potential users, especially for those who aren’t using Kafka already, and we didn’t want to make that a requirement.
• How can we balance flexibility and control over Zeebe with efficiency and maintainability? Building an engine on top of Kafka would require us to put a lot of faith into a big and complex product that we don’t have much (if any) control over. That said, it wouldn’t have made sense to build every single Zeebe component from scratch, and we pull in other awesome open-source frameworks and tools where it’s possible. Zeebe relies heavily on RocksDB and gRPC, and we’re also working on integrating Atomix to cover some core functionality. At least for now, we’ve found a balance that allows us to build the Zeebe that we want to build and that we believe will solve our users’ problems in the long-term while saving time and effort with existing libs and frameworks.
I know this doesn’t get to the question “if trying to implement a workflow engine with vanilla Kafka, where would someone hit a major roadblock?”, but to be honest, we don’t really know the answer to that because it wasn’t quite the right approach for our goals with Zeebe.
Let me know if you have any follow-up questions.
Best,
Mike