Hi @magaton, gladly! I hope this helps.
It’s very likely possible to build a workflow engine on top of e.g. Kafka + Kafka Streams, but there were some other considerations that made this approach a nonstarter for us:
• How can we build Zeebe so it’s lightweight and easy to adopt? Along with horizontal scalability, one of the reasons we’re building Zeebe the way we are (paritions + replications vs. relational database) is to limit the number of components required to use Zeebe. Requiring Kafka (and therefore ZooKeeper) to run the engine would have been a lot to ask from potential users, especially for those who aren’t using Kafka already, and we didn’t want to make that a requirement.
• How can we balance flexibility and control over Zeebe with efficiency and maintainability? Building an engine on top of Kafka would require us to put a lot of faith into a big and complex product that we don’t have much (if any) control over. That said, it wouldn’t have made sense to build every single Zeebe component from scratch, and we pull in other awesome open-source frameworks and tools where it’s possible. Zeebe relies heavily on RocksDB and gRPC, and we’re also working on integrating Atomix to cover some core functionality. At least for now, we’ve found a balance that allows us to build the Zeebe that we want to build and that we believe will solve our users’ problems in the long-term while saving time and effort with existing libs and frameworks.
I know this doesn’t get to the question “if trying to implement a workflow engine with vanilla Kafka, where would someone hit a major roadblock?”, but to be honest, we don’t really know the answer to that because it wasn’t quite the right approach for our goals with Zeebe.
Let me know if you have any follow-up questions.