How does Zeebe handle data migrations between broker versions in production?

Juraj Machac: Hi guys,
We are thinking about using Zeebe for service orchestration, hence I’m interested in the production readiness.
I’ve seen the blog which describes what does Zeebe label “production ready” for version 0.20.0, and while that definition works for me, I was a bit worried when I found that since release 0.22.0 it’s not possible to migrate data from older versions. I suppose we may need to retain the data of some business processes for audit purposes, hence I want to ask if you plan to support migrating such data in future? Or is there at least some tool how to extract the data from older version to some fairly readable format? Thanks.

Nicolas: Hey Juraj! As Zeebe does not use an external data store by default, the way to persist data for auditing purposes is to use exporters (https://docs.zeebe.io/basics/exporters.html). It would otherwise be quite costly to keep all the data for Zeebe, and it’s not a goal of ours to be a data warehouse, hence the need to export that data so we can locally get rid of it.

Currently, we only officially support exporting data to Elasticsearch, but there are various community projects for different target systems (https://github.com/zeebe-io/awesome-zeebe#exporters) - we hope however that we’ve created a simple enough API such that it is easy to create your own exporter if need be :slight_smile:

Juraj Machac: Ah I get it :slightly_smiling_face: cool!
Will look into the exporters, thanks

Nicolas: If you use Operate (https://docs.zeebe.io/operate-user-guide/index.html), then all Zeebe data is exported to Elasticsearch, so in that sense your data lives in Elasticsearch and is available for as long as possible - meaning depending on how you configured your Elasticsearch cluster

Juraj Machac: In terms of licensing - I understand that Zeebe should be free to use commercially provided that we don’t use it to power a workflow service or similar.
For Operate, I understand that we’ll need an enterprise license. Is there any pricing available or does this all go individually through support?

Nicolas: It’s currently done individually, though it’s free for evaluation purposes - you can also test drive it if you join the cloud’s public beta

Juraj Machac: Yeah, joined that. Very useful for test driving :slightly_smiling_face: though we’ll have to host it eventually, hence I’m asking.
Thanks for info

Aravindh RS: So zeebe is cloud based state machine like AWS step function?

Josh Wulf: @Juraj Machac , here is a article on writing an exporter: https://zeebe.io/blog/2020/02/incident-exporter/|https://zeebe.io/blog/2020/02/incident-exporter/

Josh Wulf: @Aravindh RS yes, it plays the same role in a system. But with BPMN instead of YAML.

2 Likes