Zeebe Broker Backup

We are looking into the use of a Zeebe Cluster on a Openshift/Kubernetes Cluster.

What will be the right way or “Best Practice” for Backup/Restoration of the Data, which is on the Zeebe Cluster?

Is it enough to copy the Data of each running Instance? Or do we have consider something more?

Thanks in advance
Severin

Hi Severin, you run a fault-tolerant cluster with replication. The cluster nodes replicate state between them. When a node fails, no data is lost.

You can export historical data to Elastic Search. This data can be used for analysis, but cannot be used to rehydrate cluster/broker state.

There is no other concept of a “back-up” in Zeebe.

What is the failure scenario you want to address?

Josh

1 Like

Hi Josh
The failure Scenario is a Desaster Recovery Backup. In case a complete cluster will fail, or there was a human error, like application bugs.

Severin

@sbirrer you can always create an exporter of events (as the one mentioned by “The Real @jwulf” ) and then back them up in any preferred storage that you use for disaster scenarios. Does that make sense? If you already have a defined procedure in your company, you can follow the same approach by connecting the Zeebe cluster with those tools using a Custom Exporter.

hi @salaboy
if i re-deploy a zeebe cluster using docker-compose down / up, can my previous workflow data still be called? my test result is: i need to redeploy the previous workflow file.

@walt-liuzw that is absolutely true, because docker compose will automatically delete the storage associated with the broker containers. If you want to keep using the data you will need to configure a volume that doesn’t get deleted.

@salaboy I deployed it using the docker-compose yaml file in https://github.com/zeebe-io/zeebe-docker-compose/tree/master/standalone-gateway.
Which configuration item do I need to modify so that Zeebe Cluster can continue to use the previous process data after redeployment?

now it throws exception:

2019-12-18 16:23:14.0430 ERROR Status(StatusCode=NotFound, Detail=“Command rejected with code ‘CREATE’: Expected to find workflow definition with process ID ‘demo-purchase-order’, but none found”)

@walt-liuzw that is the default behaviour for docker compose, it is automatically removing all the storage for the pods. You need to configure a volume so the brokers can store the data in a directory in the host system.

See here:

this is my docker-compose yaml file:
version: “2”

networks:
zeebe_network:
driver: bridge

volumes:
zeebe_data:
zeebe_elasticsearch_data:

services:
zeebe:
container_name: zeebe
image: camunda/zeebe:0.21.1
environment:
- ZEEBE_LOG_LEVEL=debug
- ZEEBE_STANDALONE_GATEWAY=true
- ZEEBE_GATEWAY_CONTACT_POINT=node0:26502
- ZEEBE_GATEWAY_CLUSTER_PORT=26502
- ZEEBE_GATEWAY_CLUSTER_HOST=zeebe
ports:
- “26500:26500”
- “9600:9600”
volumes:
- zeebe_data:/usr/local/zeebe/data
- ./gateway.cfg.toml:/usr/local/zeebe/conf/gateway.cfg.toml
depends_on:
- elasticsearch
networks:
- zeebe_network
node0:
container_name: zeebe_broker_1
image: camunda/zeebe:0.21.1
environment:
- ZEEBE_LOG_LEVEL=debug
- ZEEBE_NODE_ID=0
- ZEEBE_PARTITIONS_COUNT=8
- ZEEBE_REPLICATION_FACTOR=3
- ZEEBE_CLUSTER_SIZE=3
- ZEEBE_CONTACT_POINTS=node0:26502
ports:
- “26600:26500”
volumes:
- ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
networks:
- zeebe_network
node1:
container_name: zeebe_broker_2
image: camunda/zeebe:0.21.1
environment:
- ZEEBE_LOG_LEVEL=debug
- ZEEBE_NODE_ID=1
- ZEEBE_PARTITIONS_COUNT=8
- ZEEBE_REPLICATION_FACTOR=3
- ZEEBE_CLUSTER_SIZE=3
- ZEEBE_CONTACT_POINTS=node0:26502
volumes:
- ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
networks:
- zeebe_network
depends_on:
- node0
node2:
container_name: zeebe_broker_3
image: camunda/zeebe:0.21.1
environment:
- ZEEBE_LOG_LEVEL=debug
- ZEEBE_NODE_ID=2
- ZEEBE_PARTITIONS_COUNT=8
- ZEEBE_REPLICATION_FACTOR=3
- ZEEBE_CLUSTER_SIZE=3
- ZEEBE_CONTACT_POINTS=node0:26502
volumes:
- ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
networks:
- zeebe_network
depends_on:
- node1
operate:
image: camunda/operate:1.1.0
ports:
- “8080:8080”
volumes:
- …/lib/application.yml:/usr/local/operate/config/application.yml
networks:
- zeebe_network
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch-oss:6.7.1
ports:
- “9200:9200”
environment:
- discovery.type=single-node
- cluster.name=elasticsearch
- “ES_JAVA_OPTS=-Xms512m -Xmx512m”
volumes:
- zeebe_elasticsearch_data:/usr/share/elasticsearch/data
networks:
- zeebe_network

that still throws exceptions after restarting docker-compose
2019-12-18 16:23:14.0430 ERROR Status(StatusCode=NotFound, Detail=“Command rejected with code ‘CREATE’: Expected to find workflow definition with process ID ‘demo-purchase-order’, but none found”)

Did you redeploy the workflow at least once after modifying the docker-compose like this?

hi @jwulf
I think I have already deployed this workflow before. I just upgrade the service / restart the service. Those running workflows should not need to be redeployed.This should be a feature.

@walt-liuzw that is a feature… but the problem is docker compose getting rid of your persistent storage. If you run a broker outside docker you will see that the process definitions and instances don’t go away.

If you are running a cluster like this via docker compose, you need to provide a volume for each of the brokers. You have one persistent volume mounted to the gateway, and you are running three brokers.

Do the same thing for the three broker nodes - create a volume and mount it.

The gateway doesn’t need a persistent volume. The brokers do.

1 Like