Can I resize an existing cluster (Feb 2021 tldr; “No”)

slack-archivist · February 3, 2021, 10:42pm

Dag H. Baardsen: When “migrating” from a single node broker to a 3-node broker cluster using docker-compose, how should I migrate the data storage? Is it enough to copy the single node data directory to the first node’s data volume or should it be copied to all three node data directories? Or am I missing something? Help appreciated.

korthout: Hi @Dag H. Baardsen Zeebe does not support changing the clusterSize, partitionCount or replicationFactor for an existing cluster. It is not possible to copy the data.

Dag H. Baardsen: Ah, OK. Thanks for the answer!

Dag H. Baardsen: Does any other strategy for recreating data between cluster configuration changes exist? Like export/import, re-running workflows internally, third party software. Or is there something in the roadmap what will cater for these needs? I guess I’m not the only one with these concerns.

korthout: I’m not aware of any available strategies for this. It is a valid concern, but at this time we’re not working on a solution for it. Generally, we recommend to keep your development and production environment as similar as possible. The issue might become more important to us in the future, so if it is important to you, please create an issue for it on github

Dag H. Baardsen: I will do. BTW, this is not a “bring my data from development to production”, but more of a “my resources are exhausted in production, so I have to expand my cluster” concern.

korthout: Ah clear. Thanks for bringing it up. We do have an open feature request for dynamic resizing of an existing cluster. It might make sense to add your use case as a comment there. It’s likely that this feature will not be implemented any time soon. However, I recommend commenting your use case, because perhaps a workaround can be made available more easily while we don’t have this massive feature yet. It’s worth discussing IMO.

Josh Wulf: See also: https://github.com/zeebe-io/zeebe/issues/3254

Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!

If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.

slack-archivist · February 4, 2021, 7:43am

Dag H. Baardsen: Gotta love that slack-archivist profile photo!

jwulf · February 14, 2021, 12:14am

See this open feature request, and feel free to upvote:

github.com/camunda/zeebe

Dynamic resize of an existing zeebe cluster

opened 04:42PM - 27 Apr 20 UTC

deepthidevaki

kind/feature scope/broker

**Is your feature request related to a problem? Please describe.** I want to dy…namically scale a zeebe cluster. At present the cluster size is configured once when the cluster is initially created. It is not possible to add or remove nodes to address scalability issues. **Describe the solution you'd like** I can add or remove brokers into a running zeebe cluster and the cluster should redistribute the partitions. For example, Initially my cluster has 1 broker and N partitions with no replicas. Later we add one more broker. Then N/2 partitions (with all data) from first broker will be moved to the new one. From this point it behaves as if the cluster size is 2. If the second broker leaves the cluster, its partitions and data should be moved back to the first one. **Describe alternatives you've considered** * add new brokers with new set of partitions. IMO this is more challenging as we have some data that need to be transferred to new partitions. Scaling down is also difficult as we have to merge data from different partitions. I would also assume the proposed feature is a prerequisite to reconfiguring number of partitions. * Stop the cluster. Manually copy partitions data to target nodes. Restart with new configuration. Not sure if this works out of the box now. Even if it works it is error prone and the scaling is not dynamic. **Additional context** Why is this useful? This is first step towards making Zeebe dynamically scalable. The absolute performance (throughput vs latency) we get from a broker with 1 partition per core and N partitions per core can be similar when there is not too much load. As the load increases we can scale the cluster by adding more brokers and distributing the partitions thus distributing the load. Scale down also works similarly. POC: There is a poc for a solution [here](https://github.com/zeebe-io/zeebe/tree/dd-poc-scaling.) It works as follows: 1. Initial : clusterSize = 2 (nodeIds 0, 1), partitions = 4, replication factor = 1 2. We start one new broker nodeId = 2 3. Send a command `zbctl updateClusterSize init 2` to all 3 brokers. 4. Each broker on receiving the above command recalculates the partition matrix initially it looked like this n\p| 1 | 2 | 3 | 4 | -- | - | - | - | - | 0| x| | x| | 1| |x | |x| Which indicates that node 0 has partition 1 and 3 and node 1 has partition 2 After: n\p| 1 | 2 | 3 | 4 | -- | - | - | - | - | 0| x| | L|J | 1| |x | |L| 2| | | J| | J indicates new partitions to join L indicates partitions to leave First each node joins the partition that it is not yet member of. node 0 joins partition 4 and node 2 joins partition 3. node 0 joins partition 4 by sending a join request to node 3 which is the current member of that group. Raft implementation allows dynamic reconfiguration. Once the join process complete (it is a multi step process), partition 3 and partition 4 has 2 replicas - the new member and the old members. Quorum is also updated accordingly. Any new appends to the raft log should be replicated to the new quorum. At this point the new member might not have catch up with the log. But it has started replicating and will be an active member of the quorum. 5. Once the join is complete, we send another command 'updateClusterSize commit' to all nodes. 6. Each broker on receiving the command leaves the partition that it is no more member of. It is safe to do that as the new members are already part of the quorum and will be replicating. The raft implementation has a clean process to leave. The configuration change will be committed to the quorum before leave happens so that a leader can safely leave without any data loss. The new partition matrix looks as follows: n\p| 1 | 2 | 3 | 4 | -- | - | - | - | - | 0| x| | |x | 1| |x | | | 2| | | x| Current limitations with the POC * configuration change is not persisted. So a restart of the cluster will use the old config and might result in data loss. * Also configuration change is not handled well by many modules, as it expects static configuration. For example, Topology is not updated when a broker leaves a partition. * Join and leave are controlled manually by sending two commands to each broker. This is error prone and not safe. Ideally we should just send one command `join <newMembers>` the cluster should coordinate itself and tell us when the process is done.

system · January 31, 2024, 10:11am