Unable to resize the PVC for Zeebe Data

Camunda 8 Topics Discussion & Questions
1

Hello Team,

I have run out of Space on the Zeebe Data PVCs, due to which getting the following issues.
[raft-server-1-raft-partition-partition-3] ERROR io.atomix.raft.roles.LeaderRole - RaftServer{raft-partition-partition-3}{role=LEADER} - Error on appending entry InitializeEntry{term=5837, timestamp=2020-12-23 07:55:27,240}, retry.
io.atomix.storage.StorageException: java.io.IOException: No space left on device

I reconfirmed by logging into the pods and running the “df” command
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nvme4n1 10190100 10173716 0 100% /usr/local/zeebe/data

I have also tried editing the PVC associated with the same and getting the below error.
$ kubectl edit pvc data-zeebe-zeebe-0 -n zeebe-test
error: persistentvolumeclaims “data-zeebe-zeebe-0” could not be patched: persistentvolumeclaims “data-zeebe-zeebe-0” is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

So, wanted your help into two things

  1. To be able to resize the PVC if possible.
  2. While newly deploying Zeebe is there a configuration which can be changed so that a PVC with higher capacity is created.

Thanks.

2

@salaboy may know how to deploy with a bigger PVC.

3

Hi Karthik.prakash,
I had the some problem, I resolved follow this steps:

  1. Enable allowVolumeExpansion feature on: storageclasses.storage.k8s.io.
    1.1 kubectl get storageclasses.storage.k8s.io and kubectl edit storageclasses.storage.k8s.io <NAME>
  2. Resize your physical disk, if cloud provider like AWS resize your volume on Console or API.
  3. Resize storage on Persistent Volumes: kubectl edit persistentvolume pvc-xxxxxxxx
  4. Resize storage on Persistent Volumes Claims: kubectl edit persistentvolumeclaims

Kind regards.

4