0.23.1 - "Failed to obtain a pending snapshot directory for position ..."

I see this periodically in the broker log for Zeebe 0.23.1 running on a local machine with one broker:

2020-05-18 11:21:14.809 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] INFO  io.zeebe.logstreams.snapshot - Finished taking snapshot, need to wait until last written event position 30064885224 is committed, current commit position is 30064885224. After that snapshot can be marked as valid.
2020-05-18 11:21:14.809 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] INFO  io.zeebe.logstreams.snapshot - Current commit position 30064885224 is greater then 30064885224, snapshot is valid.
2020-05-18 11:36:14.676 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 11:51:14.729 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 12:06:14.755 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 12:21:14.802 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 12:36:14.822 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 12:51:14.831 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 13:06:14.842 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 13:21:14.922 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 13:36:14.980 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 13:51:14.983 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 14:06:14.995 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 14:21:15.447 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960
2020-05-18 14:36:15.586 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 30064888960

Any reasons for concern?

Hey @mostwired

thanks for trying Zeebe.

This looks interesting. Could you please check whether you went out of disk space?
How do you deploy Zeebe?

Greets
Chris

1 Like

No, plenty of disk space (> 1 TB left). I download the latest release from Github and run the executable on a local machine. The only change to the yaml config is enabling Elasticsearch exporter. The error sometimes comes back, but restarting broker usually gets rid of it. Sometimes it goes away by itself, followed by:

2020-05-18 23:46:00.725 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] INFO  io.zeebe.logstreams.snapshot - Current commit position 34359811960 is greater then 34359811960, snapshot is valid.

meet the same error:

2020-05-20 11:19:59.223 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94494478368
2020-05-20 11:34:59.224 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94494774432
2020-05-20 11:49:59.242 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94495073336
2020-05-20 12:04:59.253 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94495369720
2020-05-20 12:19:59.267 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94495667488
2020-05-20 12:34:59.282 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94495998208
2020-05-20 12:49:59.316 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94496329704
2020-05-20 13:04:59.348 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94496647088
2020-05-20 13:19:59.367 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94496944296
2020-05-20 13:34:59.373 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94497251888
2020-05-20 13:49:59.376 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 94497583000
2020-05-20 14:04:59.405 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98784509744
2020-05-20 14:19:59.428 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98784808368
2020-05-20 14:34:59.451 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98785106032
2020-05-20 14:49:59.479 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98785404504
2020-05-20 15:04:59.493 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98785712152
2020-05-20 15:19:59.505 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98786010752
2020-05-20 15:34:59.532 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98786308512
2020-05-20 15:49:59.544 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98786602840
2020-05-20 16:04:59.548 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-0] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98786896008
2020-05-20 16:19:59.566 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98787191968
2020-05-20 16:34:59.578 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98787489680
2020-05-20 16:49:59.602 [Broker-0-SnapshotDirector-1] [Broker-0-zb-fs-workers-1] WARN  io.zeebe.logstreams.snapshot - Failed to obtain a pending snapshot directory for position 98787785328

context:
zeebe 0.23.1
es Exporter
deployed in docker swarm

same problem here.
Any idea what is the meaning/impact of such error? Is zeebe broker losing information etc. ?
@Zelldon

Hey guys it seems to be a know issue you can track it here

@Eetay no this is just to take snasphots, if the broker is not able to take snapshots it will not be able to compact the log.

Greets
Chris

Thanks @Zelldon

Quick followup - Can this issue be in any way related to failure to show new workflows deployed, or running instances, for example, in the community monitor?
Eetay