Is it worth compressing some broker topic log contents?


#1

Hi, I just finished watching the latest webinar and at the 48 minute mark the contents of the broker topic log are shown

this file contains binary contents, but I noticed it also contains uncompressed bpmn xml.

I haven’t looked at the zeebe code, I do not know how often the log file is updated, but I recall reading somewhere that modern cpu’s can gzip compress/uncompress data very quickly. In some conditions, depending on the amount of compression, the savings in disk i/o or network i/o time (due to reduced overall file size) is greater than the time required to compress file contents.

Just throwing this out there as a ‘hmm’ thought… No idea if it’s practical or not. But given that these files are both saved to disk and replicated to other brokers, compression of xml portions might be warranted.

thanks


#2

Hey @bkcsfi,

That is a good idea and we considered it already but never approached it so far as there were more pressing things to do. We are currently in a phase where we are changing and simplifying a couple of concepts in Zeebe (this is also the reason why there were no releases in the last two months). Along with that, the log file will no longer store the entire history of the system, but rather act like a write-ahead log paired with RocksDB persistence of the latest state. The log itself will then contain much less data at a time and for the rocksdb state there will be less redundance, but nevertheless compression is also a worthy idea there (luckily provided out of the box by RocksDB).

We are planning to release a new Zeebe version mid of October and will then also release some blog posts on the concept changes.

Cheers,
Thorben