Simple monitor does not work correctly after upgrade to 0.18 with broker 0.23 - Variable too large

Camunda 8 Topics Discussion & Questions
1

Hi,
I’ve upgraded to zeebe broker 0.23 according to the zeebe-docker-compose
Simple monitor stopped working correctly - deployments succeed but don’t show up, workflows are not updated when they should.
Looking into the logs of simple-monitor, I’m getting on simple monitor (0.18.0) error that was not there when using broker version 0.22.0 and simple monitor 0.17.0:

2020-05-28 17:34:04.605  INFO 1 --- [  restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService  : Connecting to Hazelcast 'zeebe:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/app/libs/hazelcast-3.12.6.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-05-28 17:34:05.015  INFO 1 --- [  restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService  : Importing records from Hazelcast...
2020-05-28 17:34:05.096  INFO 1 --- [  restartedMain] i.z.h.connect.java.ZeebeHazelcast        : Read from ringbuffer 'zeebe' starting from sequence '0'
2020-05-28 17:34:06.721 ERROR 1 --- [pool-1-thread-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : Value too long for column "PAYLOAD_ VARCHAR(255)": "STRINGDECODE('fields {\n  key: \""UPDATE_COMPONENT_PENDING\""\n  value {\n    struct_value {\n      fields {\n        key: \""creat... (1081)"; SQL statement:
insert into message (correlation_key_, message_id_, name_, payload_, state_, timestamp_, key_) values (?, ?, ?, ?, ?, ?, ?) [22001-200]

now, I do have very large variables (few K’s of data) and one of them is certainly “payload”, but that was not a problem with the previous version

my question is: how can I overcome this with the docker image provided?
(previous answers on forum suggested code changes - is that necessary?)

Thanks
Eetay

FYI @yoavsal

2

Adding:
I’ve already tried 4 versions of hazelcast exporter jar (with the 0.18.0 simple monitor docker image):
the default one provided in …/lib in zeebe-docker-compose
zeebe-hazelcast-exporter-0.X.0-jar-with-dependencies.jar (where X is one of: 7,8,9)

I’ve also tried the 0.7.0 jar with the 0.17.0 image of simple monitor as well as the 0.18.0
E

3

finally managed to solve it in part, by moving to postgre configuration AND
patching tables like so:

ALTER TABLE message ALTER COLUMN payload_ TYPE text;

That is not enough, however - incidents table also have a problem, that I was not so far able to correct
in postgre logs:

org.postgresql.util.PSQLException: Large Objects may not be used in auto-commit mode.

and in http://simple-monitor-host:8082/views/incidents:

Error 500 - Internal Server Error
Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream

org.springframework.orm.jpa.JpaSystemException: Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream
1 Like
4

Hi @Eetay,

the simple monitor version 0.19.0 contains a fix to increase the payload column of messages (https://github.com/zeebe-io/zeebe-simple-monitor/issues/138). This version requires the Hazelcast exporter version 0.9.0.

Please create an issue in the repository if something else is not working.

Note that it is a community-driven project that is happy to get input or PRs from the community.

Best regards,
Philipp

2 Likes
5

thanks @philipp.ossler will check it out!

6

Thanks @philipp.ossler
Encountered errors with 0.19.0, decided to go back to 0.18.0 and my postgre ALTER TABLE patch for now.

7

I was trying to use simple monitor version 0.18.0 with sql server and hazelcast exporter 0.8.0.alpha1 in kubernetes. I was able to deploy workflows from there and to see instances running, but after running a couple of workflow instances the zeebe monitor stopped showing new data, not seeing any error in the log though. Any idea?

8

Hi @smolina74.

I need more information to analyze it. Please check all logs for - both for the simple monitor and the broker. If no data is coming in then it could be the exporter in the broker, something with Hazelcast or in the simple monitor.

9