Hello everyone!
Confused on some incorrect negative metrics in Grafana (see screenshot) like:
Running workflow instances
Stream Processor vs Exporter
Actually exported data is consistent and correct, but metrics are wrong.
Tested on clean just created docker instances with single deployed process with just one task.
Tried in different configurations and almost always see numbers not matching the reality.
Is there any way to fix metrics to use them as a reliable source of diagnostic data?
I would say the panel at the bottom is kind of expected. It means the exporter is faster then the stream processor. The upper one might related due to restarts, there is a known issue on metrics becoming neg. due to reprocessing. Open issues which are related to this