Negative metrics in Grafana

Hello everyone!
Confused on some incorrect negative metrics in Grafana (see screenshot) like:

  • Running workflow instances
  • Stream Processor vs Exporter

Actually exported data is consistent and correct, but metrics are wrong.
Tested on clean just created docker instances with single deployed process with just one task.

Tried in different configurations and almost always see numbers not matching the reality.

Is there any way to fix metrics to use them as a reliable source of diagnostic data?

2 Likes

Hmmm… quick question: are you running the broker at faster than the speed of light? That produces that kind of output.

Joking! I don’t know what causes that. Maybe @Zelldon knows?

1 Like

Could you please clarify what’s unusual you see in this output?
From my point it is nothing exceptional.

I’m implementing testing on 4CPU (+4 hyperthreading) single node (NVMe SSD, 40Gb RAM), 4 partitions, cpu threads 6, io threads 2
Starting from NodeJS client 10000 instances with 4 in parallel.

    const parallelFactor = this.topology.partitionsCount;
    await Parallel.each(instancesVariables, async (instanceVariables) => {
        await this.zbc.createWorkflowInstance(processDefinitionId, instanceVariables);
    }, parallelFactor);

It produces around 40-50 started processes per second. (It is way less than I expected of at least 500/sec. But this is another story.)

Hey @jetdream

I would say the panel at the bottom is kind of expected. It means the exporter is faster then the stream processor. The upper one might related due to restarts, there is a known issue on metrics becoming neg. due to reprocessing. Open issues which are related to this

Greets
Chris

2 Likes

Thank you @Zelldon, now it’s clearer.
But this noise makes it difficult to analyse the real issues.
Hope it will be fixed soon.

Agree maybe you can add a comment to the issue to give a bit more pressure :slight_smile: