Zeebe Metrics Monitoring URL works but the Host URL giving 404 error

Hi All

I am new to Zeebe and started with local project setup. I am using standalone broker with embedded gateway to try out Zeebe related functionalities. I have also enabled the Prometheus monitoring API configurations and noticed that actual metrics URL is working fine but the host URL is not working. As a result I couldn’t use it in Grafana to show dashboards related to Zeebe.

  1. As per the below link
    https://docs.zeebe.io/appendix/broker-config-template.html

I have configured the standalone broker with embedded gateway to have the monitoring API enabled. Added the configurations as follows to the appropriate yaml file

zeebe:
broker:
network:
monitoringApi:
host: <MY_HOST>
port: 9600

  1. I can see the metrics related to Zeebe in the below link
    http:<MY_HOST>:9600/metrics (This works)

but the following link doesn’t work

http:<MY_HOST>:9600 (This doesn’t work)

  1. Since the landing server URL doesn’t work when I am configuring the datasource in Grafana it is not able to add the datasource as intended. I am getting the HTTP Error Not Found.

Is there any configuration available to make the landing server URL work so that I can configure the datasource in Grafana correctly and create dashboards for Zeebe?

Setup details:
a) Operating System
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.10 (Santiago)
Release: 6.10

b) Zeebe Version: 0.23.0

c) Prometheus: prometheus-2.19.1.linux-amd64

d) Grafana: grafana-7.0.6

Please let me know if I have to enable or add any configurations to make this work.

Hey @chiranvijay

I’m not 100% sure whether I get the questions. Why you want to scrape from the root and not using the /metrics endpoint?

It is btw also described here https://docs.zeebe.io/operations/metrics.html?highlight=metrics#connecting-prometheus

Greets
Chris

Hi @Zelldon - Sorry it was due to wrong configuration and environment it wasn’t working. Now I could able to get it working fine. Thanks for your reply.

1 Like

@Zelldon - Quick question, I was testing in Zeebe 0.23.1 version and after correcting all environment and configurations it used to work perfectly fine. Now I upgraded Zeebe to latest 0.24.1 version and the configurations are retained same. Just wondering whether there is any change in behavior from Zeebe perspective where metrics URL returns HTTP status code 404.

http://<MY_HOST_IP>:9600/metrics

returns server returned HTTP status 404 in Prometheus targets.

I think we replaced the http server, it might be that it just returns values after it becomes ready.

Maybe @pihme can help here.

Thanks @Zelldon for quick reply.

@pihme - Please let me know whether we need to turn on any configurations. I have kept zeebe running for longer time and still no luck with the metrics URL in 0.24.1 version. Tried reverting back to older version to perform sanity check and it works fine. Only in newer version I am seeing this behavior. Please let me know your thoughts.

Can you tell me what gets returned when you access

[host]:9600/actuator/

This returns a JSON and should contain the string Prometheus.

If it doesn’t, and if you specify the config file via SPRING_CONFIG_LOCATION=[path to file] then change it to export SPRING_CONFIG_LOCATION='classpath:/,file:./[path to config file]'.

This was one of the reasons why we couldn’t access the metrics. PR is pending.

If it is something else, please let me know.

Hi @pihme

It didn’t return prometheus URL if I give [host]:9600/actuator, however after changing the spring config location to what you have pointed out to add classpath showed me the prometheus URL. Now with the same URL [host]:9600/metrics I could able to see it working.

Just wondering whether is it by design this has changed or something is modified? I couldn’t find or missed in Zeebe upgrade documentation.

Anyhow thanks @Zelldon and @pihme for assisting me and sorting this issue out.

Documentation needs to be updated to reflect this. We learned about this detail last minute pre-release.

Hi everyone, I was following this thread because I am having issues configuring the metrics.
I have updated my zeebe_configmap with:
zeebe:
broker:
network:
port: 9600
But I cannot still open my :9600/metrics.
I have created a rule to allow egress to 9600 but seems it is not working either. Here a screenshot of my configuration: