Zeebe-simple-monitor is trying to connect to localhost:5701

We specified the zeebe-boker’s endpoint url with 5701 port in order to connect to hazelcast. But zeebe-simple-monitor is trying to connect to localhost:5701. Please check the logs:

    `ubuntu@ip-172-31-0-78:~/zeebe-simple-monitor-helm/zeebe-simple-monitor-helm/zeebe-simple-monitor/templates$ kubectl logs zeebe-monitor-zeebe-simple-monitor-5bfcd5894b-d82zs -n zeebe-test`

___                 __
 _/  _  _ |_   _   (_  .  _   _  |  _   |\/|  _   _  . |_  _   _
/__ (- (- |_) (-   __) | ||| |_) | (-   |  | (_) | ) | |_ (_) |
                             |

================================================================
                                 (v0.20.0-SNAPSHOT)

2020-07-07 06:17:20.034  INFO 1 --- [           main] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Starting ZeebeSimpleMonitorApp v0.20.0-SNAPSHOT on zeebe-monitor-zeebe-simple-monitor-5bfcd5894b-d82zs with PID 1 (/usr/local/zeebe-simple-monitor.jar started by root in /)
2020-07-07 06:17:20.038 DEBUG 1 --- [           main] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Running with Spring Boot v2.3.0.RELEASE, Spring v5.2.6.RELEASE
2020-07-07 06:17:20.039  INFO 1 --- [           main] io.zeebe.monitor.ZeebeSimpleMonitorApp   : No active profile set, falling back to default profiles: default
2020-07-07 06:17:28.938  INFO 1 --- [           main] i.z.monitor.zeebe.ZeebeHazelcastService  : Connecting to Hazelcast 'localhost:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/usr/local/zeebe-simple-monitor.jar!/BOOT-INF/lib/hazelcast-4.0.1.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-07-07 06:17:29.706  WARN 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701
2020-07-07 06:17:29.706  WARN 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [4.0.1] Unable to get live cluster connection, retry in 15000 ms, attempt: 1 , cluster connect timeout: 36000000 seconds , max backoff millis: 60000
2020-07-07 06:17:44.707  WARN 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701
2020-07-07 06:17:44.708  WARN 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [4.0.1] Unable to get live cluster connection, retry in 30000 ms, attempt: 2 , cluster connect timeout: 36000000 seconds , max backoff millis: 60000
2020-07-07 06:18:14.709  WARN 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701 

Please check the running pods and services:

Below are the environments which we used in simple-monitor-deployment file:

env:
   - name: zeebe.client.broker.contactPoint
     value: zeebe-zeebe.zeebe-test.svc.cluster.local:26500
   - name: zeebe.worker.hazelcast.connection
     value: zeebe-zeebe.zeebe-test.svc.cluster.local:5701

Thanks in advance!!

@SujataKale97 hi Sujata, welcome to our community!
Can I ask where did you get the Simple Monitor deployment file?

name: zeebe.client.broker.contactPoint

That definitely doesn’t sounds like an ENVIRONMENT VARIABLE… so that will not work, but because we don’t know where is that being applied it is impossible to help you.
Can you elaborate on how did you get the Simple Monitor Service and Deployment deployed to Kubernetes?

@salaboy We are using https://github.com/livspaceeng/zeebe-simple-monitor-helm this helm chart to deploy simple monitor on kubernetes.
We are using that environment varaibles from this git url https://github.com/zeebe-io/zeebe-simple-monitor. But that works for docker not for kubernetes. So can you please provide those envrionments which i can use on kubernetes.

And if you have any helm chart to deploy simple-monitor on kubernetes, please let me know it

@salaboy we updated that envs as below:
image
Now it’s trying to connect to actual hazelcast but getting below errors. Please check the logs and guide us:
ubuntu@ip-172-31-0-78:~/zeebe-simple-monitor-helm/zeebe-simple-monitor-helm$ kubectl logs zeebe-monitor-zeebe-simple-monitor-7bdbbd76c9-tlc8q -n zeebe-test -f

___                 __
 _/  _  _ |_   _   (_  .  _   _  |  _   |\/|  _   _  . |_  _   _
/__ (- (- |_) (-   __) | ||| |_) | (-   |  | (_) | ) | |_ (_) |
                             |

================================================================


2020-07-07 10:37:12.208  INFO 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Starting ZeebeSimpleMonitorApp on zeebe-monitor-zeebe-simple-monitor-7bdbbd76c9-tlc8q with PID 1 (/app/classes started by root in /)
2020-07-07 10:37:12.212 DEBUG 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-07 10:37:12.214  INFO 1 --- [  restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp   : No active profile set, falling back to default profiles: default
2020-07-07 10:37:20.582  INFO 1 --- [  restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService  : Connecting to Hazelcast 'zeebe-zeebe-gateway.zeebe-test.svc.cluster.local:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/app/libs/hazelcast-3.12.6.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-07-07 10:37:26.031  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Exception during initial connection to [zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketTimeoutException
2020-07-07 10:37:26.032  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Unable to get live cluster connection, retry in 3000 ms, attempt 1 of 2.
2020-07-07 10:37:34.038  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Exception during initial connection to [zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketTimeoutException
2020-07-07 10:37:34.039  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Unable to get live cluster connection, attempt 2 of 2.
2020-07-07 10:37:34.039  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Unable to connect to any address for cluster: dev. The following addresses were tried: [[zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701]
2020-07-07 10:37:34.039  WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService      : hz.client_1 [dev] [3.12.6] Could not connect to any cluster, shutting down the client: Unable to connect to any cluster.
2020-07-07 10:37:34.304 ERROR 1 --- [  restartedMain] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zeebeHazelcastService': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Unable to connect to any cluster.
        at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:882) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
        at io.zeebe.monitor.ZeebeSimpleMonitorApp.main(ZeebeSimpleMonitorApp.java:39) ~[classes/:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.2.6.RELEASE.jar:2.2.6.RELEASE]
Caused by: java.lang.IllegalStateException: Unable to connect to any cluster.
        at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl.connectToClusterInternal(ClusterConnectorServiceImpl.java:214) ~[hazelcast-client-3.12.6.jar:3.12.6]
        at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl.access$000(ClusterConnectorServiceImpl.java:62) ~[hazelcast-client-3.12.6.jar:3.12.6]
        at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl$1.call(ClusterConnectorServiceImpl.java:279) ~[hazelcast-client-3.12.6.jar:3.12.6]
        at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl$1.call(ClusterConnectorServiceImpl.java:275) ~[hazelcast-client-3.12.6.jar:3.12.6]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
        at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64) ~[hazelcast-3.12.6.jar:3.12.6]
        at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80) ~[hazelcast-3.12.6.jar:3.12.6]

@SujataKale97 in order to work with simple monitor you need to have the hazel cast exporter in your broker… do you have that already? if so… you need to make sure that the Simple Monitor app points to the broker address that has an in-memory hazelcast instance… this is quite complicated I know but that is what it takes to get Simple Monitor working with the current setup. You might want to also create an issue in the Simple Monitor Charts that you are using, as it is a community project and the person who created it might know exactly what the problem is.

@salaboy I have seen docker-compose file for simple monitor and simple monitor is working in docker.
Do you have helm chart for zeebe simple monitor?