We specified the zeebe-boker’s endpoint url with 5701 port in order to connect to hazelcast. But zeebe-simple-monitor is trying to connect to localhost:5701. Please check the logs:
`ubuntu@ip-172-31-0-78:~/zeebe-simple-monitor-helm/zeebe-simple-monitor-helm/zeebe-simple-monitor/templates$ kubectl logs zeebe-monitor-zeebe-simple-monitor-5bfcd5894b-d82zs -n zeebe-test`
___ __
_/ _ _ |_ _ (_ . _ _ | _ |\/| _ _ . |_ _ _
/__ (- (- |_) (- __) | ||| |_) | (- | | (_) | ) | |_ (_) |
|
================================================================
(v0.20.0-SNAPSHOT)
2020-07-07 06:17:20.034 INFO 1 --- [ main] io.zeebe.monitor.ZeebeSimpleMonitorApp : Starting ZeebeSimpleMonitorApp v0.20.0-SNAPSHOT on zeebe-monitor-zeebe-simple-monitor-5bfcd5894b-d82zs with PID 1 (/usr/local/zeebe-simple-monitor.jar started by root in /)
2020-07-07 06:17:20.038 DEBUG 1 --- [ main] io.zeebe.monitor.ZeebeSimpleMonitorApp : Running with Spring Boot v2.3.0.RELEASE, Spring v5.2.6.RELEASE
2020-07-07 06:17:20.039 INFO 1 --- [ main] io.zeebe.monitor.ZeebeSimpleMonitorApp : No active profile set, falling back to default profiles: default
2020-07-07 06:17:28.938 INFO 1 --- [ main] i.z.monitor.zeebe.ZeebeHazelcastService : Connecting to Hazelcast 'localhost:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/usr/local/zeebe-simple-monitor.jar!/BOOT-INF/lib/hazelcast-4.0.1.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-07-07 06:17:29.706 WARN 1 --- [ main] c.h.c.i.c.ClientConnectionManager : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701
2020-07-07 06:17:29.706 WARN 1 --- [ main] c.h.c.i.c.ClientConnectionManager : hz.client_1 [dev] [4.0.1] Unable to get live cluster connection, retry in 15000 ms, attempt: 1 , cluster connect timeout: 36000000 seconds , max backoff millis: 60000
2020-07-07 06:17:44.707 WARN 1 --- [ main] c.h.c.i.c.ClientConnectionManager : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701
2020-07-07 06:17:44.708 WARN 1 --- [ main] c.h.c.i.c.ClientConnectionManager : hz.client_1 [dev] [4.0.1] Unable to get live cluster connection, retry in 30000 ms, attempt: 2 , cluster connect timeout: 36000000 seconds , max backoff millis: 60000
2020-07-07 06:18:14.709 WARN 1 --- [ main] c.h.c.i.c.ClientConnectionManager : hz.client_1 [dev] [4.0.1] Exception during initial connection to [localhost]:5701: com.hazelcast.core.HazelcastException: java.net.SocketException: Connection refused to address localhost/127.0.0.1:5701
Please check the running pods and services:
Below are the environments which we used in simple-monitor-deployment file:
env:
- name: zeebe.client.broker.contactPoint
value: zeebe-zeebe.zeebe-test.svc.cluster.local:26500
- name: zeebe.worker.hazelcast.connection
value: zeebe-zeebe.zeebe-test.svc.cluster.local:5701
Thanks in advance!!
@SujataKale97 hi Sujata, welcome to our community!
Can I ask where did you get the Simple Monitor deployment file?
name: zeebe.client.broker.contactPoint
That definitely doesn’t sounds like an ENVIRONMENT VARIABLE… so that will not work, but because we don’t know where is that being applied it is impossible to help you.
Can you elaborate on how did you get the Simple Monitor Service and Deployment deployed to Kubernetes?
@salaboy We are using https://github.com/livspaceeng/zeebe-simple-monitor-helm this helm chart to deploy simple monitor on kubernetes.
We are using that environment varaibles from this git url https://github.com/zeebe-io/zeebe-simple-monitor. But that works for docker not for kubernetes. So can you please provide those envrionments which i can use on kubernetes.
And if you have any helm chart to deploy simple-monitor on kubernetes, please let me know it
@salaboy we updated that envs as below:
Now it’s trying to connect to actual hazelcast but getting below errors. Please check the logs and guide us:
ubuntu@ip-172-31-0-78:~/zeebe-simple-monitor-helm/zeebe-simple-monitor-helm$ kubectl logs zeebe-monitor-zeebe-simple-monitor-7bdbbd76c9-tlc8q -n zeebe-test -f
___ __
_/ _ _ |_ _ (_ . _ _ | _ |\/| _ _ . |_ _ _
/__ (- (- |_) (- __) | ||| |_) | (- | | (_) | ) | |_ (_) |
|
================================================================
2020-07-07 10:37:12.208 INFO 1 --- [ restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp : Starting ZeebeSimpleMonitorApp on zeebe-monitor-zeebe-simple-monitor-7bdbbd76c9-tlc8q with PID 1 (/app/classes started by root in /)
2020-07-07 10:37:12.212 DEBUG 1 --- [ restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-07 10:37:12.214 INFO 1 --- [ restartedMain] io.zeebe.monitor.ZeebeSimpleMonitorApp : No active profile set, falling back to default profiles: default
2020-07-07 10:37:20.582 INFO 1 --- [ restartedMain] i.z.monitor.zeebe.ZeebeHazelcastService : Connecting to Hazelcast 'zeebe-zeebe-gateway.zeebe-test.svc.cluster.local:5701'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (file:/app/libs/hazelcast-3.12.6.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2020-07-07 10:37:26.031 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Exception during initial connection to [zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketTimeoutException
2020-07-07 10:37:26.032 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Unable to get live cluster connection, retry in 3000 ms, attempt 1 of 2.
2020-07-07 10:37:34.038 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Exception during initial connection to [zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701: com.hazelcast.core.HazelcastException: java.net.SocketTimeoutException
2020-07-07 10:37:34.039 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Unable to get live cluster connection, attempt 2 of 2.
2020-07-07 10:37:34.039 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Unable to connect to any address for cluster: dev. The following addresses were tried: [[zeebe-zeebe-gateway.zeebe-test.svc.cluster.local]:5701]
2020-07-07 10:37:34.039 WARN 1 --- [ient_1.cluster-] c.h.c.c.nio.ClusterConnectorService : hz.client_1 [dev] [3.12.6] Could not connect to any cluster, shutting down the client: Unable to connect to any cluster.
2020-07-07 10:37:34.304 ERROR 1 --- [ restartedMain] o.s.boot.SpringApplication : Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zeebeHazelcastService': Invocation of init method failed; nested exception is java.lang.IllegalStateException: Unable to connect to any cluster.
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:416) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1788) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:595) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:517) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:323) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:882) ~[spring-beans-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:878) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) ~[spring-boot-2.2.6.RELEASE.jar:2.2.6.RELEASE]
at io.zeebe.monitor.ZeebeSimpleMonitorApp.main(ZeebeSimpleMonitorApp.java:39) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.2.6.RELEASE.jar:2.2.6.RELEASE]
Caused by: java.lang.IllegalStateException: Unable to connect to any cluster.
at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl.connectToClusterInternal(ClusterConnectorServiceImpl.java:214) ~[hazelcast-client-3.12.6.jar:3.12.6]
at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl.access$000(ClusterConnectorServiceImpl.java:62) ~[hazelcast-client-3.12.6.jar:3.12.6]
at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl$1.call(ClusterConnectorServiceImpl.java:279) ~[hazelcast-client-3.12.6.jar:3.12.6]
at com.hazelcast.client.connection.nio.ClusterConnectorServiceImpl$1.call(ClusterConnectorServiceImpl.java:275) ~[hazelcast-client-3.12.6.jar:3.12.6]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64) ~[hazelcast-3.12.6.jar:3.12.6]
at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80) ~[hazelcast-3.12.6.jar:3.12.6]
@SujataKale97 in order to work with simple monitor you need to have the hazel cast exporter in your broker… do you have that already? if so… you need to make sure that the Simple Monitor app points to the broker address that has an in-memory hazelcast instance… this is quite complicated I know but that is what it takes to get Simple Monitor working with the current setup. You might want to also create an issue in the Simple Monitor Charts that you are using, as it is a community project and the person who created it might know exactly what the problem is.
@salaboy I have seen docker-compose file for simple monitor and simple monitor is working in docker.
Do you have helm chart for zeebe simple monitor?
@ashu2108 no we don’t have a helm chart for simple monitor, because it is a community project you can create and share that charts if you are interested in using helm to deploy it.
