I have modified the configmap.yaml of operate helm chart with AWS ELK but still, operate is searching elastic search in Kubernetes Cluster.
values:
host: “AWS ELK URL”
port: “443”
username: “AWS ELK username”
password: “AWS ELK password”
@ashu2108 what is the value of the host and port entries?
Can you please format correctly using triple back quotes your yaml file? It might be the case that your parameters are not being picked up because they are at the wrong level of indentation.
@ashu2108 what do you mean? You need to set them both… they are different properties, and the second ones are needed (even if they are the same) for the Zeebe Client inside Operate to connect as well.
2020-07-07 09:58:11.627 INFO 7 --- [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-07 09:58:11.842 INFO 7 --- [ main] o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext
2020-07-07 09:58:11.842 INFO 7 --- [ main] o.s.w.c.ContextLoader : Root WebApplicationContext: initialization completed in 6294 ms
2020-07-07 09:58:13.843 INFO 7 --- [ main] o.c.o.WebappModuleConfiguration : Starting module: webapp
2020-07-07 09:58:13.964 DEBUG 7 --- [ main] o.c.o.e.ElasticsearchConnector : Creating Elasticsearch connection...
2020-07-07 09:58:14.545 ERROR 7 --- [ main] o.c.o.e.ElasticsearchConnector : Error occurred while connecting to Elasticsearch: clustername [ph1-dev-elk], https://xxxxxxxxx.xxxxxxxxxx.xxxxxxxx.xxxxx-.ap-south-1.es.amazonaws.com:443. Will be retried...
java.io.IOException: https://xxxxxxx.xxxxxxxx.xxxxxxx.xxxxxxx-ap-south-1.es.amazonaws.com: Name or service not known
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]
@ashu2108 ok… so now it is a network problem… nothing to do with Zeebe … you need to ask why the host that is running Operate cannot reach that amazon url… I don’t know if I can help you with that…
Maybe try without the https:// at the beginning… but again… this is not a Zeebe problem.
@salaboy okay if it is network problem but this issue is only I’m facing in operate and the same URL, I was able to configure in broker as shown below:
@ashu2108 what is the topology of your cluster? How many nodes? Are they all hosted in the same place?
I would suggest to create a simple pod which does a rest call to that URL to test, you can use an existing container with bash to just do curl to that url from your cluster. I cannot think of any reason why it will work for a broker and not for Operate, besides a difference in the way that the rest calls are done. @svetlana have you seen something like that before?
please check the value provided in {{.Values.global.elasticsearch.clusterName}}. Does it correspond to Elasticsearch cluster name? It may be found in Elasticsearch config file or by requesting GET http://<els_host>:<els_port>.
Hi @svetlana
Same issue with that elasticsearch cluster name also. I took that clustername from GET http://<els_host>:<els_port> . but same issue is there
@svetlana yes elastic search is private so it needs authentication and is there an alternative way we can integrate with a broker and get UI running so we can access in the browser??
Thanks for the quick response.
Unfortunately currently it is not possible to connect Operate to Elasticsearch with enabled authentication. You will have to find some way to make ELS host visible to only Zeebe and Operate and to close it from all other connections.
@svetlana and @salaboy can we deploy zeebe simple monitor on kubernetes for UI purpose. Actually we deployed the simple-monitor on docker and it’s working fine but same configuration is not working on kubernetes. We are facing some issues on kubernetes please check this issue Zeebe-simple-monitor is trying to connect to localhost:5701. Please guide on it.
@svetlana we tried with public aws elastic-search which don’t take username and password. Still facing same issue
You can check the logs:
/ __| /_\ | \/ | | | | \| | \ /_\ / _ \| _ \ __| _ \ /_\_ _| __|
| (__ / _ \| |\/| | |_| | .` | |) / _ \ | (_) | _/ _|| / / _ \| | | _|
\___/_/ \_\_| |_|\___/|_|\_|___/_/ \_\ \___/|_| |___|_|_\/_/ \_\_| |___|
0.23.0
2020-07-08 13:24:00.453 INFO 7 --- [ main] o.c.o.Application : Starting Application on zeebe-operate-5dc8b97f5c-j9vvt with PID 7 (/usr/local/operate/lib/camunda-operate-webapp-0.23.0-exec.jar started by root in /usr/local/operate)
2020-07-08 13:24:00.465 DEBUG 7 --- [ main] o.c.o.Application : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-08 13:24:00.469 INFO 7 --- [ main] o.c.o.Application : The following profiles are active: auth
2020-07-08 13:24:06.187 INFO 7 --- [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-07-08 13:24:06.239 INFO 7 --- [ main] o.a.c.h.Http11NioProtocol : Initializing ProtocolHandler ["http-nio-8080"]
2020-07-08 13:24:06.241 INFO 7 --- [ main] o.a.c.c.StandardService : Starting service [Tomcat]
2020-07-08 13:24:06.243 INFO 7 --- [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-08 13:24:06.442 INFO 7 --- [ main] o.a.c.c.C.[.[.[/] : Initializing Spring embedded WebApplicationContext
2020-07-08 13:24:06.442 INFO 7 --- [ main] o.s.w.c.ContextLoader : Root WebApplicationContext: initialization completed in 5802 ms
2020-07-08 13:24:08.185 INFO 7 --- [ main] o.c.o.WebappModuleConfiguration : Starting module: webapp
2020-07-08 13:24:08.352 DEBUG 7 --- [ main] o.c.o.e.ElasticsearchConnector : Creating Elasticsearch connection...
2020-07-08 13:24:08.922 ERROR 7 --- [ main] o.c.o.e.ElasticsearchConnector : Error occurred while connecting to Elasticsearch: clustername [400307571006:es-test], https://xxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com:443. Will be retried...
java.io.IOException: https://xxxxxxxxxxxxxx.us-east-2.es.amazonaws.com: Name or service not known
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]
at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$98f21047.CGLIB$esClient$0(<generated>) ~[camunda-operate-common-0.23.0.jar!/:?]
When we deploy zeebe-full that time zeebe full helm chart will deploy its own elastic search on kubernetes and that configuration is working fine. But when we are replacing that elasticsearch host with our aws elasticsearch(public) , the zeebe-operate is failing.
So now i have question can we connect aws elasticsearch to zeebe-operate. Is that configuration available??