Zeebe operate search locally for Elastic search

ashu2108 · July 6, 2020, 9:36am

I have modified the configmap.yaml of operate helm chart with AWS ELK but still, operate is searching elastic search in Kubernetes Cluster.
values:
host: “AWS ELK URL”
port: “443”
username: “AWS ELK username”
password: “AWS ELK password”

configmap.yaml:
kind: ConfigMapmetadata:
name: {{ include “zeebe-operate.fullname” . }}
apiVersion: v1
data:
application.yml: |
# Operate configuration file
camunda.operate:
elasticsearch:
host: {{ .Values.global.elasticsearch.host }}
port: {{ .Values.global.elasticsearch.port }}
username: {{ .Values.global.elasticsearch.username }}
password: {{ .Values.global.elasticsearch.password }}
prefix: zeebe-record-operate

salaboy · July 6, 2020, 9:56am

@ashu2108 what is the value of the host and port entries?
Can you please format correctly using triple back quotes your yaml file? It might be the case that your parameters are not being picked up because they are at the wrong level of indentation.

ashu2108 · July 6, 2020, 10:02am

global:
elasticsearch:
host: “xxxxxxxxxxxxxxx.xxxxxxxxxxxx.xxxxxxx.x.ap-south-1.es.amazonaws.com”
port: 443
username: xxxxxxxxxxx
password: xxxxxxxxxxx

same values are passed and working fine with the zeebe cluster. but for operate it is searching locally

ashu2108 · July 6, 2020, 10:10am

I have deployed zeebe cluster

now I want is dashboard UI for workflow?

salaboy · July 6, 2020, 10:14am

@ashu2108 can you please format your code correctly? There is no indentation here…

ashu2108 · July 6, 2020, 10:30am

configmap:

values.yaml:

global:
  zeebe: "{{ .Release.Name }}-zeebe"
  elasticsearch:
    host: "https://xxxxxxxxxxxxxxxx.xxxxx.xxxx.xxx.ap-south-1.es.amazonaws.com"
    port: 443
    username: xxxxxxxxx
    password: xxxxxxxxxxx
    clusterName: "ph1-dev-elk"

sorry I have pasted it directly so it was not showing indentations.
Thanks in Advance!!

salaboy · July 6, 2020, 1:27pm

@ashu2108 I think that you are missing the second half of it
are you setting this also:

github.com

zeebe-io/zeebe-operate-helm/blob/master/zeebe-operate/templates/configmap.yaml#L26


        port: {{ .Values.global.elasticsearch.port }}
      # Zeebe instance
      zeebe:
        # Broker contact point
        brokerContactPoint: {{ tpl .Values.global.zeebe . }}-gateway:26500
      # ELS instance to export Zeebe data to
      zeebeElasticsearch:
        # Cluster name
        clusterName: {{ .Values.global.elasticsearch.clusterName }}
        # Host
        host: {{ .Values.global.elasticsearch.host }}
        # Transport port
        port: {{ .Values.global.elasticsearch.port }}
        # Index prefix, configured in Zeebe Elasticsearch exporter
        prefix: zeebe-record
    logging:
{{- with .Values.logging }}
{{ . | toYaml | indent 6 }}
{{- end }}
    #Spring Boot Actuator endpoints to be exposed
    management.endpoints.web.exposure.include: health,info,conditions,configprops,prometheus

??

ashu2108 · July 6, 2020, 1:36pm

I have not set for the second half but those are also using the same values as you can see both the key and values are the same.

host: {{ .Values.global.elasticsearch.host }}

Thanks in Advance!

salaboy · July 7, 2020, 7:44am

@ashu2108 what do you mean? You need to set them both… they are different properties, and the second ones are needed (even if they are the same) for the Zeebe Client inside Operate to connect as well.

ashu2108 · July 7, 2020, 10:05am

I have updated as shown below now both are using same details of elastic search.

ERROR:

2020-07-07 09:58:11.627  INFO 7 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-07 09:58:11.842  INFO 7 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-07 09:58:11.842  INFO 7 --- [           main] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 6294 ms
2020-07-07 09:58:13.843  INFO 7 --- [           main] o.c.o.WebappModuleConfiguration          : Starting module: webapp
2020-07-07 09:58:13.964 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-07 09:58:14.545 ERROR 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Error occurred while connecting to Elasticsearch: clustername [ph1-dev-elk], https://xxxxxxxxx.xxxxxxxxxx.xxxxxxxx.xxxxx-.ap-south-1.es.amazonaws.com:443. Will be retried...

java.io.IOException: https://xxxxxxx.xxxxxxxx.xxxxxxx.xxxxxxx-ap-south-1.es.amazonaws.com: Name or service not known
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]

salaboy · July 7, 2020, 1:02pm

@ashu2108 ok… so now it is a network problem… nothing to do with Zeebe … you need to ask why the host that is running Operate cannot reach that amazon url… I don’t know if I can help you with that…
Maybe try without the https:// at the beginning… but again… this is not a Zeebe problem.

ashu2108 · July 7, 2020, 2:54pm

@salaboy okay if it is network problem but this issue is only I’m facing in operate and the same URL, I was able to configure in broker as shown below:

logs of broker:

2020-07-07 12:06:37.785 [] [main] DEBUG io.zeebe.broker.exporter - Exporter configured with ElasticsearchExporterConfiguration{url='https://xxxxxxxxxxx.xxxxxxxxxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443', index=IndexConfiguration{indexPrefix='zeebe-record', createTemplate=true, command=false, event=true, rejection=false, error=true, deployment=true, incident=true, job=true, message=false, messageSubscription=false, variable=true, variableDocument=true, workflowInstance=true, workflowInstanceCreation=false, workflowInstanceSubscription=false, ignoreVariablesAbove=8191}, bulk=BulkConfiguration{delay=5, size=1000}, authentication=AuthenticationConfiguration{username='xxxxxxxxxxxx'}}

as you can see both pods are running on the same namespaces.

Correct me if I’m wrong, but If this was the network issue then the same issue broker must-have. why only operate?

Thanks in Advance!!

salaboy · July 8, 2020, 7:34am

@ashu2108 what is the topology of your cluster? How many nodes? Are they all hosted in the same place?
I would suggest to create a simple pod which does a rest call to that URL to test, you can use an existing container with bash to just do curl to that url from your cluster. I cannot think of any reason why it will work for a broker and not for Operate, besides a difference in the way that the rest calls are done.
@svetlana have you seen something like that before?

sdorokhova · July 8, 2020, 7:55am

Hi @ashu2108,

please check the value provided in {{.Values.global.elasticsearch.clusterName}}. Does it correspond to Elasticsearch cluster name? It may be found in Elasticsearch config file or by requesting GET http://<els_host>:<els_port>.

Bers regards,
Svetlana

SujataKale97 · July 8, 2020, 9:15am

Hi @svetlana
Same issue with that elasticsearch cluster name also. I took that clustername from GET http://<els_host>:<els_port> . but same issue is there

sdorokhova · July 8, 2020, 9:45am

Does your Elasticsearch require authentication? If yes, it won’t work with Operate currently. This feature is planned, but not yet implemented.

ashu2108 · July 8, 2020, 9:58am

@svetlana yes elastic search is private so it needs authentication and is there an alternative way we can integrate with a broker and get UI running so we can access in the browser??
Thanks for the quick response.

Thank you in Advance.

sdorokhova · July 8, 2020, 11:30am

Unfortunately currently it is not possible to connect Operate to Elasticsearch with enabled authentication. You will have to find some way to make ELS host visible to only Zeebe and Operate and to close it from all other connections.

SujataKale97 · July 8, 2020, 1:11pm

@svetlana and @salaboy can we deploy zeebe simple monitor on kubernetes for UI purpose. Actually we deployed the simple-monitor on docker and it’s working fine but same configuration is not working on kubernetes. We are facing some issues on kubernetes please check this issue Zeebe-simple-monitor is trying to connect to localhost:5701. Please guide on it.

Thanks

SujataKale97 · July 8, 2020, 1:27pm

@svetlana we tried with public aws elastic-search which don’t take username and password. Still facing same issue
You can check the logs:

  / __| /_\ |  \/  | | | | \| |   \ /_\    / _ \| _ \ __| _ \  /_\_   _| __|
 | (__ / _ \| |\/| | |_| | .` | |) / _ \  | (_) |  _/ _||   / / _ \| | | _|
  \___/_/ \_\_|  |_|\___/|_|\_|___/_/ \_\  \___/|_| |___|_|_\/_/ \_\_| |___|

                                                           0.23.0


2020-07-08 13:24:00.453  INFO 7 --- [           main] o.c.o.Application                        : Starting Application on zeebe-operate-5dc8b97f5c-j9vvt with PID 7 (/usr/local/operate/lib/camunda-operate-webapp-0.23.0-exec.jar started by root in /usr/local/operate)
2020-07-08 13:24:00.465 DEBUG 7 --- [           main] o.c.o.Application                        : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-08 13:24:00.469  INFO 7 --- [           main] o.c.o.Application                        : The following profiles are active: auth
2020-07-08 13:24:06.187  INFO 7 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2020-07-08 13:24:06.239  INFO 7 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
2020-07-08 13:24:06.241  INFO 7 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2020-07-08 13:24:06.243  INFO 7 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-08 13:24:06.442  INFO 7 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-08 13:24:06.442  INFO 7 --- [           main] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 5802 ms
2020-07-08 13:24:08.185  INFO 7 --- [           main] o.c.o.WebappModuleConfiguration          : Starting module: webapp
2020-07-08 13:24:08.352 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-08 13:24:08.922 ERROR 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Error occurred while connecting to Elasticsearch: clustername [400307571006:es-test], https://xxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com:443. Will be retried...

java.io.IOException: https://xxxxxxxxxxxxxx.us-east-2.es.amazonaws.com: Name or service not known
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$98f21047.CGLIB$esClient$0(<generated>) ~[camunda-operate-common-0.23.0.jar!/:?]

When we deploy zeebe-full that time zeebe full helm chart will deploy its own elastic search on kubernetes and that configuration is working fine. But when we are replacing that elasticsearch host with our aws elasticsearch(public) , the zeebe-operate is failing.

So now i have question can we connect aws elasticsearch to zeebe-operate. Is that configuration available??