Kafka - Zeebe Connector

Alan · January 24, 2020, 12:01pm

Hi

I am trying to get the demo Kafka - Zeebe Connector running. The Ping Pong application.

I have the following processes running:
C:\Camunda\elasticsearch-6.8.0>bin\elasticsearch.bat
C:\Camunda\zeebe-broker-0.20.0>bin\broker.bat
C:\Camunda\camunda-operate-1.0.0>bin\operate.bat

C:\Kafka\kafka_2.12-2.4.0>bin\windows\zookeeper-server-start.bat config\zookeeper.properties
C:\Kafka\kafka_2.12-2.4.0>bin\windows\kafka-server-start.bat config/server.properties

I initiate the workflow using:

bin\zbctl.exe create instance --variables "{\"name\": \"pong\", \"payload\": { \"foo\": \"bar\"}, \"key\": 2}" ping-pong

I can see the running instance in the Operate GUI. It is waiting at the “To Kafka” Service Task.
I cannot see any messages published to the pong topic using this command:

bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic pong --from-beginning

I copied the uber zeebe connector jar into C:\Kafka\kafka_2.12-2.4.0\libs
I copied the files
quickstart-zeebe-source.properties
quickstart-zeebe-sink.properties
into C:\Kafka\kafka_2.12-2.4.0\config

and started the connect-standalone process using:

C:\Kafka\kafka_2.12-2.4.0>bin\windows\connect-standalone.bat config\connect-standalone.properties config\quickstart-zeebe-source.properties config\quickstart-zeebe-sink.properties

There are many errors during start up but the example foo/bar file Connector example works.

Step 7.

After initiating a new Ping Pong workflow execution I see no message activity in the connector-standalone process or the kafka broker process.

Can anyone point me in the right direction please.

Thanks
Alan

salaboy · January 27, 2020, 9:48am

@Alan have you checked this file:

github.com

zeebe-io/kafka-connect-zeebe/blob/master/docker/docker-compose.yml

version: '2.1'

services:
  zeebe:
    image: camunda/zeebe:0.21.1
    hostname: zeebe
    ports:
      - "26500:26500"
      - "9600:9600"
    volumes:
      - ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
    environment:
      ZEEBE_INSECURE_CONNECTION: "true"

  operate:
    image: camunda/operate:1.1.0
    ports:
      - "8080:8080"
    volumes:
      - ./operate.yml:/usr/local/operate/config/application.yml

This file has been truncated. show original

Are you running with that docker compose?

Alan · January 27, 2020, 10:08am

Hi @salaboy, thanks for the response. No I am not using docker-compose. At the moment I need to run on my Windows desktop. The Confluent Kafka docker does not run on Windows. So I am running all of the processes manually.

elasticsearch
zeebe-broker
camunda-operate
zookeeper
kafka-server-start.bat

salaboy · January 27, 2020, 10:12am

“The Confluent Kafka docker does not run on Windows.” really? do you know why?
You can run docker compose on windows right?

Alan · January 27, 2020, 10:44am

Sorry my Windows desktop is a virtualized windows instance. I thought it was not possible to run another VM for docker on top of this. I will double check if that is correct. But I still should be able to get the zeebe integration running manually. Thanks

salaboy · January 27, 2020, 4:12pm

@Alan I am sure that you can run docker compose even if it is in a virtual machine.
If you want to get it working by manually running all the services, I am sure that you can copy each component configuration from the docker-compose.yaml file… You will need to replicate what those docker containers are providing to make it work. You probably want to use the same versions as well.

system · January 31, 2024, 10:09am