Connect to Zeebe in a docker-compose environment

BerndRuecker · August 14, 2017, 3:18pm

Hey guys.

I am a bit puzzled, maybe you can give me a quick hint (might be docker-compose related, but I give it a shot).

I created a simple monitoring application, that can connect to Zeebe and show the status in a small webapp. I now run both of them via docker:

https://github.com/camunda-consulting/zeebe-simple-monitor/blob/master/docker-compose.yml

Now I try to connect to Zeebe via

zeebe:51015

I also tried various other names like zeebesimplemonitor_zeebe_1:51015 without luck. Any idea what is wrong here?

That would be AWESOME It works perfectly when I run the Java process locally and connect to localhost:51015

Creating network "zeebesimplemonitor_default" with the default driver
Creating zeebesimplemonitor_zeebe_1 ...
Creating zeebesimplemonitor_monitor_1 ...
Creating zeebesimplemonitor_monitor_1
Creating zeebesimplemonitor_zeebe_1 ... done
Attaching to zeebesimplemonitor_monitor_1, zeebesimplemonitor_zeebe_1
...
monitor_1  | java.net.ConnectException: Connection refused
monitor_1  |    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
monitor_1  |    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
monitor_1  |    at io.zeebe.transport.impl.TransportChannel.finishConnect(TransportChannel.java:261)
monitor_1  |    at io.zeebe.transport.impl.selector.ConnectTransportPoller.processKey(ConnectTransportPoller.java:128)
monitor_1  |    at org.agrona.nio.NioSelectedKeySet.forEach(NioSelectedKeySet.java:136)
monitor_1  |    at io.zeebe.transport.impl.selector.ConnectTransportPoller.pollNow(ConnectTransportPoller.java:111)
monitor_1  |    at io.zeebe.transport.impl.selector.ConnectTransportPoller.doWork(ConnectTransportPoller.java:52)
monitor_1  |    at io.zeebe.transport.impl.actor.ClientConductor.doWork(ClientConductor.java:43)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.tryRunActor(ActorRunner.java:165)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.runActor(ActorRunner.java:145)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.doWork(ActorRunner.java:114)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.run(ActorRunner.java:71)
monitor_1  |    at java.lang.Thread.run(Thread.java:745)
monitor_1  | java.lang.NullPointerException
monitor_1  |    at io.zeebe.client.task.impl.subscription.EventSubscriptions.lambda$reopenSubscriptionsForRemote$0(EventSubscriptions.java:93)
monitor_1  |    at io.zeebe.client.task.impl.subscription.EventSubscriptions.forAllDoConsume(EventSubscriptions.java:198)
monitor_1  |    at io.zeebe.client.task.impl.subscription.EventSubscriptions.reopenSubscriptionsForRemote(EventSubscriptions.java:91)
monitor_1  |    at io.zeebe.client.task.impl.subscription.SubscriptionManager.onConnectionClosed(SubscriptionManager.java:196)
monitor_1  |    at io.zeebe.transport.impl.actor.Conductor.lambda$null$5(Conductor.java:141)
monitor_1  |    at java.util.ArrayList.forEach(ArrayList.java:1249)
monitor_1  |    at io.zeebe.transport.impl.actor.Conductor.lambda$onChannelDisconnected$6(Conductor.java:137)
monitor_1  |    at org.agrona.concurrent.ManyToOneConcurrentArrayQueue.drain(ManyToOneConcurrentArrayQueue.java:113)
monitor_1  |    at org.agrona.concurrent.ManyToOneConcurrentArrayQueue.drain(ManyToOneConcurrentArrayQueue.java:88)
monitor_1  |    at io.zeebe.util.DeferredCommandContext.doWork(DeferredCommandContext.java:66)
monitor_1  |    at io.zeebe.transport.impl.actor.Conductor.doWork(Conductor.java:67)
monitor_1  |    at io.zeebe.transport.impl.actor.ClientConductor.doWork(ClientConductor.java:41)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.tryRunActor(ActorRunner.java:165)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.runActor(ActorRunner.java:138)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.doWork(ActorRunner.java:114)
monitor_1  |    at io.zeebe.util.actor.ActorRunner.run(ActorRunner.java:71)
monitor_1  |    at java.lang.Thread.run(Thread.java:745)
monitor_1  | 2017-08-14 15:13:32.949  INFO 1 --- [-runner-default] io.zeebe.client.subscription             : Could not open subscription; aborting
monitor_1  |
monitor_1  | io.zeebe.client.cmd.ClientException: Unexpected exception during response handling
monitor_1  |    at io.zeebe.client.impl.RequestController$ExecuteRequestState.doWork(RequestController.java:298) ~[zeebe-client-java-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.client.impl.RequestController$ExecuteRequestState.doWork(RequestController.java:275) ~[zeebe-client-java-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.util.state.StateMachine.doWork(StateMachine.java:111) ~[zb-util-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.client.impl.RequestController.doWork(RequestController.java:141) ~[zeebe-client-java-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.client.impl.RequestManager.doWork(RequestManager.java:78) ~[zeebe-client-java-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.util.actor.ActorRunner.tryRunActor(ActorRunner.java:165) ~[zb-util-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.util.actor.ActorRunner.runActor(ActorRunner.java:138) ~[zb-util-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.util.actor.ActorRunner.doWork(ActorRunner.java:114) ~[zb-util-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.util.actor.ActorRunner.run(ActorRunner.java:71) ~[zb-util-0.1.0.jar!/:0.1.0]
monitor_1  |    at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
monitor_1  | Caused by: java.util.concurrent.ExecutionException: Request failed - Socket channel has been disconnected
monitor_1  |    at io.zeebe.transport.impl.ClientRequestImpl.get(ClientRequestImpl.java:254) ~[zb-transport-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.transport.impl.ClientRequestImpl.get(ClientRequestImpl.java:182) ~[zb-transport-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.transport.impl.ClientRequestImpl.get(ClientRequestImpl.java:39) ~[zb-transport-0.1.0.jar!/:0.1.0]
monitor_1  |    at io.zeebe.client.impl.RequestController$ExecuteRequestState.doWork(RequestController.java:286) ~[zeebe-client-java-0.1.0.jar!/:0.1.0]
monitor_1  |    ... 9 common frames omitted
monitor_1  |

Cheers
Bernd

menski · August 14, 2017, 3:29pm

Hi Bernd,

I cannot see the docker-compose file, is it maybe a private repo? But we expect your configuration could be a problem. How did you configured the host of the network section? It specifies how the socket is bound but also how the client API tries to connect. So localhost will not work, as the client will ask the broker where to find the leader of topic default-topic:0 and then the broker would return localhost:51015 which will not work for the client.

Cheers,
Sebastian

BerndRuecker · August 14, 2017, 3:34pm

You are right, sorry, I fixed the visibility of the repo, you can now see it. you mean the host part of the zeebe configuration? I used the default zeebe docker image:

root@d812f4db6941:/usr/local/zeebe/bin# cat /usr/local/zeebe/conf/zeebe.cfg.toml
# Default Zeebe Config File. Used if no config file is provided to the broker.

# Networking configuration ----------------------------

[network]
host = "0.0.0.0"

Or the client? There it is

    Properties clientProperties = new Properties();
    clientProperties.put(ClientProperties.BROKER_CONTACTPOINT, "zeebe:51015");

    ZeebeClient client = ZeebeClient.create(clientProperties);
    client.connect();

menski · August 14, 2017, 3:41pm

Hi Bernd,

yeah that will not work. Sorry. We have to implement kind of advertising configuration to make Zeebe work in container environment. If the client ask the broker where to find different brokers they will return this address, which would be 0.0.0.0:51015 in your case. So the client will send the next request to 0.0.0.0:51015, which is localhost in its own container and not the zeebe container. You would have to change the host in the configuration to the container name, i.e. zeebe to make your example work. But it will then not work anymore with a linked port on your localhost. Did that make sense to you?

Cheers,
Sebastian

BerndRuecker · August 14, 2017, 3:46pm

OK, that already explains why it is not working!

We are talking about the host property of the zeebe broker, right? I cannot change it via environment variable from the outside in the default docker image, right? So in order to make that change I have to create my own zeebe docker image and change the config there, right?

But it will then not work anymore with a linked port on your localhost

You mean I cannot connect to localhost:5015 any more, as this is not bound to zeebe, correct? And both at once is currently not possible? Right?

menski · August 14, 2017, 3:54pm

We are talking about the host property of the zeebe broker, right?

Yes

I cannot change it via environment variable from the outside in the default docker image, right?

You can link a custom configuration into the container:

version: '2'                                                                                                                                                                                                                       

services:

  broker-1:
    image: camunda/zeebe:0.1.0
    volumes:
        - $PWD/broker-1/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml

You mean I cannot connect to localhost:5015 any more, as this is not bound to zeebe, correct? And both at once is currently not possible?

Yes. Lets try to explain that in more detail what the client is doing. If you specify a contact point for the client it will request the topology from this contact point. This topology will contain the address of every broker known by the cluster. And these addresses are equal to the host property at the moment. So the first topology request will work but then the client tries to connect to zeebe:51015 which is not resolvable from your localhost. On linux/unix you could do a hack and add zeebe as an alias for 127.0.0.1 in your /etc/host file. But not sure how to do this in windows.

Cheers,
Sebastian

BerndRuecker · August 14, 2017, 7:14pm

OK - understood. Thank you very much for the explanation! With this I got it to work For now I can also set the host “zeebe” hardcoded to localhost (also possible in windows).

I created an additional issue as I think the host configuration should be an environment variable in the docker world, would things make much easier (no I have to copy a zeebe conf which might need to be adjusted with every version change): https://github.com/zeebe-io/zeebe/issues/402. OK?

menski · August 15, 2017, 12:18am

Thanks Bernd, yeah we agree that most of the network configuration should be configurable through env variables as this would help to operate zeebe in a container environment. Thanks for the issue.

Ingo_Richtsmeier · August 15, 2017, 11:07am

Maybe a dumb question, but woudn’t a proxy help to communicate with the broker from the local host machine?

I found a german example here: https://gist.github.com/psct/bbb859d1767a8b990a6e27951859513e (not compose, but the network setup should be similar)

Cheers, Ingo

menski · August 15, 2017, 11:16am

Hi Ingo,

what do you want to achieve by using a proxy?

Cheers,
Sebastian

BerndRuecker · August 15, 2017, 11:24am

I think a proxy cannot help the core problem, that I have to configure zeebe to use some meaningful hostname in the docker-compose network (e.g. “zeebe”) which is not known to the outside world. We could maybe run our own DNS to cheat - but that’s not worth the effort I think…

Ingo_Richtsmeier · August 15, 2017, 11:47am

I think docker (compose) offers some configuration to run all containers (brocker, client, …) in a virtual (closed) network and I want to open this network to the “public” localhost with the tcp endpoints, that are really used (without hacking the /etc/hosts…).

Maybe a proxy is too much, but the network should do some DNS stuff here. Let’s play with the details tomorrow ;-).

BerndRuecker · August 15, 2017, 12:03pm

Hi Ingo.
The problem is that you have to configure Zeebe to use an hard coded host name which is then always returned to all clients - independent of your network topology. And the network topology is different within the docker-compose network than on the outside. I think there is no way around having a hostname set on the outside. But we can also discuss F2F tomorrow - but so far I do not see a good way of resolving this.

The only workaround i have in the back of my head is to make one docker image extending zeebe and adding the monitor. This way both Java applications run on the same host and can continue using localhost. But this will not solve any problems when running all of it in the cloud, where you have to use proper host names anyway. Let’s see. That’s why I want to provision everything and not only draw boxes in ppt

Cheers
Bernd

system · January 31, 2024, 10:09am