Shishir Khandelwal: I am trying to get a 1 gateway & 3 broker setup done on an env with ipv6.
I am getting this in each broker, so I know they are starting correctly.
however when I try to exec inside the broker and try “zbctl status” or “zbctl status --insecure” I keep getting this error
root@z2943-broker1-01:/usr/local/zeebe# zbctl status
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
root@z2943-broker1-01:/usr/local/zeebe# zbctl status --insecure
Error: rpc error: code = Unknown desc =
If I try to specify the pod address…
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: address tcp/c0e0: unknown port"
root@z2943-broker1-01:/usr/local/zeebe# zbctl status --address [240b:c0e0:202:5e2b:b424:2:0:51f]
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: address [240b:c0e0: missing ']' in address"
My guess is that the brokers are not able to differentiate between a ipv4 and ipv6 address.
My settings are
for broker
image version is 0.25.1
env vars are ->
ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT: '3'
ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR: '3'
ZEEBE_BROKER_CLUSTER_CLUSTERSIZE: '3'
ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS: '{{APP_NAME}}-broker1-cl-0:26502,{{APP_NAME}}-broker2-cl-0:26502,{{APP_NAME}}-broker3-cl-0:26502'
ZEEBE_BROKER_CLUSTER_CLUSTERNAME: 'zeebe-cluster'
ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT: '26501'
ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT: '26502'
ZEEBE_BROKER_NETWORK_MONITOIRNGAPI_PORT: '9600'
for gateway
image version is 0.25.1
env vars are ->
ZEEBE_STANDALONE_GATEWAY: 'true'
ZEEBE_GATEWAY_NETWORK_HOST: '0.0.0.0'
ZEEBE_GATEWAY_NETWORK_PORT: '26500'
ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT: '{{APP_NAME}}-broker1-cl-0:26502'
ZEEBE_GATEWAY_CLUSTER_CLUSTERNAME: 'zeebe-cluster'
ZEEBE_GATEWAY_CLUSTER_MEMBERID: '3'
ZEEBE_GATEWAY_CLUSTER_HOST: '0.0.0.0'
ZEEBE_GATEWAY_CLUSTER_PORT: '26502'
I have also tried the exactly same setup inside an ipv4 env, able to do “zbctl status --insecure” over there.
Josh Wulf: @korthout - can Zeebe work over ipv6?
korthout: I really don’t know 
Maybe @Nicolas knows this
Mohit Mehra: There should be YAMLS examples also apart from kubernates …for setting cluster on different servers that i feel
Nicolas: Zeebe can work over ipv6 - I think some users mentioned issues with the Helm chart there, I don’t know. But Zeebe will just pick whatever iface you give it and bind to it - ipv4 or ipv6 doesn’t matter.
Shishir Khandelwal: @Nicolas
I am getting the below error inside gateway, it does not look like gateway is able to identify the address as an ipv6 address. And that is why treating the colon (which is inside the ipv6 address) as the colon separating ipv4 address with it’s port.
Shishir Khandelwal:
2020-12-02 07:45:11.612 [] [grpc-default-executor-0] ERROR io.grpc.internal.SerializingExecutor - Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@501e95a8
java.lang.NumberFormatException: For input string: "c0e0"
at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:?]
at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
at java.lang.Integer.parseInt(Unknown Source) ~[?:?]
at io.zeebe.gateway.EndpointManager.addBrokerInfo(EndpointManager.java:78) ~[zeebe-gateway-0.25.1.jar:0.25.1]
at io.zeebe.gateway.EndpointManager.lambda$topology$1(EndpointManager.java:253) ~[zeebe-gateway-0.25.1.jar:0.25.1]
at org.agrona.collections.IntArrayList.forEach(IntArrayList.java:588) ~[agrona-1.8.0.jar:1.8.0]
at io.zeebe.gateway.EndpointManager.topology(EndpointManager.java:250) ~[zeebe-gateway-0.25.1.jar:0.25.1]
at io.zeebe.gateway.GatewayGrpcService.topology(GatewayGrpcService.java:135) ~[zeebe-gateway-0.25.1.jar:0.25.1]
at io.zeebe.gateway.protocol.GatewayGrpc$MethodHandlers.invoke(GatewayGrpc.java:1532) ~[zeebe-gateway-protocol-impl-0.25.1.jar:0.25.1]
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182) ~[grpc-stub-1.33.0.jar:1.33.0]
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:814) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.33.0.jar:1.33.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) [grpc-core-1.33.0.jar:1.33.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Shishir Khandelwal:
kubectl get pods -o wide | grep z2943
z2943-broker1-01 1/1 Running 0 54m 240b:c0e0:202:5e2b:b424:2:0:51f
z2943-broker2-01 1/1 Running 0 54m 240b:c0e0:202:5e2b:b424:2:0:37b
z2943-broker3-01 1/1 Running 0 54m 240b:c0e0:202:5e2b:b424:2:0:c56
z2943-gateway-01 1/1 Running 0 53m 240b:c0e0:202:5e2b:b424:2:0:af3
for your reference these are my zeebe related pods 
Nicolas: Ah, in the topology manager. Interesting 
I was mostly focused on the transport layer, and figured it should work regardless, but it could be that we sometimes parse broker addresses in the topology and expect them to be ipv4
Shishir Khandelwal: Yes, need your help to overcome this issue.
Also, if there is anything else you require from logs, settings to understand the issue better?
I can provide right away.
@Nicolas
Nicolas: No I understand the issue - but it’d be very helpful if you could open an issue on our Github page https://github.com/zeebe-io/zeebe
Nicolas: I’m afraid the only workaround right now is to use ipv4 =/
Shishir Khandelwal: It is my understanding that when zeebe broker starts, it performs 11 or 14 steps before it becomes ready for use. In the 7th step (cluster services step) it looks for other brokers to connect with. In my ipv6 setup - all the brokers are able to complete this step.
-So does this not mean zeebe CAN make connections on ipv6?
Josh Wulf: Sounds like some parts of it can, and some parts assume an ipv4 address
Shishir Khandelwal: Yes, could be.
Is there some way I can verify if it is indeed ipv6 parsing issue causing this. @Nicolas @Josh Wulf
Also, I have opened an issue for this on github as you asked @Nicolas https://github.com/zeebe-io/zeebe/issues/5951
Nicolas: I looked at the code, and I can guarantee it is the parsing 
Nicolas: There’s essentially two places where we split the address by ‘:’. So you could look for these in the code .split(":") and replace them properly, then run your own artifacts and see if it works. But at least looking at these it’s obvious it won’t without changing the code first 
Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!
If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.