I am trying to send one large message(about 50mb) using java client to a broker with gateway.
I have change 'maxMessageSize` to 80MB in the config file, below is the stack trace of the error.
Jul 10, 2020 2:57:58 PM io.grpc.netty.NettyServerStream$TransportState deframeFailed
WARNING: Exception processing message
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 79571781
at io.grpc.Status.asRuntimeException(Status.java:524)
at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:387)
at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:267)
at io.grpc.internal.MessageDeframer.request(MessageDeframer.java:161)
at io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:205)
at io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:112)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
please don’t use Zeebe as storage system. It is not intended and not recommended to handle such big messages. We already have found issues with the current defaults, see for example this https://github.com/zeebe-io/zeebe/issues/4249
I would suggest that you store the data somewhere else, in a system which is intended to to handel such big data like mongo db or elastic etc. You can then use a key or id to reference the data and use zeebe only for orchestrating your services.
If you still want it, then yes you need to change the maxMessageSize.
zeebe:
broker:
gateway:
# Enable the embedded gateway to start on broker startup.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_ENABLE.
enable: true
network:
# Sets the port the embedded gateway binds to.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_NETWORK_PORT.
port: 26500
security:
# Enables TLS authentication between clients and the gateway
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED.
enabled: false
network:
# Controls the default host the broker should bind to. Can be overwritten on a
# per binding basis for client, management and replication
# This setting can also be overridden using the environment variable ZEEBE_BROKER_NETWORK_HOST.
host: localhost
maxMessageSize: 30MB
Configure maxMessageSize in this way but it does not take effect and still reports an error RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 6917673
# Zeebe Standalone Broker configuration file (with embedded gateway)
# This file is based on broker.standalone.yaml.template but stripped down to contain only a limited
# set of configuration options. These are a good starting point to get to know Zeebe.
# For advanced configuration options, have a look at the templates in this folder.
# !!! Note that this configuration is not suitable for running a standalone gateway. !!!
# If you want to run a standalone gateway node, please have a look at gateway.yaml.template
# ----------------------------------------------------
# Byte sizes
# For buffers and others must be specified as strings and follow the following
# format: "10U" where U (unit) must be replaced with KB = Kilobytes, MB = Megabytes or GB = Gigabytes.
# If unit is omitted then the default unit is simply bytes.
# Example:
# sendBufferSize = "16MB" (creates a buffer of 16 Megabytes)
#
# Time units
# Timeouts, intervals, and the likes, must be specified either in the standard ISO-8601 format used
# by java.time.Duration, or as strings with the following format: "VU", where:
# - V is a numerical value (e.g. 1, 5, 10, etc.)
# - U is the unit, one of: ms = Millis, s = Seconds, m = Minutes, or h = Hours
#
# Paths:
# Relative paths are resolved relative to the installation directory of the broker.
# ----------------------------------------------------
zeebe:
broker:
gateway:
# Enable the embedded gateway to start on broker startup.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_ENABLE.
enable: true
network:
# Sets the port the embedded gateway binds to.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_NETWORK_PORT.
port: 26500
maxMessageSize: 30MB
security:
# Enables TLS authentication between clients and the gateway
# This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED.
enabled: false
network:
# Controls the default host the broker should bind to. Can be overwritten on a
# per binding basis for client, management and replication
# This setting can also be overridden using the environment variable ZEEBE_BROKER_NETWORK_HOST.
host: localhost
maxMessageSize: 30MB
data:
# Specify a directory in which data is stored.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_DIRECTORY.
directory: data
# The size of data log segment files.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_LOGSEGMENTSIZE.
logSegmentSize: 512MB
# How often we take snapshots of streams (time unit)
# This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_SNAPSHOTPERIOD.
snapshotPeriod: 15m
cluster:
# Specifies the Zeebe cluster size.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_CLUSTERSIZE.
clusterSize: 1
# Controls the replication factor, which defines the count of replicas per partition.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR.
replicationFactor: 1
# Controls the number of partitions, which should exist in the cluster.
# This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT.
partitionsCount: 1
threads:
# Controls the number of non-blocking CPU threads to be used.
# WARNING: You should never specify a value that is larger than the number of physical cores
# available. Good practice is to leave 1-2 cores for ioThreads and the operating
# system (it has to run somewhere). For example, when running Zeebe on a machine
# which has 4 cores, a good value would be 2.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
cpuThreadCount: 2
# Controls the number of io threads to be used.
# This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_IOTHREADCOUNT
ioThreadCount: 2
The zeebe configuration file has been configured as described above. However, the following error still occurred. What is the reason?
zeebe.log
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 6917403
at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-api-1.37.0.jar:1.37.0]
at io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:391) ~[grpc-core-1.37.0.jar:1.37.0]
at io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:271) ~[grpc-core-1.37.0.jar:1.37.0]
at io.grpc.internal.MessageDeframer.request(MessageDeframer.java:161) ~[grpc-core-1.37.0.jar:1.37.0]
at io.grpc.internal.AbstractStream$TransportState$1RequestRunnable.run(AbstractStream.java:239) [grpc-core-1.37.0.jar:1.37.0]
at io.grpc.netty.NettyServerStream$TransportState$1.run(NettyServerStream.java:202) [grpc-netty-1.37.0.jar:1.37.0]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-native-epoll-4.1.63.Final-linux-x86_64.jar:4.1.63.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.63.Final.jar:4.1.63.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
client.error
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: CANCELLED: HTTP/2 error code: CANCEL
Received Rst Stream
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:59)
... 83 common frames omitted
Caused by: io.grpc.StatusRuntimeException: CANCELLED: HTTP/2 error code: CANCEL
Received Rst Stream
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 common frames omitted
The client does not support receiving responses larger than 4MB at this time (see feature request). So, while your request was handled server-side, the client could not receive the response.
From the stack trace you shared, I cannot see what command you sent and what response was expected. Can you share where in your code this happened? It seems to me that it is no longer the one you shared in your OP.
It is not clear to me how this command could produce an error with message gRPC message exceeds maximum size 4194304: 6917403 when you have configured the maxMessageSize to 30MB.
I’ve already shown that the configuration should work correctly with the create process instance command. So, please make sure to have another look at what I’ve written there. Or please provide a minimal reproducible code example. Otherwise, I am not able to help.
