Hi,
I am deploying some complex workflows with zeebe. My solution is micro services based (.net core). I have a Grpc server which deploys workflow instances. One of my micro services is responsible of image treatments and I am sending images as Base64. When I initiate a new workflow instance, I receive all the variables except the images. Looking in the traces, I found:
“”"
2020-01-15 14:29:40.560 [exporter-1] [172.22.0.5:26501-zb-fs-workers-0] WARN io.zeebe.broker.exporter.elasticsearch - Failed to flush at least one bulk request ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Document contains at least one immense term in field=“value.value” (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: ‘[91, 123, 34, 70, 105, 110, 103, 101, 114, 73, 100, 34, 58, 34, 52, 98, 52, 100, 56, 100, 97, 56, 45, 57, 49, 56, 49, 45, 52, 48]…’, original message: bytes can be at most 32766 in length; got 332913]]; nested: ElasticsearchException[Elasticsearch exception [type=max_bytes_length_exceeded_exception, reason=bytes can be at most 32766 in length; got 332913]];,
“”"
I know that this is an elasticsearch issue, but anyway has a workaround ?
Thank you
@nizare this is an interesting question… the problem as you can see is in ElasticSearchExporter… For use cases like yours, where you are sending a large amount of text, we should have some kind of mechanism to detect such situations to use another special field that can be compressed for example.
I am not aware of any work around… but can you create an issue so we can include this as a use case for the exporter?
Cheers
Hi salaboy, thank you for your comment, I have already created an issue https://github.com/zeebe-io/zeebe/issues/3673. Actually I am really blocked and looking forward to receiving any reply
@nizare while thinking about this problem, I wonder why do you need to send the image payload to a process variable, is that really needed? Usually I would avoid doing that.
I am not saying that what you reported is not an issue, but in the meantime, can you please elaborate on why do you need to send that content in the process variables?
Cheers
@salaboy I have a micro service which accepts a Person Object as argument and identifies the Person using the fingerprints (it is an ABIS system). So, in my zeebe workflow, when I am calling my micro service using the http worker, I am setting the fingerprints variable as an array of base64 WSQ images.
@nizare that is ok in principle, but what I’ve seen in different implementations of BPM engines is that you usually can add just a link to the resource, in your case the fingerprints, and then you just have a string with the link. Unless you are going to use the fingerprint content in the process to make decisions I would just deal with it as an external resource.
That, of course, will give you more work on the worker side which will need to push the fingerprint content to an external storage, in the cloud world that would be a Bucket such as S3 Bucket or hosted such as https://min.io for a distributed architecture.
Hopefully this makes sense.
@salaboy yes this makes sense, however I really prefer not to use external resources. I know that if I use hazelcast and simple monitor it works just fine because the problem is being caused by the elastic search limitations. Anyway, I am trying to find a workaround because I really like elastic search and ELK coupled with Zeebe to offer best system visibility and facilitates the administrator’s tasks. My system is complex, we have 20 microservices all based on Event Sourcing Pattern and use Raven DB and Event Store. It is being a real challenge for me to make it work correctly and process millions of transactions per day.
Kind Regards
@nizare I know that you prefer not to use an external resource, but if you want to scale you might be forced to do that… even if the ElasticSearch issue gets fixed… that is mostly because you want to avoid moving content around in your requests.