I’m having trouble filtering variables being sent to Kafka as mentioned in GitHub - camunda-community-hub/kafka-connect-zeebe: Kafka Connector for Zeebe.io. The message which gets posted on my kafka topic is -
‘{
“bpmnProcessId”: “toKafka”,
“customHeaders”: {
“sendVars”: “testID, key”,
“topic”: “pong3”
},
“deadline”: 1583748375824,
“elementId”: “Task_13zltv1”,
“elementInstanceKey”: 2251799813687492,
“key”: 2251799813687493,
“retries”: 3,
“type”: “testService”,
“variables”: “{}”,
“variablesAsMap”: {},
“worker”: “kafka-connector-1”,
“workflowDefinitionVersion”: 1,
“workflowInstanceKey”: 2251799813687486,
“workflowKey”: 2251799813687421
}’
The filtering seems to happen only inside ‘variables’ and ‘variablesAsMaps’ fields of the message. Is there a way I can get rid of all the other keys like ‘bpmnProcessId’, ‘workflowInstanceKey’, ‘workflowKey’ etc so that my message becomes something as shown below?
‘{
“type”: “testService”,
“variables”: “{}”,
“variablesAsMap”: {}
}’
This is the job metadata. The variables are the key:values in the variables field.
Sorry, no one-line answer from me.
There may be a way to accomplish what you are trying to do (format the record being sent to Kafka) through configuration, either on the Kafka side or in the connector.
Someone else may have the answer, but I would start by checking out the connector source code and opening it in IntelliJ, or whatever you use as your IDE, and going through it to understand how it works. The answers are always in the source code.
Thanks @jwulf for the prompt reply. I had already gone through the source code before posting the query. I just wanted to be sure that my understanding of ‘filter variables’ was in sync with the functionality and probably find someone who has already faced and fixed what I am facing. The links you have shared seem promising. Thanks for that.
