Clarification on Exporter API is needed

Camunda 8 Topics Discussion & Questions
1

I’m working on my exporter (NATS/NATS Streaming), implementing Exporter.export() method.
In my case the order of exports’ completions may be different than the order of records being received.
When should I call Controller#updateLastExportedRecordPosition?

Javadoc on Exporter interface says:

Exporter.export(Record record): Called at least once for every record to be exported. Once a record is guaranteed to have been exported, implementations should call Controller#updateLastExportedRecordPosition to signal that this record should not be received here ever again.

But I see two contradicting approaches on when Controller#updateLastExportedRecordPosition must be called:

  1. On each successfully exported record, meaning the order of records (their positions) does not matter
  2. On the last successfully exported record in the order records received, meaning that all the calls of Controller#updateLastExportedRecordPosition must be with monotonically increasing positions.

What approach is correct?

P.S.: my biggest concern is not to lose failed to export record with position 1 after I signaled updateLastExportedRecordPosition(2)

2

Normally you will receive a record in the exporter, then you will export it, then you will update the exporter position to mark this record as exported.

If you are doing something like batching records before exporting them, then you would update the exporter position to the last record in the batch, after the batch is exported.

If your exporter crashes mid-way through a batch, then on restart, it will reprocess the batch from the end of the last batch, because the exporter position has not been updated.

Does that answer your question?

4 Likes
3

Thank you @jwulf
Now it’s clear.
I suggest syncing the documentation with your answer.

1 Like
4