Kafka Pipeline consuming a transactional kafka topic

Hi,

I’m trying to setup a very simple kafka pipeline that consumes a topic which is populated with a transactional producer (ie the topic contains transaction markers too).

CREATE OR REPLACE PIPELINE mypipeline AS LOAD DATA KAFKA ‘broker/topic’ INTO TABLE table`;

When testing the pipeline, I’m getting the following error:

ERROR 1934 ER_EXTRACTOR_EXTRACTOR_EXTRACT: Leaf Error (127.0.0.1:3306): Leaf Error (127.0.0.1:3307): Cannot extract data for pipeline mypipeline. Stderr:
2019-10-04 10:15:02.652 Waiting for next batch, 255 batches left until expiration.
2019-10-04 10:15:02.652 Batch starting with new consumer.
2019-10-04 10:15:02.820 Partition EOF stremaing offset from Kafka, resetting to earliest known offset
2019-10-04 10:15:02.922 invalid message offset, consumed out of order

If I try the very same pipeline with a topic that’s identical to the previous one but no transactions, it works fine.

Does the kafka pipeline work with transactional topics at all? This issue may be similar: https://stackoverflow.com/a/54666226

Thanks,
Laszlo

Hello Laszlo. thanks for trying out MemSQL pipelines.

at this time, MemSQL uses non-transactional Kafka consumers, and keeps its own records of partitions and consumed offsets for each topic. Using a transactional producer is likely incompatible with this.

Could you describe a bit more about your use-case? How do you use Kafka transactions?

Hi,

We have complex streaming pipelines using kafka EOS (transactions) and we simply want to ingest some of the topics in memsql. All we need is a consumer’s capable of handling the “isolation.level”: “read_committed”, nothing else really. Not sure if this is the problem I have, but potentially may be, because I tried to ingest 2, almost identical topics: one had transaction markers in it, other didn’t, and the former failed.

Thanks,
Laszlo