I’m trying to setup a very simple kafka pipeline that consumes a topic which is populated with a transactional producer (ie the topic contains transaction markers too).
CREATE OR REPLACE PIPELINE
mypipeline AS LOAD DATA KAFKA ‘broker/topic’ INTO TABLE table`;
When testing the pipeline, I’m getting the following error:
ERROR 1934 ER_EXTRACTOR_EXTRACTOR_EXTRACT: Leaf Error (127.0.0.1:3306): Leaf Error (127.0.0.1:3307): Cannot extract data for pipeline
2019-10-04 10:15:02.652 Waiting for next batch, 255 batches left until expiration.
2019-10-04 10:15:02.652 Batch starting with new consumer.
2019-10-04 10:15:02.820 Partition EOF stremaing offset from Kafka, resetting to earliest known offset
2019-10-04 10:15:02.922 invalid message offset, consumed out of order
If I try the very same pipeline with a topic that’s identical to the previous one but no transactions, it works fine.
Does the kafka pipeline work with transactional topics at all? This issue may be similar: https://stackoverflow.com/a/54666226