Kafka Pipelines - configure kafka consumer

Hi everyone!

I am very eager to try out the Pipelines functionality in MemSQL to load some JSON-formatted data from Kafka.

However, it is quite important to me to be able to configure Kafka consumer roperties. E.g. define group.id, auto.offset.reset and certain other properties for each pipeline.

Could you advise if there is any way to do that?

Regards,
Denis

Hello Denis! thanks for trying out MemSQL Pipelines.

MemSQL pipelines does not use group.id or auto.offset.reset, but rather it has its own logic tracking the offsets for each topic. This has a few implications, e.g. only one topic per pipeline, but in general allows MemSQL to retry batches when needed and enforce exactly once semantics.

Could you explain why you need pipelines to use consumer groups? It’s possible that we can find a workable solution.

Hello @mkobyakov!

Thank you for a quick reply.

I guess, if MemSQL manages offsets all by itself, then there is no reason to keep track of the data ingestion process from my end.

Cheers,
Denis

1 Like