We are creating pipeline that uses AVRO schema to ingest data in tables from kafka. As with avro we get flexibility of evolving our schema we are looking for a similar feature with pipeline too.
But with Alter pipeline we noticed that there is no option to update schema, we can only set offsets/transforms thus our only solution left to achieve this is
2)fetch offset from all partitions
3)Create new pipeline with updated schema
4)Alter pipeline with previous pipeline offset
5)Drop old pipeline
6)start this new pipeline.
This looks tedious specially maintaining offsets. Is memsql has any plans to support schema evolution or any better way to achieve this use case?