Kafka specific question. For compacted topics, a Key:Value can be deleted by producing a Key:NULL to the topic. Given that the pipeline seems only interested in the values, and not the keys, is there a way to properly process these events?
No. In current versions of the product, we don’t consider kafka keys, except for their use in deciding the sharding within kafka (that is, for a non-compacted topic, values with the same key can be configured to be processed in order). Since MemSQL reads directly at the requested Kafka offsets, it will read topic uncompacted, so it will see each update as it comes. In this case, an SP would have to handle it.