I have a table and a pipeline that inserts data from Kafka.
CREATE OR REPLACE PIPELINE player_audit AS LOAD DATA KAFKA 'cluster/topic' BATCH_INTERVAL 30000 SKIP DUPLICATE KEY ERRORS INTO TABLE player_audit FORMAT JSON ( @v_data <- data, ) SET ... fields set using JSON EXTRACT ... ;
Sometimes this pipeline misses events.
I stopped the pipeline, checked and saw 3 events in my kafka topic, but using
test pipeline I got only 1 event in the pipeline.
ALTER PIPELINE player_audit SET OFFSETS LATEST; it suddenly worked OK.
It is important to note that this is development cluster - not prod - so it is not that powerful.
2 nodes that are not working that hard.
Any ideas why this might have happened?