Pipelines load data into table based on a condition from one or more fields in the JSON message

I have created a pipeline that as load data Kafka replace into table option and mapped JSON fields to table columns and one of the Field in JSON into a variable @cust_type. I want to load data only when @cust_type = ‘A’ and skip other messages.

I tried it however it’s failing and when I looked at the record that’s failing @cust_type is not ‘A’

Does pipeline first parse JSON fields and map to user defined variables and table columns before where condition is evaluated? I think my issue is certain fields do not exist in the JSON message when customer type is not equal to ‘A’

It’s solved now. It required me to add default clause so that when a path isn’t found in the JSON, it’s assigned with default value and then where clause is applied.

1 Like