Pipelines load data into table based on a condition from one or more fields in the JSON message

Help Connectors and Pipelines
1

Hi,
I have created a pipeline that as load data Kafka replace into table option and mapped JSON fields to table columns and one of the Field in JSON into a variable @cust_type. I want to load data only when @cust_type = ‘A’ and skip other messages.

I tried it however it’s failing and when I looked at the record that’s failing @cust_type is not ‘A’

Does pipeline first parse JSON fields and map to user defined variables and table columns before where condition is evaluated? I think my issue is certain fields do not exist in the JSON message when customer type is not equal to ‘A’

2

It’s solved now. It required me to add default clause so that when a path isn’t found in the JSON, it’s assigned with default value and then where clause is applied.

1 Like