Pipeline unable to ingest JSON data with missing fields

I am trying to ingest JSON data from Kafka but it’s failing since some messages do not have all the fields in it.

I have complete list of fields that may be part of JSON in some cases while in other cases it contains only subset of fields(subset could be 1 or more fields). How can I specify when a field doesn’t exist then use default value while mapping it to a table column?


I was able to solve it by specifying default clause and literal expr was assigned to the column when path can’t be found in input JSON

1 Like