How Kafka pipeline decides how to populate the table

Help Connectors and Pipelines
1

Hi,

I am new to MemSql but I can’t figure out the answer to my question from the documentation.
If I missed something documented in existing documentation, a link will be very much appreciated.

Suppose I have a table with colums: id, name, address, phone
Suppose I am configuring a pipeline to ingest JSONs from Kafka
How does the pipeline determines which field to put in which column?
Should the name of the field be the same as the column?
What if a field occurs in JSON that doesn’t match the column? Or vice versa?

Thanks,
Victoria

2

You can create a pipeline like this: SingleStore Pipeline from Kafka - YouTube and it’ll automatically map json properties into column names, matching by name. It also serializes types from JavaScript types into SQL types. Extra JSON data will be ignored, and missing JSON data will appear as SQL nulls.

CREATE PIPELINE mypipeline AS
LOAD DATA KAFKA '127.0.0.1/my-topic'
INTO TABLE my_table
FORMAT JSON;

Or you can pipe it into a stored procedure and handle the JSON there like this: SingleStore Pipeline to Stored Procedure - YouTube