Troubleshoot Pipelines

Off-Topic
1

Hello!

I am trying to use MemSQL Pipelines to ingest some data from Kafka and process it by a stored procedure.

Although the TEST PIPELINE works like a charm, and produces JSON messages from Kafka, when I start the pipeline, it’s shown as “running” when I check it with “show pipelines” statement, and then the state changes to “Paused due to error. Run START PIPELINE or consider setting pipelines_stop_on_error to false”.

I don’t see any error messages coming in memsql.log or tracelog.log in /var/lib/memsql/{ID}, even with set global general_log = ON.

Is there any way to see the actual error to dig up the reason behind?

Regards,
Denis

2

Oh, it turns out that I never noticed the START PIPELINE … FOREGROUND option. It showed me the actual error.

3

You can see errors from “background” pipelines in information_schema.pipelines_errors, though start pipeline foreground is indeed the better next step when first building a pipeline.

It’s worth getting to know the pipelines info schema tables in general: