That’s an interesting feature. How consistent are your headers and trailers? Can they be assumed to be just one line or are they more complex?
The simplest option for on-premise clusters is to employ a transform which strips them out. https://docs.memsql.com/v7.1/concepts/pipelines/transforms/
You can get the filename in the process, which can be passed into the stored procedure via
pipeline_source_file() builtins. Here’s an example:
MySQL [db]> create or replace procedure proc(q query(a int, batch int)) as begin insert into t select batch from q; end //
Query OK, 1 row affected (0.015 sec)
MySQL [db]> create or replace pipeline p as load data fs "/tmp/*.csv" into procedure proc(a) set batch = pipeline_batch_id()//
Query OK, 0 rows affected (0.196 sec)
Note that you can always find the source for the pipeline batch by looking at the relevant tables in INFORMATION_SCHEMA, e.g. PIPELINES_BATCHES.
However, the transform may need to do something extra for headers and trailer, e.g. tag the line with a code so that you can properly route the row in a stored procedure later.