CREATE PIPELINE fails

Hello!

I would be grateful to get an insight in why the CREATE PIPELINE actually fails to execute.

  • There is a topic “ods.siebel.s_srv_req_ct.raw” in our Kafka cluster, with a lot of JSON messages in it
  • Kafka cluster is accessible from the aggregator-node and all leaf-nodes (there are only 2 of them in my test instance). Checked it with telnet.

My SQL:
CREATE PIPELINE s_srv_req_ct_3_prod
AS LOAD DATA KAFKA ‘{host}:9092/ods.siebel.s_srv_req_ct.raw’
BATCH_INTERVAL 1
INTO PROCEDURE cdc_s_srv_req_ct_3

SQL Error I’m getting:
SQL Error [1970] [HY000]: Subprocess timed out. Stderr tail:
n.
2019-08-23 14:48:23.092 Waiting for next batch, 237 batches left until expiration.
2019-08-23 14:48:26.071 Waiting for next batch, 236 batches left until expiration.
2019-08-23 14:48:29.460 Waiting for next batch, 235 batches left until expiration.
2019-08-23 14:48:32.906 Waiting for next batch, 234 batches left until expiration.
2019-08-23 14:48:37.390 Waiting for next batch, 233 batches left until expiration.
2019-08-23 14:48:39.342 Batch starting with new consumer.

Any help would be greately appreciated.

Denis

The most likely reason for this is that you’re timing out is because there’s some issue connecting to that particular kafka broker from your MA. This looks like 6.8? It might be worth setting pipelines_debug_logging to true, running flush extractor pools and trying again to figure out why its hanging.

It seems like you have a pipeline that already works. Is that one connected to the same kafka? Whats the difference between that other pipeline and this one?