SSL Encryption For Kafka Pipelines

Help Security
1

For enabling SSL encryption (only) for Kafka pipelines, we created the pipeline like below:

CREATE PIPELINE `kafka_ssl`
AS LOAD DATA KAFKA '<kafka-host>:<kafka-port>/test'
CONFIG '{"security.protocol": "ssl"
"ssl.ca.location": "<path-to-the-ca-certificate>"}''
INTO table <t>;

from SingleStoreDB Cloud · SingleStore Documentation

Please note that we’re not looking for Client authentication (mutual TLS connection). This requires us to manually copy the CA Cert into memsql nodes and provide that path. This works fine for standalone Kafka cluster.

What is the recommended approach if we’re using AWS MSK?

Following documentation points to the client configuration for SSL encryption for MSK clients. They use client truststore location as a property which is not an identified property for memsql pipeline config json

2

We’re having the same issue. Unable to create a pipeline from Singlestore in a box to AWS MSK.

3

Hi DataSleek! :wave:
The team has confirmed that this issue was due to the Kafka topic publicly being inaccessible as the username/password had not been specified and was resolved during a live working session by removing the existing CONFIG and CREDENTIALS sections and adding this line: CONFIG '{"security.protocol": "ssl"}

We’d appreciate you marking this issue as resolved. Let us know if we can be of further support. Thank you! :pray: