MemSQL on K8S - cluster pods failing

Hi,

We’re trying to do a POC of MemSQL on kubernetes using the new operator.
We’re following the instructions here

When deploying the cluster, the pods are crashing, a closer look showed the below errors.
We’re using Azure AKS, created a StorageClass as per Azure filesystem documentation in case that’s the problem.

Events:
Type Reason Age From Message


Normal Scheduled 34m default-scheduler Successfully assigned default/node-memsql-test-aks-cluster-master-0 to aks-agentpool-25738454-1
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (a7dd34e3b6ad713798bcce9af8fcab460ad9dfdafd86c41692e654248914129d)
Warning Unhealthy 33m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (a7dd34e3b6ad713798bcce9af8fcab460ad9dfdafd86c41692e654248914129d)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (adc84edad2a1fdcd78454e4b58d01058a25c9f0a970deab273ac13eccb96f3a2)
Warning Unhealthy 33m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (adc84edad2a1fdcd78454e4b58d01058a25c9f0a970deab273ac13eccb96f3a2)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (b3b27f0b8145702a3a19869a62c486085f4329d6bb27c5724abd12be31c40f21)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (b3b27f0b8145702a3a19869a62c486085f4329d6bb27c5724abd12be31c40f21)
Normal Pulled 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Container image “memsql/dynamic-node:ae177265fe28294c42499e0e31af72a31d0a0edd” already present on machine
Normal Started 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Started container
Normal Created 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Created container
Warning Unhealthy 32m kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (c9b43903257800df73f4357aa2f4cdef4b9e2cfc6d36ffd8a6e55428d121d2b9)
Warning Unhealthy 32m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (c9b43903257800df73f4357aa2f4cdef4b9e2cfc6d36ffd8a6e55428d121d2b9)
Warning BackOff 4m (x123 over 33m) kubelet, aks-agentpool-25738454-1 Back-off restarting failed container

$ k logs node-memsql-test-aks-cluster-master-0

Would you like to continue? [Y/n]:
Creating node
colliding port: 3306
✘ Failed to create node with Memsql ID

Any help would me most appreciated.

Regards,
Naim

Hi Naim, it looks like you are getting a port collision on 3306 when creating the nodes. MemSQL uses 3306 for communication. Is this port being blocked or used by something else in your infrastructure?

first of all thank you for pointing me to a lead, I’ll investigate further and if that’s in fact the problem I’ll find it.

I doubt I’m using that port for anything, but just to be clear what do you mean by “something else in your infrastructure” ?
I’ve created this Azure AKS only for POC’ing mem, so nothing else is running there.