MemSQL on K8S - cluster pods failing


#1

Hi,

We’re trying to do a POC of MemSQL on kubernetes using the new operator.
We’re following the instructions here

When deploying the cluster, the pods are crashing, a closer look showed the below errors.
We’re using Azure AKS, created a StorageClass as per Azure filesystem documentation in case that’s the problem.

Events:
Type Reason Age From Message


Normal Scheduled 34m default-scheduler Successfully assigned default/node-memsql-test-aks-cluster-master-0 to aks-agentpool-25738454-1
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (a7dd34e3b6ad713798bcce9af8fcab460ad9dfdafd86c41692e654248914129d)
Warning Unhealthy 33m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (a7dd34e3b6ad713798bcce9af8fcab460ad9dfdafd86c41692e654248914129d)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (adc84edad2a1fdcd78454e4b58d01058a25c9f0a970deab273ac13eccb96f3a2)
Warning Unhealthy 33m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (adc84edad2a1fdcd78454e4b58d01058a25c9f0a970deab273ac13eccb96f3a2)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (b3b27f0b8145702a3a19869a62c486085f4329d6bb27c5724abd12be31c40f21)
Warning Unhealthy 33m (x2 over 33m) kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (b3b27f0b8145702a3a19869a62c486085f4329d6bb27c5724abd12be31c40f21)
Normal Pulled 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Container image “memsql/dynamic-node:ae177265fe28294c42499e0e31af72a31d0a0edd” already present on machine
Normal Started 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Started container
Normal Created 32m (x4 over 34m) kubelet, aks-agentpool-25738454-1 Created container
Warning Unhealthy 32m kubelet, aks-agentpool-25738454-1 Readiness probe errored: rpc error: code = Unknown desc = container not running (c9b43903257800df73f4357aa2f4cdef4b9e2cfc6d36ffd8a6e55428d121d2b9)
Warning Unhealthy 32m kubelet, aks-agentpool-25738454-1 Liveness probe errored: rpc error: code = Unknown desc = container not running (c9b43903257800df73f4357aa2f4cdef4b9e2cfc6d36ffd8a6e55428d121d2b9)
Warning BackOff 4m (x123 over 33m) kubelet, aks-agentpool-25738454-1 Back-off restarting failed container

$ k logs node-memsql-test-aks-cluster-master-0

Would you like to continue? [Y/n]:
Creating node
colliding port: 3306
✘ Failed to create node with Memsql ID

Any help would me most appreciated.

Regards,
Naim


#2

Hi Naim, it looks like you are getting a port collision on 3306 when creating the nodes. MemSQL uses 3306 for communication. Is this port being blocked or used by something else in your infrastructure?


#3

first of all thank you for pointing me to a lead, I’ll investigate further and if that’s in fact the problem I’ll find it.

I doubt I’m using that port for anything, but just to be clear what do you mean by “something else in your infrastructure” ?
I’ve created this Azure AKS only for POC’ing mem, so nothing else is running there.