MemSQL CIAB Container Restart

[boldy]

Docker run command used to initially start container: docker run -i --init --name memsql -v memsql-data:/var/lib/memsql -e LICENSE_KEY=<LICENSE KEY> -p 3306:3306 -p 8080:8080 memsql/cluster-in-a-box

I found myself needing to restart the server my memsql/cluster-in-a-box container was running on. I stopped the container manually, using docker stop memsql --time 240, prior to restarting the server.

When starting the container, I receive

40715851 2020-06-01 15:19:42.285 ERROR: ProcessNetworkEvents Heartbeat connection error reading heartbeat response header to 127.0.0.1:3307 (2 = End of file)
40866808 2020-06-01 15:19:42.436 ERROR: ProcessNetworkEvents Heartbeat connection error reading heartbeat response header to 127.0.0.1:3307 (2 = End of file)
41017797 2020-06-01 15:19:42.587 ERROR: ProcessNetworkEvents Heartbeat connection error reading heartbeat response header to 127.0.0.1:3307 (2 = End of file)
41062189 2020-06-01 15:19:42.631 ERROR: Thread 115055: Fn: Waiting for node to catch up with cluster state before starting replication management.
41149852 2020-06-01 15:19:42.719 ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user ‘root’@‘localhost’ (using password: NO)
: Failed to connect to MemSQL: Could not establish connection: Error 1045: Access denied for user ‘root’@‘localhost’ (using password: NO)
Traceback (most recent call last):
File “/startup”, line 113, in
start_cluster()
File “/startup”, line 79, in start_cluster
ctl(“start-node”, “–all”)
File “/startup”, line 18, in ctl
subprocess.check_output([“memsqlctl”, “-yj”] + list(args)))
File “/usr/lib64/python2.7/subprocess.py”, line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command ‘[‘memsqlctl’, ‘-yj’, ‘start-node’, ‘–all’]’ returned non-zero exit status 1

Attaching a shell to the container and using the following does not work:

• memsql-admin change-root-password --password ''
• memsql-admin change-root-password --password '' --all
• memsql-admin change-root-password --password
• memsql-admin change-root-password --password --all
• memsqlctl change-root-password --password ''
• memsqlctl change-root-password --password '' -all
• memsqlctl change-root-password --password
• memsqlctl change-root-password --password -all

Starting another container with the same volume results in the same error. Am I missing something blatantly obvious here?

[robrich]

This is really weird. I like the repro steps here. Do you have a minimum table create & insert that reproduces this problem?

[boldy]

Hi Rob! Apologies on the delay. I believe the issue was driven by changing the root password incorrectly. I think I did it in such a way that the change wasn’t at the cluster level, so when I restarted the container, the root password didn’t match between nodes. Pure conjecture, of course.

I ended up copying the files directly out of the containers volume in var/lib/docker/volumes, creating a new container, and copying the files into the new containers volume. Unfortunately I don’t remember the exact steps I took, I was panicking a little bit!