Failed to add user to the memsql group during cluster setup


#1

I’m trying to set up a three node memsql cluster (one aggregator and two leaf nodes) on CentOS 7.5. When running the memsql-deploy setup-cluster command, I’m getting the following error:
✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.

I am already in the sudoers file on all three nodes. Next, I manually added myself to the memsql group (which the deploy successfully created) on all three nodes per the instructions here:

However, when I re-run memsql-deploy setup-cluster, it yields the same error and rolls back the changes.

Here is the command I’m running:

memsql-deploy setup-cluster --license <license-key> --master-host 172.31.163.151 --leaf-hosts 172.31.163.152,172.31.163.153 --password <secure-password>

Here is the full output on the console:

    ✓ Downloaded memsql-server 6.7.15
    Installing MemSQL on all hosts...
    ✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
    ✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.152 (1/3)
    ✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
    ✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.151 (2/3)
    ✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
    ✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.153 (3/3)
    ✓ Successfully installed on 3 hosts
    Starting rollback
    ✓ Uninstalling MemSQL 6.7.15 on 172.31.163.151
    ✓ Uninstalling MemSQL 6.7.15 on 172.31.163.152
    ✓ Uninstalling MemSQL 6.7.15 on 172.31.163.153
    ✓ Unregistered hosts
    ✓ Rollback succeeded

Any idea how I get around this?

Thanks!


#2

Hi @srappaport,

Can you run the same command with highest verbosity, -vvv and paste the full output here?


#3

Hi @mbhojani,

The full, verbose output is too many character for one post so I’ll do it in two:

⠋ Downloading memsql-server 6.7.15 Debug 2019/04/04 15:44:02 downloader.go:90 Fetching package meta from https://release.memsql.com/production/index/memsqlserver/latest.json
⠼ Downloading memsql-server 6.7.15 Debug 2019/04/04 15:44:03 downloader.go:41 Downloading from Freya into /tmp/memsql-server-rpm860578428
Debug 2019/04/04 15:44:03 downloader.go:128 Downloading https://release.memsql.com/production/rpm/x86_64/memsql-server-6.7.15-d4450c6798.x86_64.rpm
✓ Downloaded memsql-server 6.7.15
Installing MemSQL on all hosts...
2019/04/04 15:44:04 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠹ Checking hosts for existing MemSQL packages 2019/04/04 15:44:05 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 1.101662951s
2019/04/04 15:44:05 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠴ Checking hosts for existing MemSQL packages 2019/04/04 15:44:06 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 1.24267764s
2019/04/04 15:44:06 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠴ Checking hosts for existing MemSQL packages 2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 1.073820247s
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''`
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''`
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''`
⠧ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''` took 152.641251ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''`
2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''` took 192.2576ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''`
⠇ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''printenv'\'' '\''SUDO_USER'\'''` took 277.007335ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''`
⠏ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''` took 162.706809ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.9QGY7La4a5'\'''`
2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''` took 166.251898ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.VN6bEvj0hN'\'''`
⠋ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:07 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.9QGY7La4a5'\'''` took 172.844279ms
2019/04/04 15:44:07 main.go:39 Running command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/3.socket /tmp/memsql-server-rpm860578428 172.31.163.153:/tmp/tmp.9QGY7La4a5`
⠙ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:08 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''mktemp'\'''` took 280.226625ms
2019/04/04 15:44:08 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.HuZFnd9YUw'\'''`
⠸ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:08 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.VN6bEvj0hN'\'''` took 365.757177ms
2019/04/04 15:44:08 main.go:39 Running command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/2.socket /tmp/memsql-server-rpm860578428 172.31.163.152:/tmp/tmp.VN6bEvj0hN`
⠴ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:08 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''chown'\'' '\''srappaport'\'' '\''/tmp/tmp.HuZFnd9YUw'\'''` took 373.446561ms
2019/04/04 15:44:08 main.go:39 Running command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/1.socket /tmp/memsql-server-rpm860578428 172.31.163.151:/tmp/tmp.HuZFnd9YUw`
⠹ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:09 main.go:39 Command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/3.socket /tmp/memsql-server-rpm860578428 172.31.163.153:/tmp/tmp.9QGY7La4a5` took 1.197229358s
2019/04/04 15:44:09 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.9QGY7La4a5'\'''`
⠴ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:09 main.go:39 Command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/2.socket /tmp/memsql-server-rpm860578428 172.31.163.152:/tmp/tmp.VN6bEvj0hN` took 1.230843271s
2019/04/04 15:44:09 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.VN6bEvj0hN'\'''`
⠦ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:09 main.go:39 Command `/usr/bin/scp -oBatchMode=yes -q -oControlPath=/tmp/memsql-toolbox145966280/1.socket /tmp/memsql-server-rpm860578428 172.31.163.151:/tmp/tmp.HuZFnd9YUw` took 1.117185659s
2019/04/04 15:44:09 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.HuZFnd9YUw'\'''`
⠋ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:12 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.9QGY7La4a5'\'''` took 3.799259313s
2019/04/04 15:44:12 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''`
⠙ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.HuZFnd9YUw'\'''` took 3.547337707s
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''`
⠹ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--install'\'' '\''--replacepkgs'\'' '\''/tmp/tmp.VN6bEvj0hN'\'''` took 3.765927287s
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''`
⠸ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''` took 251.35328ms
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''`
⠼ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''` took 404.776934ms
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''`
⠦ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''` failed with exit code 127 in 193.630422ms
✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
Debug 2019/04/04 15:44:13 install.go:85 error running command: `"/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/tmp/memsql-toolbox145966280/3.socket" "172.31.163.153" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\\'' && '\\''exec'\\'' '\\''usermod'\\'' '\\''-a'\\'' '\\''-G'\\'' '\\''memsql'\\'' '\\''srappaport'\\'''"`: exit status 127
✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.153 (1/3)
2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''getent'\'' '\''group'\'' '\''memsql'\'''` took 386.732779ms
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''`
⠇ Installing memsql-server6.7.15-d4450c6798 on 3 hosts 2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''` failed with exit code 127 in 445.321275ms
✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
Debug 2019/04/04 15:44:13 install.go:85 error running command: `"/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/tmp/memsql-toolbox145966280/1.socket" "172.31.163.151" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\\'' && '\\''exec'\\'' '\\''usermod'\\'' '\\''-a'\\'' '\\''-G'\\'' '\\''memsql'\\'' '\\''srappaport'\\'''"`: exit status 127
✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.151 (2/3)
2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''usermod'\'' '\''-a'\'' '\''-G'\'' '\''memsql'\'' '\''srappaport'\'''` failed with exit code 127 in 231.155172ms
✘ Failed to add srappaport to the memsql group. See https://docs.memsql.com/memsql-deploy-redir/memsql-group for more details.
Debug 2019/04/04 15:44:13 install.go:85 error running command: `"/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/tmp/memsql-toolbox145966280/2.socket" "172.31.163.152" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\\'' && '\\''exec'\\'' '\\''usermod'\\'' '\\''-a'\\'' '\\''-G'\\'' '\\''memsql'\\'' '\\''srappaport'\\'''"`: exit status 127
✓ Installed memsql-server6.7.15-d4450c6798 on host 172.31.163.152 (3/3)
✓ Successfully installed on 3 hosts
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''`
2019/04/04 15:44:13 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''` took 181.206765ms
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''`
2019/04/04 15:44:13 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''`
2019/04/04 15:44:14 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''` took 182.016495ms
2019/04/04 15:44:14 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''`
2019/04/04 15:44:14 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''`
2019/04/04 15:44:14 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''` took 173.748492ms
2019/04/04 15:44:14 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''` took 500.14041ms
2019/04/04 15:44:14 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''id'\'' '\''-u'\'' '\''-n'\'''` took 417.065499ms
2019/04/04 15:44:14 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''`
2019/04/04 15:44:14 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''memsqlctl'\'' '\''--json'\'' '\''--yes'\'' '\''list-nodes'\'''` took 304.562612ms
Starting rollback
2019/04/04 15:44:14 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
2019/04/04 15:44:15 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 938.009265ms
2019/04/04 15:44:15 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''`
2019/04/04 15:44:16 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''` took 231.845404ms
2019/04/04 15:44:16 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''`
2019/04/04 15:44:16 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''` took 313.773102ms
2019/04/04 15:44:16 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''`
2019/04/04 15:44:16 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''` took 385.582967ms
2019/04/04 15:44:16 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠏ Uninstalling MemSQL 6.7.15 on 172.31.163.151 2019/04/04 15:44:17 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 916.304185ms
2019/04/04 15:44:17 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''`
⠹ Uninstalling MemSQL 6.7.15 on 172.31.163.151 2019/04/04 15:44:18 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/1.socket 172.31.163.151 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''` took 343.092392ms
✓ Uninstalling MemSQL 6.7.15 on 172.31.163.151
2019/04/04 15:44:18 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
2019/04/04 15:44:18 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 944.081967ms
2019/04/04 15:44:18 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''`
2019/04/04 15:44:19 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''` took 486.895846ms
2019/04/04 15:44:19 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''`
2019/04/04 15:44:19 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''` took 221.735274ms
2019/04/04 15:44:19 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''`

#4

Here’s the remainder of the output. Thanks!

2019/04/04 15:44:19 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''` took 307.327732ms
2019/04/04 15:44:19 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠇ Uninstalling MemSQL 6.7.15 on 172.31.163.152 2019/04/04 15:44:20 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 884.430473ms
2019/04/04 15:44:20 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''`
⠸ Uninstalling MemSQL 6.7.15 on 172.31.163.152 2019/04/04 15:44:21 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/2.socket 172.31.163.152 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''` took 484.97368ms
✓ Uninstalling MemSQL 6.7.15 on 172.31.163.152
2019/04/04 15:44:21 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
2019/04/04 15:44:22 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 1.330986413s
2019/04/04 15:44:22 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''`
2019/04/04 15:44:22 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''which'\'' '\''memsqlctl'\'''` took 202.563724ms
2019/04/04 15:44:22 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''`
2019/04/04 15:44:23 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''readlink'\'' '\''--canonicalize'\'' '\''--no-newline'\'' '\''/usr/bin/memsqlctl'\'''` took 167.490713ms
2019/04/04 15:44:23 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''`
2019/04/04 15:44:23 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--queryformat'\'' '\''%{NAME}'\'' '\''--file'\'' '\''/opt/memsql-server-6.7.15-d4450c6798/memsqlctl'\'''` took 293.908429ms
2019/04/04 15:44:23 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''`
⠏ Uninstalling MemSQL 6.7.15 on 172.31.163.153 2019/04/04 15:44:24 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--query'\'' '\''--all'\'' '\''--queryformat'\'' '\''%{NAME} %{VERSION}\n'\'''` took 927.005128ms
2019/04/04 15:44:24 main.go:39 Running command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''`
⠸ Uninstalling MemSQL 6.7.15 on 172.31.163.153 2019/04/04 15:44:24 main.go:39 Command `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/tmp/memsql-toolbox145966280/3.socket 172.31.163.153 'sudo' '-S' '-k' '--' '/bin/sh' '-c' ''\''printf'\'' '\''e4ce152d-080f-41eb-ba85-00e9f57a8b2d'\'' && '\''exec'\'' '\''rpm'\'' '\''--erase'\'' '\''memsql-server6.7.15-d4450c6798'\'''` took 462.388786ms
✓ Uninstalling MemSQL 6.7.15 on 172.31.163.153
✓ Unregistered hosts
✓ Rollback succeeded

#5

Thanks for reporting this to us. Here are the notes I have so far:

  • srappaport has sudo privileges on all hosts, and is part of the memsql group that was created by a previous run of memsql-deploy setup-cluster.
  • Even though you see the message “Failed to add srappaport to the memsql group”, setup-cluster still makes progress, which is what we expect.
  • setup-cluster starts rolling back after running memsqlctl list-nodes on all three hosts.

To help us debug this issue further, could you provide the output of the following commands?

  • memsql-deploy version
  • memsql-deploy env

Also, it seems like your output is truncated. After finishing the rollback, we would normally expect setup-cluster to print out the error that caused the rollback. Since the character limit on this forum prevents you from including the whole output in one message, try running this command to put all the output in a file, copy the contents to a pastebin, and then send us the link.

memsql-deploy setup-cluster --license <license-key> --master-host 172.31.163.151 --leaf-hosts 172.31.163.152,172.31.163.153 --password <secure-password> -vvv 2>&1 | tee setup-cluster.log

After that, I’d like to unblock you by including the instructions for setting up the same cluster using the other tools. I’ve adapted them from the comprehensive install guide for your specific case:

  1. memsql-toolbox-config register-host --host 172.31.163.151
  2. memsql-toolbox-config register-host --host 172.31.163.152
  3. memsql-toolbox-config register-host --host 172.31.163.153
  4. memsql-deploy install --all
  5. memsql-admin create-node --host 172.31.163.151 --password <secure-password>
  6. memsql-admin bootstrap-aggregator --memsql-id <memsql-id from previous output> --license <license-key>
  7. memsql-admin create-node --host 172.31.163.152 --password <secure-password>
  8. memsql-admin add-leaf --memsql-id <memsql-id from previous output> --password <password from the previous command>
  9. memsql-admin create-node --host 172.31.163.153 --password <secure-password>
  10. memsql-admin add-leaf --memsql-id <memsql-id from previous output> --password <password from the previous command>

Let us know if these commands give you any trouble.


#6

Hi @jkaplan,

Here is the information you requested:

Version: 1.1.0
Commit: 71bb8a3568be5544fb811e7e4f56613dc8dd5911
+------------+----------------------------------------------------------------+
| configFile | /home/srappaport/.config/memsql-toolbox/toolbox.hcl            |
| stateFile  | /home/srappaport/.local/share/memsql-toolbox/toolbox-state.hcl |
| cacheFile  | /home/srappaport/.local/share/memsql-toolbox/toolbox-cache.hcl |
| sessionDir | /tmp/memsql-toolbox790806345                                   |
| sshBinary  | /usr/bin/ssh                                                   |
+------------+----------------------------------------------------------------+ 

Here is the output of the setup-cluster command when I copied it to a file. I do not see a message after rollback indicating what caused it.
https://pastebin.com/jxqTzf0h

Thanks for the comprehensive install instructions! I will give those a try.


#7

Hi @jkaplan,

Step 5 in the comprehensive instructions generated an error. Here is the output:

[srappaport@memsql01-px ~]$ memsql-admin create-node --host 172.31.163.151 --password <secure-password>
srappaport@172.31.163.151's password:
The target user srappaport on host 172.31.163.151 does not have the privileges to perform this action.
Please enter your password to proceed with sudo.  (For details, see https://docs.memsql.com/toolbox-redir/sudo-prompt).
sudo password for srappaport@172.31.163.151:
Toolbox is about to perform the following actions on host 172.31.163.151:
  · Run 'memsqlctl create-node --password ●●●●●●'

Would you like to continue? [y/N]: y
error running memsqlctl: error running command: `"/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/tmp/memsql-toolbox991387071/1.socket" "172.31.163.151" "'sudo'" "'-S'" "'-k'" "'--'" "'/bin/sh'" "'-c'" "''\\''printf'\\'' '\\''7637e485-6a8b-4689-8c8b-8ee74481d8de'\\'' && '\\''exec'\\'' '\\''memsqlctl'\\'' '\\''--json'\\'' '\\''--yes'\\'' '\\''create-node'\\'' '\\''--secure-password'\\'' '\\''yEMyPfx0CNV+42PsPT94Zoi7JkWDV73AwIQrjEnghtNxYt0N1mr3'\\'''"`: exit status 1
stderr: colliding port: 3306

I tried rebooting the server but I still got the same error. I’ve also tried it both with and without memsql running on the server.

Thanks again for your assistance.


#8

Hi @srappaport

We really appreciate the questions you are asking – however, would you mind surrounding blocks of code with the code block styling button? That would make the command line outputs much easier to read for our engineers.

Alternatively, you can surround code blocks with the following:

```
some code block
```

Thank you :slight_smile:


#9

Will do. Sorry about that, @Jacky.


#10

Thank you for including all the diagnostics! I’ll start with some notes:

  • You’re running the latest version of memsql-deploy.
  • The memsql-deploy setup-cluster command failed with this message:
MemSQL nodes already exist on these hosts. To view them, run 'memsql-admin list-nodes.' To remove them, run 'memsql-admin stop-node --all' then 'memsql-admin delete-node --all'
  • The comprehensive install fails at memsql-admin create-node 172.31.163.151 with
stderr: colliding port: 3306

These errors both indicate the same thing: there is already a MemSQL node associated with 172.31.163.151:3306. This may have been the result of an earlier deployment or a failed rollback in a previous setup-cluster.

To identify the node, run memsql-admin list-nodes and look for a node that corresponds to host 172.31.163.151 and port 3306.

If that node has the role Unknown, it is not part of any existing cluster, so you can continue to the next step (memsql-admin bootstrap-aggregator). Otherwise, that node is already part of a cluster, and we do not recommend running multiple clusters on the same set of hosts. To stop that node and delete it along with all of its data, use memsql-admin delete-node --stop.


#11

Thanks @jkaplan, that did the trick! The node did have the Unknown role. It was also stopped so I started it with:

memsql-admin start-node --all

Then I was able to continue with the remainder of the comprehensive steps you posted to get up and running. :slight_smile:

Thanks for all of the help!