All nodes cannot start. This node is not managed by a supported tool

Update from version 6.8.4, all node versions are updated successfully. “This node is not managed by a supported tool.” .toolbox has been updated to the latest version. Using memsql-admin, memctl cannot start the node, the same prompt.

Would you like to continue? [y/N]: y
✘ Failed to start Leaf node on 192.168.10.153 (1/2)
✘ Failed to start Leaf node on 192.168.10.154 (2/2)
error running memsqlctl: error running command: "/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/run/user/0/memsql-toolbox166256997/4.socket" "192.168.10.154" "'memsqlctl'" "'--json'" "'--yes'" "'start-node'" "'--memsql-id'" "'5C0F8482645BD8EAE94188F79BA1E75A83C6F9A1'": exit status 1
stderr: Latest errors from MemSQL tracelog:
02887404 2020-01-02 01:27:07.482 FATAL: Thread 115121: performDbUpgradeSteps: This node is not managed by a supported tool. Please use a toolbox version at least as new as 1.3.0 or an ops version at least as new as 7.0.0.
: Failed to connect to MemSQL: process exited: exit status 1
root@MemSQL-1:~# apt-get install memsql-toolbox
Reading package lists… Done
Building dependency tree
Reading state information… Done
memsql-toolbox is already the newest version (1.4.1).
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

Hello,

How exactly did you upgrade your MemSQL cluster? Can you say which commands you ran exactly?

Thank you.

Before the upgrade, “sudo apt install memsql-toolbox -y” was not executed, but “memsql-deploy upgrade” was executed directly. After executing this error, I executed “sudo apt install memsql-toolbox -y” but couldn’t solve the problem.

The database cannot be started, which has a great impact.

You can fix this by installing MemSQL 6.8.4, starting the MemSQL nodes, and re-running upgrade:

memsql-deploy install --version 6.8.4
memsql-admin start-node --all
memsql-deploy upgrade

To give some context, the upgrade failed (as the toolbox version was too low for upgrades to MemSQL 7), and the nodes remained offline.
However, when you try to start them again, they will start using the new MemSQL version, which was already installed when the upgrade failed. As the nodes start, they will notice they are in fact upgrading, but instead of being upgraded with memsql-deploy upgrade (which is the safe and recommended way), the upgrade is just being performed by memsql-admin start-nodes. The nodes will refuse to start.
So to fix it, we just start the nodes again but using the previous MemSQL version, which is reinstalled, and we re-run the upgrade command.

The problem could not be resolved, and version 7 could not be returned to 6.8.4.

root@MemSQL-1:~# memsql-deploy install --version 6.8.4
±------±---------------±-----------+
| Index | Hostname | Local Host |
±------±---------------±-----------+
| 1 | 192.168.10.151 | No |
| 2 | 192.168.10.152 | No |
| 3 | 192.168.10.153 | No |
| 4 | 192.168.10.154 | No |
| 5 | All Hosts | |
±------±---------------±-----------+
Select host(s): 5
Toolbox will perform the following actions:
· Download memsql-server 6.8.4

· Install memsql-server 6.8.4 on 192.168.10.151
· Install memsql-server 6.8.4 on 192.168.10.152
· Install memsql-server 6.8.4 on 192.168.10.153
· Install memsql-server 6.8.4 on 192.168.10.154

Would you like to continue? [y/N]: y
✓ Downloaded memsql-server production:6.8.4
✓ Found existing package memsql-server6.8.4-696e55c4e4 on host 192.168.10.151 (1/4)
✓ Found existing package memsql-server6.8.4-696e55c4e4 on host 192.168.10.152 (2/4)
✓ Found existing package memsql-server6.8.4-696e55c4e4 on host 192.168.10.153 (3/4)
✓ Found existing package memsql-server6.8.4-696e55c4e4 on host 192.168.10.154 (4/4)
✓ Successfully installed on 4 hosts
Operation completed successfully
root@MemSQL-1:~# memsql-admin list-nodes
±-----------±-----------±---------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
±-----------±-----------±---------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
| 5018155081 | Master | 192.168.10.151 | 3306 | Stopped | False | 7.0.9 | Unknown | | 0.0.0.0 |
| 234936A578 | Aggregator | 192.168.10.152 | 3306 | Stopped | False | 7.0.9 | Unknown | | 0.0.0.0 |
| 50BB75E9FC | Leaf | 192.168.10.153 | 3306 | Stopped | False | 7.0.9 | Unknown | | 0.0.0.0 |
| 5C0F848264 | Leaf | 192.168.10.154 | 3306 | Stopped | False | 7.0.9 | Unknown | | 0.0.0.0 |
±-----------±-----------±---------------±-----±--------------±-------------±--------±---------------±-------------------±-------------+
root@MemSQL-1:~# memsql-admin start-node --all
Toolbox is about to perform the following actions:
· Start all nodes in the cluster

Would you like to continue? [y/N]: y
✘ Failed to start Leaf node on 192.168.10.154 (1/2)
✘ Failed to start Leaf node on 192.168.10.153 (2/2)
error running memsqlctl: error running command: "/usr/bin/ssh" "-oBatchMode=yes" "-q" "-tt" "-oControlPath=/run/user/0/memsql-toolbox971246705/3.socket" "192.168.10.153" "'memsqlctl'" "'--json'" "'--yes'" "'start-node'" "'--memsql-id'" "'50BB75E9FCF025B4FEFE1EA1CC122FF6E8D3479A'": exit status 1
stderr: Latest errors from MemSQL tracelog:
02910109 2020-01-11 11:42:40.580 FATAL: Thread 115121: performDbUpgradeSteps: This node is not managed by a supported tool. Please use a toolbox version at least as new as 1.3.0 or an ops version at least as new as 7.0.0.
: Failed to connect to MemSQL: process exited: exit status 1
root@MemSQL-1:~#

The database cannot be started, which has a great impact.

The problem remains unsolved, help me.

Sorry for the delay, I wasn’t getting notifications for this question.

Try to uninstall 7.0 completely. You can do memsql deploy list-versions to obtain the 7.0 version you currently have and then you can uninstall it with memsql deploy uninstall --version <VERSION>.

Having the same problem after installing v7 on a machine that was previously running a v6.x.x. - 6.x.x was never uninstalled with “memsql-deploy uninstall” rather the RPMs were just removed.

Currently cannot run memsql-deploy upgrade becuase the machine does not have internet access, is there an offline alternative?

Yes, memsql-deploy upgrade supports a --file-path argument to install an rpm package.