I have an Unknown node that I’m trying to delete so I can create a new node with a comprehensive install. I can stop and start the Unknown node:
[centos@ip-10-145-80-247 /]$ memsql-admin stop-node --all
Toolbox is about to perform the following actions:
· Stop all nodes in the cluster
Would you like to continue? [y/N]: y
✓ Stopped Unknown nodes on 10.145.80.247 (1/1)
✓ Successfully stopped Unknown nodes on 1 host
✓ Stopped Unknown node
Operation completed successfully
However, I cannot delete it or list the nodes:
[centos@ip-10-145-80-247 /]$ memsql-admin delete-node --all
failed to list nodes on 1 host: 10.145.80.247
[centos@ip-10-145-80-247 /]$ memsql-admin list-nodes
✘ Failed to list nodes on all hosts: failed to list nodes on 1 host: 10.145.80.247
No nodes found
How do I delete this Unknown node so I can create the node with a fresh slate?
Thanks!
All three of these commands (stop-node --all, delete-node --all, and list-nodes) gather a list of all nodes in the cluster as part of their early operation. A failure in list-nodes usually points to a permissions issue, so I think that’s the first place to investigate.
The stop-node and delete-node commands require the privileges of the memsql user on the target hosts because they operate on processes and write to the filesystem. The list-nodes command requires the privileges of the memsql group because it reads from the node metadata file. (However, due to an implementation detail, delete-node gathers node information using memsql group privileges and only assumes memsql user privileges when deleting the node.)
It looks like the target user (centos, unless customized during memsql-toolbox-config register-node) is part of the memsql group on 10.145.80.247, but is still unable to read the node metadata file. Ensure that the memsql group can read the node metadata file on the target host (10.145.80.247). By default, the node metadata file is located at /var/lib/memsql/nodes.hcl, but you can see its configured location by running memsqlctl env.