I’m trying to setup a 3 node cluster (one master aggregator and two leaf nodes) in AWS running on CentOS 7.6. We use terraform instead of cloudformation, so using the Cloud instructions here wasn’t an option for me:
I chose r4.2xlarge for my instance sizes, which ends up providing too much memory for the free license I’m trying to test MemSQL with in our dev environment.
stderr: Error 2350: Leaf or aggregator node could not be added because you are using the MemSQL free license which has a total memory limit of 128.00 GB and after adding the node your total MemSQL memory capacity (used_cluster_capacity) would be 161.47 GB. To purchase more capacity, contact team@memsql.com. See https://docs.memsql.com/redir/memory-limit-error for instructions on how to set node memory limits to create a cluster with up to 128.00 GB of memory capacity.
I’m trying to do a comprehensive installation so I can limit the amount of memory per the instructions here:
Here are the steps I’ve done so far (from host 10.145.80.161):
I then tried to limit the memory of the master aggregator with this command:
memsql-admin update-config --set-global --key maximum_memory --value 30208 --memsql-id 84619EC4AFF18B21DE8B246E6250CD2119814C4A
This was the error I got:
no node found with given prefix
Here is the verbose output from the command:
Next, I followed the link provided in the error:
The memsql.cnf file exists and has the correct permissions:
[root@ip-10-145-80-161 6be4123d-7f86-436c-9c7e-d1c4ca7c71bc]# ls -l memsql.cnf
-rw-------. 1 memsql memsql 471 Apr 11 20:36 memsql.cnf
I also tried the ‘memsql-admin describe-node’ command and got the same error:
no node found with given prefix
Here is the verbose output of that command:
Any help with this error would be greatly appreciated! Please let me know if you need any more details.
I changed it so I am now using four r4-xlarge nodes (1 master aggregator and three leaves). Each node has 30GB of memory, so total is around 120GB. I’m still getting that same error (“no node found with given prefix”) when I try this command:
Any idea what the error “no node found with given prefix” means and how to get past it? I need to get this cluster up and running ASAP for our analytics team to start testing MemSQL in our dev environment. The next step after that will be to purchase an enterprise license for our prod environment but we won’t be able to move forward with that until I can get them going in dev.
I just tried this again with a fresh set of EC2 instances and I’m getting the same error. What does “no node found with given prefix” mean? Here are the commands I ran:
[centos@ip-10-145-80-184 data]$ memsql-admin create-node --host 10.145.80.184 --password <secure-password> --datadir /data
Toolbox is about to perform the following actions on host 10.145.80.184:
· Run 'memsqlctl create-node --datadir /data --password ●●●●●●'
Would you like to continue? [y/N]: y
+------------------------------------------+---------------+
| MemSQL ID | Host |
+------------------------------------------+---------------+
| 26C1D20F68635EB27FA0623058B802AF9A8D50BC | 10.145.80.184 |
+------------------------------------------+---------------+
[centos@ip-10-145-80-184 data]$ memsql-admin bootstrap-aggregator --memsql-id 26C1D20F68635EB27FA0623058B802AF9A8D50BC --license <license-id>
no node found with given prefix
@srappaport sorry that you are running into this issue. It appears that you are still waiting for help? Have you reached out and received any assistance via another channel?
In the meantime, the error you are receiving is saying that memsql-admin is unable to find any nodes with the provided memsql id. I realize this is very confusing especially since you are using the ID provided back to you from create-node.
The issue seems to be the permissions of the memsql.cnf file. List-nodes (which is used by admin to lookup nodes) doesn’t change to the memsql user to run. It simply assumes that the current user is in the memsql group. It appears that your memsql.cnf file does not permit group access which is causing the issue you describe.
Normal usage of memsqlctl and memsql-admin should result in that file having 644 permissions. Do you know why the file has 600 permissions in your environment?
You have two solutions to this issue:
chmod the memsql.cnf file to 644 (and make sure parent directories also permit read access to the memsql group). This solution also requires that the user you are ssh’ing into each host as is also in the memsql group.
using memsql-toolbox-config, unregister all of your hosts and then register them using the memsql user to ssh in to each of your hosts. By doing this all of our actions on remote hosts will be as the memsql user and thus not run into these permissions issues