Hey all,
I’m currently running MemSQL in a 2 server setup on the same private network, with 2 hosts total & 4 nodes total, 1 host per server & 2 nodes per server. Both servers are running CentOS 7.x:
- Host 1 (Local Host 192.168.1.3): Master & Leaf
- Host 2 (192.168.1.5): Aggregator & Leaf
I’m able to successfully configure each node and everything works fine, until I restart either the aggregator or second leaf node on Server 2 from Server 1. If I were to restart them, those nodes stay in a permanent offline state according to MemSQL Studio.
Because these nodes exist on another server on the network, I’m unsure if there’s a missed handshake due to an improper SSH configuration, or firewall config preventing those nodes on Server 2 from fully appearing online on Server 1. My SSH keys are saved in the following path on both servers as the ‘root’ user, and were configured under the ‘–identity-file’ flag during the installation process: (/root/.ssh/id_rsa).
I’d temporarily disabled the firewall across both servers, and there’s still no luck with these offline nodes. I am however able to ping each server & verify TCP ports 22, 443, 3306, 3307, 3308, 3309, 3310, and 8080 are open & accessible via ‘nc -zv’. IP routes between servers exist. The only other thing that stands out is I’m defining a custom ‘–base-install-dir’ for each node during the ‘memsql-admin create-node’ process, while giving the user ‘memsql’ full access to those directories.
In summary, I know I at least have a working configuration when creating the nodes and assigning aggregators / leaves to them, as I can see a fully online config. The problem comes if I need to restart those nodes for any particular reason; they refuse to fully come back online
Below are the steps taken during the installation process, from the SSH key generation through firewall config, and finally the MemSQL installation steps. Please let me know if there were any additional steps missed in the process for a successful 2 server cluster configuration on the same local network, thanks.
SSH Key Generation for Server 1, as ‘root’:
ssh-keygen
ssh-copy-id root@192.168.1.5
SSH Key Generation for Server 2, as ‘root’:
ssh-keygen
ssh-copy-id root@192.168.1.3
Firewall config changes on both servers:
firewall-cmd --add-port=22/tcp --permanent
firewall-cmd --add-port=443/tcp --permanent
firewall-cmd --add-port=3306/tcp --permanent
firewall-cmd --add-port=3307/tcp --permanent
firewall-cmd --add-port=3308/tcp --permanent
firewall-cmd --add-port=3309/tcp --permanent
firewall-cmd --add-port=3310/tcp --permanent
firewall-cmd --add-port=8080/tcp --permanent
firewall-cmd --reload
firewall-cmd --list-all
Folders created on both servers, giving user ‘memsql’ full access:
mkdir /memsql_base_dir
chown memsql:memsql /memsql_base_dir
Installation Steps:
sudo yum-config-manager --add-repo https://release.memsql.com/production/rpm/x86_64/repodata/memsql.repo
sudo yum repolist
rpm -q which
sudo yum install -y memsql-toolbox memsql-client memsql-studio
memsql-toolbox-config register-host --localhost --host 192.168.1.3
memsql-toolbox-config register-host --identity-file /root/.ssh/id_rsa --host 192.168.1.5
memsql-deploy install --all
memsql-admin create-node --host 192.168.1.3 --password "" -P 3307 --base-install-dir /memsql_base_dir/master
memsql-admin create-node --host 192.168.1.3 --password "" -P 3308 --base-install-dir /memsql_base_dir/leaf
memsql-admin create-node --host 192.168.1.5 --password "" -P 3309 --base-install-dir /memsql_base_dir/aggregator
memsql-admin create-node --host 192.168.1.5 --password "" -P 3310 --base-install-dir /memsql_base_dir/leaf
memsql-admin bootstrap-aggregator --memsql-id <MemSQL_ID> --license <license>
memsql-admin add-aggregator --memsql-id <MemSQL_ID>
memsql-admin add-leaf --memsql-id <MemSQL_ID>
memsql-admin add-leaf --memsql-id <MemSQL_ID>
memsql-admin optimize
memsql-admin restart-node --all
sudo systemctl enable memsql-studio
sudo systemctl start memsql-studio
