Need help with 'Parition has no master instance'

(Ported from memsql-public-chat Slack channel)

Matteo [4:18 AM]
hi all, I’m totally new of memsql and I’m testing it for a pilot project in our company

unfortunately I didn’t notice I was filling the disk with data and once noticed it was too late… now, despite the fact there’s room enough to grow, anytime I try to perform even just a select on my database, I receive the following erro:

ERROR 1777 (HY000): Partition aic_monitor:0 has no master instance.

rebooting even the whole server didn’t resolve… and both master and leaf (which are on the same machine) are alive

is there any chance I can recover my data or I’d better restart from scratch? of course that thing worries me if I have to move to production…

seth [10:01 AM]
@Matteo it sounds like you are missing master partitions for the database… which isn’t good. master partitions are the active partitions for your database.

select @@redundancy_level;

select @@default_partitions_per_leaf;

show aggregators;

show leaves;

show cluster status;

use database_name;

show partitions extended;

^^ run these commands on your master aggregator and share the output.

Matteo [11:35 AM]
thank you @seth! going to run them all as soon as I reach the dev server!

Matteo [2:10 AM]
I’ve just anonymized the IP address, that’s the status of the db

mysql> select @@redundancy_level;
+--------------------+
| @@redundancy_level |
+--------------------+
|         1 |
+--------------------+
1 row in set (0.12 sec)
mysql> select @@default_partitions_per_leaf;
+-------------------------------+
| @@default_partitions_per_leaf |
+-------------------------------+
|               8 |
+-------------------------------+
1 row in set (0.13 sec)
mysql> show aggregators;
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
| Host      | Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | Master_Aggregator | NodeId |
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
|  <VALID IP>  | 3306 | online |         1 |             NULL |         1 |   1 |
+----------------+------+--------+--------------------+------------------------------+-------------------+--------+
1 row in set (0.00 sec)
mysql> show leaves;
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
| Host      | Port | Availability_Group | Pair_Host | Pair_Port | State | Opened_Connections | Average_Roundtrip_Latency_ms | NodeId |
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
|  <VALID IP>  | 3307 |         1 | NULL   |   NULL | online |         10 |            0.349 |   2 |
+----------------+------+--------------------+-----------+-----------+--------+--------------------+------------------------------+--------+
1 row in set (0.00 sec)
mysql> show cluster status;
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
| Node ID | Host      | Port | Database   | Role    | State    | Position   | Master Host  | Master Port | Metadata Master Node ID | Metadata Master Host | Metadata Master Port | Metadata Role  | Details                     |
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
|    1 |  <VALID IP>  | 3306 | aic_monitor  | master   | online   | 30:10     | NULL      |    NULL |          NULL | NULL         |         NULL | Reference    |                         |
|    1 |  <VALID IP>  | 3306 | aic_stats   | master   | online   | 0:11650    | NULL      |    NULL |          NULL | NULL         |         NULL | Reference    |                         |
|    1 |  <VALID IP>  | 3306 | cluster    | master   | online   | 5:18     | NULL      |    NULL |          NULL | NULL         |         NULL | Reference    |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor  | sync slave | replicating | 30:10     |  <VALID IP>  |    3306 |            1 |   <VALID IP>   |         3306 | Reference    |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_0 | detached  | pending   | 439:265207550 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_1 | detached  | pending   | 439:262971139 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_2 | detached  | pending   | 439:261157609 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_3 | detached  | pending   | 439:267582197 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_4 | detached  | pending   | 439:260992115 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_5 | detached  | pending   | 439:259062765 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_6 | detached  | pending   | 440:10    | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_monitor_7 | detached  | pending   | 439:267115988 | NULL      |    NULL |          NULL | NULL         |         NULL | Detached Master |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats   | sync slave | replicating | 0:11650    |  <VALID IP>  |    3306 |            1 |   <VALID IP>   |         3306 | Reference    |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_0  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_1  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_2  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_3  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_4  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_5  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_6  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | aic_stats_7  | master   | online   | 0:11393    | NULL      |    NULL |          NULL | NULL         |         NULL | Master     |                         |
|    2 |  <VALID IP>  | 3307 | cluster    | async slave | replicating | 5:17     |  <VALID IP>  |    3306 |            1 |   <VALID IP>   |         3306 | Reference    | stage: packet wait, state: x_streaming, err: no |
+---------+----------------+------+---------------+-------------+-------------+---------------+----------------+-------------+-------------------------+----------------------+----------------------+-----------------+-------------------------------------------------+
22 rows in set (0.01 sec)
mysql> show partitions extended;
+---------+------+------+------+--------+------+--------------+------------+--------------+
| Ordinal | Host | Port | Role | Locked | Info | Last Command | Last Error | Last Message |
+---------+------+------+------+--------+------+--------------+------------+--------------+
|    0 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    1 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    2 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    3 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    4 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    5 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    6 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
|    7 | NULL | NULL | NULL |   0 | NULL | NULL     |    NULL | NULL     |
+---------+------+------+------+--------+------+--------------+------------+--------------+
8 rows in set (0.00 sec)

Matteo [3:33 AM]
@seth thank you for your hints: I’ve checked the “detached” leaves, I’ve re-detached them and attached again and it did the trick! I can now see data again!