Ran Out of Disk Space & Unable to Connect to the Master Partition to Drop Tables

Hi,

It appears one of our queries used more than expected data overnight, resulting in our hard drives nearly filling up in space. The database & nodes appear in a critical space according to the MemSQL Studio dashboard, and restarting those nodes via memsql-admin returns no luck.

Is there a way to connect to this node “offline” and drop tables while it’s in a critical state? The current hard disk if fairly large, as we’re unable to simply copy everything to an even larger RAID array. I don’t see any other unused files we could possibly delete, as 100% of this RAID array is dedicated to MemSQL.

Here’s the error I see when attempting to drop a table:

ERROR 1772 ER_DISTRIBUTED_PARTITION_MOVE_FAILED: Failed to find a master partition for db_data

Hi ymonye,

Which version of memsql are you running?

There are two things you can try to free up disk space.

  1. If running MemSQL 7.0 or higher set snapshots_to_keep to 1 on all nodes. This should free up a bunch of disk space almost right away and hopefully allow you to delete some data. This doesn’t work in memsql 6.X.

  2. Manually drop slave partitions via DROP PARTITION db_name:X on 'host':port . You’ll need to do this very carefully (don’t accidentally drop a master partition). You can see which hosts have slave partitions via select * from information_schema.distributed_partitions where ROLE='Slave';. You can restore the slave partitions via RESTORE REDUNDANCY when done freeing up some disk space.

I’m on 7.1.3. I will try the 1st, then second option, thanks

Is there an elegant way of setting the snapshots_to_keep variable to 1? I searched across the documentation online, only saw it referenced via the database call memsql> SHOW VARIABLES;

Thanks

I assume your using memsql tools and not ops? You can use this if so (ops has equivalent functionality if your using ops):

1 Like

Thank you, that did the trick & I was able to restart the nodes to then delete the extra tables.

memsql-admin update-config --all --set-global --key "snapshots_to_keep" --value "1"

memsql-admin restart-node --all