Partition log file size

Help
1

Hi,

Every time I create a database, memsql creates very large partitions log files taking a significant amount of disk space. Why are these files so large if there’s no data in them?

Example in a two-leaf cluster with default_partitions_per_leaf = 8.
I create an empty database "CREATE DATABASE ‘sam_test’;

The following is generated in each of the leaves:

-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_11_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_11_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_13_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_13_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_15_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_15_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_1_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_1_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_3_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_3_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_5_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_5_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_7_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_7_log_v1_65536
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:52 sam_test_9_log_v1_0
-rw------- 1 celer-uat celer-uat 268435456 Jul 16 08:51 sam_test_9_log_v1_65536
-rw------- 1 celer-uat celer-uat 67108864 Jul 16 08:52 sam_test_log_v1_0
-rw------- 1 celer-uat celer-uat 67108864 Jul 16 08:52 sam_test_log_v1_16384

That adds up to about 8.5GB worth of disk space for an empty database which is not scalable.
I’m running 7.1.2.

Thanks,

Sam.

2

Hi celer-tech,

in versions 7.0 and 7.1, create database disk allocation can be tuned using these engine variables.

log_file_size_partitions
log_file_size_ref_dbs

You can use memsql-admin update-config or set them from the database prompt. The default is 256MB, which is consistent with the numbers you see.

3

Thanks. Could you point me to the documentation for those? I could not find any.
Otherwise, how do you decide which value to give them?

4

We have a task filed for public documentation on these variables (and many more), and it should be published soon. In the meantime, there is some discussion on these particular variables on this forum:

and

5

Just to add a bit of context there, those log files are pre-allocated on create database so we don’t need to allocate them while committing transactions. If you write data to the database they will fill up with user data. So this only impacts things for empty or mostly empty databases (disk is very cheap these days, so the default configuration errs on the side of performance by using up 1/2 GB of disk per partition for empty databases).