Hi Team,
We have 9 node cluster of memSQL with one master aggregator, 2 child aggregator and 6 leaf nodes.
While executing query we got below error -
Error Code: 2423. Leaf Error (<ip_address>:3306): Compilation submission failed. If the error persists please restart MemSQL. This may have been due to an overusage of compilation memory or accidental process kill. If the issue continues after restart, please set the max_compilation_memory_mb flag. 0.421 sec
MemSQL Studio gives message - From master run following command -
$ memsql-ops agent-start --all
We executed command and cluster is up again, we checked all logs and MemSQL Studio for data sharding and sharing resources, we don’t find anything. Want to understand why this has happens
Thanks and Regards
Prashant
Hello ppotdar,
Before I answer your question, I noticed the cluster size that you referenced is larger than the free tier allocation. If you are a paying MemSQL customer, you are eligible for Enterprise Support through the MemSQL support team, which is where you can file these types of issues and errors that you encounter in the product.
See our Support FAQ for more details.
To your specific question:
It may be difficult to pinpoint exactly what happened without monitoring data, but the error likely indicates that there is a query that was compiling that overused the allocated compilation memory.
In MemSQL 6.5, we added the max_compilation_time_s and max_compilation_memory_mb engine variables. The former is the timeout for compiling a query. If the timeout is reached, an error is returned. The latter is the maximum amount of memory used to compile a query. If this maximum is reached, an error is returned, which is similar to the error you saw.
What was the query you were running that returned the error? We recommend running EXPLAIN on that query, and a PROFILE, if possible.
Additionally, you could also check the memsql.log to determine what other issues were occurring during the time of the error. The memsql.log lives in the tracelogs directory of your nodes, and among other errorr and warnings, it will print queries that are being compiled on the first time of your cluster. We recommend you look in the logs during the time of the issue to determine if it provides any clues. The logs will be in the following directory, assuming a standard installation: /var/lib/memsql/<node id>/tracelogs/memsql.log
Alternatively, we have some monitoring tables in information_schema that may provide some history of events in the cluster.
Monitoring MemSQL
Best,
Roxanna
Thanks Roxanna,
The cluster is enterprise licensed, I don’t know how to take support from “Enterprise Support” team, though I have created another forum account with client’s e-mail ID.
Since I don’t have privileges for Enterprise support hence asking help from my community edition account.
I will update you soon by checking memsql.log and by changing parameters max_compilation_time_s and max_compilation_memory_mb.
Thanks and Regards
Prashant
Hi Prashant,
Thanks for confirming!
The Portal/Forum credentials are currently independent of our support portal (we will merge these in the future).
In order to get access to MemSQL enterprise support, you need to navigate to support.memsql.com and create an account with your company’s email handle that purchased MemSQL software.
I asked you this in the other thread (about Spark) as well, but would you mind sharing the company you are with that bought an enterprise MemSQL license? I can reach out to your account team and make sure you have everything you need to access enterprise support.
Best,
Roxanna
I can not share name on public forum, do you have anything where I can communicate personally.
Thanks and Regards
Pashant
Hi Prashant,
Of course, that makes sense! 
I was actually able to identify your affiliation through our internal system for our public forum users. I have reached out to your account team, and they will be in touch with you and other folks on your team to make sure everyone knows how to access support.
Best,
Roxanna