Resource Pools and Pipelines: The operating system failed to allocate memory issue

[rjsv]

I am getting below error while running the a pipeline.

SQL Error [1712] [HY000]: Leaf Error (10.10.0.169:3306): The operating system failed to allocate memory (MemSQL memory use 7183.88 Mb). The request was not processed. See Memory Errors · SingleStore Documentation for additional information.

My cluster host details as below.

My cluster node details are as below.

mv_node details are as beow.

Below you can find the show resource pools; output details.

I am running the pipeline using the user who assigned to inventory _resource_pool.

But why I am getting below error. How to avoid this. I am getting this issue intermidently.

The operating system failed to allocate memory (MemSQL memory use 7183.88 Mb).

[Lejo]

Hi Rjsv-

Can you run a cluster report and see if there is all our basic OS settings have been set correctly?

SingleStoreDB Cloud · SingleStore Documentation

$ memsql-report collect --all
✓ Collected report for host 127.0.0.1
Report written to report-XXXX-XX-XXXXXXXXX.tar.gz
$ memsql-report check --all --report-path ./report-XXXX-XX-XXXXXXXXX.tar.gz

[rjsv]

I have shared it on the e-mail separately.

[Saksham]

Can you run this command
show create pipeline <pipeline name>

Do you see RESOURCE POOL in the output of this command?

If not, you probably forgot to associate the pipelines with a specific resource pool.

Looking at CREATE PIPELINE syntax

CREATE [OR REPLACE] [AGGREGATOR] PIPELINE [IF NOT EXISTS] pipeline_name AS
  LOAD DATA { kafka_configuration | s3_configuration | filesystem_configuration
            | azure_blob_configuration | hdfs_configuration | gcs_configuration }
    [BATCH_INTERVAL milliseconds]
    [MAX_PARTITIONS_PER_BATCH max_partitions_per_batch]
    [RESOURCE POOL pool_name]
    [(ENABLE|DISABLE) OUT_OF_ORDER OPTIMIZATION]
    [WITH TRANSFORM ('uri', 'executable', 'arguments [...]') ]
  [REPLACE | IGNORE | SKIP { ALL | CONSTRAINT | DUPLICATE KEY } ERRORS]
  { INTO TABLE table_name | INTO PROCEDURE proc_name }
  { json_format_options | avro_format_options | parquet_format_options | csv_format_options }
  [ (column_name, ... ) ]
  [SET  col_name = expr,...]
  [WHERE expr,...]
  [ON DUPLICATE KEY UPDATE column_name = expression, [...]]


[RESOURCE POOL pool_name]: Specifies the resource pool that is used to load pipeline data.

If the resource pool is not specified at the time of pipeline creation, then the pipeline uses the user’s default resource pool. If no default resource pool has been set for the user, then the pipeline uses the value of the resource_pool engine variable as its resource pool.
The background tasks of a pipeline runs in its resource pool. Therefore, a resource pool used by a pipeline cannot be dropped.
The user who creates the pipeline must have permissions to use the resource pool. The pipeline will continue to use the pool even if the right to use the pool for the pipeline is revoked for the user who created the pool.
For more information on resource pools, see Setting Resource Limits.

Note the [RESOURCE POOL pool_name]. Using this syntax you can associate a specific resource pool with a pipeline during it’s creation. Then, when the pipeline is launched anytime in future, it is this resource pool within which your pipeline will run.