Details
-
Bug
-
Status: Needs Feedback (View Workflow)
-
Critical
-
Resolution: Unresolved
-
10.11.18
-
None
-
None
-
Can result in hang or crash
Description
Our environment is three nodes (node1 + node2+witness) with galera cluster:
node1: MariaDB version 10.11.18, 8core+32Gmem
node2: MariaDB version 10.11.18, 8core+32Gmem
galera: galera-4-26.4.27-1.el8.x86_64
OS: Redhat 8.10 (4.18.0-553.150.1.el8_10.x86_64)
---------
OS messages log:
....
Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: chk_maria_top.s invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: oom_kill_process.cold.33+0xb/0x10
Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-ddd_testdb01.service,task=mariadbd,pid=15824,uid=3001
Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: Out of memory: Killed process 15824 (mariadbd) total-vm:64507224kB, anon-rss:31423688kB, file-rss:0kB, shmem-rss:0kB, UID:3001 pgtables:87164kB oom_score_adj:0
-------
the batch insert:
The MariaDB instance was killed by the Linux OOM killer after a single session (thread_id 654320, user CCCC) accumulated ~29.5 GB of memory in memory/sql/thd::main_mem_root over roughly 40 seconds, driving the server to ~27 GB RES (87% of RAM) and exhausting system memory (30.7 GB used out of 31.8 GB total, with swap climbing).
Time |Event |thd::main_mem_root (cur) |mariadbd RES
9:44:04 |Baseline, normal operation |~0.67 MB |5.3 GB
9:45:06 |mariadbd CPU jumps to 23.5%, |~0.70 MB |5.5 GB
DELETE FROM CCC_DDD_STAGING ... LIMIT 10000 appears
9:45:16 |CPU 170%, insert into CCC_DDD_STAGING begins |~0.70 MB |5.6 GB
9:45:27 |CPU 176%, insert ... FFF_ID between 799219275 and 799229274 |0.75 MB |5.6 GB
9:45:37 |insert ... TABLE_ROWID between 360000 and 369999 |0.81 MB |5.6 GB
9:45:47 |Memory spike starts |5,261 MB |10.8 GB
9:45:58 | |12,810 MB |18.0 GB
9:46:08 | |20,640 MB |24.3 GB
09:46:19–23 | |26,024 → 26,295 MB |27.3 GB
9:46:34 | |29,411 MB |26.9 GB
9:48:06 |OOM kill |~29.5 GB |killed
9:48:18 |mariadbd restarted (PID 842110) |— |2.3 GB
9:48:30 |restart again (PID 842254) |— |1.4 GB
Attachments
1.
|
OOM was triggerred when running 500 single-row INSERTs |
|
Open | Unassigned |