Details
-
Technical task
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11.18
-
None
-
None
Description
Lab environment is 3 nodes (node1 + node2+witness) with galera cluster:
node1: MariaDB version 10.11.18, 8core+32Gmem
node2: MariaDB version 10.11.18, 8core+32Gmem
galera: galera-4-26.4.27-1.el8.x86_64
OS: Redhat 8.10 (4.18.0-553.123.1.el8_10.x86_64)
---------
Run test
(Refer to the attached "Run Test Readme.txt", zip file "load test scripts")
---------
-
-
- Node-1 and Node-2 ###
mkdir -p /u01/oomlab /u01/csv - Upload scripts to /u01/oomlab
chmod 700 /u01/oomlab/*.sh
- Node-1 and Node-2 ###
-
-
-
- Node-1 ###
-
- Create the lab schema
cd /u01/oomlab
mysql -e "drop database if exists oomlab"
mysql < 01_schema.sql
mysql -e "show tables from oomlab"
----------------------
| Tables_in_oomlab |
----------------------
| app_rec_bomb |
| app_rec_bomb_flat |
| app_rec_bomb_jnl |
| app_var_0 |
| app_var_5 |
| seq_app_rec_bomb_jnl |
| staging_web_1 |
| staging_web_2 |
| staging_web_3 |
| staging_web_4 |
| staging_web_5 |
| stat |
| trigger_control |
----------------------
---------
[root@t1vuat-gcisdba-orderdb01-1-01 oomlab]# ./oomctl.sh
╔══════════════════════════════════════════════════════════════════════╗
║ OOM ROOT-CAUSE ISOLATION MENU ║
╠══════════════════════════════════════════════════════════════════════╣
║ Memory cap is EXTERNAL. Set before running: ║
║ systemctl set-property mariadb.service MemoryMax=2048M ║
║ systemctl set-property mariadb.service MemoryHigh=infinity ║
║ Swap must be off: run ./disable_swap_permanently.sh once ║
╠══════════════════════════════════════════════════════════════════════╣
║ Diagnostics ║
║ D1) Pre-flight ║
║ D2) Show matrix.csv ║
║ D3) Show memory now ║
║ ║
║ T0 Baseline sizing (3) 1) T0.1 2) T0.2 3) T0.3 ║
║ T1 Vector isolation (7) 4) T1.1 5) T1.2 6) T1.3 7) T1.4 ║
║ 8) T1.5 9) T1.6 10) T1.7 ║
║ ║
║ Utilities ║
║ 90) Emergency abort ║
║ 91) Collect evidence manually ║
║ 92) Recovery / cleanup (DB drop + CSV wipe — leaves cap/swap) ║
║ 93) Reset database ║
║ 0) Exit ║
╚══════════════════════════════════════════════════════════════════════╝
Select test:
10
----------
Started at 2026-10-22 17:10, mariadb's mem% is going up.
(Refer to the attached "Mariadb_top_mem_list.txt")
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2026-10-22 17:10:00 : 48417 gcis_gc+ 20 0 8198.9g 3.0g 1.3g S 72.2 9.6 161:59.37 mariadbd
2026-10-22 17:10:01 : 48417 gcis_gc+ 20 0 8198.9g 3.4g 1.3g S 81.0 10.8 162:00.78 mariadbd
2026-10-22 17:10:03 : 48417 gcis_gc+ 20 0 8199.2g 3.9g 1.3g S 100.0 12.6 162:02.12 mariadbd
2026-10-22 17:10:04 : 48417 gcis_gc+ 20 0 8199.8g 4.5g 1.3g S 115.8 14.4 162:03.57 mariadbd
2026-10-22 17:10:05 : 48417 gcis_gc+ 20 0 8200.4g 5.1g 1.3g S 122.2 16.4 162:04.99 mariadbd
2026-10-22 17:10:06 : 48417 gcis_gc+ 20 0 8201.0g 5.7g 1.3g S 110.5 18.4 162:06.40 mariadbd
......
2026-10-22 17:11:21 : 48417 gcis_gc+ 20 0 8231.2g 27.9g 0 S 105.3 89.7 163:17.60 mariadbd
2026-10-22 17:11:23 : 48417 gcis_gc+ 20 0 0 0 0 Z 94.7 0.0 163:18.90 mariadbd
2026-10-22 17:11:24 : (mariadb process not found)
2026-10-22 17:11:25 : (mariadb process not found)
2026-10-22 17:11:27 : (mariadb process not found)
2026-10-22 17:11:28 : (mariadb process not found)
2026-10-22 17:11:29 : (mariadb process not found)
2026-10-22 17:11:30 : (mariadb process not found)
2026-10-22 17:11:31 : (mariadb process not found)
......
Test result is: 2 nodes mariadb crash which was triggered from OOM issued.
----------
OS messages log:
....
Node1,
Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: vmtoolsd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: oom_kill_process.cold.33+0xb/0x10
Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-gcisdba_orderdb01.service,task=mariadbd,pid=48417,uid=3002
Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: Out of memory: Killed process 48417 (mariadbd) total-vm:8631172872kB, anon-rss:29365024kB, file-rss:0kB, shmem-rss:0kB, UID:3002 pgtables:74516kB oom_score_adj:0
Node2,
Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: mariadbd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: oom_kill_process.cold.33+0xb/0x10
Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-gcisdba_orderdb01.service,task=mariadbd,pid=48762,uid=3002
Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: Out of memory: Killed process 48762 (mariadbd) total-vm:8631499128kB, anon-rss:29670848kB, file-rss:0kB, shmem-rss:0kB, UID:3002 pgtables:75156kB oom_score_adj:0
-------
the batch insert:
500 single-row INSERTs into APP_REC_BOMB, trigger ON. Per-row trigger overhead: each INSERT also writes one journal row.
(Refer to the attached bin log file "500 single-row INSERTs.sql")
INSERT sample as below,
#Q> INSERT INTO oomlab.APP_REC_BOMB
#Q> (SCHEME_TYPE_CD,REC_ID,APPL_ID_TYPE,APPL_ID_NO,APP_NO,
#Q> SECOND_FACTOR_VALUE,ACADEMIC_YR,STUDENT_ID,STUDENT_NAME,
#Q> CHI_STUDENT_NAME,STATUS,RESULT_NOTIF_DT,ENG_MORE_INF,CHI_MORE_INF,
#Q> CREATE_BY,CREATE_DT,LAST_UPD_BY,LAST_UPD_DT,SYSTEM_SCHEME_CD,SSO_ID)
#Q> VALUES
#Q> ('ENLS', 1, 'A', 'ID1', 'APP1', 'SF1', '2026', 'STU1',
#Q> 'NAME 1', 'NAME_CHI 1', 'A', NOW(), 'ENG 1', 'CHI 1',
#Q> 'WEB01_IMPORT', NOW(6), 'WEB01_IMPORT', NOW(6), 'ENLS', 'SSO-1')
#261022 17:09:59 server id 1 end_log_pos 421219034 CRC32 0x7b774353 Table_map: `oomlab`.`seq_app_rec_bomb_jnl` mapped to number 371
- at 421219034
#261022 17:09:59 server id 1 end_log_pos 421219125 CRC32 0xda34894b Write_rows: table id 371 flags: STMT_END_F-
- INSERT INTO `oomlab`.`seq_app_rec_bomb_jnl`
- SET
- @1=70003 /* LONGINT meta=0 nullable=0 is_null=0 */
- @2=1 /* LONGINT meta=0 nullable=0 is_null=0 */
- @3=9999999999 /* LONGINT meta=0 nullable=0 is_null=0 */
- @4=1 /* LONGINT meta=0 nullable=0 is_null=0 */
- @5=1 /* LONGINT meta=0 nullable=0 is_null=0 */
- @6=0 /* LONGINT meta=0 nullable=0 is_null=0 */
- @7=0 /* TINYINT meta=0 nullable=0 is_null=0 */
- @8=0 /* LONGINT meta=0 nullable=0 is_null=0 */
-