XMLWordPrintable

Details

    • Technical task
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.18
    • None
    • None

    Description

      Lab environment is 3 nodes (node1 + node2+witness) with galera cluster:
      node1: MariaDB version 10.11.18, 8core+32Gmem
      node2: MariaDB version 10.11.18, 8core+32Gmem
      galera: galera-4-26.4.27-1.el8.x86_64
      OS: Redhat 8.10 (4.18.0-553.123.1.el8_10.x86_64)
      ---------

      Run test
      (Refer to the attached "Run Test Readme.txt", zip file "load test scripts")
      ---------

          1. Node-1 and Node-2 ###
            mkdir -p /u01/oomlab /u01/csv
          2. Upload scripts to /u01/oomlab
            chmod 700 /u01/oomlab/*.sh
          1. Node-1 ###
      1. Create the lab schema
        cd /u01/oomlab
        mysql -e "drop database if exists oomlab"
        mysql < 01_schema.sql
        mysql -e "show tables from oomlab"

      ----------------------

      Tables_in_oomlab

      ----------------------

      app_rec_bomb
      app_rec_bomb_flat
      app_rec_bomb_jnl
      app_var_0
      app_var_5
      seq_app_rec_bomb_jnl
      staging_web_1
      staging_web_2
      staging_web_3
      staging_web_4
      staging_web_5
      stat
      trigger_control

      ----------------------
      ---------

      [root@t1vuat-gcisdba-orderdb01-1-01 oomlab]# ./oomctl.sh
      ╔══════════════════════════════════════════════════════════════════════╗
      ║ OOM ROOT-CAUSE ISOLATION MENU ║
      ╠══════════════════════════════════════════════════════════════════════╣
      ║ Memory cap is EXTERNAL. Set before running: ║
      ║ systemctl set-property mariadb.service MemoryMax=2048M ║
      ║ systemctl set-property mariadb.service MemoryHigh=infinity ║
      ║ Swap must be off: run ./disable_swap_permanently.sh once ║
      ╠══════════════════════════════════════════════════════════════════════╣
      ║ Diagnostics ║
      ║ D1) Pre-flight ║
      ║ D2) Show matrix.csv ║
      ║ D3) Show memory now ║
      ║ ║
      ║ T0 Baseline sizing (3) 1) T0.1 2) T0.2 3) T0.3 ║
      ║ T1 Vector isolation (7) 4) T1.1 5) T1.2 6) T1.3 7) T1.4 ║
      ║ 8) T1.5 9) T1.6 10) T1.7 ║
      ║ ║
      ║ Utilities ║
      ║ 90) Emergency abort ║
      ║ 91) Collect evidence manually ║
      ║ 92) Recovery / cleanup (DB drop + CSV wipe — leaves cap/swap) ║
      ║ 93) Reset database ║
      ║ 0) Exit ║
      ╚══════════════════════════════════════════════════════════════════════╝
      Select test:
      10

      ----------
      Started at 2026-10-22 17:10, mariadb's mem% is going up.
      (Refer to the attached "Mariadb_top_mem_list.txt")

      PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
      2026-10-22 17:10:00 : 48417 gcis_gc+ 20 0 8198.9g 3.0g 1.3g S 72.2 9.6 161:59.37 mariadbd
      2026-10-22 17:10:01 : 48417 gcis_gc+ 20 0 8198.9g 3.4g 1.3g S 81.0 10.8 162:00.78 mariadbd
      2026-10-22 17:10:03 : 48417 gcis_gc+ 20 0 8199.2g 3.9g 1.3g S 100.0 12.6 162:02.12 mariadbd
      2026-10-22 17:10:04 : 48417 gcis_gc+ 20 0 8199.8g 4.5g 1.3g S 115.8 14.4 162:03.57 mariadbd
      2026-10-22 17:10:05 : 48417 gcis_gc+ 20 0 8200.4g 5.1g 1.3g S 122.2 16.4 162:04.99 mariadbd
      2026-10-22 17:10:06 : 48417 gcis_gc+ 20 0 8201.0g 5.7g 1.3g S 110.5 18.4 162:06.40 mariadbd
      ......
      2026-10-22 17:11:21 : 48417 gcis_gc+ 20 0 8231.2g 27.9g 0 S 105.3 89.7 163:17.60 mariadbd
      2026-10-22 17:11:23 : 48417 gcis_gc+ 20 0 0 0 0 Z 94.7 0.0 163:18.90 mariadbd
      2026-10-22 17:11:24 : (mariadb process not found)
      2026-10-22 17:11:25 : (mariadb process not found)
      2026-10-22 17:11:27 : (mariadb process not found)
      2026-10-22 17:11:28 : (mariadb process not found)
      2026-10-22 17:11:29 : (mariadb process not found)
      2026-10-22 17:11:30 : (mariadb process not found)
      2026-10-22 17:11:31 : (mariadb process not found)
      ......

      Test result is: 2 nodes mariadb crash which was triggered from OOM issued.

      ----------
      OS messages log:
      ....
      Node1,
      Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: vmtoolsd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
      Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: oom_kill_process.cold.33+0xb/0x10
      Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
      Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-gcisdba_orderdb01.service,task=mariadbd,pid=48417,uid=3002
      Oct 22 17:11:22 t1vuat-gcisdba-orderdb01-1-01 kernel: Out of memory: Killed process 48417 (mariadbd) total-vm:8631172872kB, anon-rss:29365024kB, file-rss:0kB, shmem-rss:0kB, UID:3002 pgtables:74516kB oom_score_adj:0

      Node2,
      Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: mariadbd invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
      Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: oom_kill_process.cold.33+0xb/0x10
      Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
      Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-gcisdba_orderdb01.service,task=mariadbd,pid=48762,uid=3002
      Oct 22 17:11:20 t2vuat-gcisdba-orderdb01-2-01 kernel: Out of memory: Killed process 48762 (mariadbd) total-vm:8631499128kB, anon-rss:29670848kB, file-rss:0kB, shmem-rss:0kB, UID:3002 pgtables:75156kB oom_score_adj:0

      -------
      the batch insert:
      500 single-row INSERTs into APP_REC_BOMB, trigger ON. Per-row trigger overhead: each INSERT also writes one journal row.
      (Refer to the attached bin log file "500 single-row INSERTs.sql")
      INSERT sample as below,
      #Q> INSERT INTO oomlab.APP_REC_BOMB
      #Q> (SCHEME_TYPE_CD,REC_ID,APPL_ID_TYPE,APPL_ID_NO,APP_NO,
      #Q> SECOND_FACTOR_VALUE,ACADEMIC_YR,STUDENT_ID,STUDENT_NAME,
      #Q> CHI_STUDENT_NAME,STATUS,RESULT_NOTIF_DT,ENG_MORE_INF,CHI_MORE_INF,
      #Q> CREATE_BY,CREATE_DT,LAST_UPD_BY,LAST_UPD_DT,SYSTEM_SCHEME_CD,SSO_ID)
      #Q> VALUES
      #Q> ('ENLS', 1, 'A', 'ID1', 'APP1', 'SF1', '2026', 'STU1',
      #Q> 'NAME 1', 'NAME_CHI 1', 'A', NOW(), 'ENG 1', 'CHI 1',
      #Q> 'WEB01_IMPORT', NOW(6), 'WEB01_IMPORT', NOW(6), 'ENLS', 'SSO-1')
      #261022 17:09:59 server id 1 end_log_pos 421219034 CRC32 0x7b774353 Table_map: `oomlab`.`seq_app_rec_bomb_jnl` mapped to number 371

      1. at 421219034
        #261022 17:09:59 server id 1 end_log_pos 421219125 CRC32 0xda34894b Write_rows: table id 371 flags: STMT_END_F
          1. INSERT INTO `oomlab`.`seq_app_rec_bomb_jnl`
          2. SET
          3. @1=70003 /* LONGINT meta=0 nullable=0 is_null=0 */
          4. @2=1 /* LONGINT meta=0 nullable=0 is_null=0 */
          5. @3=9999999999 /* LONGINT meta=0 nullable=0 is_null=0 */
          6. @4=1 /* LONGINT meta=0 nullable=0 is_null=0 */
          7. @5=1 /* LONGINT meta=0 nullable=0 is_null=0 */
          8. @6=0 /* LONGINT meta=0 nullable=0 is_null=0 */
          9. @7=0 /* TINYINT meta=0 nullable=0 is_null=0 */
          10. @8=0 /* LONGINT meta=0 nullable=0 is_null=0 */

      Attachments

        1. 500 single-row INSERTs.sql
          2.38 MB
          morris chen
        2. Mariadb_top_mem_list.log
          13 kB
          morris chen
        3. Run Test Readme.txt
          3 kB
          morris chen

        Activity

          People

            Unassigned Unassigned
            morrischen morris chen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.