Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41357

OOM Frequent happened when running batch insert data

    XMLWordPrintable

Details

    • Can result in hang or crash

    Description

      Our environment is three nodes (node1 + node2+witness) with galera cluster:
      node1: MariaDB version 10.11.18, 8core+32Gmem
      node2: MariaDB version 10.11.18, 8core+32Gmem
      galera: galera-4-26.4.27-1.el8.x86_64
      OS: Redhat 8.10 (4.18.0-553.150.1.el8_10.x86_64)
      ---------
      OS messages log:
      ....
      Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: chk_maria_top.s invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
      Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: oom_kill_process.cold.33+0xb/0x10
      Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
      Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mariadb-ddd_testdb01.service,task=mariadbd,pid=15824,uid=3001
      Sep 30 09:48:06 a1vuat-ddd-testdb01-1-01 kernel: Out of memory: Killed process 15824 (mariadbd) total-vm:64507224kB, anon-rss:31423688kB, file-rss:0kB, shmem-rss:0kB, UID:3001 pgtables:87164kB oom_score_adj:0

      -------
      the batch insert:
      The MariaDB instance was killed by the Linux OOM killer after a single session (thread_id 654320, user CCCC) accumulated ~29.5 GB of memory in memory/sql/thd::main_mem_root over roughly 40 seconds, driving the server to ~27 GB RES (87% of RAM) and exhausting system memory (30.7 GB used out of 31.8 GB total, with swap climbing).

      Time |Event |thd::main_mem_root (cur) |mariadbd RES
      9:44:04 |Baseline, normal operation |~0.67 MB |5.3 GB
      9:45:06 |mariadbd CPU jumps to 23.5%, |~0.70 MB |5.5 GB
      DELETE FROM CCC_DDD_STAGING ... LIMIT 10000 appears
      9:45:16 |CPU 170%, insert into CCC_DDD_STAGING begins |~0.70 MB |5.6 GB
      9:45:27 |CPU 176%, insert ... FFF_ID between 799219275 and 799229274 |0.75 MB |5.6 GB
      9:45:37 |insert ... TABLE_ROWID between 360000 and 369999 |0.81 MB |5.6 GB
      9:45:47 |Memory spike starts |5,261 MB |10.8 GB
      9:45:58 | |12,810 MB |18.0 GB
      9:46:08 | |20,640 MB |24.3 GB
      09:46:19–23 | |26,024 → 26,295 MB |27.3 GB
      9:46:34 | |29,411 MB |26.9 GB
      9:48:06 |OOM kill |~29.5 GB |killed
      9:48:18 |mariadbd restarted (PID 842110) |— |2.3 GB
      9:48:30 |restart again (PID 842254) |— |1.4 GB

      Attachments

        Activity

          People

            shipjain Shipra Jain
            John Gou John Gou
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.