[MXS-4582] Maxscale crash with OOM Created: 2023-04-13  Updated: 2023-06-08  Resolved: 2023-06-07

Status: Closed
Project: MariaDB MaxScale
Component/s: readwritesplit
Affects Version/s: 23.02.1
Fix Version/s: 2.5.26, 6.4.7, 22.08.6, 23.02.2

Type: Bug Priority: Major
Reporter: Pramod Mahto Assignee: Niclas Antti
Resolution: Fixed Votes: 1
Labels: None

Attachments: Text File dmesg.txt    
Issue Links:
Relates
relates to MXS-4586 transaction_replay_max_size default i... Closed
Sprint: MXS-SPRINT-181, MXS-SPRINT-182, MXS-SPRINT-183, MXS-SPRINT-184

 Description   

Maxscale version :- 23.02.1 crashed with OOM even after increasing the Memory to higher end.

 
Apr  6 11:26:36 ext-coremx-dc1 kernel: Worker-13 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Apr  6 11:26:36 ext-coremx-dc1 kernel: Worker-13 cpuset=/ mems_allowed=0
Apr  6 11:26:36 ext-coremx-dc1 kernel: CPU: 11 PID: 24164 Comm: Worker-13 Kdump: loaded Not tainted 3.10.0-1160.88.1.el7.x86_64 #1
Apr  6 11:26:36 ext-coremx-dc1 kernel: Hardware name: oVirt RHEL/RHEL-AV, BIOS 1.16.0-4.el9 04/01/2014
Apr  6 11:26:36 ext-coremx-dc1 kernel: Call Trace:
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2db1bec>] dump_stack+0x19/0x1f
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dacb4f>] dump_header+0x90/0x22d
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb270d482>] ? ktime_get_ts64+0x52/0xf0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27668ef>] ? delayacct_end+0x8f/0xc0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cd335>] oom_kill_process+0x2d5/0x4a0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cccfd>] ? oom_unkillable_task+0xcd/0x120
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cda3a>] out_of_memory+0x31a/0x500
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27d4844>] __alloc_pages_nodemask+0xae4/0xbf0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb28259d8>] alloc_pages_current+0x98/0x110
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27c8c57>] __page_cache_alloc+0x97/0xb0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cbce0>] filemap_fault+0x270/0x420
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffc0364ffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb26e8a0c>] ? set_next_entity+0x3c/0xe0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffc036520c>] xfs_filemap_fault+0x2c/0x40 [xfs]
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27fa15a>] __do_fault.isra.61+0x8a/0x100
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27fa70c>] do_read_fault.isra.63+0x4c/0x1b0
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2802113>] handle_mm_fault+0xa33/0x1190
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbf7d3>] __do_page_fault+0x213/0x510
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbfbb6>] trace_do_page_fault+0x56/0x150
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbf112>] do_async_page_fault+0x22/0x100
Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbb7e8>] async_page_fault+0x28/0x30
Apr  6 11:26:36 ext-coremx-dc1 kernel: Mem-Info:
Apr  6 11:26:36 ext-coremx-dc1 kernel: active_anon:15245596 inactive_anon:721364 isolated_anon:0#012 active_file:0 inactive_file:396 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:6139 slab_unreclaimable:14441#012 mapped:495 shmem:619 pagetables:193757 bounce:0#012 free:87059 free_pcp:1688 free_cma:0
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA free:15892kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 1803 64245 64245
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA32 free:252036kB min:1896kB low:2368kB high:2844kB active_anon:1179444kB inactive_anon:393160kB active_file:16kB inactive_file:160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080620kB managed:1847116kB mlocked:0kB dirty:0kB writeback:0kB mapped:72kB shmem:24kB slab_reclaimable:492kB slab_unreclaimable:1456kB kernel_stack:64kB pagetables:16088kB unstable:0kB bounce:0kB free_pcp:1660kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:71 all_unreclaimable? no
Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 0 62441 62441
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 Normal free:80308kB min:65668kB low:82084kB high:98500kB active_anon:59802940kB inactive_anon:2492296kB active_file:0kB inactive_file:5656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63940560kB mlocked:0kB dirty:0kB writeback:0kB mapped:1908kB shmem:2452kB slab_reclaimable:24064kB slab_unreclaimable:56292kB kernel_stack:5536kB pagetables:758940kB unstable:0kB bounce:0kB free_pcp:5164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? no
Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 0 0 0
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA32: 719*4kB (UEM) 338*8kB (UEM) 86*16kB (UEM) 28*32kB (UEM) 21*64kB (UEM) 11*128kB (UEM) 24*256kB (UE) 20*512kB (UE) 4*1024kB (UE) 2*2048kB (UE) 53*4096kB (M) = 252268kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 Normal: 1870*4kB (UEM) 1225*8kB (UEM) 526*16kB (UEM) 243*32kB (UEM) 328*64kB (UEM) 192*128kB (EM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 79040kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: 11789 total pagecache pages
Apr  6 11:26:36 ext-coremx-dc1 kernel: 9135 pages in swap cache
Apr  6 11:26:36 ext-coremx-dc1 kernel: Swap cache stats: add 34607001, delete 34604069, find 2506804/2926671
Apr  6 11:26:36 ext-coremx-dc1 kernel: Free swap  = 400kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: Total swap = 16777212kB
Apr  6 11:26:36 ext-coremx-dc1 kernel: 16777081 pages RAM
Apr  6 11:26:36 ext-coremx-dc1 kernel: 0 pages HighMem/MovableOnly
Apr  6 11:26:36 ext-coremx-dc1 kernel: 326185 pages reserved
Apr  6 11:26:36 ext-coremx-dc1 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  526]     0   526     9859      722      24       49             0 systemd-journal
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  562]     0   562    29161        0      25       78             0 lvmetad
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  569]     0   569    11468        2      24      248         -1000 systemd-udevd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  692]     0   692    13883        1      26      110         -1000 auditd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  738]     0   738    11055       73      25       56             0 qemu-ga
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  739]   999   739   153085       12      61     2129             0 polkitd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  743]     0   743     6596       30      19       54             0 systemd-logind
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  749]     0   749     5447       53      16       54             0 irqbalance
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  751]    81   751    14523        4      31      129          -900 dbus-daemon
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  752]    32   752    17314       16      37      120             0 rpcbind
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  764]     0   764    48802        0      36      128             0 gssproxy
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  785]     0   785    27552        1      10       31             0 agetty
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  786]     0   786    31606       28      19      137             0 crond
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  972]     0   972   143571       80      99     2758             0 tuned
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  973]     0   973    28250        8      59      251         -1000 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [  979]     0   979    57733      325      46      212             0 rsyslogd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 1203]     0  1203    22451       19      43      240             0 master
Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 1225]    89  1225    22494        8      44      245             0 qmgr
Apr  6 11:26:36 ext-coremx-dc1 kernel: [27316]     0 27316    16376        3      34      167             0 keepalived
Apr  6 11:26:36 ext-coremx-dc1 kernel: [27317]     0 27317    16376       20      35      154             0 keepalived
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12633]     0 12633    40455        9      81      349             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12639]     0 12639    28914        1      13      114             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12807]     0 12807    40455       38      83      318             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12813]     0 12813    28914        1      13      119             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [13498]     0 13498    40497       44      81      328             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [13511]     0 13511    28915       44      14       77             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [13614]     0 13614    40455        0      81      351             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [13619]     0 13619    28915        1      14      119             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [30587]     0 30587   430872     3636      98     5103             0 telegraf
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12603]     0 12603    39751       93      78      305             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [12605]     0 12605    28918       53      13       75             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 9977]     0  9977    39343      155      35       90             0 watch
Apr  6 11:26:36 ext-coremx-dc1 kernel: [30171]     0 30171    27024       11      10       17             0 tail
Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 3309]     0  3309    40533      125      33       80             0 top
Apr  6 11:26:36 ext-coremx-dc1 kernel: [11662]     0 11662    39751       30      81      312             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [11674]     0 11674    28887        0      13      114             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [11688]     0 11688    39337      164      33       63             0 watch
Apr  6 11:26:36 ext-coremx-dc1 kernel: [18476]     0 18476    39751        0      82      341             0 sshd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [18479]     0 18479    28887        2      12      117             0 bash
Apr  6 11:26:36 ext-coremx-dc1 kernel: [20309]    38 20309     7490       33      19      121             0 ntpd
Apr  6 11:26:36 ext-coremx-dc1 kernel: [24132]   997 24132 98607733 15951978  192062  4174738             0 maxscale
Apr  6 11:26:36 ext-coremx-dc1 kernel: [24195]     0 24195    27024        0      10       26             0 tail
Apr  6 11:26:36 ext-coremx-dc1 kernel: [24398]     0 24398    27024        0      10       26             0 tail
Apr  6 11:26:36 ext-coremx-dc1 kernel: [27417]    89 27417    22477      252      47        0             0 pickup
Apr  6 11:26:36 ext-coremx-dc1 kernel: [31922]     0 31922    39342       67      29       90             0 watch
Apr  6 11:26:36 ext-coremx-dc1 kernel: [31923]     0 31923    13511      613      26        0             0 maxctrl
Apr  6 11:26:36 ext-coremx-dc1 kernel: [31925]     0 31925    39336       74      28       63             0 watch
Apr  6 11:26:36 ext-coremx-dc1 kernel: [31926]     0 31926    13511      679      27        0             0 maxctrl
Apr  6 11:26:36 ext-coremx-dc1 kernel: Out of memory: Kill process 24132 (maxscale) score 984 or sacrifice child
Apr  6 11:26:36 ext-coremx-dc1 kernel: Killed process 24132 (maxscale), UID 997, total-vm:394430932kB, anon-rss:63807844kB, file-rss:0kB, shmem-rss:0kB
Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service: main process exited, code=killed, status=9/KILL
Apr  6 11:26:44 ext-coremx-dc1 systemd: Unit maxscale.service entered failed state.
Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service failed.
Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service holdoff time over, scheduling restart.
Apr  6 11:26:44 ext-coremx-dc1 systemd: Stopped MariaDB MaxScale Database Proxy.
Apr  6 11:26:44 ext-coremx-dc1 systemd: Starting MariaDB MaxScale Database Proxy...



 Comments   
Comment by Eoin Kim [ 2023-04-18 ]

I am having the same issue. Kernel log message attached. dmesg.txt

MaxScale is running on a VM - 2 CPUs, 4GB RAM.

Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-146-generic x86_64)

Generated at Thu Feb 08 04:29:41 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.