Maxscale crash with OOM

Apr  6 11:26:36 ext-coremx-dc1 kernel: Worker-13 invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0

Apr  6 11:26:36 ext-coremx-dc1 kernel: Worker-13 cpuset=/ mems_allowed=0

Apr  6 11:26:36 ext-coremx-dc1 kernel: CPU: 11 PID: 24164 Comm: Worker-13 Kdump: loaded Not tainted 3.10.0-1160.88.1.el7.x86_64 #1

Apr  6 11:26:36 ext-coremx-dc1 kernel: Hardware name: oVirt RHEL/RHEL-AV, BIOS 1.16.0-4.el9 04/01/2014

Apr  6 11:26:36 ext-coremx-dc1 kernel: Call Trace:

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2db1bec>] dump_stack+0x19/0x1f

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dacb4f>] dump_header+0x90/0x22d

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb270d482>] ? ktime_get_ts64+0x52/0xf0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27668ef>] ? delayacct_end+0x8f/0xc0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cd335>] oom_kill_process+0x2d5/0x4a0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cccfd>] ? oom_unkillable_task+0xcd/0x120

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cda3a>] out_of_memory+0x31a/0x500

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27d4844>] __alloc_pages_nodemask+0xae4/0xbf0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb28259d8>] alloc_pages_current+0x98/0x110

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27c8c57>] __page_cache_alloc+0x97/0xb0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27cbce0>] filemap_fault+0x270/0x420

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffc0364ffe>] __xfs_filemap_fault+0x7e/0x1d0 [xfs]

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb26e8a0c>] ? set_next_entity+0x3c/0xe0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffc036520c>] xfs_filemap_fault+0x2c/0x40 [xfs]

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27fa15a>] __do_fault.isra.61+0x8a/0x100

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb27fa70c>] do_read_fault.isra.63+0x4c/0x1b0

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2802113>] handle_mm_fault+0xa33/0x1190

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbf7d3>] __do_page_fault+0x213/0x510

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbfbb6>] trace_do_page_fault+0x56/0x150

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbf112>] do_async_page_fault+0x22/0x100

Apr  6 11:26:36 ext-coremx-dc1 kernel: [<ffffffffb2dbb7e8>] async_page_fault+0x28/0x30

Apr  6 11:26:36 ext-coremx-dc1 kernel: Mem-Info:

Apr  6 11:26:36 ext-coremx-dc1 kernel: active_anon:15245596 inactive_anon:721364 isolated_anon:0#012 active_file:0 inactive_file:396 isolated_file:0#012 unevictable:0 dirty:0 writeback:0 unstable:0#012 slab_reclaimable:6139 slab_unreclaimable:14441#012 mapped:495 shmem:619 pagetables:193757 bounce:0#012 free:87059 free_pcp:1688 free_cma:0

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA free:15892kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 1803 64245 64245

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA32 free:252036kB min:1896kB low:2368kB high:2844kB active_anon:1179444kB inactive_anon:393160kB active_file:16kB inactive_file:160kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080620kB managed:1847116kB mlocked:0kB dirty:0kB writeback:0kB mapped:72kB shmem:24kB slab_reclaimable:492kB slab_unreclaimable:1456kB kernel_stack:64kB pagetables:16088kB unstable:0kB bounce:0kB free_pcp:1660kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:71 all_unreclaimable? no

Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 0 62441 62441

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 Normal free:80308kB min:65668kB low:82084kB high:98500kB active_anon:59802940kB inactive_anon:2492296kB active_file:0kB inactive_file:5656kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65011712kB managed:63940560kB mlocked:0kB dirty:0kB writeback:0kB mapped:1908kB shmem:2452kB slab_reclaimable:24064kB slab_unreclaimable:56292kB kernel_stack:5536kB pagetables:758940kB unstable:0kB bounce:0kB free_pcp:5164kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:24 all_unreclaimable? no

Apr  6 11:26:36 ext-coremx-dc1 kernel: lowmem_reserve[]: 0 0 0 0

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 DMA32: 719*4kB (UEM) 338*8kB (UEM) 86*16kB (UEM) 28*32kB (UEM) 21*64kB (UEM) 11*128kB (UEM) 24*256kB (UE) 20*512kB (UE) 4*1024kB (UE) 2*2048kB (UE) 53*4096kB (M) = 252268kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 Normal: 1870*4kB (UEM) 1225*8kB (UEM) 526*16kB (UEM) 243*32kB (UEM) 328*64kB (UEM) 192*128kB (EM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 79040kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: 11789 total pagecache pages

Apr  6 11:26:36 ext-coremx-dc1 kernel: 9135 pages in swap cache

Apr  6 11:26:36 ext-coremx-dc1 kernel: Swap cache stats: add 34607001, delete 34604069, find 2506804/2926671

Apr  6 11:26:36 ext-coremx-dc1 kernel: Free swap  = 400kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: Total swap = 16777212kB

Apr  6 11:26:36 ext-coremx-dc1 kernel: 16777081 pages RAM

Apr  6 11:26:36 ext-coremx-dc1 kernel: 0 pages HighMem/MovableOnly

Apr  6 11:26:36 ext-coremx-dc1 kernel: 326185 pages reserved

Apr  6 11:26:36 ext-coremx-dc1 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  526]     0   526     9859      722      24       49             0 systemd-journal

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  562]     0   562    29161        0      25       78             0 lvmetad

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  569]     0   569    11468        2      24      248         -1000 systemd-udevd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  692]     0   692    13883        1      26      110         -1000 auditd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  738]     0   738    11055       73      25       56             0 qemu-ga

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  739]   999   739   153085       12      61     2129             0 polkitd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  743]     0   743     6596       30      19       54             0 systemd-logind

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  749]     0   749     5447       53      16       54             0 irqbalance

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  751]    81   751    14523        4      31      129          -900 dbus-daemon

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  752]    32   752    17314       16      37      120             0 rpcbind

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  764]     0   764    48802        0      36      128             0 gssproxy

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  785]     0   785    27552        1      10       31             0 agetty

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  786]     0   786    31606       28      19      137             0 crond

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  972]     0   972   143571       80      99     2758             0 tuned

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  973]     0   973    28250        8      59      251         -1000 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [  979]     0   979    57733      325      46      212             0 rsyslogd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 1203]     0  1203    22451       19      43      240             0 master

Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 1225]    89  1225    22494        8      44      245             0 qmgr

Apr  6 11:26:36 ext-coremx-dc1 kernel: [27316]     0 27316    16376        3      34      167             0 keepalived

Apr  6 11:26:36 ext-coremx-dc1 kernel: [27317]     0 27317    16376       20      35      154             0 keepalived

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12633]     0 12633    40455        9      81      349             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12639]     0 12639    28914        1      13      114             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12807]     0 12807    40455       38      83      318             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12813]     0 12813    28914        1      13      119             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [13498]     0 13498    40497       44      81      328             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [13511]     0 13511    28915       44      14       77             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [13614]     0 13614    40455        0      81      351             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [13619]     0 13619    28915        1      14      119             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [30587]     0 30587   430872     3636      98     5103             0 telegraf

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12603]     0 12603    39751       93      78      305             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [12605]     0 12605    28918       53      13       75             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 9977]     0  9977    39343      155      35       90             0 watch

Apr  6 11:26:36 ext-coremx-dc1 kernel: [30171]     0 30171    27024       11      10       17             0 tail

Apr  6 11:26:36 ext-coremx-dc1 kernel: [ 3309]     0  3309    40533      125      33       80             0 top

Apr  6 11:26:36 ext-coremx-dc1 kernel: [11662]     0 11662    39751       30      81      312             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [11674]     0 11674    28887        0      13      114             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [11688]     0 11688    39337      164      33       63             0 watch

Apr  6 11:26:36 ext-coremx-dc1 kernel: [18476]     0 18476    39751        0      82      341             0 sshd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [18479]     0 18479    28887        2      12      117             0 bash

Apr  6 11:26:36 ext-coremx-dc1 kernel: [20309]    38 20309     7490       33      19      121             0 ntpd

Apr  6 11:26:36 ext-coremx-dc1 kernel: [24132]   997 24132 98607733 15951978  192062  4174738             0 maxscale

Apr  6 11:26:36 ext-coremx-dc1 kernel: [24195]     0 24195    27024        0      10       26             0 tail

Apr  6 11:26:36 ext-coremx-dc1 kernel: [24398]     0 24398    27024        0      10       26             0 tail

Apr  6 11:26:36 ext-coremx-dc1 kernel: [27417]    89 27417    22477      252      47        0             0 pickup

Apr  6 11:26:36 ext-coremx-dc1 kernel: [31922]     0 31922    39342       67      29       90             0 watch

Apr  6 11:26:36 ext-coremx-dc1 kernel: [31923]     0 31923    13511      613      26        0             0 maxctrl

Apr  6 11:26:36 ext-coremx-dc1 kernel: [31925]     0 31925    39336       74      28       63             0 watch

Apr  6 11:26:36 ext-coremx-dc1 kernel: [31926]     0 31926    13511      679      27        0             0 maxctrl

Apr  6 11:26:36 ext-coremx-dc1 kernel: Out of memory: Kill process 24132 (maxscale) score 984 or sacrifice child

Apr  6 11:26:36 ext-coremx-dc1 kernel: Killed process 24132 (maxscale), UID 997, total-vm:394430932kB, anon-rss:63807844kB, file-rss:0kB, shmem-rss:0kB

Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service: main process exited, code=killed, status=9/KILL

Apr  6 11:26:44 ext-coremx-dc1 systemd: Unit maxscale.service entered failed state.

Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service failed.

Apr  6 11:26:44 ext-coremx-dc1 systemd: maxscale.service holdoff time over, scheduling restart.

Apr  6 11:26:44 ext-coremx-dc1 systemd: Stopped MariaDB MaxScale Database Proxy.

Apr  6 11:26:44 ext-coremx-dc1 systemd: Starting MariaDB MaxScale Database Proxy...