[MDEV-6319] MariaDB 10.x gets killed by CentOS oom-killer with my.cnf that worked well for years - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Incomplete
Affects Version/s: 10.0.10, 10.0.11
Fix Version/s: N/A
Component/s: OTHER
Labels:
- need_feedback
Environment:

Hide
      System | Dell Inc.; PowerEdge R810; vNot Specified (Rack Mount Chassis)
    Platform | Linux
     Release | CentOS release 6.5 (Final)
      Kernel | 2.6.32-431.17.1.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
   Threading | NPTL 2.12
    Compiler | GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-4).
     SELinux | Disabled
Virtualized | No virtualization detected
  Processors | physical = 2, cores = 16, virtual = 32, hyperthreading = yes
      Speeds | 32x2260.906
      Models | 32xIntel(R) Xeon(R) CPU X7560 @ 2.27GHz
      Caches | 32x24576 KB
       Total | 126.0G
        Free | 55.5G
        Used | physical = 70.5G, swap allocated = 2.0G, swap used = 701.4M, virtual = 71.2G

Show
      System | Dell Inc.; PowerEdge R810; vNot Specified (Rack Mount Chassis)     Platform | Linux      Release | CentOS release 6.5 (Final)       Kernel | 2.6.32-431.17.1.el6.x86_64 Architecture | CPU = 64-bit, OS = 64-bit    Threading | NPTL 2.12     Compiler | GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-4).      SELinux | Disabled Virtualized | No virtualization detected   Processors | physical = 2, cores = 16, virtual = 32, hyperthreading = yes       Speeds | 32x2260.906       Models | 32xIntel(R) Xeon(R) CPU X7560 @ 2.27GHz       Caches | 32x24576 KB        Total | 126.0G         Free | 55.5G         Used | physical = 70.5G, swap allocated = 2.0G, swap used = 701.4M, virtual = 71.2G

Sprint:
5.5.48-0

Description

MariaDB 10.0.11 (and I believe also 10.0.10 before that) gets killed by CentOS' out-of-memory killer with a configuration that worked well for years in several other MySQL versions like MySQL 5.5 and MariaDB 5.3, before we upgraded to MariaDB 10.x.

The server we use has 128GB of ram and MariaDB as the only real workload. Here's the original my.cnf that we've been using a very long time:

port                                = 3306

datadir                             = /var/lib/mysql

socket                              = /var/lib/mysql/mysql.sock

user                                = mysql

key_buffer                          = 1536M

max_allowed_packet                  = 64M

table_open_cache                    = 10000

table_definition_cache              = 10000

sort_buffer_size                    = 8M

read_buffer_size                    = 4M

read_rnd_buffer_size                = 8M

thread_cache_size                   = 64

query_cache_limit                   = 2M

query_cache_size                    = 384M

query_cache_type                    = 1

join_buffer_size                    = 4M

thread_concurrency                  = 64

tmp_table_size                      = 128M

max_heap_table_size                 = 128M

max_user_connections                = 350

max_connections                     = 450

long_query_time                     = 2

wait_timeout                        = 7200

interactive_timeout                 = 7200

log-error                           = /var/log/mysql/error

slow_query_log                      = 0

slow_query_log_file                 = /var/log/mysql/slow-queries

low_priority_updates                = 1

concurrent_insert                   = 2

myisam-recover

myisam_repair_threads               = 1

group_concat_max_len                = 1M

max_connect_errors                  = 999999999

collation_server                    = utf8_unicode_ci

character_set_server                = utf8

feedback                            = on

tmpdir                              = /tmp

innodb_data_home_dir                = /var/lib/mysql/

innodb_data_file_path               = ibdata1:20M:autoextend

innodb_buffer_pool_size             = 80G

innodb_buffer_pool_instances        = 16

innodb_log_group_home_dir           = /var/lib/mysql/

innodb_log_files_in_group           = 2

innodb_log_file_size                = 1G

innodb_log_buffer_size              = 8M

innodb_flush_log_at_trx_commit      = 0

innodb_flush_method                 = O_DIRECT

innodb_file_per_table               = 1

innodb_open_files                   = 10000

innodb_lock_wait_timeout            = 7200

ft_min_word_len                     = 2

ft_stopword_file                    =

We've already reduced the innodb_buffer_pool_size to 50GB in multiple steps. With the 50GB configuration the mysqltuner.pl scripts reports that the maximum amount of memory that MySQL will use is 63GB, which shouldn't be any problem at all on a 128GB server.

Still MySQL gets killed by the OOM-killer at least once a day. This is the part of the /var/log/messages kernel-log for the latest crash. At the last lines you see that mysqld used 123GB of memory at the moment of crash.

Jun  8 17:07:03 hydra kernel: mysqld invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0

Jun  8 17:07:03 hydra kernel: mysqld cpuset=/ mems_allowed=0-1

Jun  8 17:07:03 hydra kernel: Pid: 81246, comm: mysqld Not tainted 2.6.32-431.17.1.el6.x86_64 #1

Jun  8 17:07:03 hydra kernel: Call Trace:

Jun  8 17:07:03 hydra kernel: [<ffffffff810d0211>] ? cpuset_print_task_mems_allowed+0x91/0xb0

Jun  8 17:07:03 hydra kernel: [<ffffffff811225c0>] ? dump_header+0x90/0x1b0

Jun  8 17:07:03 hydra kernel: [<ffffffff8129011d>] ? __bitmap_intersects+0x1d/0xa0

Jun  8 17:07:03 hydra kernel: [<ffffffff8122761c>] ? security_real_capable_noaudit+0x3c/0x70

Jun  8 17:07:03 hydra kernel: [<ffffffff81122a42>] ? oom_kill_process+0x82/0x2a0

Jun  8 17:07:03 hydra kernel: [<ffffffff81122981>] ? select_bad_process+0xe1/0x120

Jun  8 17:07:03 hydra kernel: [<ffffffff81122e80>] ? out_of_memory+0x220/0x3c0

Jun  8 17:07:03 hydra kernel: [<ffffffff8112f79f>] ? __alloc_pages_nodemask+0x89f/0x8d0

Jun  8 17:07:03 hydra kernel: [<ffffffff8116779a>] ? alloc_pages_vma+0x9a/0x150

Jun  8 17:07:03 hydra kernel: [<ffffffff8114a89d>] ? handle_pte_fault+0x73d/0xb00

Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc8e>] ? invalidate_interrupt4+0xe/0x20

Jun  8 17:07:03 hydra kernel: [<ffffffff8114ae8a>] ? handle_mm_fault+0x22a/0x300

Jun  8 17:07:03 hydra kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480

Jun  8 17:07:03 hydra kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320

Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20

Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc6e>] ? invalidate_interrupt3+0xe/0x20

Jun  8 17:07:03 hydra kernel: [<ffffffff8152d73e>] ? do_page_fault+0x3e/0xa0

Jun  8 17:07:03 hydra kernel: [<ffffffff8152aaf5>] ? page_fault+0x25/0x30

Jun  8 17:07:03 hydra kernel: Mem-Info:

Jun  8 17:07:03 hydra kernel: Node 0 DMA per-cpu:

Jun  8 17:07:03 hydra kernel: CPU    0: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    1: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    2: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    3: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    4: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    5: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    6: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    7: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    8: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    9: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   10: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   11: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   12: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   13: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   14: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   15: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   16: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   17: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   18: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   19: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   20: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   21: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   22: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   23: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   24: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   25: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   26: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   27: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   28: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   29: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   30: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   31: hi:    0, btch:   1 usd:   0

Jun  8 17:07:03 hydra kernel: Node 0 DMA32 per-cpu:

Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:  22

Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:  30

Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:   3

Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:   8

Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:   3

Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   7

Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  22

Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: Node 0 Normal per-cpu:

Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:   7

Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:   8

Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:   5

Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   4

Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:   3

Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   1

Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  30

Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: Node 1 Normal per-cpu:

Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:  26

Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:  30

Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:  17

Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:  24

Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  30

Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:  28

Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0

Jun  8 17:07:03 hydra kernel: active_anon:31249425 inactive_anon:1380038 isolated_anon:1344

Jun  8 17:07:03 hydra kernel: active_file:3585 inactive_file:2447 isolated_file:0

Jun  8 17:07:03 hydra kernel: unevictable:0 dirty:1453 writeback:1295 unstable:0

Jun  8 17:07:03 hydra kernel: free:88927 slab_reclaimable:19247 slab_unreclaimable:20626

Jun  8 17:07:03 hydra kernel: mapped:3032 shmem:1 pagetables:77575 bounce:0

Jun  8 17:07:03 hydra kernel: Node 0 DMA free:15240kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14860kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 1963 64583 64583

Jun  8 17:07:03 hydra kernel: Node 0 DMA32 free:251840kB min:1368kB low:1708kB high:2052kB active_anon:334952kB inactive_anon:331512kB active_file:76kB inactive_file:128kB unevictable:0kB isolated(anon):512kB isolated(file):0kB present:2010560kB mlocked:0kB dirty:76kB writeback:32kB mapped:88kB shmem:0kB slab_reclaimable:184kB slab_unreclaimable:1076kB kernel_stack:48kB pagetables:2928kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? no

Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 62620 62620

Jun  8 17:07:03 hydra kernel: Node 0 Normal free:43648kB min:43660kB low:54572kB high:65488kB active_anon:61743464kB inactive_anon:2569912kB active_file:6736kB inactive_file:4912kB unevictable:0kB isolated(anon):3200kB isolated(file):0kB present:64122880kB mlocked:0kB dirty:3488kB writeback:0kB mapped:5320kB shmem:0kB slab_reclaimable:39256kB slab_unreclaimable:49908kB kernel_stack:9056kB pagetables:168272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7648 all_unreclaimable? no

Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0

Jun  8 17:07:03 hydra kernel: Node 1 Normal free:44980kB min:45068kB low:56332kB high:67600kB active_anon:62919284kB inactive_anon:2618728kB active_file:7528kB inactive_file:4748kB unevictable:0kB isolated(anon):1664kB isolated(file):0kB present:66191360kB mlocked:0kB dirty:2248kB writeback:5568kB mapped:6720kB shmem:4kB slab_reclaimable:37548kB slab_unreclaimable:31520kB kernel_stack:3688kB pagetables:139100kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8215 all_unreclaimable? no

Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0

Jun  8 17:07:03 hydra kernel: Node 0 DMA: 2*4kB 2*8kB 3*16kB 2*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 3*4096kB = 15240kB

Jun  8 17:07:03 hydra kernel: Node 0 DMA32: 66*4kB 69*8kB 75*16kB 45*32kB 57*64kB 34*128kB 17*256kB 13*512kB 6*1024kB 3*2048kB 53*4096kB = 251840kB

Jun  8 17:07:03 hydra kernel: Node 0 Normal: 9712*4kB 347*8kB 30*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 45848kB

Jun  8 17:07:03 hydra kernel: Node 1 Normal: 10142*4kB 382*8kB 6*16kB 1*32kB 0*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 47592kB

Jun  8 17:07:03 hydra kernel: 95591 total pagecache pages

Jun  8 17:07:03 hydra kernel: 90149 pages in swap cache

Jun  8 17:07:03 hydra kernel: Swap cache stats: add 13260661, delete 13170512, find 14306957/14869509

Jun  8 17:07:03 hydra kernel: Free swap  = 0kB

Jun  8 17:07:03 hydra kernel: Total swap = 2097144kB

Jun  8 17:07:03 hydra kernel: 33554431 pages RAM

Jun  8 17:07:03 hydra kernel: 519760 pages reserved

Jun  8 17:07:03 hydra kernel: 10698 pages shared

Jun  8 17:07:03 hydra kernel: 32928576 pages non-shared

Jun  8 17:07:03 hydra kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name

Jun  8 17:07:03 hydra kernel: [  853]     0   853     2780        1   1     -17         -1000 udevd

Jun  8 17:07:03 hydra kernel: [ 1610]     0  1610    23300       88  17     -17         -1000 auditd

Jun  8 17:07:03 hydra kernel: [ 1626]     0  1626    62367      904   1       0             0 rsyslogd

Jun  8 17:07:03 hydra kernel: [ 1655]    81  1655     5351        2   0       0             0 dbus-daemon

Jun  8 17:07:03 hydra kernel: [ 1692]     0  1692    49448      251  22       0             0 snmpd

Jun  8 17:07:03 hydra kernel: [ 1698]     0  1698    63771      211  13       0             0 lsi_mrdsnmpagen

Jun  8 17:07:03 hydra kernel: [ 1711]     0  1711    16652       52  18     -17         -1000 sshd

Jun  8 17:07:03 hydra kernel: [ 1745]     0  1745     5223       24   0       0             0 lsi_mrdsnmpagen

Jun  8 17:07:03 hydra kernel: [ 2113]    93  2113    23413       28  17       0             0 exim

Jun  8 17:07:03 hydra kernel: [ 2174]   496  2174     2707       35  29       0             0 epmd

Jun  8 17:07:03 hydra kernel: [ 2226]     0  2226    27109        1   0       0             0 S80rabbitmq-ser

Jun  8 17:07:03 hydra kernel: [ 2239]     0  2239    27041        2  18       0             0 bash

Jun  8 17:07:03 hydra kernel: [ 2240]     0  2240    27041        2   4       0             0 rabbitmq-server

Jun  8 17:07:03 hydra kernel: [ 2253]     0  2253    36359        2  20       0             0 su

Jun  8 17:07:03 hydra kernel: [ 2260]   496  2260   650702     6906  18       0             0 beam.smp

Jun  8 17:07:03 hydra kernel: [ 2430]   496  2430     2698        2   0       0             0 inet_gethost

Jun  8 17:07:03 hydra kernel: [ 2431]   496  2431     4280        2   0       0             0 inet_gethost

Jun  8 17:07:03 hydra kernel: [ 2439]     0  2439    29325       76  12       0             0 crond

Jun  8 17:07:03 hydra kernel: [ 2450]     0  2450     5385        9  19       0             0 atd

Jun  8 17:07:03 hydra kernel: [ 2466]     0  2466    27041        2   3       0             0 sh

Jun  8 17:07:03 hydra kernel: [ 2471]     0  2471  8801199    17711   0       0             0 java

Jun  8 17:07:03 hydra kernel: [ 2508]     0  2508    23103      349  19       0             0 miniserv.pl

Jun  8 17:07:03 hydra kernel: [ 2517]     0  2517     1016        2   2       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2519]     0  2519     1016        2   2       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2521]     0  2521     1016        2  25       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2523]     0  2523     1016        2   2       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2527]     0  2527     1016        2   0       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2529]     0  2529     1016        2  19       0             0 mingetty

Jun  8 17:07:03 hydra kernel: [ 2703]     0  2703  8779693       71   0       0             0 java

Jun  8 17:07:03 hydra kernel: [ 3414]     0  3414     2779        1   1     -17         -1000 udevd

Jun  8 17:07:03 hydra kernel: [ 3415]     0  3415     2779        1  17     -17         -1000 udevd

Jun  8 17:07:03 hydra kernel: [81327]   495 81327  7355427  1578291  19       0             0 java

Jun  8 17:07:03 hydra kernel: [30759]     0 30759    26549       83  17       0             0 mysqld_safe

Jun  8 17:07:03 hydra kernel: [30993]   498 30993 37697878 30935404   0       0             0 mysqld

Jun  8 17:07:03 hydra kernel: [31129]     0 31129    35792      475   8       0             0 munin-node

Jun  8 17:07:03 hydra kernel: [32419]     0 32419    29810      290  19       0             0 screen

Jun  8 17:07:03 hydra kernel: [32420]     0 32420    27109      216   2       0             0 bash

Jun  8 17:07:03 hydra kernel: [74824]     0 74824     5545      144   1       0             0 xinetd

Jun  8 17:07:03 hydra kernel: [77251]     0 77251    26515      130   2       0             0 memorylog

Jun  8 17:07:03 hydra kernel: [77410]     0 77410     1532       92   1       0             0 vmstat

Jun  8 17:07:03 hydra kernel: [14379]     0 14379    25227       77  16       0             0 sleep

Jun  8 17:07:03 hydra kernel: [14671]     0 14671    35046      193   1       0             0 crond

Jun  8 17:07:03 hydra kernel: [14672]     0 14672    35046      193  17       0             0 crond

Jun  8 17:07:03 hydra kernel: [14690]     0 14690    26515      135   2       0             0 backup-blieb

Jun  8 17:07:03 hydra kernel: [14691]     0 14691    26515      134  19       0             0 bash

Jun  8 17:07:03 hydra kernel: [14693]     0 14693    26515      135  18       0             0 updatehits

Jun  8 17:07:03 hydra kernel: [14733]     0 14733    56394      379   0       0             0 mysql

Jun  8 17:07:03 hydra kernel: [14734]     0 14734    31785      372  21       0             0 mysqldump

Jun  8 17:07:03 hydra kernel: [14735]     0 14735     1074       85   1       0             0 gzip

Jun  8 17:07:03 hydra kernel: [14788]     0 14788    25356     4879  18       0             0 miniserv.pl

Jun  8 17:07:03 hydra kernel: Out of memory: Kill process 30993 (mysqld) score 934 or sacrifice child

Jun  8 17:07:03 hydra kernel: Killed process 30993, UID 498, (mysqld) total-vm:150791512kB, anon-rss:123736464kB, file-rss:5148kB

The mysql error log doesn't show anything at the moment the process gets killed.

I've attached an image of the munin memory usage graph of the server, where you clearly see the MariaDB reset at 17:07. If I understand correctly, it looks like MariaDB is allocation a lot of memory, while not really using it.

Could this be a bug in MariaDB 10.x? What can I provide to help you find it?

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

extendstatus1.txt
14 kB
2014-06-23 15:53
extendstatus2.txt
14 kB
2014-06-23 15:53
graph_crash at 16h51.png
56 kB
2016-02-04 19:40
memory-day.png
47 kB
2014-06-08 21:50
memory-day (2).png
40 kB
2014-06-16 09:57
munin.jpg
65 kB
2015-10-07 12:47
munin-production.png
90 kB
2015-10-23 16:19
mycnf.txt
3 kB
2014-06-23 15:53
myisam_week.png
64 kB
2016-02-15 11:15
myisam.png
85 kB
2016-02-09 14:57

Issue Links

relates to

MDEV-18866 Chewing Through Swap

Open

MDEV-9533 Query with an aggregate values in ORDER BY clause takes more memory and time than it should

Open

Activity

People

Assignee:: Sergey Vojtovich

Reporter:: Koen Crijns

Votes:: 7 Vote for this issue

Watchers:: 18 Start watching this issue

Dates

Created:: 2014-06-08 21:42

Updated:: 2021-01-15 12:00

Resolved:: 2021-01-15 12:00

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.