Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6319

MariaDB 10.x gets killed by CentOS oom-killer with my.cnf that worked well for years

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 10.0.10, 10.0.11
    • Fix Version/s: 10.2
    • Component/s: None
    • Labels:
      None
    • Environment:
    • Sprint:
      5.5.48-0

      Description

      MariaDB 10.0.11 (and I believe also 10.0.10 before that) gets killed by CentOS' out-of-memory killer with a configuration that worked well for years in several other MySQL versions like MySQL 5.5 and MariaDB 5.3, before we upgraded to MariaDB 10.x.

      The server we use has 128GB of ram and MariaDB as the only real workload. Here's the original my.cnf that we've been using a very long time:

      port                                = 3306
      datadir                             = /var/lib/mysql
      socket                              = /var/lib/mysql/mysql.sock
      user                                = mysql
      key_buffer                          = 1536M
      max_allowed_packet                  = 64M
      table_open_cache                    = 10000
      table_definition_cache              = 10000
      sort_buffer_size                    = 8M
      read_buffer_size                    = 4M
      read_rnd_buffer_size                = 8M
      thread_cache_size                   = 64
      query_cache_limit                   = 2M
      query_cache_size                    = 384M
      query_cache_type                    = 1
      join_buffer_size                    = 4M
      thread_concurrency                  = 64
      tmp_table_size                      = 128M
      max_heap_table_size                 = 128M
      max_user_connections                = 350
      max_connections                     = 450
      long_query_time                     = 2
      wait_timeout                        = 7200
      interactive_timeout                 = 7200
      log-error                           = /var/log/mysql/error
      slow_query_log                      = 0
      slow_query_log_file                 = /var/log/mysql/slow-queries
      low_priority_updates                = 1
      concurrent_insert                   = 2
      myisam-recover
      myisam_repair_threads               = 1
      group_concat_max_len                = 1M
      max_connect_errors                  = 999999999
      collation_server                    = utf8_unicode_ci
      character_set_server                = utf8
      feedback                            = on
      tmpdir                              = /tmp
      innodb_data_home_dir                = /var/lib/mysql/
      innodb_data_file_path               = ibdata1:20M:autoextend
      innodb_buffer_pool_size             = 80G
      innodb_buffer_pool_instances        = 16
      innodb_log_group_home_dir           = /var/lib/mysql/
      innodb_log_files_in_group           = 2
      innodb_log_file_size                = 1G
      innodb_log_buffer_size              = 8M
      innodb_flush_log_at_trx_commit      = 0
      innodb_flush_method                 = O_DIRECT
      innodb_file_per_table               = 1
      innodb_open_files                   = 10000
      innodb_lock_wait_timeout            = 7200
      ft_min_word_len                     = 2
      ft_stopword_file                    =

      We've already reduced the innodb_buffer_pool_size to 50GB in multiple steps. With the 50GB configuration the mysqltuner.pl scripts reports that the maximum amount of memory that MySQL will use is 63GB, which shouldn't be any problem at all on a 128GB server.

      Still MySQL gets killed by the OOM-killer at least once a day. This is the part of the /var/log/messages kernel-log for the latest crash. At the last lines you see that mysqld used 123GB of memory at the moment of crash.

      Jun  8 17:07:03 hydra kernel: mysqld invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
      Jun  8 17:07:03 hydra kernel: mysqld cpuset=/ mems_allowed=0-1
      Jun  8 17:07:03 hydra kernel: Pid: 81246, comm: mysqld Not tainted 2.6.32-431.17.1.el6.x86_64 #1
      Jun  8 17:07:03 hydra kernel: Call Trace:
      Jun  8 17:07:03 hydra kernel: [<ffffffff810d0211>] ? cpuset_print_task_mems_allowed+0x91/0xb0
      Jun  8 17:07:03 hydra kernel: [<ffffffff811225c0>] ? dump_header+0x90/0x1b0
      Jun  8 17:07:03 hydra kernel: [<ffffffff8129011d>] ? __bitmap_intersects+0x1d/0xa0
      Jun  8 17:07:03 hydra kernel: [<ffffffff8122761c>] ? security_real_capable_noaudit+0x3c/0x70
      Jun  8 17:07:03 hydra kernel: [<ffffffff81122a42>] ? oom_kill_process+0x82/0x2a0
      Jun  8 17:07:03 hydra kernel: [<ffffffff81122981>] ? select_bad_process+0xe1/0x120
      Jun  8 17:07:03 hydra kernel: [<ffffffff81122e80>] ? out_of_memory+0x220/0x3c0
      Jun  8 17:07:03 hydra kernel: [<ffffffff8112f79f>] ? __alloc_pages_nodemask+0x89f/0x8d0
      Jun  8 17:07:03 hydra kernel: [<ffffffff8116779a>] ? alloc_pages_vma+0x9a/0x150
      Jun  8 17:07:03 hydra kernel: [<ffffffff8114a89d>] ? handle_pte_fault+0x73d/0xb00
      Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc8e>] ? invalidate_interrupt4+0xe/0x20
      Jun  8 17:07:03 hydra kernel: [<ffffffff8114ae8a>] ? handle_mm_fault+0x22a/0x300
      Jun  8 17:07:03 hydra kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
      Jun  8 17:07:03 hydra kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
      Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
      Jun  8 17:07:03 hydra kernel: [<ffffffff8100bc6e>] ? invalidate_interrupt3+0xe/0x20
      Jun  8 17:07:03 hydra kernel: [<ffffffff8152d73e>] ? do_page_fault+0x3e/0xa0
      Jun  8 17:07:03 hydra kernel: [<ffffffff8152aaf5>] ? page_fault+0x25/0x30
      Jun  8 17:07:03 hydra kernel: Mem-Info:
      Jun  8 17:07:03 hydra kernel: Node 0 DMA per-cpu:
      Jun  8 17:07:03 hydra kernel: CPU    0: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    1: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    2: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    3: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    4: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    5: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    6: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    7: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    8: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    9: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   10: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   11: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   12: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   13: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   14: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   15: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   16: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   17: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   18: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   19: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   20: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   21: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   22: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   23: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   24: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   25: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   26: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   27: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   28: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   29: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   30: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   31: hi:    0, btch:   1 usd:   0
      Jun  8 17:07:03 hydra kernel: Node 0 DMA32 per-cpu:
      Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:  22
      Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:  30
      Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:   3
      Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:   8
      Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:   3
      Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   7
      Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  22
      Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: Node 0 Normal per-cpu:
      Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:   7
      Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:   8
      Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:   5
      Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   4
      Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:   3
      Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   1
      Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  30
      Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: Node 1 Normal per-cpu:
      Jun  8 17:07:03 hydra kernel: CPU    0: hi:  186, btch:  31 usd:  26
      Jun  8 17:07:03 hydra kernel: CPU    1: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    2: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    3: hi:  186, btch:  31 usd:  30
      Jun  8 17:07:03 hydra kernel: CPU    4: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    5: hi:  186, btch:  31 usd:  17
      Jun  8 17:07:03 hydra kernel: CPU    6: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    7: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    8: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU    9: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   10: hi:  186, btch:  31 usd:  24
      Jun  8 17:07:03 hydra kernel: CPU   11: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   12: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   13: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   14: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   15: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   16: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   17: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   18: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   19: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   20: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   21: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   22: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   23: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   24: hi:  186, btch:  31 usd:  30
      Jun  8 17:07:03 hydra kernel: CPU   25: hi:  186, btch:  31 usd:  28
      Jun  8 17:07:03 hydra kernel: CPU   26: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   27: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   28: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   29: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   30: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: CPU   31: hi:  186, btch:  31 usd:   0
      Jun  8 17:07:03 hydra kernel: active_anon:31249425 inactive_anon:1380038 isolated_anon:1344
      Jun  8 17:07:03 hydra kernel: active_file:3585 inactive_file:2447 isolated_file:0
      Jun  8 17:07:03 hydra kernel: unevictable:0 dirty:1453 writeback:1295 unstable:0
      Jun  8 17:07:03 hydra kernel: free:88927 slab_reclaimable:19247 slab_unreclaimable:20626
      Jun  8 17:07:03 hydra kernel: mapped:3032 shmem:1 pagetables:77575 bounce:0
      Jun  8 17:07:03 hydra kernel: Node 0 DMA free:15240kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14860kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
      Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 1963 64583 64583
      Jun  8 17:07:03 hydra kernel: Node 0 DMA32 free:251840kB min:1368kB low:1708kB high:2052kB active_anon:334952kB inactive_anon:331512kB active_file:76kB inactive_file:128kB unevictable:0kB isolated(anon):512kB isolated(file):0kB present:2010560kB mlocked:0kB dirty:76kB writeback:32kB mapped:88kB shmem:0kB slab_reclaimable:184kB slab_unreclaimable:1076kB kernel_stack:48kB pagetables:2928kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? no
      Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 62620 62620
      Jun  8 17:07:03 hydra kernel: Node 0 Normal free:43648kB min:43660kB low:54572kB high:65488kB active_anon:61743464kB inactive_anon:2569912kB active_file:6736kB inactive_file:4912kB unevictable:0kB isolated(anon):3200kB isolated(file):0kB present:64122880kB mlocked:0kB dirty:3488kB writeback:0kB mapped:5320kB shmem:0kB slab_reclaimable:39256kB slab_unreclaimable:49908kB kernel_stack:9056kB pagetables:168272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7648 all_unreclaimable? no
      Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0
      Jun  8 17:07:03 hydra kernel: Node 1 Normal free:44980kB min:45068kB low:56332kB high:67600kB active_anon:62919284kB inactive_anon:2618728kB active_file:7528kB inactive_file:4748kB unevictable:0kB isolated(anon):1664kB isolated(file):0kB present:66191360kB mlocked:0kB dirty:2248kB writeback:5568kB mapped:6720kB shmem:4kB slab_reclaimable:37548kB slab_unreclaimable:31520kB kernel_stack:3688kB pagetables:139100kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8215 all_unreclaimable? no
      Jun  8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0
      Jun  8 17:07:03 hydra kernel: Node 0 DMA: 2*4kB 2*8kB 3*16kB 2*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 3*4096kB = 15240kB
      Jun  8 17:07:03 hydra kernel: Node 0 DMA32: 66*4kB 69*8kB 75*16kB 45*32kB 57*64kB 34*128kB 17*256kB 13*512kB 6*1024kB 3*2048kB 53*4096kB = 251840kB
      Jun  8 17:07:03 hydra kernel: Node 0 Normal: 9712*4kB 347*8kB 30*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 45848kB
      Jun  8 17:07:03 hydra kernel: Node 1 Normal: 10142*4kB 382*8kB 6*16kB 1*32kB 0*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 47592kB
      Jun  8 17:07:03 hydra kernel: 95591 total pagecache pages
      Jun  8 17:07:03 hydra kernel: 90149 pages in swap cache
      Jun  8 17:07:03 hydra kernel: Swap cache stats: add 13260661, delete 13170512, find 14306957/14869509
      Jun  8 17:07:03 hydra kernel: Free swap  = 0kB
      Jun  8 17:07:03 hydra kernel: Total swap = 2097144kB
      Jun  8 17:07:03 hydra kernel: 33554431 pages RAM
      Jun  8 17:07:03 hydra kernel: 519760 pages reserved
      Jun  8 17:07:03 hydra kernel: 10698 pages shared
      Jun  8 17:07:03 hydra kernel: 32928576 pages non-shared
      Jun  8 17:07:03 hydra kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
      Jun  8 17:07:03 hydra kernel: [  853]     0   853     2780        1   1     -17         -1000 udevd
      Jun  8 17:07:03 hydra kernel: [ 1610]     0  1610    23300       88  17     -17         -1000 auditd
      Jun  8 17:07:03 hydra kernel: [ 1626]     0  1626    62367      904   1       0             0 rsyslogd
      Jun  8 17:07:03 hydra kernel: [ 1655]    81  1655     5351        2   0       0             0 dbus-daemon
      Jun  8 17:07:03 hydra kernel: [ 1692]     0  1692    49448      251  22       0             0 snmpd
      Jun  8 17:07:03 hydra kernel: [ 1698]     0  1698    63771      211  13       0             0 lsi_mrdsnmpagen
      Jun  8 17:07:03 hydra kernel: [ 1711]     0  1711    16652       52  18     -17         -1000 sshd
      Jun  8 17:07:03 hydra kernel: [ 1745]     0  1745     5223       24   0       0             0 lsi_mrdsnmpagen
      Jun  8 17:07:03 hydra kernel: [ 2113]    93  2113    23413       28  17       0             0 exim
      Jun  8 17:07:03 hydra kernel: [ 2174]   496  2174     2707       35  29       0             0 epmd
      Jun  8 17:07:03 hydra kernel: [ 2226]     0  2226    27109        1   0       0             0 S80rabbitmq-ser
      Jun  8 17:07:03 hydra kernel: [ 2239]     0  2239    27041        2  18       0             0 bash
      Jun  8 17:07:03 hydra kernel: [ 2240]     0  2240    27041        2   4       0             0 rabbitmq-server
      Jun  8 17:07:03 hydra kernel: [ 2253]     0  2253    36359        2  20       0             0 su
      Jun  8 17:07:03 hydra kernel: [ 2260]   496  2260   650702     6906  18       0             0 beam.smp
      Jun  8 17:07:03 hydra kernel: [ 2430]   496  2430     2698        2   0       0             0 inet_gethost
      Jun  8 17:07:03 hydra kernel: [ 2431]   496  2431     4280        2   0       0             0 inet_gethost
      Jun  8 17:07:03 hydra kernel: [ 2439]     0  2439    29325       76  12       0             0 crond
      Jun  8 17:07:03 hydra kernel: [ 2450]     0  2450     5385        9  19       0             0 atd
      Jun  8 17:07:03 hydra kernel: [ 2466]     0  2466    27041        2   3       0             0 sh
      Jun  8 17:07:03 hydra kernel: [ 2471]     0  2471  8801199    17711   0       0             0 java
      Jun  8 17:07:03 hydra kernel: [ 2508]     0  2508    23103      349  19       0             0 miniserv.pl
      Jun  8 17:07:03 hydra kernel: [ 2517]     0  2517     1016        2   2       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2519]     0  2519     1016        2   2       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2521]     0  2521     1016        2  25       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2523]     0  2523     1016        2   2       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2527]     0  2527     1016        2   0       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2529]     0  2529     1016        2  19       0             0 mingetty
      Jun  8 17:07:03 hydra kernel: [ 2703]     0  2703  8779693       71   0       0             0 java
      Jun  8 17:07:03 hydra kernel: [ 3414]     0  3414     2779        1   1     -17         -1000 udevd
      Jun  8 17:07:03 hydra kernel: [ 3415]     0  3415     2779        1  17     -17         -1000 udevd
      Jun  8 17:07:03 hydra kernel: [81327]   495 81327  7355427  1578291  19       0             0 java
      Jun  8 17:07:03 hydra kernel: [30759]     0 30759    26549       83  17       0             0 mysqld_safe
      Jun  8 17:07:03 hydra kernel: [30993]   498 30993 37697878 30935404   0       0             0 mysqld
      Jun  8 17:07:03 hydra kernel: [31129]     0 31129    35792      475   8       0             0 munin-node
      Jun  8 17:07:03 hydra kernel: [32419]     0 32419    29810      290  19       0             0 screen
      Jun  8 17:07:03 hydra kernel: [32420]     0 32420    27109      216   2       0             0 bash
      Jun  8 17:07:03 hydra kernel: [74824]     0 74824     5545      144   1       0             0 xinetd
      Jun  8 17:07:03 hydra kernel: [77251]     0 77251    26515      130   2       0             0 memorylog
      Jun  8 17:07:03 hydra kernel: [77410]     0 77410     1532       92   1       0             0 vmstat
      Jun  8 17:07:03 hydra kernel: [14379]     0 14379    25227       77  16       0             0 sleep
      Jun  8 17:07:03 hydra kernel: [14671]     0 14671    35046      193   1       0             0 crond
      Jun  8 17:07:03 hydra kernel: [14672]     0 14672    35046      193  17       0             0 crond
      Jun  8 17:07:03 hydra kernel: [14690]     0 14690    26515      135   2       0             0 backup-blieb
      Jun  8 17:07:03 hydra kernel: [14691]     0 14691    26515      134  19       0             0 bash
      Jun  8 17:07:03 hydra kernel: [14693]     0 14693    26515      135  18       0             0 updatehits
      Jun  8 17:07:03 hydra kernel: [14733]     0 14733    56394      379   0       0             0 mysql
      Jun  8 17:07:03 hydra kernel: [14734]     0 14734    31785      372  21       0             0 mysqldump
      Jun  8 17:07:03 hydra kernel: [14735]     0 14735     1074       85   1       0             0 gzip
      Jun  8 17:07:03 hydra kernel: [14788]     0 14788    25356     4879  18       0             0 miniserv.pl
      Jun  8 17:07:03 hydra kernel: Out of memory: Kill process 30993 (mysqld) score 934 or sacrifice child
      Jun  8 17:07:03 hydra kernel: Killed process 30993, UID 498, (mysqld) total-vm:150791512kB, anon-rss:123736464kB, file-rss:5148kB

      The mysql error log doesn't show anything at the moment the process gets killed.

      I've attached an image of the munin memory usage graph of the server, where you clearly see the MariaDB reset at 17:07. If I understand correctly, it looks like MariaDB is allocation a lot of memory, while not really using it.

      Could this be a bug in MariaDB 10.x? What can I provide to help you find it?

        Attachments

        1. extendstatus1.txt
          14 kB
        2. extendstatus2.txt
          14 kB
        3. graph_crash at 16h51.png
          graph_crash at 16h51.png
          56 kB
        4. memory-day.png
          memory-day.png
          47 kB
        5. memory-day (2).png
          memory-day (2).png
          40 kB
        6. munin.jpg
          munin.jpg
          65 kB
        7. munin-production.png
          munin-production.png
          90 kB
        8. mycnf.txt
          3 kB
        9. myisam_week.png
          myisam_week.png
          64 kB
        10. myisam.png
          myisam.png
          85 kB

          Issue Links

            Activity

              People

              Assignee:
              svoj Sergey Vojtovich
              Reporter:
              koenc Koen Crijns
              Votes:
              7 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated: