Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Incomplete
-
10.0.10, 10.0.11
-
System | Dell Inc.; PowerEdge R810; vNot Specified (Rack Mount Chassis)
Platform | Linux
Release | CentOS release 6.5 (Final)
Kernel | 2.6.32-431.17.1.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
Compiler | GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-4).
SELinux | Disabled
Virtualized | No virtualization detected
Processors | physical = 2, cores = 16, virtual = 32, hyperthreading = yes
Speeds | 32x2260.906
Models | 32xIntel(R) Xeon(R) CPU X7560 @ 2.27GHz
Caches | 32x24576 KB
Total | 126.0G
Free | 55.5G
Used | physical = 70.5G, swap allocated = 2.0G, swap used = 701.4M, virtual = 71.2G
System | Dell Inc.; PowerEdge R810; vNot Specified (Rack Mount Chassis) Platform | Linux Release | CentOS release 6.5 (Final) Kernel | 2.6.32-431.17.1.el6.x86_64 Architecture | CPU = 64-bit, OS = 64-bit Threading | NPTL 2.12 Compiler | GNU CC version 4.4.7 20120313 (Red Hat 4.4.7-4). SELinux | Disabled Virtualized | No virtualization detected Processors | physical = 2, cores = 16, virtual = 32, hyperthreading = yes Speeds | 32x2260.906 Models | 32xIntel(R) Xeon(R) CPU X7560 @ 2.27GHz Caches | 32x24576 KB Total | 126.0G Free | 55.5G Used | physical = 70.5G, swap allocated = 2.0G, swap used = 701.4M, virtual = 71.2G
-
5.5.48-0
Description
MariaDB 10.0.11 (and I believe also 10.0.10 before that) gets killed by CentOS' out-of-memory killer with a configuration that worked well for years in several other MySQL versions like MySQL 5.5 and MariaDB 5.3, before we upgraded to MariaDB 10.x.
The server we use has 128GB of ram and MariaDB as the only real workload. Here's the original my.cnf that we've been using a very long time:
port = 3306
|
datadir = /var/lib/mysql
|
socket = /var/lib/mysql/mysql.sock
|
user = mysql
|
key_buffer = 1536M
|
max_allowed_packet = 64M
|
table_open_cache = 10000
|
table_definition_cache = 10000
|
sort_buffer_size = 8M
|
read_buffer_size = 4M
|
read_rnd_buffer_size = 8M
|
thread_cache_size = 64
|
query_cache_limit = 2M
|
query_cache_size = 384M
|
query_cache_type = 1
|
join_buffer_size = 4M
|
thread_concurrency = 64
|
tmp_table_size = 128M
|
max_heap_table_size = 128M
|
max_user_connections = 350
|
max_connections = 450
|
long_query_time = 2
|
wait_timeout = 7200
|
interactive_timeout = 7200
|
log-error = /var/log/mysql/error
|
slow_query_log = 0
|
slow_query_log_file = /var/log/mysql/slow-queries
|
low_priority_updates = 1
|
concurrent_insert = 2
|
myisam-recover
|
myisam_repair_threads = 1
|
group_concat_max_len = 1M
|
max_connect_errors = 999999999
|
collation_server = utf8_unicode_ci
|
character_set_server = utf8
|
feedback = on
|
tmpdir = /tmp
|
innodb_data_home_dir = /var/lib/mysql/
|
innodb_data_file_path = ibdata1:20M:autoextend
|
innodb_buffer_pool_size = 80G
|
innodb_buffer_pool_instances = 16
|
innodb_log_group_home_dir = /var/lib/mysql/
|
innodb_log_files_in_group = 2
|
innodb_log_file_size = 1G
|
innodb_log_buffer_size = 8M
|
innodb_flush_log_at_trx_commit = 0
|
innodb_flush_method = O_DIRECT
|
innodb_file_per_table = 1
|
innodb_open_files = 10000
|
innodb_lock_wait_timeout = 7200
|
ft_min_word_len = 2
|
ft_stopword_file =
|
We've already reduced the innodb_buffer_pool_size to 50GB in multiple steps. With the 50GB configuration the mysqltuner.pl scripts reports that the maximum amount of memory that MySQL will use is 63GB, which shouldn't be any problem at all on a 128GB server.
Still MySQL gets killed by the OOM-killer at least once a day. This is the part of the /var/log/messages kernel-log for the latest crash. At the last lines you see that mysqld used 123GB of memory at the moment of crash.
Jun 8 17:07:03 hydra kernel: mysqld invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
|
Jun 8 17:07:03 hydra kernel: mysqld cpuset=/ mems_allowed=0-1
|
Jun 8 17:07:03 hydra kernel: Pid: 81246, comm: mysqld Not tainted 2.6.32-431.17.1.el6.x86_64 #1
|
Jun 8 17:07:03 hydra kernel: Call Trace:
|
Jun 8 17:07:03 hydra kernel: [<ffffffff810d0211>] ? cpuset_print_task_mems_allowed+0x91/0xb0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff811225c0>] ? dump_header+0x90/0x1b0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8129011d>] ? __bitmap_intersects+0x1d/0xa0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8122761c>] ? security_real_capable_noaudit+0x3c/0x70
|
Jun 8 17:07:03 hydra kernel: [<ffffffff81122a42>] ? oom_kill_process+0x82/0x2a0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff81122981>] ? select_bad_process+0xe1/0x120
|
Jun 8 17:07:03 hydra kernel: [<ffffffff81122e80>] ? out_of_memory+0x220/0x3c0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8112f79f>] ? __alloc_pages_nodemask+0x89f/0x8d0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8116779a>] ? alloc_pages_vma+0x9a/0x150
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8114a89d>] ? handle_pte_fault+0x73d/0xb00
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8100bc8e>] ? invalidate_interrupt4+0xe/0x20
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8114ae8a>] ? handle_mm_fault+0x22a/0x300
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8104a8d8>] ? __do_page_fault+0x138/0x480
|
Jun 8 17:07:03 hydra kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8100bc2e>] ? invalidate_interrupt1+0xe/0x20
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8100bc6e>] ? invalidate_interrupt3+0xe/0x20
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8152d73e>] ? do_page_fault+0x3e/0xa0
|
Jun 8 17:07:03 hydra kernel: [<ffffffff8152aaf5>] ? page_fault+0x25/0x30
|
Jun 8 17:07:03 hydra kernel: Mem-Info:
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA per-cpu:
|
Jun 8 17:07:03 hydra kernel: CPU 0: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 1: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 2: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 3: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 4: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 5: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 6: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 7: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 8: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 9: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 10: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 11: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 12: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 13: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 14: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 15: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 16: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 17: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 18: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 19: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 20: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 21: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 22: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 23: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 24: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 25: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 26: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 27: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 28: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 29: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 30: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 31: hi: 0, btch: 1 usd: 0
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA32 per-cpu:
|
Jun 8 17:07:03 hydra kernel: CPU 0: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 1: hi: 186, btch: 31 usd: 22
|
Jun 8 17:07:03 hydra kernel: CPU 2: hi: 186, btch: 31 usd: 30
|
Jun 8 17:07:03 hydra kernel: CPU 3: hi: 186, btch: 31 usd: 3
|
Jun 8 17:07:03 hydra kernel: CPU 4: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 5: hi: 186, btch: 31 usd: 8
|
Jun 8 17:07:03 hydra kernel: CPU 6: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 7: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 9: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 10: hi: 186, btch: 31 usd: 3
|
Jun 8 17:07:03 hydra kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 12: hi: 186, btch: 31 usd: 7
|
Jun 8 17:07:03 hydra kernel: CPU 13: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 14: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 15: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 16: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 17: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 18: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 19: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 20: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 21: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 22: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 23: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 24: hi: 186, btch: 31 usd: 22
|
Jun 8 17:07:03 hydra kernel: CPU 25: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 26: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 27: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 28: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 29: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 30: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 31: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: Node 0 Normal per-cpu:
|
Jun 8 17:07:03 hydra kernel: CPU 0: hi: 186, btch: 31 usd: 7
|
Jun 8 17:07:03 hydra kernel: CPU 1: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 2: hi: 186, btch: 31 usd: 8
|
Jun 8 17:07:03 hydra kernel: CPU 3: hi: 186, btch: 31 usd: 5
|
Jun 8 17:07:03 hydra kernel: CPU 4: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 5: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 6: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 7: hi: 186, btch: 31 usd: 4
|
Jun 8 17:07:03 hydra kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 9: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 10: hi: 186, btch: 31 usd: 3
|
Jun 8 17:07:03 hydra kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 12: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 13: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 14: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 15: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 16: hi: 186, btch: 31 usd: 1
|
Jun 8 17:07:03 hydra kernel: CPU 17: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 18: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 19: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 20: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 21: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 22: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 23: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 24: hi: 186, btch: 31 usd: 30
|
Jun 8 17:07:03 hydra kernel: CPU 25: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 26: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 27: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 28: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 29: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 30: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 31: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: Node 1 Normal per-cpu:
|
Jun 8 17:07:03 hydra kernel: CPU 0: hi: 186, btch: 31 usd: 26
|
Jun 8 17:07:03 hydra kernel: CPU 1: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 2: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 3: hi: 186, btch: 31 usd: 30
|
Jun 8 17:07:03 hydra kernel: CPU 4: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 5: hi: 186, btch: 31 usd: 17
|
Jun 8 17:07:03 hydra kernel: CPU 6: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 7: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 8: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 9: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 10: hi: 186, btch: 31 usd: 24
|
Jun 8 17:07:03 hydra kernel: CPU 11: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 12: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 13: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 14: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 15: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 16: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 17: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 18: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 19: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 20: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 21: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 22: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 23: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 24: hi: 186, btch: 31 usd: 30
|
Jun 8 17:07:03 hydra kernel: CPU 25: hi: 186, btch: 31 usd: 28
|
Jun 8 17:07:03 hydra kernel: CPU 26: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 27: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 28: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 29: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 30: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: CPU 31: hi: 186, btch: 31 usd: 0
|
Jun 8 17:07:03 hydra kernel: active_anon:31249425 inactive_anon:1380038 isolated_anon:1344
|
Jun 8 17:07:03 hydra kernel: active_file:3585 inactive_file:2447 isolated_file:0
|
Jun 8 17:07:03 hydra kernel: unevictable:0 dirty:1453 writeback:1295 unstable:0
|
Jun 8 17:07:03 hydra kernel: free:88927 slab_reclaimable:19247 slab_unreclaimable:20626
|
Jun 8 17:07:03 hydra kernel: mapped:3032 shmem:1 pagetables:77575 bounce:0
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA free:15240kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:14860kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
|
Jun 8 17:07:03 hydra kernel: lowmem_reserve[]: 0 1963 64583 64583
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA32 free:251840kB min:1368kB low:1708kB high:2052kB active_anon:334952kB inactive_anon:331512kB active_file:76kB inactive_file:128kB unevictable:0kB isolated(anon):512kB isolated(file):0kB present:2010560kB mlocked:0kB dirty:76kB writeback:32kB mapped:88kB shmem:0kB slab_reclaimable:184kB slab_unreclaimable:1076kB kernel_stack:48kB pagetables:2928kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:448 all_unreclaimable? no
|
Jun 8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 62620 62620
|
Jun 8 17:07:03 hydra kernel: Node 0 Normal free:43648kB min:43660kB low:54572kB high:65488kB active_anon:61743464kB inactive_anon:2569912kB active_file:6736kB inactive_file:4912kB unevictable:0kB isolated(anon):3200kB isolated(file):0kB present:64122880kB mlocked:0kB dirty:3488kB writeback:0kB mapped:5320kB shmem:0kB slab_reclaimable:39256kB slab_unreclaimable:49908kB kernel_stack:9056kB pagetables:168272kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:7648 all_unreclaimable? no
|
Jun 8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0
|
Jun 8 17:07:03 hydra kernel: Node 1 Normal free:44980kB min:45068kB low:56332kB high:67600kB active_anon:62919284kB inactive_anon:2618728kB active_file:7528kB inactive_file:4748kB unevictable:0kB isolated(anon):1664kB isolated(file):0kB present:66191360kB mlocked:0kB dirty:2248kB writeback:5568kB mapped:6720kB shmem:4kB slab_reclaimable:37548kB slab_unreclaimable:31520kB kernel_stack:3688kB pagetables:139100kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:8215 all_unreclaimable? no
|
Jun 8 17:07:03 hydra kernel: lowmem_reserve[]: 0 0 0 0
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA: 2*4kB 2*8kB 3*16kB 2*32kB 0*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 1*2048kB 3*4096kB = 15240kB
|
Jun 8 17:07:03 hydra kernel: Node 0 DMA32: 66*4kB 69*8kB 75*16kB 45*32kB 57*64kB 34*128kB 17*256kB 13*512kB 6*1024kB 3*2048kB 53*4096kB = 251840kB
|
Jun 8 17:07:03 hydra kernel: Node 0 Normal: 9712*4kB 347*8kB 30*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 45848kB
|
Jun 8 17:07:03 hydra kernel: Node 1 Normal: 10142*4kB 382*8kB 6*16kB 1*32kB 0*64kB 2*128kB 0*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 47592kB
|
Jun 8 17:07:03 hydra kernel: 95591 total pagecache pages
|
Jun 8 17:07:03 hydra kernel: 90149 pages in swap cache
|
Jun 8 17:07:03 hydra kernel: Swap cache stats: add 13260661, delete 13170512, find 14306957/14869509
|
Jun 8 17:07:03 hydra kernel: Free swap = 0kB
|
Jun 8 17:07:03 hydra kernel: Total swap = 2097144kB
|
Jun 8 17:07:03 hydra kernel: 33554431 pages RAM
|
Jun 8 17:07:03 hydra kernel: 519760 pages reserved
|
Jun 8 17:07:03 hydra kernel: 10698 pages shared
|
Jun 8 17:07:03 hydra kernel: 32928576 pages non-shared
|
Jun 8 17:07:03 hydra kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
|
Jun 8 17:07:03 hydra kernel: [ 853] 0 853 2780 1 1 -17 -1000 udevd
|
Jun 8 17:07:03 hydra kernel: [ 1610] 0 1610 23300 88 17 -17 -1000 auditd
|
Jun 8 17:07:03 hydra kernel: [ 1626] 0 1626 62367 904 1 0 0 rsyslogd
|
Jun 8 17:07:03 hydra kernel: [ 1655] 81 1655 5351 2 0 0 0 dbus-daemon
|
Jun 8 17:07:03 hydra kernel: [ 1692] 0 1692 49448 251 22 0 0 snmpd
|
Jun 8 17:07:03 hydra kernel: [ 1698] 0 1698 63771 211 13 0 0 lsi_mrdsnmpagen
|
Jun 8 17:07:03 hydra kernel: [ 1711] 0 1711 16652 52 18 -17 -1000 sshd
|
Jun 8 17:07:03 hydra kernel: [ 1745] 0 1745 5223 24 0 0 0 lsi_mrdsnmpagen
|
Jun 8 17:07:03 hydra kernel: [ 2113] 93 2113 23413 28 17 0 0 exim
|
Jun 8 17:07:03 hydra kernel: [ 2174] 496 2174 2707 35 29 0 0 epmd
|
Jun 8 17:07:03 hydra kernel: [ 2226] 0 2226 27109 1 0 0 0 S80rabbitmq-ser
|
Jun 8 17:07:03 hydra kernel: [ 2239] 0 2239 27041 2 18 0 0 bash
|
Jun 8 17:07:03 hydra kernel: [ 2240] 0 2240 27041 2 4 0 0 rabbitmq-server
|
Jun 8 17:07:03 hydra kernel: [ 2253] 0 2253 36359 2 20 0 0 su
|
Jun 8 17:07:03 hydra kernel: [ 2260] 496 2260 650702 6906 18 0 0 beam.smp
|
Jun 8 17:07:03 hydra kernel: [ 2430] 496 2430 2698 2 0 0 0 inet_gethost
|
Jun 8 17:07:03 hydra kernel: [ 2431] 496 2431 4280 2 0 0 0 inet_gethost
|
Jun 8 17:07:03 hydra kernel: [ 2439] 0 2439 29325 76 12 0 0 crond
|
Jun 8 17:07:03 hydra kernel: [ 2450] 0 2450 5385 9 19 0 0 atd
|
Jun 8 17:07:03 hydra kernel: [ 2466] 0 2466 27041 2 3 0 0 sh
|
Jun 8 17:07:03 hydra kernel: [ 2471] 0 2471 8801199 17711 0 0 0 java
|
Jun 8 17:07:03 hydra kernel: [ 2508] 0 2508 23103 349 19 0 0 miniserv.pl
|
Jun 8 17:07:03 hydra kernel: [ 2517] 0 2517 1016 2 2 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2519] 0 2519 1016 2 2 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2521] 0 2521 1016 2 25 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2523] 0 2523 1016 2 2 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2527] 0 2527 1016 2 0 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2529] 0 2529 1016 2 19 0 0 mingetty
|
Jun 8 17:07:03 hydra kernel: [ 2703] 0 2703 8779693 71 0 0 0 java
|
Jun 8 17:07:03 hydra kernel: [ 3414] 0 3414 2779 1 1 -17 -1000 udevd
|
Jun 8 17:07:03 hydra kernel: [ 3415] 0 3415 2779 1 17 -17 -1000 udevd
|
Jun 8 17:07:03 hydra kernel: [81327] 495 81327 7355427 1578291 19 0 0 java
|
Jun 8 17:07:03 hydra kernel: [30759] 0 30759 26549 83 17 0 0 mysqld_safe
|
Jun 8 17:07:03 hydra kernel: [30993] 498 30993 37697878 30935404 0 0 0 mysqld
|
Jun 8 17:07:03 hydra kernel: [31129] 0 31129 35792 475 8 0 0 munin-node
|
Jun 8 17:07:03 hydra kernel: [32419] 0 32419 29810 290 19 0 0 screen
|
Jun 8 17:07:03 hydra kernel: [32420] 0 32420 27109 216 2 0 0 bash
|
Jun 8 17:07:03 hydra kernel: [74824] 0 74824 5545 144 1 0 0 xinetd
|
Jun 8 17:07:03 hydra kernel: [77251] 0 77251 26515 130 2 0 0 memorylog
|
Jun 8 17:07:03 hydra kernel: [77410] 0 77410 1532 92 1 0 0 vmstat
|
Jun 8 17:07:03 hydra kernel: [14379] 0 14379 25227 77 16 0 0 sleep
|
Jun 8 17:07:03 hydra kernel: [14671] 0 14671 35046 193 1 0 0 crond
|
Jun 8 17:07:03 hydra kernel: [14672] 0 14672 35046 193 17 0 0 crond
|
Jun 8 17:07:03 hydra kernel: [14690] 0 14690 26515 135 2 0 0 backup-blieb
|
Jun 8 17:07:03 hydra kernel: [14691] 0 14691 26515 134 19 0 0 bash
|
Jun 8 17:07:03 hydra kernel: [14693] 0 14693 26515 135 18 0 0 updatehits
|
Jun 8 17:07:03 hydra kernel: [14733] 0 14733 56394 379 0 0 0 mysql
|
Jun 8 17:07:03 hydra kernel: [14734] 0 14734 31785 372 21 0 0 mysqldump
|
Jun 8 17:07:03 hydra kernel: [14735] 0 14735 1074 85 1 0 0 gzip
|
Jun 8 17:07:03 hydra kernel: [14788] 0 14788 25356 4879 18 0 0 miniserv.pl
|
Jun 8 17:07:03 hydra kernel: Out of memory: Kill process 30993 (mysqld) score 934 or sacrifice child
|
Jun 8 17:07:03 hydra kernel: Killed process 30993, UID 498, (mysqld) total-vm:150791512kB, anon-rss:123736464kB, file-rss:5148kB
|
The mysql error log doesn't show anything at the moment the process gets killed.
I've attached an image of the munin memory usage graph of the server, where you clearly see the MariaDB reset at 17:07. If I understand correctly, it looks like MariaDB is allocation a lot of memory, while not really using it.
Could this be a bug in MariaDB 10.x? What can I provide to help you find it?
Attachments
Issue Links
- relates to
-
-
- Open
-
-
-
- Open
-