|
Setting to confirmed because the described behavior is reproducible easily. I'm leaving it to psergey to determine whether it's actually a bug.
|
|
Comparing with InnoDB: According to Marko, parts of its buffer pool are allocated in in os_mem_alloc_large()
Putting breakpoints at
- mmap() call in os_mem_alloc_large
- mlockall(MCL_CURRENT) call in init_server_components()
I can see that buffer pool is alloced before the mlockall call, so the buffer
pool (as well as other things InnoDB allocates on startup
|
|
... and a similar thing happens with MyRocks. init_server_components() first calls plugin_init for the plugins (loadable or not) and then makes the mlockall call.
So, a possible explanation could be that MyRocks allocates memory after its initialization?
|
|
Indeed, for the block_cache it seems to be true. I started a server, checked rocksdb_tbl_options->block_cache->GetUsage(), then loaded some data and checked it again, and again after some more:
AAAA block_cache size= 16680 (0 M, 0 G)
|
2018-01-18 13:35:00 140737288869632 [Note] RocksDB: Manual memtable flush.
|
AAAA block_cache size= 5988792 (5 M, 0 G)
|
AAAA block_cache size= 135323088 (129 M, 0 G)
|
The server has
MariaDB [test]> select @@rocksdb_block_cache_size / (1024* 1024*1024.) as G ;
|
+---------+
|
| G |
|
+---------+
|
| 10.0000 |
|
+---------+
|
|
How do I make sure that mariadb locks all memory so it ever gets paged out or killed by the OOM?
So, memlock=1 doesn't give you that, and it's expected behavior. I suspect there might be other reasons making it difficult to pre-alloc all the memory that RocksDB/MyRocks will potentially need.
Do you really experience MyRocks instance getting OOM-killed due to consuming too much memory?
|
|
Yes I do. The OOM killer always kills my mysqld. Also, I use largepages=1, but mariadb never allocates from HugePages_Total as in
cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 3000
HugePages_Free: 2998
HugePages_Rsvd: 1736
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 304576 kB
DirectMap2M: 83580928 kB
I do have the group correctly set in /etc/sysctl.conf. In all fairness, mariadb uses about 200 pages., but not more. The engines seem to ignore Large Pages, altough the OS uses Anon Huge Pages. But I need way to lock all memory in away that the OS may never kill it. It amazes me that the OS never shrink the disk buffers and instead kills mysqld. Any idea how to protect mariadb from being OOM-killed.
This happen this AM,
[4651176.321582] Node 0 hugepages_total=3000 hugepages_free=2998 hugepages_surp=0 hugepages_size=2048kB
[4651176.321582] 13364 total pagecache pages
[4651176.321586] 0 pages in swap cache
[4651176.321586] Swap cache stats: add 0, delete 0, find 0/0
[4651176.321587] Free swap = 5185532kB
[4651176.321587] Total swap = 5185532kB
[4651176.321588] 20971279 pages RAM
[4651176.321588] 0 pages HighMem/MovableOnly
[4651176.321588] 348934 pages reserved
[4651176.321589] 0 pages cma reserved
[4651176.321589] 0 pages hwpoisoned
[4651176.321598] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[4651176.321605] [ 517] 0 517 47187 137 92 3 0 0 systemd-journal
[4651176.321607] [ 549] 0 549 11410 508 23 3 0 -1000 systemd-udevd
[4651176.321609] [ 672] 0 672 13897 107 26 3 0 -1000 auditd
[4651176.321610] [ 694] 0 694 4901 85 13 3 0 0 irqbalance
[4651176.321612] [ 695] 81 695 12139 158 27 3 0 -900 dbus-daemon
[4651176.321613] [ 697] 996 697 26859 98 23 3 0 0 chronyd
[4651176.321614] [ 701] 0 701 6289 174 16 3 0 0 smartd
[4651176.321615] [ 703] 0 703 170080 383 133 4 0 0 rsyslogd
[4651176.321617] [ 704] 0 704 26457 396 40 3 0 0 VGAuthService
[4651176.321618] [ 706] 0 706 111799 337 64 3 0 0 abrtd
[4651176.321619] [ 707] 0 707 114369 517 69 3 0 0 NetworkManager
[4651176.321620] [ 708] 0 708 11385 134 24 3 0 0 systemd-logind
[4651176.321621] [ 710] 0 710 61036 483 60 3 0 0 vmtoolsd
[4651176.321622] [ 711] 0 711 1643 31 8 3 0 0 mcelog
[4651176.321623] [ 712] 0 712 1105 18 6 3 0 0 rngd
[4651176.321624] [ 740] 0 740 51161 123 35 3 0 0 gssproxy
[4651176.321626] [ 761] 0 761 99475 299 78 3 0 0 abrt-dump-journ
[4651176.321627] [ 762] 0 762 158342 301 125 4 0 0 abrt-dump-journ
[4651176.321628] [ 789] 998 789 132860 826 56 4 0 0 polkitd
[4651176.321630] [ 1042] 0 1042 19439 689 35 3 0 0 dhclient
[4651176.321631] [ 1234] 0 1234 33240 156 17 3 0 0 crond
[4651176.321632] [ 1238] 0 1238 6491 48 18 3 0 0 atd
[4651176.321633] [ 1243] 0 1243 29267 29 9 3 0 0 agetty
[4651176.321635] [ 1252] 0 1252 6141 83 16 3 0 0 lxc-autostart
[4651176.321636] [ 1260] 0 1260 10836 321 26 3 0 0 systemd
[4651176.321637] [ 1276] 0 1276 22525 12457 49 3 0 0 systemd-journal
[4651176.321638] [ 1306] 0 1306 10570 137 21 3 0 -1000 systemd-udevd
[4651176.321640] [ 1389] 0 1389 6114 141 15 3 0 0 systemd-logind
[4651176.321641] [ 1392] 0 1392 70485 6092 65 3 0 0 rsyslogd
[4651176.321642] [ 1393] 81 1393 6074 81 16 4 0 -900 dbus-daemon
[4651176.321643] [ 1403] 0 1403 1617 32 8 3 0 0 agetty
[4651176.321645] [ 1404] 0 1404 1617 29 8 3 0 0 agetty
[4651176.321646] [ 1408] 0 1408 1617 32 8 3 0 0 agetty
[4651176.321647] [ 1409] 0 1409 1617 31 7 3 0 0 agetty
[4651176.321648] [ 1410] 0 1410 1617 32 9 3 0 0 agetty
[4651176.321649] [ 1411] 0 1411 1617 31 8 3 0 0 agetty
[4651176.321650] [ 1412] 0 1412 1617 31 8 3 0 0 agetty
[4651176.321651] [ 1413] 0 1413 1617 31 8 3 0 0 agetty
[4651176.321661] [ 1414] 0 1414 1617 33 7 3 0 0 agetty
[4651176.321663] [ 1418] 0 1418 1617 32 8 3 0 0 agetty
[4651176.321664] [ 1419] 0 1419 1617 31 8 3 0 0 agetty
[4651176.321666] [ 2593] 0 2593 26502 245 54 3 0 -1000 sshd
[4651176.321667] [ 2594] 99 2594 3897 48 12 3 0 0 dnsmasq
[4651176.321668] [15799] 32 15799 14370 127 29 3 0 0 rpcbind
[4651176.321670] [20217] 0 20217 5664 153 15 3 0 0 crond
[4651176.321672] [ 4884] 0 4884 25166 235 51 3 0 -1000 sshd
[4651176.321673] [16416] 27 16416 20964326 18752942 38543 92 0 0 mysqld
[4651176.321675] [32474] 0 32474 5432 60 15 3 0 0 anacron
[4651176.321677] [ 662] 0 662 2381 52 9 3 0 0 run-parts
[4651176.321678] [ 666] 0 666 60915 3587 119 3 0 0 yum-cron
[4651176.321679] [ 669] 0 669 2472 23 8 3 0 0 awk
[4651176.321681] [ 1704] 0 1704 17601 224 37 4 0 0 crond
[4651176.321682] [ 1788] 0 1788 28285 58 10 4 0 0 safe_asterisk1
[4651176.321683] [ 1789] 0 1789 822037 8091 345 6 0 0 asterisk
[4651176.321685] [ 1826] 0 1826 28285 58 9 3 0 0 safe_asterisk2
[4651176.321686] [ 1827] 0 1827 822002 8065 347 6 0 0 asterisk
[4651176.321688] [ 1888] 0 1888 28285 57 10 3 0 0 safe_asterisk3
[4651176.321689] [ 1889] 0 1889 822001 8067 346 6 0 0 asterisk
[4651176.321690] [ 1946] 0 1946 28285 58 10 3 0 0 safe_asterisk4
[4651176.321691] [ 1947] 0 1947 822003 8066 345 6 0 0 asterisk
[4651176.321692] [ 2003] 0 2003 28285 57 10 3 0 0 safe_asterisk5
[4651176.321693] [ 2004] 0 2004 822001 8065 347 6 0 0 asterisk
[4651176.321695] [ 2056] 0 2056 28285 58 10 3 0 0 safe_asterisk6
[4651176.321696] [ 2058] 0 2058 822002 8065 347 6 0 0 asterisk
[4651176.321697] [ 2111] 0 2111 28285 57 10 3 0 0 safe_asterisk7
[4651176.321698] [ 2113] 0 2113 822001 8067 348 6 0 0 asterisk
[4651176.321699] [ 2168] 0 2168 28285 58 10 3 0 0 safe_asterisk8
[4651176.321700] [ 2169] 0 2169 822001 8066 346 6 0 0 asterisk
[4651176.321702] [ 2223] 0 2223 28285 57 10 3 0 0 safe_asterisk9
[4651176.321703] [ 2224] 0 2224 822001 8070 346 6 0 0 asterisk
[4651176.321704] [ 2284] 0 2284 28285 57 9 3 0 0 safe_asterisk10
[4651176.321705] [ 2285] 0 2285 822003 8067 345 6 0 0 asterisk
[4651176.321707] [ 2333] 0 2333 28285 57 10 3 0 0 safe_asterisk11
[4651176.321708] [ 2334] 0 2334 822002 8065 348 7 0 0 asterisk
[4651176.321709] [ 2388] 0 2388 28285 58 10 3 0 0 safe_asterisk12
[4651176.321710] [ 2389] 0 2389 822032 8066 346 6 0 0 asterisk
[4651176.321712] [ 2922] 0 2922 6036 116 16 3 0 0 screen
[4651176.321713] [ 2923] 0 2923 2383 55 9 3 0 0 audit.sh
[4651176.321722] [ 3454] 0 3454 2383 54 8 3 0 0 audit.sh
[4651176.321724] [ 3455] 0 3455 14437 327 33 3 0 0 mysql
[4651176.321725] [ 3456] 0 3456 2472 23 9 3 0 0 awk
[4651176.321726] Out of memory: Kill process 16416 (mysqld) score 857 or sacrifice child
[4651176.322094] Killed process 16416 (mysqld) total-vm:83857304kB, anon-rss:75011768kB, file-rss:0kB, shmem-rss:0kB
[4651182.615404] oom_reaper: reaped process 16416 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
after Mariadb restarted
free -g
total used free shared buff/cache available
Mem: 78 17 58 0 2 59
Swap: 4 0 4
This installation is using TokuDB, but I have boxes with RocksDB and Innodb and they all get killed in identical manner.
show variables like '%Toku%';
----------------------------------------
----------------------------------------
| tokudb_alter_print_error |
OFF |
| tokudb_analyze_delete_fraction |
1.000000 |
| tokudb_analyze_in_background |
ON |
| tokudb_analyze_mode |
TOKUDB_ANALYZE_STANDARD |
| tokudb_analyze_throttle |
0 |
| tokudb_analyze_time |
5 |
| tokudb_auto_analyze |
1 |
| tokudb_block_size |
4194304 |
| tokudb_bulk_fetch |
ON |
| tokudb_cache_size |
10737418240 |
| tokudb_cachetable_pool_threads |
0 |
| tokudb_cardinality_scale_percent |
50 |
| tokudb_check_jemalloc |
OFF |
| tokudb_checkpoint_lock |
OFF |
| tokudb_checkpoint_on_flush_logs |
OFF |
| tokudb_checkpoint_pool_threads |
0 |
| tokudb_checkpointing_period |
60 |
| tokudb_cleaner_iterations |
5 |
| tokudb_cleaner_period |
1 |
| tokudb_client_pool_threads |
0 |
| tokudb_commit_sync |
OFF |
| tokudb_compress_buffers_before_eviction |
ON |
| tokudb_create_index_online |
ON |
| tokudb_data_dir |
|
| tokudb_debug |
0 |
| tokudb_dir_per_db |
OFF |
| tokudb_directio |
OFF |
| tokudb_disable_hot_alter |
OFF |
| tokudb_disable_prefetching |
OFF |
| tokudb_disable_slow_alter |
OFF |
| tokudb_empty_scan |
rl |
| tokudb_enable_partial_eviction |
ON |
| tokudb_fanout |
16 |
| tokudb_fs_reserve_percent |
5 |
| tokudb_fsync_log_period |
0 |
| tokudb_hide_default_row_format |
ON |
| tokudb_killed_time |
4000 |
| tokudb_last_lock_timeout |
|
| tokudb_load_save_space |
ON |
| tokudb_loader_memory_size |
100000000 |
| tokudb_lock_timeout |
4000 |
| tokudb_lock_timeout_debug |
1 |
| tokudb_log_dir |
|
| tokudb_max_lock_memory |
1342177280 |
| tokudb_optimize_index_fraction |
1.000000 |
| tokudb_optimize_index_name |
|
| tokudb_optimize_throttle |
0 |
| tokudb_pk_insert_mode |
1 |
| tokudb_prelock_empty |
ON |
| tokudb_read_block_size |
65536 |
| tokudb_read_buf_size |
1048576 |
| tokudb_read_status_frequency |
10000 |
| tokudb_row_format |
tokudb_zlib |
| tokudb_rpl_check_readonly |
ON |
| tokudb_rpl_lookup_rows |
ON |
| tokudb_rpl_lookup_rows_delay |
0 |
| tokudb_rpl_unique_checks |
ON |
| tokudb_rpl_unique_checks_delay |
0 |
| tokudb_strip_frm_data |
OFF |
| tokudb_support_xa |
ON |
| tokudb_tmp_dir |
/tmp |
| tokudb_version |
5.6.37-82.2 |
| tokudb_write_status_frequency |
1000 |
----------------------------------------
63 rows in set (0.00 sec)
|
Yes I do. The OOM killer always kills my mysqld.
...
[4651176.322094] Killed process 16416 (mysqld) total-vm:83857304kB, anon-rss:75011768kB, file-rss:0kB, shmem-rss:0kB
total-vm:83,857,304 kB = about 83G
anon-rss:75,011,768 kB = about 75G
free -g
total used free shared buff/cache available
Mem: 78 17 58 0 2 59
Swap: 4 0 4
total mem + swap 78G= 82 G.
So it looks like mysqld's memory usage was 83G which is more than 82G that was available...
Even if we only look at RSS, which is 75G, this is already fairly close to hitting the limit.
We need to find out why did mysqld use so much memory.
|
Also, I use largepages=1, but mariadb never allocates from HugePages_Total
Checked that with other devs, including wlad. MyRocks doesn't use memory from huge (or large) pages. It is only InnoDB that is able to use it.
|
|
Coming back to the
We need to find out why did mysqld use so much memory.
philip_38, do you have any idea
- How much time mysqld runs before OOM kills it?
- Does mysqld's memory usage spikes suddenly or it grows gradually?
- Is there any specific workload that is run before the mysqld is OOM-killed ?
|
|
It crashes every single day.
I don't know if it spikes or grows slowly
Fiest I download a table with 5 numeric fields, 700 MM records into a CSV text file, then I compare that file against a new one I get from a customer, so I get two files: a delete.sql and an insert.sql file, which have millions of lines each. Then I do
mysql < delete.sql && mysql < insert.sql
Then it crashes.
I can give you root access to the box. I take full responsibility.
The process runs off cron at 6 AM.
Note: the rest of the day MariaDB is idle
|
|
[3472400.202421] mysqld invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[3472400.202425] mysqld cpuset=system55 mems_allowed=0-1
[3472400.202432] CPU: 4 PID: 12739 Comm: mysqld Not tainted 4.12.14-041214-generic #201709200843
[3472400.202433] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1607292324 07/29/2016
[3472400.202435] Call Trace:
[3472400.202447] dump_stack+0x63/0x8d
[3472400.202452] dump_header+0x97/0x21a
[3472400.202459] ? do_try_to_free_pages+0x2c2/0x350
[3472400.202464] ? security_capable_noaudit+0x45/0x60
[3472400.202468] oom_kill_process+0x20b/0x3d0
[3472400.202469] out_of_memory+0x2a9/0x4d0
[3472400.202471] __alloc_pages_slowpath+0xc5b/0xd80
[3472400.202476] ? alloc_pages_current+0x93/0x150
[3472400.202478] __alloc_pages_nodemask+0x23f/0x260
[3472400.202480] alloc_pages_current+0x93/0x150
[3472400.202481] __page_cache_alloc+0xc2/0xe0
[3472400.202483] filemap_fault+0x214/0x5e0
[3472400.202487] ? down_read+0x12/0x40
[3472400.202544] xfs_filemap_fault+0x5b/0xe0 [xfs]
[3472400.202547] __do_fault+0x1e/0xb0
[3472400.202549] __handle_mm_fault+0xb17/0xfa0
[3472400.202551] handle_mm_fault+0xb1/0x250
[3472400.202555] __do_page_fault+0x23f/0x4c0
[3472400.202557] do_page_fault+0x22/0x30
[3472400.202560] page_fault+0x28/0x30
[3472400.202562] RIP: 0033:0x7f31121c33c0
[3472400.202563] RSP: 002b:00007f30fe443098 EFLAGS: 00010286
[3472400.202564] RAX: 00000000ffffff00 RBX: 00000000ffffff00 RCX: 0000000000003930
[3472400.202565] RDX: ffffffffffffffea RSI: 00007f0e273646d6 RDI: 00007f310d2fbed0
[3472400.202566] RBP: 00007f30fe4430e0 R08: 0000000000000014 R09: 00007f310ce9a180
[3472400.202566] R10: 0000000000000002 R11: 00007f31105ff2d0 R12: 00007f30fe443150
[3472400.202567] R13: 00007f30fe443170 R14: 00007f14405b30d0 R15: 00007f144025e0d0
[3472400.202569] Mem-Info:
[3472400.202574] active_anon:32921061 inactive_anon:1325662 isolated_anon:0
[3472400.202574] active_file:50 inactive_file:0 isolated_file:0
[3472400.202574] unevictable:915 dirty:1 writeback:0 unstable:0
[3472400.202574] slab_reclaimable:11320 slab_unreclaimable:16765
[3472400.202574] mapped:25569 shmem:308363 pagetables:70091 bounce:0
[3472400.202574] free:91299 free_pcp:225 free_cma:0
[3472400.202576] Node 0 active_anon:65680024kB inactive_anon:2642420kB active_file:260kB inactive_file:0kB unevictable:3644kB isolated(anon):0kB isolated(file):0kB mapped:43064kB dirty:4kB writeback:0kB shmem:555948kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 37326848kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[3472400.202578] Node 1 active_anon:66004220kB inactive_anon:2660228kB active_file:0kB inactive_file:0kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:59212kB dirty:0kB writeback:0kB shmem:677504kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 48396288kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[3472400.202579] Node 0 DMA free:15856kB min:8kB low:20kB high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202582] lowmem_reserve[]: 0 2946 67913 67913 67913
[3472400.202585] Node 0 DMA32 free:261680kB min:1952kB low:4968kB high:7984kB active_anon:2733400kB inactive_anon:55036kB active_file:48kB inactive_file:88kB unevictable:0kB writepending:0kB present:3128768kB managed:3063200kB mlocked:0kB slab_reclaimable:532kB slab_unreclaimable:500kB kernel_stack:0kB pagetables:9816kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202588] lowmem_reserve[]: 0 0 64966 64966 64966
[3472400.202590] Node 0 Normal free:42716kB min:43060kB low:109584kB high:176108kB active_anon:62946624kB inactive_anon:2587384kB active_file:32kB inactive_file:0kB unevictable:3644kB writepending:4kB present:67633152kB managed:66525768kB mlocked:3644kB slab_reclaimable:19808kB slab_unreclaimable:34100kB kernel_stack:3576kB pagetables:126264kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202593] lowmem_reserve[]: 0 0 0 0 0
[3472400.202595] Node 1 Normal free:44944kB min:45084kB low:114732kB high:184380kB active_anon:66004220kB inactive_anon:2660228kB active_file:0kB inactive_file:48kB unevictable:16kB writepending:0kB present:70778880kB managed:69652412kB mlocked:16kB slab_reclaimable:24940kB slab_unreclaimable:32444kB kernel_stack:2280kB pagetables:144284kB bounce:0kB free_pcp:900kB local_pcp:120kB free_cma:0kB
[3472400.202597] lowmem_reserve[]: 0 0 0 0 0
[3472400.202599] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15856kB
[3472400.202607] Node 0 DMA32: 56*4kB (UME) 38*8kB (UME) 55*16kB (UE) 74*32kB (UME) 78*64kB (UME) 68*128kB (UME) 53*256kB (UE) 331*512kB (ME) 54*1024kB (UME) 3*2048kB (E) 0*4096kB = 261952kB
[3472400.202615] Node 0 Normal: 361*4kB (UME) 270*8kB (UME) 364*16kB (UME) 294*32kB (UME) 198*64kB (UME) 80*128kB (UME) 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB = 43796kB
[3472400.202622] Node 1 Normal: 567*4kB (UME) 325*8kB (UME) 360*16kB (UME) 350*32kB (UME) 129*64kB (UME) 63*128kB (UME) 18*256kB (ME) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 44292kB
[3472400.202630] Node 0 hugepages_total=250 hugepages_free=248 hugepages_surp=0 hugepages_size=2048kB
[3472400.202631] Node 1 hugepages_total=250 hugepages_free=250 hugepages_surp=0 hugepages_size=2048kB
[3472400.202632] 309947 total pagecache pages
[3472400.202633] 257 pages in swap cache
[3472400.202634] Swap cache stats: add 188412, delete 188155, find 173541/174313
[3472400.202634] Free swap = 0kB
[3472400.202634] Total swap = 372732kB
[3472400.202635] 35389199 pages RAM
[3472400.202636] 0 pages HighMem/MovableOnly
[3472400.202636] 574878 pages reserved
[3472400.202636] 0 pages cma reserved
[3472400.202637] 0 pages hwpoisoned
[3472400.202637] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[3472400.202644] [ 547] 0 547 23120 12805 49 3 1 0 systemd-journal
[3472400.202646] [ 566] 0 566 25743 152 16 3 20 0 lvmetad
[3472400.202648] [ 586] 0 586 11463 819 21 3 33 -1000 systemd-udevd
[3472400.202650] [ 1163] 100 1163 25082 357 17 3 42 0 systemd-timesyn
[3472400.202651] [ 1222] 104 1222 64100 518 26 4 80 0 rsyslogd
[3472400.202652] [ 1226] 0 1226 1101 281 7 3 37 0 acpid
[3472400.202653] [ 1228] 0 1228 7165 419 17 3 56 0 systemd-logind
[3472400.202655] [ 1229] 107 1229 10727 467 23 3 68 -900 dbus-daemon
[3472400.202656] [ 1326] 0 1326 30487 732 52 3 2087 0 vmtoolsd
[3472400.202657] [ 1327] 0 1327 114112 413 24 3 30 0 lxcfs
[3472400.202658] [ 1329] 0 1329 6512 240 17 3 45 0 atd
[3472400.202660] [ 1331] 0 1331 68943 183 36 3 1 0 accounts-daemon
[3472400.202661] [ 1337] 0 1337 7253 467 17 3 23 0 cron
[3472400.202663] [ 1393] 0 1393 16914 394 35 3 28 -1000 sshd
[3472400.202664] [ 1409] 0 1409 3345 36 10 3 0 0 mdadm
[3472400.202665] [ 1411] 0 1411 1307 401 8 3 0 0 iscsid
[3472400.202667] [ 1412] 0 1412 1432 917 8 3 0 -17 iscsid
[3472400.202668] [ 1486] 0 1486 3986 352 12 3 0 0 agetty
[3472400.202669] [ 1494] 0 1494 4909 55 13 3 43 0 irqbalance
[3472400.202671] [ 1495] 0 1495 69296 149 37 3 30 0 polkitd
[3472400.202672] [ 1503] 0 1503 6740 401 15 3 1 0 atd
[3472400.202673] [ 1518] 0 1518 13622 445 31 3 120 0 lxc-autostart
[3472400.202677] [26099] 0 26099 216749 4102 56 7 83 -900 snapd
[3472400.202678] [ 3453] 0 3453 6798 223 16 3 101 0 screen
[3472400.202679] [ 3454] 0 3454 3137 443 10 3 0 0 ubupdate.sh
[3472400.202681] [ 3683] 0 3683 70449 713 107 3 21827 0 aptitude
[3472400.202682] [ 4641] 0 4641 5048 375 15 3 589 0 dpkg
[3472400.202683] [ 4718] 0 4718 15442 425 35 3 2921 0 frontend
[3472400.202685] [ 4728] 0 4728 3225 349 11 3 145 0 grub-efi-amd64.
[3472400.202686] [ 4749] 0 4749 3200 345 10 3 121 0 ucf
[3472400.202687] [ 4793] 0 4793 5133 389 14 3 0 0 whiptail
[3472400.202689] [18390] 0 18390 10829 311 24 3 0 0 systemd
[3472400.202691] [18453] 0 18453 45675 30152 92 3 0 0 systemd-journal
[3472400.202692] [18527] 0 18527 10597 353 23 3 0 -1000 systemd-udevd
[3472400.202694] [18571] 81 18571 6105 90 17 3 0 -900 dbus-daemon
[3472400.202695] [18575] 0 18575 6054 88 15 3 0 0 systemd-logind
[3472400.202696] [18587] 0 18587 89726 14913 96 4 0 0 rsyslogd
[3472400.202697] [18601] 0 18601 1617 31 8 3 0 0 agetty
[3472400.202699] [18603] 0 18603 1617 32 8 3 0 0 agetty
[3472400.202700] [18606] 0 18606 1617 32 7 3 0 0 agetty
[3472400.202701] [18608] 0 18608 1617 32 8 3 0 0 agetty
[3472400.202702] [18609] 0 18609 1617 30 8 3 0 0 agetty
[3472400.202704] [18611] 0 18611 1617 33 7 3 0 0 agetty
[3472400.202705] [18613] 0 18613 1617 32 7 3 0 0 agetty
[3472400.202706] [18615] 0 18615 1617 30 8 4 0 0 agetty
[3472400.202707] [18616] 0 18616 1617 32 8 3 0 0 agetty
[3472400.202709] [18619] 0 18619 1617 32 8 3 0 0 agetty
[3472400.202710] [18620] 0 18620 1617 31 8 3 0 0 agetty
[3472400.202711] [18632] 0 18632 5665 157 15 3 0 0 crond
[3472400.202713] [19744] 99 19744 3897 49 12 3 0 0 dnsmasq
[3472400.202714] [19746] 0 19746 26519 247 54 3 0 -1000 sshd
[3472400.202715] [19748] 0 19748 6099 82 14 3 0 0 atd
[3472400.202717] [19929] 996 19929 36583493 33926220 68714 143 0 0 mysqld
[3472400.202718] [28948] 0 28948 26519 260 53 3 0 0 sshd
[3472400.202720] [28953] 0 28953 28883 96 14 3 0 0 bash
[3472400.202721] [ 9693] 0 9693 6036 118 16 3 0 0 screen
[3472400.202723] [ 9694] 0 9694 28314 82 12 3 0 0 mac-neustar.sh
[3472400.202725] [12736] 0 12736 28314 81 9 3 0 0 mac-neustar.sh
[3472400.202726] [12737] 0 12737 14454 293 30 3 0 0 mysql
[3472400.202728] [12738] 0 12738 2472 23 9 3 0 0 awk
[3472400.202729] Out of memory: Kill process 19929 (mysqld) score 973 or sacrifice child
[3472400.202882] Killed process 19929 (mysqld) total-vm:146333972kB, anon-rss:135704880kB, file-rss:0kB, shmem-rss:0kB
[3472404.168210] oom_reaper: reaped process 19929 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[3472400.202421] mysqld invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[3472400.202425] mysqld cpuset=system55 mems_allowed=0-1
[3472400.202432] CPU: 4 PID: 12739 Comm: mysqld Not tainted 4.12.14-041214-generic #201709200843
[3472400.202433] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1607292324 07/29/2016
[3472400.202435] Call Trace:
[3472400.202447] dump_stack+0x63/0x8d
[3472400.202452] dump_header+0x97/0x21a
[3472400.202459] ? do_try_to_free_pages+0x2c2/0x350
[3472400.202464] ? security_capable_noaudit+0x45/0x60
[3472400.202468] oom_kill_process+0x20b/0x3d0
[3472400.202469] out_of_memory+0x2a9/0x4d0
[3472400.202471] __alloc_pages_slowpath+0xc5b/0xd80
[3472400.202476] ? alloc_pages_current+0x93/0x150
[3472400.202478] __alloc_pages_nodemask+0x23f/0x260
[3472400.202480] alloc_pages_current+0x93/0x150
[3472400.202481] __page_cache_alloc+0xc2/0xe0
[3472400.202483] filemap_fault+0x214/0x5e0
[3472400.202487] ? down_read+0x12/0x40
[3472400.202544] xfs_filemap_fault+0x5b/0xe0 [xfs]
[3472400.202547] __do_fault+0x1e/0xb0
[3472400.202549] __handle_mm_fault+0xb17/0xfa0
[3472400.202551] handle_mm_fault+0xb1/0x250
[3472400.202555] __do_page_fault+0x23f/0x4c0
[3472400.202557] do_page_fault+0x22/0x30
[3472400.202560] page_fault+0x28/0x30
[3472400.202562] RIP: 0033:0x7f31121c33c0
[3472400.202563] RSP: 002b:00007f30fe443098 EFLAGS: 00010286
[3472400.202564] RAX: 00000000ffffff00 RBX: 00000000ffffff00 RCX: 0000000000003930
[3472400.202565] RDX: ffffffffffffffea RSI: 00007f0e273646d6 RDI: 00007f310d2fbed0
[3472400.202566] RBP: 00007f30fe4430e0 R08: 0000000000000014 R09: 00007f310ce9a180
[3472400.202566] R10: 0000000000000002 R11: 00007f31105ff2d0 R12: 00007f30fe443150
[3472400.202567] R13: 00007f30fe443170 R14: 00007f14405b30d0 R15: 00007f144025e0d0
[3472400.202569] Mem-Info:
[3472400.202574] active_anon:32921061 inactive_anon:1325662 isolated_anon:0
[3472400.202574] active_file:50 inactive_file:0 isolated_file:0
[3472400.202574] unevictable:915 dirty:1 writeback:0 unstable:0
[3472400.202574] slab_reclaimable:11320 slab_unreclaimable:16765
[3472400.202574] mapped:25569 shmem:308363 pagetables:70091 bounce:0
[3472400.202574] free:91299 free_pcp:225 free_cma:0
[3472400.202576] Node 0 active_anon:65680024kB inactive_anon:2642420kB active_file:260kB inactive_file:0kB unevictable:3644kB isolated(anon):0kB isolated(file):0kB mapped:43064kB dirty:4kB writeback:0kB shmem:555948kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 37326848kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[3472400.202578] Node 1 active_anon:66004220kB inactive_anon:2660228kB active_file:0kB inactive_file:0kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:59212kB dirty:0kB writeback:0kB shmem:677504kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 48396288kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[3472400.202579] Node 0 DMA free:15856kB min:8kB low:20kB high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202582] lowmem_reserve[]: 0 2946 67913 67913 67913
[3472400.202585] Node 0 DMA32 free:261680kB min:1952kB low:4968kB high:7984kB active_anon:2733400kB inactive_anon:55036kB active_file:48kB inactive_file:88kB unevictable:0kB writepending:0kB present:3128768kB managed:3063200kB mlocked:0kB slab_reclaimable:532kB slab_unreclaimable:500kB kernel_stack:0kB pagetables:9816kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202588] lowmem_reserve[]: 0 0 64966 64966 64966
[3472400.202590] Node 0 Normal free:42716kB min:43060kB low:109584kB high:176108kB active_anon:62946624kB inactive_anon:2587384kB active_file:32kB inactive_file:0kB unevictable:3644kB writepending:4kB present:67633152kB managed:66525768kB mlocked:3644kB slab_reclaimable:19808kB slab_unreclaimable:34100kB kernel_stack:3576kB pagetables:126264kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[3472400.202593] lowmem_reserve[]: 0 0 0 0 0
[3472400.202595] Node 1 Normal free:44944kB min:45084kB low:114732kB high:184380kB active_anon:66004220kB inactive_anon:2660228kB active_file:0kB inactive_file:48kB unevictable:16kB writepending:0kB present:70778880kB managed:69652412kB mlocked:16kB slab_reclaimable:24940kB slab_unreclaimable:32444kB kernel_stack:2280kB pagetables:144284kB bounce:0kB free_pcp:900kB local_pcp:120kB free_cma:0kB
[3472400.202597] lowmem_reserve[]: 0 0 0 0 0
[3472400.202599] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15856kB
[3472400.202607] Node 0 DMA32: 56*4kB (UME) 38*8kB (UME) 55*16kB (UE) 74*32kB (UME) 78*64kB (UME) 68*128kB (UME) 53*256kB (UE) 331*512kB (ME) 54*1024kB (UME) 3*2048kB (E) 0*4096kB = 261952kB
[3472400.202615] Node 0 Normal: 361*4kB (UME) 270*8kB (UME) 364*16kB (UME) 294*32kB (UME) 198*64kB (UME) 80*128kB (UME) 0*256kB 0*512kB 0*1024kB 1*2048kB (M) 0*4096kB = 43796kB
[3472400.202622] Node 1 Normal: 567*4kB (UME) 325*8kB (UME) 360*16kB (UME) 350*32kB (UME) 129*64kB (UME) 63*128kB (UME) 18*256kB (ME) 1*512kB (M) 1*1024kB (M) 0*2048kB 0*4096kB = 44292kB
[3472400.202630] Node 0 hugepages_total=250 hugepages_free=248 hugepages_surp=0 hugepages_size=2048kB
[3472400.202631] Node 1 hugepages_total=250 hugepages_free=250 hugepages_surp=0 hugepages_size=2048kB
[3472400.202632] 309947 total pagecache pages
[3472400.202633] 257 pages in swap cache
[3472400.202634] Swap cache stats: add 188412, delete 188155, find 173541/174313
[3472400.202634] Free swap = 0kB
[3472400.202634] Total swap = 372732kB
[3472400.202635] 35389199 pages RAM
[3472400.202636] 0 pages HighMem/MovableOnly
[3472400.202636] 574878 pages reserved
[3472400.202636] 0 pages cma reserved
[3472400.202637] 0 pages hwpoisoned
[3472400.202637] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[3472400.202644] [ 547] 0 547 23120 12805 49 3 1 0 systemd-journal
[3472400.202646] [ 566] 0 566 25743 152 16 3 20 0 lvmetad
[3472400.202648] [ 586] 0 586 11463 819 21 3 33 -1000 systemd-udevd
[3472400.202650] [ 1163] 100 1163 25082 357 17 3 42 0 systemd-timesyn
[3472400.202651] [ 1222] 104 1222 64100 518 26 4 80 0 rsyslogd
[3472400.202652] [ 1226] 0 1226 1101 281 7 3 37 0 acpid
[3472400.202653] [ 1228] 0 1228 7165 419 17 3 56 0 systemd-logind
[3472400.202655] [ 1229] 107 1229 10727 467 23 3 68 -900 dbus-daemon
[3472400.202656] [ 1326] 0 1326 30487 732 52 3 2087 0 vmtoolsd
[3472400.202657] [ 1327] 0 1327 114112 413 24 3 30 0 lxcfs
[3472400.202658] [ 1329] 0 1329 6512 240 17 3 45 0 atd
[3472400.202660] [ 1331] 0 1331 68943 183 36 3 1 0 accounts-daemon
[3472400.202661] [ 1337] 0 1337 7253 467 17 3 23 0 cron
[3472400.202663] [ 1393] 0 1393 16914 394 35 3 28 -1000 sshd
[3472400.202664] [ 1409] 0 1409 3345 36 10 3 0 0 mdadm
[3472400.202665] [ 1411] 0 1411 1307 401 8 3 0 0 iscsid
[3472400.202667] [ 1412] 0 1412 1432 917 8 3 0 -17 iscsid
[3472400.202668] [ 1486] 0 1486 3986 352 12 3 0 0 agetty
[3472400.202669] [ 1494] 0 1494 4909 55 13 3 43 0 irqbalance
[3472400.202671] [ 1495] 0 1495 69296 149 37 3 30 0 polkitd
[3472400.202672] [ 1503] 0 1503 6740 401 15 3 1 0 atd
[3472400.202673] [ 1518] 0 1518 13622 445 31 3 120 0 lxc-autostart
[3472400.202677] [26099] 0 26099 216749 4102 56 7 83 -900 snapd
[3472400.202678] [ 3453] 0 3453 6798 223 16 3 101 0 screen
[3472400.202679] [ 3454] 0 3454 3137 443 10 3 0 0 ubupdate.sh
[3472400.202681] [ 3683] 0 3683 70449 713 107 3 21827 0 aptitude
[3472400.202682] [ 4641] 0 4641 5048 375 15 3 589 0 dpkg
[3472400.202683] [ 4718] 0 4718 15442 425 35 3 2921 0 frontend
[3472400.202685] [ 4728] 0 4728 3225 349 11 3 145 0 grub-efi-amd64.
[3472400.202686] [ 4749] 0 4749 3200 345 10 3 121 0 ucf
[3472400.202687] [ 4793] 0 4793 5133 389 14 3 0 0 whiptail
[3472400.202689] [18390] 0 18390 10829 311 24 3 0 0 systemd
[3472400.202691] [18453] 0 18453 45675 30152 92 3 0 0 systemd-journal
[3472400.202692] [18527] 0 18527 10597 353 23 3 0 -1000 systemd-udevd
[3472400.202694] [18571] 81 18571 6105 90 17 3 0 -900 dbus-daemon
[3472400.202695] [18575] 0 18575 6054 88 15 3 0 0 systemd-logind
[3472400.202696] [18587] 0 18587 89726 14913 96 4 0 0 rsyslogd
[3472400.202697] [18601] 0 18601 1617 31 8 3 0 0 agetty
[3472400.202699] [18603] 0 18603 1617 32 8 3 0 0 agetty
[3472400.202700] [18606] 0 18606 1617 32 7 3 0 0 agetty
[3472400.202701] [18608] 0 18608 1617 32 8 3 0 0 agetty
[3472400.202702] [18609] 0 18609 1617 30 8 3 0 0 agetty
[3472400.202704] [18611] 0 18611 1617 33 7 3 0 0 agetty
[3472400.202705] [18613] 0 18613 1617 32 7 3 0 0 agetty
[3472400.202706] [18615] 0 18615 1617 30 8 4 0 0 agetty
[3472400.202707] [18616] 0 18616 1617 32 8 3 0 0 agetty
[3472400.202709] [18619] 0 18619 1617 32 8 3 0 0 agetty
[3472400.202710] [18620] 0 18620 1617 31 8 3 0 0 agetty
[3472400.202711] [18632] 0 18632 5665 157 15 3 0 0 crond
[3472400.202713] [19744] 99 19744 3897 49 12 3 0 0 dnsmasq
[3472400.202714] [19746] 0 19746 26519 247 54 3 0 -1000 sshd
[3472400.202715] [19748] 0 19748 6099 82 14 3 0 0 atd
[3472400.202717] [19929] 996 19929 36583493 33926220 68714 143 0 0 mysqld
[3472400.202718] [28948] 0 28948 26519 260 53 3 0 0 sshd
[3472400.202720] [28953] 0 28953 28883 96 14 3 0 0 bash
[3472400.202721] [ 9693] 0 9693 6036 118 16 3 0 0 screen
[3472400.202723] [ 9694] 0 9694 28314 82 12 3 0 0 mac-neustar.sh
[3472400.202725] [12736] 0 12736 28314 81 9 3 0 0 mac-neustar.sh
[3472400.202726] [12737] 0 12737 14454 293 30 3 0 0 mysql
[3472400.202728] [12738] 0 12738 2472 23 9 3 0 0 awk
[3472400.202729] Out of memory: Kill process 19929 (mysqld) score 973 or sacrifice child
[3472400.202882] Killed process 19929 (mysqld) total-vm:146333972kB, anon-rss:135704880kB, file-rss:0kB, shmem-rss:0kB
[3472404.168210] oom_reaper: reaped process 19929 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
|
|
The machine has 132G of RAM, and MariaDB has 34G assigned to RocksDB. Innodb is not enabled. Yet it OOM-crashes every single day. Something is not right.
The stats below are after I re-run the insertion and deletion post-crash
total used free shared buff/cache available
Mem: 132 37 52 1 43 93
Swap: 0 0 0
|
|
[3731920.551600] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15856kB
[3731920.551608] Node 0 DMA32: 48*4kB (UE) 47*8kB (UE) 73*16kB (UME) 73*32kB (UME) 44*64kB (UME) 268*128kB (UME) 345*256kB (UME) 143*512kB (UME) 48*1024kB (UME) 5*2048kB (ME) 0*4096kB = 262120kB
[3731920.551616] Node 0 Normal: 396*4kB (UME) 162*8kB (UME) 146*16kB (UME) 131*32kB (UME) 102*64kB (UME) 91*128kB (UME) 39*256kB (UME) 11*512kB (ME) 1*1024kB (M) 0*2048kB 0*4096kB = 44224kB
[3731920.551623] Node 1 Normal: 228*4kB (UME) 129*8kB (UME) 129*16kB (UME) 83*32kB (UME) 46*64kB (UME) 43*128kB (UME) 52*256kB (UME) 15*512kB (UE) 9*1024kB (ME) 0*2048kB 0*4096kB = 45320kB
[3731920.551632] Node 0 hugepages_total=250 hugepages_free=250 hugepages_surp=0 hugepages_size=2048kB
[3731920.551633] Node 1 hugepages_total=250 hugepages_free=248 hugepages_surp=0 hugepages_size=2048kB
[3731920.551634] 366214 total pagecache pages
[3731920.551635] 1354 pages in swap cache
[3731920.551641] Swap cache stats: add 189550, delete 188196, find 238093/238867
[3731920.551642] Free swap = 0kB
[3731920.551642] Total swap = 372732kB
[3731920.551643] 35389199 pages RAM
[3731920.551643] 0 pages HighMem/MovableOnly
[3731920.551644] 574878 pages reserved
[3731920.551644] 0 pages cma reserved
[3731920.551644] 0 pages hwpoisoned
[3731920.551645] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[3731920.551652] [ 547] 0 547 31335 18891 66 3 1 0 systemd-journal
[3731920.551653] [ 566] 0 566 25743 38 16 3 20 0 lvmetad
[3731920.551655] [ 586] 0 586 11463 231 21 3 481 -1000 systemd-udevd
[3731920.551657] [ 1163] 100 1163 25082 244 17 3 42 0 systemd-timesyn
[3731920.551658] [ 1222] 104 1222 64100 703 26 4 77 0 rsyslogd
[3731920.551659] [ 1226] 0 1226 1101 281 7 3 37 0 acpid
[3731920.551661] [ 1228] 0 1228 7165 353 17 3 55 0 systemd-logind
[3731920.551662] [ 1229] 107 1229 10727 107 23 3 68 -900 dbus-daemon
[3731920.551663] [ 1326] 0 1326 30674 994 52 3 2207 0 vmtoolsd
[3731920.551665] [ 1327] 0 1327 114112 414 24 3 30 0 lxcfs
[3731920.551666] [ 1329] 0 1329 6512 230 17 3 45 0 atd
[3731920.551668] [ 1331] 0 1331 68943 183 36 3 1 0 accounts-daemon
[3731920.551669] [ 1337] 0 1337 7253 459 17 3 23 0 cron
[3731920.551670] [ 1393] 0 1393 16914 594 35 3 25 -1000 sshd
[3731920.551672] [ 1409] 0 1409 3345 275 10 3 0 0 mdadm
[3731920.551673] [ 1411] 0 1411 1307 401 8 3 0 0 iscsid
[3731920.551675] [ 1412] 0 1412 1432 917 8 3 0 -17 iscsid
[3731920.551676] [ 1486] 0 1486 3986 352 12 3 0 0 agetty
[3731920.551677] [ 1494] 0 1494 4909 264 13 3 43 0 irqbalance
[3731920.551679] [ 1495] 0 1495 69296 149 37 3 30 0 polkitd
[3731920.551680] [ 1503] 0 1503 6740 480 15 3 1 0 atd
[3731920.551681] [ 1518] 0 1518 13622 423 31 3 142 0 lxc-autostart
[3731920.551684] [26099] 0 26099 218798 4115 57 7 84 -900 snapd
[3731920.551685] [ 3453] 0 3453 6798 127 16 3 101 0 screen
[3731920.551687] [ 3454] 0 3454 3137 443 10 3 0 0 ubupdate.sh
[3731920.551688] [ 3683] 0 3683 70449 851 107 3 21827 0 aptitude
[3731920.551690] [ 4641] 0 4641 5048 375 15 3 589 0 dpkg
[3731920.551691] [ 4718] 0 4718 15442 425 35 3 2921 0 frontend
[3731920.551692] [ 4728] 0 4728 3225 349 11 3 145 0 grub-efi-amd64.
[3731920.551694] [ 4749] 0 4749 3200 345 10 3 121 0 ucf
[3731920.551695] [ 4793] 0 4793 5133 389 14 3 0 0 whiptail
[3731920.551696] [18390] 0 18390 10935 405 23 3 0 0 systemd
[3731920.551698] [18453] 0 18453 27575 16480 59 3 3 0 systemd-journal
[3731920.551699] [18571] 81 18571 6131 93 18 3 22 -900 dbus-daemon
[3731920.551701] [18575] 0 18575 6307 319 15 3 25 0 systemd-logind
[3731920.551702] [18587] 0 18587 95700 8692 92 4 4 0 rsyslogd
[3731920.551703] [18601] 0 18601 1617 3 8 3 28 0 agetty
[3731920.551704] [18603] 0 18603 1617 2 8 3 30 0 agetty
[3731920.551705] [18606] 0 18606 1617 32 7 3 0 0 agetty
[3731920.551707] [18608] 0 18608 1617 32 8 3 0 0 agetty
[3731920.551708] [18609] 0 18609 1617 5 8 3 25 0 agetty
[3731920.551709] [18611] 0 18611 1617 3 7 3 30 0 agetty
[3731920.551711] [18613] 0 18613 1617 5 7 3 27 0 agetty
[3731920.551712] [18615] 0 18615 1617 5 8 4 25 0 agetty
[3731920.551713] [18616] 0 18616 1617 3 8 3 29 0 agetty
[3731920.551714] [18619] 0 18619 1617 4 8 3 28 0 agetty
[3731920.551716] [18620] 0 18620 1617 3 8 3 28 0 agetty
[3731920.551717] [18632] 0 18632 5665 156 15 3 0 0 crond
[3731920.551718] [19744] 99 19744 3897 22 12 3 27 0 dnsmasq
[3731920.551720] [19746] 0 19746 26519 34 54 3 212 -1000 sshd
[3731920.551721] [19748] 0 19748 7148 98 17 3 0 0 atd
[3731920.551722] [13109] 996 13109 36925583 33864409 69037 147 0 0 mysqld
[3731920.551724] [24272] 0 24272 26519 259 52 3 0 0 sshd
[3731920.551725] [24295] 0 24295 28883 96 14 3 0 0 bash
[3731920.551727] [30723] 0 30723 10356 100 21 3 0 -1000 systemd-udevd
[3731920.551728] [19626] 0 19626 5432 60 15 3 0 0 anacron
[3731920.551730] [19795] 0 19795 2381 52 9 3 0 0 run-parts
[3731920.551731] [19799] 0 19799 60935 3589 120 3 0 0 yum-cron
[3731920.551733] [19800] 0 19800 2472 22 9 3 0 0 awk
[3731920.551734] [22535] 0 22535 6036 117 16 3 0 0 screen
[3731920.551736] [22537] 0 22537 28314 82 12 3 0 0 mac-neustar.sh
[3731920.551738] [25192] 0 25192 28314 81 9 3 0 0 mac-neustar.sh
[3731920.551740] [25193] 0 25193 14454 297 31 3 0 0 mysql
[3731920.551741] [25194] 0 25194 2472 22 9 3 0 0 awk
[3731920.551743] Out of memory: Kill process 13109 (mysqld) score 972 or sacrifice child
[3731920.551895] Killed process 13109 (mysqld) total-vm:147702332kB, anon-rss:135457636kB, file-rss:0kB, shmem-rss:0kB
[3731923.214879] oom_reaper: reaped process 13109 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
|
|
Below is the full sequence of events before the OOM killer.
Now:
free -g
total used free shared buff/cache available
Mem: 132 1 129 1 1 128
Swap: 0 0 0
show engines;
-------------------
| Engine |
Support |
Comment |
Transactions |
XA |
Savepoints |
-------------------
| ROCKSDB |
DEFAULT |
RocksDB storage engine |
YES |
YES |
YES |
| CSV |
YES |
CSV storage engine |
NO |
NO |
NO |
| MRG_MyISAM |
YES |
Collection of identical MyISAM tables |
NO |
NO |
NO |
| MyISAM |
YES |
MyISAM storage engine |
NO |
NO |
NO |
| SEQUENCE |
YES |
Generated tables filled with sequential values |
YES |
NO |
YES |
| PERFORMANCE_SCHEMA |
YES |
Performance Schema |
NO |
NO |
NO |
| MEMORY |
YES |
Hash based, stored in memory, useful for temporary tables |
NO |
NO |
NO |
| Aria |
YES |
Crash-safe tables with MyISAM heritage |
NO |
NO |
NO |
| InnoDB |
NO |
Supports transactions, row-level locking, foreign keys and encryption for tables |
NULL |
NULL |
NULL |
-------------------
show variables like '%buffer%';
-----------------------------
-----------------------------
| aria_pagecache_buffer_size |
134217728 |
| aria_sort_buffer_size |
268434432 |
| bulk_insert_buffer_size |
8388608 |
| join_buffer_size |
8388608 |
| join_buffer_space_limit |
2097152 |
| key_buffer_size |
134217728 |
| mrr_buffer_size |
262144 |
| myisam_sort_buffer_size |
134216704 |
| net_buffer_length |
16384 |
| preload_buffer_size |
32768 |
| read_buffer_size |
131072 |
| read_rnd_buffer_size |
262144 |
| rocksdb_db_write_buffer_size |
0 |
| sort_buffer_size |
2097152 |
| sql_buffer_result |
OFF |
-----------------------------
from /etc/my.cnf
rocksdb_commit_in_the_middle=1
rocksdb_use_direct_io_for_flush_and_compaction=0
rocksdb_use_direct_reads=0
rocksdb_max_open_files=-1
rocksdb_allow_concurrent_memtable_write=1
rocksdb_allow_mmap_reads=1
rocksdb_allow_mmap_writes=1
rocksdb_blind_delete_primary_key=1
rocksdb_tmpdir=/tmp
rocksdb_bulk_load_size=1000
rocksdb_max_open_files=-1
rocksdb_max_total_wal_size=4G
rocksdb_block_size=16384
rocksdb_block_cache_size=14G
rocksdb_table_cache_numshardbits=6
rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=64m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=256;level0_st$
rocksdb_override_cf_options=system=
{memtable=skip_list:16}
rocksdb_commit_in_the_middle=1
rocksdb_write_disable_wal=1
rocksdb_max_total_wal_size=4G
rocksdb_table_cache_numshardbits=6
rocksdb_flush_log_at_trx_commit=0
rocksdb_strict_collation_check=off
ignore_db_dirs=.rocksdb
[4336811.878649] vmtoolsd invoked oom-killer: gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=(null), order=0, oom_score_adj=0
[4336811.878653] vmtoolsd cpuset=/ mems_allowed=0-1
[4336811.878660] CPU: 0 PID: 1326 Comm: vmtoolsd Not tainted 4.12.14-041214-generic #201709200843
[4336811.878661] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1607292324 07/29/2016
[4336811.878662] Call Trace:
[4336811.878675] dump_stack+0x63/0x8d
[4336811.878680] dump_header+0x97/0x21a
[4336811.878687] ? do_try_to_free_pages+0x2c2/0x350
[4336811.878692] ? security_capable_noaudit+0x45/0x60
[4336811.878695] oom_kill_process+0x20b/0x3d0
[4336811.878697] out_of_memory+0x2a9/0x4d0
[4336811.878698] __alloc_pages_slowpath+0xc5b/0xd80
[4336811.878704] ? alloc_pages_current+0x93/0x150
[4336811.878705] __alloc_pages_nodemask+0x23f/0x260
[4336811.878707] alloc_pages_current+0x93/0x150
[4336811.878709] __page_cache_alloc+0xc2/0xe0
[4336811.878710] filemap_fault+0x214/0x5e0
[4336811.878712] ? filemap_map_pages+0x179/0x320
[4336811.878717] ext4_filemap_fault+0x31/0x50
[4336811.878721] __do_fault+0x1e/0xb0
[4336811.878722] __handle_mm_fault+0xb17/0xfa0
[4336811.878724] handle_mm_fault+0xb1/0x250
[4336811.878730] __do_page_fault+0x23f/0x4c0
[4336811.878732] do_page_fault+0x22/0x30
[4336811.878737] page_fault+0x28/0x30
[4336811.878739] RIP: 0033:0x7fbc1579a7b0
[4336811.878739] RSP: 002b:00007ffe69eb4908 EFLAGS: 00010202
[4336811.878741] RAX: 0000000000000001 RBX: 000000000128d900 RCX: 0000000000000001
[4336811.878741] RDX: 0000000000000064 RSI: 0000000000000001 RDI: 000000000128d900
[4336811.878742] RBP: 000000007fffffff R08: 0000000000000005 R09: 0000000000000001
[4336811.878742] R10: 00007ffe69eb48d0 R11: 0000000000000001 R12: 00007ffe69eb4930
[4336811.878743] R13: 00007ffe69eb4928 R14: 0000000000000000 R15: 00007fbc15768700
[4336811.878745] Mem-Info:
[4336811.878750] active_anon:32905971 inactive_anon:1335875 isolated_anon:0
[4336811.878750] active_file:0 inactive_file:0 isolated_file:0
[4336811.878750] unevictable:915 dirty:0 writeback:0 unstable:0
[4336811.878750] slab_reclaimable:12956 slab_unreclaimable:17070
[4336811.878750] mapped:6441 shmem:403028 pagetables:70247 bounce:0
[4336811.878750] free:91326 free_pcp:138 free_cma:0
[4336811.878752] Node 0 active_anon:65693124kB inactive_anon:2649700kB active_file:24kB inactive_file:0kB unevictable:3644kB isolated(anon):0kB isolated(file):0kB mapped:18628kB dirty:0kB writeback:0kB shmem:576040kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 54325248kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[4336811.878755] Node 1 active_anon:65930760kB inactive_anon:2693800kB active_file:0kB inactive_file:0kB unevictable:16kB isolated(anon):0kB isolated(file):0kB mapped:7136kB dirty:0kB writeback:0kB shmem:1036072kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 27015168kB writeback_tmp:0kB unstable:0kB all_unreclaimable? yes
[4336811.878755] Node 0 DMA free:15856kB min:8kB low:20kB high:32kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[4336811.878758] lowmem_reserve[]: 0 2946 67913 67913 67913
[4336811.878761] Node 0 DMA32 free:261780kB min:1952kB low:4968kB high:7984kB active_anon:2754868kB inactive_anon:35900kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3128768kB managed:3063200kB mlocked:0kB slab_reclaimable:212kB slab_unreclaimable:116kB kernel_stack:0kB pagetables:8844kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[4336811.878764] lowmem_reserve[]: 0 0 64966 64966 64966
[4336811.878766] Node 0 Normal free:42772kB min:43060kB low:109584kB high:176108kB active_anon:62938148kB inactive_anon:2613800kB active_file:24kB inactive_file:196kB unevictable:3644kB writepending:0kB present:67633152kB managed:66525768kB mlocked:3644kB slab_reclaimable:18688kB slab_unreclaimable:29352kB kernel_stack:3384kB pagetables:117284kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[4336811.878770] lowmem_reserve[]: 0 0 0 0 0
[4336811.878772] Node 1 Normal free:44896kB min:45084kB low:114732kB high:184380kB active_anon:65930772kB inactive_anon:2693800kB active_file:628kB inactive_file:704kB unevictable:16kB writepending:0kB present:70778880kB managed:69652412kB mlocked:16kB slab_reclaimable:32924kB slab_unreclaimable:38796kB kernel_stack:2792kB pagetables:154860kB bounce:0kB free_pcp:552kB local_pcp:276kB free_cma:0kB
[4336811.878774] lowmem_reserve[]: 0 0 0 0 0
[4336811.878776] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15856kB
[4336811.878784] Node 0 DMA32: 2465*4kB (UME) 3571*8kB (UME) 2721*16kB (UME) 1659*32kB (UME) 808*64kB (UME) 312*128kB (UME) 36*256kB (ME) 21*512kB (UME) 11*1024kB (E) 2*2048kB (M) 0*4096kB = 262028kB
[4336811.878791] Node 0 Normal: 333*4kB (UME) 167*8kB (UE) 132*16kB (UME) 119*32kB (UME) 64*64kB (UE) 46*128kB (UME) 28*256kB (UE) 15*512kB (UME) 8*1024kB (ME) 1*2048kB (M) 0*4096kB = 43660kB
[4336811.878799] Node 1 Normal: 234*4kB (UME) 221*8kB (UME) 420*16kB (UME) 492*32kB (UME) 221*64kB (UME) 48*128kB (UME) 3*256kB (ME) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 46224kB
[4336811.878807] Node 0 hugepages_total=250 hugepages_free=249 hugepages_surp=0 hugepages_size=2048kB
[4336811.878808] Node 1 hugepages_total=250 hugepages_free=249 hugepages_surp=0 hugepages_size=2048kB
[4336811.878808] 404496 total pagecache pages
[4336811.878810] 228 pages in swap cache
[4336811.878810] Swap cache stats: add 189600, delete 189372, find 378743/379541
[4336811.878811] Free swap = 0kB
[4336811.878811] Total swap = 372732kB
[4336811.878812] 35389199 pages RAM
[4336811.878813] 0 pages HighMem/MovableOnly
[4336811.878813] 574878 pages reserved
[4336811.878813] 0 pages cma reserved
[4336811.878814] 0 pages hwpoisoned
[4336811.878814] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[4336811.878821] [ 547] 0 547 10868 3545 26 3 1 0 systemd-journal
[4336811.878823] [ 566] 0 566 25743 38 16 3 20 0 lvmetad
[4336811.878825] [ 586] 0 586 11463 271 21 3 486 -1000 systemd-udevd
[4336811.878827] [ 1163] 100 1163 25082 286 17 3 36 0 systemd-timesyn
[4336811.878828] [ 1222] 104 1222 64100 539 26 4 77 0 rsyslogd
[4336811.878829] [ 1226] 0 1226 1101 281 7 3 37 0 acpid
[4336811.878831] [ 1228] 0 1228 7165 331 17 3 55 0 systemd-logind
[4336811.878832] [ 1229] 107 1229 10727 95 23 3 68 -900 dbus-daemon
[4336811.878834] [ 1326] 0 1326 31079 1539 53 3 2207 0 vmtoolsd
[4336811.878835] [ 1327] 0 1327 114112 423 24 3 29 0 lxcfs
[4336811.878836] [ 1329] 0 1329 6512 220 17 3 45 0 atd
[4336811.878837] [ 1331] 0 1331 68943 184 36 3 0 0 accounts-daemon
[4336811.878839] [ 1337] 0 1337 7253 469 17 3 23 0 cron
[4336811.878840] [ 1393] 0 1393 16914 594 35 3 25 -1000 sshd
[4336811.878842] [ 1409] 0 1409 3345 36 10 3 0 0 mdadm
[4336811.878843] [ 1411] 0 1411 1307 401 8 3 0 0 iscsid
[4336811.878844] [ 1412] 0 1412 1432 917 8 3 0 -17 iscsid
[4336811.878846] [ 1486] 0 1486 3986 352 12 3 0 0 agetty
[4336811.878847] [ 1494] 0 1494 4909 55 13 3 43 0 irqbalance
[4336811.878848] [ 1495] 0 1495 69296 149 37 3 30 0 polkitd
[4336811.878849] [ 1503] 0 1503 6740 478 15 3 1 0 atd
[4336811.878851] [ 1518] 0 1518 13622 403 31 3 162 0 lxc-autostart
[4336811.878854] [26099] 0 26099 218798 4120 57 7 81 -900 snapd
[4336811.878855] [ 3453] 0 3453 6798 399 16 3 101 0 screen
[4336811.878856] [ 3454] 0 3454 3137 443 10 3 0 0 ubupdate.sh
[4336811.878858] [ 3683] 0 3683 70449 953 107 3 21827 0 aptitude
[4336811.878859] [ 4641] 0 4641 5048 375 15 3 589 0 dpkg
[4336811.878861] [ 4718] 0 4718 15442 425 35 3 2921 0 frontend
[4336811.878862] [ 4728] 0 4728 3225 349 11 3 145 0 grub-efi-amd64.
[4336811.878863] [ 4749] 0 4749 3200 345 10 3 121 0 ucf
[4336811.878864] [ 4793] 0 4793 5133 389 14 3 0 0 whiptail
[4336811.878866] [18390] 0 18390 10968 439 26 3 0 0 systemd
[4336811.878867] [18453] 0 18453 11259 3421 27 3 3 0 systemd-journal
[4336811.878868] [18571] 81 18571 6131 100 18 3 22 -900 dbus-daemon
[4336811.878870] [18575] 0 18575 6557 568 16 3 25 0 systemd-logind
[4336811.878871] [18587] 0 18587 97898 2758 74 4 3 0 rsyslogd
[4336811.878872] [18601] 0 18601 1617 3 8 3 28 0 agetty
[4336811.878873] [18603] 0 18603 1617 2 8 3 30 0 agetty
[4336811.878874] [18606] 0 18606 1617 32 7 3 0 0 agetty
[4336811.878876] [18608] 0 18608 1617 32 8 3 0 0 agetty
[4336811.878877] [18609] 0 18609 1617 5 8 3 25 0 agetty
[4336811.878878] [18611] 0 18611 1617 3 7 3 30 0 agetty
[4336811.878879] [18613] 0 18613 1617 5 7 3 27 0 agetty
[4336811.878880] [18615] 0 18615 1617 5 8 4 25 0 agetty
[4336811.878881] [18616] 0 18616 1617 3 8 3 29 0 agetty
[4336811.878882] [18619] 0 18619 1617 4 8 3 28 0 agetty
[4336811.878883] [18620] 0 18620 1617 3 8 3 28 0 agetty
[4336811.878885] [18632] 0 18632 5665 157 15 3 0 0 crond
[4336811.878886] [19744] 99 19744 3897 24 12 3 26 0 dnsmasq
[4336811.878888] [19746] 0 19746 26519 34 54 3 212 -1000 sshd
[4336811.878889] [19748] 0 19748 7148 104 17 3 0 0 atd
[4336811.878890] [13106] 996 13106 36598555 33823105 68776 146 0 0 mysqld
[4336811.878892] [19881] 0 19881 26519 258 54 3 0 0 sshd
[4336811.878893] [19884] 0 19884 28883 124 14 3 0 0 bash
[4336811.878895] [19900] 0 19900 10356 101 22 3 0 -1000 systemd-udevd
[4336811.878896] [ 2364] 0 2364 5432 61 15 4 0 0 anacron
[4336811.878897] [ 2669] 0 2669 2381 53 9 3 0 0 run-parts
[4336811.878899] [ 2673] 0 2673 60935 3591 121 4 0 0 yum-cron
[4336811.878900] [ 2674] 0 2674 2472 23 9 3 0 0 awk
[4336811.878901] [ 3495] 0 3495 6036 116 16 3 0 0 screen
[4336811.878903] [ 3498] 0 3498 28314 82 11 3 0 0 mac-neustar.sh
[4336811.878905] [ 5821] 0 5821 28314 81 8 3 0 0 mac-neustar.sh
[4336811.878906] [ 5822] 0 5822 14454 298 32 3 0 0 mysql
[4336811.878908] [ 5823] 0 5823 2472 22 9 3 0 0 awk
[4336811.878909] [ 5841] 0 5841 26519 248 53 3 0 0 sshd
[4336811.878910] Out of memory: Kill process 13106 (mysqld) score 970 or sacrifice child
[4336811.879074] Killed process 13106 (mysqld) total-vm:146394220kB, anon-rss:135292420kB, file-rss:0kB, shmem-rss:0kB
[4336816.264951] oom_reaper: reaped process 13106 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
|
|
Tasks: 26 total, 1 running, 25 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 13925728+total, 73712784 free, 16950448 used, 48594048 buff/cache
KiB Swap: 372732 total, 4 free, 372728 used. 11964473+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31322 mysql 20 0 24.599g 0.018t 4.094g S 0.0 14.1 16:01.57 mysqld
16 root 20 0 45036 16776 16500 S 0.0 0.0 1:43.47 systemd-journal
89 root 20 0 391592 13980 12364 S 0.0 0.0 1:22.16 rsyslogd
1 root 20 0 43980 4240 2456 S 0.0 0.0 0:44.48 systemd
80 root 20 0 26228 4120 1852 S 0.0 0.0 0:30.03 systemd-logind
12389 root 20 0 106440 3824 2496 S 0.7 0.0 0:00.20 sshd
Mariadb is using 25 GB of RAM, but rocksdb is supposed to use 15. And this is the only engine in use, no innodb.
show variables like 'rocks%cach%';
------------------------------------------------
------------------------------------------------
| rocksdb_block_cache_size |
15032385536 |
| rocksdb_cache_index_and_filter_blocks |
ON |
| rocksdb_force_compute_memtable_stats_cachetime |
60000000 |
| rocksdb_no_block_cache |
OFF |
| rocksdb_persistent_cache_path |
|
| rocksdb_persistent_cache_size_mb |
0 |
| rocksdb_pin_l0_filter_and_index_blocks_in_cache |
ON |
| rocksdb_sim_cache_size |
0 |
| rocksdb_skip_fill_cache |
OFF |
| rocksdb_table_cache_numshardbits |
6 |
| rocksdb_use_clock_cache |
OFF |
------------------------------------------------
11 rows in set (0.00 sec)
|
|
I am not sure what exactly is the cause of excessive memory consumption in this case, but the fix for MDEV-14460 (recently pushed) reduces memory consumption for transactions that have many statements.
|
|
I figured out the Achilles heel of MariaDB with any engine, and why it blows up. Every morning before it blows up, I do a large
mysql dbaase < update.sql
with hundreds of thousands of records, Some
delete from table where primar_key='xxx'
or replace table(fields) values(xxzz);
MariaDB will start requesting memory from the OS, without limit, until the OS kills the app through the OOM killer.
I found a workaround using split and then ingesting thousands of files instead of a large one. I still occasionally blows up, but far less.
So the issue is why the software cannot handle more than a few thousands lines in any *.sql file.
|
|
philip_38, I think something is really wrong with your experience here. I (and others) have loaded much larger files (on the order of gigabytes) from dump files without an issue.
Maybe your MariaDB instance is not configured correctly (some buffers are set to be very large?) Or maybe there is something very peculiar about the contents of your update.sql files...
I am not sure what to ask about in this case... In order to attempt to reproduce the problem, we will need your my.cnf, database dump, and an example of update.sql file that causes excessive memory consumption. This can be uploaded to our ftp where only MariaDB team members will have access to it. Is this too much to share?
|
|
> I found a workaround using split and then ingesting thousands of files instead of a large one. I still occasionally blows up, but far less.
I'm wondering what is the difference here between loading many smaller files and loading one large file. Are the changes loaded as one big transaction (do the files have set autocommit=0 or begin; ...end ?
Another possible difference is that each file might be using its own connection. So, if the connection accumulates some state which consumes more and more memory, one can get this effect (think of something like repeatedly running "SET @var=CONCAT(@var, 'more data')". But as you've mentioned the files have just DELETE and REPLACE commands, not such tricks...
|
|
Changing status to Cannot reproduce. philip_38, you're welcome to reopen if there is more info about how to reproduce or some details.
|