InnoDB: Warning: a long semaphore wait: --Thread 139885753087744 has waited at buf0flu.cc line 1099 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 InnoDB: Warning: semaphore wait: --Thread 139885753087744 has waited at buf0flu.cc line 1099 for 241.00 seconds the semaphore: S-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 InnoDB: Warning: semaphore wait: --Thread 139885550978816 has waited at row0ins.cc line 2410 for 236.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 InnoDB: Warning: semaphore wait: --Thread 139885633599232 has waited at row0ins.cc line 2410 for 236.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 InnoDB: Warning: semaphore wait: --Thread 139885646182144 has waited at row0ins.cc line 2410 for 233.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ................. ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 3220490 srv_active, 0 srv_shutdown, 5060747 srv_idle srv_master_thread log flush and writes: 8281237 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 4030789 --Thread 139885753087744 has waited at buf0flu.cc line 1099 for 939.00 seconds the semaphore: S-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 --Thread 139885550978816 has waited at row0ins.cc line 2410 for 934.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 --Thread 139885633599232 has waited at row0ins.cc line 2410 for 934.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 --Thread 139885646182144 has waited at row0ins.cc line 2410 for 931.00 seconds the semaphore: X-lock on RW-latch at 0x7f3a13755940 '&block->lock' a writer (thread id 139885552043776) has reserved it in mode exclusive number of readers 0, waiters flag 1, lock_word: 0 Last time read locked in file buf0flu.cc line 1093 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.1.16/storage/xtradb/row/row0ins.cc line 2410 Holder thread 0 file not yet reserved line 0 OS WAIT ARRAY INFO: signal count 7563270 Mutex spin waits 57944255, rounds 191420993, OS waits 1707054 RW-shared spins 4170955, rounds 91034528, OS waits 2142812 RW-excl spins 2504676, rounds 17939004, OS waits 70531 Spin rounds per wait: 3.30 mutex, 21.83 RW-shared, 7.16 RW-excl ------------ TRANSACTIONS ------------ ............. ---TRANSACTION 454073972, not started MySQL thread id 3388936, OS thread handle 0x7f38f537d700, query id 119099404 10.220.62.13 root cleaning up ---TRANSACTION 454073970, not started MySQL thread id 3388914, OS thread handle 0x7f38f45b6700, query id 119099402 10.220.62.13 root cleaning up ---TRANSACTION 381495043, not started MySQL thread id 1, OS thread handle 0x7f39ab9eb700, query id 0 Waiting for background binlog tasks ---TRANSACTION 480931009, ACTIVE 931 sec inserting, thread declared inside InnoDB 5000 mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s) MySQL thread id 4805271, OS thread handle 0x7f39aa3ff700, query id 119082211 10.220.62.12 root update .... Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC ---TRANSACTION 480931005, ACTIVE 934 sec inserting, thread declared inside InnoDB 5000 mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s) MySQL thread id 4805272, OS thread handle 0x7f39a97ff700, query id 119082207 10.220.62.12 root update ... Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC ---TRANSACTION 480931002, ACTIVE 934 sec inserting, thread declared inside InnoDB 5000 mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s) MySQL thread id 4805273, OS thread handle 0x7f39a4934700, query id 119082204 10.220.62.12 root update .. Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC ---TRANSACTION 480930999, ACTIVE 939 sec inserting, thread declared inside InnoDB 5000 mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s) MySQL thread id 4805274, OS thread handle 0x7f39a4a38700, query id 119082192 10.220.62.12 root update .. Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC .... -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (read thread) I/O thread 7 state: waiting for completed aio requests (read thread) I/O thread 8 state: waiting for completed aio requests (read thread) I/O thread 9 state: waiting for completed aio requests (read thread) I/O thread 10 state: waiting for completed aio requests (write thread) I/O thread 11 state: waiting for completed aio requests (write thread) I/O thread 12 state: waiting for completed aio requests (write thread) I/O thread 13 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0, 0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 77216828 OS file reads, 93310295 OS file writes, 9426013 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 4686, seg size 4688, 3661170 merges merged operations: insert 3876519, delete mark 42167625, delete 3518489 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 199632279248 Log flushed up to 199632279248 Pages flushed up to 199621096581 Last checkpoint at 199621096581 Max checkpoint age 651231499 Checkpoint age target 630880515 Modified age 11182667 Checkpoint age 11182667 0 pending log writes, 0 pending chkp writes 48890888 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 4504682496; in additional pool allocated 0 Total memory allocated by read views 3016 Internal hash tables (constant factor + variable factor) Adaptive hash index 75696848 (75692648 + 4200) Page hash 1107208 (buffer pool 0 only) Dictionary cache 19098461 (17702416 + 1396045) File system 85360 (82672 + 2688) Lock system 11467744 (10625704 + 842040) Recovery system 0 (0 + 0) Dictionary memory allocated 1396045 Buffer pool size 262140 Buffer pool size, bytes 4294901760 Free buffers 1052 Database pages 261088 Old database pages 96454 Modified db pages 4135 Percent of dirty pages(LRU & free pages): 1.577 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 71264070, not young 2181684825 0.00 youngs/s, 0.00 non-youngs/s Pages read 77217216, created 1303673, written 44421265 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261088, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 0 Database pages 65535 Old database pages 24210 Modified db pages 1175 Percent of dirty pages(LRU & free pages): 1.793 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 17798220, not young 544684514 0.00 youngs/s, 0.00 non-youngs/s Pages read 19082119, created 427446, written 21495951 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 65535, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 238 Database pages 65297 Old database pages 24123 Modified db pages 903 Percent of dirty pages(LRU & free pages): 1.378 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 17022538, not young 527933472 0.00 youngs/s, 0.00 non-youngs/s Pages read 19223850, created 286478, written 6528799 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 65297, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 357 Database pages 65178 Old database pages 24079 Modified db pages 742 Percent of dirty pages(LRU & free pages): 1.132 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 18401203, not young 586295393 0.00 youngs/s, 0.00 non-youngs/s Pages read 19588231, created 287708, written 6998904 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 65178, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 65535 Buffer pool size, bytes 1073725440 Free buffers 457 Database pages 65078 Old database pages 24042 Modified db pages 1315 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 18042109, not young 522771446 0.00 youngs/s, 0.00 non-youngs/s Pages read 19323016, created 302041, written 9397611 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 65078, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 6 queries inside InnoDB, 2320 queries in queue 2 read views open inside InnoDB 4 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 4 out of 1000 descriptors used ---OLDEST VIEW--- Normal read view Read view low limit trx n:o 480930971 Read view up limit trx id 480930971 Read view low limit trx id 480930971 Read view individually stored trx ids: ----------------- Main thread process no. 12961, id 139885826516736, state: sleeping Number of rows inserted 22227139, updated 8289059, deleted 19713716, read 31179497755 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 0, updated 0, deleted 0, read 0 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 2017-05-07 04:16:31 7f39b63fc700 InnoDB: Assertion failure in thread 139885847496448 in file srv0srv.cc line 2365 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 170507 4:16:31 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.16-MariaDB key_buffer_size=16777216 read_buffer_size=262144 max_used_connections=2379 max_threads=3001 thread_count=2379 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6992161 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x40000