InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 241.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 241.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 241.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 11:55:06 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 52 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 826 srv_idle srv_master_thread log flush and writes: 91036 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 241.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 241 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:55:26 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 846 srv_idle srv_master_thread log flush and writes: 91056 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 261.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 261 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 272.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 272.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 272.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:55:42 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 862 srv_idle srv_master_thread log flush and writes: 91072 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 277.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 277 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:56:02 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 882 srv_idle srv_master_thread log flush and writes: 91092 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 297.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 297 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 303.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 303.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 303.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:56:18 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 898 srv_idle srv_master_thread log flush and writes: 91108 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 313.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 313 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:56:38 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 918 srv_idle srv_master_thread log flush and writes: 91128 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 333.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 333 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 334.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 334.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 334.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:56:54 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 934 srv_idle srv_master_thread log flush and writes: 91144 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 349.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 349 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 365.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 365.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 365.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 11:57:10 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 950 srv_idle srv_master_thread log flush and writes: 91160 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 365.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 365 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:57:30 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 970 srv_idle srv_master_thread log flush and writes: 91180 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 385.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 385 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 396.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 396.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 396.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:57:46 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 986 srv_idle srv_master_thread log flush and writes: 91196 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 401.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not startedInnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 401 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:58:06 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1006 srv_idle srv_master_thread log flush and writes: 91216 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 421.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 421 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 427.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 427.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 427.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:58:22 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1022 srv_idle srv_master_thread log flush and writes: 91232 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 437.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 437 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:58:42 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1042 srv_idle srv_master_thread log flush and writes: 91252 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 457.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 457 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 458.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 458.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 458.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:58:58 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1058 srv_idle srv_master_thread log flush and writes: 91268 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 473.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 473 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 489.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 489.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 489.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 11:59:14 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1074 srv_idle srv_master_thread log flush and writes: 91284 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 489.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 489 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:59:34 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1094 srv_idle srv_master_thread log flush and writes: 91304 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 509.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 509 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 520.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 520.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 520.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 11:59:50 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1110 srv_idle srv_master_thread log flush and writes: 91320 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 525.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 525 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:00:10 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1130 srv_idle srv_master_thread log flush and writes: 91340 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 545.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 545 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 551.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 551.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 551.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:00:26 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1146 srv_idle srv_master_thread log flush and writes: 91356 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 561.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 561 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:00:46 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1166 srv_idle srv_master_thread log flush and writes: 91376 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 581.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 581 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 582.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 582.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 582.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:01:02 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1182 srv_idle srv_master_thread log flush and writes: 91392 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 597.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 597 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 613.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 613.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 613.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 12:01:18 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1198 srv_idle srv_master_thread log flush and writes: 91408 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 613.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 613 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] 1 out of 1000 descriptors used InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] Main thread process no. 2942, id 140411297310464, state: sleeping InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:01:38 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1218 srv_idle srv_master_thread log flush and writes: 91428 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 633.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 633 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 644.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 644.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 644.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:01:54 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1234 srv_idle srv_master_thread log flush and writes: 91444 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 649.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 649 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:02:14 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1254 srv_idle srv_master_thread log flush and writes: 91464 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 669.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 669 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 675.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 675.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 675.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:02:30 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1270 srv_idle srv_master_thread log flush and writes: 91480 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 685.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: runningInnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 685 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:02:46 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1285 srv_idle srv_master_thread log flush and writes: 91495 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 701.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 701 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 707.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 707.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 707.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:03:02 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1301 srv_idle srv_master_thread log flush and writes: 91511 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 717.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 717 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:03:22 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1321 srv_idle srv_master_thread log flush and writes: 91531 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 737.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 737 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 738.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 738.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 738.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:03:38 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1337 srv_idle srv_master_thread log flush and writes: 91547 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 753.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 753 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 769.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 769.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 769.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 12:03:54 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1353 srv_idle srv_master_thread log flush and writes: 91563 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 769.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 769 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:04:14 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1373 srv_idle srv_master_thread log flush and writes: 91583 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 789.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 789 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 800.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 800.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 800.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:04:30 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1389 srv_idle srv_master_thread log flush and writes: 91599 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 805.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 805 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000616 (84999896 + 720) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:04:50 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 3 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1409 srv_idle srv_master_thread log flush and writes: 91619 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 825.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 0, not started MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7414639 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 825 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 831.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 831.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 831.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:05:06 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1425 srv_idle srv_master_thread log flush and writes: 91635 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 841.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7414639 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 841 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:05:26 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1445 srv_idle srv_master_thread log flush and writes: 91655 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 861.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7414639 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 861 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 862.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 862.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 862.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:05:42 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1461 srv_idle srv_master_thread log flush and writes: 91671 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 877.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7414639 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 877 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 893.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 893.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 893.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 ===================================== 2016-01-10 12:05:58 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1477 srv_idle srv_master_thread log flush and writes: 91687 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 893.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7414639 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 893 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 614077696 (605537432 + 8540264) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 6927 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 812 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.032 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 846 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.959 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 901 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 893 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 849 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.053 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 906 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 850 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.093 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 870 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.906 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] 1 out of 1000 descriptors used InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] Main thread process no. 2942, id 140411297310464, state: sleeping InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:06:18 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1497 srv_idle srv_master_thread log flush and writes: 91707 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 913.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba TRANSACTION 0, not started MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7415310 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 913 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 605541632 (605537432 + 4200) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 7448 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 874 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.031 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.958 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 970 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 962 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.052 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 967 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 915 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.092 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 934 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.905 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Warning: a long semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 924.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Warning: semaphore wait: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 924.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: Warning: Writer thread is waiting this semaphore: --Thread 140413944608512 has waited at row0ins.cc line 2708 for 924.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: InnoDB: Pending preads 0, pwrites 0 InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:06:34 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 16 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1513 srv_idle srv_master_thread log flush and writes: 91723 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 929.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7415342 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 929 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 605541632 (605537432 + 4200) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 7448 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 874 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.031 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.958 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 970 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 962 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.052 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 967 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 915 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.092 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 934 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.905 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] ===================================== 2016-01-10 12:06:54 7fb40e1fb700 INNODB MONITOR OUTPUT ===================================== Per second averages calculated from the last 20 seconds ----------------- BACKGROUND THREAD ----------------- srv_master_thread loops: 90210 srv_active, 0 srv_shutdown, 1533 srv_idle srv_master_thread log flush and writes: 91743 ---------- SEMAPHORES ---------- OS WAIT ARRAY INFO: reservation count 115244 --Thread 140413944608512 has waited at row0ins.cc line 2708 for 949.00 seconds the semaphore: X-lock (wait_ex) on RW-latch at 0x7fb9d1b25d40 '&block->lock' a writer (thread id 140413944608512) has reserved it in mode wait exclusive number of readers 1, waiters flag 0, lock_word: ffffffffffffffff Last time read locked in file buf0flu.cc line 1073 Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.0.22/storage/xtradb/row/row0ins.cc line 2708 OS WAIT ARRAY INFO: signal count 117960 Mutex spin waits 498955, rounds 3159087, OS waits 89901 RW-shared spins 29131, rounds 711638, OS waits 21329 RW-excl spins 20571, rounds 361831, OS waits 2900 Spin rounds per wait: 6.33 mutex, 24.43 RW-shared, 17.59 RW-excl ------------ TRANSACTIONS ------------ Trx id counter 44256486477 Purge done for trx's n:o < 44256486477 undo n:o < 0 state: running but idle InnoDB: signal 12 (User defined signal 2), address is 0x1f500000b7e from 0x3313c0b5ba History list length 2830 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0, not started MySQL thread id 248942, OS thread handle 0x7fb4ab45a700, query id 7415342 localhost root cleaning up ---TRANSACTION 44249729795, not started MySQL thread id 1, OS thread handle 0x7fb4ab5c7700, query id 0 Waiting for background binlog tasks ---TRANSACTION 44256486472, ACTIVE 949 sec inserting mysql tables in use 1, locked 1 1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1 MySQL thread id 6, OS thread handle 0x7fb4ab4a3700, query id 7408331 executing INSERT INTO user_bind_reward ( roleName,mobile,gameId,channel,packageId,version,createDate,reserve1,reserve2,reserve3,reserve4) VALUES ( 'G-T777264549177', '', '23081', 'bbgmj', 'com.ourgame.mahjong.danji.vivo', '5.5.0', NOW(), '', '', '', '' ) Trx #rec lock waits 0 #table lock waits 0 Trx total rec lock wait time 0 SEC Trx total table lock wait time 0 SEC -------- FILE I/O -------- I/O thread 0 state: waiting for completed aio requests (insert buffer thread) I/O thread 1 state: waiting for completed aio requests (log thread) I/O thread 2 state: waiting for completed aio requests (read thread) I/O thread 3 state: waiting for completed aio requests (read thread) I/O thread 4 state: waiting for completed aio requests (read thread) I/O thread 5 state: waiting for completed aio requests (read thread) I/O thread 6 state: waiting for completed aio requests (write thread) I/O thread 7 state: waiting for completed aio requests (write thread) I/O thread 8 state: waiting for completed aio requests (write thread) I/O thread 9 state: waiting for completed aio requests (write thread) Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 4 [4, 0, 0, 0] , ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0 Pending flushes (fsync) log: 0; buffer pool: 0 2177530 OS file reads, 9763391 OS file writes, 727366 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s ------------------------------------- INSERT BUFFER AND ADAPTIVE HASH INDEX ------------------------------------- Ibuf: size 1, free list len 22305, seg size 22307, 93306 merges merged operations: insert 122991, delete mark 4741, delete 3021 discarded operations: insert 0, delete mark 0, delete 0 0.00 hash searches/s, 0.00 non-hash searches/s --- LOG --- Log sequence number 1242897315268 Log flushed up to 1242897315268 Pages flushed up to 1242823361758 Last checkpoint at 1242823361758 Max checkpoint age 5217672500 Checkpoint age target 5054620235 Modified age 73953510 Checkpoint age 73953510 0 pending log writes, 0 pending chkp writes 6845525 log i/o's done, 0.00 log i/o's/second ---------------------- BUFFER POOL AND MEMORY ---------------------- Total memory allocated 35165044736; in additional pool allocated 0 Total memory allocated by read views 192 Internal hash tables (constant factor + variable factor) Adaptive hash index 605541632 (605537432 + 4200) Page hash 4426024 (buffer pool 0 only) Dictionary cache 143799107 (141607376 + 2191731) File system 961840 (812272 + 149568) Lock system 85000976 (84999896 + 1080) Recovery system 0 (0 + 0) Dictionary memory allocated 2191731 Buffer pool size 2097144 Buffer pool size, bytes 34359607296 Free buffers 7448 Database pages 2089696 Old database pages 771252 Modified db pages 41897 Percent of dirty pages(LRU & free pages): 1.998 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 6, single page 0 Pages made young 96579, not young 207241 0.00 youngs/s, 0.00 non-youngs/s Pages read 2177344, created 67858, written 2825328 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 2089696, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---------------------- INDIVIDUAL BUFFER POOL INFO ---------------------- ---BUFFER POOL 0 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 874 Database pages 261269 Old database pages 96425 Modified db pages 5325 Percent of dirty pages(LRU & free pages): 2.031 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12406, not young 25691 0.00 youngs/s, 0.00 non-youngs/s Pages read 272711, created 8251, written 335846 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261269, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 1 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96410 Modified db pages 5133 Percent of dirty pages(LRU & free pages): 1.958 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 2, flush list 1, single page 0 Pages made young 12225, not young 26920 0.00 youngs/s, 0.00 non-youngs/s Pages read 272238, created 8593, written 351875 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 2 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 970 Database pages 261173 Old database pages 96398 Modified db pages 5279 Percent of dirty pages(LRU & free pages): 2.014 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12189, not young 27558 0.00 youngs/s, 0.00 non-youngs/s Pages read 272281, created 8535, written 358403 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261173, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 3 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 962 Database pages 261181 Old database pages 96396 Modified db pages 5260 Percent of dirty pages(LRU & free pages): 2.007 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 2, single page 0 Pages made young 12272, not young 27932 0.00 youngs/s, 0.00 non-youngs/s Pages read 272386, created 8092, written 388317 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261181, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 4 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 913 Database pages 261230 Old database pages 96415 Modified db pages 5380 Percent of dirty pages(LRU & free pages): 2.052 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11931, not young 24069 0.00 youngs/s, 0.00 non-youngs/s Pages read 272027, created 9352, written 356589 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261230, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 5 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 967 Database pages 261176 Old database pages 96392 Modified db pages 5040 Percent of dirty pages(LRU & free pages): 1.923 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 1, single page 0 Pages made young 11684, not young 23563 0.00 youngs/s, 0.00 non-youngs/s Pages read 271707, created 7707, written 339035 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261176, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 6 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 915 Database pages 261228 Old database pages 96409 Modified db pages 5485 Percent of dirty pages(LRU & free pages): 2.092 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 12252, not young 24941 0.00 youngs/s, 0.00 non-youngs/s Pages read 272367, created 9078, written 361032 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261228, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] ---BUFFER POOL 7 Buffer pool size 262143 Buffer pool size, bytes 4294950912 Free buffers 934 Database pages 261209 Old database pages 96407 Modified db pages 4995 Percent of dirty pages(LRU & free pages): 1.905 Max dirty pages percent: 75.000 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 11620, not young 26567 0.00 youngs/s, 0.00 non-youngs/s Pages read 271627, created 8250, written 334231 0.00 reads/s, 0.00 creates/s, 0.00 writes/s No buffer pool page gets since the last printout Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 261209, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0] -------------- ROW OPERATIONS -------------- 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB 1 RW transactions active inside InnoDB 0 RO transactions active inside InnoDB 1 out of 1000 descriptors used Main thread process no. 2942, id 140411297310464, state: sleeping Number of rows inserted 2415224, updated 1243477, deleted 0, read 1243478 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 3360279, updated 0, deleted 3360279, read 3360280 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s ---------------------------- END OF INNODB MONITOR OUTPUT ============================ InnoDB: Stacktrace for Thread 140413944608512 InnoDB: [bt]: (1) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (2) /lib64/libpthread.so.0(pthread_cond_wait+0xca) [0x3313c0b5ba] InnoDB: [bt]: (3) /usr/sbin/mysqld() [0x8d85cb] InnoDB: [bt]: (4) /usr/sbin/mysqld() [0x948d1b] InnoDB: [bt]: (5) /usr/sbin/mysqld() [0x94aa49] InnoDB: [bt]: (6) /usr/sbin/mysqld() [0x9a2fa7] InnoDB: [bt]: (7) /usr/sbin/mysqld() [0x984fe8] InnoDB: [bt]: (8) /usr/sbin/mysqld() [0x90bb31] InnoDB: [bt]: (9) /usr/sbin/mysqld() [0x90e101] InnoDB: [bt]: (10) /usr/sbin/mysqld() [0x90e4b8] InnoDB: [bt]: (11) /usr/sbin/mysqld() [0x919bd0] InnoDB: [bt]: (12) /usr/sbin/mysqld() [0x885068] InnoDB: [bt]: (13) /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x14f) [0x735cbf] InnoDB: [bt]: (14) /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x113) [0x7f0113] InnoDB: [bt]: (15) /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x22) [0x7f03b2] InnoDB: [bt]: (16) /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x231) [0x7e4531] InnoDB: [bt]: (17) /usr/sbin/mysqld(_Z26apply_event_and_update_posP9Log_eventP3THDP14rpl_group_infoP19rpl_parallel_thread+0x16d) [0x565d1d] InnoDB: [bt]: (18) /usr/sbin/mysqld(handle_slave_sql+0x15ab) [0x5682ab] InnoDB: [bt]: (19) /lib64/libpthread.so.0() [0x3313c079d1] InnoDB: [bt]: (20) /lib64/libc.so.6(clone+0x6d) [0x33138e89dd] InnoDB: ###### Diagnostic info printed to the standard error stream InnoDB: Error: semaphore wait has lasted > 600 seconds InnoDB: We intentionally crash the server, because it appears to be hung. 2016-01-10 12:06:59 7fb40ebfc700 InnoDB: Assertion failure in thread 140411318290176 in file srv0srv.cc line 2200 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 160110 12:06:59 [ERROR] mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.22-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=4 max_threads=4002 thread_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 8918475 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48000 /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb8065b] /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x72c9b8] /lib64/libpthread.so.0[0x3313c0f710] /lib64/libc.so.6(gsignal+0x35)[0x3313832625] /lib64/libc.so.6(abort+0x175)[0x3313833e05] /usr/sbin/mysqld[0x9409f5] /lib64/libpthread.so.0[0x3313c079d1] /lib64/libc.so.6(clone+0x6d)[0x33138e89dd] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 160110 12:06:59 mysqld_safe Number of processes running now: 0 160110 12:06:59 mysqld_safe mysqld restarted 160110 12:06:59 [Note] /usr/sbin/mysqld (mysqld 10.0.22-MariaDB-log) starting as process 31059 ... 160110 12:06:59 [Note] InnoDB: Using mutexes to ref count buffer pool pages 160110 12:06:59 [Note] InnoDB: The InnoDB memory heap is disabled 160110 12:06:59 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 160110 12:06:59 [Note] InnoDB: Memory barrier is not used 160110 12:06:59 [Note] InnoDB: Compressed tables use zlib 1.2.3 160110 12:06:59 [Note] InnoDB: Using Linux native AIO 160110 12:06:59 [Note] InnoDB: Using CPU crc32 instructions 160110 12:06:59 [Note] InnoDB: Initializing buffer pool, size = 32.0G 160110 12:07:00 [Note] InnoDB: Completed initialization of buffer pool 160110 12:07:00 [Note] InnoDB: Opened 4 undo tablespaces 160110 12:07:00 [Note] InnoDB: Highest supported file format is Barracuda. 160110 12:07:00 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1242823361758 160110 12:07:00 [Note] InnoDB: Database was not shutdown normally! 160110 12:07:00 [Note] InnoDB: Starting crash recovery. 160110 12:07:00 [Note] InnoDB: Reading tablespace information from the .ibd files... 160110 12:07:00 [Note] InnoDB: Restoring possible half-written data pages 160110 12:07:00 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 1242828604416 InnoDB: Doing recovery: scanned up to log sequence number 1242833847296 InnoDB: Doing recovery: scanned up to log sequence number 1242839090176 InnoDB: Doing recovery: scanned up to log sequence number 1242844333056 InnoDB: Doing recovery: scanned up to log sequence number 1242849575936 InnoDB: Doing recovery: scanned up to log sequence number 1242854818816 InnoDB: Doing recovery: scanned up to log sequence number 1242860061696 InnoDB: Doing recovery: scanned up to log sequence number 1242865304576 InnoDB: Doing recovery: scanned up to log sequence number 1242870547456 InnoDB: Doing recovery: scanned up to log sequence number 1242875790336 InnoDB: Doing recovery: scanned up to log sequence number 1242881033216 InnoDB: Doing recovery: scanned up to log sequence number 1242886276096 InnoDB: Doing recovery: scanned up to log sequence number 1242891518976 InnoDB: Doing recovery: scanned up to log sequence number 1242896761856 InnoDB: Doing recovery: scanned up to log sequence number 1242897315268 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 1 row operations to undo InnoDB: Trx id counter is 44256486912 160110 12:07:02 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: Last MySQL binlog file position 0 697210873, file name /data/mysql/log/mysql-bin.000063 160110 12:07:04 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2016-01-10 12:07:04 7fd2ce7ff700 InnoDB: Rolling back trx with id 44256486472, 1 rows to undo 160110 12:07:04 [Note] InnoDB: Rollback of trx with id 44256486472 completed 2016-01-10 12:07:04 7fd2ce7ff700 InnoDB: Rollback of non-prepared transactions completed 160110 12:07:04 [Note] InnoDB: Waiting for purge to start 160110 12:07:05 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.26-74.0 started; log sequence number 1242897315268 2016-01-10 12:07:05 7fd2bfff9700 InnoDB: Loading buffer pool(s) from /data/mysql/data/ib_buffer_pool 160110 12:07:05 [Note] Plugin 'FEEDBACK' is disabled. 160110 12:07:05 server_audit: MariaDB Audit Plugin version 1.3.0 STARTED. 160110 12:07:05 [Note] Recovering after a crash using /data/mysql/log/mysql-bin 160110 12:07:07 [Note] Starting crash recovery... 160110 12:07:07 [Note] Crash recovery finished. 160110 12:07:07 [Warning] Failed to create a socket for IPv6 '::': errno: 97. 160110 12:07:07 [Note] Server socket created on IP: '0.0.0.0'. 160110 12:07:07 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.0.22-MariaDB-log' socket: '/data/mysql/data/mysql.sock' port: 3306 MariaDB Server