Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Warning] InnoDB: A long semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987314886400 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Warning] InnoDB: A long semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987413726976 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Warning] InnoDB: A long semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987319494400 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Note] InnoDB: A semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987314886400 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Note] InnoDB: A semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987413726976 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Note] InnoDB: A semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987319801600 has waited at btr0cur.cc line 1357 for 235.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Note] InnoDB: A semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987325638400 has waited at btr0cur.cc line 1357 for 236.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 0:09:13 0 [Note] InnoDB: A semaphore wait: Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987319494400 has waited at btr0cur.cc line 1357 for 241.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria InnoDB: ###### Starts InnoDB Monitor for 30 secs to print diagnostic info: Jan 15 00:09:20 spt-oxy-sql-plydev maria InnoDB: Pending reads 1, writes 0 Jan 15 00:09:20 spt-oxy-sql-plydev maria ===================================== Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 00:09:17 0x7f516c47d700 INNODB MONITOR OUTPUT Jan 15 00:09:20 spt-oxy-sql-plydev maria ===================================== Jan 15 00:09:20 spt-oxy-sql-plydev maria Per second averages calculated from the last 2 seconds Jan 15 00:09:20 spt-oxy-sql-plydev maria ----------------- Jan 15 00:09:20 spt-oxy-sql-plydev maria BACKGROUND THREAD Jan 15 00:09:20 spt-oxy-sql-plydev maria ----------------- Jan 15 00:09:20 spt-oxy-sql-plydev maria srv_master_thread loops: 2906 srv_active, 0 srv_shutdown, 3372 srv_idle Jan 15 00:09:20 spt-oxy-sql-plydev maria srv_master_thread log flush and writes: 6278 Jan 15 00:09:20 spt-oxy-sql-plydev maria ---------- Jan 15 00:09:20 spt-oxy-sql-plydev maria SEMAPHORES Jan 15 00:09:20 spt-oxy-sql-plydev maria ---------- Jan 15 00:09:20 spt-oxy-sql-plydev maria OS WAIT ARRAY INFO: reservation count 1203748 Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987314886400 has waited at btr0cur.cc line 1357 for 245.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987413726976 has waited at btr0cur.cc line 1357 for 245.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987319801600 has waited at btr0cur.cc line 1357 for 239.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987325638400 has waited at btr0cur.cc line 1357 for 240.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria --Thread 139987319494400 has waited at btr0cur.cc line 1357 for 245.00 seconds the semaphore: Jan 15 00:09:20 spt-oxy-sql-plydev maria SX-lock on RW-latch at 0x7f51145ae4a0 created in file dict0dict.cc line 2130 Jan 15 00:09:20 spt-oxy-sql-plydev maria a writer (thread id 139987677333248) has reserved it in mode SX Jan 15 00:09:20 spt-oxy-sql-plydev maria number of readers 0, waiters flag 1, lock_word: 10000000 Jan 15 00:09:20 spt-oxy-sql-plydev maria Last time write locked in file dict0stats.cc line 1969 Jan 15 00:09:20 spt-oxy-sql-plydev maria OS WAIT ARRAY INFO: signal count 1074647 Jan 15 00:09:20 spt-oxy-sql-plydev maria RW-shared spins 1722908, rounds 30862565, OS waits 628657 Jan 15 00:09:20 spt-oxy-sql-plydev maria RW-excl spins 677295, rounds 2503737, OS waits 35982 Jan 15 00:09:20 spt-oxy-sql-plydev maria RW-sx spins 93191, rounds 1466674, OS waits 18801 Jan 15 00:09:20 spt-oxy-sql-plydev maria Spin rounds per wait: 17.91 RW-shared, 3.70 RW-excl, 15.74 RW-sx Jan 15 00:09:20 spt-oxy-sql-plydev maria ------------------------ Jan 15 00:09:20 spt-oxy-sql-plydev maria LATEST DETECTED DEADLOCK Jan 15 00:09:20 spt-oxy-sql-plydev maria ------------------------ Jan 15 00:09:20 spt-oxy-sql-plydev maria 2021-01-15 00:05:10 0x7f52d4370700 Jan 15 00:09:20 spt-oxy-sql-plydev maria *** (1) TRANSACTION: Jan 15 00:09:20 spt-oxy-sql-plydev maria TRANSACTION 20921264, ACTIVE 14 sec inserting Jan 15 00:09:20 spt-oxy-sql-plydev maria mysql tables in use 3, locked 3 Jan 15 00:09:20 spt-oxy-sql-plydev maria LOCK WAIT 24835 lock struct(s), heap size 2646136, 1043337 row lock(s) Jan 15 00:09:20 spt-oxy-sql-plydev maria MySQL thread id 8128, OS thread handle 139987314886400, query id 127931 (redacted) Jan 15 00:21:10 spt-oxy-sql-plydev maria 2021-01-15 0:21:05 0 [ERROR] [FATAL] InnoDB: Semaphore wait has lasted > 600 seconds. We intentionally crash the server because it appears to be hung.