InnoDB: Waiting for page_cleaner to finish flushing of buffer pool (unbelievably slow)

Mar 26 04:45:17 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service: main process exited, code=killed, status=9/KILL

Mar 26 04:45:17 opsky7.aus.stglabs.ibm.com systemd[1]: Unit mariadb.service entered failed state.

Mar 26 04:45:17 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service failed.

Mar 26 04:45:22 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service holdoff time over, scheduling restart.

Mar 26 04:45:23 opsky7.aus.stglabs.ibm.com systemd[1]: Starting MariaDB 10.3.6 database server...

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] /usr/local/mysql/bin/mysqld (mysqld 10.3.6-MariaDB) starting as process 57988 ...

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Uses event mutexes

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using Linux native AIO

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Number of pools: 1

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using SSE2 crc32 instructions

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Initializing buffer pool, total size = 128G, instances = 8, chunk size = 1G

Mar 26 04:45:57 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:57 0 [Note] InnoDB: Completed initialization of buffer pool

Mar 26 04:45:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See t

Mar 26 04:46:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=38095597389

Mar 26 04:46:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:14 0 [Note] InnoDB: Read redo log up to LSN=37328381952

Mar 26 04:46:20 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:20 0 [Note] InnoDB: Starting final batch to recover 549794 pages from redo log.

Mar 26 04:46:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:29 0 [Note] InnoDB: To recover: 537699 pages from log

Mar 26 04:46:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:44 0 [Note] InnoDB: To recover: 505862 pages from log

Mar 26 04:46:54 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service start operation timed out. Terminating.

Mar 26 04:46:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:59 0 [Note] InnoDB: To recover: 459982 pages from log

Mar 26 04:47:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:14 0 [Note] InnoDB: To recover: 414658 pages from log

Mar 26 04:47:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:29 0 [Note] InnoDB: To recover: 368367 pages from log

Mar 26 04:47:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:44 0 [Note] InnoDB: To recover: 322961 pages from log

Mar 26 04:47:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:59 0 [Note] InnoDB: To recover: 278069 pages from log

Mar 26 04:48:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:14 0 [Note] InnoDB: To recover: 237498 pages from log

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service stop-final-sigterm timed out. Skipping SIGKILL. Entering failed mode.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Failed to start MariaDB 10.3.6 database server.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Unit mariadb.service entered failed state.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service failed.

Mar 26 04:48:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:29 0 [Note] InnoDB: To recover: 192424 pages from log

Mar 26 04:48:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:44 0 [Note] InnoDB: To recover: 146854 pages from log

Mar 26 04:48:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:59 0 [Note] InnoDB: To recover: 100860 pages from log

Mar 26 04:49:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:14 0 [Note] InnoDB: To recover: 55217 pages from log

Mar 26 04:49:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:29 0 [Note] InnoDB: To recover: 8860 pages from log

Mar 26 04:49:33 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:33 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aebc0000000 size 2097152

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Waiting for purge to start

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: 5.7.21 started; log sequence number 38095618140; transaction id 96064

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Plugin 'FEEDBACK' is disabled.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Recovering after a crash using tc.log

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Starting crash recovery...

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Crash recovery finished.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Buffer pool(s) load completed at 180326  4:49:35

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] Server socket created on IP: '::'.

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Uses event mutexes

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using Linux native AIO

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Number of pools: 1

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using SSE2 crc32 instructions

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Initializing buffer pool, total size = 128G, instances = 8, chunk size = 1G

Mar 26 04:45:57 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:57 0 [Note] InnoDB: Completed initialization of buffer pool

Mar 26 04:45:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See th

Mar 26 04:46:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=38095597389

Mar 26 04:46:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:14 0 [Note] InnoDB: Read redo log up to LSN=37328381952

Mar 26 04:46:20 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:20 0 [Note] InnoDB: Starting final batch to recover 549794 pages from redo log.

Mar 26 04:46:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:29 0 [Note] InnoDB: To recover: 537699 pages from log

Mar 26 04:46:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:44 0 [Note] InnoDB: To recover: 505862 pages from log

Mar 26 04:46:54 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service start operation timed out. Terminating.

Mar 26 04:46:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:59 0 [Note] InnoDB: To recover: 459982 pages from log

Mar 26 04:47:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:14 0 [Note] InnoDB: To recover: 414658 pages from log

Mar 26 04:47:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:29 0 [Note] InnoDB: To recover: 368367 pages from log

Mar 26 04:47:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:44 0 [Note] InnoDB: To recover: 322961 pages from log

Mar 26 04:47:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:59 0 [Note] InnoDB: To recover: 278069 pages from log

Mar 26 04:48:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:14 0 [Note] InnoDB: To recover: 237498 pages from log

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service stop-final-sigterm timed out. Skipping SIGKILL. Entering failed mode.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Failed to start MariaDB 10.3.6 database server.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Unit mariadb.service entered failed state.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service failed.

Mar 26 04:48:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:29 0 [Note] InnoDB: To recover: 192424 pages from log

Mar 26 04:48:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:44 0 [Note] InnoDB: To recover: 146854 pages from log

Mar 26 04:48:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:59 0 [Note] InnoDB: To recover: 100860 pages from log

Mar 26 04:49:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:14 0 [Note] InnoDB: To recover: 55217 pages from log

Mar 26 04:49:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:29 0 [Note] InnoDB: To recover: 8860 pages from log

Mar 26 04:49:33 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:33 0 [Warning] InnoDB: Failed to set memory to DODUMP: Invalid argument ptr 0x2aebc0000000 size 2097152

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: 128 out of 128 rollback segments are active.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Creating shared tablespace for temporary tables

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Waiting for purge to start

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: 5.7.21 started; log sequence number 38095618140; transaction id 96064

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Plugin 'FEEDBACK' is disabled.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Recovering after a crash using tc.log

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Starting crash recovery...

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] Crash recovery finished.

Mar 26 04:49:35 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:49:35 0 [Note] InnoDB: Buffer pool(s) load completed at 180326  4:49:35

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] Server socket created on IP: '::'.

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] /usr/local/mysql/bin/mysqld (unknown): Normal shutdown

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] Reading of all Master_info entries succeded

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] Added new Master_info '' to hash table

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: Version: '10.3.6-MariaDB'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] Event Scheduler: Purging the queue. 0 events

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] InnoDB: FTS optimize thread exiting.

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] InnoDB: Starting shutdown...

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool

Mar 26 04:50:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:50:00 0 [Note] InnoDB: Buffer pool(s) dump completed at 180326  4:50:00

Mar 26 04:51:01 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:51:01 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:52:01 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:52:01 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:53:01 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:53:01 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:54:01 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:54:01 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:55:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:55:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:56:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:56:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:57:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:57:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:58:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:58:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:59:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:59:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 05:00:02 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  5:00:02 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 05:01:03 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  5:01:03 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 05:02:03 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  5:02:03 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

.....

Mar 26 16:21:06 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:21:06 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 16:22:06 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:22:06 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 16:23:06 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:23:06 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 16:24:07 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:24:07 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 16:25:07 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:25:07 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 16:26:07 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26 16:26:07 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Uses event mutexes

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.7

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using Linux native AIO

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Number of pools: 1

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Using SSE2 crc32 instructions

Mar 26 04:45:24 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:24 0 [Note] InnoDB: Initializing buffer pool, total size = 128G, instances = 8, chunk size = 1G

Mar 26 04:45:57 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:57 0 [Note] InnoDB: Completed initialization of buffer pool

Mar 26 04:45:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:45:59 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See th

Mar 26 04:46:00 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=38095597389

Mar 26 04:46:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:14 0 [Note] InnoDB: Read redo log up to LSN=37328381952

Mar 26 04:46:20 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:20 0 [Note] InnoDB: Starting final batch to recover 549794 pages from redo log.

Mar 26 04:46:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:29 0 [Note] InnoDB: To recover: 537699 pages from log

Mar 26 04:46:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:44 0 [Note] InnoDB: To recover: 505862 pages from log

Mar 26 04:46:54 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service start operation timed out. Terminating.

Mar 26 04:46:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:46:59 0 [Note] InnoDB: To recover: 459982 pages from log

Mar 26 04:47:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:14 0 [Note] InnoDB: To recover: 414658 pages from log

Mar 26 04:47:29 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:29 0 [Note] InnoDB: To recover: 368367 pages from log

Mar 26 04:47:44 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:44 0 [Note] InnoDB: To recover: 322961 pages from log

Mar 26 04:47:59 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:47:59 0 [Note] InnoDB: To recover: 278069 pages from log

Mar 26 04:48:14 opsky7.aus.stglabs.ibm.com mysqld[57988]: 2018-03-26  4:48:14 0 [Note] InnoDB: To recover: 237498 pages from log

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service stop-final-sigterm timed out. Skipping SIGKILL. Entering failed mode.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Failed to start MariaDB 10.3.6 database server.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: Unit mariadb.service entered failed state.

Mar 26 04:48:24 opsky7.aus.stglabs.ibm.com systemd[1]: mariadb.service failed.

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

0x00007fc29b553945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thread 17 (Thread 0x7fc296e60700 (LWP 57989)):

#0  0x00007fc29b553cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x0000000000c63c61 in inline_mysql_cond_timedwait (mutex=<optimized out>, src_line=<optimized out>, that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_file=<optimized out>, src_line=<optimized out>) at /home/dan/mariadb-server/include/mysql/psi/mysql_thread.h:1215

#2  timer_handler (arg=<optimized out>) at /home/dan/mariadb-server/mysys/thr_timer.c:292

#3  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7fc294938700 (LWP 57990)):

#0  0x00007fc29b553cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x0000000000b7a093 in inline_mysql_cond_timedwait (src_line=<optimized out>, that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_file=<optimized out>, src_line=<optimized out>) at /home/dan/mariadb-server/include/mysql/psi/mysql_thread.h:1215

#2  my_service_thread_sleep (control=0x1534cc0 <checkpoint_control>, sleep_time=<optimized out>) at /home/dan/mariadb-server/storage/maria/ma_servicethread.c:115

#3  0x0000000000b73a11 in ma_checkpoint_background (arg=0x1e) at /home/dan/mariadb-server/storage/maria/ma_checkpoint.c:707

#4  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7fc28d29b700 (LWP 58255)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fc28d29aa70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fc28d29aa70, m1=0x7fc28d29ad28, m2=<optimized out>, request=0x7fc28d29ad10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fc28d29ad28, m2=0x7fc28d29ad08, request=0x7fc28d29ad10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=0) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7fc28ca9a700 (LWP 58256)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fc28ca99a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fc28ca99a70, m1=0x7fc28ca99d28, m2=<optimized out>, request=0x7fc28ca99d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fc28ca99d28, m2=0x7fc28ca99d08, request=0x7fc28ca99d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=1) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7fbff5e06700 (LWP 58257)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff5e05a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff5e05a70, m1=0x7fbff5e05d28, m2=<optimized out>, request=0x7fbff5e05d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff5e05d28, m2=0x7fbff5e05d08, request=0x7fbff5e05d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=2) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7fbff5605700 (LWP 58258)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff5604a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff5604a70, m1=0x7fbff5604d28, m2=<optimized out>, request=0x7fbff5604d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff5604d28, m2=0x7fbff5604d08, request=0x7fbff5604d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=3) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7fbff4e04700 (LWP 58259)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff4e03a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff4e03a70, m1=0x7fbff4e03d28, m2=<optimized out>, request=0x7fbff4e03d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff4e03d28, m2=0x7fbff4e03d08, request=0x7fbff4e03d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=4) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7fbff4603700 (LWP 58260)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff4602a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff4602a70, m1=0x7fbff4602d28, m2=<optimized out>, request=0x7fbff4602d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff4602d28, m2=0x7fbff4602d08, request=0x7fbff4602d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=5) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7fbff3e02700 (LWP 58261)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff3e01a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff3e01a70, m1=0x7fbff3e01d28, m2=<optimized out>, request=0x7fbff3e01d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff3e01d28, m2=0x7fbff3e01d08, request=0x7fbff3e01d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=6) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7fbff3601700 (LWP 58262)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff3600a70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff3600a70, m1=0x7fbff3600d28, m2=<optimized out>, request=0x7fbff3600d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff3600d28, m2=0x7fbff3600d08, request=0x7fbff3600d10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=7) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7fbff2e00700 (LWP 58263)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff2dffa70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff2dffa70, m1=0x7fbff2dffd28, m2=<optimized out>, request=0x7fbff2dffd10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff2dffd28, m2=0x7fbff2dffd08, request=0x7fbff2dffd10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=8) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7fbff25ff700 (LWP 58264)):

#0  0x00007fc29b120644 in __io_getevents_0_4 () from /lib64/libaio.so.1

#1  0x00000000009bd3af in LinuxAIOHandler::collect (this=0x7fbff25fea70) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:1891

#2  0x00000000009bd7b6 in LinuxAIOHandler::poll (this=0x7fbff25fea70, m1=0x7fbff25fed28, m2=<optimized out>, request=0x7fbff25fed10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2036

#3  0x00000000009c1312 in os_aio_linux_handler (global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>, global_segment=<optimized out>, m1=<optimized out>, m2=<optimized out>, request=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:2090

#4  os_aio_handler (segment=<optimized out>, m1=0x7fbff25fed28, m2=0x7fbff25fed08, request=0x7fbff25fed10) at /home/dan/mariadb-server/storage/innobase/os/os0file.cc:5668

#5  0x0000000000afd864 in fil_aio_wait (segment=9) at /home/dan/mariadb-server/storage/innobase/fil/fil0fil.cc:5323

#6  0x0000000000a3b188 in io_handler_thread (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:339

#7  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#8  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7fbff1dfe700 (LWP 58265)):

#0  0x00007fc29b556e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x00000000009c52af in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0thread.cc:230

#2  0x0000000000ab44ca in buf_flush_page_cleaner_close () at /home/dan/mariadb-server/storage/innobase/buf/buf0flu.cc:2713

#3  buf_flush_page_cleaner_coordinator () at /home/dan/mariadb-server/storage/innobase/buf/buf0flu.cc:3426

#4  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7fbff15fd700 (LWP 58266)):

#0  0x00007fc29b553945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00000000009c4cb6 in wait (this=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0event.cc:163

#2  os_event::wait_low (this=0x17a563d30, reset_sig_count=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0event.cc:333

#3  0x0000000000ab51c3 in buf_flush_page_cleaner_worker (arg=<optimized out>) at /home/dan/mariadb-server/storage/innobase/buf/buf0flu.cc:3505

#4  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#5  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7fc133fff700 (LWP 60268)):

#0  0x00007fc29b553cf2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00000000007c58ec in inline_mysql_cond_timedwait (that=<optimized out>, mutex=<optimized out>, abstime=<optimized out>, src_file=<optimized out>, src_line=<optimized out>) at /home/danmariadb-server/include/mysql/psi/mysql_thread.h:1215

#2  timer_thread (param=0x15d85f8 <pool_timer>) at /home/dan/mariadb-server/sql/threadpool_generic.cc:655

#3  0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#4  0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7fc294137700 (LWP 60270)):

#0  0x00007fc29b556e4d in nanosleep () from /lib64/libpthread.so.0

#1  0x00000000009c52af in os_thread_sleep (tm=<optimized out>) at /home/dan/mariadb-server/storage/innobase/os/os0thread.cc:230

#2  0x00000000009ad8fd in logs_empty_and_mark_files_at_shutdown () at /home/dan/mariadb-server/storage/innobase/log/log0log.cc:1996

#3  0x0000000000a41f7d in innodb_shutdown () at /home/dan/mariadb-server/storage/innobase/srv/srv0start.cc:2808

#4  0x000000000096cc55 in innobase_end () at /home/dan/mariadb-server/storage/innobase/handler/ha_innodb.cc:4298

#5  0x00000000007f0588 in ha_finalize_handlerton (plugin=0x2e86928) at /home/dan/mariadb-server/sql/handler.cc:461

#6  0x00000000006386b9 in plugin_deinitialize (plugin=0x2e86928, ref_check=true) at /home/dan/mariadb-server/sql/sql_plugin.cc:1258

#7  0x0000000000636fba in reap_plugins () at /home/dan/mariadb-server/sql/sql_plugin.cc:1334

#8  0x0000000000638925 in plugin_shutdown () at /home/dan/mariadb-server/sql/sql_plugin.cc:2000

#9  0x000000000057dcad in clean_up (print_message=true) at /home/dan/mariadb-server/sql/mysqld.cc:2243

#10 0x000000000057dbdb in unireg_end () at /home/dan/mariadb-server/sql/mysqld.cc:2115

#11 0x000000000057db87 in kill_server (sig_ptr=<optimized out>) at /home/dan/mariadb-server/sql/mysqld.cc:2042

#12 kill_server_thread (arg=0x7fc294135498) at /home/dan/mariadb-server/sql/mysqld.cc:2065

#13 0x00007fc29b54fe25 in start_thread () from /lib64/libpthread.so.0

#14 0x00007fc29970534d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7fc29b9418c0 (LWP 57988)):

#0  0x00007fc29b553945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00000000005813fb in inline_mysql_cond_wait (that=<optimized out>, mutex=<optimized out>, src_file=<optimized out>, src_line=<optimized out>) at /home/dan/mariadb-server/include/mysql/si/mysql_thread.h:1178

#2  mysqld_main (argc=27, argv=0x2e1c658) at /home/dan/mariadb-server/sql/mysqld.cc:6180

#3  0x00007fc29962ec05 in __libc_start_main () from /lib64/libc.so.6

#4  0x000000000057c759 in _start ()

#2  0x00000000009ad8fd in logs_empty_and_mark_files_at_shutdown () at /home/dan/mariadb-server/storage/innobase/log/log0log.cc:1996

1996                    os_thread_sleep(100000);

(gdb) info locals

count = 259

lsn = <optimized out>

(gdb) p *log_sys

$1 = {pad1 = '\000' <repeats 63 times>, lsn = 38095655874, buf_free = 450, pad2 = '\000' <repeats 63 times>, mutex = {m_impl = {m_lock_word = 0, m_event = 0x17a5609a0, m_policy = {

        m_count = {m_spins = 0, m_waits = 0, m_calls = 0, m_enabled = false}, m_id = LATCH_ID_LOG_SYS}}, m_ptr = 0x0}, pad3 = '\000' <repeats 63 times>, write_mutex = {m_impl = {

      m_lock_word = 0, m_event = 0x17a560ab0, m_policy = {m_count = {m_spins = 0, m_waits = 0, m_calls = 0, m_enabled = false}, m_id = LATCH_ID_LOG_WRITE}}, m_ptr = 0x0},

  pad4 = '\000' <repeats 63 times>, log_flush_order_mutex = {m_impl = {m_lock_word = 0, m_event = 0x17a560f60, m_policy = {m_count = {m_spins = 0, m_waits = 0, m_calls = 0,

          m_enabled = false}, m_id = LATCH_ID_LOG_FLUSH_ORDER}}, m_ptr = 0x0}, buf = 0x2aeb80000000 "\004oV\312\001\302", <incomplete sequence \302>, first_in_use = true,

  buf_size = 67108864, max_buf_free = 33486848, check_flush_or_checkpoint = false, log = {n_files = 4, format = 103, file_size = 1073741824, state = LOG_GROUP_OK, lsn = 38095637015,

    lsn_offset = 3735963671, file_header_bufs_ptr = 0x17a5857e0, file_header_bufs = 0x17a585810, scanned_lsn = 38095618140, checkpoint_buf_ptr = 0x17a588080 "",

    checkpoint_buf = 0x17a588200 ""}, buf_next_to_write = 450, is_extending = false, write_lsn = 38095655874, current_flush_lsn = 38095655874, flushed_to_disk_lsn = 38095655874,

  n_pending_flushes = 0, flush_event = 0x17a561070, n_log_ios = 13372, n_log_ios_old = 13369, last_printout_time = 1522057775, log_group_capacity = 3865463194,

  max_modified_age_async = 3043432951, max_modified_age_sync = 3260821019, max_checkpoint_age_async = 3369515053, max_checkpoint_age = 3478209086, next_checkpoint_no = 965,

  last_checkpoint_lsn = 37223755687, next_checkpoint_lsn = 37223755687, append_on_checkpoint = 0x0, n_pending_checkpoint_writes = 0, checkpoint_lock = {lock_word = 536870912, waiters = 0,

    sx_recursive = 0, writer_is_wait_ex = false, writer_thread = 0, event = 0x17a5610f0, wait_ex_event = 0x17a561170,

    cfile_name = 0xecc160 "/home/dan/mariadb-server/storage/innobase/log/log0log.cc", last_x_file_name = 0xecc160 "/home/dan/mariadb-server/storage/innobase/log/log0log.cc", cline = 753,

    is_block_lock = 0, last_x_line = 1525, count_os_wait = 2, list = {prev = 0x17a584cd0, next = 0x17a5581c0}, pfs_psi = 0x0}, checkpoint_buf_ptr = 0x17a5611f0 "",

  checkpoint_buf = 0x17a561200 ""}

[mariadb]

datadir=/var/lib/mysql

thread_handling=pool-of-threads

thread_pool_stall_limit=1ms

max-connections=4096

large_pages=1

memlock=1

query_cache_type=0

# Was set to match P9's memory page size

#innodb_page_size=64k

innodb-buffer-pool-size=128G

# 1G - 32M - based off buf_chunk_init to keep the

# allocation in 1G chunks

# 16k page size

# (gdb) p (1 << 30) / (16 * 1024) * sizeof(buf_block_t) + (16384 - 1)

#$2 = 25182207

# 64k page size

#(gdb) p (1 << 30) / (65536) * sizeof(buf_block_t) + (65536 - 1)

#$3 = 6356991

#(gdb) p 1 << 30 - (32 * 1<< 20)

#$4 = 1073741824

innodb_buffer_pool_chunk_size=1073741824

innodb_file_per_table=1

innodb_flush_method=O_DIRECT

innodb_lru_scan_depth=20000

innodb_io_capacity=10k

innodb_io_capacity_max=20k

innodb_log_buffer_size=64M

innodb_thread_concurrency=0

innodb_max_dirty_pages_pct=25

innodb_adaptive_flushing=1

innodb_read_io_threads=4

innodb_write_io_threads=4

innodb_purge_threads=1

innodb_log_file_size=1G

innodb_log_files_in_group=4

performance_schema=OFF

max_prepared_stmt_count=16M