21.11.2019 - DBCL01-02 Nov 21 08:41:29 dbcl01-02 mysqld: 191121 8:41:29 [ERROR] mysqld got signal 11 ; Nov 21 08:41:29 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:41:29 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:41:29 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:41:29 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:41:29 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:41:29 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:41:29 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:41:29 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:41:29 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:41:29 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:41:29 dbcl01-02 mysqld: max_used_connections=53 Nov 21 08:41:29 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:41:29 dbcl01-02 mysqld: thread_count=69 Nov 21 08:41:29 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:41:29 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:41:29 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:41:29 dbcl01-02 mysqld: Thread pointer: 0x7f78a80009a8 Nov 21 08:41:29 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:41:29 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:41:29 dbcl01-02 mysqld: terribly wrong... Nov 21 08:41:29 dbcl01-02 mysqld: stack_bottom = 0x7f7a083bfd00 thread_stack 0x30000 Nov 21 08:41:30 dbcl01-02 kernel: mysqld[53420]: segfault at 19 ip 0000560db96799bf sp 00007f7a0806b1c0 error 4 in mysqld[560db89be000+12c9000] Nov 21 08:41:31 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:41:31 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:41:31 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:41:37 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:41:37 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:41:37 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 32425 ... Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Uses event mutexes Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:41:37 dbcl01-02 mysqld: 2019-11-21 8:41:37 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:41:43 dbcl01-02 mysqld: 2019-11-21 8:41:43 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:41:44 dbcl01-02 mysqld: 2019-11-21 8:41:44 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:41:44 dbcl01-02 mysqld: 2019-11-21 8:41:44 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873785764524 Nov 21 08:41:49 dbcl01-02 mysqld: 2019-11-21 8:41:49 0 [Note] InnoDB: Ignoring data file './db/XXXXXXXX_XXXX_XXXXXX.ibd' with space ID 19946, since the redo log references ./db/XXXXXXXX_XXXX_XXXXXX.ibd with space ID 19942. Nov 21 08:41:51 dbcl01-02 mysqld: 2019-11-21 8:41:51 0 [Note] InnoDB: Ignoring data file './db/XXXXXXXX_XXXX_XXXXXX.ibd' with space ID 19947, since the redo log references ./db/XXXXXXXX_XXXX_XXXXXX.ibd with space ID 19943. Nov 21 08:41:52 dbcl01-02 mysqld: 2019-11-21 8:41:52 0 [Note] InnoDB: Ignoring data file './db/XXXXXXXX_XXXX_XXXXXX.ibd' with space ID 19948, since the redo log references ./db/XXXXXXXX_XXXX_XXXXXX.ibd with space ID 19944. Nov 21 08:41:52 dbcl01-02 mysqld: 2019-11-21 8:41:52 0 [Note] InnoDB: Ignoring data file './db/XXXXXXXX_XXXX_XXXXXX.ibd' with space ID 19949, since the redo log references ./db/XXXXXXXX_XXXX_XXXXXX.ibd with space ID 19945. Nov 21 08:41:58 dbcl01-02 mysqld: 2019-11-21 8:41:58 0 [Note] InnoDB: Read redo log up to LSN=19872605073920 Nov 21 08:42:13 dbcl01-02 mysqld: 2019-11-21 8:42:13 0 [Note] InnoDB: Read redo log up to LSN=19873673245184 Nov 21 08:42:16 dbcl01-02 mysqld: 2019-11-21 8:42:16 0 [Note] InnoDB: Transaction 11233655361 was in the XA prepared state. Nov 21 08:42:19 dbcl01-02 mysqld: 2019-11-21 8:42:19 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo Nov 21 08:42:19 dbcl01-02 mysqld: 2019-11-21 8:42:19 0 [Note] InnoDB: Trx id counter is 11233655362 Nov 21 08:42:19 dbcl01-02 mysqld: 2019-11-21 8:42:19 0 [Note] InnoDB: Starting final batch to recover 155927 pages from redo log. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Starting in background the rollback of recovered transactions Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Rollback of non-prepared transactions completed Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873807438089; transaction id 11233655364 Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Starting crash recovery... Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Starting recovery for XA transactions... Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Transaction 11233655361 in prepared state after recovery Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: Transaction contains changes to 3 rows Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] InnoDB: 1 transactions in prepared state after recovery Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Found 1 prepared transaction(s) in InnoDB Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Crash recovery finished. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] Added new Master_info '' to hash table Nov 21 08:42:29 dbcl01-02 mysqld: 2019-11-21 8:42:29 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:42:29 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:42:29 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:46:59 dbcl01-02 mysqld: 191121 8:46:59 [ERROR] mysqld got signal 11 ; Nov 21 08:46:59 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:46:59 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:46:59 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:46:59 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:46:59 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:46:59 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:46:59 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:46:59 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:46:59 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:46:59 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:46:59 dbcl01-02 mysqld: max_used_connections=12 Nov 21 08:46:59 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:46:59 dbcl01-02 mysqld: thread_count=19 Nov 21 08:46:59 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:46:59 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:46:59 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:46:59 dbcl01-02 mysqld: Thread pointer: 0x7fbca00009a8 Nov 21 08:46:59 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:46:59 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:46:59 dbcl01-02 mysqld: terribly wrong... Nov 21 08:46:59 dbcl01-02 mysqld: stack_bottom = 0x7fbd20579d00 thread_stack 0x30000 Nov 21 08:46:59 dbcl01-02 kernel: mysqld[34517]: segfault at 19 ip 00005655547c39bf sp 00007fbd204af1c0 error 4 in mysqld[565553b08000+12c9000] Nov 21 08:47:00 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:47:00 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:47:00 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:47:05 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:47:05 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:47:05 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 34759 ... Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Uses event mutexes Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:47:06 dbcl01-02 mysqld: 2019-11-21 8:47:06 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:47:11 dbcl01-02 mysqld: 2019-11-21 8:47:11 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:47:12 dbcl01-02 mysqld: 2019-11-21 8:47:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:47:13 dbcl01-02 mysqld: 2019-11-21 8:47:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809072541 Nov 21 08:47:21 dbcl01-02 mysqld: 2019-11-21 8:47:21 0 [Note] InnoDB: Starting final batch to recover 72301 pages from redo log. Nov 21 08:47:25 dbcl01-02 mysqld: 2019-11-21 8:47:25 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809078660; transaction id 11233655367 Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Starting crash recovery... Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Crash recovery finished. Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] Added new Master_info '' to hash table Nov 21 08:47:26 dbcl01-02 mysqld: 2019-11-21 8:47:26 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:47:26 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:47:26 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:47:26 dbcl01-02 mysqld: 191121 8:47:26 [ERROR] mysqld got signal 11 ; Nov 21 08:47:26 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:47:26 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:47:26 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:47:26 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:47:26 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:47:26 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:47:26 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:47:26 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:47:26 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:47:26 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:47:26 dbcl01-02 mysqld: max_used_connections=35 Nov 21 08:47:26 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:47:26 dbcl01-02 mysqld: thread_count=42 Nov 21 08:47:26 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:47:26 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:47:26 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:47:26 dbcl01-02 mysqld: Thread pointer: 0x7f94e40009a8 Nov 21 08:47:26 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:47:26 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:47:26 dbcl01-02 mysqld: terribly wrong... Nov 21 08:47:26 dbcl01-02 kernel: mysqld[34835]: segfault at 19 ip 00005617613c09bf sp 00007fbf200ef1c0 error 4 in mysqld[561760705000+12c9000] Nov 21 08:47:27 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:47:27 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:47:27 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:47:32 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:47:32 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:47:32 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 35092 ... Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Uses event mutexes Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:47:33 dbcl01-02 mysqld: 2019-11-21 8:47:33 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:47:38 dbcl01-02 mysqld: 2019-11-21 8:47:38 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:47:39 dbcl01-02 mysqld: 2019-11-21 8:47:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:47:40 dbcl01-02 mysqld: 2019-11-21 8:47:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873314254965 Nov 21 08:47:54 dbcl01-02 mysqld: 2019-11-21 8:47:54 0 [Note] InnoDB: To recover: 72496 pages from log Nov 21 08:47:54 dbcl01-02 mysqld: 2019-11-21 8:47:54 0 [Note] InnoDB: Starting final batch to recover 72301 pages from redo log. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809078660; transaction id 11233655367 Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Starting crash recovery... Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Crash recovery finished. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] Added new Master_info '' to hash table Nov 21 08:47:58 dbcl01-02 mysqld: 2019-11-21 8:47:58 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:47:58 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:47:59 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:48:17 dbcl01-02 mysqld: 191121 8:48:17 [ERROR] mysqld got signal 11 ; Nov 21 08:48:17 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:48:17 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:48:17 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:48:17 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:48:17 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:48:17 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:48:17 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:48:17 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:48:17 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:48:17 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:48:17 dbcl01-02 mysqld: max_used_connections=5 Nov 21 08:48:17 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:48:17 dbcl01-02 mysqld: thread_count=12 Nov 21 08:48:17 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:48:17 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:48:17 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:48:17 dbcl01-02 mysqld: Thread pointer: 0x7f7ec40009a8 Nov 21 08:48:17 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:48:17 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:48:17 dbcl01-02 mysqld: terribly wrong... Nov 21 08:48:17 dbcl01-02 mysqld: stack_bottom = 0x7fa8cc19ed00 thread_stack 0x30000 Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55961df7c8be] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55961da1959f] Nov 21 08:48:17 dbcl01-02 mysqld: sigaction.c:0(__restore_rt)[0x7fa94a4ad5f0] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbb9bf)[0x55961df419bf] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcc75da)[0x55961df4d5da] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x55961da1e187] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbfe19)[0x55961df45e19] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT20init_ror_merged_scanEbP11st_mem_root+0x85)[0x55961db26e85] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT20init_ror_merged_scanEbP11st_mem_root+0x7d)[0x55961db2709d] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT5resetEv+0x5f)[0x55961db27e8f] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x48)[0x55961d881ad8] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x55961d8712d9] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x55961d892edb] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55961d8930f3] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55961d89324a] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x55961d893d5c] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(+0x4c2f4c)[0x55961d748f4c] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x668b)[0x55961d83e03b] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55961d84051b] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x55961d843106] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x55961d84404e] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x55961d918111] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55961d9181dd] Nov 21 08:48:17 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xca77ad)[0x55961df2d7ad] Nov 21 08:48:17 dbcl01-02 mysqld: pthread_create.c:0(start_thread)[0x7fa94a4a5e65] Nov 21 08:48:17 dbcl01-02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7fa94884688d] Nov 21 08:48:17 dbcl01-02 mysqld: Trying to get some variables. Nov 21 08:48:17 dbcl01-02 mysqld: Some pointers may be invalid and cause the dump to abort. Nov 21 08:48:17 dbcl01-02 kernel: mysqld[35467]: segfault at 19 ip 000055961df419bf sp 00007fa8cc19c1c0 error 4 in mysqld[55961d286000+12c9000] Nov 21 08:48:17 dbcl01-02 mysqld: Query (0x7f7ec400f550): SELECT COUNT(id) AS requests FROM api_log PARTITION(p11) WHERE kid='XXXX' AND auth_type='basic' AND auth_id='XXXX' AND created >= DATE_SUB(NOW(), INTERVAL 60 MINUTE) LIMIT 1 Nov 21 08:48:17 dbcl01-02 mysqld: Connection ID (thread ID): 19 Nov 21 08:48:17 dbcl01-02 mysqld: Status: NOT_KILLED Nov 21 08:48:17 dbcl01-02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Nov 21 08:48:17 dbcl01-02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Nov 21 08:48:17 dbcl01-02 mysqld: information that should help you find out what is causing the crash. Nov 21 08:48:17 dbcl01-02 mysqld: Writing a core file... Nov 21 08:48:17 dbcl01-02 mysqld: Working directory at /var/lib/mysql Nov 21 08:48:17 dbcl01-02 mysqld: Resource Limits: Nov 21 08:48:17 dbcl01-02 mysqld: Limit Soft Limit Hard Limit Units Nov 21 08:48:17 dbcl01-02 mysqld: Max cpu time unlimited unlimited seconds Nov 21 08:48:17 dbcl01-02 mysqld: Max file size unlimited unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max data size unlimited unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max stack size 8388608 unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max core file size 0 unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max resident set unlimited unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max processes 1030309 1030309 processes Nov 21 08:48:17 dbcl01-02 mysqld: Max open files 65535 65535 files Nov 21 08:48:17 dbcl01-02 mysqld: Max locked memory 65536 65536 bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max address space unlimited unlimited bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max file locks unlimited unlimited locks Nov 21 08:48:17 dbcl01-02 mysqld: Max pending signals 1030309 1030309 signals Nov 21 08:48:17 dbcl01-02 mysqld: Max msgqueue size 819200 819200 bytes Nov 21 08:48:17 dbcl01-02 mysqld: Max nice priority 0 0 Nov 21 08:48:17 dbcl01-02 mysqld: Max realtime priority 0 0 Nov 21 08:48:17 dbcl01-02 mysqld: Max realtime timeout unlimited unlimited us Nov 21 08:48:17 dbcl01-02 mysqld: Core pattern: core Nov 21 08:48:18 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:48:18 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:48:18 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:48:23 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:48:23 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:48:23 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 35715 ... Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Uses event mutexes Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:48:24 dbcl01-02 mysqld: 2019-11-21 8:48:24 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:48:29 dbcl01-02 mysqld: 2019-11-21 8:48:29 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:48:30 dbcl01-02 mysqld: 2019-11-21 8:48:30 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:48:31 dbcl01-02 mysqld: 2019-11-21 8:48:31 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809188786 Nov 21 08:48:37 dbcl01-02 mysqld: 2019-11-21 8:48:37 0 [Note] InnoDB: Starting final batch to recover 58353 pages from redo log. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809194905; transaction id 11233655367 Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Starting crash recovery... Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Crash recovery finished. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] Added new Master_info '' to hash table Nov 21 08:48:41 dbcl01-02 mysqld: 2019-11-21 8:48:41 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:48:41 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:48:41 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:48:43 dbcl01-02 mysqld: 191121 8:48:43 [ERROR] mysqld got signal 11 ; Nov 21 08:48:43 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:48:43 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:48:43 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:48:43 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:48:43 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:48:43 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:48:43 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:48:43 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:48:43 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:48:43 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:48:43 dbcl01-02 mysqld: max_used_connections=4 Nov 21 08:48:43 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:48:43 dbcl01-02 mysqld: thread_count=11 Nov 21 08:48:43 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:48:43 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:48:43 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:48:43 dbcl01-02 mysqld: Thread pointer: 0x7f2d900009a8 Nov 21 08:48:43 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:48:43 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:48:43 dbcl01-02 mysqld: terribly wrong... Nov 21 08:48:43 dbcl01-02 mysqld: stack_bottom = 0x7f57a00a4d00 thread_stack 0x30000 Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e9206cd8be] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x55e92016a59f] Nov 21 08:48:43 dbcl01-02 mysqld: sigaction.c:0(__restore_rt)[0x7f581faaf5f0] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbb9bf)[0x55e9206929bf] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcc75da)[0x55e92069e5da] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x55e92016f187] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbfe19)[0x55e920696e19] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT20init_ror_merged_scanEbP11st_mem_root+0x85)[0x55e920277e85] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT20init_ror_merged_scanEbP11st_mem_root+0x7d)[0x55e92027809d] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT5resetEv+0x5f)[0x55e920278e8f] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x48)[0x55e91ffd2ad8] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x55e91ffc22d9] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x55e91ffe3edb] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55e91ffe40f3] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x55e91ffe424a] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x55e91ffe4d5c] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(+0x4c2f4c)[0x55e91fe99f4c] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x668b)[0x55e91ff8f03b] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x55e91ff9151b] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x55e91ff94106] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x55e91ff9504e] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x55e920069111] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55e9200691dd] Nov 21 08:48:43 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xca77ad)[0x55e92067e7ad] Nov 21 08:48:43 dbcl01-02 mysqld: pthread_create.c:0(start_thread)[0x7f581faa7e65] Nov 21 08:48:44 dbcl01-02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f581de4888d] Nov 21 08:48:44 dbcl01-02 mysqld: Trying to get some variables. Nov 21 08:48:44 dbcl01-02 mysqld: Some pointers may be invalid and cause the dump to abort. Nov 21 08:48:44 dbcl01-02 mysqld: Query (0x7f2d9000f4f0): SELECT direction, COUNT(id) AS sr_all, SUM(success) AS sr_ack, MAX(duration_bill) AS talk_max, SUM(duration_bill) AS talk_sum, AVG(duration_bill) AS talk_avg, AVG(duration_ringing) AS ringing_avg FROM cdr_acd PARTITION(p11) WHERE kid='XXXX' AND agenten_id='XXXXX' AND date_time >= DATE_FORMAT(NOW(), '%Y-%m%-%d') AND ((direction='inbound' AND SUBSTRING(from_number, 1, 6) NOT IN('AGENT:','SKILL:') AND SUBSTRING(from_number, 1, 3) != '000') OR (direction='outbound' AND SUBSTRING(to_number, 1, 6) NOT IN('AGENT:','SKILL:') AND SUBSTRING(to_number, 1, 3) != '000')) GROUP BY direction Nov 21 08:48:44 dbcl01-02 mysqld: Connection ID (thread ID): 16 Nov 21 08:48:44 dbcl01-02 mysqld: Status: NOT_KILLED Nov 21 08:48:44 dbcl01-02 kernel: mysqld[35782]: segfault at 19 ip 000055e9206929bf sp 00007f57a00a21c0 error 4 in mysqld[55e91f9d7000+12c9000] Nov 21 08:48:44 dbcl01-02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Nov 21 08:48:44 dbcl01-02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Nov 21 08:48:44 dbcl01-02 mysqld: information that should help you find out what is causing the crash. Nov 21 08:48:44 dbcl01-02 mysqld: Writing a core file... Nov 21 08:48:44 dbcl01-02 mysqld: Working directory at /var/lib/mysql Nov 21 08:48:44 dbcl01-02 mysqld: Resource Limits: Nov 21 08:48:44 dbcl01-02 mysqld: Limit Soft Limit Hard Limit Units Nov 21 08:48:44 dbcl01-02 mysqld: Max cpu time unlimited unlimited seconds Nov 21 08:48:44 dbcl01-02 mysqld: Max file size unlimited unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max data size unlimited unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max stack size 8388608 unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max core file size 0 unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max resident set unlimited unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max processes 1030309 1030309 processes Nov 21 08:48:44 dbcl01-02 mysqld: Max open files 65535 65535 files Nov 21 08:48:44 dbcl01-02 mysqld: Max locked memory 65536 65536 bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max address space unlimited unlimited bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max file locks unlimited unlimited locks Nov 21 08:48:44 dbcl01-02 mysqld: Max pending signals 1030309 1030309 signals Nov 21 08:48:44 dbcl01-02 mysqld: Max msgqueue size 819200 819200 bytes Nov 21 08:48:44 dbcl01-02 mysqld: Max nice priority 0 0 Nov 21 08:48:44 dbcl01-02 mysqld: Max realtime priority 0 0 Nov 21 08:48:44 dbcl01-02 mysqld: Max realtime timeout unlimited unlimited us Nov 21 08:48:44 dbcl01-02 mysqld: Core pattern: core Nov 21 08:48:44 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:48:44 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:48:44 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:48:49 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:48:49 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:48:49 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 36031 ... Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Uses event mutexes Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:48:50 dbcl01-02 mysqld: 2019-11-21 8:48:50 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:48:55 dbcl01-02 systemd: Created slice User Slice of root. Nov 21 08:48:55 dbcl01-02 systemd-logind: New session 7664 of user root. Nov 21 08:48:55 dbcl01-02 systemd: Started Session 7664 of user root. Nov 21 08:48:55 dbcl01-02 mysqld: 2019-11-21 8:48:55 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:48:55 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 su: (to postgres) root on none Nov 21 08:48:56 dbcl01-02 systemd-logind: Removed session 7664. Nov 21 08:48:56 dbcl01-02 systemd: Removed slice User Slice of root. Nov 21 08:48:56 dbcl01-02 mysqld: 2019-11-21 8:48:56 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:48:57 dbcl01-02 mysqld: 2019-11-21 8:48:57 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809201024 Nov 21 08:49:03 dbcl01-02 mysqld: 2019-11-21 8:49:03 0 [Note] InnoDB: Starting final batch to recover 58338 pages from redo log. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809207143; transaction id 11233655367 Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Starting crash recovery... Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Crash recovery finished. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] Added new Master_info '' to hash table Nov 21 08:49:07 dbcl01-02 mysqld: 2019-11-21 8:49:07 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:49:07 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:49:07 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:49:08 dbcl01-02 mysqld: 191121 8:49:08 [ERROR] mysqld got signal 11 ; Nov 21 08:49:08 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:49:08 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:49:08 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:49:08 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:49:08 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:49:08 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:49:08 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:49:08 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:49:08 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:49:08 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:49:08 dbcl01-02 mysqld: max_used_connections=5 Nov 21 08:49:08 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:49:08 dbcl01-02 mysqld: thread_count=12 Nov 21 08:49:08 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:49:08 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:49:08 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:49:08 dbcl01-02 mysqld: Thread pointer: 0x7f51d00009a8 Nov 21 08:49:08 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:49:08 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:49:08 dbcl01-02 mysqld: terribly wrong... Nov 21 08:49:08 dbcl01-02 mysqld: stack_bottom = 0x7f7bc4172d00 thread_stack 0x30000 Nov 21 08:49:08 dbcl01-02 kernel: mysqld[36386]: segfault at 19 ip 00005608fcba09bf sp 00007f7bc40a81c0 error 4 in mysqld[5608fbee5000+12c9000] Nov 21 08:49:08 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:49:08 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:49:08 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:49:14 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:49:14 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:49:14 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 36621 ... Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Uses event mutexes Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:49:14 dbcl01-02 mysqld: 2019-11-21 8:49:14 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:49:20 dbcl01-02 mysqld: 2019-11-21 8:49:20 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:49:21 dbcl01-02 mysqld: 2019-11-21 8:49:21 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:49:21 dbcl01-02 mysqld: 2019-11-21 8:49:21 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873448796946 Nov 21 08:49:32 dbcl01-02 mysqld: 2019-11-21 8:49:32 0 [Note] InnoDB: Starting final batch to recover 58338 pages from redo log. Nov 21 08:49:35 dbcl01-02 mysqld: 2019-11-21 8:49:35 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809207143; transaction id 11233655367 Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Starting crash recovery... Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Crash recovery finished. Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] Added new Master_info '' to hash table Nov 21 08:49:36 dbcl01-02 mysqld: 2019-11-21 8:49:36 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:49:36 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:49:36 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:50:00 dbcl01-02 mysqld: 191121 8:50:00 [ERROR] mysqld got signal 11 ; Nov 21 08:50:00 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:50:00 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:50:00 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:50:00 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:50:00 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:50:00 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:50:00 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:50:00 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:50:00 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:50:00 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:50:00 dbcl01-02 mysqld: max_used_connections=4 Nov 21 08:50:00 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:50:00 dbcl01-02 mysqld: thread_count=11 Nov 21 08:50:00 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:50:00 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:50:00 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:50:00 dbcl01-02 mysqld: Thread pointer: 0x7fa5a40009a8 Nov 21 08:50:00 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:50:00 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:50:00 dbcl01-02 mysqld: terribly wrong... Nov 21 08:50:00 dbcl01-02 mysqld: stack_bottom = 0x7fcfb410ad00 thread_stack 0x30000 Nov 21 08:50:00 dbcl01-02 kernel: mysqld[37005]: segfault at 19 ip 000055a2db6f69bf sp 00007fcfb40a41c0 error 4 in mysqld[55a2daa3b000+12c9000] Nov 21 08:50:00 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:50:00 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:50:00 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:50:04 dbcl01-02 systemd: Created slice User Slice of root. Nov 21 08:50:04 dbcl01-02 systemd-logind: New session 7666 of user root. Nov 21 08:50:04 dbcl01-02 systemd: Started Session 7666 of user root. Nov 21 08:50:06 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:50:06 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:50:06 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 37258 ... Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Uses event mutexes Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:50:06 dbcl01-02 mysqld: 2019-11-21 8:50:06 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:50:12 dbcl01-02 mysqld: 2019-11-21 8:50:12 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:50:13 dbcl01-02 mysqld: 2019-11-21 8:50:13 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:50:13 dbcl01-02 mysqld: 2019-11-21 8:50:13 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809341745 Nov 21 08:50:17 dbcl01-02 mysqld: 2019-11-21 8:50:17 0 [Note] InnoDB: Starting final batch to recover 38034 pages from redo log. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809347864; transaction id 11233655367 Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Starting crash recovery... Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Crash recovery finished. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] Added new Master_info '' to hash table Nov 21 08:50:20 dbcl01-02 mysqld: 2019-11-21 8:50:20 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:50:20 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:50:20 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:50:26 dbcl01-02 mysqld: 191121 8:50:26 [ERROR] mysqld got signal 11 ; Nov 21 08:50:26 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:50:26 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:50:26 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:50:26 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:50:26 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:50:26 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:50:26 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:50:26 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:50:26 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:50:26 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:50:26 dbcl01-02 mysqld: max_used_connections=7 Nov 21 08:50:26 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:50:26 dbcl01-02 mysqld: thread_count=14 Nov 21 08:50:26 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:50:26 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:50:26 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:50:26 dbcl01-02 mysqld: Thread pointer: 0x7f49780009a8 Nov 21 08:50:26 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:50:26 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:50:26 dbcl01-02 mysqld: terribly wrong... Nov 21 08:50:26 dbcl01-02 mysqld: stack_bottom = 0x7f49e86fad00 thread_stack 0x30000 Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x563d999878be] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x563d9942459f] Nov 21 08:50:26 dbcl01-02 mysqld: sigaction.c:0(__restore_rt)[0x7f740d6b35f0] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbb9bf)[0x563d9994c9bf] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcc75da)[0x563d999585da] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x563d99429187] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbfe19)[0x563d99950e19] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT20init_ror_merged_scanEbP11st_mem_root+0x85)[0x563d99531e85] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT20init_ror_merged_scanEbP11st_mem_root+0x7d)[0x563d9953209d] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT5resetEv+0x5f)[0x563d99532e8f] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x48)[0x563d9928cad8] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x563d9927c2d9] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x563d9929dedb] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x563d9929e0f3] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x563d9929e24a] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x563d9929ed5c] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(+0x4c2f4c)[0x563d99153f4c] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x668b)[0x563d9924903b] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x563d9924b51b] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x563d9924e106] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x563d9924f04e] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x563d99323111] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x563d993231dd] Nov 21 08:50:26 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xca77ad)[0x563d999387ad] Nov 21 08:50:26 dbcl01-02 mysqld: pthread_create.c:0(start_thread)[0x7f740d6abe65] Nov 21 08:50:26 dbcl01-02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f740ba4c88d] Nov 21 08:50:26 dbcl01-02 mysqld: Trying to get some variables. Nov 21 08:50:26 dbcl01-02 mysqld: Some pointers may be invalid and cause the dump to abort. Nov 21 08:50:26 dbcl01-02 mysqld: Query (0x7f497800f470): SELECT COUNT(id) AS requests FROM api_log PARTITION(p11) WHERE kid='XXXX' AND auth_type='basic' AND auth_id='XXXX' AND created >= DATE_SUB(NOW(), INTERVAL 60 MINUTE) LIMIT 1 Nov 21 08:50:26 dbcl01-02 mysqld: Connection ID (thread ID): 48 Nov 21 08:50:26 dbcl01-02 mysqld: Status: NOT_KILLED Nov 21 08:50:26 dbcl01-02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Nov 21 08:50:26 dbcl01-02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Nov 21 08:50:26 dbcl01-02 mysqld: information that should help you find out what is causing the crash. Nov 21 08:50:26 dbcl01-02 mysqld: Writing a core file... Nov 21 08:50:26 dbcl01-02 mysqld: Working directory at /var/lib/mysql Nov 21 08:50:26 dbcl01-02 mysqld: Resource Limits: Nov 21 08:50:26 dbcl01-02 mysqld: Limit Soft Limit Hard Limit Units Nov 21 08:50:26 dbcl01-02 mysqld: Max cpu time unlimited unlimited seconds Nov 21 08:50:26 dbcl01-02 mysqld: Max file size unlimited unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max data size unlimited unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max stack size 8388608 unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max core file size 0 unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max resident set unlimited unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max processes 1030309 1030309 processes Nov 21 08:50:26 dbcl01-02 mysqld: Max open files 65535 65535 files Nov 21 08:50:26 dbcl01-02 mysqld: Max locked memory 65536 65536 bytes Nov 21 08:50:26 dbcl01-02 kernel: mysqld[37328]: segfault at 19 ip 0000563d9994c9bf sp 00007f49e86f81c0 error 4 in mysqld[563d98c91000+12c9000] Nov 21 08:50:26 dbcl01-02 mysqld: Max address space unlimited unlimited bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max file locks unlimited unlimited locks Nov 21 08:50:26 dbcl01-02 mysqld: Max pending signals 1030309 1030309 signals Nov 21 08:50:26 dbcl01-02 mysqld: Max msgqueue size 819200 819200 bytes Nov 21 08:50:26 dbcl01-02 mysqld: Max nice priority 0 0 Nov 21 08:50:26 dbcl01-02 mysqld: Max realtime priority 0 0 Nov 21 08:50:26 dbcl01-02 mysqld: Max realtime timeout unlimited unlimited us Nov 21 08:50:26 dbcl01-02 mysqld: Core pattern: core Nov 21 08:50:27 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:50:27 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:50:27 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:50:32 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:50:32 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:50:32 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 37577 ... Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Uses event mutexes Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:50:33 dbcl01-02 mysqld: 2019-11-21 8:50:33 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:50:38 dbcl01-02 mysqld: 2019-11-21 8:50:38 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:50:39 dbcl01-02 mysqld: 2019-11-21 8:50:39 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:50:40 dbcl01-02 mysqld: 2019-11-21 8:50:40 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809372340 Nov 21 08:50:43 dbcl01-02 mysqld: 2019-11-21 8:50:43 0 [Note] InnoDB: Starting final batch to recover 34705 pages from redo log. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809378459; transaction id 11233655367 Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Starting crash recovery... Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Crash recovery finished. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] Added new Master_info '' to hash table Nov 21 08:50:46 dbcl01-02 mysqld: 2019-11-21 8:50:46 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:50:46 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:50:46 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:50:52 dbcl01-02 mysqld: 191121 8:50:52 [ERROR] mysqld got signal 11 ; Nov 21 08:50:52 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:50:52 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:50:52 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:50:52 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:50:52 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:50:52 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:50:52 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:50:52 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:50:52 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:50:52 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:50:52 dbcl01-02 mysqld: max_used_connections=15 Nov 21 08:50:52 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:50:52 dbcl01-02 mysqld: thread_count=22 Nov 21 08:50:52 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:50:52 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:50:52 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:50:52 dbcl01-02 mysqld: Thread pointer: 0x7f6ce00009a8 Nov 21 08:50:52 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:50:52 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:50:52 dbcl01-02 mysqld: terribly wrong... Nov 21 08:50:52 dbcl01-02 mysqld: stack_bottom = 0x7f6d70067d00 thread_stack 0x30000 Nov 21 08:50:52 dbcl01-02 kernel: mysqld[37648]: segfault at 19 ip 0000559e954409bf sp 00007f9714105d70 error 4 in mysqld[559e94785000+12c9000] Nov 21 08:50:53 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:50:53 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:50:53 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:50:58 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:50:58 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:50:58 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 38181 ... Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Uses event mutexes Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:50:59 dbcl01-02 mysqld: 2019-11-21 8:50:59 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:51:04 dbcl01-02 mysqld: 2019-11-21 8:51:04 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:51:05 dbcl01-02 mysqld: 2019-11-21 8:51:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:51:06 dbcl01-02 mysqld: 2019-11-21 8:51:06 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809402935 Nov 21 08:51:10 dbcl01-02 mysqld: 2019-11-21 8:51:10 0 [Note] InnoDB: Starting final batch to recover 30655 pages from redo log. Nov 21 08:51:12 dbcl01-02 mysqld: 2019-11-21 8:51:12 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809409054; transaction id 11233655367 Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Starting crash recovery... Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Crash recovery finished. Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] Added new Master_info '' to hash table Nov 21 08:51:13 dbcl01-02 mysqld: 2019-11-21 8:51:13 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:51:13 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:51:13 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:51:18 dbcl01-02 mysqld: 191121 8:51:18 [ERROR] mysqld got signal 11 ; Nov 21 08:51:18 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:51:18 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:51:18 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:51:18 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:51:18 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:51:18 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:51:18 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:51:18 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:51:18 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:51:18 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:51:18 dbcl01-02 mysqld: max_used_connections=22 Nov 21 08:51:18 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:51:18 dbcl01-02 mysqld: thread_count=30 Nov 21 08:51:18 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:51:18 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:51:18 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:51:18 dbcl01-02 mysqld: Thread pointer: 0x7f5af40009a8 Nov 21 08:51:18 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:51:18 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:51:18 dbcl01-02 mysqld: terribly wrong... Nov 21 08:51:18 dbcl01-02 mysqld: stack_bottom = 0x7f5c60190d00 thread_stack 0x30000 Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560a631578be] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x560a62bf459f] Nov 21 08:51:19 dbcl01-02 mysqld: sigaction.c:0(__restore_rt)[0x7f86864555f0] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbb9bf)[0x560a6311c9bf] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcc75da)[0x560a631285da] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x560a62bf9187] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbfe19)[0x560a63120e19] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT20init_ror_merged_scanEbP11st_mem_root+0x85)[0x560a62d01e85] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT20init_ror_merged_scanEbP11st_mem_root+0x7d)[0x560a62d0209d] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT5resetEv+0x5f)[0x560a62d02e8f] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x48)[0x560a62a5cad8] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x560a62a4c2d9] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x560a62a6dedb] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x560a62a6e0f3] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x560a62a6e24a] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x560a62a6ed5c] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0x4c2f4c)[0x560a62923f4c] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x668b)[0x560a62a1903b] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x560a62a1b51b] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x560a62a1e106] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x560a62a1f04e] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x560a62af3111] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x560a62af31dd] Nov 21 08:51:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xca77ad)[0x560a631087ad] Nov 21 08:51:19 dbcl01-02 mysqld: pthread_create.c:0(start_thread)[0x7f868644de65] Nov 21 08:51:19 dbcl01-02 kernel: mysqld[38259]: segfault at 19 ip 0000560a6311c9bf sp 00007f5c603501c0 error 4 in mysqld[560a62461000+12c9000] Nov 21 08:51:20 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:51:20 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:51:20 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:51:24 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:57:05 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 40508 ... Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Uses event mutexes Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:57:05 dbcl01-02 mysqld: 2019-11-21 8:57:05 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:57:11 dbcl01-02 mysqld: 2019-11-21 8:57:11 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:57:12 dbcl01-02 mysqld: 2019-11-21 8:57:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:57:12 dbcl01-02 mysqld: 2019-11-21 8:57:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873809439633 Nov 21 08:57:15 dbcl01-02 mysqld: 2019-11-21 8:57:15 0 [Note] InnoDB: Starting final batch to recover 24935 pages from redo log. Nov 21 08:57:17 dbcl01-02 mysqld: 2019-11-21 8:57:17 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809445752; transaction id 11233655367 Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Starting crash recovery... Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Crash recovery finished. Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] Added new Master_info '' to hash table Nov 21 08:57:18 dbcl01-02 mysqld: 2019-11-21 8:57:18 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:57:18 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:57:18 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 08:57:18 dbcl01-02 mysqld: 191121 8:57:18 [ERROR] mysqld got signal 11 ; Nov 21 08:57:18 dbcl01-02 mysqld: This could be because you hit a bug. It is also possible that this binary Nov 21 08:57:18 dbcl01-02 mysqld: or one of the libraries it was linked against is corrupt, improperly built, Nov 21 08:57:18 dbcl01-02 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. Nov 21 08:57:18 dbcl01-02 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs Nov 21 08:57:18 dbcl01-02 mysqld: We will try our best to scrape up some info that will hopefully help Nov 21 08:57:18 dbcl01-02 mysqld: diagnose the problem, but since we have already crashed, Nov 21 08:57:18 dbcl01-02 mysqld: something is definitely wrong and this may fail. Nov 21 08:57:18 dbcl01-02 mysqld: Server version: 10.3.20-MariaDB-log Nov 21 08:57:18 dbcl01-02 mysqld: key_buffer_size=134217728 Nov 21 08:57:18 dbcl01-02 mysqld: read_buffer_size=2097152 Nov 21 08:57:18 dbcl01-02 mysqld: max_used_connections=8 Nov 21 08:57:18 dbcl01-02 mysqld: max_threads=5002 Nov 21 08:57:18 dbcl01-02 mysqld: thread_count=15 Nov 21 08:57:18 dbcl01-02 mysqld: It is possible that mysqld could use up to Nov 21 08:57:18 dbcl01-02 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20731864 K bytes of memory Nov 21 08:57:18 dbcl01-02 mysqld: Hope that's ok; if not, decrease some variables in the equation. Nov 21 08:57:18 dbcl01-02 mysqld: Thread pointer: 0x7f394c0009a8 Nov 21 08:57:18 dbcl01-02 mysqld: Attempting backtrace. You can use the following information to find out Nov 21 08:57:18 dbcl01-02 mysqld: where mysqld died. If you see no messages after this, something went Nov 21 08:57:18 dbcl01-02 mysqld: terribly wrong... Nov 21 08:57:18 dbcl01-02 mysqld: stack_bottom = 0x7f633011cd00 thread_stack 0x30000 Nov 21 08:57:18 dbcl01-02 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x562e249f28be] Nov 21 08:57:18 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x562e2448f59f] Nov 21 08:57:19 dbcl01-02 mysqld: sigaction.c:0(__restore_rt)[0x7f63b1b945f0] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbb9bf)[0x562e249b79bf] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcc75da)[0x562e249c35da] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcijP11st_mem_rootP4ListI6StringE+0x47)[0x562e24494187] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xcbfe19)[0x562e249bbe19] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT20init_ror_merged_scanEbP11st_mem_root+0x85)[0x562e2459ce85] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT20init_ror_merged_scanEbP11st_mem_root+0x7d)[0x562e2459d09d] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN26QUICK_ROR_INTERSECT_SELECT5resetEv+0x5f)[0x562e2459de8f] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x48)[0x562e242f7ad8] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x169)[0x562e242e72d9] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa7b)[0x562e24308edb] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x562e243090f3] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x11a)[0x562e2430924a] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cc)[0x562e24309d5c] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0x4c2f4c)[0x562e241bef4c] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x668b)[0x562e242b403b] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22b)[0x562e242b651b] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1ca6)[0x562e242b9106] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x13e)[0x562e242ba04e] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x221)[0x562e2438e111] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x562e2438e1dd] Nov 21 08:57:19 dbcl01-02 mysqld: /usr/sbin/mysqld(+0xca77ad)[0x562e249a37ad] Nov 21 08:57:19 dbcl01-02 mysqld: pthread_create.c:0(start_thread)[0x7f63b1b8ce65] Nov 21 08:57:19 dbcl01-02 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f63aff2d88d] Nov 21 08:57:19 dbcl01-02 mysqld: Trying to get some variables. Nov 21 08:57:19 dbcl01-02 mysqld: Some pointers may be invalid and cause the dump to abort. Nov 21 08:57:19 dbcl01-02 mysqld: Query (0x7f394c00f530): SELECT direction, COUNT(id) AS sr_all, SUM(success) AS sr_ack, MAX(duration_bill) AS talk_max, SUM(duration_bill) AS talk_sum, AVG(duration_bill) AS talk_avg, AVG(duration_ringing) AS ringing_avg FROM cdr_acd PARTITION(p11) WHERE kid='XXXX' AND agenten_id='XXXXX' AND date_time >= DATE_FORMAT(NOW(), '%Y-%m%-%d') AND ((direction='inbound' AND SUBSTRING(from_number, 1, 6) NOT IN('AGENT:','SKILL:') AND SUBSTRING(from_number, 1, 3) != '000') OR (direction='outbound' AND SUBSTRING(to_number, 1, 6) NOT IN('AGENT:','SKILL:') AND SUBSTRING(to_number, 1, 3) != '000')) GROUP BY direction Nov 21 08:57:19 dbcl01-02 mysqld: Connection ID (thread ID): 52 Nov 21 08:57:19 dbcl01-02 mysqld: Status: NOT_KILLED Nov 21 08:57:19 dbcl01-02 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on Nov 21 08:57:19 dbcl01-02 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Nov 21 08:57:19 dbcl01-02 mysqld: information that should help you find out what is causing the crash. Nov 21 08:57:19 dbcl01-02 mysqld: Writing a core file... Nov 21 08:57:19 dbcl01-02 mysqld: Working directory at /var/lib/mysql Nov 21 08:57:19 dbcl01-02 mysqld: Resource Limits: Nov 21 08:57:19 dbcl01-02 mysqld: Limit Soft Limit Hard Limit Units Nov 21 08:57:19 dbcl01-02 mysqld: Max cpu time unlimited unlimited seconds Nov 21 08:57:19 dbcl01-02 mysqld: Max file size unlimited unlimited bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max data size unlimited unlimited bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max stack size 8388608 unlimited bytes Nov 21 08:57:19 dbcl01-02 kernel: mysqld[40567]: segfault at 19 ip 0000562e249b79bf sp 00007f633011a1c0 error 4 in mysqld[562e23cfc000+12c9000] Nov 21 08:57:19 dbcl01-02 mysqld: Max core file size 0 unlimited bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max resident set unlimited unlimited bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max processes 1030309 1030309 processes Nov 21 08:57:19 dbcl01-02 mysqld: Max open files 65535 65535 files Nov 21 08:57:19 dbcl01-02 mysqld: Max locked memory 65536 65536 bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max address space unlimited unlimited bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max file locks unlimited unlimited locks Nov 21 08:57:19 dbcl01-02 mysqld: Max pending signals 1030309 1030309 signals Nov 21 08:57:19 dbcl01-02 mysqld: Max msgqueue size 819200 819200 bytes Nov 21 08:57:19 dbcl01-02 mysqld: Max nice priority 0 0 Nov 21 08:57:19 dbcl01-02 mysqld: Max realtime priority 0 0 Nov 21 08:57:19 dbcl01-02 mysqld: Max realtime timeout unlimited unlimited us Nov 21 08:57:19 dbcl01-02 mysqld: Core pattern: core Nov 21 08:57:20 dbcl01-02 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV Nov 21 08:57:20 dbcl01-02 systemd: Unit mariadb.service entered failed state. Nov 21 08:57:20 dbcl01-02 systemd: mariadb.service failed. Nov 21 08:57:25 dbcl01-02 systemd: mariadb.service holdoff time over, scheduling restart. Nov 21 08:57:25 dbcl01-02 systemd: Stopped MariaDB 10.3.20 database server. Nov 21 08:57:25 dbcl01-02 systemd: Starting MariaDB 10.3.20 database server... Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] /usr/sbin/mysqld (mysqld 10.3.20-MariaDB-log) starting as process 40820 ... Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Using Linux native AIO Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Uses event mutexes Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Compressed tables use zlib 1.2.7 Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Number of pools: 1 Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Using SSE2 crc32 instructions Nov 21 08:57:25 dbcl01-02 mysqld: 2019-11-21 8:57:25 0 [Note] InnoDB: Initializing buffer pool, total size = 160G, instances = 64, chunk size = 128M Nov 21 08:57:31 dbcl01-02 mysqld: 2019-11-21 8:57:31 0 [Note] InnoDB: Completed initialization of buffer pool Nov 21 08:57:32 dbcl01-02 mysqld: 2019-11-21 8:57:32 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). Nov 21 08:57:32 dbcl01-02 mysqld: 2019-11-21 8:57:32 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=19873691312592 Nov 21 08:57:36 dbcl01-02 mysqld: 2019-11-21 8:57:36 0 [Note] InnoDB: Starting final batch to recover 24935 pages from redo log. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Last binlog file '/var/lib/mysql/binlog-dbcl01-03.000392', position 45371881 Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Waiting for purge to start Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: 10.3.20 started; log sequence number 19873809451871; transaction id 11233655367 Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Plugin 'FEEDBACK' is disabled. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Recovering after a crash using /var/lib/mysql/binlog-dbcl01-03 Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Starting crash recovery... Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Crash recovery finished. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Server socket created on IP: '1X.XXX.XXX.82'. Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 8 [Note] Reading Master_info: 'master-from_dbcl01@002d01.info' Relay_info:'relay-log-from_dbcl01@002d01.info' Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 8 [Note] Initialized Master_info from 'master-from_dbcl01@002d01.info' Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 8 [Note] Added new Master_info 'from_dbcl01-01' to hash table Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Reading of all Master_info entries succeeded Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] Added new Master_info '' to hash table Nov 21 08:57:39 dbcl01-02 mysqld: 2019-11-21 8:57:39 0 [Note] /usr/sbin/mysqld: ready for connections. Nov 21 08:57:39 dbcl01-02 mysqld: Version: '10.3.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server Nov 21 08:57:39 dbcl01-02 systemd: Started MariaDB 10.3.20 database server. Nov 21 09:00:36 dbcl01-02 mysqld: 2019-11-21 9:00:36 11113 [Note] Master 'from_dbcl01-01': Slave I/O thread: Start asynchronous replication to master 'slave@1X.XXX.XXX.81:3306' in log 'binlog-m1.001897' at position 4 Nov 21 09:00:36 dbcl01-02 mysqld: 2019-11-21 9:00:36 11113 [Note] Master 'from_dbcl01-01': Slave I/O thread: connected to master 'slave@1X.XXX.XXX.81:3306',replication starts at GTID position '1-81-85182247,0-51-4941698647,2-86-2' Nov 21 09:00:36 dbcl01-02 mysqld: 2019-11-21 9:00:36 11114 [Note] Master 'from_dbcl01-01': Slave SQL thread initialized, starting replication in log 'binlog-m1.001897' at position 4, relay log '/var/lib/mysql/relay-bin-dbcl01-03-from_dbcl01@002d01.000001' position: 4; GTID position '0-51-4941698647,1-81-85182247,2-86-2'