Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.3.14
-
None
-
CentOS 7.8
Description
Since yesterday mariadb suddenly failed to start on our entire mariadb cluster.
We are using master-slave replication with GTID's and master-master replication between 2 datacenters.
This the error log of one of the slaves after I was able to start the master server with innodb_force_recovery = 6 set.
2020-05-28 5:58:27 0 [Note] InnoDB: Using Linux native AIO
|
2020-05-28 5:58:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
|
2020-05-28 5:58:27 0 [Note] InnoDB: Uses event mutexes
|
2020-05-28 5:58:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.7
|
2020-05-28 5:58:27 0 [Note] InnoDB: Number of pools: 1
|
2020-05-28 5:58:27 0 [Note] InnoDB: Using SSE2 crc32 instructions
|
2020-05-28 5:58:27 0 [Note] InnoDB: Initializing buffer pool, total size = 8G, instances = 4, chunk size = 128M
|
2020-05-28 5:58:28 0 [Note] InnoDB: Completed initialization of buffer pool
|
2020-05-28 5:58:28 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2020-05-28 5:58:28 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1395344903391
|
2020-05-28 5:58:34 0 [Note] InnoDB: Starting final batch to recover 81885 pages from redo log.
|
2020-05-28 5:58:37 0 [Note] InnoDB: Last binlog file './mariadb-bin.003482', position 1688
|
2020-05-28 5:58:38 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
|
2020-05-28 5:58:38 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
|
2020-05-28 5:58:38 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2020-05-28 5:58:38 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2020-05-28 5:58:38 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2020-05-28 5:58:38 0 [Note] InnoDB: 10.3.14 started; log sequence number 1395538094571; transaction id 766830539
|
2020-05-28 5:58:38 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
|
2020-05-28 5:58:38 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2020-05-28 5:58:38 0 [Note] Recovering after a crash using mariadb-bin
|
2020-05-28 5:58:38 0 [Note] Starting crash recovery...
|
2020-05-28 5:58:38 0 [Note] Crash recovery finished.
|
2020-05-28 05:58:38 0x7f09f2dcd700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.14/storage/innobase/rem/rem0rec.cc line 814
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mysqld startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
200528 5:58:38 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.3.14-MariaDB-log
|
key_buffer_size=16777216
|
read_buffer_size=131072
|
max_used_connections=0
|
max_threads=1002
|
thread_count=6
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2219062 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
Thread pointer: 0x7f09b40009a8
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f09f2dccbf0 thread_stack 0x40000
|
2020-05-28 5:58:38 0 [Note] Server socket created on IP: '0.0.0.0'.
|
2020-05-28 5:58:38 0 [Note] Reading of all Master_info entries succeded
|
2020-05-28 5:58:38 0 [Note] Added new Master_info '' to hash table
|
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x5610ac707dfe]
|
/usr/sbin/mysqld(handle_fatal_signal+0x357)[0x5610ac1a7cc7]
|
2020-05-28 5:58:38 10 [Note] Slave I/O thread: Start asynchronous replication to master 'slave@master:3306' in log 'mariadb-bin.001462' at position 424
|
2020-05-28 5:58:38 11 [Note] Slave SQL thread initialized, starting replication in log 'mariadb-bin.001462' at position 424, relay log './mariadb-relay-bin.000001' position: 4; GTID position '1-3117982927-205587455,2-2811022535-23281327,100-1-140363414'
|
2020-05-28 5:58:38 0 [Note] /usr/sbin/mysqld: ready for connections.
|
Version: '10.3.14-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
2020-05-28 5:58:38 10 [Note] Slave I/O thread: connected to master 'slave@master:3306',replication starts at GTID position '1-3117982927-205587455,2-2811022535-23281327,100-1-140363414'
|
sigaction.c:0(__restore_rt)[0x7f0c68771630]
|
:0(__GI_raise)[0x7f0c66a42387]
|
:0(__GI_abort)[0x7f0c66a43a78]
|
/usr/sbin/mysqld(+0x4c4b3a)[0x5610abee9b3a]
|
/usr/sbin/mysqld(+0x9b83c3)[0x5610ac3dd3c3]
|
/usr/sbin/mysqld(+0x99576c)[0x5610ac3ba76c]
|
/usr/sbin/mysqld(+0xa91413)[0x5610ac4b6413]
|
/usr/sbin/mysqld(+0x9f92cb)[0x5610ac41e2cb]
|
/usr/sbin/mysqld(+0x9fa8e7)[0x5610ac41f8e7]
|
/usr/sbin/mysqld(+0x9f4d52)[0x5610ac419d52]
|
/usr/sbin/mysqld(+0x9f76d3)[0x5610ac41c6d3]
|
/usr/sbin/mysqld(+0x9f859b)[0x5610ac41d59b]
|
/usr/sbin/mysqld(+0x9b3d7a)[0x5610ac3d8d7a]
|
/usr/sbin/mysqld(+0xa3eb88)[0x5610ac463b88]
|
/usr/sbin/mysqld(+0xa22de2)[0x5610ac447de2]
|
pthread_create.c:0(start_thread)[0x7f0c68769ea5]
|
/lib64/libc.so.6(clone+0x6d)[0x7f0c66b0a8dd]
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x0):
|
Connection ID (thread ID): 1
|
Status: NOT_KILLED
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
|
|
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
information that should help you find out what is causing the crash.
|
We have rebuild the entire cluster from one slave that was still working, but a few hours later the same thing happened again.
Each server has 16GB of memory, so the notice about the required memory keyspace isn't the issue.
Attachments
Issue Links
- duplicates
-
MDEV-21098 Crash in rec_get_offsets_func() due to invalid rec_get_status()
- Closed
- relates to
-
MDEV-23432 MariaDB (mysql) crash with 80000003 in several distinct installations
- Closed