[MDEV-21811] MariaDB not starting Created: 2020-02-24  Updated: 2023-04-14  Resolved: 2023-04-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Dioni Alves de Oliveira Assignee: Marko Mäkelä
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

Hots with Debian 10


Issue Links:
Relates
relates to MDEV-24449 Corruption of system tablespace or la... Closed

 Description   

Sorry about my English

Today in the morning I noticed that my zabbix server was not working. Initially having a problem with the server, I tried to restart the server or even restart the service, but it did not return to operation.

Analyzing the log, something new came to me and I can't find the problem:

2020-02-24 13:18:12 0 [Note] InnoDB: Using Linux native AIO
2020-02-24 13:18:12 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-02-24 13:18:12 0 [Note] InnoDB: Uses event mutexes
2020-02-24 13:18:12 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-02-24 13:18:12 0 [Note] InnoDB: Number of pools: 1
2020-02-24 13:18:12 0 [Note] InnoDB: Using generic crc32 instructions
2020-02-24 13:18:12 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-02-24 13:18:12 0 [Note] InnoDB: Completed initialization of buffer pool
2020-02-24 13:18:12 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-02-24 13:18:12 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=644349686286
2020-02-24 13:18:12 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 734 row operations to undo
2020-02-24 13:18:12 0 [Note] InnoDB: Trx id counter is 139251516
2020-02-24 13:18:12 0 [Note] InnoDB: Starting final batch to recover 2535 pages from redo log.
2020-02-24 13:18:12 0 [ERROR] [FATAL] InnoDB: is_short 1, info_and_status_bits 2, offset 99, o_offset 5, mismatch index 18446744073709551610, end_seg_len 19 parsed len 1
200224 13:18:12 [ERROR] mysqld got signal 6 ;

Alguem poderia me ajudar?
I have no backup of the bank.



 Comments   
Comment by Elena Stepanova [ 2020-02-29 ]

Which version is it? Please provide the rest of the error log, the part after 'signal 6' and the error log from the previous server run, which must have ended in a crash (since your server is performing crash recovery).

Comment by Dioni Alves de Oliveira [ 2020-03-10 ]

I haven't been able to solve the problem with the bank yet, zabbix is still on par. Below full log.

Complet log:

2020-03-10  8:57:27 0 [Note] InnoDB: Using Linux native AIO
2020-03-10  8:57:27 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-03-10  8:57:27 0 [Note] InnoDB: Uses event mutexes
2020-03-10  8:57:27 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-03-10  8:57:27 0 [Note] InnoDB: Number of pools: 1
2020-03-10  8:57:27 0 [Note] InnoDB: Using generic crc32 instructions
2020-03-10  8:57:27 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-03-10  8:57:27 0 [Note] InnoDB: Completed initialization of buffer pool
2020-03-10  8:57:27 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-03-10  8:57:27 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=644349686286
2020-03-10  8:57:27 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 734 row operations to undo
2020-03-10  8:57:27 0 [Note] InnoDB: Trx id counter is 139251516
2020-03-10  8:57:27 0 [Note] InnoDB: Starting final batch to recover 2535 pages from redo log.
2020-03-10  8:57:27 0 [ERROR] [FATAL] InnoDB: is_short 1, info_and_status_bits 2, offset 99, o_offset 5, mismatch index 18446744073709551610, end_seg_len 19 parsed len 1
200310  8:57:27 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.22-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467423 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d25670391e]
/usr/sbin/mysqld(handle_fatal_signal+0x54d)[0x55d2562339ed]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f5a885a6730]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b)[0x7f5a87b637bb]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x121)[0x7f5a87b4e535]
/usr/sbin/mysqld(+0x4d9a59)[0x55d255f7aa59]
/usr/sbin/mysqld(+0x4bfda2)[0x55d255f60da2]
/usr/sbin/mysqld(+0x973d29)[0x55d256414d29]
/usr/sbin/mysqld(+0x95c4e9)[0x55d2563fd4e9]
/usr/sbin/mysqld(+0x95d2ef)[0x55d2563fe2ef]
/usr/sbin/mysqld(+0x95ebfe)[0x55d2563ffbfe]
/usr/sbin/mysqld(+0x4d2d03)[0x55d255f73d03]
/usr/sbin/mysqld(+0x9027d4)[0x55d2563a37d4]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x68)[0x55d2562360b8]
/usr/sbin/mysqld(+0x5d0f59)[0x55d256071f59]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x98a)[0x55d2560730da]
/usr/sbin/mysqld(+0x512951)[0x55d255fb3951]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x3f9)[0x55d255fba509]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f5a87b5009b]
/usr/sbin/mysqld(_start+0x2a)[0x55d255fada2a]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             64054                64054                processes
Max open files            16364                16364                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       64054                64054                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: core

Comment by Marko Mäkelä [ 2021-04-15 ]

The server is intentionally killed in page_cur_parse_insert_rec() because the log record does not match the page contents.

A possible cause of this could be a race condition in InnoDB crash recovery, affecting all versions from MySQL 3.23 till the fix of MDEV-24449.

MariaDB Server 10.5 should never have been affected by this thanks to MDEV-19514, and even if it were, the symptoms should be different because the redo log format and related code was changed in MDEV-12353 and MDEV-21724.

Generated at Thu Feb 08 09:09:59 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.