[MDEV-28975] AWS RDS mariadb 10.5.12 crashes upon OS update Created: 2022-06-29  Updated: 2022-06-29  Resolved: 2022-06-29

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.12
Fix Version/s: 10.6.9, 10.7.5, 10.8.4, 10.9.2, 10.10.0

Type: Bug Priority: Major
Reporter: TAO ZHOU Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 0
Labels: corruption, crash
Environment:

AWS


Issue Links:
Relates
relates to MDEV-13542 Crashing on a corrupted page is unhel... Closed
relates to MDEV-22739 !cursor->index->is_committed() in row... Closed
relates to MDEV-26917 InnoDB: Clustered record for sec rec ... Closed
relates to MDEV-27734 Set innodb_change_buffering=none by d... Closed

 Description   

My RDS instance was showing `incompatible parameter` error on the console; users getting connection refused. I followed this to reset the parameters to default and rebooted, but still getting same error. Then I saw there's an OS update available, so I hit the `upgrade now` button. Then it starts crashing, never booted again.

here's the logs

2022-06-24 2:59:54 0 [Warning] You need to use --log-bin to make --log-slave-updates work.
2022-06-24 2:59:54 0 [Note] InnoDB: Uses event mutexes
2022-06-24 2:59:54 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-06-24 2:59:54 0 [Note] InnoDB: Number of pools: 1
2022-06-24 2:59:54 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-06-24 2:59:54 0 [Note] InnoDB: Using Linux native AIO
2022-06-24 2:59:54 0 [Note] InnoDB: Initializing buffer pool, total size = 2147483648, chunk size = 134217728
2022-06-24 2:59:54 0 [Note] InnoDB: Completed initialization of buffer pool
2022-06-24 2:59:54 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1343302636904,1343302636904
2022-06-24 3:00:00 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
2022-06-24 3:00:00 0 [Note] InnoDB: Trx id counter is 8615938331
2022-06-24 3:00:00 0 [Note] InnoDB: Starting final batch to recover 25079 pages from redo log.
2022-06-24 3:00:16 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.667099', position 4312
2022-06-24 3:00:16 0 [Note] InnoDB: 128 rollback segments are active.
2022-06-24 3:00:16 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-06-24 3:00:16 0 [Note] InnoDB: To roll back: 2 transactions, 2 rows
2022-06-24 3:00:16 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-06-24 3:00:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-06-24 3:00:16 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-06-24 3:00:16 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
2022-06-24 3:00:16 0 [ERROR] [FATAL] InnoDB: Page old data size 8606 new data size 5454, page old max ins size 7544 new max ins size 10696
220624 3:00:16 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.5.12-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11660859 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2022-06-24 3:00:16 0 [Note] InnoDB: 10.5.12 started; log sequence number 1343572670349; transaction id 8615938332
2022-06-24 3:00:16 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
2022-06-24 3:00:16 0 [Note] Server socket created on IP: '::'.
2022-06-24 3:00:16 0 [Note] Reading of all Master_info entries succeeded
2022-06-24 3:00:16 0 [Note] Added new Master_info '' to hash table
2022-06-24 3:00:16 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
Version: '10.5.12-MariaDB' socket: '/tmp/mysql.sock' port: 3306 managed by https://aws.amazon.com/rds/
mysys/stacktrace.c:213(my_print_stacktrace)[0x55b7e23a1c5e]
sql/signal_handler.cc:221(handle_fatal_signal)[0x55b7e1b5376a]
sigaction.c:0(__restore_rt)[0x2b656a1e6100]
:0(__GI_raise)[0x2b656a4285f7]
:0(__GI_abort)[0x2b656a429ce8]
:0(ib::fatal::~fatal())[0x55b7e21f57d1]
btr/btr0btr.cc:1484(btr_page_reorganize_low)[0x55b7e21fb246]
btr/btr0btr.cc:1646(btr_page_reorganize(page_cur_t*, dict_index_t*, mtr_t*))[0x55b7e21fb617]
include/dict0mem.h:1840(dict_table_t::not_redundant() const)[0x55b7e20e0f93]
ibuf/ibuf0ibuf.cc:3881(ibuf_insert_to_index_page)[0x55b7e20e6a11]
buf/buf0buf.cc:3339(buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x55b7e223052b]
buf/buf0buf.cc:3421(buf_page_get_gen(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x55b7e2230576]
btr/btr0cur.cc:1622(btr_cur_search_to_nth_level_func(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, rw_lock_t*, char const*, unsigned int, mtr_t*, unsigned long))[0x55b7e2210941]
include/btr0pcur.ic:448(btr_pcur_open_low)[0x55b7e2189ca2]
row/row0row.cc:1306(row_search_index_entry(dict_index_t*, dtuple_t const*, unsigned long, btr_pcur_t*, mtr_t*))[0x55b7e2189e0f]
row/row0uins.cc:260(row_undo_ins_remove_sec_low)[0x55b7e22faee3]
row/row0uins.cc:320(row_undo_ins_remove_sec)[0x55b7e22fb5b2]
row/row0undo.cc:420(row_undo)[0x55b7e2194d0a]
que/que0que.cc:944(que_thr_step)[0x55b7e213ba30]
trx/trx0roll.cc:631(trx_rollback_active)[0x55b7e21d1e90]
trx/trx0roll.cc:784(trx_rollback_recovered(bool))[0x55b7e21d255f]
trx/trx0roll.cc:838(trx_rollback_all_recovered)[0x55b7e21d2c6a]
pthread_create.c:0(start_thread)[0x2b656a1dedc5]
??:0(__clone)[0x2b656a4e9c9d]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /rdsdbdata/db
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size unlimited unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15735 15735 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
2022-06-24 3:01:11 0 [Note] /rdsdbbin/mysql/bin/mysqld (mysqld 10.5.12-MariaDB) starting as process 17497 ...
2022-06-24 3:01:11 0 [Warning] You need to use --log-bin to make --log-slave-updates work.
2022-06-24 3:01:11 0 [Note] InnoDB: Uses event mutexes
2022-06-24 3:01:11 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-06-24 3:01:11 0 [Note] InnoDB: Number of pools: 1
2022-06-24 3:01:11 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-06-24 3:01:11 0 [Note] InnoDB: Using Linux native AIO
2022-06-24 3:01:11 0 [Note] InnoDB: Initializing buffer pool, total size = 2147483648, chunk size = 134217728
2022-06-24 3:01:11 0 [Note] InnoDB: Completed initialization of buffer pool
2022-06-24 3:01:11 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1343302636904,1343302636904
2022-06-24 3:01:17 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
2022-06-24 3:01:17 0 [Note] InnoDB: Trx id counter is 8615938331
2022-06-24 3:01:17 0 [Note] InnoDB: Starting final batch to recover 25079 pages from redo log.
2022-06-24 3:01:33 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.667099', position 4312
2022-06-24 3:01:33 0 [Note] InnoDB: 128 rollback segments are active.
2022-06-24 3:01:33 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-06-24 3:01:33 0 [Note] InnoDB: To roll back: 2 transactions, 2 rows
2022-06-24 3:01:33 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-06-24 3:01:33 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-06-24 3:01:33 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-06-24 3:01:33 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
2022-06-24 3:01:33 0 [ERROR] [FATAL] InnoDB: Page old data size 8606 new data size 5454, page old max ins size 7544 new max ins size 10696
220624 3:01:33 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.5.12-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11660859 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2022-06-24 3:01:33 0 [Note] InnoDB: 10.5.12 started; log sequence number 1343572974007; transaction id 8615938332
2022-06-24 3:01:33 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
2022-06-24 3:01:33 0 [Note] Server socket created on IP: '::'.
2022-06-24 3:01:33 0 [Note] Reading of all Master_info entries succeeded
2022-06-24 3:01:33 0 [Note] Added new Master_info '' to hash table
2022-06-24 3:01:33 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
Version: '10.5.12-MariaDB' socket: '/tmp/mysql.sock' port: 3306 managed by https://aws.amazon.com/rds/
mysys/stacktrace.c:213(my_print_stacktrace)[0x560cd2b17c5e]
sql/signal_handler.cc:221(handle_fatal_signal)[0x560cd22c976a]
sigaction.c:0(__restore_rt)[0x2abefd95a100]
:0(__GI_raise)[0x2abefdb9c5f7]
:0(__GI_abort)[0x2abefdb9dce8]
:0(ib::fatal::~fatal())[0x560cd296b7d1]
btr/btr0btr.cc:1484(btr_page_reorganize_low)[0x560cd2971246]
btr/btr0btr.cc:1646(btr_page_reorganize(page_cur_t*, dict_index_t*, mtr_t*))[0x560cd2971617]
include/dict0mem.h:1840(dict_table_t::not_redundant() const)[0x560cd2856f93]
ibuf/ibuf0ibuf.cc:3881(ibuf_insert_to_index_page)[0x560cd285ca11]
buf/buf0buf.cc:3339(buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x560cd29a652b]
buf/buf0buf.cc:3421(buf_page_get_gen(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x560cd29a6576]
btr/btr0cur.cc:1622(btr_cur_search_to_nth_level_func(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, rw_lock_t*, char const*, unsigned int, mtr_t*, unsigned long))[0x560cd2986941]
include/btr0pcur.ic:448(btr_pcur_open_low)[0x560cd28ffca2]
row/row0row.cc:1306(row_search_index_entry(dict_index_t*, dtuple_t const*, unsigned long, btr_pcur_t*, mtr_t*))[0x560cd28ffe0f]
row/row0uins.cc:260(row_undo_ins_remove_sec_low)[0x560cd2a70ee3]
row/row0uins.cc:320(row_undo_ins_remove_sec)[0x560cd2a715b2]
row/row0undo.cc:420(row_undo)[0x560cd290ad0a]
que/que0que.cc:944(que_thr_step)[0x560cd28b1a30]
trx/trx0roll.cc:631(trx_rollback_active)[0x560cd2947e90]
trx/trx0roll.cc:784(trx_rollback_recovered(bool))[0x560cd294855f]
trx/trx0roll.cc:838(trx_rollback_all_recovered)[0x560cd2948c6a]
pthread_create.c:0(start_thread)[0x2abefd952dc5]
??:0(__clone)[0x2abefdc5dc9d]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /rdsdbdata/db
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size unlimited unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15735 15735 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
2022-06-24 3:02:28 0 [Warning] You need to use --log-bin to make --log-slave-updates work.
2022-06-24 3:02:28 0 [Note] InnoDB: Uses event mutexes
2022-06-24 3:02:28 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-06-24 3:02:28 0 [Note] InnoDB: Number of pools: 1
2022-06-24 3:02:28 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-06-24 3:02:28 0 [Note] InnoDB: Using Linux native AIO
2022-06-24 3:02:28 0 [Note] InnoDB: Initializing buffer pool, total size = 2147483648, chunk size = 134217728
2022-06-24 3:02:29 0 [Note] InnoDB: Completed initialization of buffer pool
2022-06-24 3:02:29 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1343302636904,1343302636904
2022-06-24 3:02:34 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 2 row operations to undo
2022-06-24 3:02:34 0 [Note] InnoDB: Trx id counter is 8615938331
2022-06-24 3:02:34 0 [Note] InnoDB: Starting final batch to recover 25079 pages from redo log.
2022-06-24 3:02:50 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.667099', position 4312
2022-06-24 3:02:50 0 [Note] InnoDB: 128 rollback segments are active.
2022-06-24 3:02:50 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-06-24 3:02:50 0 [Note] InnoDB: To roll back: 2 transactions, 2 rows
2022-06-24 3:02:50 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-06-24 3:02:50 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-06-24 3:02:50 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-06-24 3:02:50 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
2022-06-24 3:02:50 0 [ERROR] [FATAL] InnoDB: Page old data size 8606 new data size 5454, page old max ins size 7544 new max ins size 10696
220624 3:02:50 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.5.12-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=5002
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11660859 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x40000
2022-06-24 3:02:50 0 [Note] InnoDB: 10.5.12 started; log sequence number 1343573277665; transaction id 8615938332
2022-06-24 3:02:50 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
2022-06-24 3:02:50 0 [Note] Server socket created on IP: '::'.
2022-06-24 3:02:50 0 [Note] Reading of all Master_info entries succeeded
2022-06-24 3:02:50 0 [Note] Added new Master_info '' to hash table
2022-06-24 3:02:50 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
Version: '10.5.12-MariaDB' socket: '/tmp/mysql.sock' port: 3306 managed by https://aws.amazon.com/rds/
mysys/stacktrace.c:213(my_print_stacktrace)[0x563f02949c5e]
sql/signal_handler.cc:221(handle_fatal_signal)[0x563f020fb76a]
sigaction.c:0(__restore_rt)[0x2b828c588100]
:0(__GI_raise)[0x2b828c7ca5f7]
:0(__GI_abort)[0x2b828c7cbce8]
:0(ib::fatal::~fatal())[0x563f0279d7d1]
btr/btr0btr.cc:1484(btr_page_reorganize_low)[0x563f027a3246]
btr/btr0btr.cc:1646(btr_page_reorganize(page_cur_t*, dict_index_t*, mtr_t*))[0x563f027a3617]
include/dict0mem.h:1840(dict_table_t::not_redundant() const)[0x563f02688f93]
ibuf/ibuf0ibuf.cc:3881(ibuf_insert_to_index_page)[0x563f0268ea11]
buf/buf0buf.cc:3339(buf_page_get_low(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x563f027d852b]
buf/buf0buf.cc:3421(buf_page_get_gen(page_id_t, unsigned long, unsigned long, buf_block_t*, unsigned long, char const*, unsigned int, mtr_t*, dberr_t*, bool))[0x563f027d8576]
btr/btr0cur.cc:1622(btr_cur_search_to_nth_level_func(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, rw_lock_t*, char const*, unsigned int, mtr_t*, unsigned long))[0x563f027b8941]
include/btr0pcur.ic:448(btr_pcur_open_low)[0x563f02731ca2]
row/row0row.cc:1306(row_search_index_entry(dict_index_t*, dtuple_t const*, unsigned long, btr_pcur_t*, mtr_t*))[0x563f02731e0f]
row/row0uins.cc:260(row_undo_ins_remove_sec_low)[0x563f028a2ee3]
row/row0uins.cc:320(row_undo_ins_remove_sec)[0x563f028a35b2]
row/row0undo.cc:420(row_undo)[0x563f0273cd0a]
que/que0que.cc:944(que_thr_step)[0x563f026e3a30]
trx/trx0roll.cc:631(trx_rollback_active)[0x563f02779e90]
trx/trx0roll.cc:784(trx_rollback_recovered(bool))[0x563f0277a55f]
trx/trx0roll.cc:838(trx_rollback_all_recovered)[0x563f0277ac6a]
pthread_create.c:0(start_thread)[0x2b828c580dc5]
??:0(__clone)[0x2b828c88bc9d]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /rdsdbdata/db
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size unlimited unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes unlimited unlimited processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 15735 15735 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: /rdsdbdata/tmp/core-%e-%p



 Comments   
Comment by Marko Mäkelä [ 2022-06-29 ]

The stack trace shows that corruption was detected during a change buffer merge, while rolling back a transaction (maybe a recovered incomplete transaction).

Unfortunately, we have been unable to reproduce this corruption in our internal testing. The change buffer was disabled by default in MDEV-27734. Disabling it will not cure any previous corruption; it will only prevent the use of the change buffer in future operations. But, you could try if the database would start if you specify innodb_change_buffering=none. If that does not help, then with innodb_force_recovery=3 the database should start up, without rolling back that incomplete transaction. That should allow you to create a logical backup of the data (with mariadb-dump a.k.a. mysqldump or similar tools).

This crash (but not the corruption that triggers it) should have been fixed in MDEV-13542. Because we have not been able to reproduce this crash internally, I can’t guarantee what exactly should happen in this case. The intention is that the affected secondary index will be marked as corrupted and operations that would use the index would return an error to the SQL statement.

A support customer of ours hit a similar crash some time ago, but they had overwritten the corrupted database with a backup, so there was no chance to test the fix.

Also the reports MDEV-22739 and MDEV-26917 are due to change buffer induced corruption. If anyone can provide a set of SQL statements that reproduce this type of errors, please post them there. The test must start from initializing the database from the scratch, with no physical backup of any files.

Comment by Marko Mäkelä [ 2022-06-29 ]

I am closing this as a duplicate of MDEV-13542 because this crash should not occur with that version. For reproducing the corruption itself, MDEV-22739 or MDEV-26917 can be updated.

Generated at Thu Feb 08 10:04:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.