[MDEV-29150] DB continuously rebooting Created: 2022-07-21  Updated: 2022-08-24  Resolved: 2022-08-24

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.13
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Michael Qin (Inactive) Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None

Issue Links:
Duplicate
duplicates MDEV-13542 Crashing on a corrupted page is unhel... Closed

 Description   

2022-07-21  7:18:20 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-07-21  7:18:20 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
2022-07-21  7:18:20 0 [Note] InnoDB: Waiting for purge to start
2022-07-21  7:18:20 2 [ERROR] [FATAL] InnoDB: InnoDB is trying to free page [page id: space=484, page number=6610] though it is already marked as free in the tablespace! The tablespace free space info is corrupt. You may need to dump your tables and recreate the whole database!Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
220721  7:18:20 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.13-MariaDB-log
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=10608
thread_count=4
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24693801 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x2aec44412008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2ad5b4bdfde8 thread_stack 0x40000
2022-07-21  7:18:20 0 [Note] InnoDB: 10.3.13 started; log sequence number 130026721339925; transaction id 281148570965
2022-07-21  7:18:20 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
2022-07-21  7:18:20 0 [Note] InnoDB: Rolled back recovered transaction 281148570919
2022-07-21  7:18:20 0 [Note] InnoDB: Rolled back recovered transaction 281148570877
2022-07-21  7:18:20 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-07-21  7:18:20 0 [Warning] /rdsdbbin/mysql/bin/mysqld: unknown variable 'loose-rocksdb_log_dir=/rdsdbdata/db/log'
2022-07-21  7:18:20 0 [Note] InnoDB: Rolled back recovered transaction 281148570779
2022-07-21  7:18:20 0 [Note] InnoDB: Rolled back recovered transaction 281148570959
2022-07-21  7:18:20 0 [Note] InnoDB: Rolled back recovered transaction 281148570951
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x29)[0x55ad08e048c9]
2022-07-21  7:18:20 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
/rdsdbbin/mysql/bin/mysqld(handle_fatal_signal+0x57d)[0x55ad088cc07d]
/lib64/libpthread.so.0(+0xf100)[0x2ad35b3cf100]
/lib64/libc.so.6(gsignal+0x37)[0x2ad35b6115f7]
:0(__GI_raise)[0x2ad35b612ce8]
2022-07-21  7:18:21 0 [Note] Starting crash recovery...
2022-07-21  7:18:21 0 [Note] Crash recovery finished.
2022-07-21  7:18:21 0 [Note] Server socket created on IP: '::'.
/rdsdbbin/mysql/bin/mysqld(+0xae3d2d)[0x55ad08b73d2d]
2022-07-21  7:18:21 0 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired
2022-07-21  7:18:21 0 [Warning] Checking table:   './mysql/event'
2022-07-21  7:18:21 0 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly
2022-07-21  7:18:21 7 [Note] Event Scheduler: scheduler thread started with id 7
2022-07-21  7:18:21 0 [Note] Reading of all Master_info entries succeded
2022-07-21  7:18:21 0 [Note] Added new Master_info '' to hash table
2022-07-21  7:18:21 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
Version: '10.3.13-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 3306  Source distribution



 Comments   
Comment by Daniel Black [ 2022-07-22 ]

Looks like this SO complain.

Error is like MDEV-15826 (fixed 10.3.7), which seems unlikely to bump minorly past the version and then do the purge.

Moving beyond a 3.5 old version would be good too.

"handle_fatal_signal" in its output position looks like MariaDB is still crashing as the new instance is starting up. You probably should prevent that.

Comment by Marko Mäkelä [ 2022-07-22 ]

Starting with MDEV-13542, the server should no longer crash due to this corruption of allocation metadata, but propagate an error to the caller.

Comment by Elena Stepanova [ 2022-07-25 ]

Neither of the above (a bugfix from 10.3.7 nor a change in 10.6) probably help much to an instance which runs 10.3.13.
Still, even if you are not planning to upgrade to higher 10.x, 10.3.13 is over 3 year old and 20 minor versions behind the current 10.3, and bugs get fixed all the time. Please try to upgrade to at least the current 10.3 and see whether the problem goes away.

Generated at Thu Feb 08 10:06:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.