Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11.7
-
None
-
CloudLinux release 7.9 (Boris Yegorov)
CloudLinux release 8.9 (Anatoly Levchenko)
Description
Hello, we've had 15 separate MariaDB instances in the past 4 months run into an issue where the MariaDB server suddenly crashes, with an indication of a corrupted undo tablespace file, and starts to crash loop until it's begun with innodb_force_recovery=1 to ignore the corrupted page, as is described in the error logs. All of the servers are using `innodb_undo_tablespaces=3`in `my.cnf`.
We can't find a reliable way to reproduce the issue. We initially thought it was brought over during 10.6 -> 10.11 upgrades, but the most recent case is a month-old server that started with 10.11.7. If there's anything else you need to help debug the issue, we'll be happy to provide it.
Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 7:44:58 1447973524 [ERROR] InnoDB: Trying to read 16384 bytes at 70368744161280 outside the bounds of the file: .//undo003 |
Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 7:44:58 1447973524 [ERROR] InnoDB: File './/undo003' is corrupted |
Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 07:44:58 0x7f7f9013f700 InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB1011-10.11.7/mariadb-10.11.7/storage/innobase/trx/trx0purge.cc line 268 |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Failing assertion: undo_page |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: We intentionally generate a memory trap. |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: If you get repeated assertion failures or crashes, even |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: immediately after the mariadbd startup, there may be |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: corruption in the InnoDB tablespace. Please refer to |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: about forcing recovery. |
Apr 16 07:44:58 host1339 mysqld[173461]: 240416 7:44:58 [ERROR] mysqld got signal 6 ; |
Apr 16 07:44:58 host1339 mysqld[173461]: Sorry, we probably made a mistake, and this is a bug. |
Apr 16 07:44:58 host1339 mysqld[173461]: Your assistance in bug reporting will enable us to fix this for the next release. |
Apr 16 07:44:58 host1339 mysqld[173461]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
Apr 16 07:44:58 host1339 mysqld[173461]: We will try our best to scrape up some info that will hopefully help |
Apr 16 07:44:58 host1339 mysqld[173461]: diagnose the problem, but since we have already crashed, |
Apr 16 07:44:58 host1339 mysqld[173461]: something is definitely wrong and this may fail. |
Apr 16 07:44:58 host1339 mysqld[173461]: Server version: 10.11.7-MariaDB-cll-lve source revision: 87e13722a95af5d9378d990caf48cb6874439347 |
Apr 16 07:44:58 host1339 mysqld[173461]: key_buffer_size=2147483648 |
Apr 16 07:44:58 host1339 mysqld[173461]: read_buffer_size=4194304 |
Apr 16 07:44:58 host1339 mysqld[173461]: max_used_connections=1205 |
Apr 16 07:44:58 host1339 mysqld[173461]: max_threads=2002 |
Apr 16 07:44:58 host1339 mysqld[173461]: thread_count=320 |
Apr 16 07:44:58 host1339 mysqld[173461]: It is possible that mysqld could use up to |
Apr 16 07:44:58 host1339 mysqld[173461]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18551172 K bytes of memory |
Apr 16 07:44:58 host1339 mysqld[173461]: Hope that's ok; if not, decrease some variables in the equation. |
Apr 16 07:44:58 host1339 mysqld[173461]: Thread pointer: 0x7f81165dbc58 |
Apr 16 07:44:58 host1339 mysqld[173461]: Attempting backtrace. You can use the following information to find out |
Apr 16 07:44:58 host1339 mysqld[173461]: where mysqld died. If you see no messages after this, something went |
Apr 16 07:44:58 host1339 mysqld[173461]: terribly wrong... |
Apr 16 07:44:58 host1339 mysqld[173461]: stack_bottom = 0x7f7f9013d358 thread_stack 0x49000 |
Apr 16 07:44:58 host1339 mysqld[173461]: 2024-04-16 07:44:58 0x7f7a770c0700 InnoDB: Assertion failure in file /builddir/build/BUILD/cl-MariaDB1011-10.11.7/mariadb-10.11.7/storage/innobase/buf/buf0lru.cc line 281 |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Failing assertion: !block->page.in_file() |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: We intentionally generate a memory trap. |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/ |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: If you get repeated assertion failures or crashes, even |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: immediately after the mariadbd startup, there may be |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: corruption in the InnoDB tablespace. Please refer to |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/ |
Apr 16 07:44:58 host1339 mysqld[173461]: InnoDB: about forcing recovery. |
Apr 16 07:45:05 host1339 systemd-coredump[2183663]: Process 173461 (mysqld) of user 991 dumped core. |
...
|
Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16 8:02:23 0 [ERROR] InnoDB: Corrupted page identifier at 3543005337948; set innodb_force_recovery=1 to ignore the record. |
Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16 8:02:23 0 [Note] InnoDB: End of log at LSN=3543005337948 |
Apr 16 08:02:23 host1339 mysqld[2811986]: 2024-04-16 8:02:23 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error |