Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.3(EOL), 10.4(EOL)
Description
10.3 server crashes with a non-debug assertion failure when it starts on the attached datadir.
The datadir was created on the current 10.0 server as a part of the undo-upgrade scenario:
- start the current 10.0 server;
- create some tables and run some DML on them;
- kill the server during operation (with SIGKILL);
- restart the server with innodb-force-recovery=3, no client activity;
- shutdown the server normally;
Then the current 10.3 server is started on the same datadir. It starts, but crashes immediately afterwards as below.
|
10.3 b52bb6eb82db8 |
2018-06-12 18:15:28 0x7fb1809ea700 InnoDB: Assertion failure in file /data/src/10.3/storage/innobase/trx/trx0purge.cc line 121
|
InnoDB: Failing assertion: purge_sys.tail.commit <= purge_sys.rseg->last_commit
|
|
|
#5 0x00007fb1a1634448 in __GI_abort () at abort.c:89
|
#6 0x00007fb1a44f44fb in ut_dbg_assertion_failed (expr=0x7fb1a4b6b0c8 "purge_sys.tail.commit <= purge_sys.rseg->last_commit", file=0x7fb1a4b6af80 "/data/src/10.3/storage/innobase/trx/trx0purge.cc", line=121) at /data/src/10.3/storage/innobase/ut/ut0dbg.cc:61
|
#7 0x00007fb1a44c2dfb in TrxUndoRsegsIterator::set_next (this=0x7fb1a530c0a0 <purge_sys+416>) at /data/src/10.3/storage/innobase/trx/trx0purge.cc:121
|
#8 0x00007fb1a44c0d4b in trx_purge_choose_next_log () at /data/src/10.3/storage/innobase/trx/trx0purge.cc:1213
|
#9 0x00007fb1a44c0fa4 in trx_purge_get_next_rec (n_pages_handled=0x7fb1809e9dd0, heap=0x7fb1a85d2780) at /data/src/10.3/storage/innobase/trx/trx0purge.cc:1286
|
#10 0x00007fb1a44c11f5 in trx_purge_fetch_next_rec (roll_ptr=0x7fb17400d8f0, n_pages_handled=0x7fb1809e9dd0, heap=0x7fb1a85d2780) at /data/src/10.3/storage/innobase/trx/trx0purge.cc:1356
|
#11 0x00007fb1a44c151c in trx_purge_attach_undo_recs (n_purge_threads=4) at /data/src/10.3/storage/innobase/trx/trx0purge.cc:1429
|
#12 0x00007fb1a44c19ff in trx_purge (n_purge_threads=4, truncate=false) at /data/src/10.3/storage/innobase/trx/trx0purge.cc:1559
|
#13 0x00007fb1a4492d44 in srv_do_purge (n_total_purged=0x7fb1809e9ed0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2583
|
#14 0x00007fb1a4493172 in srv_purge_coordinator_thread (arg=0x0) at /data/src/10.3/storage/innobase/srv/srv0srv.cc:2714
|
#15 0x00007fb1a3293064 in start_thread (arg=0x7fb1809ea700) at pthread_create.c:309
|
#16 0x00007fb1a16e662d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
|
This current test was run with --innodb-page-size=8K --loose-innodb_log_compressed_pages=on --loose-innodb-change-buffering=none, I'm not sure whether any of them important. Naturally, to reproduce the crash on the attached datadir, the server needs to be also started with --innodb-page-size=8K, other two options don't make a difference; otherwise all defaults.
ib_logfile-s are compressed and attached separately just to overcome the 10M limitation in JIRA. I don't know if they are needed, the crash happens with and without them.
Similar-looking crashes upon upgrade from 10.1 have also been observed before.
10.2 doesn't crash on the same datadir.
Attachments
Issue Links
- causes
-
-
- Closed
-
-
-
- Closed
-
- is caused by
-
MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC
-
- Closed
-
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- mentioned in
-
Page Loading...
Activity
As described in MDEV-23755, we can prevent these failures by refusing InnoDB startup if any incomplete transactions have undo logs in the pre-MDEV-12288 format, with an error message saying that the older server version must be shut down cleanly, without using innodb_force_recovery=3.
It might be possible to remove the old_insert data members altogether.
We will refuse an upgrade from older versions than MariaDB 10.3 if a clean shutdown was not performed, so that the undo logs would be emptied:
2021-06-13 16:42:45 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
|
There will be only one persistent undo log per transaction. No main-memory data structures related to the partitioned persistent undo log before MDEV-12288 are retained.
It looks like this just got pulled into the 10.3 branch, but it breaks upgrades for anyone trying to go from 5.7 -> 10.3.30, or even 5.7 -> 10.3.29 -> 10.3.30
I started a topic for this in the zulip chat.
Log snippet, just in case it is helpful:
Version: '10.3.29-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
|
2021-06-21 20:58:29 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
|
...
|
2021-06-21 20:58:29 0 [Note] InnoDB: FTS optimize thread exiting.
|
2021-06-21 20:58:29 0 [Note] InnoDB: Starting shutdown...
|
2021-06-21 20:58:29 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
|
2021-06-21 20:58:29 0 [Note] InnoDB: Buffer pool(s) dump completed at 210621 20:58:29
|
2021-06-21 20:58:31 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
|
2021-06-21 20:58:31 0 [Note] InnoDB: Shutdown completed; log sequence number 3281751; transaction id 3852
|
2021-06-21 20:58:31 0 [Note] /usr/sbin/mysqld: Shutdown complete
|
...
|
(after updating to 10.3.30 via yum)
|
2021-06-21 20:58:31 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
|
2021-06-21 20:58:31 0 [ERROR] InnoDB: upgrade from older version than MariaDB 10.3 requires clean shutdown
|
2021-06-21 20:58:31 0 [Note] InnoDB: Retry with innodb_force_recovery=5
|
2021-06-21 20:58:31 0 [ERROR] InnoDB: Plugin initialization aborted with error Data structure corruption
|
2021-06-21 20:58:31 0 [Note] InnoDB: Starting shutdown...
|
2021-06-21 20:58:31 0 [ERROR] Plugin 'InnoDB' init function returned error.
|
2021-06-21 20:58:31 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
|
2021-06-21 20:58:31 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2021-06-21 20:58:31 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
|
2021-06-21 20:58:31 0 [ERROR] Unknown/unsupported storage engine: InnoDB
|
Edit: I went ahead and filed MDEV-25981 for this issue.
hborresen, thank you. I should have actually tested the upgrade. As I expected, fixing MDEV-25981 involved only minor changes to the validation logic.
We may have to introduce trx_rseg_t::old_insert_cached and some separate mechanism that will guarantee that old insert_undo pages from before the upgrade to 10.3 (or later) will eventually be freed, without interfering with the purge of transaction history.