[MDEV-31355] innodb_undo_log_truncate=ON fails to wait for purge of enough transaction history Created: 2023-05-26  Updated: 2023-06-23  Resolved: 2023-06-08

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 11.0, 11.1
Fix Version/s: 10.5.22, 10.6.15, 10.9.8, 10.10.6, 10.11.5, 11.0.3, 11.1.2

Type: Bug Priority: Critical
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 1
Labels: corruption, crash, purge, rr-profile-analyzed

Issue Links:
Blocks
blocks MDEV-31382 SET GLOBAL innodb_undo_log_truncate=O... Closed
Relates
relates to MDEV-30671 innodb_undo_log_truncate=ON fails to ... Closed
relates to MDEV-31234 InnoDB does not free UNDO after the f... Closed

 Description   

mleich produced an rr replay trace leading to a failure that demonstrates that the condition that was introduced in MDEV-30671 to trx_purge_truncate_history() is not sufficient:

    for (auto &rseg : trx_sys.rseg_array)
 
    {
 
      if (rseg.space != &space)
 
        continue;
 
 
 
      rseg.latch.rd_lock(SRW_LOCK_CALL);
 
      ut_ad(rseg.skip_allocation());
 
      if (rseg.is_referenced() || rseg.needs_purge > head.trx_no)
 
      {
 
not_free:
 
        rseg.latch.rd_unlock();
 
        return;
 
      }

In the trace that we got, we have rseg.needs_purge=0x27f5 and head.trx_no=0x2805 but purge_sys.view.m_up_limit_id=0x27f3. The thread that will crash holds a read view that is similar to the purge_sys.view. The function row_vers_build_for_consistent_read() will invoke trx_undo_prev_version_build() 3 times until it crashes: first on the newest version DB_TRX_ID=0x2844, then on DB_TRX_ID=0x2800, and finally crashing on DB_TRX_ID=0x27f4 while trying to access an undo page 0x2e that was invalidated by the undo tablespace truncation. This time, the failure looks like this:

10.6ish f410444a76b5703b803e6e43e3bbf9509b6fddb6

 
2023-05-25  9:26:56 0 [Note] InnoDB: Truncated .//undo001
 
mysqld: /data/Server/bb-10.6-MDEV-31343/storage/innobase/include/mach0data.inl:308: ib_uint32_t mach_read_next_compressed(const byte**): Assertion `val > 0x7F' failed.
 
 
 
Thread 3 received signal SIGABRT, Aborted.
 
...
 
#7  0x000055aea54e31d8 in mach_read_next_compressed (b=0x72cb31634fe0) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/include/mach0data.inl:308
 
#8  0x000055aea54eec18 in mach_u64_read_next_compressed (b=0x72cb31634fe0) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/include/mach0data.inl:449
 
#9  trx_undo_update_rec_get_sys_cols (ptr=0x6190001f5f0d "\004\200", trx_id=trx_id@entry=0x72cb31635170, roll_ptr=roll_ptr@entry=0x72cb31635190, info_bits=info_bits@entry=0x72cb316350f0 "\004W$\245\256U") at /data/Server/bb-10.6-MDEV-31343/storage/innobase/trx/trx0rec.cc:1421
 
#10 0x000055aea54fddcb in trx_undo_prev_version_build (rec=rec@entry=0x6250009d819d "\200", index=index@entry=0x6160005e5a08, offsets=0x72cb31636450, heap=heap@entry=0x6190001f5e80, old_vers=old_vers@entry=0x72cb316355d0, v_heap=v_heap@entry=0x0, vrow=<optimized out>, v_status=<optimized out>) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/trx/trx0rec.cc:2202
 
#11 0x000055aea548edb6 in row_vers_build_for_consistent_read (rec=rec@entry=0x30ef610ad579 "\200", mtr=mtr@entry=0x72cb31636730, index=index@entry=0x6160005e5a08, offsets=offsets@entry=0x72cb31635a50, view=0x41956db01e88, offset_heap=offset_heap@entry=0x72cb31635a30, in_heap=<optimized out>, old_vers=<optimized out>, vrow=<optimized out>) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/row/row0vers.cc:1202
 
#12 0x000055aea542cb7d in row_sel_build_prev_vers_for_mysql (prebuilt=prebuilt@entry=0x623000070188, clust_index=clust_index@entry=0x6160005e5a08, rec=0x30ef610ad579 "\200", offsets=offsets@entry=0x72cb31635a50, offset_heap=offset_heap@entry=0x72cb31635a30, old_vers=old_vers@entry=0x72cb31635ab0, vrow=0x0, mtr=0x72cb31636730) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/row/row0sel.cc:3292
 
#13 0x000055aea5446d40 in row_search_mvcc (buf=buf@entry=0x6260001fb138 "\376\377\377X", mode=<optimized out>, mode@entry=PAGE_CUR_UNSUPP, prebuilt=<optimized out>, match_mode=match_mode@entry=0, direction=direction@entry=1) at /data/Server/bb-10.6-MDEV-31343/storage/innobase/row/row0sel.cc:5376

It seems to me that the check here as well as in trx_purge_truncate_rseg_history() will have to be revised to something like purge_sys.view.sees(rseg.needs_purge).

 Comments   
Comment by Marko Mäkelä [ 2023-06-01 ]

vlad.lesin, this will require some nontrivial adjustments to the test innodb.cursor-restore-locking that you created. I think that this needs to be reviewed by you.

Comment by Marko Mäkelä [ 2023-06-07 ]

vlad.lesin, thank you for your help with the test. I revised the fix with that, and also with a change to the test innodb.dml_purge that would fix occasional failures in 10.6 of the following invocation:

./mtr --parallel=auto --repeat=500 {innodb.max_record_size,dynamic,innodb_gis.innodb_gis_rtree,dynamic,innodb.dml_purge}{,,,,}

Comment by Vladislav Lesin [ 2023-06-08 ]

LGTM

Generated at Thu Feb 08 10:23:13 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.