[MDEV-31574] Assertion failure on REPLACE on ROW_FORMAT=COMPRESSED table Created: 2023-06-28  Updated: 2024-01-11  Resolved: 2023-11-23

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 5.5, 10.1, 10.2, 10.3, 10.4
Fix Version/s: 10.4.33

Type: Bug Priority: Major
Reporter: Elena Stepanova Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: not-10.5+

Attachments: File r8.test    

 Description   

The test case is attached. It only contains 3 essential statements – CREATE, INSERT and REPLACE – but INSERT and REPLACE are quite big. Possibly it can be reduced when the problem is known. Also, mind the --sleep at the end of the test case, it should be replaced with a decent way to wait for the purge.

The table definition (also a part of the test case) is this:

CREATE TABLE t (
 
  c08 TEXT DEFAULT '',
 
  c04 TIMESTAMP(6) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
 
  c10 BIGINT COMMENT 'Y',
 
  c03 VARCHAR(1024) NOT NULL DEFAULT '',
 
  c07 DATE,
 
  c02 CHAR(3) NOT NULL DEFAULT '',
 
  c06 DOUBLE(16,8) COMMENT 'Percentage',
 
  c09 BIGINT COMMENT 'X',
 
  id BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
 
  c05 LONGBLOB NOT NULL DEFAULT '',
 
  c01 INT ZEROFILL NOT NULL DEFAULT 0,
 
  UNIQUE idx1 (c01,c02),
 
  PRIMARY KEY (id)
 
) ENGINE=InnoDB row_format=compressed key_block_size=4;

The failure started happening on 10.4 after this commit in 10.4.29

commit ff3d4395d808b6421d2e0714e10d48c7aa2f3c3a
 
Author: Marko Mäkelä
 
Date:   Wed Mar 22 14:31:00 2023 +0200
 
 
 
    MDEV-30882 Crash on ROLLBACK in a ROW_FORMAT=COMPRESSED table

However it is not reproducible with the provided test case on higher versions, so I'm not sure whether it's even worth fixing. Feel free to close if it isn't.

10.4 f5dceafd

 
mysqld: /data/src/10.4/storage/innobase/row/row0purge.cc:141: bool row_purge_remove_clust_if_poss_low(purge_node_t*, ulint): Assertion `rec_get_deleted_flag(rec, rec_offs_comp(offsets))' failed.
 
230628 22:12:38 [ERROR] mysqld got signal 6 ;
 
 
 
#9  0x00007f22fa853df2 in __GI___assert_fail (assertion=0x55bf452fc3c0 "rec_get_deleted_flag(rec, rec_offs_comp(offsets))", file=0x55bf452fc1c0 "/data/src/10.4/storage/innobase/row/row0purge.cc", line=141, function=0x55bf452fc360 "bool row_purge_remove_clust_if_poss_low(purge_node_t*, ulint)") at ./assert/assert.c:101
 
#10 0x000055bf44086004 in row_purge_remove_clust_if_poss_low (node=0x61a000001908, mode=2) at /data/src/10.4/storage/innobase/row/row0purge.cc:141
 
#11 0x000055bf44086577 in row_purge_remove_clust_if_poss (node=0x61a000001908) at /data/src/10.4/storage/innobase/row/row0purge.cc:198
 
#12 0x000055bf44089449 in row_purge_del_mark (node=0x61a000001908) at /data/src/10.4/storage/innobase/row/row0purge.cc:788
 
#13 0x000055bf4408e61a in row_purge_record_func (node=0x61a000001908, undo_rec=0x6250001723b0 "", thr=0x616000006ad0, updated_extern=false) at /data/src/10.4/storage/innobase/row/row0purge.cc:1332
 
#14 0x000055bf4408ec89 in row_purge (node=0x61a000001908, undo_rec=0x6250001723b0 "", thr=0x616000006ad0) at /data/src/10.4/storage/innobase/row/row0purge.cc:1398
 
#15 0x000055bf4408f2b5 in row_purge_step (thr=0x616000006ad0) at /data/src/10.4/storage/innobase/row/row0purge.cc:1476
 
#16 0x000055bf43f5ba25 in que_thr_step (thr=0x616000006ad0) at /data/src/10.4/storage/innobase/que/que0que.cc:966
 
#17 0x000055bf43f5be8c in que_run_threads_low (thr=0x616000006ad0) at /data/src/10.4/storage/innobase/que/que0que.cc:1028
 
#18 0x000055bf43f5c2e9 in que_run_threads (thr=0x616000006ad0) at /data/src/10.4/storage/innobase/que/que0que.cc:1068
 
#19 0x000055bf4411b810 in srv_task_execute (slot=0x55bf460b8d00 <srv_sys+832>) at /data/src/10.4/storage/innobase/srv/srv0srv.cc:2432
 
#20 0x000055bf4411bc1a in srv_worker_thread (arg=0x0) at /data/src/10.4/storage/innobase/srv/srv0srv.cc:2487
 
#21 0x00007f22fa8a7fd4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
 
#22 0x00007f22fa9285bc in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81



 Comments   
Comment by Marko Mäkelä [ 2023-07-10 ]

The record in question looks like this:

10.4 12a5fb4b36b573764564eab05cb2575c5a59b471

 
(rr) p/x *rec@0x42
 
$7 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x25, 
  0x3, 0x0, 0x0, 0x1, 0x37, 0x8, 0xe0, 0x2d, 0x67, 0x6c, 0x61, 0x73, 0x73, 
  0x2d, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f, 0xff, 0xff, 0xff, 0xff, 0xff, 
  0xb6}

The fields are:
(id=15,DB_TRX_ID=0x25,DB_ROLL_PTR=(update),c08='glass',…)

There is a matching record in the REPLACE statement:

REPLACE INTO `t` VALUES
 
…
 
('-glass-','0000-00-00 00:00:00.000000',-18904,'-x-','2025-08-04','nul',NULL,NULL,15,'2092433408',0400359424),
 
…

If I counted correctly (and if the secondary indexes do not play any role up to this point), the record id=15 was originally inserted by the following:

INSERT IGNORE INTO t VALUES
 
…
 
('null', '20250427190536.023154', 27747, 'null', 'null', '-n-', -29188, -4557079872945520640, NULL, '-segment-', 955318272),
 
…

Comment by Marko Mäkelä [ 2023-07-10 ]

I am able to reproduce this even if I shrink the REPLACE statement so that it ends in the following:

REPLACE INTO `t` VALUES
 
…
 
('-glass-','0000-00-00 00:00:00.000000',-18904,'-x-','2025-08-04','nul',NULL,NULL,15,'2092433408',0400359424),
 
('-1755381760','2011-07-21 16:37:36.040624',3,'m','1971-07-30','-d-',-99999999.99999999,7774901806701740032,16,'-1312358400',0000000000);

The last record is necessary for reproducing this bug, as is the UNIQUE idx1(c01,c02).

It is also possible to remove a few records from the end of the INSERT statement. The following record must be included for the crash to occur:

INSERT IGNORE INTO t VALUES
 
…,
 
('null', '20171124111824.006946', -6300817353668034560, 'null', '-2015-03-15-', 'j', 17688, -22, NULL, '-1845690368', 6);

Comment by Marko Mäkelä [ 2023-10-01 ]

Unfortunately, I haven’t been able to spend time on this so far, due to other priorities.

Comment by Marko Mäkelä [ 2023-11-23 ]

If I enable cmake -DWITH_INNODB_EXTRA_DEBUG:BOOL=ON, then I will get a validation error during the execution of the REPLACE statement:

10.4 e39c497c809511bcc37a658405c7aa4b5be2cf6a

 
#0  page_zip_fail_func (fmt=fmt@entry=0x55c2eb81ca60 "page_zip_validate: record content: 0x%02x") at /mariadb/10.4/storage/innobase/page/page0zip.cc:124
 
#1  0x000055c2ec52aeea in page_zip_validate_low (page_zip=0x7fddfdaff8d0, page=page@entry=0x7fddfe004000 "", index=index@entry=0x7fddec0bdbc0, sloppy=<optimized out>) at /mariadb/10.4/storage/innobase/page/page0zip.cc:3436
 
#2  0x000055c2ec52af0b in page_zip_validate (page_zip=<optimized out>, page=page@entry=0x7fddfe004000 "", index=index@entry=0x7fddec0bdbc0) at /mariadb/10.4/storage/innobase/page/page0zip.cc:3475
 
#3  0x000055c2ec41c215 in btr_cur_search_to_nth_level (index=index@entry=0x7fddec0bdbc0, level=<optimized out>, tuple=tuple@entry=0x7fddec088750, mode=mode@entry=PAGE_CUR_LE, latch_mode=<optimized out>, latch_mode@entry=1, cursor=cursor@entry=0x7fddf407ec00, file=<optimized out>, line=<optimized out>, mtr=<optimized out>, autoinc=<optimized out>) at /mariadb/10.4/storage/innobase/btr/btr0cur.cc:1810
 
#4  0x000055c2ec5a7ec3 in btr_pcur_open_low (index=index@entry=0x7fddec0bdbc0, level=<optimized out>, level@entry=0, tuple=tuple@entry=0x7fddec088750, mode=mode@entry=PAGE_CUR_LE, latch_mode=latch_mode@entry=1, cursor=cursor@entry=0x7fddf407ec00, file=<optimized out>, line=<optimized out>, autoinc=0, mtr=0x7fddf407ed40) at /mariadb/10.4/storage/innobase/include/btr0pcur.inl:441
 
#5  0x000055c2ec5a80f9 in row_search_on_row_ref (pcur=pcur@entry=0x7fddf407ec00, mode=mode@entry=1, table=table@entry=0x7fddec09b6e0, ref=0x7fddec088750, mtr=mtr@entry=0x7fddf407ed40) at /mariadb/10.4/storage/innobase/row/row0row.cc:1216
 
#6  0x000055c2ec5a837b in row_get_clust_rec (mode=mode@entry=1, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, clust_index=clust_index@entry=0x7fddf407ed38, mtr=mtr@entry=0x7fddf407ed40) at /mariadb/10.4/storage/innobase/row/row0row.cc:1267
 
#7  0x000055c2ec5d56a6 in row_vers_impl_x_locked (caller_trx=caller_trx@entry=0x7fddfe339110, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, offsets=offsets@entry=0x7fddf407f380) at /mariadb/10.4/storage/innobase/row/row0vers.cc:412
 
#8  0x000055c2ec4d0f00 in lock_sec_rec_some_has_impl (caller_trx=caller_trx@entry=0x7fddfe339110, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, offsets=offsets@entry=0x7fddf407f380) at /mariadb/10.4/storage/innobase/lock/lock0lock.cc:1249
 
#9  0x000055c2ec4da897 in lock_rec_convert_impl_to_expl (caller_trx=0x7fddfe339110, block=block@entry=0x7fddfdaffae8, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, offsets=offsets@entry=0x7fddf407f380) at /mariadb/10.4/storage/innobase/lock/lock0lock.cc:5666
 
#10 0x000055c2ec4dd5db in lock_sec_rec_read_check_and_lock (flags=flags@entry=0, block=block@entry=0x7fddfdaffae8, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, offsets=offsets@entry=0x7fddf407f380, mode=mode@entry=LOCK_X, gap_mode=0, thr=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/lock/lock0lock.cc:5902
 
#11 0x000055c2ec562337 in row_ins_set_exclusive_rec_lock (type=type@entry=0, block=block@entry=0x7fddfdaffae8, rec=rec@entry=0x7fddfe0080a5 "", index=index@entry=0x7fddec0a36b0, offsets=offsets@entry=0x7fddf407f380, thr=thr@entry=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:1435
 
#12 0x000055c2ec566636 in row_ins_scan_sec_index_for_duplicate (flags=flags@entry=0, index=index@entry=0x7fddec0a36b0, entry=entry@entry=0x7fddec0be0d0, thr=thr@entry=0x7fddec0c0d40, s_latch=s_latch@entry=false, mtr=mtr@entry=0x7fddf407f9e0, offsets_heap=<optimized out>) at /mariadb/10.4/storage/innobase/row/row0ins.cc:2197
 
#13 0x000055c2ec5684aa in row_ins_sec_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=2, index=index@entry=0x7fddec0a36b0, offsets_heap=<optimized out>, offsets_heap@entry=0x7fddec07eee0, heap=heap@entry=0x7fddec07c0e0, entry=entry@entry=0x7fddec0be0d0, trx_id=<optimized out>, thr=<optimized out>) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3089
 
#14 0x000055c2ec56af1c in row_ins_sec_index_entry (index=index@entry=0x7fddec0a36b0, entry=entry@entry=0x7fddec0be0d0, thr=thr@entry=0x7fddec0c0d40, check_foreign=check_foreign@entry=true) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3361
 
#15 0x000055c2ec56b0d4 in row_ins_index_entry (index=0x7fddec0a36b0, entry=0x7fddec0be0d0, thr=thr@entry=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3409
 
#16 0x000055c2ec56b220 in row_ins_index_entry_step (node=node@entry=0x7fddec0c05e0, thr=thr@entry=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3576
 
#17 0x000055c2ec56b30a in row_ins (node=node@entry=0x7fddec0c05e0, thr=thr@entry=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3713
 
#18 0x000055c2ec56b698 in row_ins_step (thr=thr@entry=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:3856
 
#19 0x000055c2ec585999 in row_insert_for_mysql (mysql_rec=mysql_rec@entry=0x7fddec042cf0 "\340\a", prebuilt=0x7fddec0bff00, ins_mode=<optimized out>) at /mariadb/10.4/storage/innobase/row/row0mysql.cc:1395
 
#20 0x000055c2ec3521f6 in ha_innobase::write_row (this=0x7fddec042170, record=0x7fddec042cf0 "\340\a") at /mariadb/10.4/storage/innobase/handler/ha_innodb.cc:8171

This occurs on the secondary index idx1 when attempting to insert the entry (c01,c02,id)=(0,'-f-',3). The row is near the start of the statement:

REPLACE INTO `t` VALUES ('-benefit-','2003-03-05 02:41:36.060910',0,'d','2014-10-26','-al',NULL,7,1,'-363986944',0000013440),
 
('321781760','2010-09-14 20:06:40.052139',-3587961528130732032,'-e-','0000-00-00','nul',NULL,8,2,'-item-',0000000000),
 
('-knife-','0000-00-00 00:00:00.000000',3,'x','2026-07-04','-f-',6466.00000000,-15407,3,'-receiver-',0000000000),

Comment by Marko Mäkelä [ 2023-11-23 ]

The inconsistency was introduced when the REPLACE statement inserted a clustered index record corresponding ot the 3rd row in the previous mini-transaction. It updated the DB_ROLL_PTR in the uncompressed page to roll_ptr=0x30000013703de:

10.4 e39c497c809511bcc37a658405c7aa4b5be2cf6a

 
#2  0x000055c2ec40ce88 in trx_write_roll_ptr (roll_ptr=844424950514654, ptr=0x7fddfe004133 "\003") at /mariadb/10.4/storage/innobase/include/trx0undo.h:88
 
#3  row_upd_rec_sys_fields (rec=rec@entry=0x7fddfe004125 "", page_zip=page_zip@entry=0x0, index=index@entry=0x7fddec0bdbc0, offsets=offsets@entry=0x7fddf407fee0, trx=0x7fddfe339110, 
    roll_ptr=roll_ptr@entry=844424950514654) at /mariadb/10.4/storage/innobase/include/row0upd.inl:195
 
#4  0x000055c2ec413c20 in btr_cur_update_in_place (flags=flags@entry=0, cursor=cursor@entry=0x7fddf407fdf0, offsets=0x7fddf407fee0, update=update@entry=0x7fddec07ef60, cmpl_info=cmpl_info@entry=0, 
    thr=thr@entry=0x7fddec0c0d40, trx_id=37, mtr=0x7fddf4080140) at /mariadb/10.4/storage/innobase/btr/btr0cur.cc:4341
 
#5  0x000055c2ec4180d0 in btr_cur_optimistic_update (flags=flags@entry=0, cursor=cursor@entry=0x7fddf407fdf0, offsets=offsets@entry=0x7fddf407fde0, heap=heap@entry=0x7fddf407fdd8, 
    update=update@entry=0x7fddec07ef60, cmpl_info=cmpl_info@entry=0, thr=0x7fddec0c0d40, trx_id=37, mtr=0x7fddf4080140) at /mariadb/10.4/storage/innobase/btr/btr0cur.cc:4628
 
#6  0x000055c2ec56258d in row_ins_clust_index_entry_by_modify (pcur=pcur@entry=0x7fddf407fdf0, flags=flags@entry=0, mode=mode@entry=2, offsets=offsets@entry=0x7fddf407fde0, 
    offsets_heap=offsets_heap@entry=0x7fddf407fdd8, heap=heap@entry=0x7fddec07eee0, entry=0x7fddec0bdf60, thr=0x7fddec0c0d40, mtr=0x7fddf4080140) at /mariadb/10.4/storage/innobase/row/row0ins.cc:321
 
#7  0x000055c2ec56742c in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=mode@entry=2, index=index@entry=0x7fddec0bdbc0, n_uniq=n_uniq@entry=1, entry=entry@entry=0x7fddec0bdf60, n_ext=n_ext@entry=0, 
    thr=0x7fddec0c0d40) at /mariadb/10.4/storage/innobase/row/row0ins.cc:2786

When the validation fails, the DB_ROLL_PTR on the temp_page that was decompressed from the compressed page is 0x30000013703a9, with a different least significant byte. I did not observe any write of the DB_ROLL_PTR to the compressed page. The reason is obvious. The following patch is all it takes to fix not only my reduced test, but the original r8.test:

diff --git a/storage/innobase/btr/btr0cur.cc b/storage/innobase/btr/btr0cur.cc
 
index 0f6cdf25ca2..38bbf2d1bc1 100644
 
--- a/storage/innobase/btr/btr0cur.cc
 
+++ b/storage/innobase/btr/btr0cur.cc
 
@@ -4338,7 +4338,7 @@ btr_cur_update_in_place(
 
 	}
 
 
 	if (!(flags & BTR_KEEP_SYS_FLAG)) {
 
-		row_upd_rec_sys_fields(rec, NULL, index, offsets,
 
+		row_upd_rec_sys_fields(rec, page_zip, index, offsets,
 
 				       thr_get_trx(thr), roll_ptr);
 
 	}

In MDEV-12353 (10.5) I had refactored this code so that we pass block instead of page_zip. Hence, the ROW_FORMAT=COMPRESSED page would be updated.

It turns out that this bug was there already when the InnoDB Plugin 1.0.4 was added to MySQL 5.1.

Generated at Thu Feb 08 10:24:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.