Details
-
Bug
-
Status: Confirmed (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11, 11.4, 11.8, 12.3, 13.1, 10.11.14
-
None
-
- MariaDB 10.11.14-MariaDB-0ubuntu0.24.04.1 (Ubuntu 24.04 package), source revision 053f9bcb5b147bf00edb99e1310bae9125b7f125
- Galera provider 26.4.16, 3-node cluster, writes go to all 3 nodes (multi-master)
- Kernel 6.8.0-147-generic, InnoDB, innodb_buffer_pool_size=3G, innodb_page_size=16384
- wsrep_slave_threads=1, innodb_flush_log_at_trx_commit=2, binlog_row_image=FULL,
wsrep_auto_increment_control=ON, tx_isolation=REPEATABLE-READ, no encryption, no page compression
- Application: jBPM / Red Hat Process Automation Manager (Hibernate). Table written by normal INSERTs.- MariaDB 10.11.14-MariaDB-0ubuntu0.24.04.1 (Ubuntu 24.04 package), source revision 053f9bcb5b147bf00edb99e1310bae9125b7f125 - Galera provider 26.4.16, 3-node cluster, writes go to all 3 nodes (multi-master) - Kernel 6.8.0-147-generic, InnoDB, innodb_buffer_pool_size=3G, innodb_page_size=16384 - wsrep_slave_threads=1, innodb_flush_log_at_trx_commit=2, binlog_row_image=FULL, wsrep_auto_increment_control=ON, tx_isolation=REPEATABLE-READ, no encryption, no page compression - Application: jBPM / Red Hat Process Automation Manager (Hibernate). Table written by normal INSERTs.
-
Can result in hang or crash
Description
ENVIRONMENT
- MariaDB 10.11.14-MariaDB-0ubuntu0.24.04.1 (Ubuntu 24.04 package), source revision 053f9bcb5b147bf00edb99e1310bae9125b7f125
- Galera provider 26.4.16, 3-node cluster, writes go to all 3 nodes (multi-master)
- Kernel 6.8.0-147-generic, InnoDB, innodb_buffer_pool_size=3G, innodb_page_size=16384
- wsrep_slave_threads=1, innodb_flush_log_at_trx_commit=2, binlog_row_image=FULL,
wsrep_auto_increment_control=ON, tx_isolation=REPEATABLE-READ, no encryption, no page compression - Application: jBPM / Red Hat Process Automation Manager (Hibernate). Table written by normal INSERTs.
TABLE
CREATE TABLE `Content` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`content` longblob DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
About 274,000 rows, about 9 GB, about 101,000 rows with off-page BLOBs, largest BLOB 12 MB.
SYMPTOM
Any SELECT that returns `content` for an affected row aborts the server:
InnoDB: Assertion failure in file ./storage/innobase/row/row0sel.cc line 3034
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
mariadbd got signal 6
Seen with a const PK lookup (handler::index_read_idx_map from JOIN::optimize_inner) and with a
multi-range read (DsMrr_impl::dsmrr_next / QUICK_RANGE_SELECT::get_next).
Range scans that end just before an affected row also abort (the next row is fetched with its BLOB
before the end-of-range check). DELETE of an affected row also aborts (full row read).
ANALYSIS (verified against the 10.11.14 source)
- row_sel_store_mysql_field() hits ut_a(...READ_UNCOMMITTED) when btr_rec_copy_externally_stored_field()
returns NULL. That happens only when the 20-byte BLOB reference is all zeros ("not written yet"). - Under REPEATABLE-READ only committed versions are read, so committed records carry a zero BLOB pointer.
- A freed BLOB sets page_no=FIL_NULL and length 0; it does not zero the whole reference, so this is not a purged BLOB.
- I reproduced the identical assertion on 10.11.14 by zeroing a BLOB reference in a copy of the tablespace.
EVIDENCE FROM ALL 3 NODES (offline scan of each node's .ibd file, all leaf pages)
- Per node: 62, 64 and 69 records with an all-zero BLOB reference. 105 distinct ids in total.
- 43 ids are bad on all 3 nodes. 62 ids are bad on 1 or 2 nodes only and are correct on the others.
=> The defect is created independently on each node, not copied by SST or replication. - No page checksum errors, no broken record chains, no other anomalies in the tablespace.
- All affected records: not delete-marked, DB_TRX_ID reset to 0 (old committed records).
- Affected rows come in groups of consecutive ids on the same page. Inside a group the ids mostly step by 3
(auto_increment_increment=3), so each group is probably one multi-row transaction from one node.
One group steps by 2 (written while the cluster had 2 members). - Using the current auto_increment_offset per node, bad rows occur on the originating node AND on
applier-only nodes. (Offsets change on membership changes, so this mapping is approximate for old rows.) - First affected rows were written in August 2026; new groups keep appearing until now.
WHAT WE RULED OUT
- Disk/page corruption (checksums clean, defect differs per node).
- rsync SST propagation (defect differs per node).
- Parallel apply (wsrep_slave_threads=1).
- NO PAD collation bugs (
MDEV-28328/MDEV-30111): BIGINT PK only, PAD collation, fix already in 10.11.11.
QUESTIONS
1. Is this a known issue? Similar reports: MDEV-29143 (open), MDEV-18932 (incomplete).
2. Could MDEV-39681 (BF-abort / orphaned lock wait under multi-master load, fixed in 10.11.19) leave
a committed record with a zero BLOB pointer?
3. Which logs or settings help to catch the moment the defect is created?
AVAILABLE ON REQUEST (private upload only, contains business data)
- Core dumps from 2 nodes, full error logs, the scan lists, the offline scanner script.
WORKAROUND USED
Copy all good rows one by one (WHERE id = N) into a new table, recover the rows that are good on
another node, RENAME TABLE to swap. Affected rows on all 3 nodes are lost.
Attachments
Issue Links
- relates to
-
MDEV-29143 mysql crash when selecting from large table
-
- Open
-