Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-42080

InnoDB: committed clustered-index records with all-zero BLOB pointer on Galera nodes; SELECT aborts at row0sel.cc:3034 (isolation_level == TRX_ISO_READ_UNCOMMITTED)

    XMLWordPrintable

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11, 11.4, 11.8, 12.3, 13.1, 10.11.14
    • 10.11, 11.4, 11.8, 12.3, 13.1
    • None
    • Can result in hang or crash

    Description

      ENVIRONMENT

      • MariaDB 10.11.14-MariaDB-0ubuntu0.24.04.1 (Ubuntu 24.04 package), source revision 053f9bcb5b147bf00edb99e1310bae9125b7f125
      • Galera provider 26.4.16, 3-node cluster, writes go to all 3 nodes (multi-master)
      • Kernel 6.8.0-147-generic, InnoDB, innodb_buffer_pool_size=3G, innodb_page_size=16384
      • wsrep_slave_threads=1, innodb_flush_log_at_trx_commit=2, binlog_row_image=FULL,
        wsrep_auto_increment_control=ON, tx_isolation=REPEATABLE-READ, no encryption, no page compression
      • Application: jBPM / Red Hat Process Automation Manager (Hibernate). Table written by normal INSERTs.

      TABLE
      CREATE TABLE `Content` (
      `id` bigint(20) NOT NULL AUTO_INCREMENT,
      `content` longblob DEFAULT NULL,
      PRIMARY KEY (`id`)
      ) ENGINE=InnoDB DEFAULT CHARSET=latin1 ROW_FORMAT=DYNAMIC;
      About 274,000 rows, about 9 GB, about 101,000 rows with off-page BLOBs, largest BLOB 12 MB.

      SYMPTOM
      Any SELECT that returns `content` for an affected row aborts the server:
      InnoDB: Assertion failure in file ./storage/innobase/row/row0sel.cc line 3034
      InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
      mariadbd got signal 6
      Seen with a const PK lookup (handler::index_read_idx_map from JOIN::optimize_inner) and with a
      multi-range read (DsMrr_impl::dsmrr_next / QUICK_RANGE_SELECT::get_next).
      Range scans that end just before an affected row also abort (the next row is fetched with its BLOB
      before the end-of-range check). DELETE of an affected row also aborts (full row read).

      ANALYSIS (verified against the 10.11.14 source)

      • row_sel_store_mysql_field() hits ut_a(...READ_UNCOMMITTED) when btr_rec_copy_externally_stored_field()
        returns NULL. That happens only when the 20-byte BLOB reference is all zeros ("not written yet").
      • Under REPEATABLE-READ only committed versions are read, so committed records carry a zero BLOB pointer.
      • A freed BLOB sets page_no=FIL_NULL and length 0; it does not zero the whole reference, so this is not a purged BLOB.
      • I reproduced the identical assertion on 10.11.14 by zeroing a BLOB reference in a copy of the tablespace.

      EVIDENCE FROM ALL 3 NODES (offline scan of each node's .ibd file, all leaf pages)

      • Per node: 62, 64 and 69 records with an all-zero BLOB reference. 105 distinct ids in total.
      • 43 ids are bad on all 3 nodes. 62 ids are bad on 1 or 2 nodes only and are correct on the others.
        => The defect is created independently on each node, not copied by SST or replication.
      • No page checksum errors, no broken record chains, no other anomalies in the tablespace.
      • All affected records: not delete-marked, DB_TRX_ID reset to 0 (old committed records).
      • Affected rows come in groups of consecutive ids on the same page. Inside a group the ids mostly step by 3
        (auto_increment_increment=3), so each group is probably one multi-row transaction from one node.
        One group steps by 2 (written while the cluster had 2 members).
      • Using the current auto_increment_offset per node, bad rows occur on the originating node AND on
        applier-only nodes. (Offsets change on membership changes, so this mapping is approximate for old rows.)
      • First affected rows were written in August 2026; new groups keep appearing until now.

      WHAT WE RULED OUT

      • Disk/page corruption (checksums clean, defect differs per node).
      • rsync SST propagation (defect differs per node).
      • Parallel apply (wsrep_slave_threads=1).
      • NO PAD collation bugs (MDEV-28328 / MDEV-30111): BIGINT PK only, PAD collation, fix already in 10.11.11.

      QUESTIONS
      1. Is this a known issue? Similar reports: MDEV-29143 (open), MDEV-18932 (incomplete).
      2. Could MDEV-39681 (BF-abort / orphaned lock wait under multi-master load, fixed in 10.11.19) leave
      a committed record with a zero BLOB pointer?
      3. Which logs or settings help to catch the moment the defect is created?

      AVAILABLE ON REQUEST (private upload only, contains business data)

      • Core dumps from 2 nodes, full error logs, the scan lists, the offline scanner script.

      WORKAROUND USED
      Copy all good rows one by one (WHERE id = N) into a new table, recover the rows that are good on
      another node, RENAME TABLE to swap. Affected rows on all 3 nodes are lost.

      Attachments

        1. mdev42080.opt
          0.0 kB
          Shipra Jain
        2. mdev42080.test
          3 kB
          Shipra Jain

        Issue Links

          Activity

            People

              thiru Thirunarayanan Balathandayuthapani
              Buvi30 Buvnesh Venkatesan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.