Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-41373

SIGSEGV in rec_copy_prefix_to_buf via btr_pcur_store_position during table scan: cursor->old_n_fields=60 for PRIMARY with n_uniq=1

    XMLWordPrintable

Details

    • Bug
    • Status: Needs Feedback (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.6, 10.11.11, 10.11.18
    • None
    • Debian 12 (bookworm), kernel 6.1.0-53-amd64 (6.1.187-1), package mariadb-server 1:10.11.18-0+deb12u1, x86_64
    • Can result in hang or crash

    Description

      Our server crashes repeatedly with SIGSEGV (NULL pointer read) during InnoDB table scans. The crash has occurred since April 2024 across several 10.11 releases (at least 10.11.6, 10.11.11, 10.11.18), always in the same code path: handler::ha_rnd_next -> ha_innobase::general_fetch -> row_search_mvcc -> btr_pcur_store_position -> rec_copy_prefix_to_buf.

      Different tables and statements are affected (SELECT, UPDATE, DELETE, and SELECT with filesort), but the stack is always identical: a sequential scan (rr_sequential or filesort) calling ha_rnd_next. There is no DDL running (Com_alter_table, Com_create_table, Com_drop_table, Com_truncate, Com_rename_table are all 0 since startup), and the adaptive hash index is OFF. No OOM, no MCE/EDAC errors.

      Symbolized backtrace (10.11.18-0+deb12u1, debug symbols via debuginfod.debian.net)

      #0  rec_copy_prefix_to_buf (rec=0x77bee3d48f4f "", index=0x77be98082080, n_fields=60, buf=0x77be9808b150, buf_size=0x77be9808b158) at storage/innobase/rem/rem0rec.cc:2051
      #1  btr_pcur_store_position (cursor=0x77be9808b098, mtr=0x77bec01edef0) at storage/innobase/btr/btr0pcur.cc:178
      #2  row_search_mvcc (buf=..., mode=PAGE_CUR_G, prebuilt=0x77be9808aec0, match_mode=0, direction=1) at storage/innobase/row/row0sel.cc:5762
      #3  ha_innobase::general_fetch (this=0x77be9807fc80, buf=..., direction=1, match_mode=0) at storage/innobase/handler/ha_innodb.cc:9272
      #4  handler::ha_rnd_next (this=0x77be9807fc80, buf=...) at sql/handler.cc:3613
      #5  rr_sequential (info=0x77be84019c28) at sql/records.cc:519
      #6  READ_RECORD::read_record at sql/records.h:81
      #7  sub_select at sql/sql_select.cc:23096
      #8  do_select at sql/sql_select.cc:22600
      #9  JOIN::exec_inner at sql/sql_select.cc:5023
      #10 JOIN::exec at sql/sql_select.cc:4807
      #11 mysql_select at sql/sql_select.cc:5285
      

      Query at crash time: Select * from mc_asset where ecoAssetID = 1149079 (ecoAssetID is not indexed, so this is a full scan of the clustered index, approx. 620k rows).

      Values from the core dump

      frame rec_copy_prefix_to_buf:
        n_fields = 60
        i = 32, col = 0x0          <- loop ran past the end of index->fields
        index->table->name = "mod_explore_production/mc_asset"
        index->name = "PRIMARY"
        index->type = 3            (DICT_CLUSTERED | DICT_UNIQUE)
        index->n_fields = 32       (30 user columns + DB_TRX_ID + DB_ROLL_PTR)
        index->n_uniq = 1          (PRIMARY KEY (AssetID))
        index->n_core_fields = 32
        index->n_nullable = 29
        index->n_core_null_bytes = 4
        index->table->instant = 0x0
        index->table->flags = 33
        index->fields[0] = {col = 0x77be9808a8d0, name = "AssetID", fixed_len = 4}
        index->fields[1] = {col = 0x77be9808acb0, name = "DB_TRX_ID", fixed_len = 6}
       
      frame btr_pcur_store_position:
        cursor->old_n_fields = 60
        cursor->rel_pos = BTR_PCUR_ON
      

      The index is not an ibuf index (type = 3), so btr_pcur_store_position should set cursor->old_n_fields = dict_index_get_n_unique_in_tree(index) = n_uniq = 1. Instead old_n_fields is 60, which makes rec_copy_prefix_to_buf iterate beyond index->n_fields and dereference a NULL dict_col_t at i = 32. The index definition itself looks consistent, so it seems old_n_fields (or the data it was derived from) was overwritten or computed from the wrong source.

      The record bytes are not available in the core because the buffer pool is excluded from core dumps (innodb_buffer_pool_in_core_file default).

      Table definition

      CREATE TABLE `mc_asset` (
        `AssetID` int(10) NOT NULL AUTO_INCREMENT,
        `ecoAssetID` int(18) DEFAULT 0,
        ... (30 columns: int, double, varchar(255/256), mediumtext, longtext, text)
        PRIMARY KEY (`AssetID`),
        KEY `ProjectID` (`ProjectID`) USING BTREE,
        KEY `ElementTypeID` (`ElementTypeID`),
        KEY `CategoryID` (`CategoryID`),
        KEY `AssetTypeID` (`AssetTypeID`)
      ) ENGINE=InnoDB ROW_FORMAT=DYNAMIC DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci
      

      Crash history

      All crashes since 2024-04-11 (about 45) show the same top frames above ha_rnd_next, with identical offsets per binary version:

      • 2024 (10.11.6): +0xd54b47 / +0xddd785 / +0xd8e392 / +0xcd0ba5 -> ha_rnd_next
      • 2025 to Sept 2026 (10.11.11): +0xe1adc7 / +0xebc6d7 / +0xe59569 / +0xd8dce7 -> ha_rnd_next
      • Sept 2026 onwards (10.11.18): +0xd887b7 / +0xe0de07 / +0xdc26ac / +0xd0d396 -> ha_rnd_next

      Affected tables (all InnoDB, ROW_FORMAT=DYNAMIC, no instant ALTER): mc_asset, mc_risk, eco_sco_pra, eco_sco_pvsp, eco_scenario_intens, eco_scenario_objekte, eco_messages, eco_projekt_log, eco_objekt_faktoren. The server sometimes crashes again a few minutes after restart under the same workload.

      Relevant configuration

      innodb_buffer_pool_size = 4G
      innodb_adaptive_hash_index = OFF
      thread_handling = one-thread-per-connection
      query_cache_type = OFF
      table_definition_cache = 400 (405 InnoDB tables)
      InnoDB uses io_uring
      

      Possibly related: MDEV-36773 (SIGSEGVs in row_search_mvcc / persistent cursor state).

      The core file and matching binary are preserved; I can provide further gdb output on request (the core contains production data, so I can't attach it).

      Attachments

        1. mariadb.err
          1.32 MB
          Matthias Gerber

        Activity

          People

            shipjain Shipra Jain
            gerber@slf.ch Matthias Gerber
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.