Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-33208

InnoDB: Trying to access update undo rec field 50 in index `PRIMARY` of table NNN but index has only 14 fields

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Cannot Reproduce
    • 10.11.6
    • N/A
    • Storage Engine - InnoDB
    • None
    • Linux firefly 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08) x86_64 GNU/Linux
      10.11.6-MariaDB-1:10.11.6+maria~deb10-log

    Description

      Hit this error:

      2024-01-09 20:15:31 0 [ERROR] InnoDB: Trying to access update undo rec field 50 in index `PRIMARY` of table `meshok`.`btc_invoice` but index has only 14 fields Submit a detailed bug report to https://jira.mariadb.org/. Run also CHECK TABLE `meshok`.`btc_invoice`. n_fields = 44, i = 0
      240109 20:15:31 [ERROR] mysqld got signal 11 ;
      Sorry, we probably made a mistake, and this is a bug.

      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 10.11.6-MariaDB-1:10.11.6+maria~deb10-log source revision: fecd78b83785d5ae96f2c6ff340375be803cd299
      key_buffer_size=131072
      read_buffer_size=131072
      max_used_connections=1368
      max_threads=5002
      thread_count=1268
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11017983 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x560e3e1e4ec8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f66850a6c38 thread_stack 0x49000
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x560e3c6bf9ae]
      /usr/sbin/mariadbd(handle_fatal_signal+0x475)[0x560e3c1b9d95]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7f9a2e5d4730]
      /usr/sbin/mariadbd(+0xdbe208)[0x560e3c535208]
      /usr/sbin/mariadbd(+0xd84a03)[0x560e3c4fba03]
      /usr/sbin/mariadbd(+0x6f5fba)[0x560e3be6cfba]
      /usr/sbin/mariadbd(+0xddb083)[0x560e3c552083]
      /usr/sbin/mariadbd(_ZN5tpool10task_group7executeEPNS_4taskE+0x8c)[0x560e3c65462c]
      /usr/sbin/mariadbd(_ZN5tpool19thread_pool_generic11worker_mainEPNS_11worker_dataE+0x5f)[0x560e3c65241f]
      /lib/x86_64-linux-gnu/libstdc++.so.6(+0xbbb2f)[0x7f9a2e4f4b2f]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3)[0x7f9a2e5c9fa3]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f9a2e1ed06f]
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): (null)
      Connection ID (thread ID): 0
      Status: NOT_KILLED

      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off

      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
      information that should help you find out what is causing the crash.

      We think the query pointer is invalid, but we will try to print it anyway.
      Query:

      Writing a core file...
      Working directory at /db/main/tables
      Resource Limits:
      Limit Soft Limit Hard Limit Units
      Max cpu time unlimited unlimited seconds
      Max file size unlimited unlimited bytes
      Max data size unlimited unlimited bytes
      Max stack size 8388608 unlimited bytes
      Max core file size 0 unlimited bytes
      Max resident set unlimited unlimited bytes
      Max processes 1031220 1031220 processes
      Max open files 32768 32768 files
      Max locked memory 65536 65536 bytes
      Max address space unlimited unlimited bytes
      Max file locks unlimited unlimited locks
      Max pending signals 1031220 1031220 signals
      Max msgqueue size 819200 819200 bytes
      Max nice priority 0 0
      Max realtime priority 0 0
      Max realtime timeout unlimited unlimited us
      Core pattern: core

      Kernel version: Linux version 4.19.0-25-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.289-2 (2023-08-08)

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33588 buf::Block_hint is a performance hog

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33771 InnoDB: Trying to access update undo rec field 32 in index `PRIMARY` of table `database`.`table` but index has only 16 fields

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33819 The purge of committed history is mis-parsing some log

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33275 buf_flush_LRU(): mysql_mutex_assert_owner(&buf_pool.mutex) failed

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 14 older comments
            arnklo Arnas Klova added a comment - - edited

            Of course, there have been multiple tables from different databases. Hope it helps.

            | wp_actionscheduler_claims | CREATE TABLE `wp_actionscheduler_claims` (
            		 
              `claim_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
            		 
              `date_created_gmt` datetime DEFAULT '0000-00-00 00:00:00',
            		 
              PRIMARY KEY (`claim_id`),
            		 
              KEY `date_created_gmt` (`date_created_gmt`)
            		 
            ) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci |


            | wp_actionscheduler_actions | CREATE TABLE `wp_actionscheduler_actions` (
            		 
              `action_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
            		 
              `hook` varchar(191) NOT NULL,
            		 
              `status` varchar(20) NOT NULL,
            		 
              `scheduled_date_gmt` datetime DEFAULT '0000-00-00 00:00:00',
            		 
              `scheduled_date_local` datetime DEFAULT '0000-00-00 00:00:00',
            		 
              `args` varchar(191) DEFAULT NULL,
            		 
              `schedule` longtext DEFAULT NULL,
            		 
              `group_id` bigint(20) unsigned NOT NULL DEFAULT 0,
            		 
              `attempts` int(11) NOT NULL DEFAULT 0,
            		 
              `last_attempt_gmt` datetime DEFAULT '0000-00-00 00:00:00',
            		 
              `last_attempt_local` datetime DEFAULT '0000-00-00 00:00:00',
            		 
              `claim_id` bigint(20) unsigned NOT NULL DEFAULT 0,
            		 
              `extended_args` varchar(8000) DEFAULT NULL,
            		 
              `priority` tinyint(3) unsigned NOT NULL DEFAULT 10,
            		 
              PRIMARY KEY (`action_id`),
            		 
              KEY `hook` (`hook`),
            		 
              KEY `status` (`status`),
            		 
              KEY `scheduled_date_gmt` (`scheduled_date_gmt`),
            		 
              KEY `args` (`args`),
            		 
              KEY `group_id` (`group_id`),
            		 
              KEY `last_attempt_gmt` (`last_attempt_gmt`),
            		 
              KEY `claim_id` (`claim_id`),
            		 
              KEY `claim_id_status_scheduled_date_gmt` (`claim_id`,`status`,`scheduled_date_gmt`),
            		 
              KEY `hook_status_scheduled_date_gmt` (`hook`(163),`status`,`scheduled_date_gmt`),
            		 
              KEY `status_scheduled_date_gmt` (`status`,`scheduled_date_gmt`)
            		 
            ) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci |

            arnklo Arnas Klova added a comment - - edited Of course, there have been multiple tables from different databases. Hope it helps. | wp_actionscheduler_claims | CREATE TABLE `wp_actionscheduler_claims` ( `claim_id` bigint( 20 ) unsigned NOT NULL AUTO_INCREMENT, `date_created_gmt` datetime DEFAULT '0000-00-00 00:00:00' , PRIMARY KEY (`claim_id`), KEY `date_created_gmt` (`date_created_gmt`) ) ENGINE=InnoDB AUTO_INCREMENT= 2 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_520_ci | | wp_actionscheduler_actions | CREATE TABLE `wp_actionscheduler_actions` ( `action_id` bigint( 20 ) unsigned NOT NULL AUTO_INCREMENT, `hook` varchar( 191 ) NOT NULL, `status` varchar( 20 ) NOT NULL, `scheduled_date_gmt` datetime DEFAULT '0000-00-00 00:00:00' , `scheduled_date_local` datetime DEFAULT '0000-00-00 00:00:00' , `args` varchar( 191 ) DEFAULT NULL, `schedule` longtext DEFAULT NULL, `group_id` bigint( 20 ) unsigned NOT NULL DEFAULT 0 , `attempts` int ( 11 ) NOT NULL DEFAULT 0 , `last_attempt_gmt` datetime DEFAULT '0000-00-00 00:00:00' , `last_attempt_local` datetime DEFAULT '0000-00-00 00:00:00' , `claim_id` bigint( 20 ) unsigned NOT NULL DEFAULT 0 , `extended_args` varchar( 8000 ) DEFAULT NULL, `priority` tinyint( 3 ) unsigned NOT NULL DEFAULT 10 , PRIMARY KEY (`action_id`), KEY `hook` (`hook`), KEY `status` (`status`), KEY `scheduled_date_gmt` (`scheduled_date_gmt`), KEY `args` (`args`), KEY `group_id` (`group_id`), KEY `last_attempt_gmt` (`last_attempt_gmt`), KEY `claim_id` (`claim_id`), KEY `claim_id_status_scheduled_date_gmt` (`claim_id`,`status`,`scheduled_date_gmt`), KEY `hook_status_scheduled_date_gmt` (`hook`( 163 ),`status`,`scheduled_date_gmt`), KEY `status_scheduled_date_gmt` (`status`,`scheduled_date_gmt`) ) ENGINE=InnoDB AUTO_INCREMENT= 9 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci |
            marko Marko Mäkelä added a comment -

            arnklo, I am sorry for the delay. The PRIMARY KEY index of the CREATE TABLE wp_actionscheduler_actions statement could in fact match the index has only 16 fields part of the error log output. The fields would be (action_id,DB_TRX_ID,DB_ROLL_PTR,hook,status,…,priority).

            Meanwhile we reproduced an https://rr-project.org trace of the bug MDEV-33819 in our internal testing. It would seem to match the symptoms here. The purge of history would attempt to interpret some arbitrary bytes in an undo log page as an undo log record. In this case it would fail because those arbitrary bytes misinterpreted as an undo log record would seem to refer to a field that does not exist in a secondary index.

            The fix of MDEV-33819 is currently present in the 10.6 development branch but not yet merged to the 10.11 branch. Hence I cannot request you to test a 10.11 development snapshot to see if this bug has no longer occurs with that. Our scheduled quarterly releases should happen within a couple of weeks.

            marko Marko Mäkelä added a comment - arnklo , I am sorry for the delay. The PRIMARY KEY index of the CREATE TABLE wp_actionscheduler_actions statement could in fact match the index has only 16 fields part of the error log output. The fields would be (action_id,DB_TRX_ID,DB_ROLL_PTR,hook,status,…,priority). Meanwhile we reproduced an https://rr-project.org trace of the bug MDEV-33819 in our internal testing. It would seem to match the symptoms here. The purge of history would attempt to interpret some arbitrary bytes in an undo log page as an undo log record. In this case it would fail because those arbitrary bytes misinterpreted as an undo log record would seem to refer to a field that does not exist in a secondary index. The fix of MDEV-33819 is currently present in the 10.6 development branch but not yet merged to the 10.11 branch. Hence I cannot request you to test a 10.11 development snapshot to see if this bug has no longer occurs with that. Our scheduled quarterly releases should happen within a couple of weeks.
            marko Marko Mäkelä added a comment -

            The fix of MDEV-33819 was meanwhile merged to the 10.11 branch, and anyone could test a recent development snapshot of 10.11 in a non-production environment. https://ci.mariadb.org/45877/ currently is the latest 10.11, and https://ci.mariadb.org/45878/ is the latest in the 10.6 series. Would this problem still occur with those builds?

            marko Marko Mäkelä added a comment - The fix of MDEV-33819 was meanwhile merged to the 10.11 branch, and anyone could test a recent development snapshot of 10.11 in a non-production environment. https://ci.mariadb.org/45877/ currently is the latest 10.11, and https://ci.mariadb.org/45878/ is the latest in the 10.6 series. Would this problem still occur with those builds?
            arnklo Arnas Klova added a comment -

            Hello Marko, sorry I forgot to reply. We've just deployed https://ci.mariadb.org/45877/ today on the problematic server; it appears to be stable without innodb_force_recovery. I will report back if the issue reoccurs, but it does seem to be very promising.

            arnklo Arnas Klova added a comment - Hello Marko, sorry I forgot to reply. We've just deployed https://ci.mariadb.org/45877/ today on the problematic server; it appears to be stable without innodb_force_recovery. I will report back if the issue reoccurs, but it does seem to be very promising.
            serg Sergei Golubchik added a comment -

            Thanks! I'm closing the issue now, but please feel free to add a comment if the issue reoccurs after all.

            serg Sergei Golubchik added a comment - Thanks! I'm closing the issue now, but please feel free to add a comment if the issue reoccurs after all.

            People

              marko Marko Mäkelä
              alexcool Alex
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.