Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-26977

mariadb 10.5.12 reboot loop in AWS | [ERROR] InnoDB: Insert buffer insert fails | Corruption of an index tree

    XMLWordPrintable

Details

    Description

      I was running a SQL proc which deleted records.

      2021-11-04 23:01:04 0 [ERROR] InnoDB: Insert buffer insert fails; page free 29, dtuple size 175
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373432303239222c2253455249414c5f4e554d424552223a22373432303239222c22455043223a22333033393544464138324639353634303030304235323844222c224e554d5f4c494e45223a352c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221742029","SERIAL_NUMBER":"742029","EPC":"30395DFA82F95640000B528D","NUM_LINE":5,"ITEM_TRANSACTION_MODE":"SALE"};;
       1: len 8; hex 800000001348b829; asc      H );;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881], size 16384, bitmap bits 0
      2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:01:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 2 fields;
       0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373431383636222c2253455249414c5f4e554d424552223a22373431383636222c22455043223a22333033393544464138324639353634303030304235314541222c224e554d5f4c494e45223a312c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221741866","SERIAL_NUMBER":"741866","EPC":"30395DFA82F95640000B51EA","NUM_LINE":1,"ITEM_TRANSACTION_MODE":"SALE"};;
       1: len 8; hex 800000001052dd2a; asc      R *;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
       0: len 8; hex 5246494454616730; asc RFIDTag0;;
       1: len 8; hex 800000000aefe10b; asc         ;;
      2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881] (721 records, index id 32581).
      2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafc58; asc     a  X;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafcd6; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafce2; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafd6a; asc     a  j;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0018; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0022; asc     a  ";;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0025; asc     a  %;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0039; asc     a  9;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0070; asc     a  p;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0433; asc     a  3;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0438; asc     a  8;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0468; asc     a  h;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb04ba; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0563; asc     a  c;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0629; asc     a  );;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb062d; asc     a  -;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0697; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb06c8; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0701; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb07dd; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0856; asc     a  V;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb08f8; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0936; asc     a  6;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0a03; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0a27; asc     a  ';;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0a2c; asc     a  ,;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0a32; asc     a  2;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
      InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cb0ae9; asc     a   ;;
       
      InnoDB: The table where this index record belongs
      InnoDB: is now probably corrupt. Please run CHECK TABLE on
      InnoDB: that table.
      2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
      2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      2021-11-04 23:41:34 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
      PHYSICAL RECORD: n_fields 2; compact format; info bits 0
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafbca; asc     a   ;;
      2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
      PHYSICAL RECORD: n_fields 3; compact format; info bits 0
       0: len 8; hex 8000000000000000; asc         ;;
       1: len 8; hex 800000003bb8f279; asc     ;  y;;
       2: len 4; hex 00089507; asc     ;;
      2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
      2021-11-04 23:41:34 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
      211104 23:41:34 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.12-MariaDB-log
      key_buffer_size=67108864
      read_buffer_size=262144
      max_used_connections=7
      max_threads=258
      thread_count=9
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x4002c0456798
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x40031d880ed8 thread_stack 0x40000
      /rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xaaaaea8cdeec]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): (null)
      Connection ID (thread ID): 0
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
       
      We think the query pointer is invalid, but we will try to print it anyway. 
      Query: 
       
      Writing a core file...
      Working directory at /rdsdbdata/db
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            unlimited            unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             unlimited            unlimited            processes 
      Max open files            65535                65535                files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       63078                63078                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: /rdsdbdata/tmp/core-%e-%p
       
      2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: starting recovery
      tables to flush: 1 0
       (0.0 seconds); 
      2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: recovery done
      2021-11-04 23:41:56 0 [Note] InnoDB: Uses event mutexes
      2021-11-04 23:41:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2021-11-04 23:41:56 0 [Note] InnoDB: Number of pools: 1
      2021-11-04 23:41:56 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
      2021-11-04 23:41:56 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)
      2021-11-04 23:41:56 0 [Note] InnoDB: Using Linux native AIO
      2021-11-04 23:41:56 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728
      2021-11-04 23:41:56 0 [Note] InnoDB: Completed initialization of buffer pool
      2021-11-04 23:41:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546169795863
      2021-11-04 23:41:57 0 [Note] InnoDB: Starting final batch to recover 27351 pages from redo log.
      2021-11-04 23:42:11 0 [Note] InnoDB: To recover: 11774 pages from log
      2021-11-04 23:42:37 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108
      2021-11-04 23:42:37 0 [Note] InnoDB: 128 rollback segments are active.
      2021-11-04 23:42:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2021-11-04 23:42:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2021-11-04 23:42:37 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2021-11-04 23:42:37 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
      2021-11-04 23:42:37 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169795973; transaction id 1115414123
      2021-11-04 23:42:37 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
      211104 23:42:37 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.
      211104 23:42:37 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.
      2021-11-04 23:42:37 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
      2021-11-04 23:42:37 0 [Note] Starting crash recovery...
      2021-11-04 23:42:37 0 [Note] Crash recovery finished.
      2021-11-04 23:42:37 0 [Note] Server socket created on IP: '::'.
      2021-11-04 23:42:37 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.
      2021-11-04 23:42:37 2 [Note] Event Scheduler: scheduler thread started with id 2
      2021-11-04 23:42:37 0 [Note] Reading of all Master_info entries succeeded
      2021-11-04 23:42:37 0 [Note] Added new Master_info '' to hash table
      2021-11-04 23:42:37 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
      Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/
      2021-11-04 23:42:40 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
      PHYSICAL RECORD: n_fields 2; compact format; info bits 0
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafbca; asc     a   ;;
      2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
      PHYSICAL RECORD: n_fields 3; compact format; info bits 0
       0: len 8; hex 8000000000000000; asc         ;;
       1: len 8; hex 800000003bb8f279; asc     ;  y;;
       2: len 4; hex 00089507; asc     ;;
      2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
      2021-11-04 23:42:40 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
      211104 23:42:40 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.5.12-MariaDB-log
      key_buffer_size=67108864
      read_buffer_size=262144
      max_used_connections=0
      max_threads=258
      thread_count=2
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x40003920e0d8
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x40031ebfded8 thread_stack 0x40000
      mysys/stacktrace.c:213(my_print_stacktrace)[0xaaaacb5e7eec]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): (null)
      Connection ID (thread ID): 0
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
      information that should help you find out what is causing the crash.
       
      We think the query pointer is invalid, but we will try to print it anyway. 
      Query: 
       
      Writing a core file...
      Working directory at /rdsdbdata/db
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            unlimited            unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             unlimited            unlimited            processes 
      Max open files            65535                65535                files     
      Max locked memory         65536                65536                bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       63078                63078                signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: /rdsdbdata/tmp/core-%e-%p
       
      2021-11-04 23:43:15 0 [Note] InnoDB: Uses event mutexes
      2021-11-04 23:43:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2021-11-04 23:43:15 0 [Note] InnoDB: Number of pools: 1
      2021-11-04 23:43:15 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
      2021-11-04 23:43:15 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)
      2021-11-04 23:43:15 0 [Note] InnoDB: Using Linux native AIO
      2021-11-04 23:43:15 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728
      2021-11-04 23:43:15 0 [Note] InnoDB: Completed initialization of buffer pool
      2021-11-04 23:43:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546111243034
      2021-11-04 23:43:16 0 [Note] InnoDB: Starting final batch to recover 27370 pages from redo log.
      2021-11-04 23:43:30 0 [Note] InnoDB: To recover: 11858 pages from log
      2021-11-04 23:43:40 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108
      2021-11-04 23:43:40 0 [Note] InnoDB: 128 rollback segments are active.
      2021-11-04 23:43:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2021-11-04 23:43:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2021-11-04 23:43:40 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2021-11-04 23:43:40 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
      2021-11-04 23:43:40 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169854425; transaction id 1115414123
      2021-11-04 23:43:40 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
      211104 23:43:40 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.
      211104 23:43:40 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.
      2021-11-04 23:43:40 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
      2021-11-04 23:43:40 0 [Note] Starting crash recovery...
      2021-11-04 23:43:40 0 [Note] Crash recovery finished.
      2021-11-04 23:43:40 0 [Note] Server socket created on IP: '::'.
      2021-11-04 23:43:40 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.
      2021-11-04 23:43:40 2 [Note] Event Scheduler: scheduler thread started with id 2
      2021-11-04 23:43:40 0 [Note] Reading of all Master_info entries succeeded
      2021-11-04 23:43:40 0 [Note] Added new Master_info '' to hash table
      2021-11-04 23:43:40 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
      Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/
      2021-11-04 23:43:43 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
      PHYSICAL RECORD: n_fields 2; compact format; info bits 0
       0: len 8; hex 8000000000000005; asc         ;;
       1: len 8; hex 8000000861cafbca; asc     a   ;;
      2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
      PHYSICAL RECORD: n_fields 3; compact format; info bits 0
       0: len 8; hex 8000000000000000; asc         ;;
       1: len 8; hex 800000003bb8f279; asc     ;  y;;
       2: len 4; hex 00089507; asc     ;;
      2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
      2021-11-04 23:43:43 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
      211104 23:43:43 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      

      AWS kept trying to restart the database without success, resulting in a reboot loop.

      Attachments

        1. 211119_mm_cds_crash_error_log.txt
          24 kB
          Heiki Laaniste
        2. 211202_mm-cds-2_reboot_loop.txt
          77 kB
          Heiki Laaniste
        3. bugreport_deleter_proc.sql
          4 kB
          Heiki Laaniste

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              heikilaaniste Heiki Laaniste
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.