[MDEV-26977] mariadb 10.5.12 reboot loop in AWS | [ERROR] InnoDB: Insert buffer insert fails | Corruption of an index tree Created: 2021-11-05  Updated: 2022-04-05  Resolved: 2022-04-05

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.12
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Heiki Laaniste Assignee: Marko Mäkelä
Resolution: Duplicate Votes: 2
Labels: None

Attachments: Text File 211119_mm_cds_crash_error_log.txt     Text File 211202_mm-cds-2_reboot_loop.txt     File bugreport_deleter_proc.sql    
Issue Links:
Duplicate
duplicates MDEV-26917 InnoDB: Clustered record for sec rec ... Closed
Relates
relates to MDEV-25283 Assertion `index->table->is_temporary... Closed
relates to MDEV-26808 InnoDB: Assertion failure in file /ho... Closed
relates to MDEV-27734 Set innodb_change_buffering=none by d... Closed
relates to MDEV-28148 InnoDB slave crash / signal 6 Closed
relates to MDEV-22986 [ERROR] InnoDB: Record in index was n... Closed
relates to MDEV-25796 Failing assertion: !cursor->index->is... Open
relates to MDEV-27765 MariaDB stopped to work randomly - mi... Closed

 Description   

I was running a SQL proc which deleted records.

2021-11-04 23:01:04 0 [ERROR] InnoDB: Insert buffer insert fails; page free 29, dtuple size 175
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373432303239222c2253455249414c5f4e554d424552223a22373432303239222c22455043223a22333033393544464138324639353634303030304235323844222c224e554d5f4c494e45223a352c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221742029","SERIAL_NUMBER":"742029","EPC":"30395DFA82F95640000B528D","NUM_LINE":5,"ITEM_TRANSACTION_MODE":"SALE"};;
 
 1: len 8; hex 800000001348b829; asc      H );;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881], size 16384, bitmap bits 0
 
2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:01:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark
 
InnoDB: tuple DATA TUPLE: 2 fields;
 
 0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373431383636222c2253455249414c5f4e554d424552223a22373431383636222c22455043223a22333033393544464138324639353634303030304235314541222c224e554d5f4c494e45223a312c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221741866","SERIAL_NUMBER":"741866","EPC":"30395DFA82F95640000B51EA","NUM_LINE":1,"ITEM_TRANSACTION_MODE":"SALE"};;
 
 1: len 8; hex 800000001052dd2a; asc      R *;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
 0: len 8; hex 5246494454616730; asc RFIDTag0;;
 
 1: len 8; hex 800000000aefe10b; asc         ;;
 
2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881] (721 records, index id 32581).
 
2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafc58; asc     a  X;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafcd6; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafce2; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafd6a; asc     a  j;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0018; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0022; asc     a  ";;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0025; asc     a  %;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0039; asc     a  9;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0070; asc     a  p;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0433; asc     a  3;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0438; asc     a  8;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0468; asc     a  h;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb04ba; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0563; asc     a  c;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0629; asc     a  );;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb062d; asc     a  -;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0697; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb06c8; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0701; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb07dd; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0856; asc     a  V;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb08f8; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0936; asc     a  6;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0a03; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0a27; asc     a  ';;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0a2c; asc     a  ,;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0a32; asc     a  2;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21
 
InnoDB: Cannot insert index record DATA TUPLE: 2 fields;
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cb0ae9; asc     a   ;;
 
 
 
InnoDB: The table where this index record belongs
 
InnoDB: is now probably corrupt. Please run CHECK TABLE on
 
InnoDB: that table.
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0
 
2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
2021-11-04 23:41:34 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
 
PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafbca; asc     a   ;;
 
2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
 
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 
 0: len 8; hex 8000000000000000; asc         ;;
 
 1: len 8; hex 800000003bb8f279; asc     ;  y;;
 
 2: len 4; hex 00089507; asc     ;;
 
2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
 
2021-11-04 23:41:34 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
 
211104 23:41:34 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.5.12-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=262144
 
max_used_connections=7
 
max_threads=258
 
thread_count=9
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x4002c0456798
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x40031d880ed8 thread_stack 0x40000
 
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xaaaaea8cdeec]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): (null)
 
Connection ID (thread ID): 0
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: 
 
 
Writing a core file...
 
Working directory at /rdsdbdata/db
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            unlimited            unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            65535                65535                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63078                63078                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
 
 
2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: starting recovery
 
tables to flush: 1 0
 
 (0.0 seconds); 
2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: recovery done
 
2021-11-04 23:41:56 0 [Note] InnoDB: Uses event mutexes
 
2021-11-04 23:41:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-11-04 23:41:56 0 [Note] InnoDB: Number of pools: 1
 
2021-11-04 23:41:56 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
 
2021-11-04 23:41:56 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)
 
2021-11-04 23:41:56 0 [Note] InnoDB: Using Linux native AIO
 
2021-11-04 23:41:56 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728
 
2021-11-04 23:41:56 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-11-04 23:41:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546169795863
 
2021-11-04 23:41:57 0 [Note] InnoDB: Starting final batch to recover 27351 pages from redo log.
 
2021-11-04 23:42:11 0 [Note] InnoDB: To recover: 11774 pages from log
 
2021-11-04 23:42:37 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108
 
2021-11-04 23:42:37 0 [Note] InnoDB: 128 rollback segments are active.
 
2021-11-04 23:42:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2021-11-04 23:42:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2021-11-04 23:42:37 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2021-11-04 23:42:37 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
 
2021-11-04 23:42:37 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169795973; transaction id 1115414123
 
2021-11-04 23:42:37 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
 
211104 23:42:37 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.
 
211104 23:42:37 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.
 
2021-11-04 23:42:37 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
 
2021-11-04 23:42:37 0 [Note] Starting crash recovery...
 
2021-11-04 23:42:37 0 [Note] Crash recovery finished.
 
2021-11-04 23:42:37 0 [Note] Server socket created on IP: '::'.
 
2021-11-04 23:42:37 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.
 
2021-11-04 23:42:37 2 [Note] Event Scheduler: scheduler thread started with id 2
 
2021-11-04 23:42:37 0 [Note] Reading of all Master_info entries succeeded
 
2021-11-04 23:42:37 0 [Note] Added new Master_info '' to hash table
 
2021-11-04 23:42:37 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
 
Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/
 
2021-11-04 23:42:40 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
 
PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafbca; asc     a   ;;
 
2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
 
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 
 0: len 8; hex 8000000000000000; asc         ;;
 
 1: len 8; hex 800000003bb8f279; asc     ;  y;;
 
 2: len 4; hex 00089507; asc     ;;
 
2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
 
2021-11-04 23:42:40 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
 
211104 23:42:40 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.5.12-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=262144
 
max_used_connections=0
 
max_threads=258
 
thread_count=2
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x40003920e0d8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x40031ebfded8 thread_stack 0x40000
 
mysys/stacktrace.c:213(my_print_stacktrace)[0xaaaacb5e7eec]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): (null)
 
Connection ID (thread ID): 0
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
 
 
We think the query pointer is invalid, but we will try to print it anyway. 
Query: 
 
 
Writing a core file...
 
Working directory at /rdsdbdata/db
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            unlimited            unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            65535                65535                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63078                63078                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
 
 
2021-11-04 23:43:15 0 [Note] InnoDB: Uses event mutexes
 
2021-11-04 23:43:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-11-04 23:43:15 0 [Note] InnoDB: Number of pools: 1
 
2021-11-04 23:43:15 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
 
2021-11-04 23:43:15 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)
 
2021-11-04 23:43:15 0 [Note] InnoDB: Using Linux native AIO
 
2021-11-04 23:43:15 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728
 
2021-11-04 23:43:15 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-11-04 23:43:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546111243034
 
2021-11-04 23:43:16 0 [Note] InnoDB: Starting final batch to recover 27370 pages from redo log.
 
2021-11-04 23:43:30 0 [Note] InnoDB: To recover: 11858 pages from log
 
2021-11-04 23:43:40 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108
 
2021-11-04 23:43:40 0 [Note] InnoDB: 128 rollback segments are active.
 
2021-11-04 23:43:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2021-11-04 23:43:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2021-11-04 23:43:40 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2021-11-04 23:43:40 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
 
2021-11-04 23:43:40 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169854425; transaction id 1115414123
 
2021-11-04 23:43:40 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
 
211104 23:43:40 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.
 
211104 23:43:40 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.
 
2021-11-04 23:43:40 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
 
2021-11-04 23:43:40 0 [Note] Starting crash recovery...
 
2021-11-04 23:43:40 0 [Note] Crash recovery finished.
 
2021-11-04 23:43:40 0 [Note] Server socket created on IP: '::'.
 
2021-11-04 23:43:40 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.
 
2021-11-04 23:43:40 2 [Note] Event Scheduler: scheduler thread started with id 2
 
2021-11-04 23:43:40 0 [Note] Reading of all Master_info entries succeeded
 
2021-11-04 23:43:40 0 [Note] Added new Master_info '' to hash table
 
2021-11-04 23:43:40 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
 
Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/
 
2021-11-04 23:43:43 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752
 
PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
 0: len 8; hex 8000000000000005; asc         ;;
 
 1: len 8; hex 8000000861cafbca; asc     a   ;;
 
2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112
 
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 
 0: len 8; hex 8000000000000000; asc         ;;
 
 1: len 8; hex 800000003bb8f279; asc     ;  y;;
 
 2: len 4; hex 00089507; asc     ;;
 
2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112
 
2021-11-04 23:43:43 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
 
211104 23:43:43 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs

AWS kept trying to restart the database without success, resulting in a reboot loop.

 Comments   
Comment by Heiki Laaniste [ 2021-11-05 ]

Attached the script of the stored procedure I was using to delete records: "bugreport_deleter_proc.sql".
I ran the script successfully before. Then I changed it a bit, imported data again, ran it again.
I can't provide the actual data, because it has sensitive information. Also the data dumps are large.

The dump size of the "master" schema is 14GB.
The dump size of the "store" schema is 2.8GB.

table1 number of records before deletion: 1203605 After deletion: 72479
table2: number of records before deletion 88897097 After deletion: 85296051
table3: number of records before deletion 10173872 After deletion: 9811062
table4: number of records before deletion 1976500 After deletion: 110654
table5: number of records before deletion 14026 After deletion: 1100
table6: number of records before deletion 3991958 After deletion: 224332

Comment by Roel Van de Paar [ 2021-11-05 ]

heikilaaniste Table definitions and data (obfuscated) would be appreciated if possible.

Comment by Marko Mäkelä [ 2021-11-05 ]

heikilaaniste, can you repeat this by loading the dump into a newly initialized MariaDB Server 10.5.12, and running the script?

Comment by Roel Van de Paar [ 2021-11-05 ]

Note: It is possible the issue is related to AWS.

Comment by Marko Mäkelä [ 2021-11-05 ]

heikilaaniste, to work around this bug, you can execute the following statement (and also add it to your start-up configuration) before loading any data:

SET GLOBAL innodb_change_buffering=none;

There are some other bug reports about change buffer corruption, but nobody (not even us internally) has been able to reproduce it. That is why I would be very interested in anything that could reproduce such problems.

Comment by Marko Mäkelä [ 2021-11-05 ]

MDEV-26808 may share a common root cause with this.

Note: If you use indexed virtual columns (if SELECT * FROM INFORMATION_SCHEMA.INNODB_SYS_VIRTUAL returns a non-empty result and some of those columns are indexed), then there are many open bugs that may explain such corruption.

Comment by Marko Mäkelä [ 2021-11-11 ]

heikilaaniste, can you reproduce the corruption if you set the following:

SET GLOBAL innodb_change_buffering=inserts;

With which values of that parameter is the corruption reproducible?

Comment by Heiki Laaniste [ 2021-11-12 ]

I tested this issue 3 times this week. First with a smaller dataset, then 2 more times with the original dataset that produced this error. I was not able to reproduce the error.
The "innodb_change_buffering" param value was:
SET GLOBAL innodb_change_buffering=all;

We've lately had a few issues with corrupt indexes causing database restarts.
Yesterday there was one too:

2021-11-11 21:10:58 31621 [ERROR] [FATAL] InnoDB: Rec offset 8026, cur1 offset 15170, cur2 offset 7906
 
211111 21:10:58 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.5.12-MariaDB-log
 
key_buffer_size=67108864
 
read_buffer_size=262144
 
max_used_connections=25
 
max_threads=258
 
thread_count=19
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x40188de085d8
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x400041082060 thread_stack 0x40000
 
/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xaaaadfa99eec]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x4018ee6dce70): INSERT INTO prcinvoice SET `idDat` = '1138099', `orgper_idDat` = '12965', `currency_id` = '1', `warehouse_id` = '2', `type_id` = '2', `state_id` = '3', `madeFr_idDat` = '0', `madeFr_idDats` = '', `order_invoiced` = '0', `regno` = '13096635', `no` = '21D0050414', `date` = '2021-11-09', `time` = '10:42:00', `deadline` = '30', `round` = '0.00', `currency_rate` = '1.000000000000', `hideprices` = '0', `confirmed` = '1', `notes` = '', `ref_no` = '', `baseTo_idDat` = '0', `orgper_idDat_contact` = '0', `address_id` = '8810', `delivery_address_id` = '0', `orgper_idDat_prod` = '0', `orgper_idDat_contact_prod` = '0', `address_id_prod` = '0', `orgper_idDat_author` = '136406', `project_idDat` = '0', `zerovat_id` = '0', `opertype_id` = '0', `transtype_id` = '0', `purvtype_id` = '5', `currency_id_transportcost` = '1', `currency_rate_transportcost` = '1.00000', `purveyance` = '', `triangularoperation` = '1', `transportcost` = '0.00', `transportcostdiv` = 'amount', `generaldiscount` = '0', `sum` = '664.80', `weight` = '0', `penalty` = '0', `rowtitles` = 'obfuscated', `paid` = '0', `printed_net` = '0.00', `printed_vat` = '0.00', `cif_addedvalue` = '0.00', `reason_id` = NULL, `jdoc` = NULL, state = 1, added = '1636665058', addedby = 'obfuscated'
 
 
 
Connection ID (thread ID): 31621
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=off,condition_pushdown_from_having=on,not_null_range_scan=off
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /rdsdbdata/db
 
Resource Limits:
 
Limit Soft Limit Hard Limit Units
 
Max cpu time unlimited unlimited seconds
 
Max file size unlimited unlimited bytes
 
Max data size unlimited unlimited bytes
 
Max stack size unlimited unlimited bytes
 
Max core file size 0 unlimited bytes
 
Max resident set unlimited unlimited bytes
 
Max processes unlimited unlimited processes
 
Max open files 65535 65535 files
 
Max locked memory 65536 65536 bytes
 
Max address space unlimited unlimited bytes
 
Max file locks unlimited unlimited locks
 
Max pending signals 510241 510241 signals
 
Max msgqueue size 819200 819200 bytes
 
Max nice priority 0 0
 
Max realtime priority 0 0
 
Max realtime timeout unlimited unlimited us
 
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
 
 
2021-11-11 21:11:09 0 [Note] InnoDB: Uses event mutexes
 
2021-11-11 21:11:09 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-11-11 21:11:09 0 [Note] InnoDB: Number of pools: 1
 
2021-11-11 21:11:09 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
 
2021-11-11 21:11:09 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)
 
2021-11-11 21:11:09 0 [Note] InnoDB: Using Linux native AIO
 
2021-11-11 21:11:09 0 [Note] InnoDB: Initializing buffer pool, total size = 99857989632, chunk size = 134217728
 
2021-11-11 21:11:10 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-11-11 21:11:10 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1127868803131,1127868803131
 
2021-11-11 21:11:12 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
 
2021-11-11 21:11:12 0 [Note] InnoDB: Trx id counter is 1818877404
 
2021-11-11 21:11:12 0 [Note] InnoDB: Starting final batch to recover 42371 pages from redo log.
 
2021-11-11 21:11:25 0 [Note] InnoDB: To recover: 5351 pages from log
 
2021-11-11 21:11:27 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.218775', position 418709
 
2021-11-11 21:11:27 0 [Note] InnoDB: 128 rollback segments are active.
 
2021-11-11 21:11:28 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
 
2021-11-11 21:11:28 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
 
2021-11-11 21:11:28 0 [Note] InnoDB: Creating shared tablespace for temporary tables
 
2021-11-11 21:11:28 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
 
2021-11-11 21:11:28 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.
 
2021-11-11 21:11:28 0 [Note] InnoDB: 10.5.12 started; log sequence number 1127943500876; transaction id 1818877405
 
2021-11-11 21:11:28 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool
 
211111 21:11:28 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.
 
211111 21:11:28 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.
 
2021-11-11 21:11:28 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog
 
2021-11-11 21:11:28 0 [Note] Starting crash recovery...
 
2021-11-11 21:11:28 0 [Note] Crash recovery finished.
 
2021-11-11 21:11:28 0 [Note] InnoDB: Rolled back recovered transaction 1818877403
 
2021-11-11 21:11:28 0 [Note] InnoDB: Rollback of non-prepared transactions completed
 
2021-11-11 21:11:28 0 [Note] Server socket created on IP: '::'.
 
2021-11-11 21:11:28 2 [Note] Event Scheduler: scheduler thread started with id 2
 
2021-11-11 21:11:28 0 [Note] Reading of all Master_info entries succeeded
 
2021-11-11 21:11:28 0 [Note] Added new Master_info '' to hash table
 
2021-11-11 21:11:28 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.
 
Version: '10.5.12-MariaDB-log' socket: '/tmp/mysql.sock' port: 13336 managed by https://aws.amazon.com/rds/
 
2021-11-11 21:11:28 7 [Note] Start binlog_dump to slave_server(101), pos(mysql-bin-changelog.218775, 418709), using_gtid(0), gtid('')
 
2021-11-11 21:11:34 34 [Warning] Aborted connection 34 to db: 'unconnected' user: 'rdsadmin' host: 'localhost' (Got an error reading communication packets)
 
2021-11-11 21:29:46 0 [Note] InnoDB: Buffer pool(s) load completed at 211111 21:29:46
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record overlaps another: 15139+0
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record offset out of bounds: 15163+10481
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Records in wrong order
 
 
 
InnoDB: previous record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 30; hex 800000008000000980000000000000008018e6eb00179800198000000080; asc ; (total 10470 bytes, external) len 20; hex 76616c75655f737472696e6700000005002100fc; asc value_string ! ;;
 
1: len 4; hex 0000000a; asc ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 30; hex 80000000800000098000000000000000801928640017a000198000000080; asc (d ; (total 11238 bytes, external) len 20; hex 312d292519120c080004b013277f000000000000; asc 1-)% ' ;;
 
1: len 4; hex 37250000; asc 7% ;;
 
 
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record offset out of bounds: 15188+11249
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Records in wrong order
 
 
 
InnoDB: previous record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 30; hex 80000000800000098000000000000000801928640017a000198000000080; asc (d ; (total 11238 bytes, external) len 20; hex 312d292519120c080004b013277f000000000000; asc 1-)% ' ;;
 
1: len 4; hex 37250000; asc 7% ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 30; hex 800000008000000980000000000000008019287b0017a800198000000080; asc ({ ; (total 100 bytes);
 
1: len 4; hex 14323144; asc 21D;;
 
 
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record overlaps another: 15214+109
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record overlaps another: 15239+132
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Record offset out of bounds: 15263+4915
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Records in wrong order
 
 
 
InnoDB: previous record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 30; hex 800000008000000980000000000000008019290d0017b8c4b18000000080; asc ) ; (total 4904 bytes);
 
1: len 4; hex 00000077; asc w;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 13; hex 80000000800000098000000000; asc ;;
 
1: len 4; hex 00000080; asc ;;
 
 
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Summed data size 42610, returned by func 15855
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Apparent corruption in space 0 page 6320287 of index `no` of table `crmx_52560`.`prcinvoice`
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: In page 6320287 of index `no` of table `crmx_52560`.`prcinvoice`
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: In pages [page id: space=0, page number=6320287] and [page id: space=0, page number=9265568] of index `no` of table `crmx_52560`.`prcinvoice`
 
InnoDB: records in wrong order on adjacent pages
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 13; hex 80000000800000098000000000; asc ;;
 
1: len 4; hex 00000080; asc ;;
 
 
 
InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 10; hex 32314645303030303231; asc 21FE000021;;
 
1: len 4; hex 800ec486; asc ;;
 
 
 
2021-11-11 21:33:33 9135 [ERROR] InnoDB: Corruption of an index tree: table `crmx_52560`.`prcinvoice` index `no`, father ptr page no 8868355, child page no 6320287
 
PHYSICAL RECORD: n_fields 2; compact format; info bits 0
 
0: len 10; hex 32314430303436313933; asc 21D0046193;;
 
1: len 4; hex 80112f0e; asc / ;;
 
2021-11-11 21:33:33 9135 [Note] InnoDB: n_owned: 0; heap_no: 2; next rec: 146
 
PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 
0: len 9; hex 56462d313936303635; asc VF-196065;;
 
1: len 4; hex 800bed0c; asc ;;
 
2: len 4; hex 00875203; asc R ;;
 
2021-11-11 21:33:33 9135 [Note] InnoDB: n_owned: 0; heap_no: 361; next rec: 112
 
2021-11-11 21:33:33 9135 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.
 
211111 21:33:33 [ERROR] mysqld got signal 6 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs

The DB instance was not in a restart loop, luckily. So the solution was:

use crmx_52560;
alter table prcinvoice drop key no;
alter table prcinvoice add key (no);

We have a virtual column (STORED GENERATED) with index in one table, in every client schema.
We have many client schemas in each production DB instance. Every schema has mostly the same structure.
So far that table has not been part of any crash cause though.

Feels like indexes becoming corrupt is the main issue for us (the company I work for) ATM.
"[ERROR] InnoDB: Insert buffer insert fails" could also be caused by corrupt index, right?
Any advice how to avoid indexes becoming corrupt?

I can try to reproduce the error with this param next week:
SET GLOBAL innodb_change_buffering=inserts;

Comment by Heiki Laaniste [ 2021-11-16 ]

Did not reproduce with
SET GLOBAL innodb_change_buffering=inserts;

Comment by Marko Mäkelä [ 2021-11-16 ]

heikilaaniste, thank you. I am not aware of frequent bugs with STORED GENERATED columns. From the InnoDB point of view, they should behave just like ‘normal’ columns.

Could you try 2 more settings:

  • innodb_change_buffering=changes (buffering of insert and delete-mark operations)
  • innodb_change_buffering=all (the original, just to double-check that the problem did not "magically" go away)

Could you provide a reproducible test case in some way? The data set can be uploaded outside this system, but it would be accessible to some employees.

I believe that the issue would still reproduce if you obfuscate (scramble) all non-key columns while keeping the same length.

Because the use of the change buffer depends on the performance and the access patterns of the buffer pool, we might lose reproducibility if you drop any columns or indexes, even if you compensated by reducing the innodb_buffer_pool_size.

If you obfuscate any key columns, I think that it should be safe to add something to each key value, in such a way that ORDER BY the key does not change. You can add a different value to each row, as long as the ordering of the obfuscated keys is unaffected. This should preserve the page access patterns and not affect the reproducibility.

Comment by Heiki Laaniste [ 2021-11-19 ]

Tried again both of these variants
innodb_change_buffering=changes (buffering of insert and delete-mark operations)
innodb_change_buffering=all (the original, just to double-check that the problem did not "magically" go away)

Does not reproduce.

Meanwhile in another RDS another crash. Different error though.
211119_mm_cds_crash_error_log.txt

Comment by Marko Mäkelä [ 2021-11-24 ]

heikilaaniste, thank you. So, this bug is very sporadic. That is the nature of the change buffer: it may only be used when the secondary index leaf page does not happen to exist in the buffer pool.

It is possible that your another crash is caused by something in the change buffer. Corruption could remain dormant for a long time, and setting innodb_change_buffering=none will not trigger a change buffer merge. A slow shutdown (innodb_fast_shutdown=0) or running CHECK TABLE…QUICK on every table would. Unfortunately, InnoDB is not too forgiving when it comes to corruption. You may fix corruption in secondary indexes by executing something like the following:

ALTER TABLE tablename DROP INDEX i1, DROP INDEX i2;
 
ALTER TABLE tablename ADD INDEX i1(a,b), ADD INDEX i2(c);

I am afraid that we can only follow a statistical approach here. Maybe you could try each of the following settings for a longer period of time (some weeks) and report how it goes:

  • innodb_change_buffering=none
  • innodb_change_buffering=inserts
  • innodb_change_buffering=changes

Note: If you are using ROW_FORMAT=COMPRESSED tables, MDEV-27059 is a bug specific to 10.5 and 10.6.

Comment by MG [ 2021-11-24 ] 

# gdb -p $(pidof mariadbd)
 
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
 
Copyright (C) 2018 Free Software Foundation, Inc.
 
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 
This is free software: you are free to change and redistribute it.
 
There is NO WARRANTY, to the extent permitted by law.
 
Type "show copying" and "show warranty" for details.
 
This GDB was configured as "x86_64-redhat-linux-gnu".
 
Type "show configuration" for configuration details.
 
For bug reporting instructions, please see:
 
<http://www.gnu.org/software/gdb/bugs/>.
 
Find the GDB manual and other documentation resources online at:
 
    <http://www.gnu.org/software/gdb/documentation/>.
 
 
 
For help, type "help".
 
Type "apropos word" to search for commands related to "word".
 
Attaching to process 100206
 
[New LWP 100207]
 
[New LWP 100208]
 
[New LWP 100209]
 
[New LWP 100210]
 
Missing separate debuginfo for target:/lib64/libpmem.so.1
 
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/48/759ec97ac1f912322698d63fe88ce26d17814e.debug
 
[Thread debugging using libthread_db enabled]
 
Using host libthread_db library "/lib64/libthread_db.so.1".
 
0x000055f727868be3 in std::_Rb_tree<page_id_t const, std::pair<page_id_t const, mlog_init_t::init>, std::_Select1st<std::pair<page_id_t const, mlog_init_t::init> >, std::less<page_id_t const>, ut_allocator<std::pair<page_id_t const, mlog_init_t::init>, true> >::_M_lower_bound (this=<optimized out>, __k=<synthetic pointer>...,
 
    __y=0x55f72b7926a8, __x=0x55f7322b4c48) at /usr/include/c++/8/bits/stl_tree.h:1899
 
1899	/usr/include/c++/8/bits/stl_tree.h: No such file or directory.
 
(gdb) continue
 
Continuing.
 
[New Thread 0x7f286a34e700 (LWP 100374)]
 
[New Thread 0x7f2869b4d700 (LWP 100375)]
 
[New Thread 0x7f286934c700 (LWP 100376)]
 
[New Thread 0x7f2868b4b700 (LWP 100377)]
 
[New Thread 0x7f286834a700 (LWP 100378)]
 
[New Thread 0x7f2867b49700 (LWP 100379)]
 
[New Thread 0x7f2867348700 (LWP 100380)]
 
[New Thread 0x7f2866b47700 (LWP 100381)]
 
[New Thread 0x7f2866346700 (LWP 100382)]
 
[New Thread 0x7f2ac0758700 (LWP 100386)]
 
 
 
Thread 13 "mariadbd" received signal SIGABRT, Aborted.
 
[Switching to Thread 0x7f2866b47700 (LWP 100381)]
 
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
50	  return ret;
 
(gdb) bt
 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
 
#1  0x00007f2abdab7db5 in __GI_abort () at abort.c:79
 
#2  0x000055f727215961 in ib::fatal::~fatal (this=<optimized out>, __in_chrg=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/innobase/ut/ut0ut.cc:519
 
#3  0x000055f7271fd9d9 in page_copy_rec_list_end_no_locks (new_block=new_block@entry=0x7f2a4ceb9e00, block=block@entry=0x7f2a4ceba580, rec=0x7f2a54794063 "infimum",
 
    index=index@entry=0x55f736ff2080, mtr=mtr@entry=0x7f2866b44370) at /usr/src/debug/MariaDB-/src_0/storage/innobase/include/ut0byte.h:92
 
#4  0x000055f7279362f8 in btr_page_reorganize_low (index=0x55f736ff2080, mtr=0x7f2866b44370, cursor=<optimized out>, cursor=<optimized out>)
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/include/page0page.h:299
 
#5  0x000055f727936ea7 in btr_page_reorganize (cursor=0x7f2866b433c0, index=<optimized out>, mtr=<optimized out>)
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/btr/btr0btr.cc:1535
 
#6  0x000055f72784a062 in ibuf_insert_to_index_page_low(dtuple_t const*, buf_block_t*, dict_index_t*, unsigned short**, mem_block_info_t*, mtr_t*, page_cur_t*) ()
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/ibuf/ibuf0ibuf.cc:3632
 
#7  0x000055f72784e73c in ibuf_merge_or_delete_for_page(buf_block_t*, page_id_t, unsigned long) () at /usr/src/debug/MariaDB-/src_0/storage/innobase/ibuf/ibuf0ibuf.cc:3834
 
#8  0x000055f727966335 in buf_page_get_low (page_id=..., zip_size=<optimized out>, rw_latch=<optimized out>, guess=<optimized out>, mode=<optimized out>,
 
    mtr=<optimized out>, err=<optimized out>, allow_ibuf_merge=<optimized out>) at /usr/src/debug/MariaDB-/src_0/storage/innobase/buf/buf0buf.cc:2959
 
#9  0x000055f727947729 in btr_cur_search_to_nth_level_func(dict_index_t*, unsigned long, dtuple_t const*, page_cur_mode_t, unsigned long, btr_cur_t*, srw_lock_impl<true>*, mtr_t*, unsigned long) () at /usr/src/debug/MariaDB-/src_0/storage/innobase/include/dict0mem.h:1176
 
#10 0x000055f7278df9c3 in btr_pcur_open_low (index=0x55f736ff3b80, tuple=<optimized out>, latch_mode=<optimized out>, cursor=0x7f2866b459e0, mtr=<optimized out>,
 
    autoinc=0, mode=PAGE_CUR_LE, level=0) at /usr/src/debug/MariaDB-/src_0/storage/innobase/include/btr0pcur.ic:439
 
#11 0x000055f7278dfa65 in row_search_index_entry (index=index@entry=0x55f736ff3b80, entry=entry@entry=0x55f72b658480, mode=mode@entry=2, pcur=pcur@entry=0x7f2866b459e0,
 
    mtr=mtr@entry=0x7f2866b45c50) at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0row.cc:1303
 
#12 0x000055f7279fe20b in row_undo_mod_del_unmark_sec_and_undo_update (mode=2, thr=0x55f72976d028, index=0x55f736ff3b80, entry=<optimized out>)
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0umod.cc:785
 
#13 0x000055f7279ff6dd in row_undo_mod_upd_exist_sec (thr=0x55f72976d028, node=0x55f733933c00) at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0umod.cc:1176
 
#14 row_undo_mod (node=node@entry=0x55f733933c00, thr=thr@entry=0x55f72976d028) at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0umod.cc:1364
 
#15 0x000055f7278ebd2b in row_undo (thr=0x55f72976d028, node=0x55f733933c00) at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0undo.cc:413
 
#16 row_undo_step (thr=thr@entry=0x55f72976d028) at /usr/src/debug/MariaDB-/src_0/storage/innobase/row/row0undo.cc:460
 
#17 0x000055f7278a01f8 in que_thr_step (thr=0x55f72976d028) at /usr/src/debug/MariaDB-/src_0/storage/innobase/que/que0que.cc:651
 
#18 que_run_threads_low (thr=0x55f72976d028) at /usr/src/debug/MariaDB-/src_0/storage/innobase/que/que0que.cc:709
 
#19 que_run_threads (thr=0x55f72976d028) at /usr/src/debug/MariaDB-/src_0/storage/innobase/que/que0que.cc:729
 
#20 0x000055f727912dc0 in trx_rollback_active (trx=0x55f72b92b120) at /usr/src/debug/MariaDB-/src_0/storage/innobase/trx/trx0roll.cc:602
 
#21 0x000055f7279132e2 in trx_rollback_recovered (all=true) at /usr/src/debug/MariaDB-/src_0/storage/innobase/trx/trx0roll.cc:740
 
#22 0x000055f727213689 in trx_rollback_all_recovered () at /usr/src/debug/MariaDB-/src_0/storage/innobase/trx/trx0roll.cc:786
 
#23 0x000055f727a0d6ec in tpool::task_group::execute (this=0x55f728dd6480 <rollback_all_recovered_group>, t=0x55f728dd6460 <rollback_all_recovered_task>)
 
    at /usr/src/debug/MariaDB-/src_0/tpool/task_group.cc:55
 
#24 0x000055f727a0c5bf in tpool::thread_pool_generic::worker_main (this=0x55f72983c380, thread_var=0x55f72b63b600)
 
    at /usr/src/debug/MariaDB-/src_0/tpool/tpool_generic.cc:549
 
#25 0x00007f2abe29fba3 in std::execute_native_thread_routine (__p=0x55f73319b890) at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
 
#26 0x00007f2abe77e17a in start_thread (arg=<optimized out>) at pthread_create.c:479
 
#27 0x00007f2abdb92dc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
(gdb) frame 4
 
#4  0x000055f7279362f8 in btr_page_reorganize_low (index=0x55f736ff2080, mtr=0x7f2866b44370, cursor=<optimized out>, cursor=<optimized out>)
 
    at /usr/src/debug/MariaDB-/src_0/storage/innobase/include/page0page.h:299
 
299		return page_is_comp(page) ? PAGE_NEW_INFIMUM : PAGE_OLD_INFIMUM;
 
(gdb) print index.name
 
$1 = {m_name = 0x55f736ff2210 "IBUF_DUMMY"}
 
(gdb) print index.table.name
 
$2 = {m_name = 0x55f732a05668 "IBUF_DUMMY", static part_suffix = "#P#"}
 
(gdb)

Comment by Marko Mäkelä [ 2021-11-24 ]

mg, thank you. That looks like a reboot loop due to a corruption in the change buffer tree itself. You can break the loop by setting innodb_force_recovery=3.
I hope that one of you will be able to submit something that can reproduce such corruption.

Comment by Heiki Laaniste [ 2021-12-02 ]

innodb_force_recovery can't be changed on AWS RDS though.
Another case of index corruption today:
211202_mm-cds-2_reboot_loop.txt
Yesterday I did mariadb upgrade 10.5.12 -> 10.5.13 on it. Could that be related? It was about 18 hours before the index corruption and crash though.

Comment by Marko Mäkelä [ 2021-12-13 ]

I do not think that anything changed between 10.5.12 and 10.5.13 with regard to the change buffer. That the error occurred after an upgrade should be a coincidence.

We still cannot proceed on this, because no self-contained repeatable test case exists.

Comment by Marko Mäkelä [ 2022-01-24 ]

I don’t think that we should close this report just because we were unable to reproduce this ourselves so far.

Any test case to reproduce this bug is still welcome.

Comment by Marko Mäkelä [ 2022-01-31 ]

My current plan with regard to this is to make innodb_change_buffering=none the default setting (MDEV-27734) and to deprecate the parameter (MDEV-27735).
In some recent performance tests on 10.5.13 and 10.6.5 on an SSD, the change buffer was able to improve throughput by at most 5% (MDEV-11634).

Change buffering could also cause excessive growth of the InnoDB system tablespace (MDEV-21952).

Comment by Marko Mäkelä [ 2022-04-05 ]

MDEV-26917 was filed earlier, and that bug was confirmed to be caused by the InnoDB change buffer.

Comment by Marko Mäkelä [ 2022-04-05 ]

I think that this report duplicates MDEV-26917. If anyone is able to reproduce this bug starting from a logical SQL dump, I would be interested. We are unable to reproduce this type of corruption internally, and in MDEV-27734 the InnoDB change buffer was disabled by default.

Generated at Thu Feb 08 09:49:23 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.