[MDEV-26977] mariadb 10.5.12 reboot loop in AWS | [ERROR] InnoDB: Insert buffer insert fails | Corruption of an index tree - Jira

XML

Word

Printable

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Duplicate
Affects Version/s: 10.5.12
Fix Version/s: N/A
Component/s: Storage Engine - InnoDB
Labels:
None

Description

I was running a SQL proc which deleted records.

2021-11-04 23:01:04 0 [ERROR] InnoDB: Insert buffer insert fails; page free 29, dtuple size 175

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373432303239222c2253455249414c5f4e554d424552223a22373432303239222c22455043223a22333033393544464138324639353634303030304235323844222c224e554d5f4c494e45223a352c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221742029","SERIAL_NUMBER":"742029","EPC":"30395DFA82F95640000B528D","NUM_LINE":5,"ITEM_TRANSACTION_MODE":"SALE"};;

 1: len 8; hex 800000001348b829; asc      H );;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881], size 16384, bitmap bits 0

2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:01:04 0 [ERROR] InnoDB: Unable to find a record to delete-mark

InnoDB: tuple DATA TUPLE: 2 fields;

 0: len 160; hex 7b224754494e223a2233353833373837373936303932222c22534754494e223a223031333538333738373739363039323231373431383636222c2253455249414c5f4e554d424552223a22373431383636222c22455043223a22333033393544464138324639353634303030304235314541222c224e554d5f4c494e45223a312c224954454d5f5452414e53414354494f4e5f4d4f4445223a2253414c45227d; asc {"GTIN":"3583787796092","SGTIN":"01358378779609221741866","SERIAL_NUMBER":"741866","EPC":"30395DFA82F95640000B51EA","NUM_LINE":1,"ITEM_TRANSACTION_MODE":"SALE"};;

 1: len 8; hex 800000001052dd2a; asc      R *;;

InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0

 0: len 8; hex 5246494454616730; asc RFIDTag0;;

 1: len 8; hex 800000000aefe10b; asc         ;;

2021-11-04 23:01:04 0 [ERROR] InnoDB: page [page id: space=0, page number=6405881] (721 records, index id 32581).

2021-11-04 23:01:04 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafc58; asc     a  X;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafcd6; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafce2; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafd6a; asc     a  j;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0018; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0022; asc     a  ";;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0025; asc     a  %;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0039; asc     a  9;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0070; asc     a  p;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0433; asc     a  3;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0438; asc     a  8;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0468; asc     a  h;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb04ba; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0563; asc     a  c;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0629; asc     a  );;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb062d; asc     a  -;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0697; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb06c8; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0701; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb07dd; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0856; asc     a  V;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb08f8; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0936; asc     a  6;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0a03; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0a27; asc     a  ';;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0a2c; asc     a  ,;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0a32; asc     a  2;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:25 0 [ERROR] InnoDB: Insert buffer insert fails; page free 19, dtuple size 21

InnoDB: Cannot insert index record DATA TUPLE: 2 fields;

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cb0ae9; asc     a   ;;

InnoDB: The table where this index record belongs

InnoDB: is now probably corrupt. Please run CHECK TABLE on

InnoDB: that table.

2021-11-04 23:41:25 0 [ERROR] InnoDB: page [page id: space=0, page number=7242752], size 16384, bitmap bits 0

2021-11-04 23:41:25 0 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2021-11-04 23:41:34 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752

PHYSICAL RECORD: n_fields 2; compact format; info bits 0

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafbca; asc     a   ;;

2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112

PHYSICAL RECORD: n_fields 3; compact format; info bits 0

 0: len 8; hex 8000000000000000; asc         ;;

 1: len 8; hex 800000003bb8f279; asc     ;  y;;

 2: len 4; hex 00089507; asc     ;;

2021-11-04 23:41:34 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112

2021-11-04 23:41:34 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.

211104 23:41:34 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB-log

key_buffer_size=67108864

read_buffer_size=262144

max_used_connections=7

max_threads=258

thread_count=9

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x4002c0456798

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x40031d880ed8 thread_stack 0x40000

/rdsdbbin/mysql/bin/mysqld(my_print_stacktrace+0x2c)[0xaaaaea8cdeec]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): (null)

Connection ID (thread ID): 0

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway.

Query:

Writing a core file...

Working directory at /rdsdbdata/db

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            unlimited            unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             unlimited            unlimited            processes

Max open files            65535                65535                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       63078                63078                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: /rdsdbdata/tmp/core-%e-%p

2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: starting recovery

tables to flush: 1 0

 (0.0 seconds);

2021-11-04 23:41:56 0 [Note] mysqld: Aria engine: recovery done

2021-11-04 23:41:56 0 [Note] InnoDB: Uses event mutexes

2021-11-04 23:41:56 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2021-11-04 23:41:56 0 [Note] InnoDB: Number of pools: 1

2021-11-04 23:41:56 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions

2021-11-04 23:41:56 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)

2021-11-04 23:41:56 0 [Note] InnoDB: Using Linux native AIO

2021-11-04 23:41:56 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728

2021-11-04 23:41:56 0 [Note] InnoDB: Completed initialization of buffer pool

2021-11-04 23:41:56 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546169795863

2021-11-04 23:41:57 0 [Note] InnoDB: Starting final batch to recover 27351 pages from redo log.

2021-11-04 23:42:11 0 [Note] InnoDB: To recover: 11774 pages from log

2021-11-04 23:42:37 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108

2021-11-04 23:42:37 0 [Note] InnoDB: 128 rollback segments are active.

2021-11-04 23:42:37 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2021-11-04 23:42:37 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2021-11-04 23:42:37 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2021-11-04 23:42:37 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.

2021-11-04 23:42:37 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169795973; transaction id 1115414123

2021-11-04 23:42:37 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool

211104 23:42:37 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.

211104 23:42:37 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.

2021-11-04 23:42:37 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog

2021-11-04 23:42:37 0 [Note] Starting crash recovery...

2021-11-04 23:42:37 0 [Note] Crash recovery finished.

2021-11-04 23:42:37 0 [Note] Server socket created on IP: '::'.

2021-11-04 23:42:37 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.

2021-11-04 23:42:37 2 [Note] Event Scheduler: scheduler thread started with id 2

2021-11-04 23:42:37 0 [Note] Reading of all Master_info entries succeeded

2021-11-04 23:42:37 0 [Note] Added new Master_info '' to hash table

2021-11-04 23:42:37 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.

Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/

2021-11-04 23:42:40 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752

PHYSICAL RECORD: n_fields 2; compact format; info bits 0

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafbca; asc     a   ;;

2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112

PHYSICAL RECORD: n_fields 3; compact format; info bits 0

 0: len 8; hex 8000000000000000; asc         ;;

 1: len 8; hex 800000003bb8f279; asc     ;  y;;

 2: len 4; hex 00089507; asc     ;;

2021-11-04 23:42:40 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112

2021-11-04 23:42:40 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.

211104 23:42:40 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help

diagnose the problem, but since we have already crashed,

something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB-log

key_buffer_size=67108864

read_buffer_size=262144

max_used_connections=0

max_threads=258

thread_count=2

It is possible that mysqld could use up to

key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 666138 K  bytes of memory

Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x40003920e0d8

Attempting backtrace. You can use the following information to find out

where mysqld died. If you see no messages after this, something went

terribly wrong...

stack_bottom = 0x40031ebfded8 thread_stack 0x40000

mysys/stacktrace.c:213(my_print_stacktrace)[0xaaaacb5e7eec]

Trying to get some variables.

Some pointers may be invalid and cause the dump to abort.

Query (0x0): (null)

Connection ID (thread ID): 0

Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains

information that should help you find out what is causing the crash.

We think the query pointer is invalid, but we will try to print it anyway.

Query:

Writing a core file...

Working directory at /rdsdbdata/db

Resource Limits:

Limit                     Soft Limit           Hard Limit           Units

Max cpu time              unlimited            unlimited            seconds

Max file size             unlimited            unlimited            bytes

Max data size             unlimited            unlimited            bytes

Max stack size            unlimited            unlimited            bytes

Max core file size        0                    unlimited            bytes

Max resident set          unlimited            unlimited            bytes

Max processes             unlimited            unlimited            processes

Max open files            65535                65535                files

Max locked memory         65536                65536                bytes

Max address space         unlimited            unlimited            bytes

Max file locks            unlimited            unlimited            locks

Max pending signals       63078                63078                signals

Max msgqueue size         819200               819200               bytes

Max nice priority         0                    0

Max realtime priority     0                    0

Max realtime timeout      unlimited            unlimited            us

Core pattern: /rdsdbdata/tmp/core-%e-%p

2021-11-04 23:43:15 0 [Note] InnoDB: Uses event mutexes

2021-11-04 23:43:15 0 [Note] InnoDB: Compressed tables use zlib 1.2.11

2021-11-04 23:43:15 0 [Note] InnoDB: Number of pools: 1

2021-11-04 23:43:15 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions

2021-11-04 23:43:15 0 [Note] mysqld: O_TMPFILE is not supported on /rdsdbdata/tmp (disabling future attempts)

2021-11-04 23:43:15 0 [Note] InnoDB: Using Linux native AIO

2021-11-04 23:43:15 0 [Note] InnoDB: Initializing buffer pool, total size = 11811160064, chunk size = 134217728

2021-11-04 23:43:15 0 [Note] InnoDB: Completed initialization of buffer pool

2021-11-04 23:43:15 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=546111243034,546111243034

2021-11-04 23:43:16 0 [Note] InnoDB: Starting final batch to recover 27370 pages from redo log.

2021-11-04 23:43:30 0 [Note] InnoDB: To recover: 11858 pages from log

2021-11-04 23:43:40 0 [Note] InnoDB: Last binlog file '/rdsdbdata/log/binlog/mysql-bin-changelog.033323', position 25851108

2021-11-04 23:43:40 0 [Note] InnoDB: 128 rollback segments are active.

2021-11-04 23:43:40 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"

2021-11-04 23:43:40 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2021-11-04 23:43:40 0 [Note] InnoDB: Setting file '/rdsdbdata/db/innodb/ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...

2021-11-04 23:43:40 0 [Note] InnoDB: File '/rdsdbdata/db/innodb/ibtmp1' size is now 12 MB.

2021-11-04 23:43:40 0 [Note] InnoDB: 10.5.12 started; log sequence number 546169854425; transaction id 1115414123

2021-11-04 23:43:40 0 [Note] InnoDB: Loading buffer pool(s) from /rdsdbdata/db/innodb/ib_buffer_pool

211104 23:43:40 server_audit: MariaDB Audit Plugin version 1.4.13 STARTED.

211104 23:43:40 server_audit: logging started to the file /rdsdbdata/log/audit/server_audit.log.

2021-11-04 23:43:40 0 [Note] Recovering after a crash using /rdsdbdata/log/binlog/mysql-bin-changelog

2021-11-04 23:43:40 0 [Note] Starting crash recovery...

2021-11-04 23:43:40 0 [Note] Crash recovery finished.

2021-11-04 23:43:40 0 [Note] Server socket created on IP: '::'.

2021-11-04 23:43:40 0 [Warning] 'proxies_priv' entry '@% root@dev-dsk-fangam-2c-159d0c3e.us-west-2.amazon.com' ignored in --skip-name-resolve mode.

2021-11-04 23:43:40 2 [Note] Event Scheduler: scheduler thread started with id 2

2021-11-04 23:43:40 0 [Note] Reading of all Master_info entries succeeded

2021-11-04 23:43:40 0 [Note] Added new Master_info '' to hash table

2021-11-04 23:43:40 0 [Note] /rdsdbbin/mysql/bin/mysqld: ready for connections.

Version: '10.5.12-MariaDB-log'  socket: '/tmp/mysql.sock'  port: 13339  managed by https://aws.amazon.com/rds/

2021-11-04 23:43:43 0 [ERROR] InnoDB: Corruption of an index tree: table `crmx_002`.`inventory_record` index `product_id`, father ptr page no 562439, child page no 7242752

PHYSICAL RECORD: n_fields 2; compact format; info bits 0

 0: len 8; hex 8000000000000005; asc         ;;

 1: len 8; hex 8000000861cafbca; asc     a   ;;

2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 756; next rec: 112

PHYSICAL RECORD: n_fields 3; compact format; info bits 0

 0: len 8; hex 8000000000000000; asc         ;;

 1: len 8; hex 800000003bb8f279; asc     ;  y;;

 2: len 4; hex 00089507; asc     ;;

2021-11-04 23:43:43 0 [Note] InnoDB: n_owned: 0; heap_no: 567; next rec: 112

2021-11-04 23:43:43 0 [ERROR] [FATAL] InnoDB: You should dump + drop + reimport the table to fix the corruption. If the crash happens at database startup. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery. Then dump + drop + reimport.

211104 23:43:43 [ERROR] mysqld got signal 6 ;

This could be because you hit a bug. It is also possible that this binary

or one of the libraries it was linked against is corrupt, improperly built,

or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

AWS kept trying to restart the database without success, resulting in a reboot loop.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

211119_mm_cds_crash_error_log.txt
2021-11-19 12:28
24 kB
Heiki Laaniste
211202_mm-cds-2_reboot_loop.txt
2021-12-02 07:50
77 kB
Heiki Laaniste
bugreport_deleter_proc.sql
2021-11-05 08:29
4 kB
Heiki Laaniste

Issue Links

duplicates

MDEV-26917 InnoDB: Clustered record for sec rec not found index

Closed

relates to

MDEV-25283 Assertion `index->table->is_temporary() || !page_is_leaf(block->frame) || !!(index)->is_primary()' failed.

Closed

MDEV-26808 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.28/storage/innobase/row/row0ins.cc line 221

Closed

MDEV-27734 Set innodb_change_buffering=none by default

Closed

MDEV-28148 InnoDB slave crash / signal 6

Closed

MDEV-22986 [ERROR] InnoDB: Record in index was not found on update in mysqld.log on slave

Closed

MDEV-25796 Failing assertion: !cursor->index->is_committed() in row0ins.cc

Open

MDEV-27765 MariaDB stopped to work randomly - misery started at "Unable to find a record to delete-mark"

Closed

(3 relates to)

Activity

People

Assignee:: Marko Mäkelä

Reporter:: Heiki Laaniste

Votes:: 2 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 2021-11-05 08:28

Updated:: 2022-04-05 12:58

Resolved:: 2022-04-05 12:58

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.