[MDEV-32591] InnoDB indexes are inconsistent after dropping partition Created: 2023-10-26  Updated: 2023-11-09  Resolved: 2023-11-09

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.16
Fix Version/s: 10.5.22

Type: Bug Priority: Major
Reporter: Conor Murphy Assignee: Matthias Leich
Resolution: Fixed Votes: 0
Labels: None
Environment:

RHEL 8



 Description   

Tables are used to store time series data. Older data is removed by dropping partitions from the table. This has been in place for a number of years.

Recently upgraded from 10.3->10.5. Since then, we've dropped a partition in two separate MariaDB instances. In both cases, after the partition was dropped, attempts to insert or delete rows from the table fails with errors like

2023-10-26 16:32:48 467 [ERROR] Found index <index> whose column info does not match that of MariaDB.
2023-10-26 16:32:48 467 [ERROR] InnoDB indexes are inconsistent with what defined in .frm for table ./<db>/<table>#P#P20230901
 
2023-10-26 15:38:46 64307678 [Warning] InnoDB: Using a partial-field key prefix in search, index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Last data field length 8 bytes, key ptr now exceeds key end by 3 bytes. Key value in the MySQL format:
 len 7; hex b50399b1740000; asc     t  ;
 
2023-10-26 15:38:46 64307678 [ERROR] InnoDB: Clustered record for sec rec not found index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */
InnoDB: sec index record PHYSICAL RECORD: n_fields 3; compact format; info bits 0
 0: len 2; hex 03b5; asc   ;;
 1: len 8; hex 87c174b199007005; asc   t   p ;;
 2: len 6; hex af05f8af05e2; asc       ;;
 
InnoDB: clust index record PHYSICAL RECORD: n_fields 10; compact format; info bits 0
 0: len 6; hex 007005af066c; asc  p   l;;
 1: len 6; hex 000000000000; asc       ;;
 2: len 7; hex 80000000000000; asc        ;;
 3: len 2; hex 03b5; asc   ;;
 4: len 4; hex 000146ef; asc   F ;;
 5: len 8; hex 87e074b199000000; asc   t     ;;
 6: len 4; hex 00000000; asc     ;;
 7: len 4; hex 00000000; asc     ;;
 8: len 4; hex 15000000; asc     ;;
 9: len 2; hex 0000; asc   ;;
 
TRANSACTION 421589657076240, ACTIVE 0 sec starting index read
mysql tables in use 8, locked 0
0 lock struct(s), heap size 1128, 0 row lock(s)
MySQL thread id 64307678, OS thread handle 140095744042752, query id 791359259 <hostname> 192.168.255.80 statsadm Sending data
SELECT
 <tableb>.hostname  AS hostname,
 MAX( <table>.numCurrentSessions ) AS max
FROM
 <table>,
 <tableb>,
 <tablec>
WHERE
 <tablec>.name = @site AND
 <table>.time BETWEEN @date AND @date + INTERVAL 1 DAY AND
 <table>.siteid = <tablec>.id AND
 <tableb>.id= <table>.serverid
GROUP BY hostname
 
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/

2023-10-26 15:40:30 64307291 [ERROR] [FATAL] InnoDB: Cannot insert tuple TUPLE (info_bits=0, 3 fields): {[2]  (0x02E2),[5]  t  (0x1DDE74B199),[6] p    (0x007005BDDCCA)}into index `<index>` of table `<db>`.`<table>` /* Partition `P20231101` */. Max size: 19
231026 15:40:30 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.5.16-MariaDB-log
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=56
max_threads=153
thread_count=30
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467873 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f6a9c39c5a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f6a94b44bd8 thread_stack 0x49000
Can't start addr2line
/usr/libexec/mysqld(my_print_stacktrace+0x41)[0x55a8ee32add1]
/usr/libexec/mysqld(handle_fatal_signal+0x4a5)[0x55a8eddc0355]
/lib64/libpthread.so.0(+0x12cf0)[0x7f6f0959fcf0]
/lib64/libc.so.6(gsignal+0x10f)[0x7f6f088fbacf]
/lib64/libc.so.6(abort+0x127)[0x7f6f088ceea5]
/usr/libexec/mysqld(+0x644be7)[0x55a8edaa8be7]
/usr/libexec/mysqld(+0x6485f6)[0x55a8edaac5f6]
/usr/libexec/mysqld(+0xce28fd)[0x55a8ee1468fd]
/usr/libexec/mysqld(+0xce78de)[0x55a8ee14b8de]
/usr/libexec/mysqld(+0xce7fe4)[0x55a8ee14bfe4]
/usr/libexec/mysqld(+0xcf8bd0)[0x55a8ee15cbd0]
/usr/libexec/mysqld(+0xc411ed)[0x55a8ee0a51ed]
/usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
/usr/libexec/mysqld(+0xb812f0)[0x55a8edfe52f0]
/usr/libexec/mysqld(_ZN7handler12ha_write_rowEPKh+0x177)[0x55a8eddcdee7]
/usr/libexec/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_infoP13select_result+0x1dd)[0x55a8edb8d34d]
/usr/libexec/mysqld(_Z10mysql_loadP3THDPK12sql_exchangeP10TABLE_LISTR4ListI4ItemES9_S9_15enum_duplicatesbb+0x17de)[0x55a8edbba71e]
/usr/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x185f)[0x55a8edbcba0e]
/usr/libexec/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x238)[0x55a8edbbc956]
/usr/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x177d)[0x55a8edbc842c]
/usr/libexec/mysqld(_Z10do_commandP3THD+0x122)[0x55a8edbc9702]
/usr/libexec/mysqld(_Z24do_handle_one_connectionP7CONNECTb+0x4e0)[0x55a8edcbaa00]
/usr/libexec/mysqld(handle_one_connection+0x5d)[0x55a8edcbadad]
/usr/libexec/mysqld(+0xb882d8)[0x55a8edfec2d8]
/lib64/libpthread.so.0(+0x81ca)[0x7f6f095951ca]
/lib64/libc.so.6(clone+0x43)[0x7f6f088e6e73]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f6a9c04a370): LOAD DATA LOCAL INFILE '/data/tmp/makeStats_12/<table>.bcp' INTO TABLE <table> (time,serverid,siteid,cpuUsed,memoryUsed,processes,numCurrentSessions)
 
Connection ID (thread ID): 64307291
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /data/db/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             94970                94970                processes
Max open files            160000               160000               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       94970                94970                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e



 Comments   
Comment by Conor Murphy [ 2023-10-26 ]

Dropping and re-creating the index seems to stop the errors/crashes.

Comment by Matthias Leich [ 2023-11-03 ]

Hi Conor,
would you please send the insert statement and the SHOW CREATE TABLE .... output for all involved base tables?
Many thanks in advance

Comment by Conor Murphy [ 2023-11-08 ]

Issue was resolved by updating to 10.5.22

Generated at Thu Feb 08 10:32:31 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.