[MDEV-26808] InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.28/storage/innobase/row/row0ins.cc line 221 Created: 2021-10-12  Updated: 2022-01-06  Resolved: 2022-01-06

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.3.28
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Allen Lee (Inactive) Assignee: Marko Mäkelä
Resolution: Incomplete Votes: 0
Labels: None
Environment:

RedHat Enterprise Linux, On Premise, Virtualized


Attachments: Text File show_global_variables_x01gppsgdb2a.log    
Issue Links:
Relates
relates to MDEV-26977 mariadb 10.5.12 reboot loop in AWS | ... Closed

 Description   

user hit the following error:

2021-10-12 08:19:24 0x7f75b6203700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.3.28/storage/innobase/row/row0ins.cc line 221
InnoDB: Failing assertion: !cursor->index->is_committed()
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
211012  8:19:24 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.3.28-MariaDB-log
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=9263
max_threads=15002
thread_count=8937
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 33047879 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f782009c7b8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f75b6202d30 thread_stack 0x40000
mysys/stacktrace.c:175(my_print_stacktrace)[0x556cb3562d7e]
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x556cb3001c1f]
/lib64/libpthread.so.0(+0xf630)[0x7f93486a5630]
:0(__GI_raise)[0x7f9346977387]
:0(__GI_abort)[0x7f9346978a78]
2021-10-12  8:19:45 40757673 [Warning] Aborted connection 40757673 to db: 'dpnsdb' user: 'appdynamicsdbadm' host: '10.56.106.111' (Got timeout reading communication packets)
ut/ut0rbt.cc:218(??)[0x556cb2d3bcf6]
row/row0ins.cc:221(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x556cb325316f]
row/row0ins.cc:3290(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x556cb3253273]
row/row0upd.cc:2558(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >::~map())[0x556cb32949e0]
row/row0upd.cc:3362(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >::~map())[0x556cb329524a]
row/row0mysql.cc:1849(void std::__introsort_loop<unsigned char**, long>(unsigned char**, unsigned char**, long))[0x556cb32673cb]
handler/ha_innodb.cc:8820(Gis_read_stream::check_next_symbol(char))[0x556cb31a753c]
sql/handler.cc:6508(handler::ha_update_row(unsigned char const*, unsigned char const*))[0x556cb300d312]
sql/sql_update.cc:956(mysql_update(THD*, TABLE_LIST*, List<Item>&, List<Item>&, Item*, unsigned int, st_order*, unsigned long long, bool, unsigned long long*, unsigned long long*))[0x556cb2ec46a7]
sql/sql_parse.cc:4328(mysql_execute_command(THD*))[0x556cb2e1fdb8]
sql/sql_parse.cc:7842(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x556cb2e2528b]
sql/sql_parse.cc:1916(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x556cb2e26b64]
sql/sql_parse.cc:1401(do_command(THD*))[0x556cb2e2873b]
sql/sql_connect.cc:1403(do_handle_one_connection(CONNECT*))[0x556cb2effad6]
sql/sql_connect.cc:1310(handle_one_connection)[0x556cb2effbed]
/lib64/libpthread.so.0(+0x7ea5)[0x7f934869dea5]
2021-10-12  8:19:55 40757800 [Warning] Aborted connection 40757800 to db: 'campaigns' user: 'appdynamicsdbadm' host: '10.56.106.111' (Got timeout reading communication packets)
2021-10-12  8:19:55 40757813 [Warning] Aborted connection 40757813 to db: 'campaigns' user: 'appdynamicsdbadm' host: '10.56.106.111' (Got timeout reading communication packets)
/lib64/libc.so.6(clone+0x6d)[0x7f9346a3f98d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f78203c0b50): update kw_elevation_check set CIN='S8105910A', FAILED_ATTEMPTS=0, LAST_LOGIN_DATE_TIME='2021-10-12 08:19:24.553', MOBILE_NO='97989163', WALLET_ID='8888880012754797' where ELEVATION_KEY=1352982
 
Connection ID (thread ID): 40755140
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /mariadb/data
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             788917               788917               processes 
Max open files            65536                65536                files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       788917               788917               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e %P %I

  • schema:

kw_elevation_check
Create Table: CREATE TABLE `kw_elevation_check` (
`ELEVATION_KEY` int(11) NOT NULL AUTO_INCREMENT,
`WALLET_ID` varchar(30) NOT NULL,
`CIN` varchar(20) DEFAULT NULL,
`MOBILE_NO` varchar(10) NOT NULL,
`FAILED_ATTEMPTS` varchar(20) DEFAULT NULL,
`LAST_LOGIN_DATE_TIME` datetime NOT NULL,
PRIMARY KEY (`ELEVATION_KEY`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

core file and global variables are attached.



 Comments   
Comment by Marko Mäkelä [ 2021-10-12 ]

In the attached stack traces (as well as based on the source code line number in the Description), I see that some secondary index is involved with this. But, the CREATE TABLE statement does not contain any INDEX, KEY, or UNIQUE keyword apart from the PRIMARY KEY.

What is the actual table definition?

In the core dump on which the thread apply all backtrace was executed, the name of the secondary index should be available if you execute the following in GDB:

print ((dict_index_t*)0x7f79f00994b8)->name

Comment by Marko Mäkelä [ 2021-10-12 ]

I see innodb_change_buffering=all in the configuration. Given that this is version 10.3.28, I believe that such corruption can be caused by the attempted fix MDEV-21069 and fixed by MDEV-25869.

Comment by Sergei Golubchik [ 2021-12-06 ]

Did the upgrade help?

Generated at Thu Feb 08 09:48:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.