[MDEV-29143] mysql crash when selecting from large table Created: 2022-07-20  Updated: 2022-08-27

Status: Open
Project: MariaDB Server
Component/s: Storage Engine - InnoDB
Affects Version/s: 10.5.13
Fix Version/s: None

Type: Bug Priority: Major
Reporter: TAO ZHOU Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Environment:

AWS RDS Mariadb



 Description   

The following query caused a crash

select count(*), year, month,day
from
	(select *, year(created_at) as year, month(created_at) as month, day(created_at) as day
	from activity) test
where object_id like '%payroll/payslips%'
and month in ('6','7')
group by year, month, day
order by count(*) desc

Here's the crash log:

2022-07-20 08:13:14 0x2b6fd11bb700  InnoDB: Assertion failure in file /local/p4clients/pkgbuild-78Qdm/workspace/src/RDSMariaDB/storage/innobase/row/row0sel.cc line 2973
InnoDB: Failing assertion: prebuilt->trx->isolation_level == TRX_ISO_READ_UNCOMMITTED
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220720  8:13:14 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.5.13-MariaDB-log
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=467
max_threads=3002
thread_count=468
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7004922 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x2b6fc68a4698
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x2b6fd11b9578 thread_stack 0x40000
mysys/stacktrace.c:213(my_print_stacktrace)[0x56456e7a6ace]
sql/signal_handler.cc:222(handle_fatal_signal)[0x56456df535db]
sigaction.c:0(__restore_rt)[0x2b6cbc2a5100]
:0(__GI_raise)[0x2b6cbc4e75f7]
:0(__GI_abort)[0x2b6cbc4e8ce8]
ut/ut0rbt.cc:458(rbt_eject_node)[0x56456dc70e8e]
row/row0sel.cc:2977(row_sel_store_mysql_field)[0x56456dc6fa9a]
row/row0sel.cc:3175(row_sel_store_mysql_rec)[0x56456dc6fda5]
row/row0sel.cc:5497(row_search_mvcc(unsigned char*, page_cur_mode_t, row_prebuilt_t*, unsigned long, unsigned long))[0x56456e5959fc]
handler/ha_innodb.cc:9096(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int))[0x56456e4aad29]
sql/handler.cc:3187(handler::ha_index_next(unsigned char*))[0x56456df5884f]
sql/sql_select.cc:21917(join_read_next)[0x56456ddac2ac]
sql/sql_select.cc:20894(sub_select(JOIN*, st_join_table*, bool))[0x56456dda2ff3]
sql/sql_select.cc:20407(do_select)[0x56456ddc406f]
sql/sql_select.cc:4297(JOIN::exec())[0x56456ddc45b0]
sql/sql_select.cc:4775(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x56456ddc28d4]
sql/sql_select.cc:444(handle_select(THD*, LEX*, select_result*, unsigned long))[0x56456ddc327e]
sql/sql_parse.cc:6351(execute_sqlcom_select)[0x56456dd6b3b1]
sql/sql_parse.cc:5985(mysql_execute_command(THD*))[0x56456dd684b9]
sql/sql_parse.cc:8224(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x56456dd6dc80]
sql/sql_parse.cc:1966(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x56456dd64f1e]
sql/sql_parse.cc:1378(do_command(THD*))[0x56456dd6372d]
sql/sql_connect.cc:1461(do_handle_one_connection(CONNECT*, bool))[0x56456de52506]
sql/sql_connect.cc:1359(handle_one_connection)[0x56456de52874]
perfschema/pfs.cc:2204(pfs_spawn_thread)[0x56456e423bec]
pthread_create.c:0(start_thread)[0x2b6cbc29ddc5]
??:0(__clone)[0x2b6cbc5a8c9d]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x2b6fc69c96b0): -- no. of transactions recorded accessing payslips at a given day
select count(*), year, month,day
from
	(select *, year(created_at) as year, month(created_at) as month, day(created_at) as day
	from activity) test
where object_id like '%payroll/payslips%'
and month in ('6','7')
group by year, month, day
order by count(*) desc
 
Connection ID (thread ID): 431
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /rdsdbdata/db
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            unlimited            unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1048576              1048576              files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       62942                62942                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: /rdsdbdata/tmp/core-%e-%p
 
2022-07-20  8:13:33 0 [Note] InnoDB: Uses event mutexes



 Comments   
Comment by Alice Sherepa [ 2022-07-20 ]

Is it possible for you to check if the crash is repeatable on the latest version of MariaDB (for 10.5 - 10.5.16 currently) ? Please also the output of SHOW CREATE TABLE activity;

Comment by TAO ZHOU [ 2022-07-21 ]

@Alice Yes, I have just upgraded it to 10.5.16 and it still crashes on the same query.

output from SHOW CREATE TABLE

activity | CREATE TABLE `activity` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `activity` varchar(255) NOT NULL,
  `user_id` int(10) unsigned DEFAULT NULL,
  `ts` bigint(20) unsigned DEFAULT NULL,
  `object` varchar(255) DEFAULT NULL,
  `object_id` varchar(255) DEFAULT NULL,
  `target` varchar(255) DEFAULT NULL,
  `target_id` varchar(255) DEFAULT NULL,
  `success` tinyint(4) NOT NULL DEFAULT 1,
  `meta` text DEFAULT NULL,
  `created_at` datetime NOT NULL DEFAULT current_timestamp(),
  PRIMARY KEY (`id`),
  KEY `activity_activity_index` (`activity`),
  KEY `activity_userid_index` (`user_id`),
  KEY `activity_userid_activity_index` (`object`,`object_id`),
  KEY `activity_target_targetid_index` (`target`,`target_id`),
  KEY `activity_activity_object_success_createdat_index` (`activity`,`object`,`success`,`created_at`),
  KEY `activity_created_at` (`created_at`),
  KEY `activity_activity_user_success_createdat_index` (`activity`,`user_id`,`success`,`created_at`),
  KEY `activity_activity_object_user_success_created_at_index` (`activity`,`object`,`object_id`,`user_id`,`success`,`created_at`)
) ENGINE=InnoDB AUTO_INCREMENT=1442718 DEFAULT CHARSET=utf8

`EXPLAIN` shows there are 600,000 records to be scanned.

Comment by TAO ZHOU [ 2022-07-21 ]

After some further experiments, I can get consistent crashes and it's related to the `like` condition in the where clause. If I get rid of the `like`, it doesn't crash; if I use `like` on a different column, it also works fine.

Comment by Elena Stepanova [ 2022-07-25 ]

If it's reproducible for you reliably, can you provide a self-contained test case (create table, populate table, run the crashing statement) and server settings necessary to reproduce it? Thanks.

Comment by TAO ZHOU [ 2022-07-28 ]

@Elena
That's what I did.
I've posted the query above and the `create table` query.

Generated at Thu Feb 08 10:06:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.