[MDEV-25120] MariaDB 10.5.6/10.5.8 crash (sig 11) during a SELECT Created: 2021-03-11  Updated: 2022-05-30

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.5.6, 10.5.8
Fix Version/s: 10.5

Type: Bug Priority: Major
Reporter: Juan Assignee: Oleksandr Byelkin
Resolution: Unresolved Votes: 1
Labels: None

Attachments: File cs0253752-20210311-ldd.out.tgz.enc     File cs0253752-20210311-mysqld-binaries.tgz     File cs0253752-20210311-proc-and-table-structures.sql.tgz.enc     File cs0253752-20210311_13020-mysqld_bt_all_threads.txt.tgz.enc     File cs0253752-20210311_23020-mysqld_bt_all_threads.txt.tgz.enc     File globvars.status.etc.out.tgz.enc    
Issue Links:
Relates
relates to MDEV-28663 Wrong data from Virtual Column by ord... Confirmed

 Description   

210209  8:10:26 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.
 
Server version: 10.5.6-MariaDB-log
 
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=101
max_threads=652
thread_count=105
 
It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1566099 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7ef5bc08e1c8
 
Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong...
 
stack_bottom = 0x7ef5b074bc90 thread_stack 0x49000
 
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x559728fbe0ae]
/usr/sbin/mariadbd(handle_fatal_signal+0x307)[0x5597289bf5d7]
/usr/java/jdk/jre/lib/amd64/server/libjvm.so(+0x942be4)[0x7efd638a2be4]
/usr/java/jdk/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x1b1)[0x7efd638a79c1]
:0(os::Linux::chained_handler(int, siginfo_t*, void*))[0x7efd6389aa58]
/lib64/libpthread.so.0(+0xf630)[0x7efd755cb630]
/usr/sbin/mariadbd(+0xfb2c4d)[0x559729003c4d]
/usr/sbin/mariadbd(my_convert_using_func+0xa2)[0x55972900c012]
??:0(my_strnxfrmlen_unicode_full_bin)[0x55972900c173]
??:0(my_convert_using_func)[0x559728845b81]
??:0(my_convert)[0x5597286fe673]
??:0(String::copy(char const*, unsigned long, charset_info_st const*, charset_info_st const*, unsigned int*))[0x5597289a4207]
??:0(Protocol::net_store_data_cs(unsigned char const*, unsigned long, charset_info_st const*, charset_info_st const*))[0x5597286ffc5f]
??:0(Field::send_binary(Protocol*))[0x5597287703e2]
??:0(Protocol::send_result_set_row(List<Item>*))[0x5597287fe716]
??:0(select_send::send_data(List<Item>&))[0x5597287ea22c]
??:0(copy_fields(TMP_TABLE_PARAM*))[0x5597287efbdd]
??:0(Item_cond_and::build_equal_items(THD*, COND_EQUAL*, bool, COND_EQUAL**))[0x55972881e57f]
??:0(sub_select(JOIN*, st_join_table*, bool))[0x55972881e7e3]
??:0(JOIN::exec_inner())[0x55972881c9a6]
??:0(JOIN::exec())[0x55972881d50c]
??:0(mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*))[0x55972868440a]
??:0(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5597287c35d5]
/usr/sbin/mariadbd(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x36)[0x55972871ac16]
??:0(mysql_execute_command(THD*))[0x55972872340a]
??:0(sp_instr_stmt::exec_core(THD*, unsigned int*))[0x559728723e8c]
??:0(sp_lex_keeper::reset_lex_and_exec_core(THD*, unsigned int*, bool, sp_instr*))[0x55972871eba8]
??:0(sp_instr_stmt::execute(THD*, unsigned int*))[0x55972872012f]
??:0(sp_head::execute(THD*, bool))[0x5597287b59fa]
??:0(comp_ne_creator(bool))[0x5597287b9aca]
??:0(Sql_cmd_call::execute(THD*))[0x5597287c1c91]
??:0(mysql_execute_command(THD*))[0x5597287da59d]
??:0(Prepared_statement::execute(String*, bool))[0x5597287da742]
??:0(Prepared_statement::execute_loop(String*, bool, unsigned char*, unsigned char*))[0x5597287db5fd]
??:0(Prepared_statement::execute_bulk_loop(String*, bool, unsigned char*, unsigned char*))[0x5597287db6b7]
??:0(mysqld_stmt_execute(THD*, char*, unsigned int))[0x5597287c7bb0]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x5597287c9feb]
??:0(do_command(THD*))[0x5597288b2b29]
??:0(do_handle_one_connection(CONNECT*, bool))[0x5597288b2db4]
??:0(handle_one_connection)[0x559728c0727d]
/lib64/libpthread.so.0(+0x7ea5)[0x7efd755c3ea5]
/lib64/libc.so.6(clone+0x6d)[0x7efd736ec96d]
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7ef53c732180): SELECT * FROM (                 SELECT `gdd`.`customerReference` AS `customerReference`, `gdd`.`policyNumber` AS `policyNumber`, NULL AS `schemeReference`, NULL AS `schemeName`, NULL AS `applicationReference`, min(`gdd`.`modifiedCreatedDateTime`) AS `modifiedCreatedDateTime`, min(`gdd`.`demandCreatedDateTime`) AS `demandCreatedDateTime`,count(distinct(`gdd`.`genericDemandId`)) AS `countDemands`, MIN(`gdd`.`genericDemandId`) AS `genericDemandId`,MIN(`gdd`.`status`) AS `status`, MAX(CASE `gdd`.`isUrgent` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isUrgent`,                                  MIN(gdd.reactivateDate) AS 'reactivateDate',                                        MAX(CASE `gdd`.`isBusinessException` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isBusinessException`                                   FROM                                                   (SELECT `ugdd`.`customerReference` AS `customerReference`,`ugdd`.`policyNumber` AS `policyNumber`, `ugdd`.`modifiedCreatedDateTime` AS `modifiedCreatedDateTime`,`ugdd`.`demandCreatedDateTime` AS `demandCreatedDateTime`,`ugdd`.`status` AS `status`, `ugdd`.`genericDemandId` AS `genericDemandId`,`ugdd`.`isUrgent` AS `isUrgent`, `ugdd`.`isBusinessException` AS `isBusinessException`,                (SELECT `ukpd`.`reactivateDate` FROM `PROD`.`UK_Pended_Demand` `ukpd` WHERE `ugdd`.`genericDemandId` = `ukpd`.`genericDemandId` AND `ukpd`.`isPend` = 1 AND `ukpd`.`reactivateDate` > UTC_TIMESTAMP() ORDER BY ukpd.ID DESC LIMIT 1) AS 'reactivateDate'               FROM `PROD`.`UK_Generic_Demand_Details` `ugdd`                                                   WHERE (`ugdd`.`initiator` =  NAME_CONST('pinUserId',_utf8'solanm2' COLLATE 'utf8_general_ci')                                                             AND `ugdd`.`status`<250)  ) gdd                                  JOIN `PROD`.`UK_CM_Policyholder_Data` `phd`                                   ON(`gdd`.`customerReference` = `phd`.`customerReference`)                                                        WHERE `phd`.`customerReference`<>''                                  GROUP BY `gdd`.`customerReference`       UNION ALL     /* Getting customers based on scheme reference */         SELECT NULL AS `customerReference`, NULL AS `policyNumber`, `gdd`.`schemeReference` AS `schemeReference`, `sd`.`schemeName` AS `schemeName`, NULL AS `applicationReference`, MIN(`gdd`.`modifiedCreatedDateTime`) AS `modifiedCreatedDateTime`,MIN(`gdd`.`demandCreatedDateTime`) AS `demandCreatedDateTime`, count(distinct(`gdd`.`genericDemandId`)) AS `countDemands`, MIN(`gdd`.`genericDemandId`) AS `genericDemandId`,MIN(`gdd`.`status`) AS `status`, MAX(CASE `gdd`.`isUrgent` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isUrgent`,MIN(gdd.reactivateDate )AS 'reactivateDate',MAX(CASE `gdd`.`isBusinessException` WHEN 1 THEN '1' WHEN 0 THEN '0'   END) AS `isBusinessException` FROM           (SELECT `ugdd`.`schemeReference` AS `schemeReference`, `ugdd`.`modifiedCreatedDateTime` AS `modifiedCreatedDateTime`,`ugdd`.`demandCreatedDateTime` AS `demandCreatedDateTime`,`ugdd`.`status` AS `status`, `ugdd`.`genericDemandId` AS `genericDemandId`,`ugdd`.`isUrgent` AS `isUrgent`, `ugdd`.`isBusinessException` AS `isBusinessException`,(SELECT `ukpd`.`reactivateDate` FROM `PROD`.`UK_Pended_Demand` `ukpd` WHERE `ugdd`.`genericDemandId` = `ukpd`.`genericDemandId` AND `ukpd`.`isPend` = 1 AND `ukpd`.`reactivateDate` > UTC_TIMESTAMP() ORDER BY ukpd.ID DESC LIMIT 1) AS 'reactivateDate'            FROM `PROD`.`UK_Generic_Demand_Details` `ugdd`                                               WHERE (`ugdd`.`initiator`= NAME_CONST('pinUserId',_utf8'solanm2' COLLATE 'utf8_general_ci') AND `ugdd`.`status`<250 AND `ugdd`.`policyNumber`='' AND `ugdd`.`applicationReference`='')           ) gdd        JOIN `PROD`.`UK_CM_Scheme_Data` `sd`        ON(`gdd`.`schemeReference` = `sd`.`schemeReference`) WHERE (`sd`.`schemeReference`<>'')       GROUP BY gdd.schemeReference               UNION ALL          /* Getting customers based on Application Reference */   /* Added `ugdd`.schemeReference <>'' in the WHERE clause AW-4764 */         SELECT NULL AS `customerReference`, NULL AS `policyNumber`, NULL AS `schemeReference`, NULL AS `schemeName`, `gdd`.`applicationReference` AS `applicationReference`, MIN(`gdd`.`modifiedCreatedDateTime`) AS `modifiedCreatedDateTime`, MIN(`gdd`.`demandCreatedDateTime`) AS `demandCreatedDateTime`,count(distinct(`gdd`.`genericDemandId`)) AS `countDemands`, MIN(`gdd`.`genericDemandId`) AS `genericDemandId`,MIN(`gdd`.`status`) AS `status`, MAX(CASE `gdd`.`isUrgent` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isUrgent`,MIN(gdd.reactivateDate) AS 'reactivateDate', MAX(CASE `gdd`.`isBusinessException` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isBusinessException` FROM                         ( SELECT `ugdd`.`applicationReference` AS `applicationReference`, `ugdd`.`modifiedCreatedDateTime` AS `modifiedCreatedDateTime`,`ugdd`.`demandCreatedDateTime` AS `demandCreatedDateTime`,`ugdd`.`status` AS `status`,`ugdd`.`genericDemandId` AS `genericDemandId`,`ugdd`.`isUrgent` AS `isUrgent`, `ugdd`.`isBusinessException` AS `isBusinessException`,(SELECT `ukpd`.`reactivateDate` FROM `PROD`.`UK_Pended_Demand` `ukpd` WHERE `ugdd`.`genericDemandId` = `ukpd`.`genericDemandId` AND `ukpd`.`isPend` = 1 AND `ukpd`.`reactivateDate` > UTC_TIMESTAMP() ORDER BY ukpd.ID DESC LIMIT 1) AS 'reactivateDate'                          FROM `PROD`.`UK_Generic_Demand_Details` `ugdd`             WHERE (`ugdd`.`initiator`= NAME_CONST('pinUserId',_utf8'solanm2' COLLATE 'utf8_general_ci') AND `ugdd`.`status`<250 AND `ugdd`.`policyNumber`='' AND `ugdd`.`applicationReference`<>'' AND `ugdd`.schemeReference <>'')           ) gdd        GROUP BY gdd.applicationReference              UNION ALL          /* Getting customers based on Aggregation Key */   /* Added `ugdd`.schemeReference ='' in the WHERE clause as for other cases AW-4764*/         SELECT NULL AS `customerReference`, NULL AS `policyNumber`, NULL AS `schemeReference`, NULL AS `schemeName`, `gdd`.`aggregationKey` AS `aggregationKey`, MIN(`gdd`.`modifiedCreatedDateTime`) AS `modifiedCreatedDateTime`, MIN(`gdd`.`demandCreatedDateTime`) AS `demandCreatedDateTime`,count(distinct(`gdd`.`genericDemandId`)) AS `countDemands`, MIN(`gdd`.`genericDemandId`) AS `genericDemandId`,MIN(`gdd`.`status`) AS `status`, MAX(CASE `gdd`.`isUrgent` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isUrgent`,MIN(gdd.reactivateDate) AS 'reactivateDate', MAX(CASE `gdd`.`isBusinessException` WHEN 1 THEN '1' WHEN 0 THEN '0' END) AS `isBusinessException` FROM                         ( SELECT `ugdd`.`aggregationKey` AS `aggregationKey`, `ugdd`.`modifiedCreatedDateTime` AS `modifiedCreatedDateTime`,`ugdd`.`demandCreatedDateTime` AS `demandCreatedDateTime`,`ugdd`.`status` AS `status`,`ugdd`.`genericDemandId` AS `genericDemandId`,`ugdd`.`isUrgent` AS `isUrgent`, `ugdd`.`isBusinessException` AS `isBusinessException`,(SELECT `ukpd`.`reactivateDate` FROM `PROD`.`UK_Pended_Demand` `ukpd` WHERE `ugdd`.`genericDemandId` = `ukpd`.`genericDemandId` AND `ukpd`.`isPend` = 1 AND `ukpd`.`reactivateDate` > UTC_TIMESTAMP() ORDER BY ukpd.ID DESC LIMIT 1) AS 'reactivateDate'                          FROM `PROD`.`UK_Generic_Demand_Details` `ugdd`             WHERE (`ugdd`.`initiator`= NAME_CONST('pinUserId',_utf8'solanm2' COLLATE 'utf8_general_ci') AND `ugdd`.`status`<250 AND `ugdd`.`policyNumber`='' AND `ugdd`.`aggregationKey`<>''  AND `ugdd`.`applicationReference`<>'' AND `ugdd`.schemeReference ='')           ) gdd        GROUP BY gdd.aggregationKey   )   ca ORDER BY `ca`.`isUrgent` DESC, if(`ca`.`modifiedCreatedDateTime` = '' or `ca`.`modifiedCreatedDateTime` is null,1,0),`ca`.`modifiedCreatedDateTime`,`ca`.`genericDemandId` 
Connection ID (thread ID): 11886737
 
Status: NOT_KILLED
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off



 Comments   
Comment by Sergei Golubchik [ 2021-03-12 ]

from the thread apply all bt full one can see that the actual statement was call MHP_SPGETASSIGNEDCUSTOMERSFORUSER('nft064'), it was prepared and executed via COM_STMT_EXECUTE. This SP contained this huge select. Also, apparently, they've loaded CONNECT engine with JDBC support, that loaded JVM, and the latter has installed its own signal handler, that's how JVM got into the stack trace,

Comment by Chris Calender (Inactive) [ 2021-03-17 ]

serg Thank you for those details, very helpful!

Btw, based on that, do you think CONNECT is involved in the bug, or are the JVM mentions purely there because CONNECT is enabled?

Fwiw, please note the customer is not using any CONNECT tables in this stored procedure (there are just 4 innodb tables with no FKs defined).

Comment by Sergei Golubchik [ 2021-03-23 ]

No, I don't think CONNECT has anything to do with it (besides loading the JVM, which installed its own SIGSEGV handler, but it only got into the play after the crash)

Generated at Thu Feb 08 09:35:18 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.