[MDEV-30399] mariadb crash observed Created: 2023-01-13  Updated: 2023-03-06  Resolved: 2023-03-06

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.6.8
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Yiu-Chung Lee Assignee: Unassigned
Resolution: Incomplete Votes: 0
Labels: None
Environment:

Ubuntu 18.04 mariadb package 1:10.6.8+maria~bionic

Attachments: File error.log     File mysql.cfg.tar.gz    

 Description   

We found a spontaneous crash in our mariadb server. It was configured as a part of galera cluster. The configuration file and error log is attached. If there is not enough information you may raise the concern to me, however the customer is likely unwilling to provide further information, at least they are unwilling to disclose their DB schema.

Dec  2 08:00:03 mariadb-server mariadbd[14288]: 221202  8:00:03 [ERROR] mysqld got signal 11 ;
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: This could be because you hit a bug. It is also possible that this binary
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: We will try our best to scrape up some info that will hopefully help
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: diagnose the problem, but since we have already crashed,
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: something is definitely wrong and this may fail.
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Server version: 10.6.8-MariaDB-1:10.6.8+maria~bionic
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: key_buffer_size=134217728
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: read_buffer_size=131072
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: max_used_connections=10
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: max_threads=153
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: thread_count=12
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: It is possible that mysqld could use up to
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K  bytes of memory
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Thread pointer: 0x7f5198001268
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Attempting backtrace. You can use the following information to find out
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: where mysqld died. If you see no messages after this, something went
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: terribly wrong...
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: stack_bottom = 0x7f519de6bd98 thread_stack 0x49000
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5626ecd424ee]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(handle_fatal_signal+0x545)[0x5626ec78aa25]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f51a1eb8980]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN13st_join_table17save_explain_dataEP20Explain_table_accessybPS_+0x30f)[0x5626ec5bbbdf]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0xbb5)[0x5626ec5bd7f5]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN17save_explain_dataEP13Explain_querybbbb+0x1e6)[0x5626ec5bdb56]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN13build_explainEv+0x7f)[0x5626ec5bdc2f]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Printing to addr2line failed
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0x62)[0x5626ec5c74b2]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(+0x6f5ee1)[0x5626ec527ee1]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0xb7)[0x5626ec5276d7]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN13st_select_lex14handle_derivedEP3LEXj+0x47)[0x5626ec545b27]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN15optimize_stage2Ev+0x17f)[0x5626ec5c10cf]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x1636)[0x5626ec5c4c86]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xc3)[0x5626ec5c7513]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xb3)[0x5626ec5c75e3]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resultm+0x141)[0x5626ec5c7d11]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(+0x738061)[0x5626ec56a061]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x1298)[0x5626ec566178]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1d6)[0x5626ec56d136]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(+0x73a802)[0x5626ec56c802]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1baf)[0x5626ec56309f]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x127)[0x5626ec560f47]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3e7)[0x5626ec6658c7]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(handle_one_connection+0x34)[0x5626ec665c54]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /usr/sbin/mariadbd(+0xb9883d)[0x5626ec9ca83d]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f51a1ead6db]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f51a12ab61f]
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Trying to get some variables.
 
Dec  2 08:00:03 mariadb-server mariadbd[14288]: Some pointers may be invalid and cause the dump to abort.



 Comments   
Comment by Alice Sherepa [ 2023-01-13 ]

Is the crash repeatable if you run the same query? If yes, could you please try to run EXPLAIN EXTENDED for this query and provide the output.

Comment by Yiu-Chung Lee [ 2023-01-16 ]

The crash is unrepeatable. Anyway this is the EXPLAIN EXTENDED result

id select_type table type possible_keys key key_len ref rows filtered Extra
1 PRIMARY <derived2> ALL NULL NULL NULL NULL 3510864 100.00
2 DERIVED bi_bc ref PRIMARY,bundle,field_basic_content_target_id bundle 130 const 344 100.00 Using index condition; Using temporary; Using filesort
2 DERIVED lri_bc ref|filter PRIMARY,bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.bi_bc.field_basic_content_target_id 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED n ref PRIMARY,node_iddefault_langcodelangcode,node_status_type PRIMARY 4 swd_rchd_d9.bi_bc.field_basic_content_target_id 1 97.39 Using where
2 DERIVED bi_node ref PRIMARY,node_iddefault_langcodelangcode,node_status_type PRIMARY 4 swd_rchd_d9.bi_bc.entity_id 1 97.39 Using where
2 DERIVED lri_node ref PRIMARY,node_iddefault_langcodelangcode,node_status_type PRIMARY 4 swd_rchd_d9.lri_bc.entity_id 1 97.39 Using where
2 DERIVED rchds_search_index ref nid nid 4 swd_rchd_d9.bi_bc.field_basic_content_target_id 7 100.00 Using where
2 DERIVED node_field_data_rchds_search_index ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.bi_bc.field_basic_content_target_id 1 100.00 Using index
2 DERIVED node__field_basic_content_node_field_data ref|filter bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.node_field_data_rchds_search_index.nid 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED node_field_data_node__field_basic_content ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.node__field_basic_content_node_field_data.entity_id 1 100.00 Using where; Using index
2 DERIVED node__field_basic_content_node_field_data_1 ref|filter bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.node_field_data_rchds_search_index.nid 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED node_field_data_node__field_basic_content_1 ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.node__field_basic_content_node_field_data_1.entity_id 1 100.00 Using where; Using index
2 DERIVED node__field_basic_content_node_field_data_2 ref|filter bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.node_field_data_rchds_search_index.nid 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED node_field_data_node__field_basic_content_2 ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.node__field_basic_content_node_field_data_2.entity_id 1 100.00 Using where; Using index
2 DERIVED node__field_basic_content_node_field_data_3 ref|filter bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.node_field_data_rchds_search_index.nid 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED node_field_data_node__field_basic_content_3 ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.node__field_basic_content_node_field_data_3.entity_id 1 100.00 Using where; Using index
2 DERIVED node__field_basic_content_node_field_data_4 ref|filter bundle,field_basic_content_target_id field_basic_content_target_id|bundle 4|130 swd_rchd_d9.node_field_data_rchds_search_index.nid 3 (14%) 14.29 Using where; Using rowid filter
2 DERIVED node_field_data_node__field_basic_content_4 ref PRIMARY,node_iddefault_langcode_langcode node_iddefault_langcode_langcode 4 swd_rchd_d9.node__field_basic_content_node_field_data_4.entity_id 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_contentnode_field_bi_telephone_number ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content.nid,const 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_content_1node_field_remarks_option1 ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_1.nid,const 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_content_1node_field_si_a_charging_min ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_1.nid,const 1 100.00 Using where
2 DERIVED node_field_data_node_field_basic_content_1node_field_si_a_charging_max ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_1.nid,const 1 100.00 Using where
2 DERIVED node_field_data_node_field_basic_contentnode_field_bi_name_sc ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content.nid,const 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_contentnode_field_bi_address_sc ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content.nid,const 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_content_3node_field_irtsp_type ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_3.nid,const 1 100.00 Using where
2 DERIVED node_field_data_node_field_basic_content_3node_field_irtsp_nature ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_3.nid,const 1 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_content_2node_field_ipar_image ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content_2.nid,const 2 100.00 Using where; Using index
2 DERIVED node_field_data_node_field_basic_contentnode_field_bi_name_en ref PRIMARY PRIMARY 5 swd_rchd_d9.node_field_data_node__field_basic_content.nid,const 1 100.00 Using where

Comment by Alice Sherepa [ 2023-01-17 ]

Probably the reason is rowid filter, smth like MDEV-26446.
It is better to upgrade to the latest version, because there were some bugs in that area fixed recently.
If the crash appear again, one could try using rowid_filter=off as a temporary workaround. To fix this bug we need a repeatable testcase to demonstrate the problem as it is dependent on table data, statistics, table structures and I had no luck to repeat the crash on my side.

Comment by Yiu-Chung Lee [ 2023-01-17 ]

By "latest version", do you mean latest patch in 10.6 series or latest 10.11 release?

Comment by Alice Sherepa [ 2023-01-17 ]

10.6.11

Generated at Thu Feb 08 10:15:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.