[MDEV-31055] ha_mroonga.so segfault Created: 2023-04-14  Updated: 2023-06-30  Resolved: 2023-06-30

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Mroonga
Affects Version/s: 10.3.38
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: bulepage Assignee: Kouhei Sutou
Resolution: Cannot Reproduce Votes: 0
Labels: crash


 Description   

After we upgraded from mariadb 10.1 to 10.3.38 we get segmentation fault in ha_mroonga

In dmesg

mysqld[29299]: segfault at 0 ip 00007f8150fe9ffc sp 00007f7e9a718110 error 4 in ha_mroonga.so[7f8150f06000+447000]

in mysql error.log

230412 15:02:28 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
 
Server version: 10.3.38-MariaDB-1:10.3.38+maria~ubu1804-log source revision: c73985f2ce8a391582787f3e310a011c1a712bec
key_buffer_size=1572864000
read_buffer_size=2097152
max_used_connections=2135
max_threads=65537
thread_count=259
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 404260895 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
 
Thread pointer: 0x7f7c882a99f8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f7e9a71ad28 thread_stack 0x49000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x560a736558fe]
/usr/sbin/mysqld(handle_fatal_signal+0x515)[0x560a730e7925]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f81c21df980]
/usr/lib/mysql/plugin/ha_mroonga.so(grn_obj_get_value+0xd1c)[0x7f8150fe9ffc]
/usr/lib/mysql/plugin/ha_mroonga.so(+0x979ed)[0x7f8150f9d9ed]
/usr/sbin/mysqld(_ZN15Item_func_match8val_realEv+0x7c)[0x560a7314254c]
/usr/sbin/mysqld(_ZN15Item_func_match7val_intEv+0xd)[0x560a73152c3d]
/usr/sbin/mysqld(_Z8filesortP3THDP5TABLEP8FilesortP16Filesort_trackerP4JOINy+0xaa7)[0x560a730e6277]
/usr/sbin/mysqld(_Z17create_sort_indexP3THDP4JOINP13st_join_tableP8Filesort+0xd5)[0x560a72f45285]
/usr/sbin/mysqld(_ZN13st_join_table10sort_tableEv+0x73)[0x560a72f45533]
/usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x29)[0x560a72f45599]
/usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x151)[0x560a72f379c1]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x932)[0x560a72f57162]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x560a72f575b3]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xf7)[0x560a72f57707]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x14d)[0x560a72f5809d]
/usr/sbin/mysqld(+0x595101)[0x560a72ef7101]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x549f)[0x560a72f03c0f]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x1ea)[0x560a72f0645a]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xc2d)[0x560a72f07dcd]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x126)[0x560a72f09ae6]
/usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0xdf)[0x560a7301e66f]
/usr/sbin/mysqld(+0x759b00)[0x560a730bbb00]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f81c21d46db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f81c1efd61f]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f7c88df8d60): SELECT nev, cim, adoszam, cegj_sz, oo_cegj_sz, arbev, allapot
FROM `cegek_ac_bov_mr`
WHERE MATCH (full) AGAINST ("+ÉPKAR* +–* +Ép*" in boolean mode)
ORDER BY `arbev` DESC
 LIMIT 10
 
Connection ID (thread ID): 2685296
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=
on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off
,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_eliminatio
n=on,extended_keys=off,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             116219               116219               processes
Max open files            32768                32768                files
Max locked memory         67108864             67108864             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       116219               116219               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E
 
Kernel version: Linux version 4.15.0-171-generic (buildd@ubuntu) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #180-Ubuntu SMP Wed Mar 2 17:25:05 UTC 2022

after this server crashed/restarted

OS: Ubuntu 18.04.6 LTS



 Comments   
Comment by Kouhei Sutou [ 2023-05-26 ]

I tried showing error lines by the following command lines but failed...

# curl -LsS https://r.mariadb.com/downloads/mariadb_repo_setup | bash -s -- --mariadb-server-version=mariadb-10.3
# apt install -y --allow-downgrades binutils mariadb-plugin-mroonga-dbgsym=1:10.3.38+maria~ubu1804 mariadb-plugin-mroonga=1:10.3.38+maria~ubu1804 mariadb-server-10.3=1:10.3.38+maria~ubu1804 mariadb-server-core-10.3=1:10.3.38+maria~ubu1804
# addr2line --exe=/usr/lib/debug/.build-id/24/193f9b13251c9eb04724cb13ceda6359d2553b.debug 0x979ed
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xa79288 vs 0x79c188)
ha_mroonga.cpp:?
# addr2line --exe=/usr/lib/debug/.build-id/24/193f9b13251c9eb04724cb13ceda6359d2553b.debug 0xe3ffc
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xa79288 vs 0x79c188)
??:?

The "mariadb-plugin-mroonga-dbgsym" package may be broken.

Sorry. I can't debug this only with the current information...

Comment by Daniel Black [ 2023-05-26 ]

Does the 447000 (decimal?) offset need subtracted from the offset?

Comment by Kouhei Sutou [ 2023-05-26 ]

Thanks but it doesn't work:

# addr2line --exe=/usr/lib/debug/.build-id/24/193f9b13251c9eb04724cb13ceda6359d2553b.debug 0x2a7d5 
addr2line: DWARF error: section .debug_info is larger than its filesize! (0xa79288 vs 0x79c188)
??:0

I think that "addr2line: DWARF error: section .debug_info is larger than its filesize! (0xa79288 vs 0x79c188)" is strange not the given address. And I think that the error is caused by broken mariadb-plugin-mroonga-dbgsym package.

Comment by Sergei Golubchik [ 2023-06-27 ]

May be it's 10.3-only issue? 10.3 has reached its end-of-life, so there will be no more releases for 10.3 and no more bugfixes.

bulepage, does this issue repeat on newer MariaDB versions?

Comment by bulepage [ 2023-06-29 ]

Our system run 10.3, can't upgrade now .
At the beginning, the error occurred 2-3 times, then not since then.

Close the issue.

Comment by Sergei Golubchik [ 2023-06-30 ]

Thanks!

Generated at Thu Feb 08 10:20:56 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.