[MDEV-31295] Program terminated with signal SIGSEGV, Segmentation fault on MariaDB 10.11 Created: 2023-05-16  Updated: 2023-05-17  Resolved: 2023-05-16

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.11.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Phil Assignee: Daniel Black
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Linux. Debian 11 Bullseye


Issue Links:
Duplicate
duplicates MDEV-31240 Crash with condition pushable into de... Closed
duplicates MDEV-31241 Crashes in subselects in choose_best_... Closed

 Description   

Repeatable crash with these tables and queries, from at least version 10.8.


ii mariadb-server 1:10.11.3+maria~deb11 amd64 MariaDB database server binaries
ii mariadb-server-core 1:10.11.3+maria~deb11 amd64 MariaDB database core server files
ii mariadb-server-core-dbgsym 1:10.11.3+maria~deb11 amd64 debug symbols for mariadb-server-core

root@netball-prod:~# uname -a
Linux netball-prod 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

root@netball-prod:~# free
total used free shared buff/cache available
Mem: 16380944 839612 14488712 26088 1052620 15231740
Swap: 1951740 0 1951740

root@netball-prod:~# my_print_defaults --mysqld
--socket=/run/mysqld/mysqld.sock
--pid-file=/run/mysqld/mysqld.pid
--basedir=/usr
--core_file=ON
--bind-address=127.0.0.1
--query_cache_type=0
--general_log_file=/var/log/mysql/mysql.log
--general_log=1
--log_error=/var/log/mysql/error.log
--log_slow_query_file=/var/log/mysql/mariadb-slow.log
--expire_logs_days=10
--character-set-server=utf8mb4
--collation-server=utf8mb4_general_ci

root@netball-prod:~# 230517 9:22:48 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.11.3-MariaDB-1:10.11.3+maria~deb11-log source revision: 0bb31039f54bd6a0dc8f0fc7d40e6b58a51998b0
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=4
max_threads=153
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468022 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fd3f4001668
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fd43c165d18 thread_stack 0x49000
Printing to addr2line failed
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55d83a27d96e]
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55d839d55d95]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7fd454d46140]
/usr/sbin/mariadbd(_ZN13st_join_table21choose_best_splittingEjyPy+0x571)[0x55d839cd3491]
/usr/sbin/mariadbd(Z16best_access_pathP4JOINP13st_join_tableyPK8POSITIONjbdPS3_S6+0x172)[0x55d839b467a2]
/usr/sbin/mariadbd(+0x85d3a3)[0x55d839b4a3a3]
/usr/sbin/mariadbd(+0x862270)[0x55d839b4f270]
/usr/sbin/mariadbd(_Z11choose_planP4JOINy+0x2f4)[0x55d839b50784]
/usr/sbin/mariadbd(+0x889832)[0x55d839b76832]
/usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x134c)[0x55d839b7b1dc]
/usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xa0)[0x55d839b7b7f0]
/usr/sbin/mariadbd(_ZN13st_select_lex31optimize_unflattened_subqueriesEb+0x10d)[0x55d839ad7ddd]
/usr/sbin/mariadbd(_ZN4JOIN15optimize_stage2Ev+0x176d)[0x55d839b785dd]
/usr/sbin/mariadbd(_ZN4JOIN14optimize_innerEv+0x145c)[0x55d839b7b2ec]
/usr/sbin/mariadbd(_ZN4JOIN8optimizeEv+0xa0)[0x55d839b7b7f0]
/usr/sbin/mariadbd(_Z12mysql_selectP3THDP10TABLE_LISTR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xca)[0x55d839b7b8da]
/usr/sbin/mariadbd(_Z13handle_selectP3THDP3LEXP13select_resulty+0x14f)[0x55d839b7c0df]
/usr/sbin/mariadbd(+0x804cb1)[0x55d839af1cb1]
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x4892)[0x55d839b00f82]
/usr/sbin/mariadbd(_ZN18Prepared_statement7executeEP6Stringb+0x4aa)[0x55d839b24baa]
/usr/sbin/mariadbd(ZN18Prepared_statement12execute_loopEP6StringbPhS2+0x9d)[0x55d839b24dcd]
/usr/sbin/mariadbd(+0x838d55)[0x55d839b25d55]
/usr/sbin/mariadbd(_Z19mysqld_stmt_executeP3THDPcj+0x2c)[0x55d839b25f3c]
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1c91)[0x55d839b05471]
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x138)[0x55d839b068e8]
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3af)[0x55d839c280bf]
/usr/sbin/mariadbd(handle_one_connection+0x5d)[0x55d839c2840d]
/usr/sbin/mariadbd(+0xc99c62)[0x55d839f86c62]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7fd454d3aea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fd454941a2f]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fd3f407d8f8): SELECT p.*, CONCAT(first_name, " ", last_name,
IF(
IFNULL(maiden_name, "") = "" OR show_maiden_name != 1,
"",
CONCAT(" (nee ", maiden_name, ")"))
)
AS `name`,GROUP_CONCAT(t.name SEPARATOR ',') AS team_names,
GROUP_CONCAT(pt.player_number SEPARATOR ',') AS team_player_numbers,
GROUP_CONCAT(pt.featured SEPARATOR ',') AS team_player_featured,(SELECT GROUP_CONCAT(cnt, ',') AS cnt FROM
(SELECT COUNT(pm.id) AS cnt, pm.player_id, pm.team_id
FROM `j25_nnz_stats_player_match` AS pm GROUP BY pm.player_id) AS q
WHERE p.id = q.player_id AND pt.team_id = q.team_id GROUP BY p.id)
AS team_caps
FROM `j25_nnz_stats_player` AS p
LEFT JOIN j25_nnz_stats_player_team AS pt ON pt.player_id = p.id
LEFT JOIN j25_nnz_stats_team AS t ON pt.team_id = t.id
WHERE p.champdata_id IN (1003361,994718,80808,1013810,1012791,994723,1006757,1006758,995763,80098,1020197,1006831,994117,80477,80708,1003470,998973,1001620,1027995,80506,998971,1010458,80640,80099,992995,1030459,80710)
GROUP BY p.id
ORDER BY `name` ASC

Connection ID (thread ID): 367
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off

The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63832 63832 processes
Max open files 32768 32768 files
Max locked memory 524288 524288 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63832 63832 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core

Kernel version: Linux version 5.10.0-23-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.179-1 (2023-05-12)

root@netball-prod:~# gdb /usr/sbin/mariadbd /var/lib/mysql/core
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/mariadbd...
Reading symbols from /usr/lib/debug/.build-id/66/64a0b06634fa089b05af0f9537cd9b79c474da.debug...

warning: Can't open file anon_inode:[io_uring] which was expanded to anon_inode:[io_uring] during file-backed mapping note processing

warning: Can't open file anon_inode:[io_uring] which was expanded to anon_inode:[io_uring] during file-backed mapping note processing
[New LWP 3437]
[New LWP 3404]
[New LWP 3402]
[New LWP 3403]
[New LWP 3406]
[New LWP 3405]
[New LWP 3409]
[New LWP 3416]
[New LWP 3401]
[New LWP 3410]
[New LWP 3419]
[New LWP 3413]
[New LWP 3490]
[New LWP 3408]
[New LWP 3493]
[New LWP 3414]
[New LWP 3412]
[New LWP 3417]
[New LWP 3411]
[New LWP 3415]
[New LWP 3407]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
-Type <RET> for more, q to quit, c to continue without paging-
Core was generated by `/usr/sbin/mariadbd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
[Current thread is 1 (Thread 0x7fa85807d700 (LWP 3437))]
(gdb)


j25_nnz_stats_player | CREATE TABLE `j25_nnz_stats_player` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`champdata_id` int(10) unsigned DEFAULT NULL,
`key` varchar(45) DEFAULT NULL,
`first_name` varchar(64) DEFAULT NULL,
`last_name` varchar(64) DEFAULT NULL,
`maiden_name` varchar(64) DEFAULT NULL,
`show_maiden_name` tinyint(3) unsigned DEFAULT 0,
`height` varchar(15) DEFAULT NULL,
`date_of_birth` varchar(15) DEFAULT NULL,
`occupation` varchar(45) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `key_UNIQUE` (`key`),
KEY `player_key_idx` (`key`)
) ENGINE=InnoDB AUTO_INCREMENT=452 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci |

j25_nnz_stats_player_team | CREATE TABLE `j25_nnz_stats_player_team` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`player_id` int(10) unsigned DEFAULT NULL,
`team_id` int(10) unsigned DEFAULT NULL,
`player_number` int(10) unsigned DEFAULT NULL,
`captain_number` int(10) unsigned DEFAULT NULL,
`debut_date` datetime DEFAULT NULL,
`debut_against_team_id` int(10) unsigned DEFAULT NULL,
`retirement_date` datetime DEFAULT NULL,
`selected_date` datetime DEFAULT NULL,
`main_positions` varchar(45) DEFAULT NULL,
`active` tinyint(3) unsigned DEFAULT NULL,
`games_played_offset` int(10) unsigned NOT NULL DEFAULT 0,
`career_years_override` varchar(128) NOT NULL DEFAULT '',
`captain_years_override` varchar(128) DEFAULT NULL,
`headshot_image` varchar(255) DEFAULT NULL,
`action_image` varchar(255) DEFAULT NULL,
`profile_image` varchar(255) DEFAULT NULL,
`bio_content` text DEFAULT NULL,
`published` tinyint(3) unsigned NOT NULL DEFAULT 1,
`featured` tinyint(3) unsigned NOT NULL DEFAULT 0,
PRIMARY KEY (`id`),
KEY `FK_stats_player_team_idx` (`player_id`),
KEY `FK_stats_team_player_idx` (`team_id`),
KEY `FK_stats_debut_against_idx` (`debut_against_team_id`),
CONSTRAINT `FK_stats_debut_against` FOREIGN KEY (`debut_against_team_id`) REFERENCES `j25_nnz_stats_team` (`id`) ON UPDATE CASCADE,
CONSTRAINT `FK_stats_player_team` FOREIGN KEY (`player_id`) REFERENCES `j25_nnz_stats_player` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION,
CONSTRAINT `FK_stats_team_player` FOREIGN KEY (`team_id`) REFERENCES `j25_nnz_stats_team` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=718 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci |

j25_nnz_stats_team | CREATE TABLE `j25_nnz_stats_team` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`champdata_id` int(10) unsigned DEFAULT NULL,
`league_id` int(10) unsigned DEFAULT NULL,
`key` varchar(45) DEFAULT NULL,
`name` varchar(45) DEFAULT NULL,
`country_id` int(10) unsigned DEFAULT NULL,
`abbreviation` varchar(10) DEFAULT NULL,
`logo_image` varchar(255) DEFAULT NULL,
`logo_image_on_dark` varchar(255) DEFAULT NULL,
`facebook` varchar(255) DEFAULT NULL,
`twitter` varchar(255) DEFAULT NULL,
`instagram` varchar(255) DEFAULT NULL,
`youtube` varchar(255) DEFAULT NULL,
`website` varchar(255) DEFAULT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `key_UNIQUE` (`key`),
KEY `FK_team_country_idx` (`country_id`),
KEY `team_key_idx` (`key`),
CONSTRAINT `FK_team_country` FOREIGN KEY (`country_id`) REFERENCES `j25_nnz_stats_country` (`id`) ON UPDATE CASCADE
) ENGINE=InnoDB AUTO_INCREMENT=160 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci |



 Comments   
Comment by Daniel Black [ 2023-05-16 ]

Thanks for the bug report pccnz especially with the good stack trace.

It looks identical to the MDEV-31240/MDEV-31241 issues that have been fixed after the release.

I've cherry picked the fix to 10.11 and some packages are now available from https://ci.mariadb.org/35089/.

Adding them as a Debian repository:

# echo "deb [trusted=yes] https://ci.mariadb.org/35089/amd64-debian-11-deb-autobake/debs ./" >> /etc/apt/sources.list.d/mariadb.list

RPM files have an included MariaDB.repo file at the top level that can be used.

Comment by Phil [ 2023-05-17 ]

Hi Dan

Happy to confirm this patch has fixed the issue for us. Thank you for such a quick response!

Cheers
Phil

Generated at Thu Feb 08 10:22:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.