[MDEV-33254] [ERROR] mysqld got signal 11 (..multi_delete..?) Created: 2024-01-16  Updated: 2024-01-16

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 11.3.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Petr Toman Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: None
Environment:

OpenStack VM machine 8vCPUx64GB
—
/etc/os-release:
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"

Attachments: HTML File tc-db--var-log-messages-filtered    

 Description   

Hi,

We had a few MariaDB crashes recently (Jan 14 00:15, Jan 15 15:32). They seem to be caused by the same query in the end. The load on the server doesn't seem to be very high at the time of the crash. I'm attaching the full err log from /var/log/messages (slightly anonymized, to satisfy our data protection policy).

Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: 240114  7:05:25 [ERROR] mysqld got signal 11 ;
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Sorry, we probably made a mistake, and this is a bug.
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Your assistance in bug reporting will enable us to fix this for the next release.
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: We will try our best to scrape up some info that will hopefully help
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: diagnose the problem, but since we have already crashed,
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: something is definitely wrong and this may fail.
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Server version: 11.3.1-MariaDB-log source revision: 67a0224a3e0073c4e083a6f7e24380251bcb2361
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: key_buffer_size=134217728
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: read_buffer_size=131072
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: max_used_connections=19
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: max_threads=102
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: thread_count=19
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: It is possible that mysqld could use up to
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 53624203 K  bytes of memory
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Hope that's ok; if not, decrease some variables in the equation.
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Thread pointer: 0x7fcd68000c58
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Attempting backtrace. You can use the following information to find out
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: where mysqld died. If you see no messages after this, something went
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: terribly wrong...
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: stack_bottom = 0x7fcdfc106bf8 thread_stack 0x49000
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55731c5d601e]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handle_fatal_signal+0x468)[0x55731c0b7ba8]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x54db0)[0x7fd4c1e54db0]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handler_rowid_filter_check+0x54)[0x55731c0c5d74]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xe5bfe4)[0x55731c460fe4]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xe5f817)[0x55731c464817]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xd9a188)[0x55731c39f188]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN7handler10ha_rnd_posEPhS0_+0x225)[0x55731c0bfa45]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z16rr_from_pointersP11READ_RECORD+0x38)[0x55731bd74888]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete16do_table_deletesEP5TABLEP9SORT_INFOb+0x94)[0x55731bdfb034]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete10do_deletesEv+0x84)[0x55731bdfb2f4]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete8send_eofEv+0x71)[0x55731bdfb3d1]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN4JOIN10exec_innerEv+0xf4d)[0x55731bec2d9d]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN4JOIN4execEv+0x3b)[0x55731bec323b]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN11Sql_cmd_dml13execute_innerEP3THD+0x76)[0x55731bec3366]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN14Sql_cmd_delete13execute_innerEP3THD+0x26)[0x55731bdfacc6]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN11Sql_cmd_dml7executeEP3THD+0xdb)[0x55731be7e2bb]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x490e)[0x55731be4782e]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x211)[0x55731be48781]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1685)[0x55731be4b175]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x134)[0x55731be4cdc4]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3bf)[0x55731bf71b3f]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x55731bf71e8d]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xcd90c2)[0x55731c2de0c2]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x9f802)[0x7fd4c1e9f802]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x3f450)[0x7fd4c1e3f450]
 
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Trying to get some variables.
 
...

The DB stores data for TeamCity CI/CD; we've used it for a long time and this seems to be the first time we face these issues. Could be related to TeamCity upgrade, or MariaDB upgrade, or a combination of both. We'll also reach to JetBrains (the TeamCity vendor) about the issue. At the time of the crash, TeamCity is performing the regular cleanup (it's configured to run once an hour but the crash occurs only sporadically).

The server.cnf:

[server]
 
 
 
[mysqld]
 
query_cache_type = OFF
 
query_cache_size = 0
 
innodb_flush_method = O_DIRECT
 
innodb_log_files_in_group = 2
 
innodb_buffer_pool_size = 26G
 
innodb_log_file_size = 6G
 
innodb_log_files_in_group = 2
 
innodb_flush_log_at_trx_commit = 2
 
innodb_data_file_path = ibdata1:12M:autoextend:max:5G
 
tmp_table_size = 64M
 
max_heap_table_size = 64M
 
sort_buffer_size = 512M
 
read_rnd_buffer_size = 256M
 
join_buffer_size = 512M
 
performance_schema = ON
 
thread_cache_size = 200
 
log-bin
 
binlog-format = mixed
 
max_binlog_size = 200M
 
expire_logs_days = 7
 
innodb_purge_threads = 1
 
table_definition_cache = 500
 
max_connections = 100
 
log_queries_not_using_indexes = 1
 
table_open_cache = 2000
 
 
 
[galera]
 
 
 
[embedded]
 
 
 
[mariadb]
 
 
 
[mariadb-10.7]

or

$ mariadbd --print-defaults
 
 
 
mariadbd would have been started with the following arguments:
 
--query_cache_type=OFF --query_cache_size=0 --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=2 --innodb_buffer_pool_size=26G --innodb_log_file_size=6G --innodb_log_files_in_group=2 --innodb_flush_log_at_trx_commit=2 --innodb_data_file_path=ibdata1:12M:autoextend:max:5G --tmp_table_size=64M --max_heap_table_size=64M --sort_buffer_size=512M --read_rnd_buffer_size=256M --join_buffer_size=512M --performance_schema=ON --thread_cache_size=200 --log-bin --binlog-format=mixed --max_binlog_size=200M --expire_logs_days=7 --innodb_purge_threads=1 --table_definition_cache=500 --max_connections=100 --log_queries_not_using_indexes=1 --table_open_cache=2000


Generated at Thu Feb 08 10:37:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.