Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Duplicate
-
11.3.1
-
None
Description
Hi,
We had a few MariaDB crashes recently (Jan 14 00:15, Jan 15 15:32). They seem to be caused by the same query in the end. The load on the server doesn't seem to be very high at the time of the crash. I'm attaching the full err log from /var/log/messages (slightly anonymized, to satisfy our data protection policy).
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: 240114 7:05:25 [ERROR] mysqld got signal 11 ;
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Sorry, we probably made a mistake, and this is a bug.
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Your assistance in bug reporting will enable us to fix this for the next release.
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: We will try our best to scrape up some info that will hopefully help
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: diagnose the problem, but since we have already crashed,
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: something is definitely wrong and this may fail.
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Server version: 11.3.1-MariaDB-log source revision: 67a0224a3e0073c4e083a6f7e24380251bcb2361
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: key_buffer_size=134217728
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: read_buffer_size=131072
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: max_used_connections=19
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: max_threads=102
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: thread_count=19
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: It is possible that mysqld could use up to
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 53624203 K bytes of memory
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Hope that's ok; if not, decrease some variables in the equation.
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Thread pointer: 0x7fcd68000c58
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Attempting backtrace. You can use the following information to find out
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: where mysqld died. If you see no messages after this, something went
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: terribly wrong...
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: stack_bottom = 0x7fcdfc106bf8 thread_stack 0x49000
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55731c5d601e]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handle_fatal_signal+0x468)[0x55731c0b7ba8]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x54db0)[0x7fd4c1e54db0]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handler_rowid_filter_check+0x54)[0x55731c0c5d74]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xe5bfe4)[0x55731c460fe4]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xe5f817)[0x55731c464817]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xd9a188)[0x55731c39f188]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN7handler10ha_rnd_posEPhS0_+0x225)[0x55731c0bfa45]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z16rr_from_pointersP11READ_RECORD+0x38)[0x55731bd74888]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete16do_table_deletesEP5TABLEP9SORT_INFOb+0x94)[0x55731bdfb034]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete10do_deletesEv+0x84)[0x55731bdfb2f4]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN12multi_delete8send_eofEv+0x71)[0x55731bdfb3d1]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN4JOIN10exec_innerEv+0xf4d)[0x55731bec2d9d]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN4JOIN4execEv+0x3b)[0x55731bec323b]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN11Sql_cmd_dml13execute_innerEP3THD+0x76)[0x55731bec3366]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN14Sql_cmd_delete13execute_innerEP3THD+0x26)[0x55731bdfacc6]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_ZN11Sql_cmd_dml7executeEP3THD+0xdb)[0x55731be7e2bb]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x490e)[0x55731be4782e]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x211)[0x55731be48781]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x1685)[0x55731be4b175]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x134)[0x55731be4cdc4]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3bf)[0x55731bf71b3f]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x55731bf71e8d]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /usr/sbin/mariadbd(+0xcd90c2)[0x55731c2de0c2]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x9f802)[0x7fd4c1e9f802]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: /lib64/libc.so.6(+0x3f450)[0x7fd4c1e3f450]
|
Jan 14 07:05:25 teamcity-db-server mariadbd[294867]: Trying to get some variables.
|
...
|
The DB stores data for TeamCity CI/CD; we've used it for a long time and this seems to be the first time we face these issues. Could be related to TeamCity upgrade, or MariaDB upgrade, or a combination of both. We'll also reach to JetBrains (the TeamCity vendor) about the issue. At the time of the crash, TeamCity is performing the regular cleanup (it's configured to run once an hour but the crash occurs only sporadically).
The server.cnf:
[server]
|
|
|
[mysqld]
|
query_cache_type = OFF
|
query_cache_size = 0
|
innodb_flush_method = O_DIRECT
|
innodb_log_files_in_group = 2
|
innodb_buffer_pool_size = 26G
|
innodb_log_file_size = 6G
|
innodb_log_files_in_group = 2
|
innodb_flush_log_at_trx_commit = 2
|
innodb_data_file_path = ibdata1:12M:autoextend:max:5G
|
tmp_table_size = 64M
|
max_heap_table_size = 64M
|
sort_buffer_size = 512M
|
read_rnd_buffer_size = 256M
|
join_buffer_size = 512M
|
performance_schema = ON
|
thread_cache_size = 200
|
log-bin
|
binlog-format = mixed
|
max_binlog_size = 200M
|
expire_logs_days = 7
|
innodb_purge_threads = 1
|
table_definition_cache = 500
|
max_connections = 100
|
log_queries_not_using_indexes = 1
|
table_open_cache = 2000
|
|
|
[galera]
|
|
|
[embedded]
|
|
|
[mariadb]
|
|
|
[mariadb-10.7]
|
or
$ mariadbd --print-defaults
|
|
|
mariadbd would have been started with the following arguments:
|
--query_cache_type=OFF --query_cache_size=0 --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=2 --innodb_buffer_pool_size=26G --innodb_log_file_size=6G --innodb_log_files_in_group=2 --innodb_flush_log_at_trx_commit=2 --innodb_data_file_path=ibdata1:12M:autoextend:max:5G --tmp_table_size=64M --max_heap_table_size=64M --sort_buffer_size=512M --read_rnd_buffer_size=256M --join_buffer_size=512M --performance_schema=ON --thread_cache_size=200 --log-bin --binlog-format=mixed --max_binlog_size=200M --expire_logs_days=7 --innodb_purge_threads=1 --table_definition_cache=500 --max_connections=100 --log_queries_not_using_indexes=1 --table_open_cache=2000
|
Attachments
Issue Links
- duplicates
-
-
- In Review
-