|
2019-10-19 12:37:01 0x7f68f5957700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx
0trx.cc line 1386
InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
191019 12:37:01 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Server version: 10.4.8-MariaDB-log
key_buffer_size=67108864
read_buffer_size=2097152
max_used_connections=209
max_threads=2002
thread_count=220
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 20615196 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f67680465d8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f68f5956cc0 thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x56300687983e]
/usr/sbin/mysqld(handle_fatal_signal+0x30f)[0x56300630fe0f]
/lib64/libpthread.so.0(+0xf5d0)[0x7f6f8ffbd5d0]
/lib64/libc.so.6(gsignal+0x37)[0x7f6f8e2902c7]
/lib64/libc.so.6(abort+0x148)[0x7f6f8e2919b8]
/usr/sbin/mysqld(+0x5a0542)[0x56300600b542]
/usr/sbin/mysqld(+0xb8905e)[0x5630065f405e]
/usr/sbin/mysqld(+0xb89217)[0x5630065f4217]
/usr/sbin/mysqld(+0xb89554)[0x5630065f4554]
/usr/sbin/mysqld(+0xa496fa)[0x5630064b46fa]
/usr/sbin/mysqld(+0xa498eb)[0x5630064b48eb]
/usr/sbin/mysqld(+0xa49b7b)[0x5630064b4b7b]
/usr/sbin/mysqld(+0xa524a1)[0x5630064bd4a1]
/usr/sbin/mysqld(_ZN7handler16ha_external_lockEP3THDi+0x9f)[0x56300631b55f]
/usr/sbin/mysqld(+0x9975fa)[0x5630064025fa]
/usr/sbin/mysqld(_Z24mysql_unlock_read_tablesP3THDP13st_mysql_lock+0x68)[0x563006402de8]
/usr/sbin/mysqld(_ZN4JOIN9join_freeEv+0x169)[0x5630061454c9]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb3f)[0x56300615f46f]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x56300615f723]
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x56300615da26]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1d7)[0x56300615e597]
/usr/sbin/mysqld(+0x58dd2a)[0x563005ff8d2a]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4591)[0x563006105641]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x24b)[0x56300610a51b]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1c3e)[0x56300610d39e]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x56300610eaac]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1fa)[0x5630061ec18a]
/usr/sbin/mysqld(handle_one_connection+0x3d)[0x5630061ec26d]
/usr/sbin/mysqld(+0xdc124d)[0x56300682c24d]
/lib64/libpthread.so.0(+0x7dd5)[0x7f6f8ffb5dd5]
/lib64/libc.so.6(clone+0x6d)[0x7f6f8e35802d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f6768055810): SELECT MAX(prefixes.prefix) as prefix, MAX(prj_card.title) as title, MAX(prefixes.pr_id) as pr_id FROM prj_card LEFT JOIN prefixes ON prefixes.card=prj_card
.card WHERE prj_card.card='n_loyalty'
Connection ID (thread ID): 381119
Status: NOT_KILLED
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_co
ndition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table
_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bk
a=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition
_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 63375 63375 processes
Max open files 16364 16364 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63375 63375 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
Core pattern: core
|
|
@magnum72 Consider SET GLOBAL read_buffer_size=DEFAULT; to reduce RAM required per connection. Reconnect to instance. Try your query. Possible positive side effect, reduction in handler_read_next RPS and reduction in time required to complete most queries.
|
|
Magnum72, we have not seen anything like this in internal testing. A somewhat repeatable test case would be useful. If you had a support contract with us, we could analyze it deeper.
The reported version MariaDB 10.4.8 includes a fix of MDEV-15326, which was a race condition that was introduced in the InnoDB transaction commit code in MySQL 5.7. I do not remember seeing and issues in our internal testing in this area since then.
|
|
I saw this assertion failure several times in August, mostly on 10.4.7-based trees, once on 10.3 (between 10.3.17 and 10.3.18 releases) and once on 10.5 development branch of that time. I tried to reproduce it for a while, but it stopped happening before I could make any progress. It roughly coincides with the timing for MDEV-15326 fix, or maybe there was something else fixed shortly before. I would certainly recommend trying a newer version.
|
|
We are currently testing our application with MariaDB 10.4.12 and we are experiencing this exact crash. I don't have steps to reproduce it yet, but it is happening pretty consistent in our environment.
|
|
Same, no way to reproduce yet.
Mar 11 09:43:22 golinux mysqld[123323]: 2020-03-11 09:43:22 0x7f5e41d31700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.8/storage/innobase/trx/trx0trx.cc line 1386
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: We intentionally generate a memory trap.
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: If you get repeated assertion failures or crashes, even
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: immediately after the mysqld startup, there may be
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: about forcing recovery.
|
Mar 11 09:43:22 golinux mysqld[123323]: 200311 9:43:22 [ERROR] mysqld got signal 6 ;
|
Mar 11 09:43:22 golinux mysqld[123323]: This could be because you hit a bug. It is also possible that this binary
|
Mar 11 09:43:22 golinux mysqld[123323]: or one of the libraries it was linked against is corrupt, improperly built,
|
Mar 11 09:43:22 golinux mysqld[123323]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Mar 11 09:43:22 golinux mysqld[123323]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Mar 11 09:43:22 golinux mysqld[123323]: We will try our best to scrape up some info that will hopefully help
|
Mar 11 09:43:22 golinux mysqld[123323]: diagnose the problem, but since we have already crashed,
|
Mar 11 09:43:22 golinux mysqld[123323]: something is definitely wrong and this may fail.
|
Mar 11 09:43:22 golinux mysqld[123323]: Server version: 10.4.8-MariaDB-1:10.4.8+maria~buster-log
|
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size=83886080
|
Mar 11 09:43:22 golinux mysqld[123323]: read_buffer_size=1048576
|
Mar 11 09:43:22 golinux mysqld[123323]: max_threads=3002
|
Mar 11 09:43:22 golinux mysqld[123323]: thread_count=1398
|
Mar 11 09:43:22 golinux mysqld[123323]: It is possible that mysqld could use up to
|
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9377822 K bytes of memory
|
Mar 11 09:43:22 golinux mysqld[123323]: Hope that's ok; if not, decrease some variables in the equation.
|
Mar 11 09:43:22 golinux mysqld[123323]: Thread pointer: 0x7f57cc0265d8
|
Mar 11 09:43:22 golinux mysqld[123323]: Attempting backtrace. You can use the following information to find out
|
Mar 11 09:43:22 golinux mysqld[123323]: where mysqld died. If you see no messages after this, something went
|
Mar 11 09:43:22 golinux mysqld[123323]: terribly wrong...
|
Mar 11 09:43:22 golinux mysqld[123323]: stack_bottom = 0x7f5e41d30dd8 thread_stack 0x40000
|
Mar 11 09:43:23 golinux mysqld[123323]: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
|
|
|
If you will get your config to these values, you may be close to reproducing.
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size=83886080
Mar 11 09:43:22 golinux mysqld[123323]: read_buffer_size=1048576
Mar 11 09:43:22 golinux mysqld[123323]: max_threads=3002 in your config - Thread_cache_size and thread_pool_max_threads
Mar 11 09:43:22 golinux mysqld[123323]: thread_count=1398 not sure which global variable controls this limit - looks like threads_connected count to me
Mar 11 09:43:22 golinux mysqld[123323]: It is possible that mysqld could use up to
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9377822 K bytes of memory
Just a tip.
Thanks,
Wilson Hauck
www.mysqlservertuning.com where we 'Make Every Moment Count' by reducing WAIT time.
205-981-0190
From: "sysadmin (Jira)" <jira@mariadb.org>
Date: Wed, Mar 11, 2020 4:50 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146291#comment-146291 ]
sysadmin commented on MDEV-20870:
---------------------------------
Same, no way to reproduce yet.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: 2020-03-11 09:43:22 0x7f5e41d31700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.8/storage/innobase/trx/trx0trx.cc line 1386
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 11 09:43:22 golinux mysqld[123323]: InnoDB: about forcing recovery.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: 200311 9:43:22 [ERROR] mysqld got signal 6 ;
|
|
Mar 11 09:43:22 golinux mysqld[123323]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 11 09:43:22 golinux mysqld[123323]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 11 09:43:22 golinux mysqld[123323]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 11 09:43:22 golinux mysqld[123323]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 11 09:43:22 golinux mysqld[123323]: diagnose the problem, but since we have already crashed,
|
|
Mar 11 09:43:22 golinux mysqld[123323]: something is definitely wrong and this may fail.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: Server version: 10.4.8-MariaDB-1:10.4.8+maria~buster-log
|
|
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size=83886080
|
|
Mar 11 09:43:22 golinux mysqld[123323]: read_buffer_size=1048576
|
|
Mar 11 09:43:22 golinux mysqld[123323]: max_threads=3002
|
|
Mar 11 09:43:22 golinux mysqld[123323]: thread_count=1398
|
|
Mar 11 09:43:22 golinux mysqld[123323]: It is possible that mysqld could use up to
|
|
Mar 11 09:43:22 golinux mysqld[123323]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9377822 K bytes of memory
|
|
Mar 11 09:43:22 golinux mysqld[123323]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 11 09:43:22 golinux mysqld[123323]: Thread pointer: 0x7f57cc0265d8
|
|
Mar 11 09:43:22 golinux mysqld[123323]: Attempting backtrace. You can use the following information to find out
|
|
Mar 11 09:43:22 golinux mysqld[123323]: where mysqld died. If you see no messages after this, something went
|
|
Mar 11 09:43:22 golinux mysqld[123323]: terribly wrong...
|
|
Mar 11 09:43:22 golinux mysqld[123323]: stack_bottom = 0x7f5e41d30dd8 thread_stack 0x40000
|
|
Mar 11 09:43:23 golinux mysqld[123323]: *** buffer overflow detected ***: /usr/sbin/mysqld terminated
|
|
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
I can confirm this is also a problem in 10.4.12:
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
debian : 10.3
|
kernel : 4.19.67-2+deb10u2
|
mem : 515733 MB.
|
disk : 2.6T (SSD in raid 10)
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
[11:07:33][root@<myservername>(6)]:/data
|
(130)#: dpkg -l|grep mariadb
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
Output of GDB where from the core dump:
(gdb) where
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:181
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
at fortify_fail.c:28
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
#11 <signal handler called>
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
--Type <RET> for more, q to quit, c to continue without paging--
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=
|
0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
(gdb) where
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:181
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
at fortify_fail.c:28
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
#11 <signal handler called>
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
--Type <RET> for more, q to quit, c to continue without paging--
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0,
|
having=0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
|
|
Update: we got multiple crashes a day.
i've uploaded a core dump on ftp.mariadb.com/uploads/mdev-20807-rw.tar.gz
extracted file size is: 40GB
|
|
Paul,
Anytime you use 80G for key_buffer_size (which is a per connection request) you should expect failures.
Wilson Hauck
www.mysqlservertuning.com where we 'Make Every Moment Count' by reducing WAIT time.
From: "Paul Veldema (Jira)" <jira@mariadb.org>
Date: Tue, Mar 17, 2020 5:21 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146874#comment-146874 ]
Paul Veldema commented on MDEV-20870:
-------------------------------------
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
sysadmin,
Anytime you use 64M for key_buffer_size (which is a per connection request) you should expect failures.
Wilson Hauck
www.mysqlservertuning.com where we 'Make Every Moment Count' by reducing WAIT time.
From: "sysadmin (Jira)" <jira@mariadb.org>
Date: Tue, Mar 17, 2020 5:52 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146881#comment-146881 ]
sysadmin commented on MDEV-20870:
---------------------------------
Update: we got multiple crashes a day.
i've uploaded a core dump on ftp.mariadb.com/uploads/mdev-20807-rw.tar.gz
extracted file size is: 40GB
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
Paul, correction
Anytime you use 80M for key_buffer_size (which is a per connection request) you should expect failures.
Wilson Hauck
www.mysqlservertuning.com where we 'Make Every Moment Count' by reducing WAIT time.
From: "Paul Veldema (Jira)" <jira@mariadb.org>
Date: Tue, Mar 17, 2020 5:21 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146874#comment-146874 ]
Paul Veldema commented on MDEV-20870:
-------------------------------------
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
From: "Paul Veldema (Jira)" <jira@mariadb.org>
Date: Tue, Mar 17, 2020 6:18 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146874#comment-146874 ]
Paul Veldema edited comment on MDEV-20870 at 3/17/20 11:17 AM:
---------------------------------------------------------------
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
Output of GDB where from the core dump:
|
|
(gdb) where
|
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated
|
|
") at ../sysdeps/posix/libc_fatal.c:181
|
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
|
at fortify_fail.c:28
|
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
|
#11 <signal handler called>
|
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
|
--Type <RET> for more, q to quit, c to continue without paging--
|
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=
|
|
0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
(gdb) where
|
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated
|
|
") at ../sysdeps/posix/libc_fatal.c:181
|
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
|
at fortify_fail.c:28
|
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
|
#11 <signal handler called>
|
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
|
--Type <RET> for more, q to quit, c to continue without paging--
|
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0,
|
|
having=0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
|
|
was (Author: JIRAUSER41020):
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
Paul,
Anytime you use 80M for key_buffer_size (which is a per connection request) you should expect failures.
Wilson Hauck
www.mysqlservertuning.com where we 'Make Every Moment Count' by reducing WAIT time.
From: "Paul Veldema (Jira)" <jira@mariadb.org>
Date: Tue, Mar 17, 2020 6:18 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146874#comment-146874 ]
Paul Veldema edited comment on MDEV-20870 at 3/17/20 11:17 AM:
---------------------------------------------------------------
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
Output of GDB where from the core dump:
|
|
(gdb) where
|
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated
|
|
") at ../sysdeps/posix/libc_fatal.c:181
|
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
|
at fortify_fail.c:28
|
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
|
#11 <signal handler called>
|
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
|
--Type <RET> for more, q to quit, c to continue without paging--
|
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0, having=
|
|
0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
(gdb) where
|
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#1 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#2 0x00007f4e042da508 in __libc_message (action=<optimized out>, fmt=fmt@entry=0x7f4e043e507b "*** %s ***: %s terminated
|
|
") at ../sysdeps/posix/libc_fatal.c:181
|
|
#3 0x00007f4e0436b80d in __GI___fortify_fail_abort (need_backtrace=need_backtrace@entry=true, msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected")
|
|
at fortify_fail.c:28
|
|
#4 0x00007f4e0436b841 in __GI___fortify_fail (msg=msg@entry=0x7f4e043e4ff8 "buffer overflow detected") at fortify_fail.c:44
|
|
#5 0x00007f4e04369940 in __GI___chk_fail () at chk_fail.c:28
|
|
#6 0x00007f4e0436b737 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
|
|
#7 0x00005589305597dd in my_addr_resolve (ptr=<optimized out>, loc=loc@entry=0x7eec3d425e40) at ./mysys/my_addr_resolve.c:234
|
|
#8 0x0000558930543743 in print_with_addr_resolve (n=<optimized out>, addrs=0x7eec3d425e60) at ./mysys/stacktrace.c:254
|
|
#9 my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=<optimized out>, silent=silent@entry=0 '\000') at ./mysys/stacktrace.c:273
|
|
#10 0x000055893003923d in handle_fatal_signal (sig=6) at ./sql/signal_handler.cc:206
|
|
#11 <signal handler called>
|
|
#12 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
|
#13 0x00007f4e04283535 in __GI_abort () at abort.c:79
|
|
#14 0x000055892fd2e1d5 in ut_dbg_assertion_failed (expr=expr@entry=0x5589307577c0 "UT_LIST_GET_LEN(trx->lock.trx_locks) == 0",
|
|
file=file@entry=0x5589307575f0 "/home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc", line=line@entry=1365)
|
|
at ./storage/innobase/ut/ut0dbg.cc:60
|
|
#15 0x000055892fd2d8b6 in trx_commit_in_memory (mtr=0x0, trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1365
|
|
#16 trx_commit_low (trx=0x7f4e01561978, mtr=0x0) at ./storage/innobase/trx/trx0trx.cc:1618
|
|
#17 0x00005589302f1a8a in trx_commit (trx=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1642
|
|
#18 0x00005589302f1c63 in trx_commit_for_mysql (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/trx/trx0trx.cc:1788
|
|
#19 0x00005589301d30f0 in innobase_commit_low (trx=trx@entry=0x7f4e01561978) at ./storage/innobase/handler/ha_innodb.cc:4385
|
|
#20 0x00005589301d33a4 in innobase_commit_ordered_2 (trx=trx@entry=0x7f4e01561978, thd=thd@entry=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4507
|
|
#21 0x00005589301dc981 in innobase_commit (commit_trx=true, hton=<optimized out>, thd=0x7ee98c02ef58) at ./storage/innobase/handler/ha_innodb.cc:4623
|
|
#22 ha_innobase::external_lock (this=<optimized out>, thd=0x7ee98c02ef58, lock_type=2) at ./storage/innobase/handler/ha_innodb.cc:15799
|
|
#23 0x0000558930045424 in handler::ha_external_lock (this=0x7ee7840be940, thd=thd@entry=0x7ee98c02ef58, lock_type=lock_type@entry=2) at ./sql/handler.cc:6410
|
|
--Type <RET> for more, q to quit, c to continue without paging--
|
|
#24 0x00005589301248bc in unlock_external (count=<optimized out>, table=0x7ee98c0ded10, thd=0x7ee98c02ef58) at ./sql/lock.cc:709
|
|
#25 mysql_unlock_read_tables (thd=0x7ee98c02ef58, sql_lock=0x7ee98c0decf0) at ./sql/lock.cc:485
|
|
#26 0x000055892fe6f42a in JOIN::join_free (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:13698
|
|
#27 0x000055892fe8c8e4 in do_select (procedure=<optimized out>, join=0x7ee98c0ded40) at ./sql/sql_select.cc:19923
|
|
#28 JOIN::exec_inner (this=0x7ee98c0ded40) at ./sql/sql_select.cc:4452
|
|
#29 0x000055892fe8cba3 in JOIN::exec (this=this@entry=0x7ee98c0ded40) at ./sql/sql_select.cc:4234
|
|
#30 0x000055892fe8b051 in mysql_select (thd=0x7ee98c02ef58, tables=0x7ee98c0dd3b8, wild_num=1, fields=..., conds=<optimized out>, og_num=0, order=0x0, group=0x0,
|
|
having=0x0, proc_param=0x0, select_options=2147748608, result=0x7ee98c0ded18, unit=0x7ee98c032cc0, select_lex=0x7ee98c0dcd88) at ./sql/sql_select.cc:4666
|
|
#31 0x000055892fe8b931 in handle_select (thd=thd@entry=0x7ee98c02ef58, lex=lex@entry=0x7ee98c032c00, result=result@entry=0x7ee98c0ded18,
|
|
setup_tables_done_option=setup_tables_done_option@entry=0) at ./sql/sql_select.cc:408
|
|
#32 0x000055892fe24a7c in execute_sqlcom_select (thd=0x7ee98c02ef58, all_tables=0x7ee98c0dd3b8) at ./sql/sql_parse.cc:6360
|
|
#33 0x000055892fe2ce1d in mysql_execute_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:3899
|
|
#34 0x000055892fe343c9 in mysql_parse (thd=thd@entry=0x7ee98c02ef58, rawbuf=<optimized out>, length=145, parser_state=parser_state@entry=0x7eec3d42a0f0,
|
|
is_com_multi=is_com_multi@entry=false, is_next_command=is_next_command@entry=false) at ./sql/sql_parse.cc:7901
|
|
#35 0x000055892fe36556 in dispatch_command (command=COM_QUERY, thd=0x7ee98c02ef58, packet=<optimized out>, packet_length=<optimized out>, is_com_multi=<optimized out>,
|
|
is_next_command=<optimized out>) at ./sql/sql_class.h:1168
|
|
#36 0x000055892fe380ae in do_command (thd=0x7ee98c02ef58) at ./sql/sql_parse.cc:1359
|
|
#37 0x000055892ff18bfe in do_handle_one_connection (connect=connect@entry=0x558b19299738) at ./sql/sql_connect.cc:1412
|
|
#38 0x000055892ff18ccd in handle_one_connection (arg=0x558b19299738) at ./sql/sql_connect.cc:1316
|
|
#39 0x00007f4e04e45fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
|
|
#40 0x00007f4e0435a4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
|
|
was (Author: JIRAUSER41020):
I can confirm this is also a problem in 10.4.12:
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 2020-03-17 09:55:35 0x7f8072605700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.12/storage/innobase/trx/trx0trx.cc line 1365
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Failing assertion: UT_LIST_GET_LEN(trx->lock.trx_locks) == 0
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: We intentionally generate a memory trap.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: If you get repeated assertion failures or crashes, even
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: immediately after the mysqld startup, there may be
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: InnoDB: about forcing recovery.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: 200317 9:55:35 [ERROR] mysqld got signal 6 ;
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: This could be because you hit a bug. It is also possible that this binary
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or one of the libraries it was linked against is corrupt, improperly built,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: We will try our best to scrape up some info that will hopefully help
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: diagnose the problem, but since we have already crashed,
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: something is definitely wrong and this may fail.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Server version: 10.4.12-MariaDB-1:10.4.12+maria~buster-log
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size=83886080
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: read_buffer_size=1048576
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_used_connections=1515
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: max_threads=3002
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: thread_count=1387
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: It is possible that mysqld could use up to
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9378221 K bytes of memory
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Hope that's ok; if not, decrease some variables in the equation.
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Thread pointer: 0x7f8268301328
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: Attempting backtrace. You can use the following information to find out
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: where mysqld died. If you see no messages after this, something went
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: terribly wrong...
|
|
Mar 17 09:55:35 <myservername> mysqld[108469]: stack_bottom = 0x7f8072604dd8 thread_stack 0x40000
|
|
|
|
This issues affected versions should be updated to include 10.4.12
Some more info on the affected server:
|
|
cpu : 1 x 32 core 64 bit HTx2 AMD EPYC 7551P 32-Core Processor
|
|
debian : 10.3
|
|
kernel : 4.19.67-2+deb10u2
|
|
mem : 515733 MB.
|
|
disk : 2.6T (SSD in raid 10)
|
|
There is only a kernel update available, for the rest all packages are up-t--date.
Here the current mariadb packages:
|
|
[11:07:33][root@<myservername>(6)]:/data
|
|
(130)#: dpkg -l|grep mariadb
|
|
ii libmariadb3:amd64 1:10.4.12+maria~buster amd64 MariaDB database client library
|
|
ii mariadb-backup 1:10.4.12+maria~buster amd64 Backup tool for MariaDB server
|
|
ii mariadb-client-10.4 1:10.4.12+maria~buster amd64 MariaDB database client binaries
|
|
ii mariadb-client-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core client binaries
|
|
ii mariadb-common 1:10.4.12+maria~buster all MariaDB database common files (e.g. /etc/mysql/conf.d/mariadb.cnf)
|
|
ii mariadb-server-10.4 1:10.4.12+maria~buster amd64 MariaDB database server binaries
|
|
ii mariadb-server-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-10.4
|
|
ii mariadb-server-core-10.4 1:10.4.12+maria~buster amd64 MariaDB database core server files
|
|
ii mariadb-server-core-10.4-dbgsym 1:10.4.12+maria~buster amd64 debug symbols for mariadb-server-core-10.4
|
|
We get these packages from here: mirror.i3d.net/pub/mariadb/repo/10.4/debian
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
Hello Wilson.
actually the key_buffer_size is not per connection.
You can see that in the calculation in the stacktrace (80MB + mem calculation per connection times max connections)
You can also see it here that is is a shared myisam setting:
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_key_buffer_size
All in all the per connection buffers plus the 80MB take 9GB of memory according to the calculation in the stacktrace.
Small compared to about 100GB of free memory (out of 500GB) available (not allocated for innodb)
with regards
Paul
|
|
Paul,
You are correct. My apologies.
According to this about 20GB would be needed to run.
Could you ask the person submitting the failure to post
SHOW CREATE TABLE call_list;
to allow verification of column datatypes?
Good luck with pinpointing the cause.
Thank you for your service.
Wilson Hauck
From: "Paul Veldema (Jira)" <jira@mariadb.org>
Date: Wed, Mar 18, 2020 8:30 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=146998#comment-146998 ]
Paul Veldema edited comment on MDEV-20870 at 3/18/20 1:29 PM:
--------------------------------------------------------------
Hello Wilson.
actually the key_buffer_size is not per connection.
You can see that in the calculation in the stacktrace (80MB + mem calculation per connection times max connections)
You can also see it here that is is a shared myisam setting:
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_key_buffer_size
All in all the per connection buffers plus the 80MB take 9GB of memory according to the calculation in the stacktrace.
Small compared to about 100GB of free memory (out of 500GB) available (not allocated for innodb)
with regards
Paul
was (Author: JIRAUSER41020):
Hello Wilson.
actually the key_buffer_size is not per connection.
You can see that in the calculation in the stacktrace (80MB + mem calculation per connection times max connections)
You can also see it here that is is a shared myisam setting:
https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_key_buffer_size
All in all the per connection buffers plus the 80MB take 9GB of memory according to the calculation in the stacktrace.
Small compared to about 100Gb of free memory (out of 500GB) available (not allocated for innodb)
with regards
Paul
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|
|
We suspect the following setting to be part of the crash problem (one of the only changes besides the version upgrade to Mariadb 10.4):
innodb_compression_algorithm = lz4
We reverted to an older version of mariadb with the default zlib compression to be stable again.
We will in the near future set a mariadb 10.4 server live with the same zlib default and see if that will run without crashes (alas only way to check for now).
With Regards
Paul
|
|
I can confirm that this issue still persists in version 10.5.4 Could somebody update the affected versions?
|
|
paulvrw, thank you for producing a resolved stack trace for this. In GDB, could you please execute the following statement in the stack frame where the assertion is failing?
print trx->lock.trx_locks
|
We expect the transaction to hold no locks once it has been committed, but something is violating it. It would be good to preserve the core dump, because we may need more information from it in order to be able to resolve the issue.
Is anyone using innodb_lock_schedule_algorithm=VATS? That is known to be buggy (see MDEV-16664).
Because nobody mentioned Galera in this ticket, I do not think that this could be related to MDEV-23101.
|
|
I think this just hit me with 10.5.5 .
This has been closed, but it is still incomplete. Can anybody explain?
|
|
For analysis, please send email to mydatalinks@mysqlservertuning.com
with the following TEXT attachments after root login and from MySQL Command Prompt,
A) SHOW GLOBAL STATUS;
B) SHOW GLOBAL VARIABLES;
C) last 400 lines of your error log
RAM size, Cores, any SSD or NVME devices used for storage?
from OS Command Prompt,
htop or top (first page minimum)
ulimit -a for a Linux/Unix list of limits,
iostat -xm 5 3for IOPS by device and core/cpu count, Thanks,
Wilson Hauck
www.mysqlservertuning.com UTC -6 hrs Birmingham, AL USA
From: "Christian Hesse (Jira)" <jira@mariadb.org>
Date: Tue, Sep 8, 2020 7:19 am
To: wilson.hauck@mysqlservertuning.com
Subject: [JIRA] (MDEV-20870) InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.4.8/storage/innobase/trx/trx 0trx.cc line 1386
[ https://jira.mariadb.org/browse/MDEV-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=165405#comment-165405 ]
Christian Hesse commented on MDEV-20870:
----------------------------------------
I think this just hit me with 10.5.5 .
This has been closed, but it is still incomplete. Can anybody explain?
–
This message was sent by Atlassian Jira
(v8.5.1#805001)
|