Details
Major Description
I debugged the main.mysqltest failure that reliably occurs on kvm-asan. It looks like a bug in GCC 5.4.0’s ASAN implementation to me:
|
10.5 6fb59d525b7047c82e350d2b721bffc50431eb12 |
mysqltest: At line 1: Error parsing replace_regex "a"
|
|
|
Program received signal SIGABRT, Aborted.
|
0x00007ffff534a428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
|
(gdb) bt
|
#0 0x00007ffff534a428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
|
#1 0x00007ffff534c02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
|
#2 0x00007ffff6f17dc9 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#3 0x00007ffff6f0a769 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#4 0x00007ffff6f0f5d2 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#5 0x00007ffff6f17558 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#6 0x00007ffff6f18a38 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#7 0x00007ffff6f18bbe in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#8 0x00007ffff6f225a6 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#9 0x00007ffff534f36a in __cxa_finalize ()
|
from /lib/x86_64-linux-gnu/libc.so.6
|
#10 0x00007ffff6e875a3 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#11 0x00007ffffffbdae0 in ?? ()
|
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
|
Even if I set a breakpoint on __cxa_finalize () and rerun, I will not get more information. There is only that one thread:
Starting program: /home/buildbot/server/build/client/mysqltest --defaults-file=/home/buildbot/server/build/mysql-test/var/my.cnf --silent --tmpdir=/home/buildbot/server/build/mysql-test/var/tmp --character-sets-dir=/home/buildbot/server/sql/share/charsets --logdir=/home/buildbot/server/build/mysql-test/var/log --database=test --timer-file=/home/buildbot/server/build/mysql-test/var/log/timer < foo
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
mysqltest: At line 1: Error parsing replace_regex "a"
|
|
|
Breakpoint 1, 0x00007ffff534f2d0 in __cxa_finalize ()
|
from /lib/x86_64-linux-gnu/libc.so.6
|
(gdb) thr a a bt
|
|
|
Thread 1 (Thread 0x7ffff7fe0780 (LWP 21098)):
|
#0 0x00007ffff534f2d0 in __cxa_finalize ()
|
from /lib/x86_64-linux-gnu/libc.so.6
|
#1 0x00007ffff6e875a3 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
|
#2 0x00007ffffffbdae0 in ?? ()
|
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
|
Test setup:
cd build/mysql-test
|
echo 'echo $MYSQL_TEST;' > ../../mysql-test/main/m.test
|
echo --replace_regex a > foo
|
./mtr --manual-gdb main.m
|
gdb -cd /home/buildbot/server/mysql-test -x /home/buildbot/server/build/mysql-test/var/tmp/gdbinit.mysqld.1 /home/buildbot/server/build/sql/mysqld
|
Then, in gdb, type
run
|
# (hit ctrl-z to suspend the server process)
|
detach
|
quit
|
and finally start the mysqltest client in another gdb:
run /home/buildbot/server/build/client/mysqltest --defaults-file=/home/buildbot/server/build/mysql-test/var/my.cnf --silent --tmpdir=/home/buildbot/server/build/mysql-test/var/tmp --character-sets-dir=/home/buildbot/server/sql/share/charsets --logdir=/home/buildbot/server/build/mysql-test/var/log --database=test --timer-file=/home/buildbot/server/build/mysql-test/var/log/timer < foo
|
The arguments are from the $MYSQL_TEST, except for the < foo redirection.
Note: If I try to invoke ./mtr main.n on the following n.test file directly, then there is no SIGABRT reported:
--replace_regex a |
It will only report a failure like this:
main.n [ fail ]
|
Test ended at 2020-03-21 08:19:40
|
|
|
CURRENT_TEST: main.n
|
mysqltest: At line 1: Error parsing replace_regex "a"
|
There is nothing in any log files either:
|
grep -ri sanitizer var |
var/tmp/check-mysqld_1.result:HAVE_SANITIZER ASAN
|
var/tmp/check-mysqld_1.log:HAVE_SANITIZER ASAN
|
var/log/main.n/tmp/check-mysqld_1.result:HAVE_SANITIZER ASAN
|
var/log/main.n/tmp/check-mysqld_1.log:HAVE_SANITIZER ASAN
|
On my local system, typically using clang version 7, 8, 9, or 10, I have experienced a hang of main.mysqltest since June 2019, on 10.5 and possibly also on 10.4. I finally decided to debug it, and it looks like it could be the same problem. Only, for some reason, the mysqltest process is hanging instead of crashing:
|
10.5 9a7d284e202d8862a7b34aaf2253d4627177a90e |
gdb /dev/shm/10.5a/client/mysqltest
|
GNU gdb (Debian 9.1-2) 9.1
|
Copyright (C) 2020 Free Software Foundation, Inc.
|
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
|
This is free software: you are free to change and redistribute it.
|
There is NO WARRANTY, to the extent permitted by law.
|
Type "show copying" and "show warranty" for details.
|
This GDB was configured as "x86_64-linux-gnu".
|
Type "show configuration" for configuration details.
|
For bug reporting instructions, please see:
|
<http://www.gnu.org/software/gdb/bugs/>.
|
Find the GDB manual and other documentation resources online at:
|
<http://www.gnu.org/software/gdb/documentation/>.
|
|
|
For help, type "help".
|
Type "apropos word" to search for commands related to "word"...
|
Reading symbols from /dev/shm/10.5a/client/mysqltest...
|
(gdb) set env ASAN_OPTIONS=abort_on_error=1
|
(gdb) run --defaults-file=/dev/shm/10.5a/mysql-test/var/my.cnf --silent --tmpdir=/dev/shm/10.5a/mysql-test/var/tmp --character-sets-dir=/mariadb/10.5m/sql/share/charsets --logdir=/dev/shm/10.5a/mysql-test/var/log --database=test --timer-file=/dev/shm/10.5a/mysql-test/var/log/timer < /mariadb/10.5m/mysql-test/main/n.test
|
Starting program: /dev/shm/10.5a/client/mysqltest --defaults-file=/dev/shm/10.5a/mysql-test/var/my.cnf --silent --tmpdir=/dev/shm/10.5a/mysql-test/var/tmp --character-sets-dir=/mariadb/10.5m/sql/share/charsets --logdir=/dev/shm/10.5a/mysql-test/var/log --database=test --timer-file=/dev/shm/10.5a/mysql-test/var/log/timer < /mariadb/10.5m/mysql-test/main/n.test
|
[Thread debugging using libthread_db enabled]
|
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
|
mysqltest: At line 1: Error parsing replace_regex "a"
|
|
|
Program received signal SIGABRT, Aborted.
|
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
50 ../sysdeps/unix/sysv/linux/raise.c: Tiedostoa tai hakemistoa ei ole.
|
(gdb) bt
|
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
|
#1 0x00007ffff770f55b in __GI_abort () at abort.c:79
|
#2 0x00000000004d1f37 in __sanitizer::Abort() ()
|
at /mariadb/10.5m/client/mysqltest.cc:829
|
#3 0x00000000004d0961 in __sanitizer::Die() ()
|
at /mariadb/10.5m/client/mysqltest.cc:829
|
#4 0x00000000004db3d4 in __lsan::CheckForLeaks() ()
|
at /mariadb/10.5m/client/mysqltest.cc:829
|
#5 0x00000000004db0a2 in __lsan::DoLeakCheck() ()
|
at /mariadb/10.5m/client/mysqltest.cc:829
|
#6 0x00007ffff7727e27 in __run_exit_handlers (status=1,
|
listp=0x7ffff78a6718 <__exit_funcs>,
|
run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
|
at exit.c:108
|
#7 0x00007ffff7727fda in __GI_exit (status=<optimized out>) at exit.c:139
|
#8 0x00000000004ed2dd in cleanup_and_exit (exit_code=1)
|
at /mariadb/10.5m/client/mysqltest.cc:1567
|
#9 0x00000000004ed0c7 in really_die (msg=<optimized out>)
|
at /mariadb/10.5m/client/mysqltest.cc:1276
|
#10 0x00000000004eae92 in die (fmt=<optimized out>)
|
at /mariadb/10.5m/client/mysqltest.cc:1615
|
#11 0x000000000051f88a in append_replace_regex (expr=<optimized out>,
|
expr_end=<optimized out>, res=<optimized out>, buf_p=<optimized out>)
|
at /mariadb/10.5m/client/mysqltest.cc:10269
|
#12 0x000000000051a268 in init_replace_regex (expr=0x604000000e36 "a")
|
at /mariadb/10.5m/client/mysqltest.cc:10188
|
#13 do_get_replace_regex (command=0x612000000058)
|
at /mariadb/10.5m/client/mysqltest.cc:10349
|
#14 0x0000000000517cca in main (argc=<optimized out>, argv=0x7fffffffdfe0)
|
at /mariadb/10.5m/client/mysqltest.cc:9633
|
This particular one was built cmake -DWITH_PCRE2=bundled. I have seen (but not debugged) the hang also with the system-provided library.
Workaround: Set ASAN_OPTIONS=detect_leaks=0 for the mysqltest invocation, and it will not crash or hang.
I can run the entire test without problems, using
ASAN_OPTIONS=detect_leaks=0 ./mtr main.mysqltest
|