[MDEV-9091] mysqld crashes on shutdown after running TokuDB tests on Ubuntu Created: 2015-11-06  Updated: 2022-11-10  Resolved: 2022-11-10

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - TokuDB
Affects Version/s: 10.0.22
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Alexander Barkov Assignee: Vicențiu Ciorbaru
Resolution: Won't Fix Votes: 0
Labels: portability

Issue Links:
Duplicate
duplicates MDEV-9891 Official Ubuntu 16.04 builds unsucces... Closed
Relates
relates to MDEV-7550 TokuDB crashes in build tests on Laun... Closed
Sprint: 10.1.9-3

 Description   

Login to a Ubuntu-15.10 x86_64 machine, e.g. like this:

bar@ubuntu15:~$ uname -a
Linux ubuntu15 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
bar@ubuntu15:~$ cat /etc/debian_version 
jessie/sid

Run the following commands:

git clone -b 10.0 git@github.com:MariaDB/server.git
mkdir builddir
cd builddir/
export CFLAGS="-g -Wl,-Bsymbolic-functions "
export CXXFLAGS="-g -Wl,-Bsymbolic-functions "
cmake -DCMAKE_INSTALL_PREFIX=/usr \
    -DWITH_SSL=bundled \
    -DWITH_ZLIB=system \
    -DBUILD_CONFIG=mysql_release \
     ..
make -j6
cd mysql-test
 ./mtr  tokudb_bugs.db823

It generates the following output:

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
tokudb_bugs.db823 'innodb_plugin'        [ pass ]    158
***Warnings generated in error logs during shutdown after running tests: tokudb_bugs.db823
 
151106 13:24:13 [ERROR] mysqld got signal 11 ;
Attempting backtrace. You can use the following information to find out
 
 
Only  1  of 2 completed.
--------------------------------------------------------------------------
The servers were restarted 0 times
Spent 0.158 of 6 seconds executing testcases
 
Warnings in log: All 1 tests were successful.
 
Errors/warnings were found in logfiles during server shutdown after running the
following sequence(s) of tests:
    tokudb_bugs.db823
mysql-test-run: *** ERROR: There where errors/warnings in server logs after running test cases.

Now If I run with gdb:

 ./mtr  --gdb tokudb_bugs.db823

it displays the following stack trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7f33700 (LWP 26951)]
0x00007fffef97b920 in ?? ()
(gdb) where
#0  0x00007fffef97b920 in ?? ()
#1  0x00007ffff70fc359 in __nptl_deallocate_tsd () at pthread_create.c:175
#2  0x00007ffff70fd76d in __nptl_deallocate_tsd ()
    at ../sysdeps/unix/sysv/linux/exit-thread.h:36
#3  start_thread (arg=0x7ffff7f33700) at pthread_create.c:346
#4  0x00007ffff67a8eed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
 
(gdb) info thread
  Id   Target Id         Frame 
* 50   Thread 0x7ffff7f33700 (LWP 26951) "mysqld" (Exiting) 0x00007fffef97b920 in ?? ()
  1    Thread 0x7ffff7fd9780 (LWP 26889) "mysqld" 0x00007ffff67a3817 in madvise
    () at ../sysdeps/unix/syscall-template.S:81
 
(gdb) t 1
[Switching to thread 1 (Thread 0x7ffff7fd9780 (LWP 26889))]
#0  0x00007ffff67a3817 in madvise () at ../sysdeps/unix/syscall-template.S:81
81      in ../sysdeps/unix/syscall-template.S
(gdb) where
#0  0x00007ffff67a3817 in madvise () at ../sysdeps/unix/syscall-template.S:81
#1  0x0000000000c60c0e in je_pages_purge ()
#2  0x0000000000c5c072 in arena_purge ()
#3  0x0000000000c5d76d in je_arena_dalloc_large ()
#4  0x0000000000991dee in cleanup_instruments ()
    at /home/bar/maria-bzr/server/storage/perfschema/pfs_instr.cc:534
#5  0x0000000000999907 in cleanup_performance_schema ()
    at /home/bar/maria-bzr/server/storage/perfschema/pfs_server.cc:165
#6  0x000000000099a191 in shutdown_performance_schema ()
    at /home/bar/maria-bzr/server/storage/perfschema/pfs_server.cc:196
#7  0x0000000000524334 in mysqld_exit (exit_code=exit_code@entry=0)
    at /home/bar/maria-bzr/server/sql/mysqld.cc:1975
#8  0x000000000052f3ea in mysqld_main (argc=138, argv=0x7ffff5c6ba58)
    at /home/bar/maria-bzr/server/sql/mysqld.cc:5626
#9  0x00007ffff66c2a40 in __libc_start_main (
    main=0x514540 <main(int, char**)>, argc=22, argv=0x7fffffffb8a8, 
    init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffb898) at libc-start.c:289
#10 0x0000000000523c39 in _start ()



 Comments   
Comment by Alexander Barkov [ 2015-11-06 ]

If I compile with the standard debug mode:

cmake .. -DCMAKE_BUILD_TYPE=Debug

or with the release mode without setting the "-g -Wl,-Bsymbolic-functions " flags:

cmake -DCMAKE_INSTALL_PREFIX=/usr \
    -DWITH_SSL=bundled \
    -DWITH_ZLIB=system \
    -DBUILD_CONFIG=mysql_release \
     ..

the problem disappears.

Comment by Otto Kekäläinen [ 2016-11-25 ]

This is still present in https://launchpadlibrarian.net/293123979/buildlog_ubuntu-zesty-amd64.mariadb-10.0_10.0.28-2_BUILDING.txt.gz

tokudb_bugs.db823                        [ disabled ]  MDEV-9891 - massive crashes on shutdown
tokudb_bugs.db917                        [ disabled ]  MDEV-9891 - massive crashes on shutdown
tokudb_bugs.db938                        [ disabled ]  MDEV-9891 - massive crashes on shutdown
tokudb_bugs.db945                        [ disabled ]  MDEV-9891 - massive crashes on shutdown

Comment by Vicențiu Ciorbaru [ 2016-11-28 ]

This was fixed by:

commit 83d5b963bd38e327a673c5d4f4d70c8223f45dd2
Author: Vicențiu Ciorbaru <vicentiu@mariadb.org>
Date:   Mon Sep 19 17:15:18 2016 +0200
 
    Fix tokudb jemalloc linking
 
    Linking tokudb with jemalloc privately causes problems on library
    load/unload. To prevent dangling destructor pointers, link with the same
    library as the server is using.

We should re-enable the tests and see if we get crashes again.

Comment by Vicențiu Ciorbaru [ 2016-11-28 ]

The way this bug happens is that TokuDB on loading of the .so file calls malloc (jemalloc's malloc). Jemalloc in turn sets some thread specific data and attaches destructors to them. The destructor functions come from TokuDB's internally linked jemalloc library.

On library unload however, the thread lives on a bit longer and we lose the destructor functions. When the thread finally ends it has no more destructor functions available, so it crashes. Without -Bsymbolic-functions it works, because the server keeps its own copy of jemalloc and without the compile flag, the destructor function is actually from the first occurrence of the symbol (the server's jemalloc library).

Comment by Elena Stepanova [ 2022-11-10 ]

TokuDB is not provided since 10.5.

Generated at Thu Feb 08 07:32:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.