Details
-
Bug
-
Status: Closed (View Workflow)
-
Minor
-
Resolution: Fixed
-
10.0.16
-
10.0.22, 10.1.9-1, 10.1.9-2, 10.1.9-3
Description
(Originally reported on mailing list, see thread at https://lists.launchpad.net/maria-developers/msg08111.html)
I repeatedly get these test failures when running MariaDB builds on
Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533
tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb
tokudb_bugs.5733_innodb
The build run log can be viewed at
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all
and it links to the individual build logs. All amd64 builds fail while
all i386 (where TokuDB is disabled) pass OK. Sounds very much like a
TokuDB-related issue. Or could the problem be something else and the
TokuDB tests are just the first victim?
Example of a recent log:
https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.16-1~exp3~vivid1~1422913379_FAILEDTOBUILD.txt.gz
One example of the test outputs:
rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ]
|
Test ended at 2015-02-02 22:29:06
|
|
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
|
|
|
Failed to start mysqld.1
|
mysqltest failed but provided no output
|
|
|
- saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
|
to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
|
- found 'core' (1/5)
|
|
Trying 'dbx' to get a backtrace
|
|
Trying 'gdb' to get a backtrace
|
Compressed file
|
/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core
|
***Warnings generated in error logs during shutdown after running
|
tests: rpl-tokudb.tokudb_innodb_xa_crash
|
|
150202 22:29:05 [ERROR] mysqld got signal 11 ;
|
Attempting backtrace. You can use the following information to find out
|
150202 22:29:05 [ERROR] mysqld got signal 11 ;
|
Attempting backtrace. You can use the following information to find out
|
Make options are visible in
https://github.com/ottok/mariadb-10.0/blob/master/debian/rules
Getting core dumps from Launchpad is not possible, nobody has file system access to builds. Only output channel is build logs.
The old issue might be related: https://mariadb.atlassian.net/browse/MDEV-5618 and https://tokutek.atlassian.net/browse/DB-509
Attachments
Issue Links
- relates to
-
MDEV-14383 tokudb_bugs. tests failed in buildbot, lost connection to server
-
- Closed
-
-
MDEV-9091 mysqld crashes on shutdown after running TokuDB tests on Ubuntu
-
- Closed
-
- links to
It seems the crash is caused by this linker option:
-Wl,-Bsymbolic-functions
A better stack trace:
(gdb) where
#0 0x00007fffef990a4c in je_extent_tree_ad_search () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#1 0x00007fffef991c88 in je_huge_salloc () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#2 0x00007fffef97c895 in free () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#3 0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
#4 toku_os_get_processor_frequency(unsigned long*) (hzret=0x7fffffff99b8)
at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:409
#5 0x00007fffef946e2d in toku_portability_init() () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:139
#6 0x00007fffef8f633c in toku_ft_layer_init () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/ft/ft-ops.cc:4535
#7 0x00007fffef88b9c9 in libtokuft_init() [clone .lto_priv.234] () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/src/ydb_lib.cc:102
#8 0x00007ffff7de95ba in call_init (l=<optimized out>, argc=argc@entry=24, argv=argv@entry=0x7fffffffb5c8, env=env@entry=0x7fffffffb690)
at dl-init.c:72
#9 0x00007ffff7de96cb in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:30
#10 _dl_init (main_map=main_map@entry=0x7ffff5c26000, argc=24, argv=0x7fffffffb5c8, env=0x7fffffffb690) at dl-init.c:120
#11 0x00007ffff7dee587 in dl_open_worker (a=a@entry=0x7fffffff9ca8) at dl-open.c:579
#12 0x00007ffff7de9464 in _dl_catch_error (objname=objname@entry=0x7fffffff9c98, errstring=errstring@entry=0x7fffffff9ca0,
mallocedp=mallocedp@entry=0x7fffffff9c97, operate=operate@entry=0x7ffff7dee0a0 <dl_open_worker>, args=args@entry=0x7fffffff9ca8)
at dl-error.c:187
#13 0x00007ffff7ded9a3 in _dl_open (file=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so",
mode=-2147483646, caller_dlopen=0x5d0273 <plugin_dl_add(LEX_STRING const*, int)+451>, nsid=-2, argc=<optimized out>, argv=<optimized out>,
env=0x7fffffffb690) at dl-open.c:663
#14 0x00007ffff7314fc9 in dlopen_doit (a=a@entry=0x7fffffff9ec0) at dlopen.c:66
#15 0x00007ffff7de9464 in _dl_catch_error (objname=0x7ffff5c46210, errstring=0x7ffff5c46218, mallocedp=0x7ffff5c46208,
operate=0x7ffff7314f70 <dlopen_doit>, args=0x7fffffff9ec0) at dl-error.c:187
#16 0x00007ffff731562d in _dlerror_run (operate=operate@entry=0x7ffff7314f70 <dlopen_doit>, args=args@entry=0x7fffffff9ec0) at dlerror.c:163
#17 0x00007ffff7315061 in __dlopen (
file=file@entry=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", mode=mode@entry=2)
at dlopen.c:87
#18 0x00000000005d0273 in plugin_dl_add (dl=dl@entry=0x7fffffffa700, report=report@entry=1)
at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:740
#19 0x00000000005d0b31 in plugin_add (tmp_root=tmp_root@entry=0x7fffffffa730, name=name@entry=0x7fffffffa6e0, dl=dl@entry=0x7fffffffa700,
report=1) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1058
#20 0x00000000005d6863 in plugin_load_list (list=0x0, tmp_root=0x7fffffffa730) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1839
#21 plugin_init (argc=argc@entry=0x1407598 <remaining_argc>, argv=0x7ffff5c61a58, flags=0)
at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1636
#22 0x0000000000528203 in init_server_components () at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:4837
#23 0x000000000052decb in mysqld_main (argc=140, argv=0x7ffff5c61a58) at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:5432
#24 0x00007ffff66c2a40 in __libc_start_main (main=0x513c90 <main(int, char**)>, argc=24, argv=0x7fffffffb5c8, init=<optimized out>,
fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffb5b8) at libc-start.c:289
#25 0x0000000000523389 in _start ()
(gdb) f 3
#3 0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
371 free(buf);
This patch removing the linker flag seems to fix the problem on Ubuntu:
--- a/debian/rules
+++ b/debian/rules
@@ -8,6 +8,9 @@ export DEB_BUILD_MAINT_OPTIONS = hardening=+all,-pie
DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/buildflags.mk
+# MDEV-7550 TokuDB crashes in build tests on Launchpad
+export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions
+
ARCH := $(shell dpkg-architecture -qDEB_BUILD_ARCH)
ARCH_OS := $(shell dpkg-architecture -qDEB_BUILD_ARCH_OS)
BUILDDIR := builddir
@@ -47,7 +50,7 @@ endif
# Completely disable TokuDB as it fails to build on some platforms and
# upstream developers do not seem to have fixing those on their roadmap
# See https://lists.launchpad.net/maria-developers/msg08390.html
-CMAKEFLAGS += -DWITHOUT_TOKUDB=true
+# CMAKEFLAGS += -DWITHOUT_TOKUDB=true
# Add support for verbose builds
MAKEFLAGS += VERBOSE=1
However, this is a workaround only. We should try to fix the problem in the code.