[MDEV-7550] TokuDB crashes in build tests on Launchpad Created: 2015-02-06  Updated: 2017-11-13  Resolved: 2015-11-16

Status: Closed
Project: MariaDB Server
Component/s: Packaging, Platform Debian, Storage Engine - TokuDB
Affects Version/s: 10.0.16
Fix Version/s: 10.0.23

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: Alexander Barkov
Resolution: Fixed Votes: 0
Labels: upstream

Issue Links:
Relates
relates to MDEV-14383 tokudb_bugs. tests failed in buildbot... Closed
relates to MDEV-9091 mysqld crashes on shutdown after runn... Closed
Sprint: 10.0.22, 10.1.9-1, 10.1.9-2, 10.1.9-3

 Description   

(Originally reported on mailing list, see thread at https://lists.launchpad.net/maria-developers/msg08111.html)

I repeatedly get these test failures when running MariaDB builds on
Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533
tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb
tokudb_bugs.5733_innodb

The build run log can be viewed at
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all
and it links to the individual build logs. All amd64 builds fail while
all i386 (where TokuDB is disabled) pass OK. Sounds very much like a
TokuDB-related issue. Or could the problem be something else and the
TokuDB tests are just the first victim?

Example of a recent log:
https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.16-1~exp3~vivid1~1422913379_FAILEDTOBUILD.txt.gz

One example of the test outputs:

rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ]
        Test ended at 2015-02-02 22:29:06
 
CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
 
 
Failed to start mysqld.1
mysqltest failed but provided no output
 
 
 - saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
 - found 'core' (1/5)
 
Trying 'dbx' to get a backtrace
 
Trying 'gdb' to get a backtrace
Compressed file
/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core
***Warnings generated in error logs during shutdown after running
tests: rpl-tokudb.tokudb_innodb_xa_crash
 
150202 22:29:05 [ERROR] mysqld got signal 11 ;
Attempting backtrace. You can use the following information to find out
150202 22:29:05 [ERROR] mysqld got signal 11 ;
Attempting backtrace. You can use the following information to find out

Make options are visible in
https://github.com/ottok/mariadb-10.0/blob/master/debian/rules

Getting core dumps from Launchpad is not possible, nobody has file system access to builds. Only output channel is build logs.

The old issue might be related: https://mariadb.atlassian.net/browse/MDEV-5618 and https://tokutek.atlassian.net/browse/DB-509



 Comments   
Comment by Otto Kekäläinen [ 2015-02-06 ]

Failing TokuDB tests are now stopping MariaDB 10.0 from entering
Ubuntu, as the official builds fail too:

https://launchpad.net/ubuntu/+source/mariadb-10.0/10.0.16-1
https://launchpadlibrarian.net/196473592/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.16-1_FAILEDTOBUILD.txt.gz

This time the failures are different:

Completed: Failed 7/4551 tests, 99.85% were successful.
Failing test(s): tokudb.rows-32m-0 tokudb.rows-32m-1
tokudb.savepoint-5 tokudb_bugs.frm_store tokudb_bugs.frm_store2
tokudb_bugs.frm_store3 plugins.show_all_plugins

Others yield warnings. I have no idea what to do about these. Only fix
I can do at the moment if to disable TokuDB completely.

I remember that TokuDB has created a lot of work for me during the
last 2 years or so. Could TokuDB devs consider setting up their own
Launchpad PPA (it's free) and run development builds there every now
and then, so that these issues would have a chance to be caught before
release?

Comment by Elena Stepanova [ 2015-02-06 ]

Assigning to serg who, according to the mailing list, was going to provide a debugging patch

Comment by Otto Kekäläinen [ 2015-08-30 ]

I checked the status of this issue by temporarily enabling TokuDB in Debian, but it still fails to build on Launchpad amd64: https://launchpadlibrarian.net/215874882/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.21-3~vivid1~1440875684.7954da1_BUILDING.txt.gz

Too many failed: Failed 10/1210 tests, 99.17% were successful.
Failing test(s): rpl-tokudb.tokudb_innodb_xa_crash rpl-tokudb.rpl_tokudb_read_only_ft tokudb_bugs.mdev4533 tokudb_bugs.db811 tokudb_bugs.db811s

Comment by Alexander Barkov [ 2015-10-22 ]

This log has a stack trace, but not very helpful:
https://launchpadlibrarian.net/196929183/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.16-1~exp4~vivid1~1423323331_FAILEDTOBUILD.txt.gz

Comment by Alexander Barkov [ 2015-10-28 ]

It seems the crash is caused by this linker option:

-Wl,-Bsymbolic-functions 

A better stack trace:

(gdb) where
#0  0x00007fffef990a4c in je_extent_tree_ad_search () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#1  0x00007fffef991c88 in je_huge_salloc () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#2  0x00007fffef97c895 in free () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
#3  0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
    at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
#4  toku_os_get_processor_frequency(unsigned long*) (hzret=0x7fffffff99b8)
    at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:409
#5  0x00007fffef946e2d in toku_portability_init() () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:139
#6  0x00007fffef8f633c in toku_ft_layer_init () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/ft/ft-ops.cc:4535
#7  0x00007fffef88b9c9 in libtokuft_init() [clone .lto_priv.234] () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/src/ydb_lib.cc:102
#8  0x00007ffff7de95ba in call_init (l=<optimized out>, argc=argc@entry=24, argv=argv@entry=0x7fffffffb5c8, env=env@entry=0x7fffffffb690)
    at dl-init.c:72
#9  0x00007ffff7de96cb in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:30
#10 _dl_init (main_map=main_map@entry=0x7ffff5c26000, argc=24, argv=0x7fffffffb5c8, env=0x7fffffffb690) at dl-init.c:120
#11 0x00007ffff7dee587 in dl_open_worker (a=a@entry=0x7fffffff9ca8) at dl-open.c:579
#12 0x00007ffff7de9464 in _dl_catch_error (objname=objname@entry=0x7fffffff9c98, errstring=errstring@entry=0x7fffffff9ca0, 
    mallocedp=mallocedp@entry=0x7fffffff9c97, operate=operate@entry=0x7ffff7dee0a0 <dl_open_worker>, args=args@entry=0x7fffffff9ca8)
    at dl-error.c:187
#13 0x00007ffff7ded9a3 in _dl_open (file=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", 
    mode=-2147483646, caller_dlopen=0x5d0273 <plugin_dl_add(LEX_STRING const*, int)+451>, nsid=-2, argc=<optimized out>, argv=<optimized out>, 
    env=0x7fffffffb690) at dl-open.c:663
#14 0x00007ffff7314fc9 in dlopen_doit (a=a@entry=0x7fffffff9ec0) at dlopen.c:66
#15 0x00007ffff7de9464 in _dl_catch_error (objname=0x7ffff5c46210, errstring=0x7ffff5c46218, mallocedp=0x7ffff5c46208, 
    operate=0x7ffff7314f70 <dlopen_doit>, args=0x7fffffff9ec0) at dl-error.c:187
#16 0x00007ffff731562d in _dlerror_run (operate=operate@entry=0x7ffff7314f70 <dlopen_doit>, args=args@entry=0x7fffffff9ec0) at dlerror.c:163
#17 0x00007ffff7315061 in __dlopen (
    file=file@entry=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", mode=mode@entry=2)
    at dlopen.c:87
#18 0x00000000005d0273 in plugin_dl_add (dl=dl@entry=0x7fffffffa700, report=report@entry=1)
    at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:740
#19 0x00000000005d0b31 in plugin_add (tmp_root=tmp_root@entry=0x7fffffffa730, name=name@entry=0x7fffffffa6e0, dl=dl@entry=0x7fffffffa700, 
    report=1) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1058
#20 0x00000000005d6863 in plugin_load_list (list=0x0, tmp_root=0x7fffffffa730) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1839
#21 plugin_init (argc=argc@entry=0x1407598 <remaining_argc>, argv=0x7ffff5c61a58, flags=0)
    at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1636
#22 0x0000000000528203 in init_server_components () at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:4837
#23 0x000000000052decb in mysqld_main (argc=140, argv=0x7ffff5c61a58) at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:5432
#24 0x00007ffff66c2a40 in __libc_start_main (main=0x513c90 <main(int, char**)>, argc=24, argv=0x7fffffffb5c8, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffb5b8) at libc-start.c:289
#25 0x0000000000523389 in _start ()
(gdb) f 3
#3  0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
    at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
371                 free(buf);

This patch removing the linker flag seems to fix the problem on Ubuntu:

--- a/debian/rules
+++ b/debian/rules
@@ -8,6 +8,9 @@ export DEB_BUILD_MAINT_OPTIONS = hardening=+all,-pie
 DPKG_EXPORT_BUILDFLAGS = 1
 include /usr/share/dpkg/buildflags.mk
 
+# MDEV-7550 TokuDB crashes in build tests on Launchpad
+export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions
+
 ARCH := $(shell dpkg-architecture -qDEB_BUILD_ARCH)
 ARCH_OS := $(shell dpkg-architecture -qDEB_BUILD_ARCH_OS)
 BUILDDIR := builddir
@@ -47,7 +50,7 @@ endif
 # Completely disable TokuDB as it fails to build on some platforms and
 # upstream developers do not seem to have fixing those on their roadmap
 # See https://lists.launchpad.net/maria-developers/msg08390.html
-CMAKEFLAGS += -DWITHOUT_TOKUDB=true
+# CMAKEFLAGS += -DWITHOUT_TOKUDB=true
 
 # Add support for verbose builds
 MAKEFLAGS += VERBOSE=1

However, this is a workaround only. We should try to fix the problem in the code.

Comment by Alexander Barkov [ 2015-10-28 ]

How-to-repeat instructions:

Login to a Ubuntu-15.10 x86_64 machine, e.g. like this:

bar@ubuntu15:~$ uname -a
Linux ubuntu15 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
bar@ubuntu15:~$ cat /etc/debian_version 
jessie/sid

Make sure that you have libjemalloc_pic.a installed.

bar@ubuntu15:/usr/lib/x86_64-linux-gnu$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a 
-rw-r--r-- 1 root root 420266 июля  24  2014 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a

Run the following commands:

git clone -b 10.0 git@github.com:MariaDB/server.git
mkdir builddir
cd builddir/
export CFLAGS="-g -Wl,-Bsymbolic-functions "
export CXXFLAGS="-g -Wl,-Bsymbolic-functions "
cmake -DCMAKE_INSTALL_PREFIX=/usr \
    -DWITH_SSL=bundled \
    -DWITH_ZLIB=system \
    -DBUILD_CONFIG=mysql_release \
     ..
make -j6
cd mysql-test
./mtr rpl-tokudb.tokudb_innodb_xa_crash
./mtr --gdb rpl-tokudb.tokudb_innodb_xa_crash

Comment by Alexander Barkov [ 2015-11-06 ]

Sergei reported the problem to the upstream:
https://bugs.launchpad.net/percona-server/+bug/1510915

Comment by Alexander Barkov [ 2015-11-06 ]

--- a/storage/tokudb/ft-index/portability/portability.cc
+++ b/storage/tokudb/ft-index/portability/portability.cc
@@ -355,9 +355,16 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) {
         r = get_error_errno();
     } else {
         uint64_t maxhz = 0;
-        char *buf = NULL;
-        size_t n = 0;
-        while (getline(&buf, &n, fp) >= 0) {
+        /*
+          Some lines in the "/proc/cpuinfo" output can be long, e.g.:
+            "flags  : fpu vme de pse tsc ms .... smep erms"
+          In case a line does not fit into "buf", it will be read
+          in parts by multiple "fgets" calls. This is ok, as
+          it is very unlikely that a non-leading substring of a line
+          will match again the pattern "processor: %u".
+        */
+        char buf[512];
+        while (fgets(buf, (int) sizeof(buf), fp) != NULL) {
             unsigned int cpu;
             sscanf(buf, "processor : %u", &cpu);
             unsigned int ma, mb;
@@ -367,8 +374,6 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) {
                     maxhz = hz;
             }
         }
-        if (buf)
-            free(buf);
         fclose(fp);
         *hzret = maxhz;
         r = maxhz == 0 ? ENOENT : 0;;

Comment by Sergei Golubchik [ 2015-11-06 ]

ok to push, thanks!

Generated at Thu Feb 08 07:20:27 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.