Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7550

TokuDB crashes in build tests on Launchpad

Details

    • 10.0.22, 10.1.9-1, 10.1.9-2, 10.1.9-3

    Description

      (Originally reported on mailing list, see thread at https://lists.launchpad.net/maria-developers/msg08111.html)

      I repeatedly get these test failures when running MariaDB builds on
      Launchpad: rpl-tokudb.tokudb_innodb_xa_crash tokudb_bugs.mdev4533
      tokudb.simple_join_tokudb_innodb tokudb.truncate_txn_rollback_innodb
      tokudb_bugs.5733_innodb

      The build run log can be viewed at
      https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.0/+builds?build_text=&build_state=all
      and it links to the individual build logs. All amd64 builds fail while
      all i386 (where TokuDB is disabled) pass OK. Sounds very much like a
      TokuDB-related issue. Or could the problem be something else and the
      TokuDB tests are just the first victim?

      Example of a recent log:
      https://launchpadlibrarian.net/196425361/buildlog_ubuntu-vivid-amd64.mariadb-10.0_10.0.16-1~exp3~vivid1~1422913379_FAILEDTOBUILD.txt.gz

      One example of the test outputs:

      rpl-tokudb.tokudb_innodb_xa_crash 'mix,xtradb' [ fail ]
              Test ended at 2015-02-02 22:29:06
       
      CURRENT_TEST: rpl-tokudb.tokudb_innodb_xa_crash
       
       
      Failed to start mysqld.1
      mysqltest failed but provided no output
       
       
       - saving '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
      to '/build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/'
       - found 'core' (1/5)
       
      Trying 'dbx' to get a backtrace
       
      Trying 'gdb' to get a backtrace
      Compressed file
      /build/buildd/mariadb-10.0-10.0.16/builddir/mysql-test/var/log/rpl-tokudb.tokudb_innodb_xa_crash-mix,xtradb/mysqld.1/data/core
      ***Warnings generated in error logs during shutdown after running
      tests: rpl-tokudb.tokudb_innodb_xa_crash
       
      150202 22:29:05 [ERROR] mysqld got signal 11 ;
      Attempting backtrace. You can use the following information to find out
      150202 22:29:05 [ERROR] mysqld got signal 11 ;
      Attempting backtrace. You can use the following information to find out

      Make options are visible in
      https://github.com/ottok/mariadb-10.0/blob/master/debian/rules

      Getting core dumps from Launchpad is not possible, nobody has file system access to builds. Only output channel is build logs.

      The old issue might be related: https://mariadb.atlassian.net/browse/MDEV-5618 and https://tokutek.atlassian.net/browse/DB-509

      Attachments

        Issue Links

          Activity

            bar Alexander Barkov added a comment - - edited

            It seems the crash is caused by this linker option:

            -Wl,-Bsymbolic-functions 

            A better stack trace:

            (gdb) where
            #0  0x00007fffef990a4c in je_extent_tree_ad_search () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
            #1  0x00007fffef991c88 in je_huge_salloc () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
            #2  0x00007fffef97c895 in free () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so
            #3  0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
                at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
            #4  toku_os_get_processor_frequency(unsigned long*) (hzret=0x7fffffff99b8)
                at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:409
            #5  0x00007fffef946e2d in toku_portability_init() () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:139
            #6  0x00007fffef8f633c in toku_ft_layer_init () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/ft/ft-ops.cc:4535
            #7  0x00007fffef88b9c9 in libtokuft_init() [clone .lto_priv.234] () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/src/ydb_lib.cc:102
            #8  0x00007ffff7de95ba in call_init (l=<optimized out>, argc=argc@entry=24, argv=argv@entry=0x7fffffffb5c8, env=env@entry=0x7fffffffb690)
                at dl-init.c:72
            #9  0x00007ffff7de96cb in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:30
            #10 _dl_init (main_map=main_map@entry=0x7ffff5c26000, argc=24, argv=0x7fffffffb5c8, env=0x7fffffffb690) at dl-init.c:120
            #11 0x00007ffff7dee587 in dl_open_worker (a=a@entry=0x7fffffff9ca8) at dl-open.c:579
            #12 0x00007ffff7de9464 in _dl_catch_error (objname=objname@entry=0x7fffffff9c98, errstring=errstring@entry=0x7fffffff9ca0, 
                mallocedp=mallocedp@entry=0x7fffffff9c97, operate=operate@entry=0x7ffff7dee0a0 <dl_open_worker>, args=args@entry=0x7fffffff9ca8)
                at dl-error.c:187
            #13 0x00007ffff7ded9a3 in _dl_open (file=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", 
                mode=-2147483646, caller_dlopen=0x5d0273 <plugin_dl_add(LEX_STRING const*, int)+451>, nsid=-2, argc=<optimized out>, argv=<optimized out>, 
                env=0x7fffffffb690) at dl-open.c:663
            #14 0x00007ffff7314fc9 in dlopen_doit (a=a@entry=0x7fffffff9ec0) at dlopen.c:66
            #15 0x00007ffff7de9464 in _dl_catch_error (objname=0x7ffff5c46210, errstring=0x7ffff5c46218, mallocedp=0x7ffff5c46208, 
                operate=0x7ffff7314f70 <dlopen_doit>, args=0x7fffffff9ec0) at dl-error.c:187
            #16 0x00007ffff731562d in _dlerror_run (operate=operate@entry=0x7ffff7314f70 <dlopen_doit>, args=args@entry=0x7fffffff9ec0) at dlerror.c:163
            #17 0x00007ffff7315061 in __dlopen (
                file=file@entry=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", mode=mode@entry=2)
                at dlopen.c:87
            #18 0x00000000005d0273 in plugin_dl_add (dl=dl@entry=0x7fffffffa700, report=report@entry=1)
                at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:740
            #19 0x00000000005d0b31 in plugin_add (tmp_root=tmp_root@entry=0x7fffffffa730, name=name@entry=0x7fffffffa6e0, dl=dl@entry=0x7fffffffa700, 
                report=1) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1058
            #20 0x00000000005d6863 in plugin_load_list (list=0x0, tmp_root=0x7fffffffa730) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1839
            #21 plugin_init (argc=argc@entry=0x1407598 <remaining_argc>, argv=0x7ffff5c61a58, flags=0)
                at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1636
            #22 0x0000000000528203 in init_server_components () at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:4837
            #23 0x000000000052decb in mysqld_main (argc=140, argv=0x7ffff5c61a58) at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:5432
            #24 0x00007ffff66c2a40 in __libc_start_main (main=0x513c90 <main(int, char**)>, argc=24, argv=0x7fffffffb5c8, init=<optimized out>, 
                fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffb5b8) at libc-start.c:289
            #25 0x0000000000523389 in _start ()
            (gdb) f 3
            #3  0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8)
                at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371
            371                 free(buf);

            This patch removing the linker flag seems to fix the problem on Ubuntu:

            --- a/debian/rules
            +++ b/debian/rules
            @@ -8,6 +8,9 @@ export DEB_BUILD_MAINT_OPTIONS = hardening=+all,-pie
             DPKG_EXPORT_BUILDFLAGS = 1
             include /usr/share/dpkg/buildflags.mk
             
            +# MDEV-7550 TokuDB crashes in build tests on Launchpad
            +export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions
            +
             ARCH := $(shell dpkg-architecture -qDEB_BUILD_ARCH)
             ARCH_OS := $(shell dpkg-architecture -qDEB_BUILD_ARCH_OS)
             BUILDDIR := builddir
            @@ -47,7 +50,7 @@ endif
             # Completely disable TokuDB as it fails to build on some platforms and
             # upstream developers do not seem to have fixing those on their roadmap
             # See https://lists.launchpad.net/maria-developers/msg08390.html
            -CMAKEFLAGS += -DWITHOUT_TOKUDB=true
            +# CMAKEFLAGS += -DWITHOUT_TOKUDB=true
             
             # Add support for verbose builds
             MAKEFLAGS += VERBOSE=1

            However, this is a workaround only. We should try to fix the problem in the code.

            bar Alexander Barkov added a comment - - edited It seems the crash is caused by this linker option: -Wl,-Bsymbolic-functions A better stack trace: (gdb) where #0 0x00007fffef990a4c in je_extent_tree_ad_search () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so #1 0x00007fffef991c88 in je_huge_salloc () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so #2 0x00007fffef97c895 in free () from /home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so #3 0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8) at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371 #4 toku_os_get_processor_frequency(unsigned long*) (hzret=0x7fffffff99b8) at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:409 #5 0x00007fffef946e2d in toku_portability_init() () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:139 #6 0x00007fffef8f633c in toku_ft_layer_init () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/ft/ft-ops.cc:4535 #7 0x00007fffef88b9c9 in libtokuft_init() [clone .lto_priv.234] () at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/src/ydb_lib.cc:102 #8 0x00007ffff7de95ba in call_init (l=<optimized out>, argc=argc@entry=24, argv=argv@entry=0x7fffffffb5c8, env=env@entry=0x7fffffffb690) at dl-init.c:72 #9 0x00007ffff7de96cb in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:30 #10 _dl_init (main_map=main_map@entry=0x7ffff5c26000, argc=24, argv=0x7fffffffb5c8, env=0x7fffffffb690) at dl-init.c:120 #11 0x00007ffff7dee587 in dl_open_worker (a=a@entry=0x7fffffff9ca8) at dl-open.c:579 #12 0x00007ffff7de9464 in _dl_catch_error (objname=objname@entry=0x7fffffff9c98, errstring=errstring@entry=0x7fffffff9ca0, mallocedp=mallocedp@entry=0x7fffffff9c97, operate=operate@entry=0x7ffff7dee0a0 <dl_open_worker>, args=args@entry=0x7fffffff9ca8) at dl-error.c:187 #13 0x00007ffff7ded9a3 in _dl_open (file=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", mode=-2147483646, caller_dlopen=0x5d0273 <plugin_dl_add(LEX_STRING const*, int)+451>, nsid=-2, argc=<optimized out>, argv=<optimized out>, env=0x7fffffffb690) at dl-open.c:663 #14 0x00007ffff7314fc9 in dlopen_doit (a=a@entry=0x7fffffff9ec0) at dlopen.c:66 #15 0x00007ffff7de9464 in _dl_catch_error (objname=0x7ffff5c46210, errstring=0x7ffff5c46218, mallocedp=0x7ffff5c46208, operate=0x7ffff7314f70 <dlopen_doit>, args=0x7fffffff9ec0) at dl-error.c:187 #16 0x00007ffff731562d in _dlerror_run (operate=operate@entry=0x7ffff7314f70 <dlopen_doit>, args=args@entry=0x7fffffff9ec0) at dlerror.c:163 #17 0x00007ffff7315061 in __dlopen ( file=file@entry=0x7fffffffa000 "/home/bar/maria-bzr/mariadb-10.0/builddir4/mysql-test/var/plugins/ha_tokudb.so", mode=mode@entry=2) at dlopen.c:87 #18 0x00000000005d0273 in plugin_dl_add (dl=dl@entry=0x7fffffffa700, report=report@entry=1) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:740 #19 0x00000000005d0b31 in plugin_add (tmp_root=tmp_root@entry=0x7fffffffa730, name=name@entry=0x7fffffffa6e0, dl=dl@entry=0x7fffffffa700, report=1) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1058 #20 0x00000000005d6863 in plugin_load_list (list=0x0, tmp_root=0x7fffffffa730) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1839 #21 plugin_init (argc=argc@entry=0x1407598 <remaining_argc>, argv=0x7ffff5c61a58, flags=0) at /home/bar/maria-bzr/mariadb-10.0/sql/sql_plugin.cc:1636 #22 0x0000000000528203 in init_server_components () at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:4837 #23 0x000000000052decb in mysqld_main (argc=140, argv=0x7ffff5c61a58) at /home/bar/maria-bzr/mariadb-10.0/sql/mysqld.cc:5432 #24 0x00007ffff66c2a40 in __libc_start_main (main=0x513c90 <main(int, char**)>, argc=24, argv=0x7fffffffb5c8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffb5b8) at libc-start.c:289 #25 0x0000000000523389 in _start () (gdb) f 3 #3 0x00007fffef9454ee in toku_get_processor_frequency_cpuinfo (hzret=0x7fffffff99b8) at /home/bar/maria-bzr/mariadb-10.0/storage/tokudb/ft-index/portability/portability.cc:371 371 free(buf); This patch removing the linker flag seems to fix the problem on Ubuntu: --- a/debian/rules +++ b/debian/rules @@ -8,6 +8,9 @@ export DEB_BUILD_MAINT_OPTIONS = hardening=+all,-pie DPKG_EXPORT_BUILDFLAGS = 1 include /usr/share/dpkg/buildflags.mk +# MDEV-7550 TokuDB crashes in build tests on Launchpad +export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions + ARCH := $(shell dpkg-architecture -qDEB_BUILD_ARCH) ARCH_OS := $(shell dpkg-architecture -qDEB_BUILD_ARCH_OS) BUILDDIR := builddir @@ -47,7 +50,7 @@ endif # Completely disable TokuDB as it fails to build on some platforms and # upstream developers do not seem to have fixing those on their roadmap # See https://lists.launchpad.net/maria-developers/msg08390.html -CMAKEFLAGS += -DWITHOUT_TOKUDB=true +# CMAKEFLAGS += -DWITHOUT_TOKUDB=true # Add support for verbose builds MAKEFLAGS += VERBOSE=1 However, this is a workaround only. We should try to fix the problem in the code.
            bar Alexander Barkov added a comment - - edited

            How-to-repeat instructions:

            Login to a Ubuntu-15.10 x86_64 machine, e.g. like this:

            bar@ubuntu15:~$ uname -a
            Linux ubuntu15 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
            bar@ubuntu15:~$ cat /etc/debian_version 
            jessie/sid

            Make sure that you have libjemalloc_pic.a installed.

            bar@ubuntu15:/usr/lib/x86_64-linux-gnu$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a 
            -rw-r--r-- 1 root root 420266 июля  24  2014 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a

            Run the following commands:

            git clone -b 10.0 git@github.com:MariaDB/server.git
            mkdir builddir
            cd builddir/
            export CFLAGS="-g -Wl,-Bsymbolic-functions "
            export CXXFLAGS="-g -Wl,-Bsymbolic-functions "
            cmake -DCMAKE_INSTALL_PREFIX=/usr \
                -DWITH_SSL=bundled \
                -DWITH_ZLIB=system \
                -DBUILD_CONFIG=mysql_release \
                 ..
            make -j6
            cd mysql-test
            ./mtr rpl-tokudb.tokudb_innodb_xa_crash
            ./mtr --gdb rpl-tokudb.tokudb_innodb_xa_crash

            bar Alexander Barkov added a comment - - edited How-to-repeat instructions: Login to a Ubuntu-15.10 x86_64 machine, e.g. like this: bar@ubuntu15:~$ uname -a Linux ubuntu15 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux bar@ubuntu15:~$ cat /etc/debian_version jessie/sid Make sure that you have libjemalloc_pic.a installed. bar@ubuntu15:/usr/lib/x86_64-linux-gnu$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a -rw-r--r-- 1 root root 420266 июля 24 2014 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a Run the following commands: git clone -b 10.0 git@github.com:MariaDB/server.git mkdir builddir cd builddir/ export CFLAGS="-g -Wl,-Bsymbolic-functions " export CXXFLAGS="-g -Wl,-Bsymbolic-functions " cmake -DCMAKE_INSTALL_PREFIX=/usr \ -DWITH_SSL=bundled \ -DWITH_ZLIB=system \ -DBUILD_CONFIG=mysql_release \ .. make -j6 cd mysql-test ./mtr rpl-tokudb.tokudb_innodb_xa_crash ./mtr --gdb rpl-tokudb.tokudb_innodb_xa_crash

            Sergei reported the problem to the upstream:
            https://bugs.launchpad.net/percona-server/+bug/1510915

            bar Alexander Barkov added a comment - Sergei reported the problem to the upstream: https://bugs.launchpad.net/percona-server/+bug/1510915
            bar Alexander Barkov added a comment - - edited

            --- a/storage/tokudb/ft-index/portability/portability.cc
            +++ b/storage/tokudb/ft-index/portability/portability.cc
            @@ -355,9 +355,16 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) {
                     r = get_error_errno();
                 } else {
                     uint64_t maxhz = 0;
            -        char *buf = NULL;
            -        size_t n = 0;
            -        while (getline(&buf, &n, fp) >= 0) {
            +        /*
            +          Some lines in the "/proc/cpuinfo" output can be long, e.g.:
            +            "flags  : fpu vme de pse tsc ms .... smep erms"
            +          In case a line does not fit into "buf", it will be read
            +          in parts by multiple "fgets" calls. This is ok, as
            +          it is very unlikely that a non-leading substring of a line
            +          will match again the pattern "processor: %u".
            +        */
            +        char buf[512];
            +        while (fgets(buf, (int) sizeof(buf), fp) != NULL) {
                         unsigned int cpu;
                         sscanf(buf, "processor : %u", &cpu);
                         unsigned int ma, mb;
            @@ -367,8 +374,6 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) {
                                 maxhz = hz;
                         }
                     }
            -        if (buf)
            -            free(buf);
                     fclose(fp);
                     *hzret = maxhz;
                     r = maxhz == 0 ? ENOENT : 0;;

            bar Alexander Barkov added a comment - - edited --- a/storage/tokudb/ft-index/portability/portability.cc +++ b/storage/tokudb/ft-index/portability/portability.cc @@ -355,9 +355,16 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) { r = get_error_errno(); } else { uint64_t maxhz = 0; - char *buf = NULL; - size_t n = 0; - while (getline(&buf, &n, fp) >= 0) { + /* + Some lines in the "/proc/cpuinfo" output can be long, e.g.: + "flags : fpu vme de pse tsc ms .... smep erms" + In case a line does not fit into "buf", it will be read + in parts by multiple "fgets" calls. This is ok, as + it is very unlikely that a non-leading substring of a line + will match again the pattern "processor: %u". + */ + char buf[512]; + while (fgets(buf, (int) sizeof(buf), fp) != NULL) { unsigned int cpu; sscanf(buf, "processor : %u", &cpu); unsigned int ma, mb; @@ -367,8 +374,6 @@ toku_get_processor_frequency_cpuinfo(uint64_t *hzret) { maxhz = hz; } } - if (buf) - free(buf); fclose(fp); *hzret = maxhz; r = maxhz == 0 ? ENOENT : 0;;

            ok to push, thanks!

            serg Sergei Golubchik added a comment - ok to push, thanks!

            People

              bar Alexander Barkov
              otto Otto Kekäläinen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.