Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7914

spider/bg.ha, spider/bg.ha_part crash server sporadically in buildbot

Details

    • 10.0.20, 2017-02, 10.3.6-1

    Description

      http://buildbot.askmonty.org/buildbot/builders/kvm-deb-precise-x86/builds/2516/steps/test_4/logs/stdio

      spider/bg.ha                             w3 [ fail ]
              Test ended at 2015-04-04 02:59:26
       
      CURRENT_TEST: spider/bg.ha
       
       
      The server [mysqld.3.2 - pid: 5373, winpid: 5373, exit: 256] crashed while running 'check warnings'
      Server log from this test:
      ----------SERVER LOG START-----------
      150404  2:59:25 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.1.4-MariaDB-1~precise-wsrep-log
      key_buffer_size=1048576
      read_buffer_size=131072
      max_used_connections=2
      max_threads=153
      thread_count=2
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61901 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0xaea8e008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      ----------SERVER LOG END-------------
      mysqltest failed but provided no output
      The result from queries just before the failure was:
      < snip >
      for master_1
      for child2
      child2_1
      child2_2
      child2_3
      for child3
      child3_1
      child3_2
      child3_3
      for master_1
      for child2
      child2_1
      child2_2
      child2_3
      for child3
      child3_1
      child3_2
      child3_3
       
      end of test
       
       
       
       - saving '/run/shm/var/3/log/spider/bg.ha/' to '/run/shm/var/log/bg.ha/'

      Attachments

        Issue Links

          Activity

            elenst Elena Stepanova created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value

            For the record, the occasion with a stack trace, however poor:
            http://buildbot.askmonty.org/buildbot/builders/kvm-deb-lucid-x86/builds/4295/steps/test_4/logs/stdio

            The server [mysqld.3.2 - pid: 4457, winpid: 4457, exit: 256] crashed while running 'check warnings'
            Server log from this test:
            ----------SERVER LOG START-----------
            150426  1:26:10 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see http://kb.askmonty.org/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.5-MariaDB-1~lucid-wsrep-log
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=2
            max_threads=153
            thread_count=2
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61901 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0xaee6e008
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0xb647c2fc thread_stack 0x48000
            /usr/sbin/mysqld(my_print_stacktrace+0x2d)[0xb747537d]
            /usr/sbin/mysqld(handle_fatal_signal+0x50f)[0xb6f3bc6f]
            [0xb6acb400]
            [0xb64b1530]
            [0xb64d2007]
            /usr/sbin/mysqld(+0x457708)[0xb6f40708]
            /usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1fa)[0xb6d77aca]
            /usr/sbin/mysqld(_Z19ha_close_connectionP3THD+0x3f)[0xb6f3ecaf]
            /usr/sbin/mysqld(_ZN3THDD0Ev+0x1f7)[0xb6d2b827]
            /usr/sbin/mysqld(_Z10unlink_thdP3THD+0x193)[0xb6cb5f93]
            /usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x2d)[0xb6cb60bd]
            /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x140)[0xb6e58f20]
            /usr/sbin/mysqld(handle_one_connection+0x50)[0xb6e591f0]
            /lib/tls/i686/cmov/libpthread.so.0(+0x596e)[0xb6ab196e]
            /lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb66e1a4e]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x0): 
            Connection ID (thread ID): 32
            Status: KILL_CONNECTION

            elenst Elena Stepanova added a comment - For the record, the occasion with a stack trace, however poor: http://buildbot.askmonty.org/buildbot/builders/kvm-deb-lucid-x86/builds/4295/steps/test_4/logs/stdio The server [mysqld.3.2 - pid: 4457, winpid: 4457, exit: 256] crashed while running 'check warnings' Server log from this test: ----------SERVER LOG START----------- 150426 1:26:10 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.5-MariaDB-1~lucid-wsrep-log key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 61901 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0xaee6e008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0xb647c2fc thread_stack 0x48000 /usr/sbin/mysqld(my_print_stacktrace+0x2d)[0xb747537d] /usr/sbin/mysqld(handle_fatal_signal+0x50f)[0xb6f3bc6f] [0xb6acb400] [0xb64b1530] [0xb64d2007] /usr/sbin/mysqld(+0x457708)[0xb6f40708] /usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13st_plugin_intPvEijS3_+0x1fa)[0xb6d77aca] /usr/sbin/mysqld(_Z19ha_close_connectionP3THD+0x3f)[0xb6f3ecaf] /usr/sbin/mysqld(_ZN3THDD0Ev+0x1f7)[0xb6d2b827] /usr/sbin/mysqld(_Z10unlink_thdP3THD+0x193)[0xb6cb5f93] /usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x2d)[0xb6cb60bd] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x140)[0xb6e58f20] /usr/sbin/mysqld(handle_one_connection+0x50)[0xb6e591f0] /lib/tls/i686/cmov/libpthread.so.0(+0x596e)[0xb6ab196e] /lib/tls/i686/cmov/libc.so.6(clone+0x5e)[0xb66e1a4e]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 32 Status: KILL_CONNECTION

            Minimized test case:

            --connect (master_1, localhost, root, , , $MASTER_1_MYPORT, $MASTER_1_MYSOCK)
            --echo # Done

            (it needs to be put under spider/bg/t/).

            elenst Elena Stepanova added a comment - Minimized test case: --connect (master_1, localhost, root, , , $MASTER_1_MYPORT, $MASTER_1_MYSOCK) --echo # Done (it needs to be put under spider/bg/t/ ).
            elenst Elena Stepanova added a comment - - edited

            More notes:

            The most common stack trace that I'm getting with the test above (on perro, on a release server from 10.1.4 bintar) is this:

            #2  <signal handler called>
            #3  0x00007fda40152496 in ?? () from /usr/lib/libjemalloc.so.1
            #4  0x00007fda4015389c in ?? () from /usr/lib/libjemalloc.so.1
            #5  0x00007fda40150094 in free () from /usr/lib/libjemalloc.so.1
            #6  0x0000000000519bc5 in net_end (net=0x7fda37e11240) at /home/buildbot/buildbot/build/sql/net_serv.cc:171
            #7  0x000000000057991f in THD::~THD (this=0x7fda37e11008, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/sql/sql_class.cc:1665
            #8  0x0000000000519018 in unlink_thd (thd=0x7fda37e11008) at /home/buildbot/buildbot/build/sql/mysqld.cc:2867
            #9  0x000000000051907c in one_thread_per_connection_end (thd=0x7fda3ec00af8, put_in_cache=56) at /home/buildbot/buildbot/build/sql/mysqld.cc:2980
            #10 0x000000000067ac7a in do_handle_one_connection (thd_arg=<optimized out>) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1366
            #11 0x000000000067af22 in handle_one_connection (arg=0x7fda37e11008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1258
            #12 0x0000000000ad2409 in pfs_spawn_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1860
            #13 0x00007fda40777e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
            #14 0x00007fda3f443cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x0): 
            Connection ID (thread ID): 1143
            Status: KILL_CONNECTION

            It turns out to have nothing to do with Spider, the test can be put in the main suite.

            -----------------------------------

            Twice I saw it fail this way:

            Could not execute 'check-warnings' for testcase 'main.ha9' (res: 1) server: 'mysqld.1:
            mysqltest: Could not open connection 'default': 2013 Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110 "Connection timed out"
            not ok
            mysqltest failed but provided no output

            ------------------------------

            Here is another variation that I'm getting:

            bug.ha9                                  [ fail ]
                    Test ended at 2015-04-30 00:54:47
             
            CURRENT_TEST: bug.ha9
             
             
            The server [mysqld.1 - pid: 30606, winpid: 30606, exit: 256] crashed while running 'check warnings'
            Server log from this test:
            ----------SERVER LOG START-----------
            150430  0:54:47 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see http://kb.askmonty.org/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.4-MariaDB-wsrep-log
            ----------SERVER LOG END-------------
            mysqltest failed but provided no output
             
             
             - saving '/home/elenst/mariadb-10.1.4-linux-x86_64/mysql-test/var/log/bug.ha9/' to '/home/elenst/mariadb-10.1.4-linux-x86_64/mysql-test/var/log/bug.ha9/'
             - found 'core' (0/5)
             
            Trying 'dbx' to get a backtrace
             
            Trying 'gdb' to get a backtrace
            Core generated by '/home/elenst/mariadb-10.1.4-linux-x86_64/bin/mysqld'
            Output from gdb follows. The first stack trace is from the failing thread.
            The following stack traces are from all threads (so the failing one is
            duplicated).
            --------------------------
            [New LWP 36834]
            [New LWP 36835]
            [New LWP 30607]
            [New LWP 30608]
            [New LWP 30609]
            [New LWP 30610]
             
            warning: Can't read pathname for load map: Unknown error -1.
            [New LWP -1]
            Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
             
            warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
            To enable execution of this file add
            	add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so
            line to your configuration file "/home/elenst/.gdbinit".
            To completely disable this security protection add
            	set auto-load safe-path /
            line to your configuration file "/home/elenst/.gdbinit".
            For more information about this security protection see the
            "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
            	info "(gdb)Auto-loading safe path"
             
            warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
             
            warning: Couldn't find general-purpose registers in core file.
            Error in re-setting breakpoint -1: PC register is not available
            Error in re-setting breakpoint -2: PC register is not available
            Error in re-setting breakpoint -3: PC register is not available
            Error in re-setting breakpoint -4: PC register is not available
            Error in re-setting breakpoint -5: PC register is not available
            Error in re-setting breakpoint -6: PC register is not available
            Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error
             
            warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
             
            warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
            Error in re-setting breakpoint -1: PC register is not available
            Error in re-setting breakpoint -2: PC register is not available
            Error in re-setting breakpoint -3: PC register is not available
            Error in re-setting breakpoint -4: PC register is not available
            Error in re-setting breakpoint -5: PC register is not available
            Error in re-setting breakpoint -6: PC register is not available
            Error in re-setting breakpoint -7: PC register is not available
            Error in re-setting breakpoint -8: PC register is not available
            Error in re-setting breakpoint -9: PC register is not available
            Error in re-setting breakpoint -10: PC register is not available
            Error in re-setting breakpoint -11: PC register is not available
            Error in re-setting breakpoint -12: PC register is not available
            Core was generated by `/home/elenst/mariadb-10.1.4-linux-x86_64/bin/mysqld --defaults-group-suffix=.1'.
            Program terminated with signal SIGSEGV, Segmentation fault.
            #0  0x00007ff33f52eac0 in ?? () from /usr/lib/libjemalloc.so.1
            #0  0x00007ff33f52eac0 in ?? () from /usr/lib/libjemalloc.so.1
            #1  0x00007ff339404d20 in ?? ()
            #2  0x00007ff300000001 in ?? ()
            #3  0x00007ff3370032c8 in ?? ()
            #4  0x00000000ffffffff in ?? ()
            #5  0x00007ff339404d20 in ?? ()
            #6  0x0000000000000001 in ?? ()
            #7  0x00007ff3370032c8 in ?? ()
            #8  0x00007ff3ffffffff in ?? ()
            #9  0x00007ff339404d20 in ?? ()
            #10 0x00007ff300000001 in ?? ()
            #11 0x00007ff3370032c8 in ?? ()
            #12 0x00007ff3ffffffff in ?? ()
            #13 0x00007ff339404d20 in ?? ()
            #14 0x0000000000000001 in ?? ()
            #15 0x00007ff3370032c8 in ?? ()
            #16 0x00007ff3ffffffff in ?? ()
            #17 0x00007ff339404d20 in ?? ()
            #18 0x00007ff300000001 in ?? ()
            #19 0x00007ff3370032c8 in ?? ()
            #20 0x00007ff3ffffffff in ?? ()
            #21 0x00007ff339404d20 in ?? ()
            #22 0x00007ff300000001 in ?? ()
            #23 0x00007ff3370032c8 in ?? ()
            #24 0x00007ff3ffffffff in ?? ()
            #25 0x00007ff339404d20 in ?? ()
            #26 0x00007ff300000001 in ?? ()
            #27 0x00007ff3370032c8 in ?? ()
            #28 0x00007ff3ffffffff in ?? ()
            #29 0x00007ff339404d20 in ?? ()
            #30 0x0000000000000001 in ?? ()
            #31 0x00007ff3370032c8 in ?? ()
            #32 0x00007ff3ffffffff in ?? ()
            #33 0x00007ff339404d20 in ?? ()
            #34 0x0000000000000001 in ?? ()
            #35 0x00007ff3370032c8 in ?? ()
            #36 0x01007ff3ffffffff in ?? ()
            #37 0x00007ff339404d20 in ?? ()
            #38 0x00007ff300000001 in ?? ()
            #39 0x00007ff3370032c8 in ?? ()
            #40 0x00007ff3ffffffff in ?? ()
            #41 0x00007ff339404d20 in ?? ()
            #42 0x00007ff300000001 in ?? ()
            #43 0x00007ff3370032c8 in ?? ()
            #44 0x01007ff3ffffffff in ?? ()
            #45 0x00007ff339404d20 in ?? ()
            #46 0x00007ff300000001 in ?? ()
            #47 0x00007ff3370032c8 in ?? ()
            #48 0x00000000ffffffff in ?? ()
            #49 0x00007ff339404d20 in ?? ()
            #50 0x0000000000000001 in ?? ()
            #51 0x00007ff3370032c8 in ?? ()
            #52 0x00007ff3ffffffff in ?? ()
            #53 0x00007ff339404d20 in ?? ()
            #54 0x00007ff300000001 in ?? ()
            #55 0x00007ff3370032c8 in ?? ()
            #56 0x00000000ffffffff in ?? ()
            #57 0x00007ff339404d20 in ?? ()
            #58 0x00007ff300000001 in ?? ()
            #59 0x00007ff3370032c8 in ?? ()
            #60 0x00007ff3ffffffff in ?? ()
            #61 0x00007ff339404d20 in ?? ()
            #62 0x00007ff300000001 in ?? ()
            #63 0x00007ff3370032c8 in ?? ()
            #64 0x00007ff3ffffffff in ?? ()
            #65 0x00007ff339404d20 in ?? ()
            #66 0x00007ff300000001 in ?? ()
            #67 0x00007ff3370032c8 in ?? ()
            #68 0x00007ff3ffffffff in ?? ()
            #69 0x00007ff339404d20 in ?? ()
            #70 0x00007ff300000001 in ?? ()
            #71 0x00007ff3370032c8 in ?? ()
            #72 0x00000000ffffffff in ?? ()
            #73 0x00007ff339404d20 in ?? ()
            #74 0x00007ff300000001 in ?? ()
            #75 0x00007ff3370032c8 in ?? ()
            #76 0x00007ff3ffffffff in ?? ()
            #77 0x00007ff339404d20 in ?? ()
            #78 0x00007ff300000001 in ?? ()
            #79 0x00007ff3370032c8 in ?? ()
            #80 0x00007ff3ffffffff in ?? ()
            #81 0x00007ff339404d20 in ?? ()
            #82 0x00007ff300000001 in ?? ()
            #83 0x00007ff3370032c8 in ?? ()
            #84 0x00007ff3ffffffff in ?? ()
            #85 0x00007ff339404d20 in ?? ()
            #86 0x00007ff300000001 in ?? ()
            #87 0x00007ff3370032c8 in ?? ()
            #88 0x00000000ffffffff in ?? ()
            #89 0x00007ff339404d20 in ?? ()
            #90 0x0000000000000001 in ?? ()
            #91 0x00007ff3370032c8 in ?? ()
            #92 0x00000000ffffffff in ?? ()
            #93 0x00007ff339404d20 in ?? ()
            #94 0x0000000000000001 in ?? ()
            #95 0x00007ff3370032c8 in ?? ()
            #96 0x00000000ffffffff in ?? ()
            #97 0x00007ff339404d20 in ?? ()
            #98 0x0000000000000001 in ?? ()
            #99 0x00007ff3370032c8 in ?? ()
            #100 0x00000000ffffffff in ?? ()
            #101 0x00007ff339404d20 in ?? ()
            #102 0x0000000000000001 in ?? ()
            #103 0x00007ff3370032c8 in ?? ()
            #104 0x00000000ffffffff in ?? ()
            #105 0x00007ff339404d20 in ?? ()
            #106 0x0000000000000001 in ?? ()
            #107 0x00007ff3370032c8 in ?? ()
            #108 0x00000000ffffffff in ?? ()
            #109 0x00007ff339404d20 in ?? ()
            #110 0x0000000000000001 in ?? ()
            #111 0x00007ff3370032c8 in ?? ()
            #112 0x00000000ffffffff in ?? ()
            #113 0x00007ff339404d20 in ?? ()
            #114 0x00007ff300000001 in ?? ()
            #115 0x00007ff3370032c8 in ?? ()
            #116 0xbff7ef60ffffffff in ?? ()
            #117 0x00007ff339404d20 in ?? ()
            #118 0x0000000000000001 in ?? ()
            #119 0x00007ff3370032c8 in ?? ()
            #120 0x00000000ffffffff in ?? ()
            #121 0x00007ff339404d20 in ?? ()
            #122 0x0000000000000001 in ?? ()
            #123 0x00007ff3370032c8 in ?? ()
            #124 0x00000000ffffffff in ?? ()
            #125 0x00007ff339404d20 in ?? ()
            #126 0x0000000000000001 in ?? ()
            #127 0x00007ff3370032c8 in ?? ()
            #128 0x00000000ffffffff in ?? ()
            #129 0x00007ff339404d20 in ?? ()
            #130 0x0000000000000001 in ?? ()
            #131 0x00007ff3370032c8 in ?? ()
            #132 0x00000000ffffffff in ?? ()
            #133 0x00007ff339404d20 in ?? ()
            #134 0x0000000000000001 in ?? ()
            #135 0x00007ff3370032c8 in ?? ()
            #136 0x00000000ffffffff in ?? ()
            #137 0x00007ff339404d20 in ?? ()
            #138 0x0000000000000001 in ?? ()
            #139 0x00007ff3370032c8 in ?? ()
            #140 0x00000000ffffffff in ?? ()
            #141 0x00007ff339404d20 in ?? ()
            #142 0x0000000000000001 in ?? ()
            #143 0x00007ff3370032c8 in ?? ()
            #144 0x00000000ffffffff in ?? ()
            #145 0x00007ff339404d20 in ?? ()
            #146 0x0000000000000001 in ?? ()
            #147 0x00007ff3370032c8 in ?? ()
            #148 0x00000000ffffffff in ?? ()
            #149 0x00007ff339404d20 in ?? ()
            #150 0x0000000000000001 in ?? ()
            #151 0x00007ff3370032c8 in ?? ()
            #152 0x00000000ffffffff in ?? ()
            #153 0x00007ff339404d20 in ?? ()
            #154 0x0000000000000001 in ?? ()
            #155 0x00007ff3370032c8 in ?? ()
            #156 0x00000000ffffffff in ?? ()
            #157 0x00007ff339404d20 in ?? ()
            #158 0x0000000000000001 in ?? ()
            #159 0x00007ff3370032c8 in ?? ()
            #160 0x00000000ffffffff in ?? ()
            #161 0x00007ff339404d20 in ?? ()
            #162 0x0000000000000001 in ?? ()
            #163 0x00007ff3370032c8 in ?? ()
            #164 0x00000000ffffffff in ?? ()
            #165 0x00007ff339404d20 in ?? ()
            #166 0x0000000000000001 in ?? ()
            #167 0x00007ff3370032c8 in ?? ()
            #168 0x00000000ffffffff in ?? ()
            #169 0x00007ff339404d20 in ?? ()
            #170 0x0000000000000001 in ?? ()
            #171 0x00007ff3370032c8 in ?? ()
            #172 0x00000000ffffffff in ?? ()
            #173 0x00007ff339404d20 in ?? ()
            #174 0x0000000000000001 in ?? ()
            #175 0x00007ff3370032c8 in ?? ()
            #176 0x00000000ffffffff in ?? ()
            #177 0x00007ff339404d20 in ?? ()
            #178 0x0000000000000001 in ?? ()
            #179 0x00007ff3370032c8 in ?? ()
            #180 0x00000000ffffffff in ?? ()
            #181 0x00007ff339404d20 in ?? ()
            #182 0x0000000000000001 in ?? ()
            #183 0x00007ff3370032c8 in ?? ()
            #184 0x00000000ffffffff in ?? ()
            #185 0x00007ff339404d20 in ?? ()
            #186 0x0000000000000001 in ?? ()
            #187 0x00007ff3370032c8 in ?? ()
            #188 0x00000000ffffffff in ?? ()
            #189 0x00007ff339404d20 in ?? ()
            #190 0x0000000000000001 in ?? ()
            #191 0x00007ff3370032c8 in ?? ()
            #192 0x00000000ffffffff in ?? ()
            #193 0x00007ff339404d20 in ?? ()
            #194 0x0000000000000001 in ?? ()
            #195 0x00007ff3370032c8 in ?? ()
            #196 0x00000000ffffffff in ?? ()
            #197 0x00007ff339404d20 in ?? ()
            #198 0x0000000000000001 in ?? ()
            #199 0x00007ff3370032c8 in ?? ()
            #200 0x00000000ffffffff in ?? ()
            #201 0x00007ff339404d20 in ?? ()
            #202 0x0000000000000001 in ?? ()
            #203 0x00007ff3370032c8 in ?? ()
            #204 0x00000000ffffffff in ?? ()
            #205 0x00007ff339404d20 in ?? ()
            #206 0x0000000000000001 in ?? ()
            #207 0x00007ff3370032c8 in ?? ()
            #208 0x00000000ffffffff in ?? ()
            #209 0x00007ff339404d20 in ?? ()
            #210 0x0000000000000001 in ?? ()
            #211 0x00007ff3370032c8 in ?? ()
            #212 0x00000000ffffffff in ?? ()
            #213 0x00007ff339404d20 in ?? ()
            #214 0x0000000000000001 in ?? ()
            #215 0x00007ff3370032c8 in ?? ()
            #216 0x00000000ffffffff in ?? ()
            #217 0x00007ff339404d20 in ?? ()
            #218 0x0000000000000001 in ?? ()
            #219 0x00007ff3370032c8 in ?? ()
            #220 0x00000000ffffffff in ?? ()
            #221 0x00007ff339404d20 in ?? ()
            #222 0x0000000000000001 in ?? ()
            #223 0x00007ff3370032c8 in ?? ()
            #224 0x00000000ffffffff in ?? ()
            #225 0x00007ff339404d20 in ?? ()
            #226 0x0000000000000001 in ?? ()
            #227 0x00007ff3370032c8 in ?? ()
            #228 0x00000000ffffffff in ?? ()
            #229 0x00007ff339404d20 in ?? ()
            #230 0x0000000000000001 in ?? ()
            #231 0x00007ff3370032c8 in ?? ()
            #232 0x00000000ffffffff in ?? ()
            #233 0x00007ff339404d20 in ?? ()
            #234 0x0000000000000001 in ?? ()
            #235 0x00007ff3370032c8 in ?? ()
            #236 0x00000000ffffffff in ?? ()
            #237 0x00007ff339404d20 in ?? ()
            #238 0x0000000000000001 in ?? ()
            #239 0x00007ff3370032c8 in ?? ()
            #240 0x00000000ffffffff in ?? ()
            #241 0x00007ff339404d20 in ?? ()
            #242 0x0000000000000001 in ?? ()
            #243 0x00007ff3370032c8 in ?? ()
            #244 0x00000000ffffffff in ?? ()
            #245 0x00007ff339404d20 in ?? ()
            #246 0x0000000000000001 in ?? ()
            #247 0x00007ff3370032c8 in ?? ()
            #248 0x00000000ffffffff in ?? ()
            #249 0x00007ff339404d20 in ?? ()
            #250 0x0000000000000001 in ?? ()
            #251 0x00007ff3370032c8 in ?? ()
            #252 0x00000000ffffffff in ?? ()
            #253 0x00007ff339404d20 in ?? ()
            #254 0x0000000000000001 in ?? ()
            #255 0x00007ff3370032c8 in ?? ()
            #256 0x00000000ffffffff in ?? ()
            #257 0x00007ff339404d20 in ?? ()
            #258 0x0000000000000001 in ?? ()
            #259 0x00007ff3370032c8 in ?? ()
            #260 0x00000000ffffffff in ?? ()
            #261 0x00007ff339404d20 in ?? ()
            #262 0x0000000000000001 in ?? ()
            #263 0x00007ff3370032c8 in ?? ()
            #264 0x00000000ffffffff in ?? ()
            #265 0x00007ff339404d20 in ?? ()
            #266 0x0000000000000001 in ?? ()
            #267 0x00007ff3370032c8 in ?? ()
            #268 0x00000000ffffffff in ?? ()
            #269 0x00007ff339404d20 in ?? ()
            #270 0x0000000000000001 in ?? ()
            #271 0x00007ff3370032c8 in ?? ()
            #272 0x00000000ffffffff in ?? ()
            #273 0x00007ff339404d20 in ?? ()
            #274 0x0000000000000001 in ?? ()
            #275 0x00007ff3370032c8 in ?? ()
            #276 0x00000000ffffffff in ?? ()
            #277 0x00007ff339404d20 in ?? ()
            #278 0x0000000000000001 in ?? ()
            #279 0x00007ff3370032c8 in ?? ()
            #280 0x00000000ffffffff in ?? ()
            #281 0x00007ff339404d20 in ?? ()
            #282 0x0000000000000001 in ?? ()
            #283 0x00007ff3370032c8 in ?? ()
            #284 0x00000000ffffffff in ?? ()
            #285 0x00007ff339404d20 in ?? ()
            #286 0x0000000000000001 in ?? ()
            #287 0x00007ff3370032c8 in ?? ()
            #288 0x00000000ffffffff in ?? ()
            #289 0x00007ff339404d20 in ?? ()
            #290 0x0000000000000001 in ?? ()
            #291 0x00007ff3370032c8 in ?? ()
            #292 0x00000000ffffffff in ?? ()
            #293 0x00007ff339404d20 in ?? ()
            #294 0x0000000000000001 in ?? ()
            #295 0x00007ff3370032c8 in ?? ()
            #296 0x00000000ffffffff in ?? ()
            #297 0x00007ff339404d20 in ?? ()
            #298 0x0000000000000001 in ?? ()
            #299 0x00007ff3370032c8 in ?? ()
            #300 0x00000000ffffffff in ?? ()
            #301 0x00007ff339404d20 in ?? ()
            #302 0x0000000000000001 in ?? ()
            #303 0x00007ff3370032c8 in ?? ()
            #304 0x00000000ffffffff in ?? ()
            #305 0x00007ff339404d20 in ?? ()
            #306 0x0000000000000001 in ?? ()
            #307 0x00007ff3370032c8 in ?? ()
            #308 0x00000000ffffffff in ?? ()
            #309 0x00007ff339404d20 in ?? ()
            #310 0x0000000000000001 in ?? ()
            #311 0x00007ff3370032c8 in ?? ()
            #312 0x00000000ffffffff in ?? ()
            #313 0x00007ff339404d20 in ?? ()
            #314 0x0000000000000001 in ?? ()
            #315 0x00007ff3370032c8 in ?? ()
            #316 0x00000000ffffffff in ?? ()
            #317 0x00007ff339404d20 in ?? ()
            #318 0x0000000000000001 in ?? ()
            #319 0x00007ff3370032c8 in ?? ()
            #320 0x00000000ffffffff in ?? ()
            #321 0x00007ff339404d20 in ?? ()
            #322 0x0000000000000001 in ?? ()
            #323 0x00007ff3370032c8 in ?? ()
            #324 0x00000000ffffffff in ?? ()
            #325 0x00007ff339404d20 in ?? ()
            #326 0x0000000000000001 in ?? ()
            #327 0x00007ff3370032c8 in ?? ()
            #328 0x00000000ffffffff in ?? ()
            #329 0x00007ff339404d20 in ?? ()
            #330 0x0000000000000001 in ?? ()
            #331 0x00007ff3370032c8 in ?? ()
            #332 0x00000000ffffffff in ?? ()
            #333 0x00007ff339404d20 in ?? ()
            #334 0x0000000000000001 in ?? ()
            #335 0x00007ff3370032c8 in ?? ()
            #336 0x00007ff3ffffffff in ?? ()
            #337 0x00007ff339404d20 in ?? ()
            #338 0x0000000000000001 in ?? ()
            #339 0x00007ff3370032c8 in ?? ()
            #340 0x00007ff3ffffffff in ?? ()
            #341 0x00007ff339404d20 in ?? ()
            #342 0x0000000000000001 in ?? ()
            #343 0x00007ff3370032c8 in ?? ()
            #344 0x00000000ffffffff in ?? ()
            #345 0x00007ff339404d20 in ?? ()
            #346 0x0000000000000001 in ?? ()
            #347 0x00007ff3370032c8 in ?? ()
            #348 0x00000000ffffffff in ?? ()
            #349 0x00007ff339404d20 in ?? ()
            #350 0x0000000000000001 in ?? ()
            #351 0x00007ff3370032c8 in ?? ()
            #352 0x00000000ffffffff in ?? ()
            #353 0x00007ff339404d20 in ?? ()
            #354 0x00007ff300000001 in ?? ()
            #355 0x00007ff3370032c8 in ?? ()
            #356 0x00000000ffffffff in ?? ()
            #357 0x00007ff339404d20 in ?? ()
            #358 0x00007ff300000001 in ?? ()
            #359 0x00007ff3370032c8 in ?? ()
            #360 0x00007ff3ffffffff in ?? ()
            #361 0x00007ff339404d20 in ?? ()
            #362 0x0000000000000001 in ?? ()
            #363 0x00007ff3370032c8 in ?? ()
            #364 0xfc882008ffffffff in ?? ()
            #365 0x00007ff339404d20 in ?? ()
            #366 0x0000000000000001 in ?? ()
            #367 0x00007ff3370032c8 in ?? ()
            #368 0x00000000ffffffff in ?? ()
            #369 0x00007ff339404d20 in ?? ()
            #370 0x0000000000000001 in ?? ()
            #371 0x00007ff3370032c8 in ?? ()
            #372 0x00000000ffffffff in ?? ()
            #373 0x00007ff339404d20 in ?? ()
            #374 0x0000000000000001 in ?? ()
            #375 0x00007ff3370032c8 in ?? ()
            #376 0x00000000ffffffff in ?? ()
            #377 0x00007ff339404d20 in ?? ()
            #378 0x0000000000000001 in ?? ()
            #379 0x00007ff3370032c8 in ?? ()
            #380 0x00000000ffffffff in ?? ()
            #381 0x00007ff339404d20 in ?? ()
            #382 0x0000000000000001 in ?? ()
            #383 0x00007ff3370032c8 in ?? ()
            #384 0x00000000ffffffff in ?? ()
            #385 0x00007ff339404d20 in ?? ()
            #386 0x0000000000000001 in ?? ()
            #387 0x00007ff3370032c8 in ?? ()
            #388 0x00000000ffffffff in ?? ()
            #389 0x00007ff339404d20 in ?? ()
            #390 0x0000000000000001 in ?? ()
            #391 0x00007ff3370032c8 in ?? ()
            #392 0x00000000ffffffff in ?? ()
            #393 0x00007ff339404d20 in ?? ()
            #394 0x0000000000000001 in ?? ()
            #395 0x00007ff3370032c8 in ?? ()
            #396 0x00000000ffffffff in ?? ()
            #397 0x00007ff339404d20 in ?? ()
            #398 0x0000000000000001 in ?? ()
            #399 0x00007ff3370032c8 in ?? ()
            #400 0x00000000ffffffff in ?? ()
            #401 0x00007ff339404d20 in ?? ()
            #402 0x0000000000000001 in ?? ()
            #403 0x00007ff3370032c8 in ?? ()
            #404 0x00000000ffffffff in ?? ()
            #405 0x00007ff339404d20 in ?? ()
            #406 0x0000000000000001 in ?? ()
            #407 0x00007ff3370032c8 in ?? ()
            #408 0x00000000ffffffff in ?? ()
            #409 0x00007ff339404d20 in ?? ()
            #410 0x0000000000000001 in ?? ()
            #411 0x00007ff3370032c8 in ?? ()
            #412 0x00000000ffffffff in ?? ()
            #413 0x00007ff339404d20 in ?? ()
            #414 0x0000000000000001 in ?? ()
            #415 0x00007ff3370032c8 in ?? ()
            #416 0x00000000ffffffff in ?? ()
            #417 0x00007ff339404d20 in ?? ()
            #418 0x0000000000000001 in ?? ()
            #419 0x00007ff3370032c8 in ?? ()
            #420 0x00000000ffffffff in ?? ()
            #421 0x00007ff339404d20 in ?? ()
            #422 0x0000000000000001 in ?? ()
            #423 0x00007ff3370032c8 in ?? ()
            #424 0x00000000ffffffff in ?? ()
            #425 0x00007ff339404d20 in ?? ()
            #426 0x0000000000000001 in ?? ()
            #427 0x00007ff3370032c8 in ?? ()
            #428 0x00000000ffffffff in ?? ()
            #429 0x00007ff339404d20 in ?? ()
            #430 0x0000000000000001 in ?? ()
            #431 0x00007ff3370032c8 in ?? ()
            #432 0x00000000ffffffff in ?? ()
            #433 0x00007ff339404d20 in ?? ()
            #434 0x0000000000000001 in ?? ()
            #435 0x00007ff3370032c8 in ?? ()
            #436 0x00000000ffffffff in ?? ()
            #437 0x00007ff339404d20 in ?? ()
            #438 0x0000000000000001 in ?? ()
            #439 0x00007ff3370032c8 in ?? ()
            #440 0x00000000ffffffff in ?? ()
            #441 0x00007ff339404d20 in ?? ()
            #442 0x0000000000000001 in ?? ()
            #443 0x00007ff3370032c8 in ?? ()
            #444 0x00000000ffffffff in ?? ()
            #445 0x00007ff339404d20 in ?? ()
            #446 0x00007ff300000001 in ?? ()
            #447 0x00007ff3370032c8 in ?? ()
            #448 0x0000778fffffffff in ?? ()
            #449 0x00007ff339404d20 in ?? ()
            #450 0x00007ff300000001 in ?? ()
            #451 0x00007ff3370032c8 in ?? ()
            #452 0x00000000ffffffff in ?? ()
            #453 0x00007ff339404d20 in ?? ()
            #454 0x00007ff300000001 in ?? ()
            #455 0x00007ff3370032c8 in ?? ()
            #456 0x00000000ffffffff in ?? ()
            #457 0x00007ff339404d20 in ?? ()
            #458 0x0000000000000001 in ?? ()
            #459 0x00007ff3370032c8 in ?? ()
            #460 0x00000000ffffffff in ?? ()
            #461 0x00007ff339404d20 in ?? ()
            #462 0x0000000000000001 in ?? ()
            #463 0x00007ff3370032c8 in ?? ()
            #464 0x00000000ffffffff in ?? ()
            #465 0x00007ff339404d20 in ?? ()
            #466 0x0000000000000001 in ?? ()
            #467 0x00007ff3370032c8 in ?? ()
            #468 0x00000000ffffffff in ?? ()
            #469 0x00007ff339404d20 in ?? ()
            #470 0x0000000000000001 in ?? ()
            #471 0x00007ff3370032c8 in ?? ()
            #472 0x00000000ffffffff in ?? ()
            #473 0x00007ff339404d20 in ?? ()
            #474 0x0000000000000001 in ?? ()
            #475 0x00007ff3370032c8 in ?? ()
            #476 0x00000000ffffffff in ?? ()
            #477 0x00007ff339404d20 in ?? ()
            #478 0x0000000000000001 in ?? ()
            #479 0x00007ff3370032c8 in ?? ()
            #480 0x00000000ffffffff in ?? ()
            #481 0x00007ff339404d20 in ?? ()
            #482 0x0000000000000001 in ?? ()
            #483 0x00007ff3370032c8 in ?? ()
            #484 0x00000000ffffffff in ?? ()
            #485 0x00007ff339404d20 in ?? ()
            #486 0x0000000000000001 in ?? ()
            #487 0x00007ff3370032c8 in ?? ()
            #488 0x00000000ffffffff in ?? ()
            #489 0x00007ff339404d20 in ?? ()
            #490 0x0000000000000001 in ?? ()
            #491 0x00007ff3370032c8 in ?? ()
            #492 0x00000000ffffffff in ?? ()
            #493 0x00007ff339404d20 in ?? ()
            #494 0x0000000000000001 in ?? ()
            #495 0x00007ff3370032c8 in ?? ()
            #496 0x00000000ffffffff in ?? ()
            #497 0x00007ff339404d20 in ?? ()
            #498 0x0000000000000001 in ?? ()
            #499 0x00007ff3370032c8 in ?? ()
            #500 0x00000000ffffffff in ?? ()
            #501 0x00007ff339404d20 in ?? ()
            #502 0x0000000000000001 in ?? ()
            #503 0x00007ff3370032c8 in ?? ()
            #504 0x00000000ffffffff in ?? ()
            #505 0x00007ff339404d20 in ?? ()
            #506 0x0000000000000001 in ?? ()
            #507 0x00007ff3370032c8 in ?? ()
            #508 0x00000000ffffffff in ?? ()
            #509 0x00007ff339404d20 in ?? ()
            #510 0x0000000000000001 in ?? ()
            #511 0x00007ff3370032c8 in ?? ()
            #512 0x00000000ffffffff in ?? ()
            #513 0x00007ff339404d20 in ?? ()
            #514 0x0000000000000001 in ?? ()
            #515 0x00007ff3370032c8 in ?? ()
            #516 0x00000000ffffffff in ?? ()
            #517 0x00007ff339404d20 in ?? ()
            #518 0x0000000000000001 in ?? ()
            #519 0x00007ff3370032c8 in ?? ()
            #520 0x00007ff3ffffffff in ?? ()
            #521 0x00007ff339404d20 in ?? ()
            #522 0x0000000000000001 in ?? ()
            #523 0x00007ff3370032c8 in ?? ()
            #524 0x00000000ffffffff in ?? ()
            #525 0x00007ff339404d20 in ?? ()
            #526 0x0000000000000001 in ?? ()
            #527 0x00007ff3370032c8 in ?? ()
            #528 0x00000000ffffffff in ?? ()
            #529 0x00007ff339404d20 in ?? ()
            #530 0x0000000000000001 in ?? ()
            #531 0x00007ff3370032c8 in ?? ()
            #532 0x00000000ffffffff in ?? ()
            #533 0x00007ff339404d20 in ?? ()
            #534 0x0000000000000001 in ?? ()
            #535 0x00007ff3370032c8 in ?? ()
            #536 0x00000000ffffffff in ?? ()
            #537 0x00007ff339404d20 in ?? ()
            #538 0x0000000000000001 in ?? ()
            #539 0x00007ff3370032c8 in ?? ()
            #540 0x00000000ffffffff in ?? ()
            #541 0x00007ff339404d20 in ?? ()
            #542 0x0000000000000001 in ?? ()
            #543 0x00007ff3370032c8 in ?? ()
            #544 0x00000000ffffffff in ?? ()
            #545 0x00007ff339404d20 in ?? ()
            #546 0x0000000000000001 in ?? ()
            #547 0x00007ff3370032c8 in ?? ()
            #548 0x00000000ffffffff in ?? ()
            #549 0x00007ff339404d20 in ?? ()
            #550 0x0000000000000001 in ?? ()
            #551 0x00007ff3370032c8 in ?? ()
            #552 0x00000000ffffffff in ?? ()
            #553 0x00007ff339404d20 in ?? ()
            #554 0x00f6640300000001 in ?? ()
            #555 0x00007ff3370032c8 in ?? ()
            #556 0x00000000ffffffff in ?? ()
            #557 0x00007ff339404d20 in ?? ()
            #558 0x0000000000000001 in ?? ()
            #559 0x00007ff3370032c8 in ?? ()
            #560 0x00000000ffffffff in ?? ()
            #561 0x00007ff339404d20 in ?? ()
            #562 0x0000000000000001 in ?? ()
            #563 0x00007ff3370032c8 in ?? ()
            #564 0x00000000ffffffff in ?? ()
            #565 0x00007ff339404d20 in ?? ()
            #566 0x00007ff300000001 in ?? ()
            #567 0x00007ff3370032c8 in ?? ()
            #568 0x00007ff3ffffffff in ?? ()
            #569 0x00007ff339404d20 in ?? ()
            #570 0x0000000000000001 in ?? ()
            #571 0x00007ff3370032c8 in ?? ()
            #572 0x00000000ffffffff in ?? ()
            #573 0x00007ff339404d20 in ?? ()
            #574 0x0000000000000001 in ?? ()
            #575 0x00007ff3370032c8 in ?? ()
            #576 0x00000000ffffffff in ?? ()
            #577 0x00007ff339404d20 in ?? ()
            #578 0x0000000000000001 in ?? ()
            #579 0x00007ff3370032c8 in ?? ()
            #580 0x00000000ffffffff in ?? ()
            #581 0x00007ff339404d20 in ?? ()
            #582 0x0000000000000001 in ?? ()
            #583 0x00007ff3370032c8 in ?? ()
            #584 0x00000000ffffffff in ?? ()
            #585 0x00007ff339404d20 in ?? ()
            #586 0x0000000000000001 in ?? ()
            #587 0x00007ff3370032c8 in ?? ()
            #588 0x00000000ffffffff in ?? ()
            #589 0x00007ff339404d20 in ?? ()
            #590 0x0000000000000001 in ?? ()
            #591 0x00007ff3370032c8 in ?? ()
            #592 0x00000000ffffffff in ?? ()
            #593 0x00007ff339404d20 in ?? ()
            #594 0x0000000000000001 in ?? ()
            #595 0x00007ff3370032c8 in ?? ()
            #596 0x00000000ffffffff in ?? ()
            #597 0x00007ff339404d20 in ?? ()
            #598 0x0000000000000001 in ?? ()
            #599 0x00007ff3370032c8 in ?? ()
            #600 0x00000000ffffffff in ?? ()
            #601 0x00007ff339404d20 in ?? ()
            #602 0x0000000000000001 in ?? ()
            #603 0x00007ff3370032c8 in ?? ()
            #604 0x00000000ffffffff in ?? ()
            #605 0x00007ff339404d20 in ?? ()
            #606 0x0000000000000001 in ?? ()
            #607 0x00007ff3370032c8 in ?? ()
            #608 0x00000000ffffffff in ?? ()
            #609 0x00007ff339404d20 in ?? ()
            #610 0x0000000000000001 in ?? ()
            #611 0x00007ff3370032c8 in ?? ()
            #612 0x00000000ffffffff in ?? ()
            #613 0x00007ff339404d20 in ?? ()
            #614 0x0000000000000001 in ?? ()
            #615 0x00007ff3370032c8 in ?? ()
            #616 0x00000000ffffffff in ?? ()
            #617 0x00007ff339404d20 in ?? ()
            #618 0x0000000000000001 in ?? ()
            #619 0x00007ff3370032c8 in ?? ()
            #620 0x00000000ffffffff in ?? ()
            #621 0x00007ff339404d20 in ?? ()
            #622 0x0000000000000001 in ?? ()
            #623 0x00007ff3370032c8 in ?? ()
            #624 0x00000000ffffffff in ?? ()
            #625 0x00007ff339404d20 in ?? ()
            #626 0x0000000000000001 in ?? ()
            #627 0x00007ff3370032c8 in ?? ()
            #628 0x00000000ffffffff in ?? ()
            #629 0x00007ff339404d20 in ?? ()
            #630 0x0000000000000001 in ?? ()
            #631 0x00007ff3370032c8 in ?? ()
            #632 0x00000000ffffffff in ?? ()
            #633 0x00007ff339404d20 in ?? ()
            #634 0x0000000000000001 in ?? ()
            #635 0x00007ff3370032c8 in ?? ()
            #636 0x00000000ffffffff in ?? ()
            #637 0x00007ff339404d20 in ?? ()
            #638 0x0000000000000001 in ?? ()
            #639 0x00007ff3370032c8 in ?? ()
            #640 0x00000000ffffffff in ?? ()
            #641 0x00007ff339404d20 in ?? ()
            #642 0x0000000000000001 in ?? ()
            #643 0x00007ff3370032c8 in ?? ()
            #644 0x00000000ffffffff in ?? ()
            #645 0x00007ff339404d20 in ?? ()
            #646 0x0000000000000000 in ?? ()
            /tmp/gdNTzSQOIS:2: Error in sourced command file:
            PC register is not available
            PC register is not available

            elenst Elena Stepanova added a comment - - edited More notes: The most common stack trace that I'm getting with the test above (on perro, on a release server from 10.1.4 bintar) is this: #2 <signal handler called> #3 0x00007fda40152496 in ?? () from /usr/lib/libjemalloc.so.1 #4 0x00007fda4015389c in ?? () from /usr/lib/libjemalloc.so.1 #5 0x00007fda40150094 in free () from /usr/lib/libjemalloc.so.1 #6 0x0000000000519bc5 in net_end (net=0x7fda37e11240) at /home/buildbot/buildbot/build/sql/net_serv.cc:171 #7 0x000000000057991f in THD::~THD (this=0x7fda37e11008, __in_chrg=<optimized out>) at /home/buildbot/buildbot/build/sql/sql_class.cc:1665 #8 0x0000000000519018 in unlink_thd (thd=0x7fda37e11008) at /home/buildbot/buildbot/build/sql/mysqld.cc:2867 #9 0x000000000051907c in one_thread_per_connection_end (thd=0x7fda3ec00af8, put_in_cache=56) at /home/buildbot/buildbot/build/sql/mysqld.cc:2980 #10 0x000000000067ac7a in do_handle_one_connection (thd_arg=<optimized out>) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1366 #11 0x000000000067af22 in handle_one_connection (arg=0x7fda37e11008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1258 #12 0x0000000000ad2409 in pfs_spawn_thread (arg=<optimized out>) at /home/buildbot/buildbot/build/storage/perfschema/pfs.cc:1860 #13 0x00007fda40777e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #14 0x00007fda3f443cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 1143 Status: KILL_CONNECTION It turns out to have nothing to do with Spider, the test can be put in the main suite. ----------------------------------- Twice I saw it fail this way: Could not execute 'check-warnings' for testcase 'main.ha9' (res: 1) server: 'mysqld.1: mysqltest: Could not open connection 'default': 2013 Lost connection to MySQL server at 'waiting for initial communication packet', system error: 110 "Connection timed out" not ok mysqltest failed but provided no output ------------------------------ Here is another variation that I'm getting: bug.ha9 [ fail ] Test ended at 2015-04-30 00:54:47   CURRENT_TEST: bug.ha9     The server [mysqld.1 - pid: 30606, winpid: 30606, exit: 256] crashed while running 'check warnings' Server log from this test: ----------SERVER LOG START----------- 150430 0:54:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see http://kb.askmonty.org/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.4-MariaDB-wsrep-log ----------SERVER LOG END------------- mysqltest failed but provided no output     - saving '/home/elenst/mariadb-10.1.4-linux-x86_64/mysql-test/var/log/bug.ha9/' to '/home/elenst/mariadb-10.1.4-linux-x86_64/mysql-test/var/log/bug.ha9/' - found 'core' (0/5)   Trying 'dbx' to get a backtrace   Trying 'gdb' to get a backtrace Core generated by '/home/elenst/mariadb-10.1.4-linux-x86_64/bin/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 36834] [New LWP 36835] [New LWP 30607] [New LWP 30608] [New LWP 30609] [New LWP 30610]   warning: Can't read pathname for load map: Unknown error -1. [New LWP -1] Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error   warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load". To enable execution of this file add add-auto-load-safe-path /lib/x86_64-linux-gnu/libthread_db-1.0.so line to your configuration file "/home/elenst/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/home/elenst/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path"   warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.   warning: Couldn't find general-purpose registers in core file. Error in re-setting breakpoint -1: PC register is not available Error in re-setting breakpoint -2: PC register is not available Error in re-setting breakpoint -3: PC register is not available Error in re-setting breakpoint -4: PC register is not available Error in re-setting breakpoint -5: PC register is not available Error in re-setting breakpoint -6: PC register is not available Warning: couldn't activate thread debugging using libthread_db: Cannot find new threads: generic error   warning: File "/lib/x86_64-linux-gnu/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".   warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Error in re-setting breakpoint -1: PC register is not available Error in re-setting breakpoint -2: PC register is not available Error in re-setting breakpoint -3: PC register is not available Error in re-setting breakpoint -4: PC register is not available Error in re-setting breakpoint -5: PC register is not available Error in re-setting breakpoint -6: PC register is not available Error in re-setting breakpoint -7: PC register is not available Error in re-setting breakpoint -8: PC register is not available Error in re-setting breakpoint -9: PC register is not available Error in re-setting breakpoint -10: PC register is not available Error in re-setting breakpoint -11: PC register is not available Error in re-setting breakpoint -12: PC register is not available Core was generated by `/home/elenst/mariadb-10.1.4-linux-x86_64/bin/mysqld --defaults-group-suffix=.1'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007ff33f52eac0 in ?? () from /usr/lib/libjemalloc.so.1 #0 0x00007ff33f52eac0 in ?? () from /usr/lib/libjemalloc.so.1 #1 0x00007ff339404d20 in ?? () #2 0x00007ff300000001 in ?? () #3 0x00007ff3370032c8 in ?? () #4 0x00000000ffffffff in ?? () #5 0x00007ff339404d20 in ?? () #6 0x0000000000000001 in ?? () #7 0x00007ff3370032c8 in ?? () #8 0x00007ff3ffffffff in ?? () #9 0x00007ff339404d20 in ?? () #10 0x00007ff300000001 in ?? () #11 0x00007ff3370032c8 in ?? () #12 0x00007ff3ffffffff in ?? () #13 0x00007ff339404d20 in ?? () #14 0x0000000000000001 in ?? () #15 0x00007ff3370032c8 in ?? () #16 0x00007ff3ffffffff in ?? () #17 0x00007ff339404d20 in ?? () #18 0x00007ff300000001 in ?? () #19 0x00007ff3370032c8 in ?? () #20 0x00007ff3ffffffff in ?? () #21 0x00007ff339404d20 in ?? () #22 0x00007ff300000001 in ?? () #23 0x00007ff3370032c8 in ?? () #24 0x00007ff3ffffffff in ?? () #25 0x00007ff339404d20 in ?? () #26 0x00007ff300000001 in ?? () #27 0x00007ff3370032c8 in ?? () #28 0x00007ff3ffffffff in ?? () #29 0x00007ff339404d20 in ?? () #30 0x0000000000000001 in ?? () #31 0x00007ff3370032c8 in ?? () #32 0x00007ff3ffffffff in ?? () #33 0x00007ff339404d20 in ?? () #34 0x0000000000000001 in ?? () #35 0x00007ff3370032c8 in ?? () #36 0x01007ff3ffffffff in ?? () #37 0x00007ff339404d20 in ?? () #38 0x00007ff300000001 in ?? () #39 0x00007ff3370032c8 in ?? () #40 0x00007ff3ffffffff in ?? () #41 0x00007ff339404d20 in ?? () #42 0x00007ff300000001 in ?? () #43 0x00007ff3370032c8 in ?? () #44 0x01007ff3ffffffff in ?? () #45 0x00007ff339404d20 in ?? () #46 0x00007ff300000001 in ?? () #47 0x00007ff3370032c8 in ?? () #48 0x00000000ffffffff in ?? () #49 0x00007ff339404d20 in ?? () #50 0x0000000000000001 in ?? () #51 0x00007ff3370032c8 in ?? () #52 0x00007ff3ffffffff in ?? () #53 0x00007ff339404d20 in ?? () #54 0x00007ff300000001 in ?? () #55 0x00007ff3370032c8 in ?? () #56 0x00000000ffffffff in ?? () #57 0x00007ff339404d20 in ?? () #58 0x00007ff300000001 in ?? () #59 0x00007ff3370032c8 in ?? () #60 0x00007ff3ffffffff in ?? () #61 0x00007ff339404d20 in ?? () #62 0x00007ff300000001 in ?? () #63 0x00007ff3370032c8 in ?? () #64 0x00007ff3ffffffff in ?? () #65 0x00007ff339404d20 in ?? () #66 0x00007ff300000001 in ?? () #67 0x00007ff3370032c8 in ?? () #68 0x00007ff3ffffffff in ?? () #69 0x00007ff339404d20 in ?? () #70 0x00007ff300000001 in ?? () #71 0x00007ff3370032c8 in ?? () #72 0x00000000ffffffff in ?? () #73 0x00007ff339404d20 in ?? () #74 0x00007ff300000001 in ?? () #75 0x00007ff3370032c8 in ?? () #76 0x00007ff3ffffffff in ?? () #77 0x00007ff339404d20 in ?? () #78 0x00007ff300000001 in ?? () #79 0x00007ff3370032c8 in ?? () #80 0x00007ff3ffffffff in ?? () #81 0x00007ff339404d20 in ?? () #82 0x00007ff300000001 in ?? () #83 0x00007ff3370032c8 in ?? () #84 0x00007ff3ffffffff in ?? () #85 0x00007ff339404d20 in ?? () #86 0x00007ff300000001 in ?? () #87 0x00007ff3370032c8 in ?? () #88 0x00000000ffffffff in ?? () #89 0x00007ff339404d20 in ?? () #90 0x0000000000000001 in ?? () #91 0x00007ff3370032c8 in ?? () #92 0x00000000ffffffff in ?? () #93 0x00007ff339404d20 in ?? () #94 0x0000000000000001 in ?? () #95 0x00007ff3370032c8 in ?? () #96 0x00000000ffffffff in ?? () #97 0x00007ff339404d20 in ?? () #98 0x0000000000000001 in ?? () #99 0x00007ff3370032c8 in ?? () #100 0x00000000ffffffff in ?? () #101 0x00007ff339404d20 in ?? () #102 0x0000000000000001 in ?? () #103 0x00007ff3370032c8 in ?? () #104 0x00000000ffffffff in ?? () #105 0x00007ff339404d20 in ?? () #106 0x0000000000000001 in ?? () #107 0x00007ff3370032c8 in ?? () #108 0x00000000ffffffff in ?? () #109 0x00007ff339404d20 in ?? () #110 0x0000000000000001 in ?? () #111 0x00007ff3370032c8 in ?? () #112 0x00000000ffffffff in ?? () #113 0x00007ff339404d20 in ?? () #114 0x00007ff300000001 in ?? () #115 0x00007ff3370032c8 in ?? () #116 0xbff7ef60ffffffff in ?? () #117 0x00007ff339404d20 in ?? () #118 0x0000000000000001 in ?? () #119 0x00007ff3370032c8 in ?? () #120 0x00000000ffffffff in ?? () #121 0x00007ff339404d20 in ?? () #122 0x0000000000000001 in ?? () #123 0x00007ff3370032c8 in ?? () #124 0x00000000ffffffff in ?? () #125 0x00007ff339404d20 in ?? () #126 0x0000000000000001 in ?? () #127 0x00007ff3370032c8 in ?? () #128 0x00000000ffffffff in ?? () #129 0x00007ff339404d20 in ?? () #130 0x0000000000000001 in ?? () #131 0x00007ff3370032c8 in ?? () #132 0x00000000ffffffff in ?? () #133 0x00007ff339404d20 in ?? () #134 0x0000000000000001 in ?? () #135 0x00007ff3370032c8 in ?? () #136 0x00000000ffffffff in ?? () #137 0x00007ff339404d20 in ?? () #138 0x0000000000000001 in ?? () #139 0x00007ff3370032c8 in ?? () #140 0x00000000ffffffff in ?? () #141 0x00007ff339404d20 in ?? () #142 0x0000000000000001 in ?? () #143 0x00007ff3370032c8 in ?? () #144 0x00000000ffffffff in ?? () #145 0x00007ff339404d20 in ?? () #146 0x0000000000000001 in ?? () #147 0x00007ff3370032c8 in ?? () #148 0x00000000ffffffff in ?? () #149 0x00007ff339404d20 in ?? () #150 0x0000000000000001 in ?? () #151 0x00007ff3370032c8 in ?? () #152 0x00000000ffffffff in ?? () #153 0x00007ff339404d20 in ?? () #154 0x0000000000000001 in ?? () #155 0x00007ff3370032c8 in ?? () #156 0x00000000ffffffff in ?? () #157 0x00007ff339404d20 in ?? () #158 0x0000000000000001 in ?? () #159 0x00007ff3370032c8 in ?? () #160 0x00000000ffffffff in ?? () #161 0x00007ff339404d20 in ?? () #162 0x0000000000000001 in ?? () #163 0x00007ff3370032c8 in ?? () #164 0x00000000ffffffff in ?? () #165 0x00007ff339404d20 in ?? () #166 0x0000000000000001 in ?? () #167 0x00007ff3370032c8 in ?? () #168 0x00000000ffffffff in ?? () #169 0x00007ff339404d20 in ?? () #170 0x0000000000000001 in ?? () #171 0x00007ff3370032c8 in ?? () #172 0x00000000ffffffff in ?? () #173 0x00007ff339404d20 in ?? () #174 0x0000000000000001 in ?? () #175 0x00007ff3370032c8 in ?? () #176 0x00000000ffffffff in ?? () #177 0x00007ff339404d20 in ?? () #178 0x0000000000000001 in ?? () #179 0x00007ff3370032c8 in ?? () #180 0x00000000ffffffff in ?? () #181 0x00007ff339404d20 in ?? () #182 0x0000000000000001 in ?? () #183 0x00007ff3370032c8 in ?? () #184 0x00000000ffffffff in ?? () #185 0x00007ff339404d20 in ?? () #186 0x0000000000000001 in ?? () #187 0x00007ff3370032c8 in ?? () #188 0x00000000ffffffff in ?? () #189 0x00007ff339404d20 in ?? () #190 0x0000000000000001 in ?? () #191 0x00007ff3370032c8 in ?? () #192 0x00000000ffffffff in ?? () #193 0x00007ff339404d20 in ?? () #194 0x0000000000000001 in ?? () #195 0x00007ff3370032c8 in ?? () #196 0x00000000ffffffff in ?? () #197 0x00007ff339404d20 in ?? () #198 0x0000000000000001 in ?? () #199 0x00007ff3370032c8 in ?? () #200 0x00000000ffffffff in ?? () #201 0x00007ff339404d20 in ?? () #202 0x0000000000000001 in ?? () #203 0x00007ff3370032c8 in ?? () #204 0x00000000ffffffff in ?? () #205 0x00007ff339404d20 in ?? () #206 0x0000000000000001 in ?? () #207 0x00007ff3370032c8 in ?? () #208 0x00000000ffffffff in ?? () #209 0x00007ff339404d20 in ?? () #210 0x0000000000000001 in ?? () #211 0x00007ff3370032c8 in ?? () #212 0x00000000ffffffff in ?? () #213 0x00007ff339404d20 in ?? () #214 0x0000000000000001 in ?? () #215 0x00007ff3370032c8 in ?? () #216 0x00000000ffffffff in ?? () #217 0x00007ff339404d20 in ?? () #218 0x0000000000000001 in ?? () #219 0x00007ff3370032c8 in ?? () #220 0x00000000ffffffff in ?? () #221 0x00007ff339404d20 in ?? () #222 0x0000000000000001 in ?? () #223 0x00007ff3370032c8 in ?? () #224 0x00000000ffffffff in ?? () #225 0x00007ff339404d20 in ?? () #226 0x0000000000000001 in ?? () #227 0x00007ff3370032c8 in ?? () #228 0x00000000ffffffff in ?? () #229 0x00007ff339404d20 in ?? () #230 0x0000000000000001 in ?? () #231 0x00007ff3370032c8 in ?? () #232 0x00000000ffffffff in ?? () #233 0x00007ff339404d20 in ?? () #234 0x0000000000000001 in ?? () #235 0x00007ff3370032c8 in ?? () #236 0x00000000ffffffff in ?? () #237 0x00007ff339404d20 in ?? () #238 0x0000000000000001 in ?? () #239 0x00007ff3370032c8 in ?? () #240 0x00000000ffffffff in ?? () #241 0x00007ff339404d20 in ?? () #242 0x0000000000000001 in ?? () #243 0x00007ff3370032c8 in ?? () #244 0x00000000ffffffff in ?? () #245 0x00007ff339404d20 in ?? () #246 0x0000000000000001 in ?? () #247 0x00007ff3370032c8 in ?? () #248 0x00000000ffffffff in ?? () #249 0x00007ff339404d20 in ?? () #250 0x0000000000000001 in ?? () #251 0x00007ff3370032c8 in ?? () #252 0x00000000ffffffff in ?? () #253 0x00007ff339404d20 in ?? () #254 0x0000000000000001 in ?? () #255 0x00007ff3370032c8 in ?? () #256 0x00000000ffffffff in ?? () #257 0x00007ff339404d20 in ?? () #258 0x0000000000000001 in ?? () #259 0x00007ff3370032c8 in ?? () #260 0x00000000ffffffff in ?? () #261 0x00007ff339404d20 in ?? () #262 0x0000000000000001 in ?? () #263 0x00007ff3370032c8 in ?? () #264 0x00000000ffffffff in ?? () #265 0x00007ff339404d20 in ?? () #266 0x0000000000000001 in ?? () #267 0x00007ff3370032c8 in ?? () #268 0x00000000ffffffff in ?? () #269 0x00007ff339404d20 in ?? () #270 0x0000000000000001 in ?? () #271 0x00007ff3370032c8 in ?? () #272 0x00000000ffffffff in ?? () #273 0x00007ff339404d20 in ?? () #274 0x0000000000000001 in ?? () #275 0x00007ff3370032c8 in ?? () #276 0x00000000ffffffff in ?? () #277 0x00007ff339404d20 in ?? () #278 0x0000000000000001 in ?? () #279 0x00007ff3370032c8 in ?? () #280 0x00000000ffffffff in ?? () #281 0x00007ff339404d20 in ?? () #282 0x0000000000000001 in ?? () #283 0x00007ff3370032c8 in ?? () #284 0x00000000ffffffff in ?? () #285 0x00007ff339404d20 in ?? () #286 0x0000000000000001 in ?? () #287 0x00007ff3370032c8 in ?? () #288 0x00000000ffffffff in ?? () #289 0x00007ff339404d20 in ?? () #290 0x0000000000000001 in ?? () #291 0x00007ff3370032c8 in ?? () #292 0x00000000ffffffff in ?? () #293 0x00007ff339404d20 in ?? () #294 0x0000000000000001 in ?? () #295 0x00007ff3370032c8 in ?? () #296 0x00000000ffffffff in ?? () #297 0x00007ff339404d20 in ?? () #298 0x0000000000000001 in ?? () #299 0x00007ff3370032c8 in ?? () #300 0x00000000ffffffff in ?? () #301 0x00007ff339404d20 in ?? () #302 0x0000000000000001 in ?? () #303 0x00007ff3370032c8 in ?? () #304 0x00000000ffffffff in ?? () #305 0x00007ff339404d20 in ?? () #306 0x0000000000000001 in ?? () #307 0x00007ff3370032c8 in ?? () #308 0x00000000ffffffff in ?? () #309 0x00007ff339404d20 in ?? () #310 0x0000000000000001 in ?? () #311 0x00007ff3370032c8 in ?? () #312 0x00000000ffffffff in ?? () #313 0x00007ff339404d20 in ?? () #314 0x0000000000000001 in ?? () #315 0x00007ff3370032c8 in ?? () #316 0x00000000ffffffff in ?? () #317 0x00007ff339404d20 in ?? () #318 0x0000000000000001 in ?? () #319 0x00007ff3370032c8 in ?? () #320 0x00000000ffffffff in ?? () #321 0x00007ff339404d20 in ?? () #322 0x0000000000000001 in ?? () #323 0x00007ff3370032c8 in ?? () #324 0x00000000ffffffff in ?? () #325 0x00007ff339404d20 in ?? () #326 0x0000000000000001 in ?? () #327 0x00007ff3370032c8 in ?? () #328 0x00000000ffffffff in ?? () #329 0x00007ff339404d20 in ?? () #330 0x0000000000000001 in ?? () #331 0x00007ff3370032c8 in ?? () #332 0x00000000ffffffff in ?? () #333 0x00007ff339404d20 in ?? () #334 0x0000000000000001 in ?? () #335 0x00007ff3370032c8 in ?? () #336 0x00007ff3ffffffff in ?? () #337 0x00007ff339404d20 in ?? () #338 0x0000000000000001 in ?? () #339 0x00007ff3370032c8 in ?? () #340 0x00007ff3ffffffff in ?? () #341 0x00007ff339404d20 in ?? () #342 0x0000000000000001 in ?? () #343 0x00007ff3370032c8 in ?? () #344 0x00000000ffffffff in ?? () #345 0x00007ff339404d20 in ?? () #346 0x0000000000000001 in ?? () #347 0x00007ff3370032c8 in ?? () #348 0x00000000ffffffff in ?? () #349 0x00007ff339404d20 in ?? () #350 0x0000000000000001 in ?? () #351 0x00007ff3370032c8 in ?? () #352 0x00000000ffffffff in ?? () #353 0x00007ff339404d20 in ?? () #354 0x00007ff300000001 in ?? () #355 0x00007ff3370032c8 in ?? () #356 0x00000000ffffffff in ?? () #357 0x00007ff339404d20 in ?? () #358 0x00007ff300000001 in ?? () #359 0x00007ff3370032c8 in ?? () #360 0x00007ff3ffffffff in ?? () #361 0x00007ff339404d20 in ?? () #362 0x0000000000000001 in ?? () #363 0x00007ff3370032c8 in ?? () #364 0xfc882008ffffffff in ?? () #365 0x00007ff339404d20 in ?? () #366 0x0000000000000001 in ?? () #367 0x00007ff3370032c8 in ?? () #368 0x00000000ffffffff in ?? () #369 0x00007ff339404d20 in ?? () #370 0x0000000000000001 in ?? () #371 0x00007ff3370032c8 in ?? () #372 0x00000000ffffffff in ?? () #373 0x00007ff339404d20 in ?? () #374 0x0000000000000001 in ?? () #375 0x00007ff3370032c8 in ?? () #376 0x00000000ffffffff in ?? () #377 0x00007ff339404d20 in ?? () #378 0x0000000000000001 in ?? () #379 0x00007ff3370032c8 in ?? () #380 0x00000000ffffffff in ?? () #381 0x00007ff339404d20 in ?? () #382 0x0000000000000001 in ?? () #383 0x00007ff3370032c8 in ?? () #384 0x00000000ffffffff in ?? () #385 0x00007ff339404d20 in ?? () #386 0x0000000000000001 in ?? () #387 0x00007ff3370032c8 in ?? () #388 0x00000000ffffffff in ?? () #389 0x00007ff339404d20 in ?? () #390 0x0000000000000001 in ?? () #391 0x00007ff3370032c8 in ?? () #392 0x00000000ffffffff in ?? () #393 0x00007ff339404d20 in ?? () #394 0x0000000000000001 in ?? () #395 0x00007ff3370032c8 in ?? () #396 0x00000000ffffffff in ?? () #397 0x00007ff339404d20 in ?? () #398 0x0000000000000001 in ?? () #399 0x00007ff3370032c8 in ?? () #400 0x00000000ffffffff in ?? () #401 0x00007ff339404d20 in ?? () #402 0x0000000000000001 in ?? () #403 0x00007ff3370032c8 in ?? () #404 0x00000000ffffffff in ?? () #405 0x00007ff339404d20 in ?? () #406 0x0000000000000001 in ?? () #407 0x00007ff3370032c8 in ?? () #408 0x00000000ffffffff in ?? () #409 0x00007ff339404d20 in ?? () #410 0x0000000000000001 in ?? () #411 0x00007ff3370032c8 in ?? () #412 0x00000000ffffffff in ?? () #413 0x00007ff339404d20 in ?? () #414 0x0000000000000001 in ?? () #415 0x00007ff3370032c8 in ?? () #416 0x00000000ffffffff in ?? () #417 0x00007ff339404d20 in ?? () #418 0x0000000000000001 in ?? () #419 0x00007ff3370032c8 in ?? () #420 0x00000000ffffffff in ?? () #421 0x00007ff339404d20 in ?? () #422 0x0000000000000001 in ?? () #423 0x00007ff3370032c8 in ?? () #424 0x00000000ffffffff in ?? () #425 0x00007ff339404d20 in ?? () #426 0x0000000000000001 in ?? () #427 0x00007ff3370032c8 in ?? () #428 0x00000000ffffffff in ?? () #429 0x00007ff339404d20 in ?? () #430 0x0000000000000001 in ?? () #431 0x00007ff3370032c8 in ?? () #432 0x00000000ffffffff in ?? () #433 0x00007ff339404d20 in ?? () #434 0x0000000000000001 in ?? () #435 0x00007ff3370032c8 in ?? () #436 0x00000000ffffffff in ?? () #437 0x00007ff339404d20 in ?? () #438 0x0000000000000001 in ?? () #439 0x00007ff3370032c8 in ?? () #440 0x00000000ffffffff in ?? () #441 0x00007ff339404d20 in ?? () #442 0x0000000000000001 in ?? () #443 0x00007ff3370032c8 in ?? () #444 0x00000000ffffffff in ?? () #445 0x00007ff339404d20 in ?? () #446 0x00007ff300000001 in ?? () #447 0x00007ff3370032c8 in ?? () #448 0x0000778fffffffff in ?? () #449 0x00007ff339404d20 in ?? () #450 0x00007ff300000001 in ?? () #451 0x00007ff3370032c8 in ?? () #452 0x00000000ffffffff in ?? () #453 0x00007ff339404d20 in ?? () #454 0x00007ff300000001 in ?? () #455 0x00007ff3370032c8 in ?? () #456 0x00000000ffffffff in ?? () #457 0x00007ff339404d20 in ?? () #458 0x0000000000000001 in ?? () #459 0x00007ff3370032c8 in ?? () #460 0x00000000ffffffff in ?? () #461 0x00007ff339404d20 in ?? () #462 0x0000000000000001 in ?? () #463 0x00007ff3370032c8 in ?? () #464 0x00000000ffffffff in ?? () #465 0x00007ff339404d20 in ?? () #466 0x0000000000000001 in ?? () #467 0x00007ff3370032c8 in ?? () #468 0x00000000ffffffff in ?? () #469 0x00007ff339404d20 in ?? () #470 0x0000000000000001 in ?? () #471 0x00007ff3370032c8 in ?? () #472 0x00000000ffffffff in ?? () #473 0x00007ff339404d20 in ?? () #474 0x0000000000000001 in ?? () #475 0x00007ff3370032c8 in ?? () #476 0x00000000ffffffff in ?? () #477 0x00007ff339404d20 in ?? () #478 0x0000000000000001 in ?? () #479 0x00007ff3370032c8 in ?? () #480 0x00000000ffffffff in ?? () #481 0x00007ff339404d20 in ?? () #482 0x0000000000000001 in ?? () #483 0x00007ff3370032c8 in ?? () #484 0x00000000ffffffff in ?? () #485 0x00007ff339404d20 in ?? () #486 0x0000000000000001 in ?? () #487 0x00007ff3370032c8 in ?? () #488 0x00000000ffffffff in ?? () #489 0x00007ff339404d20 in ?? () #490 0x0000000000000001 in ?? () #491 0x00007ff3370032c8 in ?? () #492 0x00000000ffffffff in ?? () #493 0x00007ff339404d20 in ?? () #494 0x0000000000000001 in ?? () #495 0x00007ff3370032c8 in ?? () #496 0x00000000ffffffff in ?? () #497 0x00007ff339404d20 in ?? () #498 0x0000000000000001 in ?? () #499 0x00007ff3370032c8 in ?? () #500 0x00000000ffffffff in ?? () #501 0x00007ff339404d20 in ?? () #502 0x0000000000000001 in ?? () #503 0x00007ff3370032c8 in ?? () #504 0x00000000ffffffff in ?? () #505 0x00007ff339404d20 in ?? () #506 0x0000000000000001 in ?? () #507 0x00007ff3370032c8 in ?? () #508 0x00000000ffffffff in ?? () #509 0x00007ff339404d20 in ?? () #510 0x0000000000000001 in ?? () #511 0x00007ff3370032c8 in ?? () #512 0x00000000ffffffff in ?? () #513 0x00007ff339404d20 in ?? () #514 0x0000000000000001 in ?? () #515 0x00007ff3370032c8 in ?? () #516 0x00000000ffffffff in ?? () #517 0x00007ff339404d20 in ?? () #518 0x0000000000000001 in ?? () #519 0x00007ff3370032c8 in ?? () #520 0x00007ff3ffffffff in ?? () #521 0x00007ff339404d20 in ?? () #522 0x0000000000000001 in ?? () #523 0x00007ff3370032c8 in ?? () #524 0x00000000ffffffff in ?? () #525 0x00007ff339404d20 in ?? () #526 0x0000000000000001 in ?? () #527 0x00007ff3370032c8 in ?? () #528 0x00000000ffffffff in ?? () #529 0x00007ff339404d20 in ?? () #530 0x0000000000000001 in ?? () #531 0x00007ff3370032c8 in ?? () #532 0x00000000ffffffff in ?? () #533 0x00007ff339404d20 in ?? () #534 0x0000000000000001 in ?? () #535 0x00007ff3370032c8 in ?? () #536 0x00000000ffffffff in ?? () #537 0x00007ff339404d20 in ?? () #538 0x0000000000000001 in ?? () #539 0x00007ff3370032c8 in ?? () #540 0x00000000ffffffff in ?? () #541 0x00007ff339404d20 in ?? () #542 0x0000000000000001 in ?? () #543 0x00007ff3370032c8 in ?? () #544 0x00000000ffffffff in ?? () #545 0x00007ff339404d20 in ?? () #546 0x0000000000000001 in ?? () #547 0x00007ff3370032c8 in ?? () #548 0x00000000ffffffff in ?? () #549 0x00007ff339404d20 in ?? () #550 0x0000000000000001 in ?? () #551 0x00007ff3370032c8 in ?? () #552 0x00000000ffffffff in ?? () #553 0x00007ff339404d20 in ?? () #554 0x00f6640300000001 in ?? () #555 0x00007ff3370032c8 in ?? () #556 0x00000000ffffffff in ?? () #557 0x00007ff339404d20 in ?? () #558 0x0000000000000001 in ?? () #559 0x00007ff3370032c8 in ?? () #560 0x00000000ffffffff in ?? () #561 0x00007ff339404d20 in ?? () #562 0x0000000000000001 in ?? () #563 0x00007ff3370032c8 in ?? () #564 0x00000000ffffffff in ?? () #565 0x00007ff339404d20 in ?? () #566 0x00007ff300000001 in ?? () #567 0x00007ff3370032c8 in ?? () #568 0x00007ff3ffffffff in ?? () #569 0x00007ff339404d20 in ?? () #570 0x0000000000000001 in ?? () #571 0x00007ff3370032c8 in ?? () #572 0x00000000ffffffff in ?? () #573 0x00007ff339404d20 in ?? () #574 0x0000000000000001 in ?? () #575 0x00007ff3370032c8 in ?? () #576 0x00000000ffffffff in ?? () #577 0x00007ff339404d20 in ?? () #578 0x0000000000000001 in ?? () #579 0x00007ff3370032c8 in ?? () #580 0x00000000ffffffff in ?? () #581 0x00007ff339404d20 in ?? () #582 0x0000000000000001 in ?? () #583 0x00007ff3370032c8 in ?? () #584 0x00000000ffffffff in ?? () #585 0x00007ff339404d20 in ?? () #586 0x0000000000000001 in ?? () #587 0x00007ff3370032c8 in ?? () #588 0x00000000ffffffff in ?? () #589 0x00007ff339404d20 in ?? () #590 0x0000000000000001 in ?? () #591 0x00007ff3370032c8 in ?? () #592 0x00000000ffffffff in ?? () #593 0x00007ff339404d20 in ?? () #594 0x0000000000000001 in ?? () #595 0x00007ff3370032c8 in ?? () #596 0x00000000ffffffff in ?? () #597 0x00007ff339404d20 in ?? () #598 0x0000000000000001 in ?? () #599 0x00007ff3370032c8 in ?? () #600 0x00000000ffffffff in ?? () #601 0x00007ff339404d20 in ?? () #602 0x0000000000000001 in ?? () #603 0x00007ff3370032c8 in ?? () #604 0x00000000ffffffff in ?? () #605 0x00007ff339404d20 in ?? () #606 0x0000000000000001 in ?? () #607 0x00007ff3370032c8 in ?? () #608 0x00000000ffffffff in ?? () #609 0x00007ff339404d20 in ?? () #610 0x0000000000000001 in ?? () #611 0x00007ff3370032c8 in ?? () #612 0x00000000ffffffff in ?? () #613 0x00007ff339404d20 in ?? () #614 0x0000000000000001 in ?? () #615 0x00007ff3370032c8 in ?? () #616 0x00000000ffffffff in ?? () #617 0x00007ff339404d20 in ?? () #618 0x0000000000000001 in ?? () #619 0x00007ff3370032c8 in ?? () #620 0x00000000ffffffff in ?? () #621 0x00007ff339404d20 in ?? () #622 0x0000000000000001 in ?? () #623 0x00007ff3370032c8 in ?? () #624 0x00000000ffffffff in ?? () #625 0x00007ff339404d20 in ?? () #626 0x0000000000000001 in ?? () #627 0x00007ff3370032c8 in ?? () #628 0x00000000ffffffff in ?? () #629 0x00007ff339404d20 in ?? () #630 0x0000000000000001 in ?? () #631 0x00007ff3370032c8 in ?? () #632 0x00000000ffffffff in ?? () #633 0x00007ff339404d20 in ?? () #634 0x0000000000000001 in ?? () #635 0x00007ff3370032c8 in ?? () #636 0x00000000ffffffff in ?? () #637 0x00007ff339404d20 in ?? () #638 0x0000000000000001 in ?? () #639 0x00007ff3370032c8 in ?? () #640 0x00000000ffffffff in ?? () #641 0x00007ff339404d20 in ?? () #642 0x0000000000000001 in ?? () #643 0x00007ff3370032c8 in ?? () #644 0x00000000ffffffff in ?? () #645 0x00007ff339404d20 in ?? () #646 0x0000000000000000 in ?? () /tmp/gdNTzSQOIS:2: Error in sourced command file: PC register is not available PC register is not available
            elenst Elena Stepanova added a comment - - edited

            I think I might have been misguided by another issue specific to perro (or rather, to libjemalloc on perro) - MDEV-7274.
            No I've upgraded libjemalloc to 3.6.0 and since then can't reproduce the crash there.

            However, it's not a proper explanation for what's happening in buildbot, because at least on /kvm/vms/vm-wheezy-amd64-install.qcow2 where one of crashes happened, libjemalloc is 3.0.0, which is supposed to be okay.

            So, even though they look so alike, these are probably different problems.

            Before digging into it on your local machine, please check which libjemalloc you have.

            elenst Elena Stepanova added a comment - - edited I think I might have been misguided by another issue specific to perro (or rather, to libjemalloc on perro) - MDEV-7274 . No I've upgraded libjemalloc to 3.6.0 and since then can't reproduce the crash there. However, it's not a proper explanation for what's happening in buildbot, because at least on /kvm/vms/vm-wheezy-amd64-install.qcow2 where one of crashes happened, libjemalloc is 3.0.0, which is supposed to be okay. So, even though they look so alike, these are probably different problems. Before digging into it on your local machine, please check which libjemalloc you have.

            Trace for all threads on my side:

            Program terminated with signal SIGSEGV, Segmentation fault.
            #0  0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
            #0  0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
            #1  0x00007f419dd576f8 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
            #2  0x00007f419e067e26 in __GI___backtrace (array=array@entry=0x7f419f2e8060, size=size@entry=128) at ../sysdeps/x86_64/backtrace.c:109
            #3  0x0000000000c1209e in my_print_stacktrace (stack_bottom=0x7f419f2e8e50 "\b\020,\222A\177", thread_stack=294912) at /home/svoj/devel/maria/mariadb/mysys/stacktrace.c:246
            #4  0x0000000000736eb0 in handle_fatal_signal (sig=11) at /home/svoj/devel/maria/mariadb/sql/signal_handler.cc:153
            #5  <signal handler called>
            #6  0x00007f419c99184a in ?? ()
            #7  0x0000000000000000 in ?? ()
             
            Thread 21 (Thread 0x7f419f332700 (LWP 21166)):
            #0  0x00007f419e04a12d in poll () at ../sysdeps/unix/syscall-template.S:81
            #1  0x0000000000c59a85 in poll (__timeout=28800000, __nfds=1, __fds=0x7f419f331be0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46
            #2  vio_io_wait (vio=vio@entry=0x7f4198e23a08, event=event@entry=VIO_IO_EVENT_READ, timeout=28800000) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:969
            #3  0x0000000000c59b18 in vio_socket_io_wait (vio=vio@entry=0x7f4198e23a08, event=event@entry=VIO_IO_EVENT_READ) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:108
            #4  0x0000000000c59c20 in vio_read (vio=0x7f4198e23a08, buf=0x7f41922bc008 "\a", size=4) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:181
            #5  0x000000000053d088 in my_real_read (net=0x7f419c7a8238, complen=complen@entry=0x7f419f331dc0, header=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/net_serv.cc:872
            #6  0x000000000053e02d in my_net_read_packet (net=0x7f419c7a8238, read_from_server=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/net_serv.cc:1088
            #7  0x00000000005d793c in do_command (thd=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:929
            #8  0x000000000069f013 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1375
            #9  0x000000000069f0f9 in handle_one_connection (arg=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1289
            #10 0x00007f419e934182 in start_thread (arg=0x7f419f332700) at pthread_create.c:312
            #11 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 20 (Thread 0x7f419f37b700 (LWP 11426)):
            #0  0x00007f419e93c0d1 in do_sigwait (sig=0x7f419f37ad70, set=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60
            #1  __sigwait (set=0x7f419f37add0, sig=0x7f419f37ad70) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97
            #2  0x00000000005333f3 in signal_hand (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3343
            #3  0x00007f419e934182 in start_thread (arg=0x7f419f37b700) at pthread_create.c:312
            #4  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 19 (Thread 0x7f41937fc700 (LWP 11412)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41937fbd70, fast_mutex=0x7f4198e22600, cond=0x7f4198e22640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f4198e22600, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000099ff21 in ib_wqueue_timedwait (wq=wq@entry=0x7f419d4da7e8, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/ut/ut0wqueue.cc:154
            #4  0x0000000000a53e6e in fts_optimize_thread (arg=0x7f419d4da7e8) at /home/svoj/devel/maria/mariadb/storage/xtradb/fts/fts0opt.cc:3003
            #5  0x00007f419e934182 in start_thread (arg=0x7f41937fc700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 18 (Thread 0x7f4193ffd700 (LWP 11411)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a93c in os_cond_wait_timed (abstime=0x7f4193ffcdc0, fast_mutex=0x7f419c7cb800, cond=0x7f419c7cb840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f419c7cb800, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x0000000000a1e2d3 in dict_stats_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/dict/dict0stats_bg.cc:349
            #4  0x00007f419e934182 in start_thread (arg=0x7f4193ffd700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 17 (Thread 0x7f41947fe700 (LWP 11410)):
            #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
            #1  0x000000000090a801 in os_cond_wait (fast_mutex=0x7f419d5fe700, cond=0x7f419d5fe740) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214
            #2  os_event_wait_low (event=0x7f419d5fe700, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610
            #3  0x00000000009e1fce in buf_dump_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0dump.cc:594
            #4  0x00007f419e934182 in start_thread (arg=0x7f41947fe700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 16 (Thread 0x7f41963fa700 (LWP 11402)):
            #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
            #1  0x000000000090a801 in os_cond_wait (fast_mutex=0x7f419d5fe580, cond=0x7f419d5fe5c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214
            #2  os_event_wait_low (event=0x7f419d5fe580, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610
            #3  0x0000000000972883 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=0x7f419d40f8f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3327
            #4  srv_purge_coordinator_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3449
            #5  0x00007f419e934182 in start_thread (arg=0x7f41963fa700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 15 (Thread 0x7f4196bfb700 (LWP 11401)):
            #0  0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x0000000000971360 in srv_master_sleep () at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2970
            #3  srv_master_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3018
            #4  0x00007f419e934182 in start_thread (arg=0x7f4196bfb700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 14 (Thread 0x7f4191fff700 (LWP 11425)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x0000000000aa29ff in inline_mysql_cond_timedwait (src_file=0xe09088 "/home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c", src_line=119, abstime=0x7f4191ffed60, mutex=0x1ca3640 <LOCK_checkpoint>, that=0x1ca3600 <COND_checkpoint>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1202
            #2  my_service_thread_sleep (control=control@entry=0x13ee720 <checkpoint_control>, sleep_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c:118
            #3  0x0000000000a9a876 in ma_checkpoint_background (arg=0x1e) at /home/svoj/devel/maria/mariadb/storage/maria/ma_checkpoint.c:709
            #4  0x00007f419e934182 in start_thread (arg=0x7f4191fff700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 13 (Thread 0x7f41973fc700 (LWP 11400)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41973fbde0, fast_mutex=0x7f419d5fe680, cond=0x7f419d5fe6c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f419d5fe680, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000096f906 in srv_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:1978
            #4  0x00007f419e934182 in start_thread (arg=0x7f41973fc700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 12 (Thread 0x7f4194fff700 (LWP 11409)):
            #0  0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x00000000009e8faf in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583
            #3  buf_flush_lru_manager_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2854
            #4  0x00007f419e934182 in start_thread (arg=0x7f4194fff700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 11 (Thread 0x7f4197bfd700 (LWP 11399)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a93c in os_cond_wait_timed (abstime=0x7f4197bfcd80, fast_mutex=0x7f419d5fe600, cond=0x7f419d5fe640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f419d5fe600, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000096fe86 in srv_error_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2242
            #4  0x00007f419e934182 in start_thread (arg=0x7f4197bfd700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 10 (Thread 0x7f4198bff700 (LWP 11408)):
            #0  0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x00000000009e807a in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583
            #3  buf_flush_page_cleaner_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2702
            #4  0x00007f419e934182 in start_thread (arg=0x7f4198bff700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 9 (Thread 0x7f41983fe700 (LWP 11398)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41983fdda0, fast_mutex=0x7f419d7dd800, cond=0x7f419d7dd840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=event@entry=0x7f419d7dd800, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x00000000008e6b1c in lock_wait_timeout_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/lock/lock0wait.cc:499
            #4  0x00007f419e934182 in start_thread (arg=0x7f41983fe700) at pthread_create.c:312
            #5  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 8 (Thread 0x7f4199bfa700 (LWP 11391)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=5, message1=message1@entry=0x7f4199bf9de0, message2=message2@entry=0x7f4199bf9df0, type=type@entry=0x7f4199bf9e00, space_id=space_id@entry=0x7f4199bf9e10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=5) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f4199bfa700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 7 (Thread 0x7f419abfc700 (LWP 11389)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=3, message1=message1@entry=0x7f419abfbde0, message2=message2@entry=0x7f419abfbdf0, type=type@entry=0x7f419abfbe00, space_id=space_id@entry=0x7f419abfbe10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=3) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f419abfc700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 6 (Thread 0x7f419a3fb700 (LWP 11390)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=4, message1=message1@entry=0x7f419a3fade0, message2=message2@entry=0x7f419a3fadf0, type=type@entry=0x7f419a3fae00, space_id=space_id@entry=0x7f419a3fae10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=4) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f419a3fb700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 5 (Thread 0x7f419b3fd700 (LWP 11388)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=2, message1=message1@entry=0x7f419b3fcde0, message2=message2@entry=0x7f419b3fcdf0, type=type@entry=0x7f419b3fce00, space_id=space_id@entry=0x7f419b3fce10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=2) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f419b3fd700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 4 (Thread 0x7f419bbfe700 (LWP 11387)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434150) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=1, message1=message1@entry=0x7f419bbfdde0, message2=message2@entry=0x7f419bbfddf0, type=type@entry=0x7f419bbfde00, space_id=space_id@entry=0x7f419bbfde10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f419bbfe700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 3 (Thread 0x7f419c3ff700 (LWP 11386)):
            #0  0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341a0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=0, message1=message1@entry=0x7f419c3fede0, message2=message2@entry=0x7f419c3fedf0, type=type@entry=0x7f419c3fee00, space_id=space_id@entry=0x7f419c3fee10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2523a in fil_aio_wait (segment=segment@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f419e934182 in start_thread (arg=0x7f419c3ff700) at pthread_create.c:312
            #6  0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 2 (Thread 0x7f419f3887c0 (LWP 11385)):
            #0  0x00007f419e04a12d in poll () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000053861c in poll (__timeout=-1, __nfds=2, __fds=0x7fffff430900) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
            #2  handle_connections_sockets () at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:6112
            #3  0x000000000053c4c5 in mysqld_main (argc=123, argv=0x7f419d457588) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:5544
            #4  0x00007f419df7eec5 in __libc_start_main (main=0x51e4a0 <main(int, char**)>, argc=8, argv=0x7fffff430c78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffff430c68) at libc-start.c:287
            #5  0x00000000005312a6 in _start ()
             
            Thread 1 (Thread 0x7f419f2e9700 (LWP 21174)):
            #0  0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
            #1  0x00007f419dd576f8 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1
            #2  0x00007f419e067e26 in __GI___backtrace (array=array@entry=0x7f419f2e8060, size=size@entry=128) at ../sysdeps/x86_64/backtrace.c:109
            #3  0x0000000000c1209e in my_print_stacktrace (stack_bottom=0x7f419f2e8e50 "\b\020,\222A\177", thread_stack=294912) at /home/svoj/devel/maria/mariadb/mysys/stacktrace.c:246
            #4  0x0000000000736eb0 in handle_fatal_signal (sig=11) at /home/svoj/devel/maria/mariadb/sql/signal_handler.cc:153
            #5  <signal handler called>
            #6  0x00007f419c99184a in ?? ()
            #7  0x0000000000000000 in ?? ()

            svoj Sergey Vojtovich added a comment - Trace for all threads on my side: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1 #0 0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1 #1 0x00007f419dd576f8 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1 #2 0x00007f419e067e26 in __GI___backtrace (array=array@entry=0x7f419f2e8060, size=size@entry=128) at ../sysdeps/x86_64/backtrace.c:109 #3 0x0000000000c1209e in my_print_stacktrace (stack_bottom=0x7f419f2e8e50 "\b\020,\222A\177", thread_stack=294912) at /home/svoj/devel/maria/mariadb/mysys/stacktrace.c:246 #4 0x0000000000736eb0 in handle_fatal_signal (sig=11) at /home/svoj/devel/maria/mariadb/sql/signal_handler.cc:153 #5 <signal handler called> #6 0x00007f419c99184a in ?? () #7 0x0000000000000000 in ?? ()   Thread 21 (Thread 0x7f419f332700 (LWP 21166)): #0 0x00007f419e04a12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000000c59a85 in poll (__timeout=28800000, __nfds=1, __fds=0x7f419f331be0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:46 #2 vio_io_wait (vio=vio@entry=0x7f4198e23a08, event=event@entry=VIO_IO_EVENT_READ, timeout=28800000) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:969 #3 0x0000000000c59b18 in vio_socket_io_wait (vio=vio@entry=0x7f4198e23a08, event=event@entry=VIO_IO_EVENT_READ) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:108 #4 0x0000000000c59c20 in vio_read (vio=0x7f4198e23a08, buf=0x7f41922bc008 "\a", size=4) at /home/svoj/devel/maria/mariadb/vio/viosocket.c:181 #5 0x000000000053d088 in my_real_read (net=0x7f419c7a8238, complen=complen@entry=0x7f419f331dc0, header=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/net_serv.cc:872 #6 0x000000000053e02d in my_net_read_packet (net=0x7f419c7a8238, read_from_server=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/net_serv.cc:1088 #7 0x00000000005d793c in do_command (thd=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:929 #8 0x000000000069f013 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1375 #9 0x000000000069f0f9 in handle_one_connection (arg=0x7f419c7a8008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1289 #10 0x00007f419e934182 in start_thread (arg=0x7f419f332700) at pthread_create.c:312 #11 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 20 (Thread 0x7f419f37b700 (LWP 11426)): #0 0x00007f419e93c0d1 in do_sigwait (sig=0x7f419f37ad70, set=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60 #1 __sigwait (set=0x7f419f37add0, sig=0x7f419f37ad70) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97 #2 0x00000000005333f3 in signal_hand (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3343 #3 0x00007f419e934182 in start_thread (arg=0x7f419f37b700) at pthread_create.c:312 #4 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 19 (Thread 0x7f41937fc700 (LWP 11412)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41937fbd70, fast_mutex=0x7f4198e22600, cond=0x7f4198e22640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f4198e22600, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000099ff21 in ib_wqueue_timedwait (wq=wq@entry=0x7f419d4da7e8, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/ut/ut0wqueue.cc:154 #4 0x0000000000a53e6e in fts_optimize_thread (arg=0x7f419d4da7e8) at /home/svoj/devel/maria/mariadb/storage/xtradb/fts/fts0opt.cc:3003 #5 0x00007f419e934182 in start_thread (arg=0x7f41937fc700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 18 (Thread 0x7f4193ffd700 (LWP 11411)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a93c in os_cond_wait_timed (abstime=0x7f4193ffcdc0, fast_mutex=0x7f419c7cb800, cond=0x7f419c7cb840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f419c7cb800, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x0000000000a1e2d3 in dict_stats_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/dict/dict0stats_bg.cc:349 #4 0x00007f419e934182 in start_thread (arg=0x7f4193ffd700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 17 (Thread 0x7f41947fe700 (LWP 11410)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000090a801 in os_cond_wait (fast_mutex=0x7f419d5fe700, cond=0x7f419d5fe740) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214 #2 os_event_wait_low (event=0x7f419d5fe700, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610 #3 0x00000000009e1fce in buf_dump_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0dump.cc:594 #4 0x00007f419e934182 in start_thread (arg=0x7f41947fe700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 16 (Thread 0x7f41963fa700 (LWP 11402)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000090a801 in os_cond_wait (fast_mutex=0x7f419d5fe580, cond=0x7f419d5fe5c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214 #2 os_event_wait_low (event=0x7f419d5fe580, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610 #3 0x0000000000972883 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=0x7f419d40f8f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3327 #4 srv_purge_coordinator_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3449 #5 0x00007f419e934182 in start_thread (arg=0x7f41963fa700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 15 (Thread 0x7f4196bfb700 (LWP 11401)): #0 0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x0000000000971360 in srv_master_sleep () at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2970 #3 srv_master_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3018 #4 0x00007f419e934182 in start_thread (arg=0x7f4196bfb700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 14 (Thread 0x7f4191fff700 (LWP 11425)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000aa29ff in inline_mysql_cond_timedwait (src_file=0xe09088 "/home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c", src_line=119, abstime=0x7f4191ffed60, mutex=0x1ca3640 <LOCK_checkpoint>, that=0x1ca3600 <COND_checkpoint>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1202 #2 my_service_thread_sleep (control=control@entry=0x13ee720 <checkpoint_control>, sleep_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c:118 #3 0x0000000000a9a876 in ma_checkpoint_background (arg=0x1e) at /home/svoj/devel/maria/mariadb/storage/maria/ma_checkpoint.c:709 #4 0x00007f419e934182 in start_thread (arg=0x7f4191fff700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 13 (Thread 0x7f41973fc700 (LWP 11400)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41973fbde0, fast_mutex=0x7f419d5fe680, cond=0x7f419d5fe6c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f419d5fe680, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000096f906 in srv_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:1978 #4 0x00007f419e934182 in start_thread (arg=0x7f41973fc700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 12 (Thread 0x7f4194fff700 (LWP 11409)): #0 0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x00000000009e8faf in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583 #3 buf_flush_lru_manager_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2854 #4 0x00007f419e934182 in start_thread (arg=0x7f4194fff700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 11 (Thread 0x7f4197bfd700 (LWP 11399)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a93c in os_cond_wait_timed (abstime=0x7f4197bfcd80, fast_mutex=0x7f419d5fe600, cond=0x7f419d5fe640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f419d5fe600, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000096fe86 in srv_error_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2242 #4 0x00007f419e934182 in start_thread (arg=0x7f4197bfd700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 10 (Thread 0x7f4198bff700 (LWP 11408)): #0 0x00007f419e04eda3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b98f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x00000000009e807a in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583 #3 buf_flush_page_cleaner_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2702 #4 0x00007f419e934182 in start_thread (arg=0x7f4198bff700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 9 (Thread 0x7f41983fe700 (LWP 11398)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a93c in os_cond_wait_timed (abstime=0x7f41983fdda0, fast_mutex=0x7f419d7dd800, cond=0x7f419d7dd840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=event@entry=0x7f419d7dd800, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x00000000008e6b1c in lock_wait_timeout_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/lock/lock0wait.cc:499 #4 0x00007f419e934182 in start_thread (arg=0x7f41983fe700) at pthread_create.c:312 #5 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 8 (Thread 0x7f4199bfa700 (LWP 11391)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=5, message1=message1@entry=0x7f4199bf9de0, message2=message2@entry=0x7f4199bf9df0, type=type@entry=0x7f4199bf9e00, space_id=space_id@entry=0x7f4199bf9e10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=5) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f4199bfa700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 7 (Thread 0x7f419abfc700 (LWP 11389)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=3, message1=message1@entry=0x7f419abfbde0, message2=message2@entry=0x7f419abfbdf0, type=type@entry=0x7f419abfbe00, space_id=space_id@entry=0x7f419abfbe10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=3) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f419abfc700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 6 (Thread 0x7f419a3fb700 (LWP 11390)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=4, message1=message1@entry=0x7f419a3fade0, message2=message2@entry=0x7f419a3fadf0, type=type@entry=0x7f419a3fae00, space_id=space_id@entry=0x7f419a3fae10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=4) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f419a3fb700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 5 (Thread 0x7f419b3fd700 (LWP 11388)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=2, message1=message1@entry=0x7f419b3fcde0, message2=message2@entry=0x7f419b3fcdf0, type=type@entry=0x7f419b3fce00, space_id=space_id@entry=0x7f419b3fce10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=2) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f419b3fd700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 4 (Thread 0x7f419bbfe700 (LWP 11387)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d434150) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=1, message1=message1@entry=0x7f419bbfdde0, message2=message2@entry=0x7f419bbfddf0, type=type@entry=0x7f419bbfde00, space_id=space_id@entry=0x7f419bbfde10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f419bbfe700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 3 (Thread 0x7f419c3ff700 (LWP 11386)): #0 0x00007f419ef8758a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908391 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f419d4341a0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=0, message1=message1@entry=0x7f419c3fede0, message2=message2@entry=0x7f419c3fedf0, type=type@entry=0x7f419c3fee00, space_id=space_id@entry=0x7f419c3fee10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2523a in fil_aio_wait (segment=segment@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974185 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f419e934182 in start_thread (arg=0x7f419c3ff700) at pthread_create.c:312 #6 0x00007f419e05747d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 2 (Thread 0x7f419f3887c0 (LWP 11385)): #0 0x00007f419e04a12d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000053861c in poll (__timeout=-1, __nfds=2, __fds=0x7fffff430900) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #2 handle_connections_sockets () at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:6112 #3 0x000000000053c4c5 in mysqld_main (argc=123, argv=0x7f419d457588) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:5544 #4 0x00007f419df7eec5 in __libc_start_main (main=0x51e4a0 <main(int, char**)>, argc=8, argv=0x7fffff430c78, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffff430c68) at libc-start.c:287 #5 0x00000000005312a6 in _start ()   Thread 1 (Thread 0x7f419f2e9700 (LWP 21174)): #0 0x00007f419dd56688 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1 #1 0x00007f419dd576f8 in _Unwind_Backtrace () from /lib/x86_64-linux-gnu/libgcc_s.so.1 #2 0x00007f419e067e26 in __GI___backtrace (array=array@entry=0x7f419f2e8060, size=size@entry=128) at ../sysdeps/x86_64/backtrace.c:109 #3 0x0000000000c1209e in my_print_stacktrace (stack_bottom=0x7f419f2e8e50 "\b\020,\222A\177", thread_stack=294912) at /home/svoj/devel/maria/mariadb/mysys/stacktrace.c:246 #4 0x0000000000736eb0 in handle_fatal_signal (sig=11) at /home/svoj/devel/maria/mariadb/sql/signal_handler.cc:153 #5 <signal handler called> #6 0x00007f419c99184a in ?? () #7 0x0000000000000000 in ?? ()

            And now if I disable SIGSEGV handler:

            #0  0x0000000000c86cbf in arena_run_dalloc ()
            #0  0x0000000000c86cbf in arena_run_dalloc ()
            #1  0x0000000000c9db20 in je_tcache_bin_flush_large ()
            #2  0x0000000000c9e06b in je_tcache_destroy ()
            #3  0x0000000000c9e1b3 in je_tcache_thread_cleanup ()
            #4  0x00007f296f87ff82 in __nptl_deallocate_tsd () at pthread_create.c:158
            #5  0x00007f296f880195 in start_thread (arg=0x7f297027e700) at pthread_create.c:325
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 20 (Thread 0x7f29702c7700 (LWP 22957)):
            #0  0x00007f296f8880d1 in do_sigwait (sig=0x7f29702c6d70, set=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60
            #1  __sigwait (set=0x7f29702c6dd0, sig=0x7f29702c6d70) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97
            #2  0x00000000005333c3 in signal_hand (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3343
            #3  0x00007f296f880182 in start_thread (arg=0x7f29702c7700) at pthread_create.c:312
            #4  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 19 (Thread 0x7f29657fe700 (LWP 22945)):
            #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
            #1  0x000000000090a7d1 in os_cond_wait (fast_mutex=0x7f296e5fe700, cond=0x7f296e5fe740) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214
            #2  os_event_wait_low (event=0x7f296e5fe700, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610
            #3  0x00000000009e1f9e in buf_dump_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0dump.cc:594
            #4  0x00007f296f880182 in start_thread (arg=0x7f29657fe700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 18 (Thread 0x7f2969bff700 (LWP 22943)):
            #0  0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x00000000009e804a in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583
            #3  buf_flush_page_cleaner_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2702
            #4  0x00007f296f880182 in start_thread (arg=0x7f2969bff700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 17 (Thread 0x7f2962fff700 (LWP 22956)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x0000000000aa29cf in inline_mysql_cond_timedwait (src_file=0xe09048 "/home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c", src_line=119, abstime=0x7f2962ffed60, mutex=0x1ca3640 <LOCK_checkpoint>, that=0x1ca3600 <COND_checkpoint>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1202
            #2  my_service_thread_sleep (control=control@entry=0x13ee720 <checkpoint_control>, sleep_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c:118
            #3  0x0000000000a9a846 in ma_checkpoint_background (arg=0x1e) at /home/svoj/devel/maria/mariadb/storage/maria/ma_checkpoint.c:709
            #4  0x00007f296f880182 in start_thread (arg=0x7f2962fff700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 16 (Thread 0x7f29673fa700 (LWP 22936)):
            #0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
            #1  0x000000000090a7d1 in os_cond_wait (fast_mutex=0x7f296e5fe580, cond=0x7f296e5fe5c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214
            #2  os_event_wait_low (event=0x7f296e5fe580, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610
            #3  0x0000000000972853 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=0x7f296e40f8f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3327
            #4  srv_purge_coordinator_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3449
            #5  0x00007f296f880182 in start_thread (arg=0x7f29673fa700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 15 (Thread 0x7f29647fc700 (LWP 22947)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29647fbd70, fast_mutex=0x7f2969e22600, cond=0x7f2969e22640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f2969e22600, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000099fef1 in ib_wqueue_timedwait (wq=wq@entry=0x7f296e4da7e8, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/ut/ut0wqueue.cc:154
            #4  0x0000000000a53e3e in fts_optimize_thread (arg=0x7f296e4da7e8) at /home/svoj/devel/maria/mariadb/storage/xtradb/fts/fts0opt.cc:3003
            #5  0x00007f296f880182 in start_thread (arg=0x7f29647fc700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 14 (Thread 0x7f2967bfb700 (LWP 22935)):
            #0  0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x0000000000971330 in srv_master_sleep () at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2970
            #3  srv_master_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3018
            #4  0x00007f296f880182 in start_thread (arg=0x7f2967bfb700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 13 (Thread 0x7f2964ffd700 (LWP 22946)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a90c in os_cond_wait_timed (abstime=0x7f2964ffcdc0, fast_mutex=0x7f296d7cb800, cond=0x7f296d7cb840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f296d7cb800, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x0000000000a1e2a3 in dict_stats_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/dict/dict0stats_bg.cc:349
            #4  0x00007f296f880182 in start_thread (arg=0x7f2964ffd700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 12 (Thread 0x7f2968bfd700 (LWP 22933)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a90c in os_cond_wait_timed (abstime=0x7f2968bfcd80, fast_mutex=0x7f296e5fe600, cond=0x7f296e5fe640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f296e5fe600, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000096fe56 in srv_error_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2242
            #4  0x00007f296f880182 in start_thread (arg=0x7f2968bfd700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 11 (Thread 0x7f2965fff700 (LWP 22944)):
            #0  0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285
            #2  0x00000000009e8f7f in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583
            #3  buf_flush_lru_manager_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2854
            #4  0x00007f296f880182 in start_thread (arg=0x7f2965fff700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 10 (Thread 0x7f29693fe700 (LWP 22932)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29693fdda0, fast_mutex=0x7f296e7dd800, cond=0x7f296e7dd840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=event@entry=0x7f296e7dd800, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x00000000008e6aec in lock_wait_timeout_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/lock/lock0wait.cc:499
            #4  0x00007f296f880182 in start_thread (arg=0x7f29693fe700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 9 (Thread 0x7f29683fc700 (LWP 22934)):
            #0  pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238
            #1  0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29683fbde0, fast_mutex=0x7f296e5fe680, cond=0x7f296e5fe6c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177
            #2  os_event_wait_time_low (event=0x7f296e5fe680, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723
            #3  0x000000000096f8d6 in srv_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:1978
            #4  0x00007f296f880182 in start_thread (arg=0x7f29683fc700) at pthread_create.c:312
            #5  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 8 (Thread 0x7f296bbfc700 (LWP 22923)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=3, message1=message1@entry=0x7f296bbfbde0, message2=message2@entry=0x7f296bbfbdf0, type=type@entry=0x7f296bbfbe00, space_id=space_id@entry=0x7f296bbfbe10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=3) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296bbfc700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 7 (Thread 0x7f296abfa700 (LWP 22925)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=5, message1=message1@entry=0x7f296abf9de0, message2=message2@entry=0x7f296abf9df0, type=type@entry=0x7f296abf9e00, space_id=space_id@entry=0x7f296abf9e10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=5) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296abfa700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 6 (Thread 0x7f296c3fd700 (LWP 22922)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=2, message1=message1@entry=0x7f296c3fcde0, message2=message2@entry=0x7f296c3fcdf0, type=type@entry=0x7f296c3fce00, space_id=space_id@entry=0x7f296c3fce10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=2) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296c3fd700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 5 (Thread 0x7f296b3fb700 (LWP 22924)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=4, message1=message1@entry=0x7f296b3fade0, message2=message2@entry=0x7f296b3fadf0, type=type@entry=0x7f296b3fae00, space_id=space_id@entry=0x7f296b3fae10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=4) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296b3fb700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 4 (Thread 0x7f296d3ff700 (LWP 22920)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341a0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=0, message1=message1@entry=0x7f296d3fede0, message2=message2@entry=0x7f296d3fedf0, type=type@entry=0x7f296d3fee00, space_id=space_id@entry=0x7f296d3fee10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296d3ff700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 3 (Thread 0x7f296cbfe700 (LWP 22921)):
            #0  0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1
            #1  0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434150) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092
            #2  os_aio_linux_handle (global_seg=global_seg@entry=1, message1=message1@entry=0x7f296cbfdde0, message2=message2@entry=0x7f296cbfddf0, type=type@entry=0x7f296cbfde00, space_id=space_id@entry=0x7f296cbfde10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238
            #3  0x0000000000a2520a in fil_aio_wait (segment=segment@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686
            #4  0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524
            #5  0x00007f296f880182 in start_thread (arg=0x7f296cbfe700) at pthread_create.c:312
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
             
            Thread 2 (Thread 0x7f29702d47c0 (LWP 22913)):
            #0  0x00007f296ef9612d in poll () at ../sysdeps/unix/syscall-template.S:81
            #1  0x00000000005385ec in poll (__timeout=-1, __nfds=2, __fds=0x7fffcebcb0d0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41
            #2  handle_connections_sockets () at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:6112
            #3  0x000000000053c495 in mysqld_main (argc=123, argv=0x7f296e457588) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:5544
            #4  0x00007f296eecaec5 in __libc_start_main (main=0x51e4a0 <main(int, char**)>, argc=8, argv=0x7fffcebcb448, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffcebcb438) at libc-start.c:287
            #5  0x00000000005312a6 in _start ()
             
            Thread 1 (Thread 0x7f297027e700 (LWP 30371)):
            #0  0x0000000000c86cbf in arena_run_dalloc ()
            #1  0x0000000000c9db20 in je_tcache_bin_flush_large ()
            #2  0x0000000000c9e06b in je_tcache_destroy ()
            #3  0x0000000000c9e1b3 in je_tcache_thread_cleanup ()
            #4  0x00007f296f87ff82 in __nptl_deallocate_tsd () at pthread_create.c:158
            #5  0x00007f296f880195 in start_thread (arg=0x7f297027e700) at pthread_create.c:325
            #6  0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

            svoj Sergey Vojtovich added a comment - And now if I disable SIGSEGV handler: #0 0x0000000000c86cbf in arena_run_dalloc () #0 0x0000000000c86cbf in arena_run_dalloc () #1 0x0000000000c9db20 in je_tcache_bin_flush_large () #2 0x0000000000c9e06b in je_tcache_destroy () #3 0x0000000000c9e1b3 in je_tcache_thread_cleanup () #4 0x00007f296f87ff82 in __nptl_deallocate_tsd () at pthread_create.c:158 #5 0x00007f296f880195 in start_thread (arg=0x7f297027e700) at pthread_create.c:325 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 20 (Thread 0x7f29702c7700 (LWP 22957)): #0 0x00007f296f8880d1 in do_sigwait (sig=0x7f29702c6d70, set=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:60 #1 __sigwait (set=0x7f29702c6dd0, sig=0x7f29702c6d70) at ../nptl/sysdeps/unix/sysv/linux/../../../../../sysdeps/unix/sysv/linux/sigwait.c:97 #2 0x00000000005333c3 in signal_hand (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:3343 #3 0x00007f296f880182 in start_thread (arg=0x7f29702c7700) at pthread_create.c:312 #4 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 19 (Thread 0x7f29657fe700 (LWP 22945)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000090a7d1 in os_cond_wait (fast_mutex=0x7f296e5fe700, cond=0x7f296e5fe740) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214 #2 os_event_wait_low (event=0x7f296e5fe700, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610 #3 0x00000000009e1f9e in buf_dump_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0dump.cc:594 #4 0x00007f296f880182 in start_thread (arg=0x7f29657fe700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 18 (Thread 0x7f2969bff700 (LWP 22943)): #0 0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x00000000009e804a in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583 #3 buf_flush_page_cleaner_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2702 #4 0x00007f296f880182 in start_thread (arg=0x7f2969bff700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 17 (Thread 0x7f2962fff700 (LWP 22956)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x0000000000aa29cf in inline_mysql_cond_timedwait (src_file=0xe09048 "/home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c", src_line=119, abstime=0x7f2962ffed60, mutex=0x1ca3640 <LOCK_checkpoint>, that=0x1ca3600 <COND_checkpoint>) at /home/svoj/devel/maria/mariadb/include/mysql/psi/mysql_thread.h:1202 #2 my_service_thread_sleep (control=control@entry=0x13ee720 <checkpoint_control>, sleep_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/maria/ma_servicethread.c:118 #3 0x0000000000a9a846 in ma_checkpoint_background (arg=0x1e) at /home/svoj/devel/maria/mariadb/storage/maria/ma_checkpoint.c:709 #4 0x00007f296f880182 in start_thread (arg=0x7f2962fff700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 16 (Thread 0x7f29673fa700 (LWP 22936)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x000000000090a7d1 in os_cond_wait (fast_mutex=0x7f296e5fe580, cond=0x7f296e5fe5c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:214 #2 os_event_wait_low (event=0x7f296e5fe580, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:610 #3 0x0000000000972853 in srv_purge_coordinator_suspend (rseg_history_len=<optimized out>, slot=0x7f296e40f8f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3327 #4 srv_purge_coordinator_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3449 #5 0x00007f296f880182 in start_thread (arg=0x7f29673fa700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 15 (Thread 0x7f29647fc700 (LWP 22947)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29647fbd70, fast_mutex=0x7f2969e22600, cond=0x7f2969e22640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f2969e22600, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000099fef1 in ib_wqueue_timedwait (wq=wq@entry=0x7f296e4da7e8, wait_in_usecs=wait_in_usecs@entry=5000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/ut/ut0wqueue.cc:154 #4 0x0000000000a53e3e in fts_optimize_thread (arg=0x7f296e4da7e8) at /home/svoj/devel/maria/mariadb/storage/xtradb/fts/fts0opt.cc:3003 #5 0x00007f296f880182 in start_thread (arg=0x7f29647fc700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 14 (Thread 0x7f2967bfb700 (LWP 22935)): #0 0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x0000000000971330 in srv_master_sleep () at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2970 #3 srv_master_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:3018 #4 0x00007f296f880182 in start_thread (arg=0x7f2967bfb700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 13 (Thread 0x7f2964ffd700 (LWP 22946)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a90c in os_cond_wait_timed (abstime=0x7f2964ffcdc0, fast_mutex=0x7f296d7cb800, cond=0x7f296d7cb840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f296d7cb800, time_in_usec=time_in_usec@entry=10000000, reset_sig_count=<optimized out>, reset_sig_count@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x0000000000a1e2a3 in dict_stats_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/dict/dict0stats_bg.cc:349 #4 0x00007f296f880182 in start_thread (arg=0x7f2964ffd700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 12 (Thread 0x7f2968bfd700 (LWP 22933)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a90c in os_cond_wait_timed (abstime=0x7f2968bfcd80, fast_mutex=0x7f296e5fe600, cond=0x7f296e5fe640) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f296e5fe600, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000096fe56 in srv_error_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:2242 #4 0x00007f296f880182 in start_thread (arg=0x7f2968bfd700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 11 (Thread 0x7f2965fff700 (LWP 22944)): #0 0x00007f296ef9ada3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000090b95f in os_thread_sleep (tm=tm@entry=1000000) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0thread.cc:285 #2 0x00000000009e8f7f in page_cleaner_sleep_if_needed (next_loop_time=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2583 #3 buf_flush_lru_manager_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/buf/buf0flu.cc:2854 #4 0x00007f296f880182 in start_thread (arg=0x7f2965fff700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 10 (Thread 0x7f29693fe700 (LWP 22932)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29693fdda0, fast_mutex=0x7f296e7dd800, cond=0x7f296e7dd840) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=event@entry=0x7f296e7dd800, time_in_usec=time_in_usec@entry=1000000, reset_sig_count=<optimized out>, reset_sig_count@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x00000000008e6aec in lock_wait_timeout_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/lock/lock0wait.cc:499 #4 0x00007f296f880182 in start_thread (arg=0x7f29693fe700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 9 (Thread 0x7f29683fc700 (LWP 22934)): #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x000000000090a90c in os_cond_wait_timed (abstime=0x7f29683fbde0, fast_mutex=0x7f296e5fe680, cond=0x7f296e5fe6c0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:177 #2 os_event_wait_time_low (event=0x7f296e5fe680, time_in_usec=time_in_usec@entry=5000000, reset_sig_count=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0sync.cc:723 #3 0x000000000096f8d6 in srv_monitor_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0srv.cc:1978 #4 0x00007f296f880182 in start_thread (arg=0x7f29683fc700) at pthread_create.c:312 #5 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 8 (Thread 0x7f296bbfc700 (LWP 22923)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=3, message1=message1@entry=0x7f296bbfbde0, message2=message2@entry=0x7f296bbfbdf0, type=type@entry=0x7f296bbfbe00, space_id=space_id@entry=0x7f296bbfbe10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=3) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296bbfc700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 7 (Thread 0x7f296abfa700 (LWP 22925)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=5, message1=message1@entry=0x7f296abf9de0, message2=message2@entry=0x7f296abf9df0, type=type@entry=0x7f296abf9e00, space_id=space_id@entry=0x7f296abf9e10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=5) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296abfa700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 6 (Thread 0x7f296c3fd700 (LWP 22922)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434100) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=2, message1=message1@entry=0x7f296c3fcde0, message2=message2@entry=0x7f296c3fcdf0, type=type@entry=0x7f296c3fce00, space_id=space_id@entry=0x7f296c3fce10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=2) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296c3fd700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 5 (Thread 0x7f296b3fb700 (LWP 22924)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341f0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=4, message1=message1@entry=0x7f296b3fade0, message2=message2@entry=0x7f296b3fadf0, type=type@entry=0x7f296b3fae00, space_id=space_id@entry=0x7f296b3fae10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=4) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296b3fb700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 4 (Thread 0x7f296d3ff700 (LWP 22920)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e4341a0) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=0, message1=message1@entry=0x7f296d3fede0, message2=message2@entry=0x7f296d3fedf0, type=type@entry=0x7f296d3fee00, space_id=space_id@entry=0x7f296d3fee10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=0) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296d3ff700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 3 (Thread 0x7f296cbfe700 (LWP 22921)): #0 0x00007f296fed358a in ?? () from /lib/x86_64-linux-gnu/libaio.so.1 #1 0x0000000000908361 in os_aio_linux_collect (seg_size=256, segment=<optimized out>, array=0x7f296e434150) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5092 #2 os_aio_linux_handle (global_seg=global_seg@entry=1, message1=message1@entry=0x7f296cbfdde0, message2=message2@entry=0x7f296cbfddf0, type=type@entry=0x7f296cbfde00, space_id=space_id@entry=0x7f296cbfde10) at /home/svoj/devel/maria/mariadb/storage/xtradb/os/os0file.cc:5238 #3 0x0000000000a2520a in fil_aio_wait (segment=segment@entry=1) at /home/svoj/devel/maria/mariadb/storage/xtradb/fil/fil0fil.cc:5686 #4 0x0000000000974155 in io_handler_thread (arg=<optimized out>) at /home/svoj/devel/maria/mariadb/storage/xtradb/srv/srv0start.cc:524 #5 0x00007f296f880182 in start_thread (arg=0x7f296cbfe700) at pthread_create.c:312 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111   Thread 2 (Thread 0x7f29702d47c0 (LWP 22913)): #0 0x00007f296ef9612d in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000005385ec in poll (__timeout=-1, __nfds=2, __fds=0x7fffcebcb0d0) at /usr/include/x86_64-linux-gnu/bits/poll2.h:41 #2 handle_connections_sockets () at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:6112 #3 0x000000000053c495 in mysqld_main (argc=123, argv=0x7f296e457588) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:5544 #4 0x00007f296eecaec5 in __libc_start_main (main=0x51e4a0 <main(int, char**)>, argc=8, argv=0x7fffcebcb448, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffcebcb438) at libc-start.c:287 #5 0x00000000005312a6 in _start ()   Thread 1 (Thread 0x7f297027e700 (LWP 30371)): #0 0x0000000000c86cbf in arena_run_dalloc () #1 0x0000000000c9db20 in je_tcache_bin_flush_large () #2 0x0000000000c9e06b in je_tcache_destroy () #3 0x0000000000c9e1b3 in je_tcache_thread_cleanup () #4 0x00007f296f87ff82 in __nptl_deallocate_tsd () at pthread_create.c:158 #5 0x00007f296f880195 in start_thread (arg=0x7f297027e700) at pthread_create.c:325 #6 0x00007f296efa347d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            ratzpo Rasmus Johansson (Inactive) made changes -
            Workflow MariaDB v2 [ 60360 ] MariaDB v3 [ 61625 ]
            serg Sergei Golubchik made changes -
            Sprint 10.0.20 [ 5 ]
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            svoj Sergey Vojtovich made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Sprint 10.0.20 [ 5 ] 10.0.20, 10.1.6-1 [ 5, 6 ]
            serg Sergei Golubchik made changes -
            Rank Ranked higher
            serg Sergei Golubchik made changes -
            Sprint 10.0.20, 10.1.6-1 [ 5, 6 ] 10.0.20 [ 5 ]
            svoj Sergey Vojtovich added a comment - - edited

            I got some reasonable trace now. One connection is being closed and another connection issues UNINSTALL PLUGIN spider.

            The reason for this crash is that being closed connection didn't finalize transaction yet, while spider plugin was removed concurrently. To avoid this spider must call thd_set_ha_data() when it is manipulating per-connection data.

            #0  spider_free_trx_alloc (trx=trx@entry=0x7f9ab504a018) at /home/svoj/devel/maria/mariadb/storage/spider/spd_trx.cc:1130
            #1  0x00007f9ab6162457 in spider_free_trx (trx=trx@entry=0x7f9ab504a018, need_lock=need_lock@entry=true) at /home/svoj/devel/maria/mariadb/storage/spider/spd_trx.cc:1530
            #2  0x00007f9ab617f814 in spider_close_connection (hton=<optimized out>, thd=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/storage/spider/spd_table.cc:5964
            #3  0x00007f9ab9517a01 in closecon_handlerton (thd=thd@entry=0x7f9aafee9008, plugin=<optimized out>, unused=unused@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/handler.cc:776
            #4  0x00007f9ab93a69eb in plugin_foreach_with_mask (thd=thd@entry=0x7f9aafee9008, func=func@entry=0x7f9ab95179c0 <closecon_handlerton(THD*, plugin_ref, void*)>, type=type@entry=1, state_mask=4294967287,
                state_mask@entry=8, arg=arg@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2363
            #5  0x00007f9ab951963e in ha_close_connection (thd=thd@entry=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/handler.cc:789
            #6  0x00007f9ab93605ed in THD::~THD (this=0x7f9aafee9008, __in_chrg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1648
            #7  0x00007f9ab9360991 in THD::~THD (this=0x7f9aafee9008, __in_chrg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1696
            #8  0x00007f9ab92d9f6c in unlink_thd (thd=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2881
            #9  0x00007f9ab92d9fd5 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2994
            #10 0x00007f9ab9467bde in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1369
            #11 0x00007f9ab9467e10 in handle_one_connection (arg=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1262
            #12 0x00007f9ab80546aa in start_thread (arg=0x7f9ab8ea9700) at pthread_create.c:333
            #13 0x00007f9ab7772eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
             
            #0  _dl_close_worker (map=map@entry=0x7f9aafe15000) at dl-close.c:745
            #1  0x00007f9ab8d60e58 in _dl_close_worker (map=0x7f9aafe15000) at dl-close.c:125
            #2  _dl_close (_map=0x7f9aafe15000) at dl-close.c:775
            #3  0x00007f9ab8d5a464 in _dl_catch_error (objname=0x7f9ab6fc5b30, errstring=0x7f9ab6fc5b38, mallocedp=0x7f9ab6fc5b28, operate=0x7f9ab826c090 <dlclose_doit>, args=0x7f9aafe15000) at dl-error.c:187
            #4  0x00007f9ab826c62d in _dlerror_run (operate=operate@entry=0x7f9ab826c090 <dlclose_doit>, args=0x7f9aafe15000) at dlerror.c:163
            #5  0x00007f9ab826c0bf in __dlclose (handle=<optimized out>) at dlclose.c:46
            #6  0x00007f9ab93a75be in free_plugin_mem (p=0x7f9ab6de6d68) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:500
            #7  plugin_dl_del (plugin_dl=0x7f9ab6de6d68) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:880
            #8  plugin_del (plugin=0x7f9ab6de6da8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:1243
            #9  0x00007f9ab93a77e4 in reap_plugins () at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:1292
            #10 0x00007f9ab93a7eaa in mysql_uninstall_plugin (thd=thd@entry=0x7f9aafe23008, name=<optimized out>, dl_arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2304
            #11 0x00007f9ab938e90c in mysql_execute_command (thd=thd@entry=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:5527
            #12 0x00007f9ab9393c0f in mysql_parse (thd=0x7f9aafe23008, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x7f9ab8ef1230) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:7205
            #13 0x00007f9ab9396888 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f9aafe23008, packet=packet@entry=0x7f9aafef1009 "UNINSTALL PLUGIN spider",
                packet_length=packet_length@entry=23) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:1475
            #14 0x00007f9ab9396fed in do_command (thd=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:1097
            #15 0x00007f9ab9467c52 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1350
            #16 0x00007f9ab9467e10 in handle_one_connection (arg=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1262
            #17 0x00007f9ab80546aa in start_thread (arg=0x7f9ab8ef2700) at pthread_create.c:333
            #18 0x00007f9ab7772eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

            svoj Sergey Vojtovich added a comment - - edited I got some reasonable trace now. One connection is being closed and another connection issues UNINSTALL PLUGIN spider. The reason for this crash is that being closed connection didn't finalize transaction yet, while spider plugin was removed concurrently. To avoid this spider must call thd_set_ha_data() when it is manipulating per-connection data. #0 spider_free_trx_alloc (trx=trx@entry=0x7f9ab504a018) at /home/svoj/devel/maria/mariadb/storage/spider/spd_trx.cc:1130 #1 0x00007f9ab6162457 in spider_free_trx (trx=trx@entry=0x7f9ab504a018, need_lock=need_lock@entry=true) at /home/svoj/devel/maria/mariadb/storage/spider/spd_trx.cc:1530 #2 0x00007f9ab617f814 in spider_close_connection (hton=<optimized out>, thd=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/storage/spider/spd_table.cc:5964 #3 0x00007f9ab9517a01 in closecon_handlerton (thd=thd@entry=0x7f9aafee9008, plugin=<optimized out>, unused=unused@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/handler.cc:776 #4 0x00007f9ab93a69eb in plugin_foreach_with_mask (thd=thd@entry=0x7f9aafee9008, func=func@entry=0x7f9ab95179c0 <closecon_handlerton(THD*, plugin_ref, void*)>, type=type@entry=1, state_mask=4294967287, state_mask@entry=8, arg=arg@entry=0x0) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2363 #5 0x00007f9ab951963e in ha_close_connection (thd=thd@entry=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/handler.cc:789 #6 0x00007f9ab93605ed in THD::~THD (this=0x7f9aafee9008, __in_chrg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1648 #7 0x00007f9ab9360991 in THD::~THD (this=0x7f9aafee9008, __in_chrg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_class.cc:1696 #8 0x00007f9ab92d9f6c in unlink_thd (thd=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2881 #9 0x00007f9ab92d9fd5 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/mysqld.cc:2994 #10 0x00007f9ab9467bde in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1369 #11 0x00007f9ab9467e10 in handle_one_connection (arg=0x7f9aafee9008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1262 #12 0x00007f9ab80546aa in start_thread (arg=0x7f9ab8ea9700) at pthread_create.c:333 #13 0x00007f9ab7772eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109   #0 _dl_close_worker (map=map@entry=0x7f9aafe15000) at dl-close.c:745 #1 0x00007f9ab8d60e58 in _dl_close_worker (map=0x7f9aafe15000) at dl-close.c:125 #2 _dl_close (_map=0x7f9aafe15000) at dl-close.c:775 #3 0x00007f9ab8d5a464 in _dl_catch_error (objname=0x7f9ab6fc5b30, errstring=0x7f9ab6fc5b38, mallocedp=0x7f9ab6fc5b28, operate=0x7f9ab826c090 <dlclose_doit>, args=0x7f9aafe15000) at dl-error.c:187 #4 0x00007f9ab826c62d in _dlerror_run (operate=operate@entry=0x7f9ab826c090 <dlclose_doit>, args=0x7f9aafe15000) at dlerror.c:163 #5 0x00007f9ab826c0bf in __dlclose (handle=<optimized out>) at dlclose.c:46 #6 0x00007f9ab93a75be in free_plugin_mem (p=0x7f9ab6de6d68) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:500 #7 plugin_dl_del (plugin_dl=0x7f9ab6de6d68) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:880 #8 plugin_del (plugin=0x7f9ab6de6da8) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:1243 #9 0x00007f9ab93a77e4 in reap_plugins () at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:1292 #10 0x00007f9ab93a7eaa in mysql_uninstall_plugin (thd=thd@entry=0x7f9aafe23008, name=<optimized out>, dl_arg=<optimized out>) at /home/svoj/devel/maria/mariadb/sql/sql_plugin.cc:2304 #11 0x00007f9ab938e90c in mysql_execute_command (thd=thd@entry=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:5527 #12 0x00007f9ab9393c0f in mysql_parse (thd=0x7f9aafe23008, rawbuf=<optimized out>, length=<optimized out>, parser_state=0x7f9ab8ef1230) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:7205 #13 0x00007f9ab9396888 in dispatch_command (command=command@entry=COM_QUERY, thd=thd@entry=0x7f9aafe23008, packet=packet@entry=0x7f9aafef1009 "UNINSTALL PLUGIN spider", packet_length=packet_length@entry=23) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:1475 #14 0x00007f9ab9396fed in do_command (thd=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_parse.cc:1097 #15 0x00007f9ab9467c52 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1350 #16 0x00007f9ab9467e10 in handle_one_connection (arg=0x7f9aafe23008) at /home/svoj/devel/maria/mariadb/sql/sql_connect.cc:1262 #17 0x00007f9ab80546aa in start_thread (arg=0x7f9ab8ef2700) at pthread_create.c:333 #18 0x00007f9ab7772eed in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
            svoj Sergey Vojtovich made changes -
            Assignee Sergey Vojtovich [ svoj ] Kentoku [ kentoku ]
            svoj Sergey Vojtovich made changes -
            elenst Elena Stepanova made changes -
            Summary spider/bg.ha fails sporadically in buildbot spider/bg.ha, spider/bg.ha_part fail sporadically in buildbot

            Fresh occurrence:
            http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/1806/steps/test/logs/stdio

            spider/bg.ha_part                        w4 [ fail ]
                    Test ended at 2016-11-04 13:40:10
             
            CURRENT_TEST: spider/bg.ha_part
             
             
            Server [mysqld.3.2 - pid: 11519, winpid: 11519, exit: 256] failed during test run
            Server log from this test:
            ----------SERVER LOG START-----------
            161104 13:40:07 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.19-MariaDB-debug
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=2
            max_threads=153
            thread_count=2
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62983 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0x3fff917a8070
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x3fff8a1b6a70 thread_stack 0x48400
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(my_print_stacktrace-0x80a410)[0x21930a60]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(handle_fatal_signal-0x107d120)[0x210244b0]
            linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fff92a40478]
            [0x3fff7ea68f14]
            [0x3fff7ea68f14]
            [0x3fff7ea6e9ac]
            [0x3fff7ea70830]
            [0x3fff7eae3ee0]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(+0x9e83a0)[0x210283a0]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_-0x1335d68)[0x20d42a20]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z19ha_close_connectionP3THD-0x107a2cc)[0x21028414]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_ZN3THDD1Ev-0x13ab9c4)[0x20cc596c]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_ZN3THDD0Ev-0x13ab504)[0x20cc5e74]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z10unlink_thdP3THD-0x145da68)[0x20c06428]
            /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z29one_thread_per_connection_endP3THDb-0x145d58c)[0x20c06934]
            mysys/stacktrace.c:267(my_print_stacktrace)[0x20eb45e4]
            sql/signal_handler.cc:166(handle_fatal_signal)[0x20eb40c8]
            sql/handler.cc:772(closecon_handlerton)[0x3fff92a0b9e0]
            /opt/at8.0/lib64/power8/libc.so.6(clone-0xaa8e8)[0x3fff921a18d0]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x0): 
            Connection ID (thread ID): 37
            Status: KILL_CONNECTION
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
             
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            Writing a core file
            ----------SERVER LOG END-------------
            mysqltest failed but provided no output
            The result from queries just before the failure was:
            < snip >
            for master_1
            for child2
            child2_1
            child2_2
            child2_3
            for child3
            child3_1
            child3_2
            child3_3
            for master_1
            for child2
            child2_1
            child2_2
            child2_3
            for child3
            child3_1
            child3_2
            child3_3
             
            end of test
             
             
             
             - saving '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/4/log/spider/bg.ha_part/' to '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/log/bg.ha_part/'
             - found 'core.11526' (0/5)
             
            Trying 'dbx' to get a backtrace
             
            Trying 'gdb' to get a backtrace
            Core generated by '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld'
            Output from gdb follows. The first stack trace is from the failing thread.
            The following stack traces are from all threads (so the failing one is
            duplicated).
            --------------------------
            [New LWP 13250]
            [New LWP 11554]
            [New LWP 11568]
            [New LWP 11631]
            [New LWP 11641]
            [New LWP 11564]
            [New LWP 11566]
            [New LWP 11642]
            [New LWP 11567]
            [New LWP 11664]
            [New LWP 11685]
            [New LWP 11569]
            [New LWP 11624]
            [New LWP 11632]
            [New LWP 11690]
            [New LWP 11547]
            [New LWP 11565]
            [New LWP 11625]
            [New LWP 11626]
            [New LWP 11663]
            [New LWP 11665]
            [New LWP 11666]
            [New LWP 11526]
             
            warning: Could not load shared library symbols for linux-vdso64.so.1.
            Do you need "set solib-search-path" or "set sysroot"?
             
            warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
            To enable execution of this file add
            	add-auto-load-safe-path /opt/at8.0/lib64/power8/libthread_db-1.0.so
            line to your configuration file "/home/buildbot/.gdbinit".
            To completely disable this security protection add
            	set auto-load safe-path /
            line to your configuration file "/home/buildbot/.gdbinit".
            For more information about this security protection see the
            "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
            	info "(gdb)Auto-loading safe path"
             
            warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
             
            warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".
             
            warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
            Core was generated by `/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld --default'.
            Program terminated with signal 11, Segmentation fault.
            #0  0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
            60	../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory.
            #0  0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
            #1  0x0000000021930bf8 in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477
            #2  0x0000000021024a20 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:296
            #3  <signal handler called>
            #4  0x00003fff7ea68f14 in ?? ()
            #5  0x00003fff7ea6e9ac in ?? ()
            #6  0x00003fff7ea70830 in ?? ()
            #7  0x00003fff7eae3ee0 in ?? ()
            #8  0x00000000210283a0 in closecon_handlerton (thd=0x43d8a1b6350, plugin=0x3fff8a1b6350, unused=0x3fff7ebf4cc0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/handler.cc:772
            Backtrace stopped: previous frame inner to this frame (corrupt stack?)
             
            Thread 23 (LWP 11526):
            #0  0x00003fff92191d10 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81
            #1  0x0000000020c0f4d0 in handle_connections_sockets () at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:6533
            #2  0x0000000020c0e000 in mysqld_main (argc=132, argv=0x3fff91b80ac0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:5958
            #3  0x0000000020bfdbe8 in main (argc=8, argv=0x3fffc02d65b8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/main.cc:25
             
            Thread 22 (LWP 11666):
            #0  0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307
            #2  0x000000002178dc3c in btr_defragment_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/btr/btr0defragment.cc:758
            #3  0x00003fff92a0b9e0 in start_thread (arg=0x3fff82bff140) at pthread_create.c:312
            #4  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 21 (LWP 11665):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed1c48, mutex=0x3fff8eed1c10, abstime=0x3fff833fe550) at pthread_cond_timedwait.c:199
            #1  0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed1c48, fast_mutex=0x3fff8eed1c10, abstime=0x3fff833fe550) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
            #2  0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed1c10, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000021730884 in ib_wqueue_timedwait (wq=0x3fff91ae4078, wait_in_usecs=5000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/ut/ut0wqueue.cc:154
            #4  0x00000000218923e4 in fts_optimize_thread (arg=0x3fff91ae4078) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fts/fts0opt.cc:3064
            #5  0x00003fff92a0b9e0 in start_thread (arg=0x3fff833ff140) at pthread_create.c:312
            #6  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 20 (LWP 11663):
            #0  0x00003fff92a13470 in __pthread_cond_wait (cond=0x3fff8eed02e8, mutex=0x3fff8eed02b0) at pthread_cond_wait.c:188
            #1  0x00000000215c7148 in os_cond_wait (cond=0x3fff8eed02e8, fast_mutex=0x3fff8eed02b0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:196
            #2  0x00000000215c7714 in os_event_wait_low (event=0x3fff8eed02b0, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:477
            #3  0x00000000217b0398 in buf_dump_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0dump.cc:703
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff843ff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 19 (LWP 11626):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed0278, mutex=0x3fff8eed0240, abstime=0x3fff883fe610) at pthread_cond_timedwait.c:199
            #1  0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed0278, fast_mutex=0x3fff8eed0240, abstime=0x3fff883fe610) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
            #2  0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed0240, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
            #3  0x00000000216cdc64 in srv_monitor_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2141
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff883ff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 18 (LWP 11625):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed0208, mutex=0x3fff8eed01d0, abstime=0x3fff88bfe5f0) at pthread_cond_timedwait.c:199
            #1  0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed0208, fast_mutex=0x3fff8eed01d0, abstime=0x3fff88bfe5f0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
            #2  0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed01d0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
            #3  0x00000000216ce6c0 in srv_error_monitor_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2417
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff88bff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 17 (LWP 11565):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef50100, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=1, message1=0x3fff8cb3e638, message2=0x3fff8cb3e640, type=0x3fff8cb3e648, space_id=0x3fff8cb3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c17a0 <n+8>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8cb3f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 16 (LWP 11547):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x229e6960 <COND_timer>, mutex=0x229e68d8 <LOCK_timer+40>, abstime=0x3fff907fe700) at pthread_cond_timedwait.c:199
            #1  0x00000000219388cc in safe_cond_timedwait (cond=0x229e6960 <COND_timer>, mp=0x229e68b0 <LOCK_timer>, abstime=0x3fff907fe700, file=0x21c40d18 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1202) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:547
            #2  0x000000002193a1c8 in inline_mysql_cond_timedwait (that=0x229e6960 <COND_timer>, mutex=0x229e68b0 <LOCK_timer>, abstime=0x3fff907fe700, src_file=0x21c40d88 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1202
            #3  0x000000002193b12c in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c:292
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff907ff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 15 (LWP 11690):
            #0  0x00003fff92a13470 in __pthread_cond_wait (cond=0x22147788 <COND_slave_background>, mutex=0x2214a760 <LOCK_slave_background+40>) at pthread_cond_wait.c:188
            #1  0x00000000219384fc in safe_cond_wait (cond=0x22147788 <COND_slave_background>, mp=0x2214a738 <LOCK_slave_background>, file=0x21a23060 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1165) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:493
            #2  0x0000000020c3d7c4 in inline_mysql_cond_wait (that=0x22147788 <COND_slave_background>, mutex=0x2214a738 <LOCK_slave_background>, src_file=0x21a23a18 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc", src_line=341) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1165
            #3  0x0000000020c3f004 in handle_slave_background (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc:341
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8a277540) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 14 (LWP 11632):
            #0  0x00003fff92a13470 in __pthread_cond_wait (cond=0x3fff8eed0198, mutex=0x3fff8eed0160) at pthread_cond_wait.c:188
            #1  0x00000000215c7148 in os_cond_wait (cond=0x3fff8eed0198, fast_mutex=0x3fff8eed0160) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:196
            #2  0x00000000215c7714 in os_event_wait_low (event=0x3fff8eed0160, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:477
            #3  0x00000000216d18dc in srv_purge_coordinator_suspend (slot=0x3fff91a482c8, rseg_history_len=6) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3548
            #4  0x00000000216d2004 in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3670
            #5  0x00003fff92a0b9e0 in start_thread (arg=0x3fff873ff140) at pthread_create.c:312
            #6  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 13 (LWP 11624):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed17e8, mutex=0x3fff8eed17b0, abstime=0x3fff893fe640) at pthread_cond_timedwait.c:199
            #1  0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed17e8, fast_mutex=0x3fff8eed17b0, abstime=0x3fff893fe640) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
            #2  0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed17b0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
            #3  0x0000000021584274 in lock_wait_timeout_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/lock/lock0wait.cc:536
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff893ff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 12 (LWP 11569):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef501a0, segment=1, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=5, message1=0x3fff8ab3e638, message2=0x3fff8ab3e640, type=0x3fff8ab3e648, space_id=0x3fff8ab3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=5) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c17c0 <n+40>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8ab3f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 11 (LWP 11685):
            #0  0x00003fff92a19554 in do_sigwait (sig=0x3fff8a2d6a60, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61
            #1  __sigwait (set=<optimized out>, sig=0x3fff8a2d6a60) at ../sysdeps/unix/sysv/linux/sigwait.c:98
            #2  0x0000000020c0764c in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:3465
            #3  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8a2d7540) at pthread_create.c:312
            #4  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 10 (LWP 11664):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed1a88, mutex=0x3fff8eed1a50, abstime=0x3fff83bfe660) at pthread_cond_timedwait.c:199
            #1  0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed1a88, fast_mutex=0x3fff8eed1a50, abstime=0x3fff83bfe660) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159
            #2  0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed1a50, time_in_usec=10000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591
            #3  0x000000002182abd4 in dict_stats_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/dict/dict0stats_bg.cc:546
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff83bff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 9 (LWP 11567):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef500b0, segment=1, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=3, message1=0x3fff8bb3e638, message2=0x3fff8bb3e640, type=0x3fff8bb3e648, space_id=0x3fff8bb3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c17b0 <n+24>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8bb3f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 8 (LWP 11642):
            #0  0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307
            #2  0x00000000217bb6cc in page_cleaner_sleep_if_needed (next_loop_time=1478281208169) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2645
            #3  0x00000000217bc198 in buf_flush_lru_manager_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2919
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff85fff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 7 (LWP 11566):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef500b0, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=2, message1=0x3fff8c33e638, message2=0x3fff8c33e640, type=0x3fff8c33e648, space_id=0x3fff8c33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=2) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c17a8 <n+16>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8c33f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 6 (LWP 11564):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef50150, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=0, message1=0x3fff8d33e638, message2=0x3fff8d33e640, type=0x3fff8d33e648, space_id=0x3fff8d33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c1798 <n>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8d33f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 5 (LWP 11641):
            #0  0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307
            #2  0x00000000217bb6cc in page_cleaner_sleep_if_needed (next_loop_time=1478281208255) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2645
            #3  0x00000000217bbb3c in buf_flush_page_cleaner_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2769
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff89bff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 4 (LWP 11631):
            #0  0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81
            #1  0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307
            #2  0x00000000216d08b4 in srv_master_sleep () at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3188
            #3  0x00000000216d0aec in srv_master_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3238
            #4  0x00003fff92a0b9e0 in start_thread (arg=0x3fff87bff140) at pthread_create.c:312
            #5  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 3 (LWP 11568):
            #0  0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1
            #1  0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1
            #2  0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef501a0, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432
            #3  0x00000000215c3d48 in os_aio_linux_handle (global_seg=4, message1=0x3fff8b33e638, message2=0x3fff8b33e640, type=0x3fff8b33e648, space_id=0x3fff8b33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585
            #4  0x0000000021846bac in fil_aio_wait (segment=4) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135
            #5  0x00000000216d6420 in io_handler_thread (arg=0x229c17b8 <n+32>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545
            #6  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8b33f140) at pthread_create.c:312
            #7  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 2 (LWP 11554):
            #0  0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x22975798 <COND_checkpoint>, mutex=0x22975710 <LOCK_checkpoint+40>, abstime=0x3fff8ebfe5f0) at pthread_cond_timedwait.c:199
            #1  0x00000000219388cc in safe_cond_timedwait (cond=0x22975798 <COND_checkpoint>, mp=0x229756e8 <LOCK_checkpoint>, abstime=0x3fff8ebfe5f0, file=0x21b49738 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1202) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:547
            #2  0x0000000021330cb4 in inline_mysql_cond_timedwait (that=0x22975798 <COND_checkpoint>, mutex=0x229756e8 <LOCK_checkpoint>, abstime=0x3fff8ebfe5f0, src_file=0x21b497b8 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1202
            #3  0x0000000021331324 in my_service_thread_sleep (control=0x21e9cc38 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c:115
            #4  0x0000000021320188 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_checkpoint.c:708
            #5  0x00003fff92a0b9e0 in start_thread (arg=0x3fff8ebff140) at pthread_create.c:312
            #6  0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
             
            Thread 1 (LWP 13250):
            #0  0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60
            #1  0x0000000021930bf8 in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477
            #2  0x0000000021024a20 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:296
            #3  <signal handler called>
            #4  0x00003fff7ea68f14 in ?? ()
            #5  0x00003fff7ea6e9ac in ?? ()
            #6  0x00003fff7ea70830 in ?? ()
            #7  0x00003fff7eae3ee0 in ?? ()
            #8  0x00000000210283a0 in closecon_handlerton (thd=0x43d8a1b6350, plugin=0x3fff8a1b6350, unused=0x3fff7ebf4cc0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/handler.cc:772
            Backtrace stopped: previous frame inner to this frame (corrupt stack?)
            

            elenst Elena Stepanova added a comment - Fresh occurrence: http://buildbot.askmonty.org/buildbot/builders/p8-rhel7-bintar-debug/builds/1806/steps/test/logs/stdio spider/bg.ha_part w4 [ fail ] Test ended at 2016-11-04 13:40:10   CURRENT_TEST: spider/bg.ha_part     Server [mysqld.3.2 - pid: 11519, winpid: 11519, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 161104 13:40:07 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.19-MariaDB-debug key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62983 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x3fff917a8070 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x3fff8a1b6a70 thread_stack 0x48400 /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(my_print_stacktrace-0x80a410)[0x21930a60] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(handle_fatal_signal-0x107d120)[0x210244b0] linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fff92a40478] [0x3fff7ea68f14] [0x3fff7ea68f14] [0x3fff7ea6e9ac] [0x3fff7ea70830] [0x3fff7eae3ee0] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(+0x9e83a0)[0x210283a0] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_-0x1335d68)[0x20d42a20] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z19ha_close_connectionP3THD-0x107a2cc)[0x21028414] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_ZN3THDD1Ev-0x13ab9c4)[0x20cc596c] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_ZN3THDD0Ev-0x13ab504)[0x20cc5e74] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z10unlink_thdP3THD-0x145da68)[0x20c06428] /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld(_Z29one_thread_per_connection_endP3THDb-0x145d58c)[0x20c06934] mysys/stacktrace.c:267(my_print_stacktrace)[0x20eb45e4] sql/signal_handler.cc:166(handle_fatal_signal)[0x20eb40c8] sql/handler.cc:772(closecon_handlerton)[0x3fff92a0b9e0] /opt/at8.0/lib64/power8/libc.so.6(clone-0xaa8e8)[0x3fff921a18d0]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x0): Connection ID (thread ID): 37 Status: KILL_CONNECTION   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off   The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file ----------SERVER LOG END------------- mysqltest failed but provided no output The result from queries just before the failure was: < snip > for master_1 for child2 child2_1 child2_2 child2_3 for child3 child3_1 child3_2 child3_3 for master_1 for child2 child2_1 child2_2 child2_3 for child3 child3_1 child3_2 child3_3   end of test       - saving '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/4/log/spider/bg.ha_part/' to '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysql-test/var/log/bg.ha_part/' - found 'core.11526' (0/5)   Trying 'dbx' to get a backtrace   Trying 'gdb' to get a backtrace Core generated by '/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 13250] [New LWP 11554] [New LWP 11568] [New LWP 11631] [New LWP 11641] [New LWP 11564] [New LWP 11566] [New LWP 11642] [New LWP 11567] [New LWP 11664] [New LWP 11685] [New LWP 11569] [New LWP 11624] [New LWP 11632] [New LWP 11690] [New LWP 11547] [New LWP 11565] [New LWP 11625] [New LWP 11626] [New LWP 11663] [New LWP 11665] [New LWP 11666] [New LWP 11526]   warning: Could not load shared library symbols for linux-vdso64.so.1. Do you need "set solib-search-path" or "set sysroot"?   warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py". To enable execution of this file add add-auto-load-safe-path /opt/at8.0/lib64/power8/libthread_db-1.0.so line to your configuration file "/home/buildbot/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/home/buildbot/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path"   warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.   warning: File "/opt/at8.0/lib64/power8/libthread_db-1.0.so" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py".   warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by `/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld --default'. Program terminated with signal 11, Segmentation fault. #0 0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60 60 ../sysdeps/unix/sysv/linux/pthread_kill.c: No such file or directory. #0 0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60 #1 0x0000000021930bf8 in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477 #2 0x0000000021024a20 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:296 #3 <signal handler called> #4 0x00003fff7ea68f14 in ?? () #5 0x00003fff7ea6e9ac in ?? () #6 0x00003fff7ea70830 in ?? () #7 0x00003fff7eae3ee0 in ?? () #8 0x00000000210283a0 in closecon_handlerton (thd=0x43d8a1b6350, plugin=0x3fff8a1b6350, unused=0x3fff7ebf4cc0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/handler.cc:772 Backtrace stopped: previous frame inner to this frame (corrupt stack?)   Thread 23 (LWP 11526): #0 0x00003fff92191d10 in .__GI___poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x0000000020c0f4d0 in handle_connections_sockets () at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:6533 #2 0x0000000020c0e000 in mysqld_main (argc=132, argv=0x3fff91b80ac0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:5958 #3 0x0000000020bfdbe8 in main (argc=8, argv=0x3fffc02d65b8) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/main.cc:25   Thread 22 (LWP 11666): #0 0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307 #2 0x000000002178dc3c in btr_defragment_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/btr/btr0defragment.cc:758 #3 0x00003fff92a0b9e0 in start_thread (arg=0x3fff82bff140) at pthread_create.c:312 #4 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 21 (LWP 11665): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed1c48, mutex=0x3fff8eed1c10, abstime=0x3fff833fe550) at pthread_cond_timedwait.c:199 #1 0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed1c48, fast_mutex=0x3fff8eed1c10, abstime=0x3fff833fe550) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159 #2 0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed1c10, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591 #3 0x0000000021730884 in ib_wqueue_timedwait (wq=0x3fff91ae4078, wait_in_usecs=5000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/ut/ut0wqueue.cc:154 #4 0x00000000218923e4 in fts_optimize_thread (arg=0x3fff91ae4078) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fts/fts0opt.cc:3064 #5 0x00003fff92a0b9e0 in start_thread (arg=0x3fff833ff140) at pthread_create.c:312 #6 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 20 (LWP 11663): #0 0x00003fff92a13470 in __pthread_cond_wait (cond=0x3fff8eed02e8, mutex=0x3fff8eed02b0) at pthread_cond_wait.c:188 #1 0x00000000215c7148 in os_cond_wait (cond=0x3fff8eed02e8, fast_mutex=0x3fff8eed02b0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:196 #2 0x00000000215c7714 in os_event_wait_low (event=0x3fff8eed02b0, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:477 #3 0x00000000217b0398 in buf_dump_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0dump.cc:703 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff843ff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 19 (LWP 11626): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed0278, mutex=0x3fff8eed0240, abstime=0x3fff883fe610) at pthread_cond_timedwait.c:199 #1 0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed0278, fast_mutex=0x3fff8eed0240, abstime=0x3fff883fe610) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159 #2 0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed0240, time_in_usec=5000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591 #3 0x00000000216cdc64 in srv_monitor_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2141 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff883ff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 18 (LWP 11625): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed0208, mutex=0x3fff8eed01d0, abstime=0x3fff88bfe5f0) at pthread_cond_timedwait.c:199 #1 0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed0208, fast_mutex=0x3fff8eed01d0, abstime=0x3fff88bfe5f0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159 #2 0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed01d0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591 #3 0x00000000216ce6c0 in srv_error_monitor_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:2417 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff88bff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 17 (LWP 11565): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef50100, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=1, message1=0x3fff8cb3e638, message2=0x3fff8cb3e640, type=0x3fff8cb3e648, space_id=0x3fff8cb3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c17a0 <n+8>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8cb3f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 16 (LWP 11547): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x229e6960 <COND_timer>, mutex=0x229e68d8 <LOCK_timer+40>, abstime=0x3fff907fe700) at pthread_cond_timedwait.c:199 #1 0x00000000219388cc in safe_cond_timedwait (cond=0x229e6960 <COND_timer>, mp=0x229e68b0 <LOCK_timer>, abstime=0x3fff907fe700, file=0x21c40d18 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1202) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:547 #2 0x000000002193a1c8 in inline_mysql_cond_timedwait (that=0x229e6960 <COND_timer>, mutex=0x229e68b0 <LOCK_timer>, abstime=0x3fff907fe700, src_file=0x21c40d88 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c", src_line=292) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1202 #3 0x000000002193b12c in timer_handler (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_timer.c:292 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff907ff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 15 (LWP 11690): #0 0x00003fff92a13470 in __pthread_cond_wait (cond=0x22147788 <COND_slave_background>, mutex=0x2214a760 <LOCK_slave_background+40>) at pthread_cond_wait.c:188 #1 0x00000000219384fc in safe_cond_wait (cond=0x22147788 <COND_slave_background>, mp=0x2214a738 <LOCK_slave_background>, file=0x21a23060 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1165) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:493 #2 0x0000000020c3d7c4 in inline_mysql_cond_wait (that=0x22147788 <COND_slave_background>, mutex=0x2214a738 <LOCK_slave_background>, src_file=0x21a23a18 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc", src_line=341) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1165 #3 0x0000000020c3f004 in handle_slave_background (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/slave.cc:341 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8a277540) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 14 (LWP 11632): #0 0x00003fff92a13470 in __pthread_cond_wait (cond=0x3fff8eed0198, mutex=0x3fff8eed0160) at pthread_cond_wait.c:188 #1 0x00000000215c7148 in os_cond_wait (cond=0x3fff8eed0198, fast_mutex=0x3fff8eed0160) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:196 #2 0x00000000215c7714 in os_event_wait_low (event=0x3fff8eed0160, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:477 #3 0x00000000216d18dc in srv_purge_coordinator_suspend (slot=0x3fff91a482c8, rseg_history_len=6) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3548 #4 0x00000000216d2004 in srv_purge_coordinator_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3670 #5 0x00003fff92a0b9e0 in start_thread (arg=0x3fff873ff140) at pthread_create.c:312 #6 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 13 (LWP 11624): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed17e8, mutex=0x3fff8eed17b0, abstime=0x3fff893fe640) at pthread_cond_timedwait.c:199 #1 0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed17e8, fast_mutex=0x3fff8eed17b0, abstime=0x3fff893fe640) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159 #2 0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed17b0, time_in_usec=1000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591 #3 0x0000000021584274 in lock_wait_timeout_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/lock/lock0wait.cc:536 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff893ff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 12 (LWP 11569): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef501a0, segment=1, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=5, message1=0x3fff8ab3e638, message2=0x3fff8ab3e640, type=0x3fff8ab3e648, space_id=0x3fff8ab3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=5) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c17c0 <n+40>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8ab3f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 11 (LWP 11685): #0 0x00003fff92a19554 in do_sigwait (sig=0x3fff8a2d6a60, set=<optimized out>) at ../sysdeps/unix/sysv/linux/sigwait.c:61 #1 __sigwait (set=<optimized out>, sig=0x3fff8a2d6a60) at ../sysdeps/unix/sysv/linux/sigwait.c:98 #2 0x0000000020c0764c in signal_hand (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/mysqld.cc:3465 #3 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8a2d7540) at pthread_create.c:312 #4 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 10 (LWP 11664): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x3fff8eed1a88, mutex=0x3fff8eed1a50, abstime=0x3fff83bfe660) at pthread_cond_timedwait.c:199 #1 0x00000000215c6fdc in os_cond_wait_timed (cond=0x3fff8eed1a88, fast_mutex=0x3fff8eed1a50, abstime=0x3fff83bfe660) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:159 #2 0x00000000215c79a0 in os_event_wait_time_low (event=0x3fff8eed1a50, time_in_usec=10000000, reset_sig_count=1) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0sync.cc:591 #3 0x000000002182abd4 in dict_stats_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/dict/dict0stats_bg.cc:546 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff83bff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 9 (LWP 11567): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef500b0, segment=1, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=3, message1=0x3fff8bb3e638, message2=0x3fff8bb3e640, type=0x3fff8bb3e648, space_id=0x3fff8bb3e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=3) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c17b0 <n+24>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8bb3f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 8 (LWP 11642): #0 0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307 #2 0x00000000217bb6cc in page_cleaner_sleep_if_needed (next_loop_time=1478281208169) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2645 #3 0x00000000217bc198 in buf_flush_lru_manager_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2919 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff85fff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 7 (LWP 11566): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef500b0, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=2, message1=0x3fff8c33e638, message2=0x3fff8c33e640, type=0x3fff8c33e648, space_id=0x3fff8c33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=2) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c17a8 <n+16>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8c33f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 6 (LWP 11564): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef50150, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=0, message1=0x3fff8d33e638, message2=0x3fff8d33e640, type=0x3fff8d33e648, space_id=0x3fff8d33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c1798 <n>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8d33f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 5 (LWP 11641): #0 0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307 #2 0x00000000217bb6cc in page_cleaner_sleep_if_needed (next_loop_time=1478281208255) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2645 #3 0x00000000217bbb3c in buf_flush_page_cleaner_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/buf/buf0flu.cc:2769 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff89bff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 4 (LWP 11631): #0 0x00003fff92194384 in .__select () at ../sysdeps/unix/syscall-template.S:81 #1 0x00000000215c8610 in os_thread_sleep (tm=1000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0thread.cc:307 #2 0x00000000216d08b4 in srv_master_sleep () at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3188 #3 0x00000000216d0aec in srv_master_thread (arg=0x0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0srv.cc:3238 #4 0x00003fff92a0b9e0 in start_thread (arg=0x3fff87bff140) at pthread_create.c:312 #5 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 3 (LWP 11568): #0 0x00003fff92950998 in .__io_getevents_0_4 () from /lib64/libaio.so.1 #1 0x00003fff929509f8 in .io_getevents () from /lib64/libaio.so.1 #2 0x00000000215c3830 in os_aio_linux_collect (array=0x3fff8ef501a0, segment=0, seg_size=256) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5432 #3 0x00000000215c3d48 in os_aio_linux_handle (global_seg=4, message1=0x3fff8b33e638, message2=0x3fff8b33e640, type=0x3fff8b33e648, space_id=0x3fff8b33e650) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/os/os0file.cc:5585 #4 0x0000000021846bac in fil_aio_wait (segment=4) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/fil/fil0fil.cc:6135 #5 0x00000000216d6420 in io_handler_thread (arg=0x229c17b8 <n+32>) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/xtradb/srv/srv0start.cc:545 #6 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8b33f140) at pthread_create.c:312 #7 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 2 (LWP 11554): #0 0x00003fff92a13960 in __pthread_cond_timedwait (cond=0x22975798 <COND_checkpoint>, mutex=0x22975710 <LOCK_checkpoint+40>, abstime=0x3fff8ebfe5f0) at pthread_cond_timedwait.c:199 #1 0x00000000219388cc in safe_cond_timedwait (cond=0x22975798 <COND_checkpoint>, mp=0x229756e8 <LOCK_checkpoint>, abstime=0x3fff8ebfe5f0, file=0x21b49738 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h", line=1202) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/thr_mutex.c:547 #2 0x0000000021330cb4 in inline_mysql_cond_timedwait (that=0x22975798 <COND_checkpoint>, mutex=0x229756e8 <LOCK_checkpoint>, abstime=0x3fff8ebfe5f0, src_file=0x21b497b8 "/home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c", src_line=116) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/include/mysql/psi/mysql_thread.h:1202 #3 0x0000000021331324 in my_service_thread_sleep (control=0x21e9cc38 <checkpoint_control>, sleep_time=30000000000) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_servicethread.c:115 #4 0x0000000021320188 in ma_checkpoint_background (arg=0x1e) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/storage/maria/ma_checkpoint.c:708 #5 0x00003fff92a0b9e0 in start_thread (arg=0x3fff8ebff140) at pthread_create.c:312 #6 0x00003fff921a18d0 in .__clone () at ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96   Thread 1 (LWP 13250): #0 0x00003fff92a14f00 in __pthread_kill (threadid=<optimized out>, signo=<optimized out>) at ../sysdeps/unix/sysv/linux/pthread_kill.c:60 #1 0x0000000021930bf8 in my_write_core (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/mysys/stacktrace.c:477 #2 0x0000000021024a20 in handle_fatal_signal (sig=11) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/signal_handler.cc:296 #3 <signal handler called> #4 0x00003fff7ea68f14 in ?? () #5 0x00003fff7ea6e9ac in ?? () #6 0x00003fff7ea70830 in ?? () #7 0x00003fff7eae3ee0 in ?? () #8 0x00000000210283a0 in closecon_handlerton (thd=0x43d8a1b6350, plugin=0x3fff8a1b6350, unused=0x3fff7ebf4cc0) at /home/buildbot/maria-slave/power8-vlp03-bintar-debug/build/sql/handler.cc:772 Backtrace stopped: previous frame inner to this frame (corrupt stack?)
            dthompson David Thompson (Inactive) made changes -
            Sprint 10.0.20 [ 5 ] 10.0.20, 10.0.21 [ 5, 129 ]
            dthompson David Thompson (Inactive) made changes -
            Assignee Kentoku [ kentoku ] David Thompson [ dthompson ]
            dthompson David Thompson (Inactive) made changes -
            Status In Progress [ 3 ] Stalled [ 10000 ]
            dthompson David Thompson (Inactive) made changes -
            Assignee David Thompson [ dthompson ] Jacob Mathew [ jacob-mathew ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.0 [ 16000 ]
            Fix Version/s 10.1 [ 16100 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -

            Hi Kentoku,

            Please review the changes that I committed to fix this bug: https://github.com/MariaDB/server/commit/67d7277d7c7dc67a97ffb65e7621f2909cfcedf3

            Thanks,
            Jacob

            jacob-mathew Jacob Mathew (Inactive) added a comment - Hi Kentoku, Please review the changes that I committed to fix this bug: https://github.com/MariaDB/server/commit/67d7277d7c7dc67a97ffb65e7621f2909cfcedf3 Thanks, Jacob
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Assignee Jacob Mathew [ jacob-mathew ] Kentoku [ kentoku ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            Hi Jacob,

            Thank you for creating the changes.
            I need to ask you first before finishing to review. Does it really need such a mechanism for all plugins? For such as freeing global locks, memories at "uninstall plugin" command.

            Thanks,
            Kentoku

            Kentoku Kentoku Shiba (Inactive) added a comment - Hi Jacob, Thank you for creating the changes. I need to ask you first before finishing to review. Does it really need such a mechanism for all plugins? For such as freeing global locks, memories at "uninstall plugin" command. Thanks, Kentoku
            alice Alice Sherepa added a comment -

            spider/bg.ha_part failed in buildbot http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/9804/steps/test_5/logs/stdio

            spider/bg.ha_part                        w2 [ fail ]
                    Test ended at 2017-08-14 17:48:51
             
            CURRENT_TEST: spider/bg.ha_part
            mysqltest: In included file "/mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/../include/../../include/deinit_spider.inc": 
            included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/../include/ha_deinit_child3_2.inc at line 2:
            included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/ha_test_deinit.inc at line 27:
            included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/ha_part.test at line 1011:
            At line 10: query 'DROP TABLE IF EXISTS mysql.spider_link_mon_servers' failed: 2013: Lost connection to MySQL server during query
             
            The result from queries just before the failure was:
            < snip >
            8	g	2011-05-05 21:33:30
            9	h	2011-05-05 22:32:10
            DROP TABLE ta_l2;
             
            deinit
            DROP DATABASE IF EXISTS auto_test_local;
            DROP DATABASE IF EXISTS auto_test_remote;
            DROP DATABASE IF EXISTS auto_test_remote2;
            DROP DATABASE IF EXISTS auto_test_remote3;
            DROP DATABASE IF EXISTS auto_test_local;
            DROP DATABASE IF EXISTS auto_test_local;
            DROP DATABASE IF EXISTS auto_test_local;
            for master_1
            for child2
            child2_1
            child2_2
            child2_3
            for child3
            child3_1
            child3_2
             
            More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.1.27/mysql-test/var/2/log/ha_part.log
             
             
            Server [mysqld.3.2 - pid: 2149, winpid: 2149, exit: 256] failed during test run
            Server log from this test:
            ----------SERVER LOG START-----------
            170814 17:48:50 [ERROR] mysqld got signal 11 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.27-MariaDB
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=2
            max_threads=153
            thread_count=2
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62909 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7f80ffb0e008
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            ----------SERVER LOG END-------------
             
            
            

            alice Alice Sherepa added a comment - spider/bg.ha_part failed in buildbot http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/9804/steps/test_5/logs/stdio spider/bg.ha_part w2 [ fail ] Test ended at 2017-08-14 17:48:51   CURRENT_TEST: spider/bg.ha_part mysqltest: In included file "/mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/../include/../../include/deinit_spider.inc": included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/../include/ha_deinit_child3_2.inc at line 2: included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/ha_test_deinit.inc at line 27: included from /mnt/buildbot/build/mariadb-10.1.27/storage/spider/mysql-test/spider/bg/t/ha_part.test at line 1011: At line 10: query 'DROP TABLE IF EXISTS mysql.spider_link_mon_servers' failed: 2013: Lost connection to MySQL server during query   The result from queries just before the failure was: < snip > 8 g 2011-05-05 21:33:30 9 h 2011-05-05 22:32:10 DROP TABLE ta_l2;   deinit DROP DATABASE IF EXISTS auto_test_local; DROP DATABASE IF EXISTS auto_test_remote; DROP DATABASE IF EXISTS auto_test_remote2; DROP DATABASE IF EXISTS auto_test_remote3; DROP DATABASE IF EXISTS auto_test_local; DROP DATABASE IF EXISTS auto_test_local; DROP DATABASE IF EXISTS auto_test_local; for master_1 for child2 child2_1 child2_2 child2_3 for child3 child3_1 child3_2   More results from queries before failure can be found in /mnt/buildbot/build/mariadb-10.1.27/mysql-test/var/2/log/ha_part.log     Server [mysqld.3.2 - pid: 2149, winpid: 2149, exit: 256] failed during test run Server log from this test: ----------SERVER LOG START----------- 170814 17:48:50 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.27-MariaDB key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=2 max_threads=153 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62909 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f80ffb0e008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... ----------SERVER LOG END-------------  
            elenst Elena Stepanova made changes -
            Affects Version/s 10.2 [ 14601 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Fix Version/s 10.3 [ 22126 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Assignee Kentoku [ kentoku ] Jacob Mathew [ jacob-mathew ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Sprint 10.0.20, 2017-02 [ 5, 129 ] 10.0.20, 2017-02, 10.3.6 [ 5, 129, 237 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Summary spider/bg.ha, spider/bg.ha_part fail sporadically in buildbot spider/bg.ha, spider/bg.ha_part crash server sporadically in buildbot

            Kentoku, please review my fix for this problem in commit 986b651 on my branch.

            jacob-mathew Jacob Mathew (Inactive) added a comment - Kentoku, please review my fix for this problem in commit 986b651 on my branch .
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Assignee Jacob Mathew [ jacob-mathew ] Kentoku [ kentoku ]
            Status In Progress [ 3 ] In Review [ 10002 ]

            Hi Jacob,
            I think this changes makes it impossible to uninstall plugin if plugin->ref_count is not 0 at do_uninstall(). Would you please check it?

            Kentoku Kentoku Shiba (Inactive) added a comment - Hi Jacob, I think this changes makes it impossible to uninstall plugin if plugin->ref_count is not 0 at do_uninstall(). Would you please check it?
            Kentoku Kentoku Shiba (Inactive) made changes -
            Assignee Kentoku [ kentoku ] Jacob Mathew [ jacob-mathew ]
            Status In Review [ 10002 ] Stalled [ 10000 ]

            Kentoku, these changes are designed to ensure that Spider's plugin->ref_count is correct. They do not change the do_uninstall() function, which will issue the following warning if plugin->ref_count is not 0:

            Plugin is busy and will be uninstalled on shutdown
            

            jacob-mathew Jacob Mathew (Inactive) added a comment - Kentoku, these changes are designed to ensure that Spider's plugin->ref_count is correct. They do not change the do_uninstall() function, which will issue the following warning if plugin->ref_count is not 0: Plugin is busy and will be uninstalled on shutdown

            Fix is pushed to 10.3.

            jacob-mathew Jacob Mathew (Inactive) added a comment - Fix is pushed to 10.3.
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Fix Version/s 10.3.6 [ 23003 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.3 [ 22126 ]
            Resolution Fixed [ 1 ]
            Status Stalled [ 10000 ] Closed [ 6 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Fix Version/s 10.2.16 [ 23110 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            Fix Version/s 10.1.34 [ 23100 ]
            jacob-mathew Jacob Mathew (Inactive) made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 61625 ] MariaDB v4 [ 149003 ]
            nayuta-yanagisawa Nayuta Yanagisawa (Inactive) made changes -
            ycp Yuchen Pei made changes -

            People

              jacob-mathew Jacob Mathew (Inactive)
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.