Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-4179

Server crash / memory corruption with MariaDB-Galera

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 5.5.28a-galera
    • N/A
    • Galera
    • None
    • CentOS 5, Ubuntu 11.10

    Description

      130218 7:14:08 [Note] Slave I/O thread: connected to master 'replication@10.240.170.40:53306',replication started in log 'mysql-bin.001384' at position 796660420
      130218 7:14:08 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.

      To report this bug, see http://kb.askmonty.org/en/reporting-bugs

      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.

      Server version: 5.5.28a-MariaDB-log
      key_buffer_size=32768
      read_buffer_size=32768
      max_used_connections=0
      max_threads=102
      thread_count=0
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2511512 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x0x56ceb010
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x2b4098605080 thread_stack 0x48000
      130218 7:14:08 InnoDB: Starting shutdown...
      ??:0(my_print_stacktrace)[0xab166e]
      ??:0(handle_fatal_signal)[0x6eb91c]
      :0()[0x37e080eca0]
      ??:0(my_real_read(st_net*, unsigned long*))[0x50609e]
      ??:0(my_net_read)[0x5065ce]
      ??:0(cli_safe_read)[0x6c4416]
      ??:0(read_event(st_mysql*, Master_info*, bool*))[0x5125ee]
      ??:0(handle_slave_io)[0x51eb11]
      ??:0(pfs_spawn_thread)[0x85e608]
      :0()[0x37e080683d]
      :0()[0x37e00d503d]

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x0): is an invalid pointer
      Connection ID (thread ID): 2
      Status: NOT_KILLED

      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=on,mrr_cost_based=on,mrr_sort_keys=on,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.

      [root@WPDB03 ~]# *** glibc detected *** /usr/sbin/mysqld: malloc(): memory corruption (fast): 0x0000000058a58cd0 ***
      ======= Backtrace: =========
      /lib64/libc.so.6[0x37e0072c3b]
      /lib64/libc.so.6(__libc_malloc+0x6e)[0x37e0073f6e]
      /usr/sbin/mysqld(my_malloc+0x42)[0xaada92]
      /usr/sbin/mysqld(_Z17flush_master_infoP11Master_infobb+0x10b)[0x64269b]
      /usr/sbin/mysqld(handle_slave_io+0x12f9)[0x51ebd9]
      /usr/sbin/mysqld[0x85e608]
      /lib64/libpthread.so.0[0x37e080683d]
      /lib64/libc.so.6(clone+0x6d)[0x37e00d503d]
      ======= Memory map: ========
      00400000-00f4c000 r-xp 00000000 08:03 11798179 /usr/sbin/mysqld
      0114b000-01197000 rw-p 00b4b000 08:03 11798179 /usr/sbin/mysqld
      01197000-019e0000 rw-p 01197000 00:00 0
      01b96000-01c14000 rw-p 00b96000 08:03 11798179 /usr/sbin/mysqld
      1cf5e000-58d51000 rw-p 1cf5e000 00:00 0 [heap]
      37dfc00000-37dfc1c000 r-xp 00000000 08:03 134611002 /lib64/ld-2.5.so
      37dfe1c000-37dfe1d000 r--p 0001c000 08:03 134611002 /lib64/ld-2.5.so
      37dfe1d000-37dfe1e000 rw-p 0001d000 08:03 134611002 /lib64/ld-2.5.so
      37e0000000-37e014f000 r-xp 00000000 08:03 134611021 /lib64/libc-2.5.so
      37e014f000-37e034f000 ---p 0014f000 08:03 134611021 /lib64/libc-2.5.so
      37e034f000-37e0353000 r--p 0014f000 08:03 134611021 /lib64/libc-2.5.so
      37e0353000-37e0354000 rw-p 00153000 08:03 134611021 /lib64/libc-2.5.so
      37e0354000-37e0359000 rw-p 37e0354000 00:00 0
      37e0400000-37e0402000 r-xp 00000000 08:03 134611025 /lib64/libdl-2.5.so
      37e0402000-37e0602000 ---p 00002000 08:03 134611025 /lib64/libdl-2.5.so
      37e0602000-37e0603000 r--p 00002000 08:03 134611025 /lib64/libdl-2.5.so
      37e0603000-37e0604000 rw-p 00003000 08:03 134611025 /lib64/libdl-2.5.so
      37e0800000-37e0816000 r-xp 00000000 08:03 134611023 /lib64/libpthread-2.5.so
      37e0816000-37e0a16000 ---p 00016000 08:03 134611023 /lib64/libpthread-2.5.so
      37e0a16000-37e0a17000 r--p 00016000 08:03 134611023 /lib64/libpthread-2.5.so
      37e0a17000-37e0a18000 rw-p 00017000 08:03 134611023 /lib64/libpthread-2.5.so
      37e0a18000-37e0a1c000 rw-p 37e0a18000 00:00 0
      37e0c00000-37e0c3b000 r-xp 00000000 08:03 134611028 /lib64/libsepol.so.1
      37e0c3b000-37e0e3b000 ---p 0003b000 08:03 134611028 /lib64/libsepol.so.1
      37e0e3b000-37e0e3c000 rw-p 0003b000 08:03 134611028 /lib64/libsepol.so.1
      37e0e3c000-37e0e46000 rw-p 37e0e3c000 00:00 0
      37e1000000-37e1015000 r-xp 00000000 08:03 134611029 /lib64/libselinux.so.1
      37e1015000-37e1215000 ---p 00015000 08:03 134611029 /lib64/libselinux.so.1
      37e1215000-37e1217000 rw-p 00015000 08:03 134611029 /lib64/libselinux.so.1
      37e1217000-37e1218000 rw-p 37e1217000 00:00 0
      37e1400000-37e1482000 r-xp 00000000 08:03 134611040 /lib64/libm-2.5.so
      37e1482000-37e1681000 ---p 00082000 08:03 134611040 /lib64/libm-2.5.so
      37e1681000-37e1682000 r--p 00081000 08:03 134611040 /lib64/libm-2.5.so
      37e1682000-37e1683000 rw-p 00082000 08:03 134611040 /lib64/libm-2.5.so
      37e1800000-37e1814000 r-xp 00000000 08:03 134611031 /lib64/libz.so.1.2.3
      37e1814000-37e1a13000 ---p 00014000 08:03 134611031 /lib64/libz.so.1.2.3
      37e1a13000-37e1a14000 rw-p 00013000 08:03 134611031 /lib64/libz.so.1.2.3
      37e1c00000-37e1c07000 r-xp 00000000 08:03 134611024 /lib64/librt-2.5.so
      37e1c07000-37e1e07000 ---p 00007000 08:03 134611024 /lib64/librt-2.5.so
      37e1e07000-37e1e08000 r--p 00007000 08:03 134611024 /lib64/librt-2.5.so
      37e1e08000-37e1e09000 rw-p 00008000 08:03 134611024 /lib64/librt-2.5.so
      37e2000000-37e200d000 r-xp 00000000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
      37e200d000-37e220d000 ---p 0000d000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
      37e220d000-37e220e000 rw-p 0000d000 08:03 134611041 /lib64/libgcc_s-4.1.2-20080825.so.1
      37e2400000-37e2409000 r-xp 00000000 08:03 134611034 /lib64/libcrypt-2.5.so
      37e2409000-37e2608000 ---p 00009000 08:03 134611034 /lib64/libcrypt-2.5.so
      37e2608000-37e2609000 r--p 00008000 08:03 134611034 /lib64/libcrypt-2.5.so
      37e2609000-37e260a000 rw-p 00009000 08:03 134611034 /lib64/libcrypt-2.5.so
      37e260a000-37e2638000 rw-p 37e260a000 00:00 0
      37e2800000-37e28e6000 r-xp 00000000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
      37e28e6000-37e2ae5000 ---p 000e6000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
      37e2ae5000-37e2aeb000 r--p 000e5000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
      37e2aeb000-37e2aee000 rw-p 000eb000 08:03 11798342 /usr/lib64/libstdc++.so.6.0.8
      37e2aee000-37e2b00000 rw-p 37e2aee000 00:00 0
      37e2c00000-37e2c11000 r-xp 00000000 08:03 134611027 /lib64/libresolv-2.5.so
      37e2c11000-37e2e11000 ---p 00011000 08:03 134611027 /lib64/libresolv-2.5.so
      37e2e11000-37e2e12000 r--p 00011000 08:03 134611027 /lib64/libresolv-2.5.so
      37e2e12000-37e2e13000 rw-p 00012000 08:03 134611027 /lib64/libresolv-2.5.so
      37e2e13000-37e2e15000 rw-p 37e2e13000 00:00 0
      37e3000000-37e3002000 r-xp 00000000 08:03 134611026 /lib64/libkeyutils-1.2.so
      37e3002000-37e3201000 ---p 00002000 08:03 134611026 /lib64/libkeyutils-1.2.so
      37e3201000-37e3202000 rw-p 00001000 08:03 134611026 /lib64/libkeyutils-1.2.so
      37e3400000-37e3402000 r-xp 00000000 08:03 134611030 /lib64/libcom_err.so.2.1
      37e3402000-37e3601000 ---p 00002000 08:03 134611030 /lib64/libcom_err.so.2.1
      37e3601000-37e3602000 rw-p 00001000 08:03 134611030 /lib64/libcom_err.so.2.1
      37e3800000-37e3940000 r-xp 00000000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
      37e3940000-37e3b40000 ---p 00140000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
      37e3b40000-37e3b62000 rw-p 00140000 08:03 134611032 /lib64/libcrypto.so.0.9.8x
      37e3b62000-37e3b66000 rw-p 37e3b62000 00:00 0
      37e4000000-37e4024000 r-xp 00000000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
      37e4024000-37e4223000 ---p 00024000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
      37e4223000-37e4225000 rw-p 00023000 08:03 11798306 /usr/lib64/libk5crypto.so.3.1
      37e4400000-37e442c000 r-xp 00000000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
      37e442c000-37e462c000 ---p 0002c000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
      37e462c000-37e462e000 rw-p 0002c000 08:03 11798308 /usr/lib64/libgssapi_krb5.so.2.2
      37e4800000-37e4891000 r-xp 00000000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
      37e4891000-37e4a91000 ---p 00091000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
      37e4a91000-37e4a95000 rw-p 00091000 08:03 11798307 /usr/lib64/libkrb5.so.3.3
      37e4c00000-37e4c08000 r-xp 00000000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
      37e4c08000-37e4e07000 ---p 00008000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
      37e4e07000-37e4e08000 rw-p 00007000 08:03 11798305 /usr/lib64/libkrb5support.so.0.1
      37e5800000-37e584f000 r-xp 00000000 08:03 134611033 /lib64/libssl.so.0.9.8x
      37e584f000-37e5a4f000 ---p 0004f000 08:03 134611033 /lib64/libssl.so.0.9.8x
      37e5a4f000-37e5a55000 rw-p 0004f000 08:03 134611033 /lib64/libssl.so.0.9.8x
      2b3572849000-2b357284b000 rw-p 2b3572849000 00:00 0
      2b3572853000-2b357285b000 rw-p 2b3572853000 00:00 0
      2b357285b000-2b357285c000 ---p 2b357285b000 00:00 0
      2b357285c000-2b357325c000 rwxp 2b357285c000 00:00 0
      2b357325c000-2b358abef000 rw-p 2b357325c000 00:00 0
      2b358abf6000-2b358ac00000 r-xp 00000000 08:03 134610979 /lib64/libnss_files-2.5.so
      2b358ac00000-2b358adff000 ---p 0000a000 08:03 134610979 /lib64/libnss_files-2.5.so
      2b358adff000-2b358ae00000 r--p 00009000 08:03 134610979 /lib64/libnss_files-2.5.so
      2b358ae00000-2b358ae01000 rw-p 0000a000 08:03 134610979 /lib64/libnss_files-2.5.so
      2b358ae01000-2b3ce9cdc000 rw-p 2b358ae01000 00:00 0
      2b3ce9cdc000-2b3ce9cdd000 ---p 2b3ce9cdc000 00:00 0
      2b3ce9cdd000-2b3cea6dd000 rwxp 2b3ce9cdd000 00:00 0
      2b3cea6dd000-2b3cea6de000 ---p 2b3cea6dd000 00:00 0
      2b3cea6de000-2b3ceb0de000 rwxp 2b3cea6de000 00:00 0
      2b3ceb0de000-2b3ceb0df000 ---p 2b3ceb0de000 00:00 0
      2b3ceb0df000-2b3cebadf000 rwxp 2b3ceb0df000 00:00 0
      2b3cebadf000-2b3cebae0000 ---p 2b3cebadf000 00:00 0
      2b3cebae0000-2b3cec4e0000 rwxp 2b3cebae0000 00:00 0
      2b3cec4e0000-2b3cec4e1000 ---p 2b3cec4e0000 00:00 0
      2b3cec4e1000-2b3cecee1000 rwxp 2b3cec4e1000 00:00 0
      2b3cecee1000-2b3cecee2000 ---p 2b3cecee1000 00:00 0
      2b3cecee2000-2b3ced8e2000 rwxp 2b3cecee2000 00:00 0
      2b3ced8e2000-2b3ced8e3000 ---p 2b3ced8e2000 00:00 0
      2b3ced8e3000-2b3cee2e3000 rwxp 2b3ced8e3000 00:00 0
      2b3cee2e3000-2b3cee2e4000 ---p 2b3cee2e3000 00:00 0
      2b3cee2e4000-2b3ceece4000 rwxp 2b3cee2e4000 00:00 0
      2b3ceece4000-2b3ceece5000 ---p 2b3ceece4000 00:00 0
      2b3ceece5000-2b3cef6e5000 rwxp 2b3ceece5000 00:00 0
      2b3cef6e5000-2b3cef6e6000 ---p 2b3cef6e5000 00:00 0
      2b3cef6e6000-2b3cf00e6000 rwxp 2b3cef6e6000 00:00 0
      2b3cf00e6000-2b3cf00e7000 ---p 2b3cf00e6000 00:00 0
      2b3cf00e7000-2b3cf0ae7000 rwxp 2b3cf00e7000 00:00 0
      2b3cf0ae7000-2b3cf0ae8000 ---p 2b3cf0ae7000 00:00 0
      2b3cf0ae8000-2b3cf14e8000 rwxp 2b3cf0ae8000 00:00 0
      2b3cf14e8000-2b3cf14e9000 ---p 2b3cf14e8000 00:00 0
      2b3cf14e9000-2b3cf1ee9000 rwxp 2b3cf14e9000 00:00 0
      2b3cf1ee9000-2b3cf1eea000 ---p 2b3cf1ee9000 00:00 0
      2b3cf1eea000-2b3cf28ea000 rwxp 2b3cf1eea000 00:00 0
      2b3cf28ea000-2b3cf28eb000 ---p 2b3cf28ea000 00:00 0
      2b3cf28eb000-2b3cf32eb000 rwxp 2b3cf28eb000 00:00 0
      2b3cf32eb000-2b3cf32ec000 ---p 2b3cf32eb000 00:00 0
      2b3cf32ec000-2b3cf3cec000 rwxp 2b3cf32ec000 00:00 0
      2b3cf3cec000-2b3cf3ced000 ---p 2b3cf3cec000 00:00 0
      2b3cf3ced000-2b3cf46ed000 rwxp 2b3cf3ced000 00:00 0
      2b3cf46ed000-2b3cf46ee000 ---p 2b3cf46ed000 00:00 0
      2b3cf46ee000-2b3cf4736000 rwxp 2b3cf46ee000 00:00 0
      2b3cf4736000-2b3cf4737000 ---p 2b3cf4736000 00:00 0
      2b3cf4737000-2b3cf477f000 rwxp 2b3cf4737000 00:00 0
      2b3cf477f000-2b3cf4780000 ---p 2b3cf477f000 00:00 0
      2b3cf4780000-2b3cf47c8000 rwxp 2b3cf4780000 00:00 0
      2b3cf8000000-2b3cf8034000 rw-p 2b3cf8000000 00:00 0
      2b3cf8034000-2b3cfc000000 ---p 2b3cf8034000 00:00 0
      2b3d0c6d8000-2b3d13a5c000 rw-p 2b3d0c6d8000 00:00 0
      7fff16415000-7fff1642a000 rwxp 7ffffffe9000 00:00 0 [stack]
      7fff16436000-7fff16439000 r-xp 7fff16436000 00:00 0 [vdso]
      ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vsyscall]

      Attachments

        1. my.cnf
          4 kB
          Aleksey Sanin

        Activity

          seppo Seppo Jaakola added a comment -

          There are two issues:
          1. slave IO thread failed for segfault during reading of replication event
          2. after that, server restart fails for a bug in wsrep recovery processing

          The issue #2 has been reported in: https://bugs.launchpad.net/codership-mysql/+bug/1132974 and the fix was merged as a part of revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3383
          Issue #1 is not possible to analyze further with the given information. I can only see that slave IO thread has corrupted memory, but what caused the memory corruption is an open question.

          seppo Seppo Jaakola added a comment - There are two issues: 1. slave IO thread failed for segfault during reading of replication event 2. after that, server restart fails for a bug in wsrep recovery processing The issue #2 has been reported in: https://bugs.launchpad.net/codership-mysql/+bug/1132974 and the fix was merged as a part of revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3383 Issue #1 is not possible to analyze further with the given information. I can only see that slave IO thread has corrupted memory, but what caused the memory corruption is an open question.
          guoxiaoqiao Xiaoqiao Guo (Inactive) added a comment - - edited

          my mariadb cluster 5.5.29 with galera 23.2.4 also crashed, and cannot get stack info.

          130425 9:04:25 [Warning] WSREP: last seen seqno below limit for trx source: d07d44b1-acab-11e2-0800-a4b4a09fdbf6 version: 2 local: 1 state: CERTIFYING flags: 129 conn_id: 5 trx_id: 23378603 seqnos (l: 89318, g: 95459, s: 95448, d: -1, ts: 1366851863915610973)
          130425 9:58:02 [ERROR] mysqld got signal 11 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.

          To report this bug, see http://kb.askmonty.org/en/reporting-bugs

          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed,
          something is definitely wrong and this may fail.

          Server version: 5.5.29-MariaDB-log
          key_buffer_size=134217728
          read_buffer_size=131072
          max_used_connections=25
          max_threads=1026
          thread_count=22
          It is possible that mysqld could use up to
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2378518 K bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.

          Thread pointer: 0x0x7f86c00428d0
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          stack_bottom = 0x7f86b412dd68 thread_stack 0x48000
          :0()[0x7f86bb91774e]
          :0()[0x7f86bb55f1ec]
          :0()[0x7f86bac7d500]
          :0()[0x7f86bb4c3d08]
          :0()[0x7f86bb415e65]
          :0()[0x7f86bb41655c]
          :0()[0x7f867a7e02ad]
          :0()[0x7f867a7e362f]
          :0()[0x7f867a7f9b06]
          :0()[0x7f86bb41dfcb]
          :0()[0x7f86bb41ec50]
          :0()[0x7f86bb4209c7]
          :0()[0x7f86bb421283]
          :0()[0x7f86bb4d1c47]
          :0()[0x7f86bb4d1e21]
          :0()[0x7f86bac75851]
          :0()[0x7f86b95e990d]

          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x7f8618004c38): is an invalid pointer
          Connection ID (thread ID): 15
          Status: NOT_KILLED

          Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

          The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
          information that should help you find out what is causing the crash.
          130425 09:58:03 mysqld_safe Number of processes running now: 0
          130425 09:58:03 mysqld_safe WSREP: not restarting wsrep node automatically
          130425 09:58:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
          130425 10:03:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql

          and my.cnf content:

          [mysqld]
          datadir=/var/lib/mysql
          socket=/var/lib/mysql/mysql.sock
          user=mysql

          1. Disabling symbolic-links is recommended to prevent assorted security risks
            symbolic-links=0

          wsrep_provider=/usr/lib64/galera/libgalera_smm.so
          wsrep_cluster_address=gcomm://
          wsrep_node_address=gcomm://192.168.100.1
          wsrep_sst_auth=galera:passwd
          wsrep_node_incoming_address=192.168.100.1
          wsrep_sst_receive_address=192.168.100.1

          general_log=1
          general_log_file=/var/log/mysql/mysql.log
          slow_query_log=1
          slow_query_log_file=/var/log/mysql/mysql-slow.log
          long_query_time=1

          skip-name-resolve
          character-set-server=utf8
          max_connection=1024
          #max_connect_errors=1000

          #log-bin=/var/log/mysql/mysql-bin
          #expire_logs_day=3

          innodb_buffer_pool_size = 512M
          innodb_file_per_table = 1

          [mysqld_safe]
          log-error=/var/log/mysqld.log
          pid-file=/var/run/mysqld/mysqld.pid

          guoxiaoqiao Xiaoqiao Guo (Inactive) added a comment - - edited my mariadb cluster 5.5.29 with galera 23.2.4 also crashed, and cannot get stack info. 130425 9:04:25 [Warning] WSREP: last seen seqno below limit for trx source: d07d44b1-acab-11e2-0800-a4b4a09fdbf6 version: 2 local: 1 state: CERTIFYING flags: 129 conn_id: 5 trx_id: 23378603 seqnos (l: 89318, g: 95459, s: 95448, d: -1, ts: 1366851863915610973) 130425 9:58:02 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 5.5.29-MariaDB-log key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=25 max_threads=1026 thread_count=22 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2378518 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7f86c00428d0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f86b412dd68 thread_stack 0x48000 :0( ) [0x7f86bb91774e] :0( ) [0x7f86bb55f1ec] :0( ) [0x7f86bac7d500] :0( ) [0x7f86bb4c3d08] :0( ) [0x7f86bb415e65] :0( ) [0x7f86bb41655c] :0( ) [0x7f867a7e02ad] :0( ) [0x7f867a7e362f] :0( ) [0x7f867a7f9b06] :0( ) [0x7f86bb41dfcb] :0( ) [0x7f86bb41ec50] :0( ) [0x7f86bb4209c7] :0( ) [0x7f86bb421283] :0( ) [0x7f86bb4d1c47] :0( ) [0x7f86bb4d1e21] :0( ) [0x7f86bac75851] :0( ) [0x7f86b95e990d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f8618004c38): is an invalid pointer Connection ID (thread ID): 15 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 130425 09:58:03 mysqld_safe Number of processes running now: 0 130425 09:58:03 mysqld_safe WSREP: not restarting wsrep node automatically 130425 09:58:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended 130425 10:03:30 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql and my.cnf content: [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql Disabling symbolic-links is recommended to prevent assorted security risks symbolic-links=0 wsrep_provider=/usr/lib64/galera/libgalera_smm.so wsrep_cluster_address=gcomm:// wsrep_node_address=gcomm://192.168.100.1 wsrep_sst_auth=galera:passwd wsrep_node_incoming_address=192.168.100.1 wsrep_sst_receive_address=192.168.100.1 general_log=1 general_log_file=/var/log/mysql/mysql.log slow_query_log=1 slow_query_log_file=/var/log/mysql/mysql-slow.log long_query_time=1 skip-name-resolve character-set-server=utf8 max_connection=1024 #max_connect_errors=1000 #log-bin=/var/log/mysql/mysql-bin #expire_logs_day=3 innodb_buffer_pool_size = 512M innodb_file_per_table = 1 [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid

          it crashed when run a little time, total crashed 3 time this week.

          guoxiaoqiao Xiaoqiao Guo (Inactive) added a comment - it crashed when run a little time, total crashed 3 time this week.
          seppo Seppo Jaakola added a comment -

          @Xiaoqiao, it looks like your issue is not related to the slave processing bug reported in this tracker.

          To troubleshoot your crash further, make first sure that you have binlog_format=ROW.
          And, if you can enable core files, it helps in analyzing your crash.
          As you have general log enabled, Can you see if there is a specific SQL statement, which always triggers this crash?

          seppo Seppo Jaakola added a comment - @Xiaoqiao, it looks like your issue is not related to the slave processing bug reported in this tracker. To troubleshoot your crash further, make first sure that you have binlog_format=ROW. And, if you can enable core files, it helps in analyzing your crash. As you have general log enabled, Can you see if there is a specific SQL statement, which always triggers this crash?

          No feedback, so closing

          serg Sergei Golubchik added a comment - No feedback, so closing

          People

            sachin.setiya.007 Sachin Setiya (Inactive)
            aleksey.sanin Aleksey Sanin (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.