Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10349

buffer overflow detected ***: /usr/sbin/mysqld terminated

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.15
    • 10.1.16
    • Optimizer
    • None
    • CentOS7 VPS in XenServer 7
      32GB RAM, 12CPU
      cPanel 56.25
      CloudLinux (form cloudlinux-testing repository)

    Description

      Quite frequently mariadb began killing itself on our production server. This happening between 1 to 12 hours for the last 2 days, but did not happen before. I have commented out the most of custom variables that were added in server.cnf to see if this helps. The latest crash report from mysql log is:

      Server version: 10.1.15-MariaDB
      key_buffer_size=67108864
      read_buffer_size=131072
      max_used_connections=11
      max_threads=202
      thread_count=6
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 509205 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7f1a9f7e3008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f1f3da42100 thread_stack 0x48400
      *** buffer overflow detected ***: /usr/sbin/mysqld terminated
      ======= Backtrace: =========
      /lib64/libc.so.6(__fortify_fail+0x37)[0x7f1f3bbae597]
      /lib64/libc.so.6(+0x10c750)[0x7f1f3bbac750]
      /lib64/libc.so.6(+0x10e507)[0x7f1f3bbae507]
      /usr/sbin/mysqld(my_addr_resolve+0x48)[0x7f1f3e64d388]
      /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7f1f3e639a92]
      /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7f1f3e16095d]
      /lib64/libpthread.so.0(+0xf100)[0x7f1f3d77e100]
      /usr/sbin/mysqld(+0x6a0ae5)[0x7f1f3e24fae5]
      /usr/sbin/mysqld(_ZN10SQL_SELECT17test_quick_selectEP3THD6BitmapILj64EEyybbb+0x1d14)[0x7f1f3e25d734]
      /usr/sbin/mysqld(+0x476a9e)[0x7f1f3e025a9e]
      /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x2b70)[0x7f1f3e034ca0]
      /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x1b)[0x7f1f3e03511b]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x8f)[0x7f1f3e03526f]
      /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x245)[0x7f1f3e035dc5]
      /usr/sbin/mysqld(+0x4295c1)[0x7f1f3dfd85c1]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5f8f)[0x7f1f3dfe41cf]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x28e)[0x7f1f3dfe7aae]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x25db)[0x7f1f3dfeaf6b]
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7f1f3dfeb7e9]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f1f3e0af2aa]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f1f3e0af480]
      /usr/sbin/mysqld(+0x9684ed)[0x7f1f3e5174ed]
      /lib64/libpthread.so.0(+0x7dc5)[0x7f1f3d776dc5]
      /lib64/libc.so.6(clone+0x6d)[0x7f1f3bb96ced]
      ======= Memory map: ========
      7f1a80000000-7f1a94c00000 rw-p 00000000 00:00 0 
      7f1a94ff5000-7f1a94ff6000 ---p 00000000 00:00 0 
      7f1a94ff6000-7f1a957f6000 rw-p 00000000 00:00 0                          [stack:652357]
      7f1a957f6000-7f1a957f7000 ---p 00000000 00:00 0 
      7f1a957f7000-7f1a95ff7000 rw-p 00000000 00:00 0                          [stack:652356]
      7f1a95ff7000-7f1a95ff8000 ---p 00000000 00:00 0 
      7f1a95ff8000-7f1a967f8000 rw-p 00000000 00:00 0                          [stack:652355]
      7f1a967f8000-7f1a967f9000 ---p 00000000 00:00 0 
      7f1a967f9000-7f1a96ff9000 rw-p 00000000 00:00 0                          [stack:652354]
      7f1a96ff9000-7f1a96ffa000 ---p 00000000 00:00 0 
      7f1a96ffa000-7f1a977fa000 rw-p 00000000 00:00 0                          [stack:652353]
      7f1a977fa000-7f1a977fb000 ---p 00000000 00:00 0 
      7f1a977fb000-7f1a97ffb000 rw-p 00000000 00:00 0                          [stack:652351]
      7f1a97ffb000-7f1a97ffc000 ---p 00000000 00:00 0 
      7f1a97ffc000-7f1a987fc000 rw-p 00000000 00:00 0                          [stack:652350]
      7f1a987fc000-7f1a987fd000 ---p 00000000 00:00 0 
      7f1a987fd000-7f1a98ffd000 rw-p 00000000 00:00 0                          [stack:652349]
      7f1a98ffd000-7f1a98ffe000 ---p 00000000 00:00 0 
      7f1a98ffe000-7f1a997fe000 rw-p 00000000 00:00 0                          [stack:652348]
      7f1a997fe000-7f1a997ff000 ---p 00000000 00:00 0 
      7f1a997ff000-7f1a99fff000 rw-p 00000000 00:00 0                          [stack:652347]
      7f1a99fff000-7f1a9a000000 ---p 00000000 00:00 0 
      7f1a9a000000-7f1ab1c00000 rw-p 00000000 00:00 0                          [stack:652352]
      7f1ab1ff6000-7f1ab1ff7000 ---p 00000000 00:00 0 
      7f1ab1ff7000-7f1ab27f7000 rw-p 00000000 00:00 0                          [stack:652303]
      7f1ab27f7000-7f1ab27f8000 ---p 00000000 00:00 0 
      7f1ab27f8000-7f1ab2ff8000 rw-p 00000000 00:00 0                          [stack:652302]
      7f1ab2ff8000-7f1ab2ff9000 ---p 00000000 00:00 0 
      7f1ab2ff9000-7f1ab37f9000 rw-p 00000000 00:00 0                          [stack:652301]
      7f1ab37f9000-7f1ab37fa000 ---p 00000000 00:00 0 
      7f1ab37fa000-7f1ab3ffa000 rw-p 00000000 00:00 0                          [stack:652300]
      7f1ab3ffa000-7f1ab3ffb000 ---p 00000000 00:00 0 
      7f1ab3ffb000-7f1ab47fb000 rw-p 00000000 00:00 0                          [stack:652299]
      7f1ab47fb000-7f1ab47fc000 ---p 00000000 00:00 0 
      7f1ab47fc000-7f1ab4ffc000 rw-p 00000000 00:00 0                          [stack:652298]
      7f1ab4ffc000-7f1ab4ffd000 ---p 00000000 00:00 0 
      7f1ab4ffd000-7f1ab57fd000 rw-p 00000000 00:00 0                          [stack:652297]
      7f1ab57fd000-7f1ab57fe000 ---p 00000000 00:00 0 
      7f1ab57fe000-7f1ab5ffe000 rw-p 00000000 00:00 0                          [stack:652296]
      7f1ab5ffe000-7f1ab5fff000 ---p 00000000 00:00 0 
      7f1ab5fff000-7f1ab67ff000 rw-p 00000000 00:00 0                          [stack:652295]
      7f1ab67ff000-7f1ab6800000 ---p 00000000 00:00 0 
      7f1ab6800000-7f1acdc00000 rw-p 00000000 00:00 0                          [stack:652294]
      7f1acde00000-7f1b11400000 rw-p 00000000 00:00 0 
      7f1b11600000-7f1bdb400000 rw-p 00000000 00:00 0 
      7f1bdb600000-7f1ca5400000 rw-p 00000000 00:00 0 
      7f1ca5600000-7f1d6f400000 rw-p 00000000 00:00 0 
      7f1d6f600000-7f1e39400000 rw-p 00000000 00:00 0 
      7f1e39600000-7f1f03400000 rw-p 00000000 00:00 0 
      7f1f037d4000-7f1f037d5000 ---p 00000000 00:00 0 
      7f1f037d5000-7f1f03fd5000 rw-p 00000000 00:00 0                          [stack:652274]
      7f1f03fd5000-7f1f03fd6000 ---p 00000000 00:00 0 
      7f1f03fd6000-7f1f047d6000 rw-p 00000000 00:00 0                          [stack:652273]
      7f1f047d6000-7f1f047d7000 ---p 00000000 00:00 0 
      7f1f047d7000-7f1f04fd7000 rw-p 00000000 00:00 0                          [stack:652272]
      7f1f04fd7000-7f1f04fd8000 ---p 00000000 00:00 0 
      7f1f04fd8000-7f1f057d8000 rw-p 00000000 00:00 0                          [stack:652271]
      7f1f057d8000-7f1f057d9000 ---p 00000000 00:00 0 
      7f1f057d9000-7f1f05fd9000 rw-p 00000000 00:00 0                          [stack:652270]
      7f1f05fd9000-7f1f05fda000 ---p 00000000 00:00 0 
      7f1f05fda000-7f1f067da000 rw-p 00000000 00:00 0                          [stack:652269]
      7f1f067da000-7f1f067db000 ---p 00000000 00:00 0 
      7f1f067db000-7f1f06fdb000 rw-p 00000000 00:00 0                          [stack:652268]
      7f1f06fdb000-7f1f06fdc000 ---p 00000000 00:00 0 
      7f1f06fdc000-7f1f077dc000 rw-p 00000000 00:00 0                          [stack:652267]
      7f1f077dc000-7f1f077dd000 ---p 00000000 00:00 0 
      7f1f077dd000-7f1f07fdd000 rw-p 00000000 00:00 0                          [stack:652266]
      7f1f07fdd000-7f1f07fde000 ---p 00000000 00:00 0 
      7f1f07fde000-7f1f087de000 rw-p 00000000 00:00 0                          [stack:652265]
      7f1f087de000-7f1f087df000 ---p 00000000 00:00 0 
      7f1f087df000-7f1f08fdf000 rw-p 00000000 00:00 0                          [stack:652264]
      7f1f08fdf000-7f1f08fe0000 ---p 00000000 00:00 0 
      7f1f08fe0000-7f1f097e0000 rw-p 00000000 00:00 0                          [stack:652263]
      7f1f097e0000-7f1f097e1000 ---p 00000000 00:00 0 
      7f1f097e1000-7f1f09fe1000 rw-p 00000000 00:00 0                          [stack:652262]
      7f1f09fe1000-7f1f09fe2000 ---p 00000000 00:00 0 
      7f1f09fe2000-7f1f0a7e2000 rw-p 00000000 00:00 0                          [stack:652261]
      7f1f0a7e2000-7f1f0a7e3000 ---p 00000000 00:00 0 
      7f1f0a7e3000-7f1f0afe3000 rw-p 00000000 00:00 0                          [stack:652260]
      7f1f0afe3000-7f1f0afe4000 ---p 00000000 00:00 0 
      7f1f0afe4000-7f1f0b7e4000 rw-p 00000000 00:00 0                          [stack:652259]
      7f1f0b7e4000-7f1f0b7e5000 ---p 00000000 00:00 0 
      7f1f0b7e5000-7f1f0bfe5000 rw-p 00000000 00:00 0                          [stack:652258]
      7f1f0bfe5000-7f1f0bfe6000 ---p 00000000 00:00 0 
      7f1f0bfe6000-7f1f0c7e6000 rw-p 00000000 00:00 0                          [stack:652257]
      7f1f0c7e6000-7f1f0c7e7000 ---p 00000000 00:00 0 
      7f1f0c7e7000-7f1f0cfe7000 rw-p 00000000 00:00 0                          [stack:652256]
      7f1f0cfe7000-7f1f0cfe8000 ---p 00000000 00:00 0 
      7f1f0cfe8000-7f1f0d7e8000 rw-p 00000000 00:00 0                          [stack:652255]
      7f1f0d7e8000-7f1f0d7e9000 ---p 00000000 00:00 0 
      7f1f0d7e9000-7f1f0dfe9000 rw-p 00000000 00:00 0                          [stack:652254]
      7f1f0dfe9000-7f1f0dfea000 ---p 00000000 00:00 0 
      7f1f0dfea000-7f1f0e7ea000 rw-p 00000000 00:00 0                          [stack:652253]
      7f1f0e7ea000-7f1f0e7eb000 ---p 00000000 00:00 0 
      7f1f0e7eb000-7f1f0efeb000 rw-p 00000000 00:00 0                          [stack:652252]
      7f1f0efeb000-7f1f0efec000 ---p 00000000 00:00 0 
      7f1f0efec000-7f1f0f7ec000 rw-p 00000000 00:00 0                          [stack:652251]
      7f1f0f7ec000-7f1f0f7ed000 ---p 00000000 00:00 0 
      7f1f0f7ed000-7f1f0ffed000 rw-p 00000000 00:00 0                          [stack:652250]
      7f1f0ffed000-7f1f0ffee000 ---p 00000000 00:00 0 
      7f1f0ffee000-7f1f107ee000 rw-p 00000000 00:00 0                          [stack:652249]
      7f1f107ee000-7f1f107ef000 ---p 00000000 00:00 0 
      7f1f107ef000-7f1f10fef000 rw-p 00000000 00:00 0                          [stack:652248]
      7f1f10fef000-7f1f10ff0000 ---p 00000000 00:00 0 
      7f1f10ff0000-7f1f117f0000 rw-p 00000000 00:00 0                          [stack:652247]
      7f1f117f0000-7f1f117f1000 ---p 00000000 00:00 0 
      7f1f117f1000-7f1f11ff1000 rw-p 00000000 00:00 0                          [stack:652246]
      7f1f11ff1000-7f1f11ff2000 ---p 00000000 00:00 0 
      7f1f11ff2000-7f1f127f2000 rw-p 00000000 00:00 0                          [stack:652245]
      7f1f127f2000-7f1f127f3000 ---p 00000000 00:00 0 
      7f1f127f3000-7f1f12ff3000 rw-p 00000000 00:00 0                          [stack:652244]
      7f1f12ff3000-7f1f12ff4000 ---p 00000000 00:00 0 
      7f1f12ff4000-7f1f137f4000 rw-p 00000000 00:00 0                          [stack:652243]
      7f1f137f4000-7f1f137f5000 ---p 00000000 00:00 0 
      7f1f137f5000-7f1f13ff5000 rw-p 00000000 00:00 0                          [stack:652242]
      7f1f13ff5000-7f1f13ff6000 ---p 00000000 00:00 0 
      7f1f13ff6000-7f1f147f6000 rw-p 00000000 00:00 0                          [stack:652241]
      7f1f147f6000-7f1f147f7000 ---p 00000000 00:00 0 
      7f1f147f7000-7f1f14ff7000 rw-p 00000000 00:00 0                          [stack:652240]
      7f1f14ff7000-7f1f14ff8000 ---p 00000000 00:00 0 
      7f1f14ff8000-7f1f157f8000 rw-p 00000000 00:00 0                          [stack:652239]
      7f1f157f8000-7f1f157f9000 ---p 00000000 00:00 0 
      7f1f157f9000-7f1f15ff9000 rw-p 00000000 00:00 0                          [stack:652238]
      7f1f15ff9000-7f1f15ffa000 ---p 00000000 00:00 0 
      7f1f15ffa000-7f1f167fa000 rw-p 00000000 00:00 0                          [stack:652237]
      7f1f167fa000-7f1f167fb000 ---p 00000000 00:00 0 
      7f1f167fb000-7f1f16ffb000 rw-p 00000000 00:00 0                          [stack:652236]
      7f1f16ffb000-7f1f16ffc000 ---p 00000000 00:00 0 
      7f1f16ffc000-7f1f177fc000 rw-p 00000000 00:00 0                          [stack:652235]
      7f1f177fc000-7f1f177fd000 ---p 00000000 00:00 0 
      7f1f177fd000-7f1f17ffd000 rw-p 00000000 00:00 0                          [stack:652234]
      7f1f17ffd000-7f1f17ffe000 ---p 00000000 00:00 0 
      7f1f17ffe000-7f1f187fe000 rw-p 00000000 00:00 0                          [stack:652233]
      7f1f187fe000-7f1f187ff000 ---p 00000000 00:00 0 
      7f1f187ff000-7f1f18fff000 rw-p 00000000 00:00 0                          [stack:652232]
      7f1f18fff000-7f1f19000000 ---p 00000000 00:00 0 
      7f1f19000000-7f1f20000000 rw-p 00000000 00:00 0                          [stack:652231]
      7f1f203ff000-7f1f20400000 ---p 00000000 00:00 0 
      7f1f20400000-7f1f21000000 rw-p 00000000 00:00 0                          [stack:652230]
      7f1f213ff000-7f1f21400000 ---p 00000000 00:00 0 
      7f1f21400000-7f1f2f000000 rw-p 00000000 00:00 0                          [stack:652229]
      7f1f2f006000-7f1f2f007000 ---p 00000000 00:00 0 
      7f1f2f007000-7f1f2f050000 rw-p 00000000 00:00 0                          [stack:863330]
      7f1f2f050000-7f1f2f1ee000 r-xp 00000000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f1ee000-7f1f2f3ee000 ---p 0019e000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3ee000-7f1f2f3f3000 r--p 0019e000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3f3000-7f1f2f3f6000 rw-p 001a3000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3f6000-7f1f2f3ff000 rw-p 00000000 00:00 0 
      7f1f2f3ff000-7f1f2f400000 ---p 00000000 00:00 0 
      7f1f2f400000-7f1f39400000 rw-p 00000000 00:00 0                          [stack:652228]
      7f1f39432000-7f1f39433000 ---p 00000000 00:00 0 
      7f1f39433000-7f1f3947c000 rw-p 00000000 00:00 0                          [stack:863329]
      7f1f3947c000-7f1f3947d000 ---p 00000000 00:00 0 
      7f1f3947d000-7f1f394c6000 rw-p 00000000 00:00 0                          [stack:655783]
      7f1f394c6000-7f1f394c7000 ---p 00000000 00:00 0 
      7f1f394c7000-7f1f39510000 rw-p 00000000 00:00 0                          [stack:655279]
      7f1f39510000-7f1f39511000 ---p 00000000 00:00 0 
      7f1f39511000-7f1f3955a000 rw-p 00000000 00:00 0                          [stack:655278]
      7f1f3955a000-7f1f3955b000 ---p 00000000 00:00 0 
      7f1f3955b000-7f1f395a4000 rw-p 00000000 00:00 0                          [stack:652782]
      7f1f395a4000-7f1f395a5000 ---p 00000000 00:00 0 
      7f1f395a5000-7f1f395ee000 rw-p 00000000 00:00 0                          [stack:865914]
      7f1f395ee000-7f1f3964e000 r-xp 00000000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3964e000-7f1f3984d000 ---p 00060000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984d000-7f1f3984e000 r--p 0005f000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984e000-7f1f3984f000 rw-p 00060000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984f000-7f1f39870000 r-xp 00000000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39870000-7f1f39a70000 ---p 00021000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a70000-7f1f39a71000 r--p 00021000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a71000-7f1f39a72000 rw-p 00022000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a72000-7f1f39a74000 rw-p 00000000 00:00 0 
      7f1f39a74000-7f1f39a83000 r-xp 00000000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39a83000-7f1f39c82000 ---p 0000f000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c82000-7f1f39c83000 r--p 0000e000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c83000-7f1f39c84000 rw-p 0000f000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c84000-7f1f39c99000 r-xp 00000000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39c99000-7f1f39e98000 ---p 00015000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e98000-7f1f39e99000 r--p 00014000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e99000-7f1f39e9a000 rw-p 00015000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e9a000-7f1f39e9e000 r-xp 00000000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f39e9e000-7f1f3a09d000 ---p 00004000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09d000-7f1f3a09e000 r--p 00003000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09e000-7f1f3a09f000 rw-p 00004000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09f000-7f1f3a0b5000 r-xp 00000000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a0b5000-7f1f3a2b5000 ---p 00016000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b5000-7f1f3a2b6000 r--p 00016000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b6000-7f1f3a2b7000 rw-p 00017000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b7000-7f1f3a2b9000 rw-p 00000000 00:00 0 
      7f1f3a2b9000-7f1f3a2bc000 r-xp 00000000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a2bc000-7f1f3a4bb000 ---p 00003000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bb000-7f1f3a4bc000 r--p 00002000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bc000-7f1f3a4bd000 rw-p 00003000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bd000-7f1f3a4ca000 r-xp 00000000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a4ca000-7f1f3a6ca000 ---p 0000d000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6ca000-7f1f3a6cb000 r--p 0000d000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6cb000-7f1f3a6cc000 rw-p 0000e000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6cc000-7f1f3a6e1000 r-xp 00000000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a6e1000-7f1f3a8e0000 ---p 00015000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e0000-7f1f3a8e1000 r--p 00014000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e1000-7f1f3a8e2000 rw-p 00015000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e2000-7f1f3a927000 r-xp 00000000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3a927000-7f1f3ab26000 ---p 00045000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab26000-7f1f3ab28000 r--p 00044000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab28000-7f1f3ab29000 rw-p 00046000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab29000-7f1f3ab2d000 r-xp 00000000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ab2d000-7f1f3ad2c000 ---p 00004000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2c000-7f1f3ad2d000 r--p 00003000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2d000-7f1f3ad2e000 rw-p 00004000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2e000-7f1f3ad35000 r-xp 00000000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3ad35000-7f1f3af34000 ---p 00007000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af34000-7f1f3af35000 r--p 00006000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af35000-7f1f3af36000 rw-p 00007000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af36000-7f1f3af65000 r-xp 00000000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3af65000-7f1f3b164000 ---p 0002f000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b164000-7f1f3b166000 r--p 0002e000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b166000-7f1f3b167000 rw-p 00030000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b167000-7f1f3b168000 rw-p 00000000 00:00 0 
      7f1f3b168000-7f1f3b16b000 r-xp 00000000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b16b000-7f1f3b36a000 ---p 00003000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36a000-7f1f3b36b000 r--p 00002000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36b000-7f1f3b36c000 rw-p 00003000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36c000-7f1f3b441000 r-xp 00000000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b441000-7f1f3b641000 ---p 000d5000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b641000-7f1f3b64e000 r--p 000d5000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b64e000-7f1f3b651000 rw-p 000e2000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b651000-7f1f3b69a000 r-xp 00000000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b69a000-7f1f3b89a000 ---p 00049000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89a000-7f1f3b89b000 r--p 00049000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89b000-7f1f3b89d000 rw-p 0004a000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89d000-7f1f3b89f000 r-xp 00000000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3b89f000-7f1f3ba9e000 ---p 00002000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3ba9e000-7f1f3ba9f000 r--p 00001000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3ba9f000-7f1f3baa0000 rw-p 00002000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3baa0000-7f1f3bc57000 r-xp 00000000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3bc57000-7f1f3be57000 ---p 001b7000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be57000-7f1f3be5b000 r--p 001b7000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be5b000-7f1f3be5d000 rw-p 001bb000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be5d000-7f1f3be62000 rw-p 00000000 00:00 0 
      7f1f3be62000-7f1f3bf63000 r-xp 00000000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3bf63000-7f1f3c162000 ---p 00101000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c162000-7f1f3c163000 r--p 00100000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c163000-7f1f3c164000 rw-p 00101000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c164000-7f1f3c24d000 r-xp 00000000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c24d000-7f1f3c44d000 ---p 000e9000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c44d000-7f1f3c456000 r--p 000e9000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c456000-7f1f3c458000 rw-p 000f2000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c458000-7f1f3c46d000 rw-p 00000000 00:00 0 
      7f1f3c46d000-7f1f3c470000 r-xp 00000000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c470000-7f1f3c66f000 ---p 00003000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c66f000-7f1f3c670000 r--p 00002000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c670000-7f1f3c671000 rw-p 00003000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c671000-7f1f3c82f000 r-xp 00000000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3c82f000-7f1f3ca2f000 ---p 001be000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca2f000-7f1f3ca49000 r--p 001be000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca49000-7f1f3ca55000 rw-p 001d8000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca55000-7f1f3ca59000 rw-p 00000000 00:00 0 
      7f1f3ca59000-7f1f3cabc000 r-xp 00000000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3cabc000-7f1f3ccbb000 ---p 00063000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccbb000-7f1f3ccbf000 r--p 00062000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccbf000-7f1f3ccc6000 rw-p 00066000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccc6000-7f1f3ccce000 r-xp 00000000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3ccce000-7f1f3cecd000 ---p 00008000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cecd000-7f1f3cece000 r--p 00007000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cece000-7f1f3cecf000 rw-p 00008000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cecf000-7f1f3cefd000 rw-p 00000000 00:00 0 
      7f1f3cefd000-7f1f3cf2f000 r-xp 00000000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3cf2f000-7f1f3d12e000 ---p 00032000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d12e000-7f1f3d130000 r--p 00031000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d130000-7f1f3d131000 rw-p 00033000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d131000-7f1f3d132000 rw-p 00000000 00:00 0 
      7f1f3d132000-7f1f3d147000 r-xp 00000000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d147000-7f1f3d346000 ---p 00015000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d346000-7f1f3d347000 r--p 00014000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d347000-7f1f3d348000 rw-p 00015000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d348000-7f1f3d349000 r-xp 00000000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d349000-7f1f3d548000 ---p 00001000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d548000-7f1f3d549000 r--p 00000000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d549000-7f1f3d54a000 rw-p 00001000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d54a000-7f1f3d56e000 r-xp 00000000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d56e000-7f1f3d76d000 ---p 00024000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76d000-7f1f3d76e000 r--p 00023000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76e000-7f1f3d76f000 rw-p 00024000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76f000-7f1f3d785000 r-xp 00000000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d785000-7f1f3d985000 ---p 00016000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d985000-7f1f3d986000 r--p 00016000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d986000-7f1f3d987000 rw-p 00017000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d987000-7f1f3d98b000 rw-p 00000000 00:00 0 
      7f1f3d98b000-7f1f3d9ac000 r-xp 00000000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3d9b0000-7f1f3d9b1000 ---p 00000000 00:00 0 
      7f1f3d9b1000-7f1f3d9fa000 rw-p 00000000 00:00 0                          [stack:652511]
      7f1f3d9fa000-7f1f3d9fb000 ---p 00000000 00:00 0 
      7f1f3d9fb000-7f1f3da44000 rw-p 00000000 00:00 0                          [stack:181434]
      7f1f3da44000-7f1f3da45000 ---p 00000000 00:00 0 
      7f1f3da45000-7f1f3da8e000 rw-p 00000000 00:00 0                          [stack:652425]
      7f1f3da8e000-7f1f3da8f000 ---p 00000000 00:00 0 
      7f1f3da8f000-7f1f3dad8000 rw-p 00000000 00:00 0                          [stack:652402]
      7f1f3dad8000-7f1f3dad9000 ---p 00000000 00:00 0 
      7f1f3dad9000-7f1f3db22000 rw-p 00000000 00:00 0                          [stack:652358]
      7f1f3db22000-7f1f3db57000 r--s 00000000 ca:21 537306668                  /var/db/nscd/services
      7f1f3db57000-7f1f3db5d000 rw-s 00000000 ca:21 805384595                  /var/lib/mysql/tc.log
      7f1f3db5d000-7f1f3db62000 rw-s 00000000 00:0a 383893695                  /[aio] (deleted)
      7f1f3db62000-7f1f3db67000 rw-s 00000000 00:0a 383893694                  /[aio] (deleted)
      7f1f3db67000-7f1f3db6c000 rw-s 00000000 00:0a 383893693                  /[aio] (deleted)
      7f1f3db6c000-7f1f3db71000 rw-s 00000000 00:0a 383893692                  /[aio] (deleted)
      7f1f3db71000-7f1f3db76000 rw-s 00000000 00:0a 383893691                  /[aio] (deleted)
      7f1f3db76000-7f1f3db7b000 rw-s 00000000 00:0a 383893690                  /[aio] (deleted)
      7f1f3db7b000-7f1f3db80000 rw-s 00000000 00:0a 383893689                  /[aio] (deleted)
      7f1f3db80000-7f1f3db85000 rw-s 00000000 00:0a 383893688                  /[aio] (deleted)
      7f1f3db85000-7f1f3db93000 rw-p 00000000 00:00 0 
      7f1f3db93000-7f1f3db97000 r-xp 00000000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db97000-7f1f3db98000 ---p 00004000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db98000-7f1f3db99000 r--p 00004000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db99000-7f1f3db9a000 rw-p 00005000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db9a000-7f1f3db9e000 rw-p 00000000 00:00 0 
      7f1f3db9e000-7f1f3dba0000 rw-s 00000000 00:0a 383893696                  /[aio] (deleted)
      7f1f3dba0000-7f1f3dba5000 rw-s 00000000 00:0a 383893687                  /[aio] (deleted)
      7f1f3dba5000-7f1f3dbaa000 rw-s 00000000 00:0a 383893686                  /[aio] (deleted)
      7f1f3dbaa000-7f1f3dbab000 rw-s 00000000 00:0a 383893685                  /[aio] (deleted)
      7f1f3dbab000-7f1f3dbac000 rw-p 00000000 00:00 0 
      7f1f3dbac000-7f1f3dbad000 r--p 00021000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3dbad000-7f1f3dbae000 rw-p 00022000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3dbae000-7f1f3dbaf000 rw-p 00000000 00:00 0 
      7f1f3dbaf000-7f1f3eb26000 r-xp 00000000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3ed26000-7f1f3edd6000 r--p 00f77000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3edd6000-7f1f3ee87000 rw-p 01027000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3ee87000-7f1f3f6fd000 rw-p 00000000 00:00 0 
      7ffe6a7f8000-7ffe6a819000 rw-p 00000000 00:00 0                          [stack]
      7ffe6a907000-7ffe6a909000 r-xp 00000000 00:00 0                          [vdso]
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      Fatal signal 6 while backtracing
      Sat Jul  9 09:43:01 2016 PerconaFT recovery starting in env /var/lib/mysql/
      Sat Jul  9 09:43:01 2016 PerconaFT recovery scanning backward from 562001
      Sat Jul  9 09:43:01 2016 PerconaFT recovery bw_end_checkpoint at 562001 timestamp 1468021178926184 xid 561997 (bw_newer)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery bw_begin_checkpoint at 561997 timestamp 1468021178926159 (bw_between)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery turning around at begin checkpoint 561997 time 25
      Sat Jul  9 09:43:01 2016 PerconaFT recovery starts scanning forward to 562001 from 561997 left 4 (fw_between)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery closing 2 dictionaries
      Sat Jul  9 09:43:01 2016 PerconaFT recovery making a checkpoint
      Sat Jul  9 09:43:01 2016 PerconaFT recovery done
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Memory barrier is not used
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using Linux native AIO
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using SSE crc32 instructions
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Initializing buffer pool, size = 16.0G
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Completed initialization of buffer pool
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Highest supported file format is Barracuda.
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Log scan progressed past the checkpoint lsn 171599114149
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Database was not shutdown normally!
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Starting crash recovery.
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Reading tablespace information from the .ibd files...
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: Restoring possible half-written data pages 
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 171599131045
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB: 128 rollback segment(s) are active.
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB: Waiting for purge to start
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 171599131045
      2016-07-09  9:43:06 139874141181696 [Note] InnoDB: Dumping buffer pool(s) not yet started
      2016-07-09  9:43:06 139894138476672 [Note] Plugin 'FEEDBACK' is disabled.
      2016-07-09  9:43:06 139894138476672 [Note] Recovering after a crash using tc.log
      2016-07-09  9:43:06 139894138476672 [Note] Starting crash recovery...
      2016-07-09  9:43:06 139894138476672 [Note] Crash recovery finished.
      2016-07-09  9:43:06 139894138476672 [Note] Server socket created on IP: '127.0.0.1'.
      2016-07-09  9:43:06 139894138476672 [Warning] 'user' entry 'root@cpanel.redy.host' ignored in --skip-name-resolve mode.
      

      The entire [mariadb] block was active upon latest crash:

      [mariadb]
      max_connections=200
      skip-external-locking
      skip-innodb_doublewrite
      key_buffer_size = 64M
      max_allowed_packet = 128M
      table_open_cache = 64
      thread_cache_size = 8
      query_cache_limit = 16M
      tmp_table_size = 128M
      max_heap_table_size = 128M
      join_buffer_size = 16M
      default-storage-engine = innodb
      innodb_log_file_size = 2G
      innodb_flush_method = O_DIRECT
      innodb_file_per_table = 1
      innodb_open_files = 6000
      innodb_flush_log_at_trx_commit = 2
      innodb_log_buffer_size = 16M
      innodb_additional_mem_pool_size = 8M
      innodb_buffer_pool_size = 16G
      innodb_thread_concurrency = 0
      innodb_buffer_pool_instances=16
      query_cache_type = 1
      

      Attachments

        Issue Links

          Activity

            philsweeney Phil Sweeney added a comment -

            ivangrynenko are you aware that MariaDB 10.1.15 was pulled due to a crash regression bug? Not sure if that's what you're hitting but if it only happened last couple days (and assume only on 10.1.15), that could be it.

            See https://jira.mariadb.org/browse/MDEV-10324

            philsweeney Phil Sweeney added a comment - ivangrynenko are you aware that MariaDB 10.1.15 was pulled due to a crash regression bug? Not sure if that's what you're hitting but if it only happened last couple days (and assume only on 10.1.15), that could be it. See https://jira.mariadb.org/browse/MDEV-10324

            I agree with philsweeney – the visible part of the stack trace looks similar to MDEV-10324, and given that it only started happening recently, it's most likely the same problem.

            ivangrynenko, if you want to experiment, the fix has now been pushed into 10.1 tree. Otherwise, I'm afraid the best advice is to roll back to 10.1.14 until the next release of 10.1 is out.
            Sorry for the trouble.

            elenst Elena Stepanova added a comment - I agree with philsweeney – the visible part of the stack trace looks similar to MDEV-10324 , and given that it only started happening recently, it's most likely the same problem. ivangrynenko , if you want to experiment, the fix has now been pushed into 10.1 tree. Otherwise, I'm afraid the best advice is to roll back to 10.1.14 until the next release of 10.1 is out. Sorry for the trouble.

            People

              Unassigned Unassigned
              ivangrynenko Ivan Grynenko
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.