Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10349

buffer overflow detected ***: /usr/sbin/mysqld terminated

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.1.15
    • Fix Version/s: 10.1.16
    • Component/s: Optimizer
    • Labels:
    • Environment:
      CentOS7 VPS in XenServer 7
      32GB RAM, 12CPU
      cPanel 56.25
      CloudLinux (form cloudlinux-testing repository)

      Description

      Quite frequently mariadb began killing itself on our production server. This happening between 1 to 12 hours for the last 2 days, but did not happen before. I have commented out the most of custom variables that were added in server.cnf to see if this helps. The latest crash report from mysql log is:

      Server version: 10.1.15-MariaDB
      key_buffer_size=67108864
      read_buffer_size=131072
      max_used_connections=11
      max_threads=202
      thread_count=6
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 509205 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7f1a9f7e3008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f1f3da42100 thread_stack 0x48400
      *** buffer overflow detected ***: /usr/sbin/mysqld terminated
      ======= Backtrace: =========
      /lib64/libc.so.6(__fortify_fail+0x37)[0x7f1f3bbae597]
      /lib64/libc.so.6(+0x10c750)[0x7f1f3bbac750]
      /lib64/libc.so.6(+0x10e507)[0x7f1f3bbae507]
      /usr/sbin/mysqld(my_addr_resolve+0x48)[0x7f1f3e64d388]
      /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7f1f3e639a92]
      /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7f1f3e16095d]
      /lib64/libpthread.so.0(+0xf100)[0x7f1f3d77e100]
      /usr/sbin/mysqld(+0x6a0ae5)[0x7f1f3e24fae5]
      /usr/sbin/mysqld(_ZN10SQL_SELECT17test_quick_selectEP3THD6BitmapILj64EEyybbb+0x1d14)[0x7f1f3e25d734]
      /usr/sbin/mysqld(+0x476a9e)[0x7f1f3e025a9e]
      /usr/sbin/mysqld(_ZN4JOIN14optimize_innerEv+0x2b70)[0x7f1f3e034ca0]
      /usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x1b)[0x7f1f3e03511b]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x8f)[0x7f1f3e03526f]
      /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x245)[0x7f1f3e035dc5]
      /usr/sbin/mysqld(+0x4295c1)[0x7f1f3dfd85c1]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5f8f)[0x7f1f3dfe41cf]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x28e)[0x7f1f3dfe7aae]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x25db)[0x7f1f3dfeaf6b]
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7f1f3dfeb7e9]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7f1f3e0af2aa]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f1f3e0af480]
      /usr/sbin/mysqld(+0x9684ed)[0x7f1f3e5174ed]
      /lib64/libpthread.so.0(+0x7dc5)[0x7f1f3d776dc5]
      /lib64/libc.so.6(clone+0x6d)[0x7f1f3bb96ced]
      ======= Memory map: ========
      7f1a80000000-7f1a94c00000 rw-p 00000000 00:00 0 
      7f1a94ff5000-7f1a94ff6000 ---p 00000000 00:00 0 
      7f1a94ff6000-7f1a957f6000 rw-p 00000000 00:00 0                          [stack:652357]
      7f1a957f6000-7f1a957f7000 ---p 00000000 00:00 0 
      7f1a957f7000-7f1a95ff7000 rw-p 00000000 00:00 0                          [stack:652356]
      7f1a95ff7000-7f1a95ff8000 ---p 00000000 00:00 0 
      7f1a95ff8000-7f1a967f8000 rw-p 00000000 00:00 0                          [stack:652355]
      7f1a967f8000-7f1a967f9000 ---p 00000000 00:00 0 
      7f1a967f9000-7f1a96ff9000 rw-p 00000000 00:00 0                          [stack:652354]
      7f1a96ff9000-7f1a96ffa000 ---p 00000000 00:00 0 
      7f1a96ffa000-7f1a977fa000 rw-p 00000000 00:00 0                          [stack:652353]
      7f1a977fa000-7f1a977fb000 ---p 00000000 00:00 0 
      7f1a977fb000-7f1a97ffb000 rw-p 00000000 00:00 0                          [stack:652351]
      7f1a97ffb000-7f1a97ffc000 ---p 00000000 00:00 0 
      7f1a97ffc000-7f1a987fc000 rw-p 00000000 00:00 0                          [stack:652350]
      7f1a987fc000-7f1a987fd000 ---p 00000000 00:00 0 
      7f1a987fd000-7f1a98ffd000 rw-p 00000000 00:00 0                          [stack:652349]
      7f1a98ffd000-7f1a98ffe000 ---p 00000000 00:00 0 
      7f1a98ffe000-7f1a997fe000 rw-p 00000000 00:00 0                          [stack:652348]
      7f1a997fe000-7f1a997ff000 ---p 00000000 00:00 0 
      7f1a997ff000-7f1a99fff000 rw-p 00000000 00:00 0                          [stack:652347]
      7f1a99fff000-7f1a9a000000 ---p 00000000 00:00 0 
      7f1a9a000000-7f1ab1c00000 rw-p 00000000 00:00 0                          [stack:652352]
      7f1ab1ff6000-7f1ab1ff7000 ---p 00000000 00:00 0 
      7f1ab1ff7000-7f1ab27f7000 rw-p 00000000 00:00 0                          [stack:652303]
      7f1ab27f7000-7f1ab27f8000 ---p 00000000 00:00 0 
      7f1ab27f8000-7f1ab2ff8000 rw-p 00000000 00:00 0                          [stack:652302]
      7f1ab2ff8000-7f1ab2ff9000 ---p 00000000 00:00 0 
      7f1ab2ff9000-7f1ab37f9000 rw-p 00000000 00:00 0                          [stack:652301]
      7f1ab37f9000-7f1ab37fa000 ---p 00000000 00:00 0 
      7f1ab37fa000-7f1ab3ffa000 rw-p 00000000 00:00 0                          [stack:652300]
      7f1ab3ffa000-7f1ab3ffb000 ---p 00000000 00:00 0 
      7f1ab3ffb000-7f1ab47fb000 rw-p 00000000 00:00 0                          [stack:652299]
      7f1ab47fb000-7f1ab47fc000 ---p 00000000 00:00 0 
      7f1ab47fc000-7f1ab4ffc000 rw-p 00000000 00:00 0                          [stack:652298]
      7f1ab4ffc000-7f1ab4ffd000 ---p 00000000 00:00 0 
      7f1ab4ffd000-7f1ab57fd000 rw-p 00000000 00:00 0                          [stack:652297]
      7f1ab57fd000-7f1ab57fe000 ---p 00000000 00:00 0 
      7f1ab57fe000-7f1ab5ffe000 rw-p 00000000 00:00 0                          [stack:652296]
      7f1ab5ffe000-7f1ab5fff000 ---p 00000000 00:00 0 
      7f1ab5fff000-7f1ab67ff000 rw-p 00000000 00:00 0                          [stack:652295]
      7f1ab67ff000-7f1ab6800000 ---p 00000000 00:00 0 
      7f1ab6800000-7f1acdc00000 rw-p 00000000 00:00 0                          [stack:652294]
      7f1acde00000-7f1b11400000 rw-p 00000000 00:00 0 
      7f1b11600000-7f1bdb400000 rw-p 00000000 00:00 0 
      7f1bdb600000-7f1ca5400000 rw-p 00000000 00:00 0 
      7f1ca5600000-7f1d6f400000 rw-p 00000000 00:00 0 
      7f1d6f600000-7f1e39400000 rw-p 00000000 00:00 0 
      7f1e39600000-7f1f03400000 rw-p 00000000 00:00 0 
      7f1f037d4000-7f1f037d5000 ---p 00000000 00:00 0 
      7f1f037d5000-7f1f03fd5000 rw-p 00000000 00:00 0                          [stack:652274]
      7f1f03fd5000-7f1f03fd6000 ---p 00000000 00:00 0 
      7f1f03fd6000-7f1f047d6000 rw-p 00000000 00:00 0                          [stack:652273]
      7f1f047d6000-7f1f047d7000 ---p 00000000 00:00 0 
      7f1f047d7000-7f1f04fd7000 rw-p 00000000 00:00 0                          [stack:652272]
      7f1f04fd7000-7f1f04fd8000 ---p 00000000 00:00 0 
      7f1f04fd8000-7f1f057d8000 rw-p 00000000 00:00 0                          [stack:652271]
      7f1f057d8000-7f1f057d9000 ---p 00000000 00:00 0 
      7f1f057d9000-7f1f05fd9000 rw-p 00000000 00:00 0                          [stack:652270]
      7f1f05fd9000-7f1f05fda000 ---p 00000000 00:00 0 
      7f1f05fda000-7f1f067da000 rw-p 00000000 00:00 0                          [stack:652269]
      7f1f067da000-7f1f067db000 ---p 00000000 00:00 0 
      7f1f067db000-7f1f06fdb000 rw-p 00000000 00:00 0                          [stack:652268]
      7f1f06fdb000-7f1f06fdc000 ---p 00000000 00:00 0 
      7f1f06fdc000-7f1f077dc000 rw-p 00000000 00:00 0                          [stack:652267]
      7f1f077dc000-7f1f077dd000 ---p 00000000 00:00 0 
      7f1f077dd000-7f1f07fdd000 rw-p 00000000 00:00 0                          [stack:652266]
      7f1f07fdd000-7f1f07fde000 ---p 00000000 00:00 0 
      7f1f07fde000-7f1f087de000 rw-p 00000000 00:00 0                          [stack:652265]
      7f1f087de000-7f1f087df000 ---p 00000000 00:00 0 
      7f1f087df000-7f1f08fdf000 rw-p 00000000 00:00 0                          [stack:652264]
      7f1f08fdf000-7f1f08fe0000 ---p 00000000 00:00 0 
      7f1f08fe0000-7f1f097e0000 rw-p 00000000 00:00 0                          [stack:652263]
      7f1f097e0000-7f1f097e1000 ---p 00000000 00:00 0 
      7f1f097e1000-7f1f09fe1000 rw-p 00000000 00:00 0                          [stack:652262]
      7f1f09fe1000-7f1f09fe2000 ---p 00000000 00:00 0 
      7f1f09fe2000-7f1f0a7e2000 rw-p 00000000 00:00 0                          [stack:652261]
      7f1f0a7e2000-7f1f0a7e3000 ---p 00000000 00:00 0 
      7f1f0a7e3000-7f1f0afe3000 rw-p 00000000 00:00 0                          [stack:652260]
      7f1f0afe3000-7f1f0afe4000 ---p 00000000 00:00 0 
      7f1f0afe4000-7f1f0b7e4000 rw-p 00000000 00:00 0                          [stack:652259]
      7f1f0b7e4000-7f1f0b7e5000 ---p 00000000 00:00 0 
      7f1f0b7e5000-7f1f0bfe5000 rw-p 00000000 00:00 0                          [stack:652258]
      7f1f0bfe5000-7f1f0bfe6000 ---p 00000000 00:00 0 
      7f1f0bfe6000-7f1f0c7e6000 rw-p 00000000 00:00 0                          [stack:652257]
      7f1f0c7e6000-7f1f0c7e7000 ---p 00000000 00:00 0 
      7f1f0c7e7000-7f1f0cfe7000 rw-p 00000000 00:00 0                          [stack:652256]
      7f1f0cfe7000-7f1f0cfe8000 ---p 00000000 00:00 0 
      7f1f0cfe8000-7f1f0d7e8000 rw-p 00000000 00:00 0                          [stack:652255]
      7f1f0d7e8000-7f1f0d7e9000 ---p 00000000 00:00 0 
      7f1f0d7e9000-7f1f0dfe9000 rw-p 00000000 00:00 0                          [stack:652254]
      7f1f0dfe9000-7f1f0dfea000 ---p 00000000 00:00 0 
      7f1f0dfea000-7f1f0e7ea000 rw-p 00000000 00:00 0                          [stack:652253]
      7f1f0e7ea000-7f1f0e7eb000 ---p 00000000 00:00 0 
      7f1f0e7eb000-7f1f0efeb000 rw-p 00000000 00:00 0                          [stack:652252]
      7f1f0efeb000-7f1f0efec000 ---p 00000000 00:00 0 
      7f1f0efec000-7f1f0f7ec000 rw-p 00000000 00:00 0                          [stack:652251]
      7f1f0f7ec000-7f1f0f7ed000 ---p 00000000 00:00 0 
      7f1f0f7ed000-7f1f0ffed000 rw-p 00000000 00:00 0                          [stack:652250]
      7f1f0ffed000-7f1f0ffee000 ---p 00000000 00:00 0 
      7f1f0ffee000-7f1f107ee000 rw-p 00000000 00:00 0                          [stack:652249]
      7f1f107ee000-7f1f107ef000 ---p 00000000 00:00 0 
      7f1f107ef000-7f1f10fef000 rw-p 00000000 00:00 0                          [stack:652248]
      7f1f10fef000-7f1f10ff0000 ---p 00000000 00:00 0 
      7f1f10ff0000-7f1f117f0000 rw-p 00000000 00:00 0                          [stack:652247]
      7f1f117f0000-7f1f117f1000 ---p 00000000 00:00 0 
      7f1f117f1000-7f1f11ff1000 rw-p 00000000 00:00 0                          [stack:652246]
      7f1f11ff1000-7f1f11ff2000 ---p 00000000 00:00 0 
      7f1f11ff2000-7f1f127f2000 rw-p 00000000 00:00 0                          [stack:652245]
      7f1f127f2000-7f1f127f3000 ---p 00000000 00:00 0 
      7f1f127f3000-7f1f12ff3000 rw-p 00000000 00:00 0                          [stack:652244]
      7f1f12ff3000-7f1f12ff4000 ---p 00000000 00:00 0 
      7f1f12ff4000-7f1f137f4000 rw-p 00000000 00:00 0                          [stack:652243]
      7f1f137f4000-7f1f137f5000 ---p 00000000 00:00 0 
      7f1f137f5000-7f1f13ff5000 rw-p 00000000 00:00 0                          [stack:652242]
      7f1f13ff5000-7f1f13ff6000 ---p 00000000 00:00 0 
      7f1f13ff6000-7f1f147f6000 rw-p 00000000 00:00 0                          [stack:652241]
      7f1f147f6000-7f1f147f7000 ---p 00000000 00:00 0 
      7f1f147f7000-7f1f14ff7000 rw-p 00000000 00:00 0                          [stack:652240]
      7f1f14ff7000-7f1f14ff8000 ---p 00000000 00:00 0 
      7f1f14ff8000-7f1f157f8000 rw-p 00000000 00:00 0                          [stack:652239]
      7f1f157f8000-7f1f157f9000 ---p 00000000 00:00 0 
      7f1f157f9000-7f1f15ff9000 rw-p 00000000 00:00 0                          [stack:652238]
      7f1f15ff9000-7f1f15ffa000 ---p 00000000 00:00 0 
      7f1f15ffa000-7f1f167fa000 rw-p 00000000 00:00 0                          [stack:652237]
      7f1f167fa000-7f1f167fb000 ---p 00000000 00:00 0 
      7f1f167fb000-7f1f16ffb000 rw-p 00000000 00:00 0                          [stack:652236]
      7f1f16ffb000-7f1f16ffc000 ---p 00000000 00:00 0 
      7f1f16ffc000-7f1f177fc000 rw-p 00000000 00:00 0                          [stack:652235]
      7f1f177fc000-7f1f177fd000 ---p 00000000 00:00 0 
      7f1f177fd000-7f1f17ffd000 rw-p 00000000 00:00 0                          [stack:652234]
      7f1f17ffd000-7f1f17ffe000 ---p 00000000 00:00 0 
      7f1f17ffe000-7f1f187fe000 rw-p 00000000 00:00 0                          [stack:652233]
      7f1f187fe000-7f1f187ff000 ---p 00000000 00:00 0 
      7f1f187ff000-7f1f18fff000 rw-p 00000000 00:00 0                          [stack:652232]
      7f1f18fff000-7f1f19000000 ---p 00000000 00:00 0 
      7f1f19000000-7f1f20000000 rw-p 00000000 00:00 0                          [stack:652231]
      7f1f203ff000-7f1f20400000 ---p 00000000 00:00 0 
      7f1f20400000-7f1f21000000 rw-p 00000000 00:00 0                          [stack:652230]
      7f1f213ff000-7f1f21400000 ---p 00000000 00:00 0 
      7f1f21400000-7f1f2f000000 rw-p 00000000 00:00 0                          [stack:652229]
      7f1f2f006000-7f1f2f007000 ---p 00000000 00:00 0 
      7f1f2f007000-7f1f2f050000 rw-p 00000000 00:00 0                          [stack:863330]
      7f1f2f050000-7f1f2f1ee000 r-xp 00000000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f1ee000-7f1f2f3ee000 ---p 0019e000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3ee000-7f1f2f3f3000 r--p 0019e000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3f3000-7f1f2f3f6000 rw-p 001a3000 ca:02 137979136                  /usr/lib64/mysql/plugin/ha_tokudb.so
      7f1f2f3f6000-7f1f2f3ff000 rw-p 00000000 00:00 0 
      7f1f2f3ff000-7f1f2f400000 ---p 00000000 00:00 0 
      7f1f2f400000-7f1f39400000 rw-p 00000000 00:00 0                          [stack:652228]
      7f1f39432000-7f1f39433000 ---p 00000000 00:00 0 
      7f1f39433000-7f1f3947c000 rw-p 00000000 00:00 0                          [stack:863329]
      7f1f3947c000-7f1f3947d000 ---p 00000000 00:00 0 
      7f1f3947d000-7f1f394c6000 rw-p 00000000 00:00 0                          [stack:655783]
      7f1f394c6000-7f1f394c7000 ---p 00000000 00:00 0 
      7f1f394c7000-7f1f39510000 rw-p 00000000 00:00 0                          [stack:655279]
      7f1f39510000-7f1f39511000 ---p 00000000 00:00 0 
      7f1f39511000-7f1f3955a000 rw-p 00000000 00:00 0                          [stack:655278]
      7f1f3955a000-7f1f3955b000 ---p 00000000 00:00 0 
      7f1f3955b000-7f1f395a4000 rw-p 00000000 00:00 0                          [stack:652782]
      7f1f395a4000-7f1f395a5000 ---p 00000000 00:00 0 
      7f1f395a5000-7f1f395ee000 rw-p 00000000 00:00 0                          [stack:865914]
      7f1f395ee000-7f1f3964e000 r-xp 00000000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3964e000-7f1f3984d000 ---p 00060000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984d000-7f1f3984e000 r--p 0005f000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984e000-7f1f3984f000 rw-p 00060000 ca:02 798994                     /usr/lib64/libpcre.so.1.2.0
      7f1f3984f000-7f1f39870000 r-xp 00000000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39870000-7f1f39a70000 ---p 00021000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a70000-7f1f39a71000 r--p 00021000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a71000-7f1f39a72000 rw-p 00022000 ca:02 1991                       /usr/lib64/libselinux.so.1
      7f1f39a72000-7f1f39a74000 rw-p 00000000 00:00 0 
      7f1f39a74000-7f1f39a83000 r-xp 00000000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39a83000-7f1f39c82000 ---p 0000f000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c82000-7f1f39c83000 r--p 0000e000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c83000-7f1f39c84000 rw-p 0000f000 ca:02 2034                       /usr/lib64/libbz2.so.1.0.6
      7f1f39c84000-7f1f39c99000 r-xp 00000000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39c99000-7f1f39e98000 ---p 00015000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e98000-7f1f39e99000 r--p 00014000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e99000-7f1f39e9a000 rw-p 00015000 ca:02 1967248                    /usr/lib64/libelf-0.163.so
      7f1f39e9a000-7f1f39e9e000 r-xp 00000000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f39e9e000-7f1f3a09d000 ---p 00004000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09d000-7f1f3a09e000 r--p 00003000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09e000-7f1f3a09f000 rw-p 00004000 ca:02 2100                       /usr/lib64/libattr.so.1.1.0
      7f1f3a09f000-7f1f3a0b5000 r-xp 00000000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a0b5000-7f1f3a2b5000 ---p 00016000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b5000-7f1f3a2b6000 r--p 00016000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b6000-7f1f3a2b7000 rw-p 00017000 ca:02 1704189                    /usr/lib64/libresolv-2.17.so
      7f1f3a2b7000-7f1f3a2b9000 rw-p 00000000 00:00 0 
      7f1f3a2b9000-7f1f3a2bc000 r-xp 00000000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a2bc000-7f1f3a4bb000 ---p 00003000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bb000-7f1f3a4bc000 r--p 00002000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bc000-7f1f3a4bd000 rw-p 00003000 ca:02 34544                      /usr/lib64/libkeyutils.so.1.5
      7f1f3a4bd000-7f1f3a4ca000 r-xp 00000000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a4ca000-7f1f3a6ca000 ---p 0000d000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6ca000-7f1f3a6cb000 r--p 0000d000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6cb000-7f1f3a6cc000 rw-p 0000e000 ca:02 1550036                    /usr/lib64/libkrb5support.so.0.1
      7f1f3a6cc000-7f1f3a6e1000 r-xp 00000000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a6e1000-7f1f3a8e0000 ---p 00015000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e0000-7f1f3a8e1000 r--p 00014000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e1000-7f1f3a8e2000 rw-p 00015000 ca:02 1969368                    /usr/lib64/libgcc_s-4.8.5-20150702.so.1
      7f1f3a8e2000-7f1f3a927000 r-xp 00000000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3a927000-7f1f3ab26000 ---p 00045000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab26000-7f1f3ab28000 r--p 00044000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab28000-7f1f3ab29000 rw-p 00046000 ca:02 193538                     /usr/lib64/libdw-0.163.so
      7f1f3ab29000-7f1f3ab2d000 r-xp 00000000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ab2d000-7f1f3ad2c000 ---p 00004000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2c000-7f1f3ad2d000 r--p 00003000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2d000-7f1f3ad2e000 rw-p 00004000 ca:02 2105                       /usr/lib64/libcap.so.2.22
      7f1f3ad2e000-7f1f3ad35000 r-xp 00000000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3ad35000-7f1f3af34000 ---p 00007000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af34000-7f1f3af35000 r--p 00006000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af35000-7f1f3af36000 rw-p 00007000 ca:02 1704191                    /usr/lib64/librt-2.17.so
      7f1f3af36000-7f1f3af65000 r-xp 00000000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3af65000-7f1f3b164000 ---p 0002f000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b164000-7f1f3b166000 r--p 0002e000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b166000-7f1f3b167000 rw-p 00030000 ca:02 1947                       /usr/lib64/libk5crypto.so.3.1
      7f1f3b167000-7f1f3b168000 rw-p 00000000 00:00 0 
      7f1f3b168000-7f1f3b16b000 r-xp 00000000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b16b000-7f1f3b36a000 ---p 00003000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36a000-7f1f3b36b000 r--p 00002000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36b000-7f1f3b36c000 rw-p 00003000 ca:02 1871                       /usr/lib64/libcom_err.so.2.1
      7f1f3b36c000-7f1f3b441000 r-xp 00000000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b441000-7f1f3b641000 ---p 000d5000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b641000-7f1f3b64e000 r--p 000d5000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b64e000-7f1f3b651000 rw-p 000e2000 ca:02 2037                       /usr/lib64/libkrb5.so.3.3
      7f1f3b651000-7f1f3b69a000 r-xp 00000000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b69a000-7f1f3b89a000 ---p 00049000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89a000-7f1f3b89b000 r--p 00049000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89b000-7f1f3b89d000 rw-p 0004a000 ca:02 86405                      /usr/lib64/libgssapi_krb5.so.2.2
      7f1f3b89d000-7f1f3b89f000 r-xp 00000000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3b89f000-7f1f3ba9e000 ---p 00002000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3ba9e000-7f1f3ba9f000 r--p 00001000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3ba9f000-7f1f3baa0000 rw-p 00002000 ca:02 1018064                    /usr/lib64/libfreebl3.so
      7f1f3baa0000-7f1f3bc57000 r-xp 00000000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3bc57000-7f1f3be57000 ---p 001b7000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be57000-7f1f3be5b000 r--p 001b7000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be5b000-7f1f3be5d000 rw-p 001bb000 ca:02 1412                       /usr/lib64/libc-2.17.so
      7f1f3be5d000-7f1f3be62000 rw-p 00000000 00:00 0 
      7f1f3be62000-7f1f3bf63000 r-xp 00000000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3bf63000-7f1f3c162000 ---p 00101000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c162000-7f1f3c163000 r--p 00100000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c163000-7f1f3c164000 rw-p 00101000 ca:02 1679900                    /usr/lib64/libm-2.17.so
      7f1f3c164000-7f1f3c24d000 r-xp 00000000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c24d000-7f1f3c44d000 ---p 000e9000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c44d000-7f1f3c456000 r--p 000e9000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c456000-7f1f3c458000 rw-p 000f2000 ca:02 34355                      /usr/lib64/libstdc++.so.6.0.19
      7f1f3c458000-7f1f3c46d000 rw-p 00000000 00:00 0 
      7f1f3c46d000-7f1f3c470000 r-xp 00000000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c470000-7f1f3c66f000 ---p 00003000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c66f000-7f1f3c670000 r--p 00002000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c670000-7f1f3c671000 rw-p 00003000 ca:02 1679898                    /usr/lib64/libdl-2.17.so
      7f1f3c671000-7f1f3c82f000 r-xp 00000000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3c82f000-7f1f3ca2f000 ---p 001be000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca2f000-7f1f3ca49000 r--p 001be000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca49000-7f1f3ca55000 rw-p 001d8000 ca:02 722013                     /usr/lib64/libcrypto.so.1.0.1e
      7f1f3ca55000-7f1f3ca59000 rw-p 00000000 00:00 0 
      7f1f3ca59000-7f1f3cabc000 r-xp 00000000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3cabc000-7f1f3ccbb000 ---p 00063000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccbb000-7f1f3ccbf000 r--p 00062000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccbf000-7f1f3ccc6000 rw-p 00066000 ca:02 1534958                    /usr/lib64/libssl.so.1.0.1e
      7f1f3ccc6000-7f1f3ccce000 r-xp 00000000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3ccce000-7f1f3cecd000 ---p 00008000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cecd000-7f1f3cece000 r--p 00007000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cece000-7f1f3cecf000 rw-p 00008000 ca:02 1417                       /usr/lib64/libcrypt-2.17.so
      7f1f3cecf000-7f1f3cefd000 rw-p 00000000 00:00 0 
      7f1f3cefd000-7f1f3cf2f000 r-xp 00000000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3cf2f000-7f1f3d12e000 ---p 00032000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d12e000-7f1f3d130000 r--p 00031000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d130000-7f1f3d131000 rw-p 00033000 ca:02 1192502                    /usr/lib64/libjemalloc.so.1
      7f1f3d131000-7f1f3d132000 rw-p 00000000 00:00 0 
      7f1f3d132000-7f1f3d147000 r-xp 00000000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d147000-7f1f3d346000 ---p 00015000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d346000-7f1f3d347000 r--p 00014000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d347000-7f1f3d348000 rw-p 00015000 ca:02 1993                       /usr/lib64/libz.so.1.2.7
      7f1f3d348000-7f1f3d349000 r-xp 00000000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d349000-7f1f3d548000 ---p 00001000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d548000-7f1f3d549000 r--p 00000000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d549000-7f1f3d54a000 rw-p 00001000 ca:02 158268                     /usr/lib64/libaio.so.1.0.1
      7f1f3d54a000-7f1f3d56e000 r-xp 00000000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d56e000-7f1f3d76d000 ---p 00024000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76d000-7f1f3d76e000 r--p 00023000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76e000-7f1f3d76f000 rw-p 00024000 ca:02 3038                       /usr/lib64/liblzma.so.5.0.99
      7f1f3d76f000-7f1f3d785000 r-xp 00000000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d785000-7f1f3d985000 ---p 00016000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d985000-7f1f3d986000 r--p 00016000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d986000-7f1f3d987000 rw-p 00017000 ca:02 1824                       /usr/lib64/libpthread-2.17.so
      7f1f3d987000-7f1f3d98b000 rw-p 00000000 00:00 0 
      7f1f3d98b000-7f1f3d9ac000 r-xp 00000000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3d9b0000-7f1f3d9b1000 ---p 00000000 00:00 0 
      7f1f3d9b1000-7f1f3d9fa000 rw-p 00000000 00:00 0                          [stack:652511]
      7f1f3d9fa000-7f1f3d9fb000 ---p 00000000 00:00 0 
      7f1f3d9fb000-7f1f3da44000 rw-p 00000000 00:00 0                          [stack:181434]
      7f1f3da44000-7f1f3da45000 ---p 00000000 00:00 0 
      7f1f3da45000-7f1f3da8e000 rw-p 00000000 00:00 0                          [stack:652425]
      7f1f3da8e000-7f1f3da8f000 ---p 00000000 00:00 0 
      7f1f3da8f000-7f1f3dad8000 rw-p 00000000 00:00 0                          [stack:652402]
      7f1f3dad8000-7f1f3dad9000 ---p 00000000 00:00 0 
      7f1f3dad9000-7f1f3db22000 rw-p 00000000 00:00 0                          [stack:652358]
      7f1f3db22000-7f1f3db57000 r--s 00000000 ca:21 537306668                  /var/db/nscd/services
      7f1f3db57000-7f1f3db5d000 rw-s 00000000 ca:21 805384595                  /var/lib/mysql/tc.log
      7f1f3db5d000-7f1f3db62000 rw-s 00000000 00:0a 383893695                  /[aio] (deleted)
      7f1f3db62000-7f1f3db67000 rw-s 00000000 00:0a 383893694                  /[aio] (deleted)
      7f1f3db67000-7f1f3db6c000 rw-s 00000000 00:0a 383893693                  /[aio] (deleted)
      7f1f3db6c000-7f1f3db71000 rw-s 00000000 00:0a 383893692                  /[aio] (deleted)
      7f1f3db71000-7f1f3db76000 rw-s 00000000 00:0a 383893691                  /[aio] (deleted)
      7f1f3db76000-7f1f3db7b000 rw-s 00000000 00:0a 383893690                  /[aio] (deleted)
      7f1f3db7b000-7f1f3db80000 rw-s 00000000 00:0a 383893689                  /[aio] (deleted)
      7f1f3db80000-7f1f3db85000 rw-s 00000000 00:0a 383893688                  /[aio] (deleted)
      7f1f3db85000-7f1f3db93000 rw-p 00000000 00:00 0 
      7f1f3db93000-7f1f3db97000 r-xp 00000000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db97000-7f1f3db98000 ---p 00004000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db98000-7f1f3db99000 r--p 00004000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db99000-7f1f3db9a000 rw-p 00005000 ca:02 693482                     /usr/lib64/libsystemd-daemon.so.0.0.12
      7f1f3db9a000-7f1f3db9e000 rw-p 00000000 00:00 0 
      7f1f3db9e000-7f1f3dba0000 rw-s 00000000 00:0a 383893696                  /[aio] (deleted)
      7f1f3dba0000-7f1f3dba5000 rw-s 00000000 00:0a 383893687                  /[aio] (deleted)
      7f1f3dba5000-7f1f3dbaa000 rw-s 00000000 00:0a 383893686                  /[aio] (deleted)
      7f1f3dbaa000-7f1f3dbab000 rw-s 00000000 00:0a 383893685                  /[aio] (deleted)
      7f1f3dbab000-7f1f3dbac000 rw-p 00000000 00:00 0 
      7f1f3dbac000-7f1f3dbad000 r--p 00021000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3dbad000-7f1f3dbae000 rw-p 00022000 ca:02 1679856                    /usr/lib64/ld-2.17.so
      7f1f3dbae000-7f1f3dbaf000 rw-p 00000000 00:00 0 
      7f1f3dbaf000-7f1f3eb26000 r-xp 00000000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3ed26000-7f1f3edd6000 r--p 00f77000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3edd6000-7f1f3ee87000 rw-p 01027000 ca:02 797471                     /usr/sbin/mysqld
      7f1f3ee87000-7f1f3f6fd000 rw-p 00000000 00:00 0 
      7ffe6a7f8000-7ffe6a819000 rw-p 00000000 00:00 0                          [stack]
      7ffe6a907000-7ffe6a909000 r-xp 00000000 00:00 0                          [vdso]
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      Fatal signal 6 while backtracing
      Sat Jul  9 09:43:01 2016 PerconaFT recovery starting in env /var/lib/mysql/
      Sat Jul  9 09:43:01 2016 PerconaFT recovery scanning backward from 562001
      Sat Jul  9 09:43:01 2016 PerconaFT recovery bw_end_checkpoint at 562001 timestamp 1468021178926184 xid 561997 (bw_newer)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery bw_begin_checkpoint at 561997 timestamp 1468021178926159 (bw_between)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery turning around at begin checkpoint 561997 time 25
      Sat Jul  9 09:43:01 2016 PerconaFT recovery starts scanning forward to 562001 from 561997 left 4 (fw_between)
      Sat Jul  9 09:43:01 2016 PerconaFT recovery closing 2 dictionaries
      Sat Jul  9 09:43:01 2016 PerconaFT recovery making a checkpoint
      Sat Jul  9 09:43:01 2016 PerconaFT recovery done
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: The InnoDB memory heap is disabled
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Memory barrier is not used
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using Linux native AIO
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Using SSE crc32 instructions
      2016-07-09  9:43:01 139894138476672 [Note] InnoDB: Initializing buffer pool, size = 16.0G
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Completed initialization of buffer pool
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Highest supported file format is Barracuda.
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Log scan progressed past the checkpoint lsn 171599114149
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Database was not shutdown normally!
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Starting crash recovery.
      2016-07-09  9:43:02 139894138476672 [Note] InnoDB: Reading tablespace information from the .ibd files...
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: Restoring possible half-written data pages 
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: from the doublewrite buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 171599131045
      2016-07-09  9:43:05 139894138476672 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB: 128 rollback segment(s) are active.
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB: Waiting for purge to start
      2016-07-09  9:43:06 139894138476672 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.30-76.3 started; log sequence number 171599131045
      2016-07-09  9:43:06 139874141181696 [Note] InnoDB: Dumping buffer pool(s) not yet started
      2016-07-09  9:43:06 139894138476672 [Note] Plugin 'FEEDBACK' is disabled.
      2016-07-09  9:43:06 139894138476672 [Note] Recovering after a crash using tc.log
      2016-07-09  9:43:06 139894138476672 [Note] Starting crash recovery...
      2016-07-09  9:43:06 139894138476672 [Note] Crash recovery finished.
      2016-07-09  9:43:06 139894138476672 [Note] Server socket created on IP: '127.0.0.1'.
      2016-07-09  9:43:06 139894138476672 [Warning] 'user' entry 'root@cpanel.redy.host' ignored in --skip-name-resolve mode.
      

      The entire [mariadb] block was active upon latest crash:

      [mariadb]
      max_connections=200
      skip-external-locking
      skip-innodb_doublewrite
      key_buffer_size = 64M
      max_allowed_packet = 128M
      table_open_cache = 64
      thread_cache_size = 8
      query_cache_limit = 16M
      tmp_table_size = 128M
      max_heap_table_size = 128M
      join_buffer_size = 16M
      default-storage-engine = innodb
      innodb_log_file_size = 2G
      innodb_flush_method = O_DIRECT
      innodb_file_per_table = 1
      innodb_open_files = 6000
      innodb_flush_log_at_trx_commit = 2
      innodb_log_buffer_size = 16M
      innodb_additional_mem_pool_size = 8M
      innodb_buffer_pool_size = 16G
      innodb_thread_concurrency = 0
      innodb_buffer_pool_instances=16
      query_cache_type = 1
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ivangrynenko Ivan Grynenko
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: