Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-14075

buffer overflow detected with unlink_thd function

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.1.21, 10.1.26
    • None
    • None
    • None

    Description

      We have MariaDB instance with very often 'signal 11' crashes. Most of them happen with reference to unlink_thd() function, which is not present anymore in MySQL 5.6+, so I assume this may be specific to MariaDB.

      171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
      Version: '10.1.26-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
      

      It is not clear what exactly queries/transactions cause these crashes.

      Example crash with related thread gdb traces (no full gdb attached due to sensitive information) below. Additional, slightly different occurences in attachment.

      171010 12:28:49 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.1.26-MariaDB
      key_buffer_size=1073741824
      read_buffer_size=131072
      max_used_connections=1058
      max_threads=2002
      thread_count=457
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5446182 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7f282328f008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f2783c90080 thread_stack 0x48400
      *** 	 ***: /usr/sbin/mysqld terminated
      ======= Backtrace: =========
      /lib64/libc.so.6(__fortify_fail+0x37)[0x7f5e7fc7cd87]
      /lib64/libc.so.6(+0x10df40)[0x7f5e7fc7af40]
      /lib64/libc.so.6(+0x10fcf7)[0x7f5e7fc7ccf7]
      /usr/sbin/mysqld(my_addr_resolve+0x48)[0x55f45696b088]
      /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x55f4569576c2]
      /usr/sbin/mysqld(handle_fatal_signal+0x305)[0x55f45647aed5]
      /lib64/libpthread.so.0(+0xf5e0)[0x7f5e818c95e0]
      /usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xab)[0x55f45647da1b]
      /usr/sbin/mysqld(_Z14trans_rollbackP3THD+0x5a)[0x55f4563d6a9a]
      /usr/sbin/mysqld(_ZN3THD7cleanupEv+0xe2)[0x55f4562c51b2]
      /usr/sbin/mysqld(_Z10unlink_thdP3THD+0x17)[0x55f456257ad7]
      /usr/sbin/mysqld(_Z29one_thread_per_connection_endP3THDb+0x20)[0x55f456257c90]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x113)[0x55f4563c8963]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x55f4563c8b80]
      /usr/sbin/mysqld(+0x74be8d)[0x55f456607e8d]
      /lib64/libpthread.so.0(+0x7e25)[0x7f5e818c1e25]
      /lib64/libc.so.6(clone+0x6d)[0x7f5e7fc6534d]
      ======= Memory map: ========
      55f455ebc000-55f456e46000 r-xp 00000000 fb:00 134770637                  /usr/sbin/mysqld
      55f457046000-55f4570f7000 r--p 00f8a000 fb:00 134770637                  /usr/sbin/mysqld
      55f4570f7000-55f4571a9000 rw-p 0103b000 fb:00 134770637                  /usr/sbin/mysqld
      (...)
      ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
      Fatal signal 6 while backtracing
      2017-10-10 12:32:44 139933296474368 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using mutexes to ref count buffer pool pages
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: The InnoDB memory heap is disabled
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Compressed tables use zlib 1.2.7
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using Linux native AIO
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Using SSE crc32 instructions
      2017-10-10 12:32:44 139933296474368 [Note] InnoDB: Initializing buffer pool, size = 200.0G
      2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Completed initialization of buffer pool
      2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Highest supported file format is Barracuda.
      2017-10-10 12:32:51 139933296474368 [Note] InnoDB: Starting crash recovery from checkpoint LSN=49954283878443
      2017-10-10 12:32:52 139933296474368 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
      InnoDB: 4 transaction(s) which must be rolled back or cleaned up
      InnoDB: in total 4 row operations to undo
      InnoDB: Trx id counter is 36935782400
      2017-10-10 12:33:04 139933296474368 [Note] InnoDB: Starting final batch to recover 127478 pages from redo log
      2017-10-10 12:33:06 139701654628096 [Note] InnoDB: To recover: 92197 pages from log
      InnoDB: Last MySQL binlog file position 0 575403140, file name /home/mysql/logs/mysql.002397
      2017-10-10 12:33:11 139933296474368 [Note] InnoDB: 128 rollback segment(s) are active.
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Starting in background the rollback of recovered transactions
      2017-10-10 12:33:11 7f0e04ffe700  InnoDB: Rolling back trx with id 36935781973, 1 rows to undo
      2017-10-10 12:33:11 139933296474368 [Note] InnoDB: Waiting for purge to start
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781973 completed
      2017-10-10 12:33:11 7f0e04ffe700  InnoDB: Rolling back trx with id 36935781972, 1 rows to undo
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781972 completed
      2017-10-10 12:33:11 7f0e04ffe700  InnoDB: Rolling back trx with id 36935781970, 1 rows to undo
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781970 completed
      2017-10-10 12:33:11 7f0e04ffe700  InnoDB: Rolling back trx with id 36935781952, 1 rows to undo
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of trx with id 36935781952 completed
      2017-10-10 12:33:11 139698190149376 [Note] InnoDB: Rollback of non-prepared transactions completed
      2017-10-10 12:33:11 139933296474368 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.1 started; log sequence number 49954847609095
      2017-10-10 12:33:21 139933296474368 [Note] Plugin 'FEEDBACK' is disabled.
      171010 12:33:21 server_audit: MariaDB Audit Plugin version 1.4.1 STARTED.
      2017-10-10 12:33:21 139697242212096 [Note] InnoDB: Dumping buffer pool(s) not yet started
      2017-10-10 12:33:21 139933296474368 [Note] Recovering after a crash using /home/mysql/logs/mysql
      2017-10-10 12:33:22 139933296474368 [Note] Starting crash recovery...
      2017-10-10 12:33:22 139933296474368 [Note] Crash recovery finished.
      2017-10-10 12:33:22 139933296474368 [Note] Server socket created on IP: '::'.
      2017-10-10 12:33:22 139933296474368 [Note] /usr/sbin/mysqld: ready for connections.
      Version: '10.1.26-MariaDB'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MariaDB Server
      

      Related thread gdb standard:

      Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
      +bt
      #0  0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
      #1  0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
      #2  0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
      #3  0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
      #4  0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
      #5  0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
      #6  0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
      #7  0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
      #8  my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
      #9  0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
      #10 <signal handler called>
      #11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
      #12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
      #13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
      #14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
      #15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
      #16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
      #17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
      #18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
      #19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
      #20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
      #21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
      +thread apply all bt
      

      gdb full:

      Thread 1 (Thread 0x7f2783c90b00 (LWP 22144)):
      +bt full
      #0  0x00007f5e7fba21f7 in raise () from /lib64/libc.so.6
      No symbol table info available.
      #1  0x00007f5e7fba38e8 in abort () from /lib64/libc.so.6
      No symbol table info available.
      #2  0x00007f5e7fbe1f47 in __libc_message () from /lib64/libc.so.6
      No symbol table info available.
      #3  0x00007f5e7fc7cd87 in __fortify_fail () from /lib64/libc.so.6
      No symbol table info available.
      #4  0x00007f5e7fc7af40 in __chk_fail () from /lib64/libc.so.6
      No symbol table info available.
      #5  0x00007f5e7fc7ccf7 in __fdelt_warn () from /lib64/libc.so.6
      No symbol table info available.
      #6  0x000055f45696b088 in my_addr_resolve (ptr=0x55f45695752e <my_print_stacktrace+46>, loc=loc@entry=0x7f2783c8f3f0) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/my_addr_resolve.c:162
              __d = <optimized out>
              input =           "\270\316\362\177^\177\000\000\000\000\000\000\000\000\000\000\270\316\362\177^\177\000\000\062\302\302\177^\177\000"
              len = <optimized out>
              total_bytes_read = 0
              extra_bytes_read = 0
              set = {
                fds_bits =             {[0] = 0 <repeats 16 times>}
              }
              timeout = {
                tv_sec = 0, 
                tv_usec = 0
              }
              filename_start = -1
              line_number_start = -1
              i = <optimized out>
      #7  0x000055f4569576c2 in print_with_addr_resolve (n=<optimized out>, addrs=0x7f2783c8f410) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:253
              loc = {
                file = 0x7f2800000000 "\300\317\333|^\177", 
                func = 0x675f61d6be26ab00 <Address 0x675f61d6be26ab00 out of bounds>, 
                line = 1461355122
              }
              i = <optimized out>
              err = <optimized out>
      #8  my_print_stacktrace (stack_bottom=<optimized out>, thread_stack=295936) at /home/buildbot/buildbot/build/mariadb-10.1.26/mysys/stacktrace.c:271
              addrs =           {[0] = 0x55f45695752e <my_print_stacktrace+46>,
                [1] = 0x55f45647aed5 <handle_fatal_signal(int)+773>,
                [2] = 0x7f5e818c95e0 <__restore_rt>,
                [3] = 0x55f45647da1b <ha_rollback_trans(THD*, bool)+171>,
                [4] = 0x55f4563d6a9a <trans_rollback(THD*)+90>,
                [5] = 0x55f4562c51b2 <THD::cleanup()+226>,
                [6] = 0x55f456257ad7 <unlink_thd(THD*)+23>,
                [7] = 0x55f456257c90 <one_thread_per_connection_end(THD*, bool)+32>,
                [8] = 0x55f4563c8963 <do_handle_one_connection(THD*)+275>,
                [9] = 0x55f4563c8b80 <handle_one_connection(void*)+64>,
                [10] = 0x55f456607e8d <pfs_spawn_thread(void*)+365>,
                [11] = 0x7f5e818c1e25 <start_thread+197>,
                [12] = 0x7f5e7fc6534d <clone+109>,
                [13] = 0x0,
                [14] = 0x7f284705c0c2,
                [15] = 0x7f284705c020,
                [16] = 0x101000000a2,
                [17] = 0x7f284705c128,
                [18] = 0x7f284705c1ca,
                [19] = 0x7f284705c1ca,
                [20] = 0x7f284705c1ca,
                [21] = 0x7f284705c1ca,
                [22] = 0x0,
                [23] = 0x112fd57b3f,
                [24] = 0x0,
                [25] = 0x55f40000000c,
                [26] = 0x0,
                [27] = 0x100007f,
                [28] = 0x7f5e00000000,
                [29] = 0x7f284705c1c5,
                [30] = 0x7f284705c1ca,
                [31] = 0x675f61d6be26ab00,
                [32] = 0x0,
                [33] = 0x7f2783c8f570,
                [34] = 0x55f457136330 <dflt_key_cache>,
                [35] = 0x55f4569570ec <my_write_stderr+28>,
                [36] = 0x7f2783c8f570,
                [37] = 0x7f5e818c86b9 <write+57>,
                [38] = 0x0,
                [39] = 0x55f4569571c3 <my_safe_printf_stderr+195>,
                [40] = 0x0,
                [41] = 0x3000000010,
                [42] = 0x7f2783c8f850,
                [43] = 0x7f2783c8f780,
                [44] = 0x6974706d65747441,
                [45] = 0x746b63616220676e,
                [46] = 0x6f59202e65636172,
                [47] = 0x7375206e61632075,
                [48] = 0x6f66206568742065,
                [49] = 0x20676e69776f6c6c,
                [50] = 0x74616d726f666e69,
                [51] = 0x66206f74206e6f69,
                [52] = 0xa74756f20646e69,
                [53] = 0x796d206572656877,
                [54] = 0x65696420646c7173,
                [55] = 0x6f79206649202e64,
                [56] = 0x6f6e206565732075,
                [57] = 0x6567617373656d20,
                [58] = 0x2072657466612073,
                [59] = 0x6f73202c73696874,
                [60] = 0x20676e696874656d,
                [61] = 0x7265740a746e6577,
                [62] = 0x727720796c626972,
                [63] = 0xa2e2e2e676e6f,
                [64] = 0x616d207369687420,
                [65] = 0xa2e6c6961662079,
                [66] = 0x61206e616320000a,
                [67] = 0x63206562206f736c,
                [68] = 0x7962206465737561,
                [69] = 0x636e75666c616d20,
                [70] = 0x20676e696e6f6974,
                [71] = 0x6572617764726168,
                [72] = 0xa0a2e,
                [73] = 0x0 <repeats 36 times>,
                [109] = 0x675f61d6be26ab00,
                [110] = 0xca,
                [111] = 0x55f456a45a40,
                [112] = 0x1f,
                [113] = 0x7f5e818c86ad <write+45>,
                [114] = 0x70,
                [115] = 0x0,
                [116] = 0x0,
                [117] = 0x7f2783c8f810,
                [118] = 0xc9,
                [119] = 0x59dcaef1,
                [120] = 0x18,
                [121] = 0x7f2783c8f830,
                [122] = 0x7f2783c8f7f0,
                [123] = 0x7f2783c8f878,
                [124] = 0x1,
                [125] = 0x0,
                [126] = 0x2f,
                [127] = 0x55f456a4e8d4}
              strings = 0x0
              n = <optimized out>
      #9  0x000055f45647aed5 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/signal_handler.cc:166
              curr_time = 1507634929
              tm = {
                tm_sec = 49, 
                tm_min = 28, 
                tm_hour = 12, 
                tm_mday = 10, 
                tm_mon = 9, 
                tm_year = 117, 
                tm_wday = 2, 
                tm_yday = 282, 
                tm_isdst = 1, 
                tm_gmtoff = 3600, 
                tm_zone = 0x7f5e7c83b070 "BST"
              }
              print_invalid_query_pointer = false
      #10 <signal handler called>
      No symbol table info available.
      #11 ha_rollback_trans (thd=thd@entry=0x7f282328f008, all=all@entry=true) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/handler.cc:1591
      No locals.
      #12 0x000055f4563d6a9a in trans_rollback (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/transaction.cc:340
              res = <optimized out>
      #13 0x000055f4562c51b2 in THD::cleanup (this=this@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_class.cc:1582
      No locals.
      #14 0x000055f4562578ea in thd_cleanup (thd=thd@entry=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2756
      No locals.
      #15 0x000055f456257ad7 in unlink_thd (thd=0x7f282328f008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2824
      No locals.
      #16 0x000055f456257c90 in one_thread_per_connection_end (thd=<optimized out>, put_in_cache=<optimized out>) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/mysqld.cc:2952
              wsrep_applier = false
      #17 0x000055f4563c8963 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1368
              create_user = <optimized out>
              thd = 0x7f282328f008
      #18 0x000055f4563c8b80 in handle_one_connection (arg=arg@entry=0x7f28084ed008) at /home/buildbot/buildbot/build/mariadb-10.1.26/sql/sql_connect.cc:1261
              thd = 0x7f28084ed008
      #19 0x000055f456607e8d in pfs_spawn_thread (arg=0x7f283af7cc08) at /home/buildbot/buildbot/build/mariadb-10.1.26/storage/perfschema/pfs.cc:1860
              typed_arg = 0x7f283af7cc08
              user_arg = 0x7f28084ed008
              pfs = <optimized out>
              user_start_routine = 0x55f4563c8b40 <handle_one_connection(void*)>
              klass = <optimized out>
      #20 0x00007f5e818c1e25 in start_thread () from /lib64/libpthread.so.0
      No symbol table info available.
      #21 0x00007f5e7fc6534d in clone () from /lib64/libc.so.6
      No symbol table info available.
      +quit
      

      Attachments

        1. other_err_log_occurrences.txt
          17 kB
          Przemek
        2. server.cnf
          3 kB
          Przemek

        Issue Links

          Activity

            People

              Unassigned Unassigned
              przemek@mysqlmaniac.com Przemek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.