Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11896

thd_get_error_context_description race condition

Details

    Description

      A Galera Cluster node crashed with the stack trace below

      170118 10:01:53 [ERROR] mysqld got signal 11 ;
      		 
      This could be because you hit a bug. It is also possible that this binary
      		 
      or one of the libraries it was linked against is corrupt, improperly built,
      		 
      or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
       
      		 
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
       
      		 
      We will try our best to scrape up some info that will hopefully help
      		 
      diagnose the problem, but since we have already crashed,
      		 
      something is definitely wrong and this may fail.
      		 
       
      		 
      Server version: 10.0.28-MariaDB-1~trusty-wsrep
      		 
      key_buffer_size=33554432
      		 
      read_buffer_size=4194304
      		 
      max_used_connections=2
      		 
      max_threads=2085
      		 
      thread_count=3
      		 
      It is possible that mysqld could use up to
      		 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17155321 K  bytes of memory
      		 
      Hope that's ok; if not, decrease some variables in the equation.
      		 
       
      		 
      Thread pointer: 0x0x7f55bc8cb008
      		 
      Attempting backtrace. You can use the following information to find out
      		 
      where mysqld died. If you see no messages after this, something went
      		 
      terribly wrong...
      		 
      stack_bottom = 0x7f63055eadc0 thread_stack 0x48000
      		 
      /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f63061cd67e]
      		 
      /usr/sbin/mysqld(handle_fatal_signal+0x433)[0x7f6305cf5853]
      		 
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f6304493330]
      		 
      /lib/x86_64-linux-gnu/libc.so.6(+0x97ee0)[0x7f6303b4bee0]
      		 
      /usr/sbin/mysqld(_ZN6String6appendEPKcj+0xb8)[0x7f6305be8708]
      		 
      /usr/sbin/mysqld(thd_get_error_context_description+0x28e)[0x7f6305b2ccbe]
      		 
      /usr/sbin/mysqld(+0x80e5c4)[0x7f6305f675c4]
      		 
      /usr/sbin/mysqld(+0x846b41)[0x7f6305f9fb41]
      		 
      /usr/sbin/mysqld(+0x8dfcde)[0x7f6306038cde]
      		 
      /usr/sbin/mysqld(+0x8077b9)[0x7f6305f607b9]
      		 
      /usr/sbin/mysqld(_Z14ha_show_statusP3THDP10handlerton12ha_stat_type+0x3e5)[0x7f6305cffd95]
      		 
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1e35)[0x7f6305b65fa5]
      		 
      /usr/sbin/mysqld(+0x41606b)[0x7f6305b6f06b]
      		 
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1fcd)[0x7f6305b7167d]
      		 
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x2fd)[0x7f6305b7244d]
      		 
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x34b)[0x7f6305c4307b]
      		 
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f6305c43160]
      		 
      /usr/sbin/mysqld(+0x74bd4d)[0x7f6305ea4d4d]
      		 
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f630448b184]
      		 
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6303bae37d]
      		 
       
      		 
      Trying to get some variables.
      		 
      Some pointers may be invalid and cause the dump to abort.
      		 
      Query (0x7f55c08cb020): is an invalid pointer
      		 
      Connection ID (thread ID): 417
      		 
      Status: NOT_KILLED

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MDEV-10699 Mariadb crash with assertion error

          • Major - Major loss of function.
          • Closed
          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13983 Mariadb becomes unresponsive

          • Major - Major loss of function.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-13196 Remove handler::store_lock()

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-13754 TRANSACTION sometimes stay in "KILLLED' STATE for inifinitve time!

          • Major - Major loss of function.
          • Closed
          links to

          Web Link upstream bug 64125

          Activity

            Ascending order - Click to sort in descending order
            elenst Elena Stepanova added a comment -

            Is it reproducible, or happens periodically, or happened only once?
            Did only one node crash?
            Do you know what was was happening on the node right before the crash?

            elenst Elena Stepanova added a comment - Is it reproducible, or happens periodically, or happened only once? Did only one node crash? Do you know what was was happening on the node right before the crash?
            tbanza TYNDALE BANZA added a comment -

            This error is not reproducible . It happened only once .

            Only one node crashed . The log files do not show anything significant before the crash .

            tbanza TYNDALE BANZA added a comment - This error is not reproducible . It happened only once . Only one node crashed . The log files do not show anything significant before the crash .
            elenst Elena Stepanova added a comment -

            I'm afraid there isn't much we can do about it at the moment, only wait till somebody else reports something similar and then compare circumstances and information.

            elenst Elena Stepanova added a comment - I'm afraid there isn't much we can do about it at the moment, only wait till somebody else reports something similar and then compare circumstances and information.
            danblack Daniel Black added a comment -

            Looking at the stack trace thd_get_error_context_description calls string append and then segfaults. Under http://www.cplusplus.com/reference/string/string/append/ a string.append with insufficient space results in undefined behaviour, in this case a segfault.

            https://github.com/MariaDB/server/blob/10.1/sql/sql_class.cc#L715 does append a lot of information like IP, username and query is appended to the str of length provided to it by its caller and at no point during this function is there an examination as to how much space is left.

            At least one call passes a 1K buffer - myisam::_mi_report_crashed. Innodb and xtradb also pass a 1k buffer (innobase_mysql_print_thd). Queries of over 1K as messy as they are are plausible.

            Rewriting thd_get_error_context_description to protect against buffer overflows even if all the information doesn't get in the buffer seems like a prudent and feasible solution.

            danblack Daniel Black added a comment - Looking at the stack trace thd_get_error_context_description calls string append and then segfaults. Under http://www.cplusplus.com/reference/string/string/append/ a string.append with insufficient space results in undefined behaviour, in this case a segfault. https://github.com/MariaDB/server/blob/10.1/sql/sql_class.cc#L715 does append a lot of information like IP, username and query is appended to the str of length provided to it by its caller and at no point during this function is there an examination as to how much space is left. At least one call passes a 1K buffer - myisam::_mi_report_crashed. Innodb and xtradb also pass a 1k buffer (innobase_mysql_print_thd). Queries of over 1K as messy as they are are plausible. Rewriting thd_get_error_context_description to protect against buffer overflows even if all the information doesn't get in the buffer seems like a prudent and feasible solution.
            danblack Daniel Black added a comment -

            Also appears in 5.5 as thd_security_context

            danblack Daniel Black added a comment - Also appears in 5.5 as thd_security_context
            danblack Daniel Black added a comment -

            quite a few other upstream mysql bugs too but still appears in mysql-5.7

            danblack Daniel Black added a comment - quite a few other upstream mysql bugs too but still appears in mysql-5.7
            serg Sergei Golubchik added a comment -

            String::append() reallocates automatically, it can never overflow a buffer. That's why this code at the end:

            thd_security_context()

            		 
            754
            		 
              if (str.c_ptr_safe() == buffer)
            		 
            755
            		 
                return buffer;
            		 
            756
            		 
             
            		 
            757
            		 
              /*
            		 
            758
            		 
                We have to copy the new string to the destination buffer because the string
            		 
            759
            		 
                was reallocated to a larger buffer to be able to fit.
            		 
            760
            		 
              */
            		 
            761
            		 
              DBUG_ASSERT(buffer != NULL);
            		 
            762
            		 
              length= min(str.length(), length-1);
            		 
            763
            		 
              memcpy(buffer, str.c_ptr_quick(), length);
            		 
            764
            		 
              /* Make sure that the new string is null terminated */
            		 
            765
            		 
              buffer[length]= '\0';
            		 
            766
            		 
              return buffer;

            serg Sergei Golubchik added a comment - String::append() reallocates automatically, it can never overflow a buffer. That's why this code at the end: thd_security_context() 754 if (str.c_ptr_safe() == buffer) 755 return buffer; 756   757 /* 758 We have to copy the new string to the destination buffer because the string 759 was reallocated to a larger buffer to be able to fit. 760 */ 761 DBUG_ASSERT(buffer != NULL); 762 length= min(str.length(), length-1); 763 memcpy (buffer, str.c_ptr_quick(), length); 764 /* Make sure that the new string is null terminated */ 765 buffer[length]= '\0' ; 766 return buffer;
            marko Marko Mäkelä added a comment -

            The attempt to fix this caused server hangs when InnoDB detects a deadlock between transactions (MDEV-13983/MDEV-16467), and it looks like the change was reverted to fix that hang.

            tbanza, if you can repeat this problem with 10.1.41, 10.2.25, or 10.3 or later, please file a new bug and mark it related to this one.

            marko Marko Mäkelä added a comment - The attempt to fix this caused server hangs when InnoDB detects a deadlock between transactions ( MDEV-13983 / MDEV-16467 ), and it looks like the change was reverted to fix that hang . tbanza , if you can repeat this problem with 10.1.41, 10.2.25, or 10.3 or later, please file a new bug and mark it related to this one.

            People

              serg Sergei Golubchik
              tbanza TYNDALE BANZA
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.