[MDEV-11896] thd_get_error_context_description race condition Created: 2017-01-24  Updated: 2020-12-17  Resolved: 2017-07-20

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 5.5.54, 10.1.21, 10.2.4, 10.0.28-galera
Fix Version/s: 10.1.26, 10.0.32, 10.2.8

Type: Bug Priority: Major
Reporter: TYNDALE BANZA Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: contribution, foundation, patch, security
Environment:

OS Ubuntu 14.04.5 LTS
EC2 m4.2xlarge

Issue Links:
Blocks
blocks MDEV-10699 Mariadb crash with assertion error Closed
Problem/Incident
causes MDEV-13983 Mariadb becomes unresponsive Closed
Relates
relates to MDEV-13196 Remove handler::store_lock() Open
relates to MDEV-13754 TRANSACTION sometimes stay in "KILLL... Closed

 Description   

A Galera Cluster node crashed with the stack trace below

170118 10:01:53 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed,
 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.0.28-MariaDB-1~trusty-wsrep
 
key_buffer_size=33554432
 
read_buffer_size=4194304
 
max_used_connections=2
 
max_threads=2085
 
thread_count=3
 
It is possible that mysqld could use up to
 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 17155321 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x0x7f55bc8cb008
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f63055eadc0 thread_stack 0x48000
 
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f63061cd67e]
 
/usr/sbin/mysqld(handle_fatal_signal+0x433)[0x7f6305cf5853]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f6304493330]
 
/lib/x86_64-linux-gnu/libc.so.6(+0x97ee0)[0x7f6303b4bee0]
 
/usr/sbin/mysqld(_ZN6String6appendEPKcj+0xb8)[0x7f6305be8708]
 
/usr/sbin/mysqld(thd_get_error_context_description+0x28e)[0x7f6305b2ccbe]
 
/usr/sbin/mysqld(+0x80e5c4)[0x7f6305f675c4]
 
/usr/sbin/mysqld(+0x846b41)[0x7f6305f9fb41]
 
/usr/sbin/mysqld(+0x8dfcde)[0x7f6306038cde]
 
/usr/sbin/mysqld(+0x8077b9)[0x7f6305f607b9]
 
/usr/sbin/mysqld(_Z14ha_show_statusP3THDP10handlerton12ha_stat_type+0x3e5)[0x7f6305cffd95]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1e35)[0x7f6305b65fa5]
 
/usr/sbin/mysqld(+0x41606b)[0x7f6305b6f06b]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1fcd)[0x7f6305b7167d]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x2fd)[0x7f6305b7244d]
 
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x34b)[0x7f6305c4307b]
 
/usr/sbin/mysqld(handle_one_connection+0x40)[0x7f6305c43160]
 
/usr/sbin/mysqld(+0x74bd4d)[0x7f6305ea4d4d]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f630448b184]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f6303bae37d]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f55c08cb020): is an invalid pointer
 
Connection ID (thread ID): 417
 
Status: NOT_KILLED



 Comments   
Comment by Elena Stepanova [ 2017-01-30 ]

Is it reproducible, or happens periodically, or happened only once?
Did only one node crash?
Do you know what was was happening on the node right before the crash?

Comment by TYNDALE BANZA [ 2017-01-31 ]

This error is not reproducible . It happened only once .

Only one node crashed . The log files do not show anything significant before the crash .

Comment by Elena Stepanova [ 2017-02-24 ]

I'm afraid there isn't much we can do about it at the moment, only wait till somebody else reports something similar and then compare circumstances and information.

Comment by Daniel Black [ 2017-02-25 ]

Looking at the stack trace thd_get_error_context_description calls string append and then segfaults. Under http://www.cplusplus.com/reference/string/string/append/ a string.append with insufficient space results in undefined behaviour, in this case a segfault.

https://github.com/MariaDB/server/blob/10.1/sql/sql_class.cc#L715 does append a lot of information like IP, username and query is appended to the str of length provided to it by its caller and at no point during this function is there an examination as to how much space is left.

At least one call passes a 1K buffer - myisam::_mi_report_crashed. Innodb and xtradb also pass a 1k buffer (innobase_mysql_print_thd). Queries of over 1K as messy as they are are plausible.

Rewriting thd_get_error_context_description to protect against buffer overflows even if all the information doesn't get in the buffer seems like a prudent and feasible solution.

Comment by Daniel Black [ 2017-02-25 ]

Also appears in 5.5 as thd_security_context

Comment by Daniel Black [ 2017-02-25 ]

quite a few other upstream mysql bugs too but still appears in mysql-5.7

Comment by Sergei Golubchik [ 2017-03-09 ]

String::append() reallocates automatically, it can never overflow a buffer. That's why this code at the end:

thd_security_context()

 
754
 
  if (str.c_ptr_safe() == buffer)
 
755
 
    return buffer;
 
756
 
 
 
757
 
  /*
 
758
 
    We have to copy the new string to the destination buffer because the string
 
759
 
    was reallocated to a larger buffer to be able to fit.
 
760
 
  */
 
761
 
  DBUG_ASSERT(buffer != NULL);
 
762
 
  length= min(str.length(), length-1);
 
763
 
  memcpy(buffer, str.c_ptr_quick(), length);
 
764
 
  /* Make sure that the new string is null terminated */
 
765
 
  buffer[length]= '\0';
 
766
 
  return buffer;

Comment by Marko Mäkelä [ 2020-01-30 ]

The attempt to fix this caused server hangs when InnoDB detects a deadlock between transactions (MDEV-13983/MDEV-16467), and it looks like the change was reverted to fix that hang.

tbanza, if you can repeat this problem with 10.1.41, 10.2.25, or 10.3 or later, please file a new bug and mark it related to this one.

Generated at Thu Feb 08 07:53:30 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.