[MDEV-17413] Crash on shutdown in my_malloc_size_cb_func, after "Forcing close of thread".. Created: 2018-10-09  Updated: 2018-10-09  Resolved: 2018-10-09

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.2.14
Fix Version/s: 10.3.11, 10.1.37, 10.2.19

Type: Bug Priority: Major
Reporter: Vladislav Vaintroub Assignee: Vladislav Vaintroub
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-4011 per-thread memory usage statistics Closed
relates to MDEV-4149 Assertion `thd' fails in my_malloc_si... Closed
relates to MDEV-6580 Assertion `thd' failed in my_malloc_s... Closed

 Description   

From the file attached to MDEV-17407 , https://jira.mariadb.org/secure/attachment/46573/CE-APP.err

2018-10-09 10:07:02 3224 [Note] Event Scheduler: Purging the queue. 0 events
2018-10-09 10:07:02 6120 [Note] InnoDB: FTS optimize thread exiting.
2018-10-09 10:07:33 3224 [Warning] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: Forcing close of thread 10  user: 'root'
 
2018-10-09 10:07:33 3224 [Warning] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: Forcing close of thread 9  user: 'bCEDAR57Z1C85I2'
 
181009 10:07:33 [ERROR] mysqld got exception 0xc0000005 ;
 
<...>
 
Server version: 10.2.14-MariaDB
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=2
max_threads=65537
thread_count=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136058 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...mysqld.exe!my_malloc_size_cb_func()[mysqld.cc:4033]
mysqld.exe!my_malloc()[my_malloc.c:126]
mysqld.exe!net_real_write()[net_serv.cc:636]
mysqld.exe!net_flush()[net_serv.cc:382]
mysqld.exe!net_write_command()[net_serv.cc:518]
mysqld.exe!net_send_error_packet()[protocol.cc:467]
mysqld.exe!net_send_error()[protocol.cc:175]
mysqld.exe!close_connection()[mysqld.cc:2824]
mysqld.exe!close_connections()[mysqld.cc:1820]
mysqld.exe!kill_server()[mysqld.cc:2016]
mysqld.exe!handle_shutdown()[mysqld.cc:3673]
mysqld.exe!pthread_start()[my_winthread.c:62]
mysqld.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>()[thread.cpp:115]
KERNEL32.dll!BaseThreadInitThunk()

Can be reproduced with non-responsive threads (i.e adding large sleep() before connection close), and compression being used on connection.
net_read_write would try to allocate thread-specific memory, for compression and then crash in my_malloc_size_cb_func(), since current_thd is NULL.

Normally, connections are closed from inside the connection itself, i.e current_thd != NULL, but in the specific case of closing connections from outside
it is not the case.
This can be reproduced also in 10.0, but just with DBUG version , but in 10.1+ also in release version

Commit b4ff64568c88 is the culprit, since it stopped checking for current_thd == NULL if thread-specfic memory is allocated.



 Comments   
Comment by Elena Stepanova [ 2018-10-09 ]

I requested the config files in MDEV-17407.

Comment by Vladislav Vaintroub [ 2018-10-09 ]

I can reproduce it, with one hanging (non-responsive) thread, and client --compress option

A patch to emulate hanging thread:

--- a/sql/threadpool_common.cc
+++ b/sql/threadpool_common.cc
@@ -209,6 +209,7 @@ void threadpool_cleanup_connection(THD *thd)
 {
   thd->net.reading_or_writing = 0;
   end_connection(thd);
+  Sleep(100000);
   close_connection(thd, 0);
   unlink_thd(thd);
   mysql_cond_broadcast(&COND_thread_count);

0. apply a patch
1. start server with mysqld --console
2. connect with a client (on Windows)
mysql -uroot --compress
3.CTRL-C in the server console to initiate shutdown
wait for long time, maybe 20-30 seconds
see the crash

This issue should not be specific to Windows (just the patch to simulate a hang is windows-specific)

Comment by Vladislav Vaintroub [ 2018-10-09 ]

In 10.0, this also crashes, but only with DBUG assertion

Generated at Thu Feb 08 08:36:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.