Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17413

Crash on shutdown in my_malloc_size_cb_func, after "Forcing close of thread"..

Details

    Description

      From the file attached to MDEV-17407 , https://jira.mariadb.org/secure/attachment/46573/CE-APP.err

      2018-10-09 10:07:02 3224 [Note] Event Scheduler: Purging the queue. 0 events
      2018-10-09 10:07:02 6120 [Note] InnoDB: FTS optimize thread exiting.
      2018-10-09 10:07:33 3224 [Warning] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: Forcing close of thread 10  user: 'root'
       
      2018-10-09 10:07:33 3224 [Warning] C:\Program Files\MariaDB 10.2\bin\mysqld.exe: Forcing close of thread 9  user: 'bCEDAR57Z1C85I2'
       
      181009 10:07:33 [ERROR] mysqld got exception 0xc0000005 ;
       
      <...>
       
      Server version: 10.2.14-MariaDB
      key_buffer_size=134217728
      read_buffer_size=131072
      max_used_connections=2
      max_threads=65537
      thread_count=2
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 136058 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...mysqld.exe!my_malloc_size_cb_func()[mysqld.cc:4033]
      mysqld.exe!my_malloc()[my_malloc.c:126]
      mysqld.exe!net_real_write()[net_serv.cc:636]
      mysqld.exe!net_flush()[net_serv.cc:382]
      mysqld.exe!net_write_command()[net_serv.cc:518]
      mysqld.exe!net_send_error_packet()[protocol.cc:467]
      mysqld.exe!net_send_error()[protocol.cc:175]
      mysqld.exe!close_connection()[mysqld.cc:2824]
      mysqld.exe!close_connections()[mysqld.cc:1820]
      mysqld.exe!kill_server()[mysqld.cc:2016]
      mysqld.exe!handle_shutdown()[mysqld.cc:3673]
      mysqld.exe!pthread_start()[my_winthread.c:62]
      mysqld.exe!thread_start<unsigned int (__cdecl*)(void * __ptr64)>()[thread.cpp:115]
      KERNEL32.dll!BaseThreadInitThunk()
      

      Can be reproduced with non-responsive threads (i.e adding large sleep() before connection close), and compression being used on connection.
      net_read_write would try to allocate thread-specific memory, for compression and then crash in my_malloc_size_cb_func(), since current_thd is NULL.

      Normally, connections are closed from inside the connection itself, i.e current_thd != NULL, but in the specific case of closing connections from outside
      it is not the case.
      This can be reproduced also in 10.0, but just with DBUG version , but in 10.1+ also in release version

      Commit b4ff64568c88 is the culprit, since it stopped checking for current_thd == NULL if thread-specfic memory is allocated.

      Attachments

        Issue Links

          Activity

            I requested the config files in MDEV-17407.

            elenst Elena Stepanova added a comment - I requested the config files in MDEV-17407 .
            wlad Vladislav Vaintroub added a comment - - edited

            I can reproduce it, with one hanging (non-responsive) thread, and client --compress option

            A patch to emulate hanging thread:

            --- a/sql/threadpool_common.cc
            +++ b/sql/threadpool_common.cc
            @@ -209,6 +209,7 @@ void threadpool_cleanup_connection(THD *thd)
             {
               thd->net.reading_or_writing = 0;
               end_connection(thd);
            +  Sleep(100000);
               close_connection(thd, 0);
               unlink_thd(thd);
               mysql_cond_broadcast(&COND_thread_count);
            

            0. apply a patch
            1. start server with mysqld --console
            2. connect with a client (on Windows)
            mysql -uroot --compress
            3.CTRL-C in the server console to initiate shutdown
            wait for long time, maybe 20-30 seconds
            see the crash

            This issue should not be specific to Windows (just the patch to simulate a hang is windows-specific)

            wlad Vladislav Vaintroub added a comment - - edited I can reproduce it, with one hanging (non-responsive) thread, and client --compress option A patch to emulate hanging thread: --- a/sql/threadpool_common.cc +++ b/sql/threadpool_common.cc @@ -209,6 +209,7 @@ void threadpool_cleanup_connection(THD *thd) { thd->net.reading_or_writing = 0; end_connection(thd); + Sleep(100000); close_connection(thd, 0); unlink_thd(thd); mysql_cond_broadcast(&COND_thread_count); 0. apply a patch 1. start server with mysqld --console 2. connect with a client (on Windows) mysql -uroot --compress 3.CTRL-C in the server console to initiate shutdown wait for long time, maybe 20-30 seconds see the crash This issue should not be specific to Windows (just the patch to simulate a hang is windows-specific)

            In 10.0, this also crashes, but only with DBUG assertion

            wlad Vladislav Vaintroub added a comment - In 10.0, this also crashes, but only with DBUG assertion

            People

              wlad Vladislav Vaintroub
              wlad Vladislav Vaintroub
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.