Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-599

LP:625107 - xtradb crashes on shutdown

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Won't Fix
    • 5.2.13, 5.1.66, 5.3.12
    • 5.1.73, 5.2.15, 5.3.13
    • None
    • Windows, non-default build with XtraDB as a dynamic plugin

    Description

      on windows, mariadb-5.1, built with default options (win\configure.js) - xtradb is built as dll.
      On load it prints "Mutexes and rw_locks use InnoDB's own implementation"
      On shutdown it crashes.

      In innobase_shutdown_for_mysql() there is:

      2192         dict_close();
      2193         btr_search_sys_free();
      2194 
      2195         /* 3. Free all InnoDB's own mutexes and the os_fast_mutexes inside
      2196         them */
      2197         os_aio_free();
      2198         sync_close();
      2199         srv_free();

      Now, btr_search_sys_free() is

       182 btr_search_sys_free(void)
       184 {
       185         mem_free(btr_search_latch_temp);
       186         btr_search_latch_temp = NULL;

      It crashes later, in sync_close():

      1449         mutex = UT_LIST_GET_FIRST(mutex_list); 
      1451         while (mutex) {
      1458                 mutex_free(mutex);
      1459                 mutex = UT_LIST_GET_FIRST(mutex_list);
      1460         }

      because one of the mutexes (3rd in the list) is filled with 0xFEEEFEEE. This is the value that windows uses to fill the freed memory. And this mutex is freed in btr_search_sys_free(), line 185. When I comment this line (185) out, xtradb doesn't crash anymore.

      Attachments

        Activity

          stewart Stewart Smith added a comment -

          Re: xtradb crashes on shutdown
          All development of XtraDB has moved under the Percona Server project - https://launchpad.net/percona-server - If this bug can be reproduced against current Percona Server, please file this bug against percona-server (you can simply do so by using the "Also affects project" link above).

          Thanks,
          Stewart Smith
          Director of Server Development
          Percona.

          stewart Stewart Smith added a comment - Re: xtradb crashes on shutdown All development of XtraDB has moved under the Percona Server project - https://launchpad.net/percona-server - If this bug can be reproduced against current Percona Server, please file this bug against percona-server (you can simply do so by using the "Also affects project" link above). Thanks, Stewart Smith Director of Server Development Percona.

          Launchpad bug id: 625107

          ratzpo Rasmus Johansson (Inactive) added a comment - Launchpad bug id: 625107

          Elena, could you please check whether it's still repeatable?

          serg Sergei Golubchik added a comment - Elena, could you please check whether it's still repeatable?
          elenst Elena Stepanova added a comment - - edited

          Now XtraDB is built statically by default, so the described problem cannot happen in a regular scenario.
          However,

          1. If I force building it as dll on current 5.1 / 5.3, it does crash on shutdown. Stack trace I'm getting does not contain sync_close() though – maybe because of the assertion that happens before (stack trace is below).

          2. InnoDB plugin on 5.1 / 5.3 also crashes on shutdown, both on Windows and Linux.

          For 5.5, I could not build XtraDB as a dll on Windows (my best attempt failed with an unresolved symbol); but InnoDB 5.5 Windows does not crash, and neither InnoDB nor XtraDB 5.5 Linux. So, I suppose the problem is fixed in 5.5. Still, I can try XtraDB 5.5 Windows if somebody tells me how to build it as dll properly.

          Error log and stack trace from current 5.1:

          Version: '5.1.66-MariaDB-debug' socket: '' port: 3306 Source distribution
          InnoDB: The InnoDB memory heap is disabled
          InnoDB: Mutexes and rw_locks use InnoDB's own implementation
          InnoDB: Compressed tables use zlib 1.2.3
          InnoDB: Windows native async i/o is disabled as default.
          InnoDB: It is not applicable for the current multi io threads implementation.
          130105 9:41:32 InnoDB: Initializing buffer pool, size = 128.0M
          130105 9:41:32 InnoDB: Completed initialization of buffer pool
          InnoDB: The first specified data file .\ibdata1 did not exist:
          InnoDB: a new database to be created!
          130105 9:41:32 InnoDB: Setting file .\ibdata1 size to 10 MB
          InnoDB: Database physically writes the file full: wait...
          130105 9:41:32 InnoDB: Log file .\ib_logfile0 did not exist: new to be created
          InnoDB: Setting log file .\ib_logfile0 size to 5 MB
          InnoDB: Database physically writes the file full: wait...
          130105 9:41:32 InnoDB: Log file .\ib_logfile1 did not exist: new to be created
          InnoDB: Setting log file .\ib_logfile1 size to 5 MB
          InnoDB: Database physically writes the file full: wait...
          InnoDB: Doublewrite buffer not found: creating new
          InnoDB: Doublewrite buffer created
          InnoDB: Creating foreign key constraint system tables
          InnoDB: Foreign key constraint system tables created
          130105 9:41:33 Percona XtraDB (http://www.percona.com) 1.0.17-14.1 started; log sequence number 0
          130105 9:42:00 InnoDB: Starting shutdown...
          130105 9:42:00 InnoDB: Shutdown completed; log sequence number 45342
          f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c(1322) : Assertion failed: _CrtIsValidHeapPointer(pUserData)
          f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c(1328) : Assertion failed: _BLOCK_TYPE_IS_VALID(pHead->nBlockUs
          130105 9:42:00 [ERROR] mysqld got exception 0xc0000005 ;
          This could be because you hit a bug. It is also possible that this binary
          or one of the libraries it was linked against is corrupt, improperly built,
          or misconfigured. This error can also be caused by malfunctioning hardware.

          To report this bug, see http://kb.askmonty.org/en/reporting-bugs

          We will try our best to scrape up some info that will hopefully help
          diagnose the problem, but since we have already crashed,
          something is definitely wrong and this may fail.

          key_buffer_size=134213632
          read_buffer_size=131072
          max_used_connections=1
          max_threads=152
          thread_count=1
          connection_count=1
          It is possible that mysqld could use up to
          key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 462849 K bytes of memory
          Hope that's ok; if not, decrease some variables in the equation.

          Thread pointer: 0x0238ECC0
          Attempting backtrace. You can use the following information to find out
          where mysqld died. If you see no messages after this, something went
          terribly wrong...
          010F3226 mysqld.exe!_output_s_l()[output.c:1644]
          010E3E36 mysqld.exe!_vsnprintf_helper()[vsprintf.c:140]
          010E4360 mysqld.exe!_vsnprintf_s_l()[vsprintf.c:288]
          010E45D0 mysqld.exe!_vsnprintf_s()[vsprintf.c:340]
          010DE9BE mysqld.exe!_VCrtDbgReportA()[dbgrptt.c:298]
          010DAE22 mysqld.exe!_CrtDbgReportV()[dbgrpt.c:241]
          010DADEB mysqld.exe!_CrtDbgReport()[dbgrpt.c:258]
          010D3C7B mysqld.exe!_free_dbg_nolock()[dbgheap.c:1345]
          010D3A30 mysqld.exe!_free_dbg()[dbgheap.c:1265]
          010E8730 mysqld.exe!free()[dbgfree.c:49]
          01616F08 mysqld.exe!my_no_flags_free()[my_malloc.c:74]
          0113F80B mysqld.exe!plugin_vars_free_values()[sql_plugin.cc:2679]
          0113A9D9 mysqld.exe!plugin_del()[sql_plugin.cc:906]
          0113A7EE mysqld.exe!reap_plugins()[sql_plugin.cc:967]
          0113E9B2 mysqld.exe!mysql_uninstall_plugin()[sql_plugin.cc:1844]
          012733E0 mysqld.exe!mysql_execute_command()[sql_parse.cc:5075]
          01276EEB mysqld.exe!mysql_parse()[sql_parse.cc:6222]
          0126A923 mysqld.exe!dispatch_command()[sql_parse.cc:1294]
          01269ECA mysqld.exe!do_command()[sql_parse.cc:906]
          0116C092 mysqld.exe!handle_one_connection()[sql_connect.cc:1208]
          0161238D mysqld.exe!pthread_start()[my_winthread.c:90]
          015E90D3 mysqld.exe!_callthreadstart()[thread.c:259]
          015E9069 mysqld.exe!_threadstart()[thread.c:243]
          752A8543 KERNEL32.DLL!BaseThreadInitThunk()
          7728AC69 ntdll.dll!RtlInitializeExceptionChain()
          7728AC3C ntdll.dll!RtlInitializeExceptionChain()

          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (02390A90): uninstall plugin innodbConnection ID (thread ID): 1
          Status: NOT_KILLED

          The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
          information that should help you find out what is causing the crash.

          Steps to reproduce on 5.1 / 5.3:

          bzr branch lp:maria/<version>

          To make XtraDB be built as dll, I removed the following line from storage/xtradb/plug.in:
          MYSQL_PLUGIN_STATIC(xtradb, [libxtradb.la])
          It's not the right way, the right way is supposed to be setting -DWITH_XTRADB_STORAGE_ENGINE=0.

          cd <basedir>

          win\configure.js
          cmake .
          cmake --build . --config debug
          cd mysql-test
          perl mysql-test-run.pl --vs-config=debug 1st
          cd ..
          .\sql\Debug\mysqld.exe --plugin-dir=<basedir>\mysql-test\var\plugins --datadir=<basedir>\mysql-test\var\mysqld.1\data --console

          in a client:

          install plugin innodb soname 'ha_xtradb.dll';

          1. Query OK, 0 rows affected (0.16 sec)

          uninstall plugin innodb

          1. ERROR 2013 (HY000): Lost connection to MySQL server during query
          elenst Elena Stepanova added a comment - - edited Now XtraDB is built statically by default, so the described problem cannot happen in a regular scenario. However, 1. If I force building it as dll on current 5.1 / 5.3, it does crash on shutdown. Stack trace I'm getting does not contain sync_close() though – maybe because of the assertion that happens before (stack trace is below). 2. InnoDB plugin on 5.1 / 5.3 also crashes on shutdown, both on Windows and Linux. For 5.5, I could not build XtraDB as a dll on Windows (my best attempt failed with an unresolved symbol); but InnoDB 5.5 Windows does not crash, and neither InnoDB nor XtraDB 5.5 Linux. So, I suppose the problem is fixed in 5.5. Still, I can try XtraDB 5.5 Windows if somebody tells me how to build it as dll properly. Error log and stack trace from current 5.1: Version: '5.1.66-MariaDB-debug' socket: '' port: 3306 Source distribution InnoDB: The InnoDB memory heap is disabled InnoDB: Mutexes and rw_locks use InnoDB's own implementation InnoDB: Compressed tables use zlib 1.2.3 InnoDB: Windows native async i/o is disabled as default. InnoDB: It is not applicable for the current multi io threads implementation. 130105 9:41:32 InnoDB: Initializing buffer pool, size = 128.0M 130105 9:41:32 InnoDB: Completed initialization of buffer pool InnoDB: The first specified data file .\ibdata1 did not exist: InnoDB: a new database to be created! 130105 9:41:32 InnoDB: Setting file .\ibdata1 size to 10 MB InnoDB: Database physically writes the file full: wait... 130105 9:41:32 InnoDB: Log file .\ib_logfile0 did not exist: new to be created InnoDB: Setting log file .\ib_logfile0 size to 5 MB InnoDB: Database physically writes the file full: wait... 130105 9:41:32 InnoDB: Log file .\ib_logfile1 did not exist: new to be created InnoDB: Setting log file .\ib_logfile1 size to 5 MB InnoDB: Database physically writes the file full: wait... InnoDB: Doublewrite buffer not found: creating new InnoDB: Doublewrite buffer created InnoDB: Creating foreign key constraint system tables InnoDB: Foreign key constraint system tables created 130105 9:41:33 Percona XtraDB ( http://www.percona.com ) 1.0.17-14.1 started; log sequence number 0 130105 9:42:00 InnoDB: Starting shutdown... 130105 9:42:00 InnoDB: Shutdown completed; log sequence number 45342 f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c(1322) : Assertion failed: _CrtIsValidHeapPointer(pUserData) f:\dd\vctools\crt_bld\self_x86\crt\src\dbgheap.c(1328) : Assertion failed: _BLOCK_TYPE_IS_VALID(pHead->nBlockUs 130105 9:42:00 [ERROR] mysqld got exception 0xc0000005 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=134213632 read_buffer_size=131072 max_used_connections=1 max_threads=152 thread_count=1 connection_count=1 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 462849 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0238ECC0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... 010F3226 mysqld.exe!_output_s_l() [output.c:1644] 010E3E36 mysqld.exe!_vsnprintf_helper() [vsprintf.c:140] 010E4360 mysqld.exe!_vsnprintf_s_l() [vsprintf.c:288] 010E45D0 mysqld.exe!_vsnprintf_s() [vsprintf.c:340] 010DE9BE mysqld.exe!_VCrtDbgReportA() [dbgrptt.c:298] 010DAE22 mysqld.exe!_CrtDbgReportV() [dbgrpt.c:241] 010DADEB mysqld.exe!_CrtDbgReport() [dbgrpt.c:258] 010D3C7B mysqld.exe!_free_dbg_nolock() [dbgheap.c:1345] 010D3A30 mysqld.exe!_free_dbg() [dbgheap.c:1265] 010E8730 mysqld.exe!free() [dbgfree.c:49] 01616F08 mysqld.exe!my_no_flags_free() [my_malloc.c:74] 0113F80B mysqld.exe!plugin_vars_free_values() [sql_plugin.cc:2679] 0113A9D9 mysqld.exe!plugin_del() [sql_plugin.cc:906] 0113A7EE mysqld.exe!reap_plugins() [sql_plugin.cc:967] 0113E9B2 mysqld.exe!mysql_uninstall_plugin() [sql_plugin.cc:1844] 012733E0 mysqld.exe!mysql_execute_command() [sql_parse.cc:5075] 01276EEB mysqld.exe!mysql_parse() [sql_parse.cc:6222] 0126A923 mysqld.exe!dispatch_command() [sql_parse.cc:1294] 01269ECA mysqld.exe!do_command() [sql_parse.cc:906] 0116C092 mysqld.exe!handle_one_connection() [sql_connect.cc:1208] 0161238D mysqld.exe!pthread_start() [my_winthread.c:90] 015E90D3 mysqld.exe!_callthreadstart() [thread.c:259] 015E9069 mysqld.exe!_threadstart() [thread.c:243] 752A8543 KERNEL32.DLL!BaseThreadInitThunk() 7728AC69 ntdll.dll!RtlInitializeExceptionChain() 7728AC3C ntdll.dll!RtlInitializeExceptionChain() Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (02390A90): uninstall plugin innodbConnection ID (thread ID): 1 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Steps to reproduce on 5.1 / 5.3: bzr branch lp:maria/<version> To make XtraDB be built as dll, I removed the following line from storage/xtradb/plug.in: MYSQL_PLUGIN_STATIC(xtradb, [libxtradb.la] ) It's not the right way, the right way is supposed to be setting -DWITH_XTRADB_STORAGE_ENGINE=0. cd <basedir> win\configure.js cmake . cmake --build . --config debug cd mysql-test perl mysql-test-run.pl --vs-config=debug 1st cd .. .\sql\Debug\mysqld.exe --plugin-dir=<basedir>\mysql-test\var\plugins --datadir=<basedir>\mysql-test\var\mysqld.1\data --console in a client: install plugin innodb soname 'ha_xtradb.dll'; Query OK, 0 rows affected (0.16 sec) uninstall plugin innodb ERROR 2013 (HY000): Lost connection to MySQL server during query

          After Wlad's fix (revision-id: wlad@montyprogram.com-20130105225225-c2fjxdrft8ktq6kg revno: 3607) XtraDB in 5.5 can be built as dll on Windows, so I was able to check the final part – no crash on plugin shutdown in 5.5.

          elenst Elena Stepanova added a comment - After Wlad's fix (revision-id: wlad@montyprogram.com-20130105225225-c2fjxdrft8ktq6kg revno: 3607) XtraDB in 5.5 can be built as dll on Windows, so I was able to check the final part – no crash on plugin shutdown in 5.5.

          People

            serg Sergei Golubchik
            serg Sergei Golubchik
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.