Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13733

Server Crash - mysqld got exception 0xe06d7363

Details

    Description

      We have a crash (happened 2 times in 4 days...) of MariaDB Server. Logs attached.
      For information, this crash happened a few hours after a migration from MySQL 5.7.16 to MariaDB 10.1.22 (migration done by creating a MySQL dump and reimport in MariaDB).

      Here's my ini file, don't hesitate if someone has remarks on it (even if not related to this bug) :

      my.ini

      [mysqld]
      datadir=C:/Program Files/MariaDB 10.1/data
      port=3306
      sql_mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION"
      default_storage_engine=innodb
      innodb_buffer_pool_size=2048M
      innodb_log_file_size=128M
      character-set-server=utf8

      max_allowed_packet=8M
      tmp_table_size=69M
      innodb_thread_concurrency=8

      transaction-isolation=READ-COMMITTED

      slow_query_log=1
      long_query_time=1
      log_output=TABLE

      [client]
      port=3306
      plugin-dir=C:/Program Files/MariaDB 10.1/lib/plugin

      thanks in advance for your help.

      Attachments

        Issue Links

          Activity

            rcb73 Renaud added a comment -

            We had again the crash this night, please help...
            Here's the log :

            170907  2:20:02 [ERROR] mysqld got exception 0xe06d7363 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.
             
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.1.22-MariaDB
            key_buffer_size=134217728
            read_buffer_size=131072
            max_used_connections=12
            max_threads=1001
            thread_count=12
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2311984 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x0
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            KERNELBASE.dll!RaiseException()
            mysqld.exe!_CxxThrowException()[throw.cpp:136]
            mysqld.exe!std::_Xout_of_range()[xthrow.cpp:25]
            mysqld.exe!dict_stats_analyze_index_for_n_prefix()[dict0stats.cc:1757]
            mysqld.exe!dict_stats_analyze_index()[dict0stats.cc:2086]
            mysqld.exe!dict_stats_update_persistent()[dict0stats.cc:2289]
            mysqld.exe!dict_stats_update()[dict0stats.cc:3239]
            mysqld.exe!dict_stats_process_entry_from_recalc_pool()[dict0stats_bg.cc:458]
            mysqld.exe!dict_stats_thread()[dict0stats_bg.cc:551]
            kernel32.dll!BaseThreadInitThunk()
            ntdll.dll!RtlUserThreadStart()
            The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
            information that should help you find out what is causing the crash.
            2017-09-07  6:07:05 4668 [Note] InnoDB: Using mutexes to ref count buffer pool pages
            2017-09-07  6:07:05 4668 [Note] InnoDB: The InnoDB memory heap is disabled
            2017-09-07  6:07:05 4668 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions
            2017-09-07  6:07:05 4668 [Note] InnoDB: _mm_lfence() and _mm_sfence() are used for memory barrier
            2017-09-07  6:07:05 4668 [Note] InnoDB: Compressed tables use zlib 1.2.3
            2017-09-07  6:07:05 4668 [Note] InnoDB: Using generic crc32 instructions
            2017-09-07  6:07:05 4668 [Note] InnoDB: Initializing buffer pool, size = 2.0G
            2017-09-07  6:07:05 4668 [Note] InnoDB: Completed initialization of buffer pool
            2017-09-07  6:07:06 4668 [Note] InnoDB: Highest supported file format is Barracuda.
            2017-09-07  6:07:06 4668 [Note] InnoDB: Starting crash recovery from checkpoint LSN=44449667045
            2017-09-07  6:07:06 4668 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
            InnoDB: 4 transaction(s) which must be rolled back or cleaned up
            InnoDB: in total 4 row operations to undo
            InnoDB: Trx id counter is 1337856
            2017-09-07  6:07:07 4668 [Note] InnoDB: Starting final batch to recover 192 pages from redo log
            2017-09-07  6:07:08 4668 [Note] InnoDB: 128 rollback segment(s) are active.
            InnoDB: Starting in background the rollback of uncommitted transactions
            2017-09-07 06:07:08 24c  InnoDB: Rolling back trx with id 1337308, 1 rows to undo
            2017-09-07  6:07:08 4668 [Note] InnoDB: Waiting for purge to start
            2017-09-07  6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337308 completed
            2017-09-07 06:07:08 24c  InnoDB: Rolling back trx with id 1337306, 1 rows to undo
            2017-09-07  6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337306 completed
            2017-09-07 06:07:08 24c  InnoDB: Rolling back trx with id 1337305, 1 rows to undo
            2017-09-07  6:07:08 4668 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 44450925376
            2017-09-07  6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337305 completed
            2017-09-07 06:07:08 24c  InnoDB: Rolling back trx with id 1337214, 1 rows to undo
            2017-09-07  6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337214 completed
            2017-09-07 06:07:08 24c  InnoDB: Rollback of non-prepared transactions completed
            2017-09-07  6:07:08 5848 [Note] InnoDB: Dumping buffer pool(s) not yet started
            2017-09-07  6:07:08 4668 [Note] Plugin 'FEEDBACK' is disabled.
            2017-09-07  6:07:08 4668 [Note] Server socket created on IP: '::'.
            2017-09-07  6:07:08 4668 [Note] C:\Program Files\MariaDB 10.1\bin\mysqld.exe: ready for connections.
            Version: '10.1.22-MariaDB'  socket: ''  port: 3306  mariadb.org binary distribution
            

            rcb73 Renaud added a comment - We had again the crash this night, please help... Here's the log : 170907 2:20:02 [ERROR] mysqld got exception 0xe06d7363 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.   To report this bug, see https://mariadb.com/kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.1.22-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=12 max_threads=1001 thread_count=12 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2311984 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... KERNELBASE.dll!RaiseException() mysqld.exe!_CxxThrowException()[throw.cpp:136] mysqld.exe!std::_Xout_of_range()[xthrow.cpp:25] mysqld.exe!dict_stats_analyze_index_for_n_prefix()[dict0stats.cc:1757] mysqld.exe!dict_stats_analyze_index()[dict0stats.cc:2086] mysqld.exe!dict_stats_update_persistent()[dict0stats.cc:2289] mysqld.exe!dict_stats_update()[dict0stats.cc:3239] mysqld.exe!dict_stats_process_entry_from_recalc_pool()[dict0stats_bg.cc:458] mysqld.exe!dict_stats_thread()[dict0stats_bg.cc:551] kernel32.dll!BaseThreadInitThunk() ntdll.dll!RtlUserThreadStart() The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 2017-09-07 6:07:05 4668 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2017-09-07 6:07:05 4668 [Note] InnoDB: The InnoDB memory heap is disabled 2017-09-07 6:07:05 4668 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions 2017-09-07 6:07:05 4668 [Note] InnoDB: _mm_lfence() and _mm_sfence() are used for memory barrier 2017-09-07 6:07:05 4668 [Note] InnoDB: Compressed tables use zlib 1.2.3 2017-09-07 6:07:05 4668 [Note] InnoDB: Using generic crc32 instructions 2017-09-07 6:07:05 4668 [Note] InnoDB: Initializing buffer pool, size = 2.0G 2017-09-07 6:07:05 4668 [Note] InnoDB: Completed initialization of buffer pool 2017-09-07 6:07:06 4668 [Note] InnoDB: Highest supported file format is Barracuda. 2017-09-07 6:07:06 4668 [Note] InnoDB: Starting crash recovery from checkpoint LSN=44449667045 2017-09-07 6:07:06 4668 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer... InnoDB: 4 transaction(s) which must be rolled back or cleaned up InnoDB: in total 4 row operations to undo InnoDB: Trx id counter is 1337856 2017-09-07 6:07:07 4668 [Note] InnoDB: Starting final batch to recover 192 pages from redo log 2017-09-07 6:07:08 4668 [Note] InnoDB: 128 rollback segment(s) are active. InnoDB: Starting in background the rollback of uncommitted transactions 2017-09-07 06:07:08 24c InnoDB: Rolling back trx with id 1337308, 1 rows to undo 2017-09-07 6:07:08 4668 [Note] InnoDB: Waiting for purge to start 2017-09-07 6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337308 completed 2017-09-07 06:07:08 24c InnoDB: Rolling back trx with id 1337306, 1 rows to undo 2017-09-07 6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337306 completed 2017-09-07 06:07:08 24c InnoDB: Rolling back trx with id 1337305, 1 rows to undo 2017-09-07 6:07:08 4668 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.35-80.0 started; log sequence number 44450925376 2017-09-07 6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337305 completed 2017-09-07 06:07:08 24c InnoDB: Rolling back trx with id 1337214, 1 rows to undo 2017-09-07 6:07:08 588 [Note] InnoDB: Rollback of trx with id 1337214 completed 2017-09-07 06:07:08 24c InnoDB: Rollback of non-prepared transactions completed 2017-09-07 6:07:08 5848 [Note] InnoDB: Dumping buffer pool(s) not yet started 2017-09-07 6:07:08 4668 [Note] Plugin 'FEEDBACK' is disabled. 2017-09-07 6:07:08 4668 [Note] Server socket created on IP: '::'. 2017-09-07 6:07:08 4668 [Note] C:\Program Files\MariaDB 10.1\bin\mysqld.exe: ready for connections. Version: '10.1.22-MariaDB' socket: '' port: 3306 mariadb.org binary distribution
            elenst Elena Stepanova added a comment - - edited

            rcb73,

            As our InnoDB expert marko has found out, it is most likely a duplicate of MySQL bug which was fixed here:

            commit 29acdaaaeef9afe32b42785f1da3d79d56ed7e59
            Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
            Date:   Wed Feb 8 12:00:52 2017 +0530
             
                Bug #24585978       INNODB: ASSERTION TOTAL_RECS > 0 FAILURE
                                        IN FILE DICT0STATS.CC
            

            This bugfix should be in versions 10.0.31, 10.1.24, 10.2.7, 10.3.1. You have 10.1.22, so please try to upgrade to the latest 10.1 and see if it helps.

            elenst Elena Stepanova added a comment - - edited rcb73 , As our InnoDB expert marko has found out, it is most likely a duplicate of MySQL bug which was fixed here: commit 29acdaaaeef9afe32b42785f1da3d79d56ed7e59 Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com> Date: Wed Feb 8 12:00:52 2017 +0530   Bug #24585978 INNODB: ASSERTION TOTAL_RECS > 0 FAILURE IN FILE DICT0STATS.CC This bugfix should be in versions 10.0.31, 10.1.24, 10.2.7, 10.3.1. You have 10.1.22, so please try to upgrade to the latest 10.1 and see if it helps.
            rcb73 Renaud added a comment -

            Thanks Elena,

            we have done a lot of tests (regression, performance, etc...) with version 10.1.22, so i was wondering if there's a risk to upgrade from 10.1.22 to 10.1.26 ?
            Is it only bug fixes ? or is there any evolutions ? For example is there some parameters changes ? (default values change, new parameters, etc...)

            thanks for your help.

            rcb73 Renaud added a comment - Thanks Elena, we have done a lot of tests (regression, performance, etc...) with version 10.1.22, so i was wondering if there's a risk to upgrade from 10.1.22 to 10.1.26 ? Is it only bug fixes ? or is there any evolutions ? For example is there some parameters changes ? (default values change, new parameters, etc...) thanks for your help.

            rcb73,

            For defaults, there was one change between these versions, the default value of innodb_empty_free_list_algorithm has changed from BACKOFF to LEGACY. If you are using some dynamic plugins/engines, there could have been changes as well.

            Naturally, any upgrade involves risks – even if there are no known regressions between versions, there certainly can be hidden ones, maybe something very specific to your particular workflow. However, sooner or later you will have to upgrade – if not on any other reason, then because of security fixes, there have been quite a few already since 10.1.22, and they will keep coming. Given that and the crashes that you encounter, you could just as well start evaluating the new version now.

            elenst Elena Stepanova added a comment - rcb73 , For defaults, there was one change between these versions, the default value of innodb_empty_free_list_algorithm has changed from BACKOFF to LEGACY . If you are using some dynamic plugins/engines, there could have been changes as well. Naturally, any upgrade involves risks – even if there are no known regressions between versions, there certainly can be hidden ones, maybe something very specific to your particular workflow. However, sooner or later you will have to upgrade – if not on any other reason, then because of security fixes, there have been quite a few already since 10.1.22, and they will keep coming. Given that and the crashes that you encounter, you could just as well start evaluating the new version now.

            People

              Unassigned Unassigned
              rcb73 Renaud
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.