Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.0.14
    • N/A
    • OTHER
    • Gentoo Linux, MariaDB installed by package system (emerge).
      8x Intel E5-2630, 8GB RAM and /var/lib/mysql on SSD harddrive

    Description

      Hi,

      our server MariaDB 10.0.14 crashed today, 1 months after upgrade from MySQL 5.1.61, which had 2 years uptime.

      This server contains only 2 databases with low load and total size about 1GB. One database with Magento e-shop database and one with our database (100x InnoDB tables, 1x Memory, 1x CONNECT and 1x TokuDB).

      I think, that it is not related to CONNECT, or TokuDB engine.

      MariaDB crashed during standard mysqldump-based backup proces.

      Below is also backtrace and MySQL process-list 1-3 seconds before crash.

      There was no load, just backup simple InnoDB tables of Magento database.

      Thank you for your help.

      141024  2:21:13 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 10.0.14-MariaDB
      key_buffer_size=134217728
      read_buffer_size=4194304
      max_used_connections=69
      max_threads=202
      thread_count=4
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4272191 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7fa9570f3008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fa954579e28 thread_stack 0x80000
      /usr/sbin/mysqld(my_print_stacktrace+0x26)[0xb16713]
      /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6f3658]
      /lib64/libpthread.so.0(+0x10150)[0x7fa9fa18b150]
      /usr/sbin/mysqld[0x88facb]
      /usr/sbin/mysqld[0x89079f]
      /usr/sbin/mysqld[0x890a1b]
      /usr/sbin/mysqld(_ZN7handler22ha_rnd_init_with_errorEb+0x15)[0x6f84a5]
      /usr/sbin/mysqld(_Z16init_read_recordP11READ_RECORDP3THDP5TABLEP10SQL_SELECTibb+0x4c2)[0x7c9f52]
      /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x70)[0x5f1d10]
      /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x161)[0x5f1ed1]
      /usr/sbin/mysqld[0x5fd82d]
      /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa08)[0x60e508]
      /usr/sbin/mysqld(_ZN4JOIN4execEv+0x9)[0x610019]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x148)[0x60d068]
      /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x294)[0x610314]
      /usr/sbin/mysqld[0x5bff2b]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x50fe)[0x5cac5e]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1fc)[0x5cd46c]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1530)[0x5ce9f0]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x19d)[0x674abd]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x674b80]
      /lib64/libpthread.so.0(+0x8bb5)[0x7fa9fa183bb5]
      /lib64/libc.so.6(clone+0x6d)[0x7fa9f8e76d7d]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7fa9634ba020): is an invalid pointer
      Connection ID (thread ID): 328170
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_me
      rge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_so
      rt_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.

      MySQL process list 1-3 seconds before crash.

      +--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
      | Id     | User            | Host      | db              | Command | Time | State                       | Info                                                               | Progress |
      +--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
      | 1      | event_scheduler | localhost |                 | Daemon  | 14   | Waiting for next activation |                                                                    | 0.000    |
      | 328170 | root            | localhost | client_cz_new   | Query   | 1    | Sending data                | SELECT /*!40001 SQL_NO_CACHE */ * FROM `ig_cashondelivery_foreign` | 0.000    |
      | 328175 | root            | localhost |                 | Query   | 0    | init                        | show processlist                                                   | 0.000    |
      | 328176 | root            | localhost |                 | Sleep   | 1    |                             |                                                                    | 0.000    |
      +--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+

      Attachments

        Activity

          Hi,

          How often do you run the same backup procedure? Did it happen before, or was it the first attempt after the upgrade to 10.0.14?

          Did you re-run the backup after the crash? If you did, did it succeed?

          How confidential is the data? Would you be able to upload it to our ftp.askmonty.org/private? (only MariaDB developers will have access to it)?

          If you can't provide all data, maybe at least client_cz_new.ig_cashondelivery_foreign which appears in the processlist – it is no guarantee that it was the last, but there is a chance.

          In any case, please also attach your cnf file(s) and the exact mysqldump command your process uses (or the backup script which performs it).

          Thank you.

          elenst Elena Stepanova added a comment - Hi, How often do you run the same backup procedure? Did it happen before, or was it the first attempt after the upgrade to 10.0.14? Did you re-run the backup after the crash? If you did, did it succeed? How confidential is the data? Would you be able to upload it to our ftp.askmonty.org/private? (only MariaDB developers will have access to it)? If you can't provide all data, maybe at least client_cz_new.ig_cashondelivery_foreign which appears in the processlist – it is no guarantee that it was the last, but there is a chance. In any case, please also attach your cnf file(s) and the exact mysqldump command your process uses (or the backup script which performs it). Thank you.
          jan.reges Ján Regeš added a comment -

          Hi Elena,

          backup procedure is executed every 6 hours. It happen just once.

          After re-run everything works fine.

          I uploaded "my.cnf" and our backup script "make-daily-mysql-backup" into ftp.askmonty.org/private (by jan.reges@siteone.cz).

          Btw, table "ig_cashondelivery_foreign" is empty, so it's not used. We store whole process-list every 2-3 seconds and it was last before crash.

          Also, maybe it can depends. We have some UDF:

          SELECT * FROM mysql.func;
           
          lib_mysqludf_preg_info	0	lib_mysqludf_preg.so	function
          preg_capture	0	lib_mysqludf_preg.so	function
          preg_check	2	lib_mysqludf_preg.so	function
          preg_replace	0	lib_mysqludf_preg.so	function
          preg_rlike	2	lib_mysqludf_preg.so	function
          preg_position	2	lib_mysqludf_preg.so	function
          microtime	1	mysql_udf_microtime.so	function
          microtime	1	mysql_udf_microtime.so

          jan.reges Ján Regeš added a comment - Hi Elena, backup procedure is executed every 6 hours. It happen just once. After re-run everything works fine. I uploaded "my.cnf" and our backup script "make-daily-mysql-backup" into ftp.askmonty.org/private (by jan.reges@siteone.cz). Btw, table "ig_cashondelivery_foreign" is empty, so it's not used. We store whole process-list every 2-3 seconds and it was last before crash. Also, maybe it can depends. We have some UDF: SELECT * FROM mysql.func;   lib_mysqludf_preg_info 0 lib_mysqludf_preg.so function preg_capture 0 lib_mysqludf_preg.so function preg_check 2 lib_mysqludf_preg.so function preg_replace 0 lib_mysqludf_preg.so function preg_rlike 2 lib_mysqludf_preg.so function preg_position 2 lib_mysqludf_preg.so function microtime 1 mysql_udf_microtime.so function microtime 1 mysql_udf_microtime.so

          Hi Ján,

          Thanks for the info.

          There are several things here I have difficulties explaining.

          First, if the table ig_cashondelivery_foreign is empty, it should not take 1 second or more to select from it, and it should be quite hard to catch it at 'Sending data' phase.
          Since you have lots of processlist logs (normal ones, not crash-related) – can you grep them to see if select from this table appears often?

          Secondly, I don't see anything in the backup process that should cause the stack trace like that. You had 4 connections, is it possible that one of the two others woke up and ran something? E.g. event scheduler started working? (The time is odd, which makes it rather unlikely but not impossible). Or could it be the other connection, the one that's sleeping in the processlist?

          Finally, about your select from mysql.func – it looks strange too. The first column in the table, 'name', is a primary key, so you should not have two functions called 'microtime'. Do you really, or is it a copy-paste error? If you seem to have two, could you please run check table mysql.func extended?

          I presume you could not provide the data dump, otherwise you would have, right?
          Then, could you please see in one of the dumps which table is queried next after ig_cashondelivery_foreign, and send both tables' structures (SHOW CREATE TABLE .. output and SHOW INDEX IN .. output)? Or, if it's not a table but some other object, it's even more interesting.

          Thanks.

          elenst Elena Stepanova added a comment - Hi Ján, Thanks for the info. There are several things here I have difficulties explaining. First, if the table ig_cashondelivery_foreign is empty, it should not take 1 second or more to select from it, and it should be quite hard to catch it at 'Sending data' phase. Since you have lots of processlist logs (normal ones, not crash-related) – can you grep them to see if select from this table appears often? Secondly, I don't see anything in the backup process that should cause the stack trace like that. You had 4 connections, is it possible that one of the two others woke up and ran something? E.g. event scheduler started working? (The time is odd, which makes it rather unlikely but not impossible). Or could it be the other connection, the one that's sleeping in the processlist? Finally, about your select from mysql.func – it looks strange too. The first column in the table, 'name', is a primary key, so you should not have two functions called 'microtime'. Do you really, or is it a copy-paste error? If you seem to have two, could you please run check table mysql.func extended ? I presume you could not provide the data dump, otherwise you would have, right? Then, could you please see in one of the dumps which table is queried next after ig_cashondelivery_foreign , and send both tables' structures ( SHOW CREATE TABLE .. output and SHOW INDEX IN .. output)? Or, if it's not a table but some other object, it's even more interesting. Thanks.

          Closing for now as incomplete, please comment to re-open if you have more information.

          elenst Elena Stepanova added a comment - Closing for now as incomplete, please comment to re-open if you have more information.

          People

            elenst Elena Stepanova
            jan.reges Ján Regeš
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.