Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9728

Hard crash in metadata_lock_info

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.1.12, 10.0, 10.1
    • 10.0.26, 10.1.15
    • Locking
    • None
    • 10.0.26

    Description

      Our server crashed today with the following stack trace:

      Server version: 10.1.12-MariaDB-1~trusty
      key_buffer_size=16777216
      read_buffer_size=131072
      max_used_connections=227
      max_threads=502
      thread_count=59
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1118998 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7fd005084008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7fcfe1e6bdf0 thread_stack 0x80000
      *** buffer overflow detected ***: /usr/sbin/mysqld terminated
      ======= Backtrace: =========
      /lib/x86_64-linux-gnu/libc.so.6(+0x7338f)[0x7fde8d17c38f]
      /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fde8d213c9c]
      /lib/x86_64-linux-gnu/libc.so.6(+0x109b60)[0x7fde8d212b60]
      /lib/x86_64-linux-gnu/libc.so.6(+0x10abe7)[0x7fde8d213be7]
      /usr/sbin/mysqld(my_addr_resolve+0x48)[0x7fde8fa78af8]
      /usr/sbin/mysqld(my_print_stacktrace+0x1c2)[0x7fde8fa651f2]
      /usr/sbin/mysqld(handle_fatal_signal+0x38d)[0x7fde8f5921bd]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fde8dae8340]
      /usr/sbin/mysqld(_ZN11MDL_context11find_ticketEP11MDL_requestP17enum_mdl_duration+0x68)[0x7fde8f4e9948]
      /usr/lib/mysql/plugin/metadata_lock_info.so(_Z31i_s_metadata_lock_info_fill_rowP10MDL_ticketPv+0x74)[0x7fde88dfdfa4]
      /usr/sbin/mysqld(+0x5057cd)[0x7fde8f4e77cd]
      /usr/sbin/mysqld(lf_hash_iterate+0x193)[0x7fde8fa6ec03]
      /usr/sbin/mysqld(_Z11mdl_iteratePFiP10MDL_ticketPvES1_+0x8f)[0x7fde8f4e7a6f]
      /usr/lib/mysql/plugin/metadata_lock_info.so(_Z33i_s_metadata_lock_info_fill_tableP3THDP10TABLE_LISTP4Item+0x27)[0x7fde88dfe197]
      /usr/sbin/mysqld(_Z24get_schema_tables_resultP4JOIN23enum_schema_table_state+0x28e)[0x7fde8f48222e]
      /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x6bd)[0x7fde8f46885d]
      /usr/sbin/mysqld(_ZN4JOIN4execEv+0x54)[0x7fde8f46ace4]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x112)[0x7fde8f467392]
      /usr/sbin/mysqld(_Z18mysql_derived_fillP3THDP3LEXP10TABLE_LIST+0x11b)[0x7fde8f3ef86b]
      /usr/sbin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0xe4)[0x7fde8f3f0b74]
      /usr/sbin/mysqld(_ZN13st_join_table12preread_initEv+0xcf)[0x7fde8f44b0df]
      /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x2f8)[0x7fde8f44b568]
      /usr/sbin/mysqld(+0x476fed)[0x7fde8f458fed]
      /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xb4c)[0x7fde8f468cec]
      /usr/sbin/mysqld(_ZN4JOIN4execEv+0x54)[0x7fde8f46ace4]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x112)[0x7fde8f467392]
      /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x245)[0x7fde8f467e75]
      /usr/sbin/mysqld(+0x428351)[0x7fde8f40a351]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x61e4)[0x7fde8f416854]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x7fde8f419fed]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x7fde8f41d330]
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7fde8f41dae9]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x7fde8f4e10fa]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x7fde8f4e12d0]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fde8dae0182]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fde8d20347d]
      

      My.cnf files can be found in my other issues about other crashes.

      We are monitoring the server status with this query:

      SELECT
        `information_schema`.`metadata_lock_info`.`THREAD_ID`     AS `THREAD_ID`,
        CONCAT(`information_schema`.`processlist`.`USER`,'@',`information_schema`.`processlist`.`HOST`) AS `USER`,
        REPLACE(`information_schema`.`processlist`.`INFO`,'\n',' ') AS `QUERY`,
        `information_schema`.`metadata_lock_info`.`LOCK_MODE`     AS `LOCK_MODE`,
        `information_schema`.`metadata_lock_info`.`LOCK_TYPE`     AS `LOCK_TYPE`,
        GROUP_CONCAT(CONCAT(`information_schema`.`metadata_lock_info`.`TABLE_SCHEMA`,'.',`information_schema`.`metadata_lock_info`.`TABLE_NAME`) SEPARATOR ',') AS `LOCK_TABLES`,
        `information_schema`.`metadata_lock_info`.`LOCK_DURATION` AS `LOCK_DURATION`,
        ROUND((`information_schema`.`processlist`.`TIME_MS` / 1000),2) AS `DURATION`
      FROM (`information_schema`.`metadata_lock_info`
         JOIN `information_schema`.`processlist`
           ON ((`information_schema`.`metadata_lock_info`.`THREAD_ID` = `information_schema`.`processlist`.`ID`)))
      GROUP BY `information_schema`.`metadata_lock_info`.`THREAD_ID`,`information_schema`.`metadata_lock_info`.`LOCK_MODE`,`information_schema`.`metadata_lock_info`.`LOCK_TYPE`$$
      

      But the server didn't crash when I was executing this query, so it might be something else. Come to think of it, there might have been a very small network hiccup. (Which is why it crashed on an addr_resolve)

      Attachments

        Issue Links

          Activity

            People

              svoj Sergey Vojtovich
              CrewOne Sander Pilon
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.