Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6455

MariaDB/TokuDB crash during "join update/set" operation

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 10.0.12
    • 10.0.14
    • CentOS 6.5 running in Citrix Xen VM, 16 GB RAM, 4 CPU cores, 200GB HDD

    Description

      During a somewhat big "join update/set" operation, MariaDB/TokuDB crashes for unknown reason.
      While watching the processlist, the initial join-query succeeds with some million rows; but the update operation causes the crash after some 100.000s updates.
      Here is information from the mysqld.log file:

      /home/buildbot/buildbot/build/mariadb-10.0.12/storage/tokudb/ft-index/ft/ft-flusher.cc:1949 toku_ft_flush_node_on_background_thread: Assertion `toku_bnc_n_entries(BNC(parent, childnum))>0' failed (errno=0)
      : Success
      Backtrace: (Note: toku_do_assert=0x0x7fa91e8791a0)
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fa91e876b2b]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c53)[0x7fa91e876c53]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0x71ccd)[0x7fa91e876ccd]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z39toku_ft_flush_node_on_background_threadP2ftP6ftnode+0x270)[0x7fa91e902490]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0xfd610)[0x7fa91e902610]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0x109430)[0x7fa91e90e430]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0xdfcda)[0x7fa91e8e4cda]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0xe0268)[0x7fa91e8e5268]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z20toku_ft_root_put_msgP2ftP6ft_msgP11txn_gc_info+0x531)[0x7fa91e8e6081]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z19toku_ft_send_insertP9ft_handleP10__toku_dbtS2_P6xids_t11ft_msg_typeP11txn_gc_info+0x2f)[0x7fa91e8e718f]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z20toku_ft_maybe_insertP9ft_handleP10__toku_dbtS2_P7tokutxnb10__toku_lsnb11ft_msg_type+0x1ce)[0x7fa91e9381ae]
      /usr/lib64/mysql/plugin/ha_tokudb.so(+0x45e51)[0x7fa91e84ae51]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_Z19env_update_multipleP13__toku_db_envP9__toku_dbP13__toku_db_txnP10__toku_dbtS6_S6_S6_jPS2_PjjP9DBT_ARRAYjSA_+0xeb8)
      [0x7fa91e9315a8]
      /usr/lib64/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb10update_rowEPKhPh+0x510)[0x7fa91e869c10]
      /usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x162)[0x72eb32]
      /usr/sbin/mysqld(_ZN12multi_update10do_updatesEv+0x63f)[0x66562f]
      /usr/sbin/mysqld(_ZN12multi_update8send_eofEv+0x288)[0x6659d8]
      /usr/sbin/mysqld[0x60c74e]
      /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa3d)[0x61f5cd]
      /usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x621441]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1dd)
      [0x61e00d]
      /usr/sbin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lexPP12multi_update+0x1ba
      )[0x664e5a]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4667)[0x5d3a27]
      /usr/sbin/mysqld[0x5d5b12]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1b20)[0x5d7cd0]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x6950e3]
      /usr/sbin/mysqld(handle_one_connection+0x42)[0x6951b2]
      /lib64/libpthread.so.0[0x3b530079d1]
      /lib64/libc.so.6(clone+0x6d)[0x3b52ce8b5d]

      Attachments

        Activity

          rtgman Thomas Trede added a comment -

          Hello Elena,
          for the time being, I prefer to keep the coredumps private.
          Thank you & kind regards,
          Thomas

          rtgman Thomas Trede added a comment - Hello Elena, for the time being, I prefer to keep the coredumps private. Thank you & kind regards, Thomas
          prohaska7 Rich Prohaska added a comment -

          Thanks for getting the core files. We examined them and found the bug in the TokuDB fractal tree indexing code. We are tracking the fix in Tokutek's bug tracking system as https://tokutek.atlassian.net/browse/FT-590.

          prohaska7 Rich Prohaska added a comment - Thanks for getting the core files. We examined them and found the bug in the TokuDB fractal tree indexing code. We are tracking the fix in Tokutek's bug tracking system as https://tokutek.atlassian.net/browse/FT-590 .
          rtgman Thomas Trede added a comment -

          Hello Rich,
          fine, that you have found the root-cause/bug for this issue.
          Now in the Tokutek bug tracking, they tell, that this will be fixed in tokudb 7.5.0.
          In MariaDB 10 we are somewhere at tokudb-version 7.1.5; when will we see this fix in MariaDB?
          It looks like, that this is very, very far away...
          Thank you & kind regards,
          Thomas

          rtgman Thomas Trede added a comment - Hello Rich, fine, that you have found the root-cause/bug for this issue. Now in the Tokutek bug tracking, they tell, that this will be fixed in tokudb 7.5.0. In MariaDB 10 we are somewhere at tokudb-version 7.1.5; when will we see this fix in MariaDB? It looks like, that this is very, very far away... Thank you & kind regards, Thomas
          prohaska7 Rich Prohaska added a comment -

          TokuDB 7.5.0 will be available in the middle of sept from Tokutek and I expect that it will be pulled into Mariadb 10 sometime after that.

          If you want to experiment on your own, the patch to the tokudb source is very simple. See the commit here: https://tokutek.atlassian.net/browse/FT-590

          prohaska7 Rich Prohaska added a comment - TokuDB 7.5.0 will be available in the middle of sept from Tokutek and I expect that it will be pulled into Mariadb 10 sometime after that. If you want to experiment on your own, the patch to the tokudb source is very simple. See the commit here: https://tokutek.atlassian.net/browse/FT-590
          prohaska7 Rich Prohaska added a comment -

          Fixed 10.0.14

          prohaska7 Rich Prohaska added a comment - Fixed 10.0.14

          People

            Unassigned Unassigned
            rtgman Thomas Trede
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.