Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-5673

Crash while parallel dropping multiple tables under heavy load

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Cannot Reproduce
    • 5.5.34, 5.5.35
    • 5.5.39
    • None
    • None
    • SW: Centos 6.5 x86_64 + Official MariaDB repo HW: 2xIntel(R) Xeon(R) CPU E5-2430, 64GB RAM, 4x2GB 7200RPM SATA Disk in RAID10

    Description

      When the crash happens, there are two sets of queries running on the server.

      One set is populating Business Intelligence aggregate tables, the other is refreshing a development copy of a database, using parallel loading.

      The two sets of queries are running on different, independent databases.

      The crash does not occur when only one set of queries (e.g. only the db load queries, or only the BI aggregator queries ) are running.

      The tables that I try to drop are in compressed format.

      The system is IO bound when the crash occurs (iostat consistently shows 100% disk utilization), but not CPU bound.

      The crash occurs when I try to drop several big (multi gigabyte) tables in parallel, as you can see in the attached error log.

      Attachments

        3. server.cnf
          1 kB
          Tóth István

        Activity

          Ascending order - Click to sort in descending order
          View 10 older comments
          elenst Elena Stepanova added a comment -

          Long semaphore waits for Jan's expert reading (logs are attached).

          elenst Elena Stepanova added a comment - Long semaphore waits for Jan's expert reading (logs are attached).
          stoty Tóth István added a comment -

          FYI, I've just had a very similar crash on Oracle Mysql 5.6.17. I can attach the logs for it if it is interesting to you.

          stoty Tóth István added a comment - FYI, I've just had a very similar crash on Oracle Mysql 5.6.17. I can attach the logs for it if it is interesting to you.
          jplindst Jan Lindström (Inactive) added a comment -

          Hi,

          Why they are running these huge inserts like:

          ---TRANSACTION 705FC, ACTIVE 1492 sec
          mysql tables in use 9, locked 9
          69176 lock struct(s), heap size 6175160, 3804151 row lock(s)

          If these inserts can't be splitted, you need to increase the long semaphore wait to higher number.

          R: Jan

          jplindst Jan Lindström (Inactive) added a comment - Hi, Why they are running these huge inserts like: ---TRANSACTION 705FC, ACTIVE 1492 sec mysql tables in use 9, locked 9 69176 lock struct(s), heap size 6175160, 3804151 row lock(s) If these inserts can't be splitted, you need to increase the long semaphore wait to higher number. R: Jan
          stoty Tóth István added a comment -

          These are the queries that transform the raw data to the star schema required by the BI software. It may be possible to re-write them, but since mysql 5.5, and MariaDB 10.x does not exhibit the crash under the same circumstances, they are probably triggering a bug specific to mysql 5.6 (sometimes crashes) and MariaDB 5.5 (crashes every time).

          However, accoring to our tests the inserts in themselves are not the problem, the crash only happens when they are running in parallel with the drop table statements.

          stoty Tóth István added a comment - These are the queries that transform the raw data to the star schema required by the BI software. It may be possible to re-write them, but since mysql 5.5, and MariaDB 10.x does not exhibit the crash under the same circumstances, they are probably triggering a bug specific to mysql 5.6 (sometimes crashes) and MariaDB 5.5 (crashes every time). However, accoring to our tests the inserts in themselves are not the problem, the crash only happens when they are running in parallel with the drop table statements.
          jplindst Jan Lindström (Inactive) added a comment -

          revno: 4230
          committer: Jan Lindström <jplindst@mariadb.org>
          branch nick: 5.5
          timestamp: Wed 2014-07-23 09:04:59 +0300
          message:
          MDEV-5673: Crash while parallel dropping multiple tables under heavy load

          Improve long semaphore wait output to include all semaphore waits
          and try to find out if there is a sequence of waiters.

          jplindst Jan Lindström (Inactive) added a comment - revno: 4230 committer: Jan Lindström <jplindst@mariadb.org> branch nick: 5.5 timestamp: Wed 2014-07-23 09:04:59 +0300 message: MDEV-5673 : Crash while parallel dropping multiple tables under heavy load Improve long semaphore wait output to include all semaphore waits and try to find out if there is a sequence of waiters.

          People

            jplindst Jan Lindström (Inactive)
            stoty Tóth István
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.