Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10843

XtraDB Semaphore Stalls with innodb_use_mtflush enabled

    XMLWordPrintable

    Details

      Description

      I have 3 servers in production all recently upgraded to MariaDB 10.1 to, among other things, make use of InnoDB Page Compression.

      All 3 use XFS on the data partition, over the top of hardware RAID10 with SSDs.

      CentOS boxes are 16 cores/32 threads, 256GB RAM
      Gentoo box is 20 cores/40 threads, 256GB RAM

      I periodically need to run through and OPTIMIZE several of our InnoDB tables. Some due to heavy delete operations in production, others, if I need to restore from snapshot, tar doesn't preserve the sparseness of the InnoDB files so I run an optimize across them to get the space back.

      However, when running a mass optimize, I kept running into hangs, with InnoDB bitching about semaphore locks etc in the error log.

      I've since discovered if I disable innodb_use_mtflush then the stall doesn't occur, at all, across the entire DB (there's a few thousand tables using page compression).

      I've attached an innodb status from when the semaphore hang occurs.

      I've experimented with various values for adaptive hash index partitions, innodb thread concurrency, io read and write threads etc. The only thing that appears to make the problem happen or go away, is whether mtflush is enabled or not.

      Apologies if mtflush is ONLY for FusionIO but it isn't readily apparent that you shouldn't, only that it was created to improve performance with FusionIO.

        Attachments

        1. innodb-status
          39 kB
        2. mysql-status
          97 kB
        3. mysql-variables
          19 kB

          Issue Links

            Activity

              People

              Assignee:
              jplindst Jan Lindström
              Reporter:
              Ninpo Alex Boag-Munroe
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Git Integration