Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29652

data-at-rest enabled high CPU use for long times

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Won't Fix
    • 10.3.36
    • N/A
    • None
    • Oracle Linux Server release 7.9
      CentOS Linux release 7.9.2009

    Description

      Either when creating a new database to have data-at-rest enabled, or adding new tables to database with other data-at-rest tables, or database startup with data-at-rest employed - there are times when all of the data-at-rest background tasks are consuming 80-90% of multiple relatively fast CPU's anywhere from 20 minutes to 6 hours on systems with 32GB RAM.

      There doesn't appear to be any consistent and determinable cause for the overly long utlizations.

      Attachments

        Issue Links

          Activity

            DBrunner Jim Dutton added a comment -

            Have uploaded "MDEV-29652_sample_bundle.tar" which consists of 3 files.

            DBrunner Jim Dutton added a comment - Have uploaded " MDEV-29652 _sample_bundle.tar" which consists of 3 files.
            DBrunner Jim Dutton added a comment - - edited

            The observed MariaDB encryption thread "thrashing" occured under CentOS-7.9 and Oracle EL-7.9.

            The previous notes about 10.5/10.9 were on Oracle EL-9, Alama-9, and/or RHEL-8.6/9.

            The Centos-7.9 DB server has just had its MariaDB instance changed from 10.3.x to 10.5.x and it is not exhibiting the previously noted excessive encryption thread "thrashing". Three databases (one populated, others empty) each with different encryption KeyIDs.

            DBrunner Jim Dutton added a comment - - edited The observed MariaDB encryption thread "thrashing" occured under CentOS-7.9 and Oracle EL-7.9. The previous notes about 10.5/10.9 were on Oracle EL-9, Alama-9, and/or RHEL-8.6/9. The Centos-7.9 DB server has just had its MariaDB instance changed from 10.3.x to 10.5.x and it is not exhibiting the previously noted excessive encryption thread "thrashing". Three databases (one populated, others empty) each with different encryption KeyIDs.

            DBrunner, thank you. The InnoDB I/O was heavily refactored by MDEV-15053, MDEV-15058, MDEV-23399 and MDEV-23855 and a few follow-up tickets in MariaDB Server 10.5.

            What you are observing in MariaDB Server 10.3 might be something similar to MDEV-27466, which was observed in MariaDB Server 10.4. I am reluctant to touch the I/O layer in MariaDB Server before version 10.5 or 10.6. MariaDB Server 10.6 included some more performance improvements, such as simpler buffer page latches (MDEV-24142).

            marko Marko Mäkelä added a comment - DBrunner , thank you. The InnoDB I/O was heavily refactored by MDEV-15053 , MDEV-15058 , MDEV-23399 and MDEV-23855 and a few follow-up tickets in MariaDB Server 10.5. What you are observing in MariaDB Server 10.3 might be something similar to MDEV-27466 , which was observed in MariaDB Server 10.4. I am reluctant to touch the I/O layer in MariaDB Server before version 10.5 or 10.6. MariaDB Server 10.6 included some more performance improvements, such as simpler buffer page latches ( MDEV-24142 ).
            DBrunner Jim Dutton added a comment -

            I have rebuilt my databases on two servers using MariaDB-10.5 without any problems. I have observed system and MariaDB startups and have not noticed any long running encryption threads. I have monitored the database servers as I worked through my application. There were "normal" increases in database activity but at most only a 0.7 percent increase in CPU utilization and for only a few seconds. The two database servers are now running with MariaDB-10.5.17.

            I tested one database server on MariaDB-10.9 and that also had no encryption-thread-thrashing issue.

            Since my application is now running successfully using MariaDB-10.5, I do not intend to (re-) install 10.3 again. Unfortunately for me - I now have the task of upgrading all of my application documentation to reflect the DB changes. Oh well.

            As far as I am concerned, MariaDB-10.5 has relieved my systems and application of the excessive long-running CPU utilization issues. In this case, I consider my issue and this problem solved. (now if I could just disable the various keyboard shorcuts various applications have implemented so as to stop popping up things as I type ....)

            You may close this problem out but I would suggest you do it with a stipulation that a MariaDB-10.3 user/site that exhibits the problems noted here should be "encouraged" to migrate to MariaDB >=10.5.

            DBrunner Jim Dutton added a comment - I have rebuilt my databases on two servers using MariaDB-10.5 without any problems. I have observed system and MariaDB startups and have not noticed any long running encryption threads. I have monitored the database servers as I worked through my application. There were "normal" increases in database activity but at most only a 0.7 percent increase in CPU utilization and for only a few seconds. The two database servers are now running with MariaDB-10.5.17. I tested one database server on MariaDB-10.9 and that also had no encryption-thread-thrashing issue. Since my application is now running successfully using MariaDB-10.5, I do not intend to (re-) install 10.3 again. Unfortunately for me - I now have the task of upgrading all of my application documentation to reflect the DB changes. Oh well. As far as I am concerned, MariaDB-10.5 has relieved my systems and application of the excessive long-running CPU utilization issues. In this case, I consider my issue and this problem solved. (now if I could just disable the various keyboard shorcuts various applications have implemented so as to stop popping up things as I type ....) You may close this problem out but I would suggest you do it with a stipulation that a MariaDB-10.3 user/site that exhibits the problems noted here should be "encouraged" to migrate to MariaDB >=10.5.

            DBrunner, thank you. It is a good idea to upgrade from 10.3 anyway, because it will reach its End of Life (EOL) in May 2023, 5 years after the initial 10.3.7 generally available (GA) release.

            Personally, I would recommend 10.6.10 or later. The MariaDB Server 10.6 release series is the first one where most DDL operations are crash-safe (MDEV-17567).

            marko Marko Mäkelä added a comment - DBrunner , thank you. It is a good idea to upgrade from 10.3 anyway, because it will reach its End of Life (EOL) in May 2023, 5 years after the initial 10.3.7 generally available (GA) release . Personally, I would recommend 10.6.10 or later. The MariaDB Server 10.6 release series is the first one where most DDL operations are crash-safe ( MDEV-17567 ).

            People

              marko Marko Mäkelä
              DBrunner Jim Dutton
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.