Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10499

Extreme memory use when rebuilding a large table

    XMLWordPrintable

Details

    Description

      When rebuilding a very large table as a result of executing DDL, eg:

      alter table <table> add column boris int(11) null;
      

      it gets to this point with very little memory use:

      2016-08-04 13:17:46 139771367865088 [Note] InnoDB: Online DDL : Start building index `PRIMARY` (1 / 4), estimated cost : 22.1238
      

      At which point it begins to consume all the available ram and then all the available swap and if there isn't enough it gets OOM'd by the kernel.

      We have run this test many times with a 160GB table on nodes that have 64GB ram and no swap and it exhausted all of it and got OOM'd by the kernel. We succeeded once with an innodb buffer pool lowered from 50GB to 16GB and a 16GB swap volume added, but attempts to repeat it failed. It does succeed if we add a really big swap volume, for example one of the size of the data length of the table which is around ~90GB.

      It therefore seems as if rebuilds of the primary index (which includes all of the table data) are being done purely in-memory rather than on-disk.

      Attachments

        Activity

          People

            Unassigned Unassigned
            m4rkw Mark Wadham
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.