Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18866

Chewing Through Swap

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.2.14
    • 10.2
    • OTHER
    • None
    • Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

    Description

      I'm running 10.2.14, with roughly 300GB data (1000K +/- tables). 95% tables are innodb. I have 64GB RAM, with INNODB buffer pool size set to 46GB (full my.cnf attached). The OS is Ubuntu 16.04.4. This is a dedicated MariaDB server.

      I have swappiness set to 0.

      I'm suffering from MariaDB 10.2.14 ripping threw allocated memory and overflowing into swap. innodb_numa_interleave=1 did not solve the problem. Actually, the problem has become even more aggressive recently. Ever since enabling full text search on thousands of tables, MariaDB has become way more aggressive in over consuming memory. I'm faced with either hitting OOM Killer, feeding it by creating more swap, or restarting MariaDB every other day. Of course, all three scenarios are unsustainable.

      Not sure if this is related to to indicated issues, or where to take this issue from here.

      As you can see in on Mar 7, I grab the following numa maps summary to see how unbalanced it might be when swap was eaten up:

      N0        :      7861090 ( 29.99 GB)
      N1        :      5629622 ( 21.48 GB)
      active    :     10546055 ( 40.23 GB)
      anon      :     13441950 ( 51.28 GB)
      dirty     :     13176616 ( 50.26 GB)
      kernelpagesize_kB:         3628 (  0.01 GB)
      mapmax    :          244 (  0.00 GB)
      mapped    :        66876 (  0.26 GB)
      swapcache :       265334 (  1.01 GB)
      
      

      Thanks,

      Mike

      Attachments

        1. available-mem.png
          36 kB
          Michael Caplan
        2. feeding-swap.png
          23 kB
          Michael Caplan
        3. globalstatus.txt
          57 kB
          Michael Caplan
        4. globalvars.txt
          75 kB
          Michael Caplan
        5. innodbstatus.txt
          12 kB
          Michael Caplan
        6. my.cnf
          2 kB
          Michael Caplan
        7. Selection_144.png
          34 kB
          Michael Caplan
        8. Selection_145.png
          31 kB
          Michael Caplan
        9. swap.png
          64 kB
          Michael Caplan

        Issue Links

          Activity

            People

              Unassigned Unassigned
              michaelcaplan Michael Caplan
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.