Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18866

Chewing Through Swap

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.2.14
    • 10.2
    • OTHER
    • None
    • Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

    Description

      I'm running 10.2.14, with roughly 300GB data (1000K +/- tables). 95% tables are innodb. I have 64GB RAM, with INNODB buffer pool size set to 46GB (full my.cnf attached). The OS is Ubuntu 16.04.4. This is a dedicated MariaDB server.

      I have swappiness set to 0.

      I'm suffering from MariaDB 10.2.14 ripping threw allocated memory and overflowing into swap. innodb_numa_interleave=1 did not solve the problem. Actually, the problem has become even more aggressive recently. Ever since enabling full text search on thousands of tables, MariaDB has become way more aggressive in over consuming memory. I'm faced with either hitting OOM Killer, feeding it by creating more swap, or restarting MariaDB every other day. Of course, all three scenarios are unsustainable.

      Not sure if this is related to to indicated issues, or where to take this issue from here.

      As you can see in on Mar 7, I grab the following numa maps summary to see how unbalanced it might be when swap was eaten up:

      N0        :      7861090 ( 29.99 GB)
      N1        :      5629622 ( 21.48 GB)
      active    :     10546055 ( 40.23 GB)
      anon      :     13441950 ( 51.28 GB)
      dirty     :     13176616 ( 50.26 GB)
      kernelpagesize_kB:         3628 (  0.01 GB)
      mapmax    :          244 (  0.00 GB)
      mapped    :        66876 (  0.26 GB)
      swapcache :       265334 (  1.01 GB)
      
      

      Thanks,

      Mike

      Attachments

        1. available-mem.png
          available-mem.png
          36 kB
        2. feeding-swap.png
          feeding-swap.png
          23 kB
        3. globalstatus.txt
          57 kB
        4. globalvars.txt
          75 kB
        5. innodbstatus.txt
          12 kB
        6. my.cnf
          2 kB
        7. Selection_144.png
          Selection_144.png
          34 kB
        8. Selection_145.png
          Selection_145.png
          31 kB
        9. swap.png
          swap.png
          64 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              michaelcaplan Michael Caplan
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.