Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18866

Chewing Through Swap

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Stalled (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.2.14
    • Fix Version/s: 10.2
    • Component/s: OTHER
    • Labels:
      None
    • Environment:
      Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

      Description

      I'm running 10.2.14, with roughly 300GB data (1000K +/- tables). 95% tables are innodb. I have 64GB RAM, with INNODB buffer pool size set to 46GB (full my.cnf attached). The OS is Ubuntu 16.04.4. This is a dedicated MariaDB server.

      I have swappiness set to 0.

      I'm suffering from MariaDB 10.2.14 ripping threw allocated memory and overflowing into swap. innodb_numa_interleave=1 did not solve the problem. Actually, the problem has become even more aggressive recently. Ever since enabling full text search on thousands of tables, MariaDB has become way more aggressive in over consuming memory. I'm faced with either hitting OOM Killer, feeding it by creating more swap, or restarting MariaDB every other day. Of course, all three scenarios are unsustainable.

      Not sure if this is related to to indicated issues, or where to take this issue from here.

      As you can see in on Mar 7, I grab the following numa maps summary to see how unbalanced it might be when swap was eaten up:

      N0        :      7861090 ( 29.99 GB)
      N1        :      5629622 ( 21.48 GB)
      active    :     10546055 ( 40.23 GB)
      anon      :     13441950 ( 51.28 GB)
      dirty     :     13176616 ( 50.26 GB)
      kernelpagesize_kB:         3628 (  0.01 GB)
      mapmax    :          244 (  0.00 GB)
      mapped    :        66876 (  0.26 GB)
      swapcache :       265334 (  1.01 GB)
      
      

      Thanks,

      Mike

        Attachments

        1. available-mem.png
          available-mem.png
          36 kB
        2. feeding-swap.png
          feeding-swap.png
          23 kB
        3. globalstatus.txt
          57 kB
        4. globalvars.txt
          75 kB
        5. innodbstatus.txt
          12 kB
        6. my.cnf
          2 kB
        7. Selection_144.png
          Selection_144.png
          34 kB
        8. Selection_145.png
          Selection_145.png
          31 kB
        9. swap.png
          swap.png
          64 kB

          Issue Links

            Activity

              People

              Assignee:
              svoj Sergey Vojtovich
              Reporter:
              michaelcaplan Michael Caplan
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated: