Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12205

MariaDB 10.1.21 Per-query timeout prefixes leak memory (confirmed)

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.1.21
    • None
    • None
    • None
    • OS: CentOS 7
      Physical/Virtual: Can reproduce on both vm and hardware, but primarily on physical. 1TB memory, dual 12x core CPU.

    Description

      I have a 720GB database, 808GB innodb_buffer_cache, 1TB system memory and 10 tables. For all I can figure we shouldn’t be running out of memory, however, the mysqld process steadily increases in memory consumption until OOM killer rightfully claims it (we now restart it before that happens). After finding some relevant memory leak fixes released in 10.1.21 we upgraded to that from 10.1.13, but unfortunately it didn’t help.

      Running mysqltuner.pl (https://github.com/major/MySQLTuner-perl) seems to agree that we shouldn’t be using more than 809 GB.

      mysqltuner snippet

      [--] Total buffers: 806.4G global + 2.9M per thread (1000 max threads)
      [OK] Maximum reached memory usage: 808.2G (80.21% of installed RAM)
      [OK] Maximum possible memory usage: 809.2G (80.31% of installed RAM)
      

      At the time this data was gathered, the mysqld process was at 980GB. (gathered with `ps -eo pid,comm,rss`), leaving the available system memory at 17GB.

      I can consistently reproduce this with similar workload on a scaled down test instance.

      I have attached a binlog from my test instance, show variables, and a few grafana graphs.

      Thanks for taking a look!

      Attachments

        1. binlog.000099.gz
          6.30 MB
          Andrew Gabbitas
        2. master_db_memory.png
          42 kB
          Andrew Gabbitas
        3. master_db_mysqld_memory.png
          48 kB
          Andrew Gabbitas
        4. show_variables.csv
          17 kB
          Andrew Gabbitas

        Activity

          People

            elenst Elena Stepanova
            andygabby Andrew Gabbitas
            Votes:
            2 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.