Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8965

cluster node (async replication slave) high memory usage

Details

    Description

      We are seeing high memory usage on cluster member running as async replication slave node, whereas other cluster member's memory usage is normal.

      – Cluster member asynchronous replication slave node

      {{tprod-dba-test-sa-03:~$ top -n5 | grep mysql
      32021 mysql 20 0 6161m 5.3g 115m S 3 67.4 562:46.43 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 1 67.4 562:46.47 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 10 67.4 562:46.77 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 5 67.4 562:46.93 massif-amd64-li}}

      — Cluster member normal node

      {{itprod-dba-test-sa-02:~$ top -n5 | grep mysql
      8707 mysql 20 0 2115m 1.4g 137m S 2 18.4 350:05.71 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 3 18.4 350:05.79 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 1 18.4 350:05.82 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 1 18.4 350:05.84 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 2 18.4 350:05.89 massif-amd64-li}}

      I profiled memory usage of 2x nodes cluster using Valgrind massif tool:
      https://www.percona.com/blog/2013/01/09/profiling-mysql-memory-usage-with-valgrind-massif/

      ms_print outputs are attached:
      1. massif_cluster_normal_node_.out
      2. massif_cluster_asych_slave_node.out

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9162 MariaDB Galera Cluster memory leak on async slave node

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            aftab.khan aftab khan added a comment -

            memory usage on cluster member (running as async replication slave) steadily increase and then server starts swapping, so after almost 2 weeks or so we have to restart mysql to fix this issue.

            aftab.khan aftab khan added a comment - memory usage on cluster member (running as async replication slave) steadily increase and then server starts swapping, so after almost 2 weeks or so we have to restart mysql to fix this issue.
            elenst Elena Stepanova added a comment -

            Thanks for the report.

            nirbhay_c, at the very first glance it looks like the problem might be related to Galera/wsrep, could you please check it out? If it turns out to be unrelated, please reassign to Jan as InnoDB will be the next suspect.

            elenst Elena Stepanova added a comment - Thanks for the report. nirbhay_c , at the very first glance it looks like the problem might be related to Galera/wsrep, could you please check it out? If it turns out to be unrelated, please reassign to Jan as InnoDB will be the next suspect.
            nirbhay_c Nirbhay Choubey (Inactive) added a comment -

            https://github.com/MariaDB/server/commit/18173ddfc4081407832d9a6703d1b8356b7defe9

            nirbhay_c Nirbhay Choubey (Inactive) added a comment - https://github.com/MariaDB/server/commit/18173ddfc4081407832d9a6703d1b8356b7defe9

            People

              nirbhay_c Nirbhay Choubey (Inactive)
              aftab.khan aftab khan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.