Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8965

cluster node (async replication slave) high memory usage

Details

    Description

      We are seeing high memory usage on cluster member running as async replication slave node, whereas other cluster member's memory usage is normal.

      – Cluster member asynchronous replication slave node

      {{tprod-dba-test-sa-03:~$ top -n5 | grep mysql
      32021 mysql 20 0 6161m 5.3g 115m S 3 67.4 562:46.43 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 1 67.4 562:46.47 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 10 67.4 562:46.77 massif-amd64-li
      32021 mysql 20 0 6161m 5.3g 115m S 5 67.4 562:46.93 massif-amd64-li}}

      — Cluster member normal node

      {{itprod-dba-test-sa-02:~$ top -n5 | grep mysql
      8707 mysql 20 0 2115m 1.4g 137m S 2 18.4 350:05.71 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 3 18.4 350:05.79 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 1 18.4 350:05.82 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 1 18.4 350:05.84 massif-amd64-li
      8707 mysql 20 0 2115m 1.4g 137m S 2 18.4 350:05.89 massif-amd64-li}}

      I profiled memory usage of 2x nodes cluster using Valgrind massif tool:
      https://www.percona.com/blog/2013/01/09/profiling-mysql-memory-usage-with-valgrind-massif/

      ms_print outputs are attached:
      1. massif_cluster_normal_node_.out
      2. massif_cluster_asych_slave_node.out

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9162 MariaDB Galera Cluster memory leak on async slave node

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            aftab.khan aftab khan created issue -
            aftab.khan aftab khan added a comment -

            memory usage on cluster member (running as async replication slave) steadily increase and then server starts swapping, so after almost 2 weeks or so we have to restart mysql to fix this issue.

            aftab.khan aftab khan added a comment - memory usage on cluster member (running as async replication slave) steadily increase and then server starts swapping, so after almost 2 weeks or so we have to restart mysql to fix this issue.
            elenst Elena Stepanova added a comment -

            Thanks for the report.

            nirbhay_c, at the very first glance it looks like the problem might be related to Galera/wsrep, could you please check it out? If it turns out to be unrelated, please reassign to Jan as InnoDB will be the next suspect.

            elenst Elena Stepanova added a comment - Thanks for the report. nirbhay_c , at the very first glance it looks like the problem might be related to Galera/wsrep, could you please check it out? If it turns out to be unrelated, please reassign to Jan as InnoDB will be the next suspect.
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Component/s Galera [ 10124 ]
            Component/s Storage Engine - InnoDB [ 10129 ]
            Component/s wsrep [ 11500 ]
            Assignee Nirbhay Choubey [ nirbhay_c ]
            aftab.khan aftab khan made changes -
            nirbhay_c Nirbhay Choubey (Inactive) made changes -
            Affects Version/s 10.0.21-galera [ 19501 ]
            Affects Version/s 10.0.21 [ 19406 ]
            nirbhay_c Nirbhay Choubey (Inactive) added a comment -

            https://github.com/MariaDB/server/commit/18173ddfc4081407832d9a6703d1b8356b7defe9

            nirbhay_c Nirbhay Choubey (Inactive) added a comment - https://github.com/MariaDB/server/commit/18173ddfc4081407832d9a6703d1b8356b7defe9
            nirbhay_c Nirbhay Choubey (Inactive) made changes -
            Fix Version/s 10.0.23-galera [ 20600 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 72200 ] MariaDB v4 [ 149726 ]

            People

              nirbhay_c Nirbhay Choubey (Inactive)
              aftab.khan aftab khan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.