Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-40178

OOM at secondary node of replication because of growing RSS

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.4, 11.8, 12.3
    • None
    • None
    • ubuntun 24 LTS

    Description

      get source code:
      #git clone -b bb-11.4-duckdb-jemalloc https://github.com/drrtuy/mdb-server.git
      #cd mdb-server
      #git clone --recurse-submodules https://github.com/MariaDB/duckdb-engine storage/duckdb/duckdb
      #./storage/duckdb/duckdb/build.sh -D

      (if you hit cryto related issue, hack
      #sed -i 's/libxcrypt-dev/libcrypt-dev/g' /test-data/clones/mdb-server/storage/duckdb/duckdb/build.sh)

      ./storage/duckdb/duckdb/build.sh -S -t RelWithDebInfo or refer ./storage/duckdb/duckdb/build.sh -h

      got to build folder and pack build
      #cd /test-data/clones/DuckdbBuildOf_mdb-server
      #cpack

      refer: https://github.com/MariaDB/duckdb-engine – for building

      1. setup 2 node async replication - 1 primary and 1 secondary
      2. use sysbench-lua BMK (http://dimitrik.free.fr/blog/posts/mysql-perf-bmk-kit.html) , the reason I have taken BMK version of sysbench-lua and not https://github.com/akopytov/sysbench because we have 3 phases, table creations(create), data loading (prepare) and run
      3. point primary and 1st run BMK's create , and then alter tables to have storage=duckdb
      4. Run populate data ( prepare) and this will replicate data to secondary. Get highest gtid and wait on replica until this highest GTID replicated
      5. stop sql_thread in secondary and the run sysbench test (update_key/nokey/write-only ) without autoincreament. pointing to primary and watch highest GTID of primary reached to relay-log of secondary
      6. start sql_thread and you will see RSS growing and after 3/5 min OOM happened and you can check OOM ( dmesg -T |grep -i oom | tail -10)

      Attaching all scripts

      Attachments

        1. _run.sh
          0.6 kB
        2. alter-sysbench-duckdb.sql
          0.9 kB
        3. config-settings.sql
          1 kB
        4. duck-udf.sql
          0.1 kB
        5. prepare.sh
          2 kB
        6. run-key-notxn.sh
          0.6 kB
        7. run-key-txn.sh
          0.6 kB
        8. run-write-only-txn.sh
          0.6 kB
        9. setup-prim.sh
          1.0 kB
        10. setup-sec.sh
          1 kB
        11. start-all-monitor.sh
          0.0 kB
        12. start-sql-monitor.sh
          0.7 kB
        13. start-top-monitor.sh
          1 kB
        14. stop-all-monitor.sh
          0.1 kB
        15. stop-sql-monitor.sh
          0.3 kB
        16. stop-top-monitor.sh
          0.0 kB
        17. test-replica-key.sh
          0.3 kB
        18. test-simulation-key-notxn.sh
          1 kB
        19. test-simulation-key-txn-heap.sh
          1 kB
        20. test-simulation-key-txn.sh
          1 kB
        21. test-simulation-write-only.sh
          1 kB
        22. users-prim.sql
          1 kB
        23. worker.sql
          0.3 kB
        24. replica-worker.ods
          40 kB
        25. replica-memory-mariadb-duck.ods
          108 kB
        26. flame-graph-leak.png
          flame-graph-leak.png
          164 kB
        27. flame-temp-allocation.png
          flame-temp-allocation.png
          435 kB
        28. flrame-allocation.png
          flrame-allocation.png
          397 kB
        29. heaptrack-allocation.jpeg
          heaptrack-allocation.jpeg
          241 kB
        30. heaptrack-consumed.jpeg
          heaptrack-consumed.jpeg
          254 kB
        31. heaptrack-tmp-allocation.jpeg
          heaptrack-tmp-allocation.jpeg
          283 kB
        32. summary.png
          summary.png
          336 kB

        Activity

          People

            drrtuy Roman
            amroo76 Amrendra Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.