Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-5343

Jmalloc profiling

Details

    • Task
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 6.3.1
    • 23.02.2
    • None
    • None
    • SkySQL AWS 32x 128 single node

    Description

      tools to track memory

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MCOL-5285 SkySQL OOM Crash? Memory not being released? testing

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            alexey.vorovich alexey vorovich (Inactive) created issue -
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Field Original Value New Value
            Description Currently theres a customer whose memory appears not to be released in skysql.

            The current work around is that RDBA/ SRE has to manually mcsShutdown and mcsStart every couple days. However the customer often has to file a ticket saying its crashed and to restart before the scheduled stop/start to clear memory.

            *Link to Logs & Configs in comment below*             		tools to track memory
            alexey.vorovich alexey vorovich (Inactive) made changes -
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked higher
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked higher
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked lower
            Andrey Andrey Piskunov (Inactive) added a comment -

            The heap reports are being generated in `/var/run/columnstore/heapprofs/` and will be sent to s3 as well as their svg versions processed by jeprof.

            Andrey Andrey Piskunov (Inactive) added a comment - The heap reports are being generated in `/var/run/columnstore/heapprofs/` and will be sent to s3 as well as their svg versions processed by jeprof.
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            alexey.vorovich alexey vorovich (Inactive) added a comment - - edited

            action plan per leonid.fedorov
            -implement singe static link binary in drone CI
            -discuss same with MDEV for jenkins CI
            -enable jmalloc profiling in drone
            -keep it disabled in Jenkins

            Discuss with Sky possible experiments to catch the leak

            -install binary on the customer k8s sky pod
            vs
            -create new image and restart the statefull set with that

            alexey.vorovich alexey vorovich (Inactive) added a comment - - edited action plan per leonid.fedorov -implement singe static link binary in drone CI -discuss same with MDEV for jenkins CI -enable jmalloc profiling in drone -keep it disabled in Jenkins Discuss with Sky possible experiments to catch the leak -install binary on the customer k8s sky pod vs -create new image and restart the statefull set with that
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked lower
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Fix Version/s 23.02.1 [ 28701 ]
            Fix Version/s 23.02 [ 28209 ]
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Issue Type Bug [ 1 ] Task [ 3 ]
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked lower
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked higher
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked higher
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Fix Version/s 23.02 [ 28209 ]
            Fix Version/s 23.02.1 [ 28701 ]
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Fix Version/s 23.02.2 [ 28713 ]
            Fix Version/s 23.02 [ 28209 ]
            alexey.vorovich alexey vorovich (Inactive) made changes -
            Component/s advanced tools [ 13515 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            leonid.fedorov Leonid Fedorov made changes -
            Component/s advanced tools [ 13515 ]
            julien.fritsch Julien Fritsch made changes -
            Labels mcs_cs_datamesh

            People

              leonid.fedorov Leonid Fedorov
              allen.herrera Allen Herrera
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.