Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-4043

Memory leaks Part 1. & errors -> need a valgrind pass

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 1.4.4, 1.5.3
    • 1.5.3
    • ExeMgr
    • None

    Description

      I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

      I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

      At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

      Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 makes ExeMgr allocate around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.

      When a particular patch merges easily into 1.2, be sure to include it.

      Attachments

        Issue Links

          Activity

            pleblanc Patrick LeBlanc (Inactive) created issue -
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Field Original Value New Value
            Assignee Gagan Goel [ tntnatbry ]
            tntnatbry Gagan Goel (Inactive) made changes -
            Attachment exemgr_valgrind.log [ 52270 ]
            tntnatbry Gagan Goel (Inactive) made changes -
            Fix Version/s 1.5 [ 22800 ]
            Fix Version/s 1.4 [ 22101 ]
            tntnatbry Gagan Goel (Inactive) made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            tntnatbry Gagan Goel (Inactive) made changes -
            Assignee Gagan Goel [ tntnatbry ] Patrick LeBlanc [ pleblanc ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Description I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.
            I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 allocates around 1GB/min that it doesn't use.
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Description I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 allocates around 1GB/min that it doesn't use.
            I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 allocates around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Description I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 allocates around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.
            I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 makes ExeMgr allocate around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Description I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 makes ExeMgr allocate around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.
            I ran our regression test on our 1.5 RC, and at the end of it, ExeMgr was using 45GB of virt mem, and about 2.6GB of real mem when it was idle. Gagan compared 1.4 & 1.5, found mem usage to be similar. Provoked an investigation.

            I attached the valgrind log from running VG on ExeMgr, on our latest 1.4.4 RC (enterprise server @ 6c7e73d7, engine @ 8ab3cc301). For this test I ran all of working_tpch1, and working_tpch1_compareLogOnly, except for MCOL_3503.sql. moda() has leaks not recorded in this log because I found that early and commented out the test that uses it the most to see how much of a difference it made in the final mem usage.

            At the end of that test, ExeMgr was using around 1.2GB when idle. The leaks reported in the log do not add up to anywhere close to 1.2GB. My suspicion is that all of these little bits of mem over time are causing a good deal of mem fragmentation. It could also be that a good deal of mem still has a reference, so VG doesn't consider it leaked. Need to start fixing these little things we have in the log, then reevaluate.

            Jun 22 update: Testing @ the columnstore-1.5.2-1 tag, I noticed test211 makes ExeMgr allocate around 1GB/min that it doesn't use. To be clear, the mem usage shows up as virtual mem usage, not resident mem usage. Resident mem usage also appears to be climbing but with a much lower slope.

            When a particular patch merges easily into 1.2, be sure to include it.
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Assignee Patrick LeBlanc [ pleblanc ] Gagan Goel [ tntnatbry ]
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Status In Review [ 10002 ] Needs Feedback [ 10501 ]
            pleblanc Patrick LeBlanc (Inactive) made changes -
            Status Needs Feedback [ 10501 ] Open [ 1 ]
            toddstoffel Todd Stoffel (Inactive) made changes -
            Rank Ranked higher
            David.Hall David Hall (Inactive) made changes -
            gdorman Gregory Dorman (Inactive) made changes -
            Summary Memory leaks & errors -> need a valgrind pass Memory leaks Part 1. & errors -> need a valgrind pass
            gdorman Gregory Dorman (Inactive) made changes -
            Fix Version/s 1.4 [ 22101 ]
            gdorman Gregory Dorman (Inactive) made changes -
            Fix Version/s 1.5.3 [ 24412 ]
            Fix Version/s 1.5 [ 22800 ]
            gdorman Gregory Dorman (Inactive) made changes -
            Component/s ExeMgr [ 13506 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            toddstoffel Todd Stoffel (Inactive) made changes -
            Affects Version/s 1.5.3 [ 24412 ]
            Affects Version/s 1.5 [ 22800 ]

            People

              tntnatbry Gagan Goel (Inactive)
              pleblanc Patrick LeBlanc (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.