Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34529

Shrink the system tablespace when system tablespace contains MDEV-30671 leaked undo pages

Details

    Description

      Looks like the fix for MDEV-14795 didn't handle leaked undo pages (because even their extent has never been marked as freed). So MDEV-14795 assumes that it is used by server. This does NOT allow to shrink ibdata1 in a newer versions, even after a slow shutdown and even when UNDO is already moved to separate tablespaces (so instance re-creation is still required if the instance was ever affected by MDEV-30671.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-35289 innodb_fast_shutdown=0 might corrupt the system tablespace on 32-bit systems

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-14795 InnoDB system tablespace cannot be shrunk

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31234 InnoDB does not free UNDO after the fix of MDEV-30671, thus shared tablespace (ibdata1) may grow indefinitely for no good reason

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-35398 Improve shrinking of system tablespace

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            thiru Thirunarayanan Balathandayuthapani added a comment - - edited

            I can repeat the issue.
            Followed the steps in MDEV-31234 in affected version.

            1) started the server with --innodb-immediate-scrub-data-uncompressed=1. --innodb_undo_tablespace=0.
            2) Ran sysbench. did slow shutdown and ended up with 1528 to_purge undo log pages. (leaked undo log corrupt pages)
            Start the latest 10.6 against the data directory, did slow shutdown

            2024-07-04 14:29:50 0 [Note] InnoDB: Truncating system tablespace from 4864 to 2496 pages
            		 
            2024-07-04 14:29:50 0 [Note] InnoDB: System tablespace truncated successfully

            Ran innochecksum tool against shrinked system tablespace and we end up with same

             to_purge Undo page state: 0 active, 128 cached, 1528 to_purge, 0 prepared, 148 other

            So verified what was exist before 2496 page, it was all leaked undo pages and previous extent state has been marked as "XDES_FREE_FRAG" and 12 pages has been used in that extent.

            #::2484         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2485         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2486         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2487         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2488         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2489         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2490         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2491         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2492         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2493         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2494         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -
            		 
            #::2495         |               Undo log page                   |, Will be freed in purge when all undodata in it is removed, -

            MDEV-14795 didn't handle leaked undo pages because even their extent has never been marked as freed. So MDEV-14795 assumes that it is used by server.
            MDEV-14795 finds the last used extent in the system tablespace and truncates the remaining file.
            Attached the ibdata1 of the leaky undo pages for future reference.

            thiru Thirunarayanan Balathandayuthapani added a comment - - edited I can repeat the issue. Followed the steps in MDEV-31234 in affected version. 1) started the server with --innodb-immediate-scrub-data-uncompressed=1. --innodb_undo_tablespace=0. 2) Ran sysbench. did slow shutdown and ended up with 1528 to_purge undo log pages. (leaked undo log corrupt pages) Start the latest 10.6 against the data directory, did slow shutdown 2024-07-04 14:29:50 0 [Note] InnoDB: Truncating system tablespace from 4864 to 2496 pages 2024-07-04 14:29:50 0 [Note] InnoDB: System tablespace truncated successfully Ran innochecksum tool against shrinked system tablespace and we end up with same to_purge Undo page state: 0 active, 128 cached, 1528 to_purge, 0 prepared, 148 other So verified what was exist before 2496 page, it was all leaked undo pages and previous extent state has been marked as "XDES_FREE_FRAG" and 12 pages has been used in that extent. #::2484 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2485 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2486 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2487 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2488 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2489 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2490 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2491 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2492 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2493 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2494 | Undo log page |, Will be freed in purge when all undodata in it is removed, - #::2495 | Undo log page |, Will be freed in purge when all undodata in it is removed, - MDEV-14795 didn't handle leaked undo pages because even their extent has never been marked as freed. So MDEV-14795 assumes that it is used by server. MDEV-14795 finds the last used extent in the system tablespace and truncates the remaining file. Attached the ibdata1 of the leaky undo pages for future reference.
            tiandiwonder Shaohua Wang added a comment -

            So we should fix the undo page leak bug?

            tiandiwonder Shaohua Wang added a comment - So we should fix the undo page leak bug?
            marko Marko Mäkelä added a comment -

            The undo page leak bug MDEV-31234 was fixed a long time ago. This fix is only freeing up the previously leaked pages in the system tablespace.

            marko Marko Mäkelä added a comment - The undo page leak bug MDEV-31234 was fixed a long time ago. This fix is only freeing up the previously leaked pages in the system tablespace.
            mleich Matthias Leich added a comment - - edited

            origin/11.2-MDEV-34529 c1fc29443a213367c7b0c521d0d9c1c1636bf48b 2024-10-10T19:21:33+05:30
            performed well in RQG testing. No new problems.
            No objections against pushing the fix.

            mleich Matthias Leich added a comment - - edited origin/11.2- MDEV-34529 c1fc29443a213367c7b0c521d0d9c1c1636bf48b 2024-10-10T19:21:33+05:30 performed well in RQG testing. No new problems. No objections against pushing the fix.

            People

              thiru Thirunarayanan Balathandayuthapani
              valerii Valerii Kravchuk
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.