Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-24156

trx_undo_left() fails to prevent overflow

    XMLWordPrintable

    Details

      Description

      The function trx_undo_left() checks whether an undo log record (or a part of it) would fit in the undo log page. If not, a new undo log page will be allocated.

      There is some wishful thinking in the design of the function, all the way from the very start of the public history of InnoDB:

      /**************************************************************************
      Calculates the free space left for extending an undo log record. */
      UNIV_INLINE
      ulint
      trx_undo_left(
      /*==========*/
      			/* out: bytes left */
      	page_t* page,	/* in: undo log page */
      	byte*	ptr)	/* in: pointer to page */
      {
      	/* The '- 10' is a safety margin, in case we have some small
      	calculation error below */
       
      	return(UNIV_PAGE_SIZE - (ptr - page) - 10 - FIL_PAGE_DATA_END);
      }
      

      Due to the unsigned return type, the additional 10 bytes actually constitute an unsafety margin that causes us to waste 10 bytes in every undo log page. If the ptr is already past those 10+4 bytes from the end of the page, the function will return a large positive value. But the numerous trx_undo_left(...) < ... checks would expect a negative value.

      I think that we must add debug assertions to the function in all maintained versions (backport parts of MDEV-24096), and make the function return 0 when ptr is already past the end, to make the overflow checks work in a desirable way.

      Also, we should re-review all code that calls the function, especially the code that writes indexed virtual columns to the undo log.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Git Integration