trx_undo_left() fails to prevent overflow

The function trx_undo_left() checks whether an undo log record (or a part of it) would fit in the undo log page. If not, a new undo log page will be allocated.

There is some wishful thinking in the design of the function, all the way from the very start of the public history of InnoDB:

/**************************************************************************

Calculates the free space left for extending an undo log record. */

UNIV_INLINE

ulint

trx_undo_left(

/*==========*/

			/* out: bytes left */

	page_t* page,	/* in: undo log page */

	byte*	ptr)	/* in: pointer to page */

{

	/* The '- 10' is a safety margin, in case we have some small

	calculation error below */

	return(UNIV_PAGE_SIZE - (ptr - page) - 10 - FIL_PAGE_DATA_END);

}

Due to the unsigned return type, the additional 10 bytes actually constitute an unsafety margin that causes us to waste 10 bytes in every undo log page. If the ptr is already past those 10+4 bytes from the end of the page, the function will return a large positive value. But the numerous trx_undo_left(...) < ... checks would expect a negative value.

I think that we must add debug assertions to the function in all maintained versions (backport parts of MDEV-24096), and make the function return 0 when ptr is already past the end, to make the overflow checks work in a desirable way.

Also, we should re-review all code that calls the function, especially the code that writes indexed virtual columns to the undo log.