Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-12353

Efficient InnoDB redo log record format

Details

    Description

      The InnoDB crash recovery performance can be improved a little while not changing the file format. MDEV-12699 removed unnecessary reads of pages that can be initialized via the redo log. MDEV-19586 will make recovery read the pages in more sequential order.

      We should fix some fundamental issues that exist with the current InnoDB redo log record format:

      • Records do not contain their length. When buffering records, we must painstakingly parse entire records in order to determine the length. This idea was mentioned in MySQL Bug#82937.
      • For B-tree operations, we are writing a lot of redundant data for mlog_parse_index(). We should use a lower-level format and lower-level apply functions. MySQL Bug#82176 merely speeds up the code around mlog_parse_index().
      • If a mini-transaction is writing multiple records to a page, the page identifier is being repeated for every record. We should omit the page identifier if multiple consecutive records are modifying the same page.

      In this task, we will only improve the redo log record format. Format changes to the redo log blocks and files will be covered by MDEV-14425.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21724 Optimize page_cur_insert_rec_low() redo logging

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21725 Optimize btr_page_reorganize_low() redo logging

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21727 Optimize redo logging for ROW_FORMAT=COMPRESSED

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21744 Assertion `!rec_offs_nth_sql_null(offsets, n)' failed in btr_cur_upd_rec_in_place

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21748 ASAN use-after-poison in PageBulk::insertPage on table rebuild

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21749 storage/innobase/page/page0cur.cc:1306: rec_t* page_cur_insert_rec_low(const page_cur_t*, dict_index_t*, const rec_t*, offset_t*, mtr_t*): Assertion `rdm - rd + bd <= insert_buf + rec_size' failed.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21779 Assertion `!memcmp(insert_buf, rec - extra_size, rec_size)' failed in page_cur_insert_rec_low

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21850 ASAN use-after-poison in page_cur_insert_rec_low

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21892 Assertion `index != clust_index || row_get_rec_trx_id(rec, index, offsets)' failed in row_search_mvcc

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21893 Assertion `log_block_get_start_lsn(lsn, log_block_get_hdr_no(buf)) == lsn' failed upon attempt to start on encrypted datadir from previous versions

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21945 Assertion w==OPT failed in trx_purge_add_undo_to_history()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21949 key rotation for innodb_encrypt_log is not working in 10.5

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22107 Restore accidentally orphaned MTR_MEMO_MODIFY

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22108 Crash recovery fails with [ERROR] InnoDB: Malformed log record

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22242 B-trees can become extremely skewed

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23806 Undo page corruption on recovery

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24196 WITH_UBSAN runtime error: member call on null pointer of type 'struct log_phys_t'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24412 Mariadb 10.5: InnoDB: Upgrade after a crash is not supported

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24652 mtr fails while reusing the cached undo log block

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-24848 Assertion `rlen < llen' failed in log_phys_t::apply_status log_phys_t::apply upon MariaBackup prepare

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27059 page_zip_dir_insert() may corrupt ROW_FORMAT=COMPRESSED tables

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27444 Perform backup prepare using mariabbackup 10.4 version when performing rolling upgrade on joiner node with 10.5

          • Major - Major loss of function.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27486 Refuse Galera SST if major version of donor and joiner are different

          • Critical - Crashes, loss of data, severe memory leak.
          • Stalled

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28731 Race condition on log checkpoint

          • Major - Major loss of function.
          • Closed
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19747 Deprecate and ignore innodb_log_optimize_ddl

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-21174 Refactor mlog_write_ulint, mlog_memset, mlog_write_string

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21351 Replace recv_sys.heap with list of buf_block_t*

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21674 purge_sys.stop() no longer waits for purge workers to complete

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12699 Improve crash recovery of corrupted data pages

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-14425 Change the InnoDB redo log format to reduce write amplification

          • Blocker - Blocks development and/or testing work, production could not run.
          • Closed

          Task - A task that needs to be done. MDEV-18976 Implement a CHECKSUM redo log record for improved validation

          • Major - Major loss of function.
          • Closed

          Task - A task that needs to be done. MDEV-19586 Replace recv_sys_t::addr_hash with a std::map

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20562 btr_cur_open_at_rnd_pos() fails to return error for corrupted page

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20584 Clearing delete-mark on R-tree pages does not appear crash-safe

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20636 Potential SPATIAL INDEX corruption with ROW_FORMAT=COMPRESSED

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21024 InnoDB is issuing redundant writes to redo log

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22126 Rename confusing constant mtr_t::OPT

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23136 InnoDB init fail after upgrade from 10.4 to 10.5

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23986 [ERROR] [FATAL] InnoDB: Page ... name ... page_type ... key_version 1 lsn ... compressed_len ...

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-27437 Galera snapshot transfer fails to upgrade between some major versions

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28256 Galera SST fails with 'InnoDB: Upgrade after a crash is not supported'

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-29153 2022-07-22 18:12:56 0x7f3319fff700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.25/storage/innobase/page/page0cur.cc line 11

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32144 Debug assertion failure w == MAYBE_NOP in mtr_t::memcpy(), trx_undo_write_xid()

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-32445 InnoDB may corrupt its log before upgrading it on startup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33274 The test encryption.innodb-redo-nokeys often fails

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15274 innodb_gis.rtree_recovery failed in buildbot with error on check table

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19747 Deprecate and ignore innodb_log_optimize_ddl

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-20608 innodb_log_optimize_ddl=OFF may omit some redo log

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-21899 INSERT into a secondary index with zero-data-length key is not crash-safe

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-22097 Not applying DELETE_ROW_FORMAT_REDUNDANT due to corruption

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23474 InnoDB fails to start with Invalid log block checksum after unsetting innodb_log_checksums dynamically

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Task - A task that needs to be done. MDEV-24719 backport MDEV-24705 from 10.5 to 10.2, 10.3, 10.4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-26322 Last binlog file and position are "empty" in mariabackup --prepare output

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-31042 Certain tables are missing in innodb storage engine of a TSE enabled database after an incremental backup restore

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            Ascending order - Click to sort in descending order
            marko Marko Mäkelä created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            julien.fritsch Julien Fritsch made changes -
            Comment [ A comment with security level 'Developers' was removed. ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Fix Version/s 10.4 [ 22408 ]
            Fix Version/s 10.2 [ 14601 ]
            Fix Version/s 10.1 [ 16100 ]
            Fix Version/s 10.3 [ 22126 ]
            marko Marko Mäkelä made changes -
            Affects Version/s 10.3 [ 22126 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Fix Version/s 10.4 [ 22408 ]
            NRE Projects RM_105_CANDIDATE
            Affects Version/s 10.4 [ 22408 ]
            elenst Elena Stepanova made changes -
            Fix Version/s 10.5 [ 23123 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Affects Version/s 10.2 [ 14601 ]
            Affects Version/s 10.0 [ 16000 ]
            Affects Version/s 10.1 [ 16100 ]
            Affects Version/s 10.3 [ 22126 ]
            Affects Version/s 10.4 [ 22408 ]
            Issue Type Bug [ 1 ] Task [ 3 ]
            marko Marko Mäkelä made changes -
            Description The InnoDB crash recovery performance could be improved. Some ideas (file format changes cannot be done in GA versions):
            # Read the to-be-recovered pages sorted by page number. Currently, recv_apply_hashed_log_recs() picks a ‘random page number’ from recv_sys->addr_hash and then reads a number of pages starting from that number.
            # Use a simpler redo log record format, and reduce the number of operations that require mlog_write_index(). For example, use MLOG_1BYTE instead of MLOG_REC_SEC_DELETE_MARK. This should also reduce the redo log volume.
            # Look at some contributed patches, such as [MySQL Bug#82937|https://bugs.mysql.com/bug.php?id=82937] (format change), [MySQL Bug#82176|https://bugs.mysql.com/bug.php?id=82176].             		The InnoDB crash recovery performance can be improved a little while not changing the file format. MDEV-12699 removed unnecessary reads of pages that can be initialized via the redo log. MDEV-19586 will make recovery read the pages in more sequential order.

            We should fix some fundamental issues that exist with the current InnoDB redo log record format:
            * Records do not contain their length. When buffering records, we must painstakingly parse entire records in order to determine the length. This idea was mentioned in [MySQL Bug#82937|https://bugs.mysql.com/bug.php?id=82937].
            * For B-tree operations, we are writing a lot of redundant data for {{mlog_parse_index()}}. We should use a lower-level format and lower-level apply functions. [MySQL Bug#82176|https://bugs.mysql.com/bug.php?id=82176] merely speeds up the code around {{mlog_parse_index()}}.
            * If a mini-transaction is writing multiple records to a page, the page identifier is being repeated for every record. We should omit the page identifier if multiple consecutive records are modifying the same page.

            In this task, we will only improve the redo log record format. Format changes to the redo log blocks and files will be covered by MDEV-14425.
            Summary Speed up InnoDB crash recovery Efficient InnoDB redo log record format
            marko Marko Mäkelä made changes -
            Comment [ While fixing MDEV-17680, it occurred to me that in {{recv_apply_hashed_log_recs()}} we could easily filter out any log records that precede {{MLOG_INIT_FILE_PAGE2}}. If such a record is present for a page, we can simply create the page and apply the log records. There is absolutely no need to read the page, and possibly unnecessarily fail if the to-be-ignored page was corrupted. This should be fixed in MDEV-12699. ]
            marko Marko Mäkelä made changes -
            Elkin Andrei Elkin made changes -
            Assignee Marko Mäkelä [ marko ] Andrei Elkin [ elkin ]
            Elkin Andrei Elkin made changes -
            Assignee Andrei Elkin [ elkin ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            serg Sergei Golubchik made changes -
            Priority Major [ 3 ] Critical [ 2 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            mleich Matthias Leich made changes -
            Attachment simp_page_rec_set_n_owned.test [ 50212 ]
            mleich Matthias Leich made changes -
            Attachment simp_page_rec_set_n_owned-master.opt [ 50213 ]
            mleich Matthias Leich made changes -
            Attachment prt [ 50214 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2020-02-14 05:01:19.0 2020-02-14 05:01:19.48
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5.1 [ 24029 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5.2 [ 24030 ]
            Fix Version/s 10.5.1 [ 24029 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            thiru Thirunarayanan Balathandayuthapani made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels performance recovery ServiceNow performance recovery
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            Labels ServiceNow performance recovery 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z performance recovery
            serg Sergei Golubchik made changes -
            Labels 76qDvLB8Gju6Hs7nk3VY3EX42G795W5z performance recovery performance recovery
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 80109 ] MariaDB v4 [ 133192 ]
            jplindst Jan Lindström (Inactive) made changes -
            jplindst Jan Lindström (Inactive) made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            midenok Aleksey Midenkov made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            rob.schwyzer@mariadb.com Rob Schwyzer (Inactive) made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 201658 167030
            Zendesk active tickets 201658

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              4 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.