Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32570

Fragment ROW replication events larger than max_packet_size

    XMLWordPrintable

Details

    • Hide
      For transactions in ROW binlog_format that are too large to be either 1) sent to the replica (i.e. greater than slave_max_allowed_packet, max 1GB) or 2) binlogged (i.e. greater than 4GB), this feature adds support to fragment such events into sub-events of new type Partial_rows_log_event. A new system variable, binlog_row_event_fragment_threshold, is added to control the size of each Partial_rows_log_event.
      Show
      For transactions in ROW binlog_format that are too large to be either 1) sent to the replica (i.e. greater than slave_max_allowed_packet, max 1GB) or 2) binlogged (i.e. greater than 4GB), this feature adds support to fragment such events into sub-events of new type Partial_rows_log_event. A new system variable, binlog_row_event_fragment_threshold, is added to control the size of each Partial_rows_log_event.

    Description

      When ROW based replication is used and the transaction size, converted to base64 encoding, exceeds max_packet_size, replication breaks:

      Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master; the first event 'mariadb-bin.109824' at 4, the last event read from 'mariadb-bin.109825' at 15113554, the last byte read from 'mariadb-bin.109825' at 15113573.'

      Unfortunately, max_packet_size maximum allowed value is 1G.

      A blob over 500M large will cause the binlogged event to be larger than 1G so that it will crash replication, with NO workaround other than manually extracting the transaction from master's binlog and apply to slave.

      This ticket involves fragmenting row events larger than max_packet_size into multiple events. Additionally, Update_rows_log_events can be split into two series of events: 1) a series of deletes that cover the OLD rows of the update, and then 2) a series of inserts that cover the NEW rows of the update.

      Original solution kept for reference:

      My proposal: before binlogging the event on master, when the computed base64-encoded size is greater than max_packet_size, just convert format to STATEMENT for this event only.

      Link to corresponding Zulip topic is here.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38232 Partial_row_log_event to Include Total Row Event Size

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38609 Master should not fragment replication events based on slave_max_allowed_packet

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38615 Assertion `skip_res == EVENT_SKIP_NOT' failed.

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38621 Position mismatch in binlog for partial_row_log_events

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38625 Mariadb-binlog cant filter partial_log_events

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38639 Slave fails with Broken Partial_rows_log_event stream

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38665 mariadb-binlog --read-from-remote-server | mariadb fails

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-28768 mysqlbinlog can produce events larger than max_allowed_packet

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. MDEV-37583 Streaming Replication

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-38677 DRAFT: SIGSEGV|bg_slave_kill|handle_manager|start_thread|clone3

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MDEV-10963 mysqlbinlog can produce events larger than max_allowed_packet for mysql

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          has action item

          DOCS-5924 Loading...

          Activity

            People

              Deepthi ES Deepthi Eranti Sreenivas
              rpizzi Rick Pizzi (Inactive)
              Brandon Nesterenko Brandon Nesterenko
              Hemant Dangi Hemant Dangi
              Deepthi Eranti Sreenivas Deepthi Eranti Sreenivas
              Votes:
              3 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.