Uploaded image for project: 'MariaDB Connector/J'
  1. MariaDB Connector/J
  2. CONJ-1348

Reassembling packets larger than 16 MB copies the accumulated content once per fragment

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • None
    • 3.5.11
    • Other
    • None

    Description

      When a server packet exceeds 16 MB (0xFFFFFF), protocol splits it into fragments of exactly 16 MB followed by a terminating shorter fragment. Reader.readPacket() reassembles them by allocating a new array of the accumulated size for every fragment and copying all previously received bytes into it.

      For a payload of n bytes this copies about n² / 32 MB bytes in total. A 512 MB BLOB row (33 fragments) copies roughly 8 GB of memory and spends about 2 seconds in reassembly, on top of the network transfer.

      The fix buffers the fragments in a list as they arrive and copies them once into the final array. The maxAllowedPacket check is still applied to the running total before each fragment is allocated. Packets under 16 MB do not enter this code path, so normal rows are unaffected. Peak transient memory is unchanged (about twice the payload during the final copy).

      Measured with a synthetic fragmented stream fed into Reader.readPacket() (median of 5 runs):

      Payload Fragments Before After
      64 MB 5 106 ms 58 ms
      128 MB 9 217 ms 149 ms
      256 MB 17 546 ms 331 ms
      512 MB 33 2172 ms 690 ms

      About 390 ms of the "after" figures is the cost of generating the test stream itself, so the reassembly overhead at 512 MB drops from roughly 1.8 s to 0.3 s and will now scales linearly with payload size.

      Unit tests added for content integrity across fragment boundaries, a payload of exactly 16 MB with its empty terminating fragment, and a stream truncated mid-fragment.

      Attachments

        Activity

          People

            diego dupin Diego Dupin
            diego dupin Diego Dupin
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.