Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
Description
When a server packet exceeds 16 MB (0xFFFFFF), protocol splits it into fragments of exactly 16 MB followed by a terminating shorter fragment. Reader.readPacket() reassembles them by allocating a new array of the accumulated size for every fragment and copying all previously received bytes into it.
For a payload of n bytes this copies about n² / 32 MB bytes in total. A 512 MB BLOB row (33 fragments) copies roughly 8 GB of memory and spends about 2 seconds in reassembly, on top of the network transfer.
The fix buffers the fragments in a list as they arrive and copies them once into the final array. The maxAllowedPacket check is still applied to the running total before each fragment is allocated. Packets under 16 MB do not enter this code path, so normal rows are unaffected. Peak transient memory is unchanged (about twice the payload during the final copy).
Measured with a synthetic fragmented stream fed into Reader.readPacket() (median of 5 runs):
| Payload | Fragments | Before | After |
|---|---|---|---|
| 64 MB | 5 | 106 ms | 58 ms |
| 128 MB | 9 | 217 ms | 149 ms |
| 256 MB | 17 | 546 ms | 331 ms |
| 512 MB | 33 | 2172 ms | 690 ms |
About 390 ms of the "after" figures is the cost of generating the test stream itself, so the reassembly overhead at 512 MB drops from roughly 1.8 s to 0.3 s and will now scales linearly with payload size.
Unit tests added for content integrity across fragment boundaries, a payload of exactly 16 MB with its empty terminating fragment, and a stream truncated mid-fragment.