Inefficient memory management by parallel slave

Summary

The parallel slave may be inefficient with handling memory.
That also includes the presentation side as a significant part of the
allocated memory is uninstrumented, invisible to P_S selects.
On top of that present configuration options may confuse users
about the actual memory status on their intended setups.

Specifically the following points are made upon source code analysis.

1. Q_max the cap is denominated in relay-log bytes, not in memory

slave_parallel_max_queued bounds relay-log bytes: queued_size
accumulates qev->event_size in rpl_parallel_thread::enqueue(), and
the cap test in rpl_parallel_entry::choose_thread_internal() is the only
backpressure on the SQL (aka driver) thread. The resource it actually governs
is resident memory, which is alpha times greater. So an increase of the
parameter forces the driver thread to queue proportionally more transactions
and their events, and the two settings multiply as |W| * Q_max, |W| being the number of workers.

There are three major contributions to alpha:

• the payload is held twice — the raw serialized copy the event was read into, and the decoded row image;
• a fixed about 1.4 KB per event "envelope" (queue element plus event object), independent of the event's size;
• a fixed about 6 KB per transaction (rgi - group descriptor plus table map), independent of the transaction's size.

The 3rd factor can be a major one for a small e.g single-statement trx, where
the amplifier alpha can grow up to 30 and therefore the estimate of the
total size of events in progress (read from relay-log and not yet committed)
can reach with |W| = 48, Q_max = 64MB, alpha = 30 to 90GB.
The estimate assumes all workers reach
the cap at some point during the run.

Note too memory allocated to rgi and its members, and to the event
envelope, is recycled through the per-worker free lists rather than freed,
and is released only at STOP SLAVE, by the teardown loop in
rpl_parallel_change_thread_count(). The payload copies of the 1st
contribution are freed per event at apply time, in
delete_or_keep_event_post_apply(). Resident memory consequently settles
at the historical maximum backlog and does not come back when the replica
catches up.

2. The table map is misplaced into rgi

The mentioned contribution from the table maps (ROW format) is due to
apparent misplacement of the table_mapping object into rgi. The map
holds no cross-group state — clear_tables() runs on every
rpl_group_info::cleanup_context() — and all server-side uses are
worker-local, both the set_table() and the get_table() calls sitting
in Rows_log_event::do_apply_event().

3. Neither meter accounts for the parallel slave

Neither

• SHOW GLOBAL STATUS LIKE 'Memory_used'
• SELECT ... FROM performance_schema.memory_summary_global_by_event_name ...

provides accurate account for how much the parallel slave is the memory
consumer, and the two fail differently:

• Memory_used sees most of the volume — everything allocated through the my_malloc family — but is a single global scalar attributing nothing to replication;
• memory_summary_* attributes, but only for named keys, of which this path has two: table_mapping::m_mem_root and String::value;
• neither sees rpl_group_info or the Log_event objects, which go through the plain global operator new.

Suggested improvements

1. Consider introducing a count of trx:s/events parameter to serve as a cap alongside slave_parallel_max_queued. There is no point in continuing to saturate workers with more assignments when they are already loaded significantly. The byte cap has to stay, for the single oversized trx that must be admitted regardless.
2. Make table_mapping one instance per worker (rpl_parallel_thread) instead of one per rgi. Two constraints:
   • sharing one map across workers is not viable — the value stored by set_table() is a TABLE handle private to that worker's THD, so the same table_id denotes a different object in each worker;
   • copy_data_between_tables() builds an rpl_group_info on the stack for online ALTER, so the member cannot simply be removed.
3. Remove the two file-name arrays queued_event::event_relay_log_name and queued_event::future_event_master_log_name from the event envelope. They are 1024 of its 1096 bytes, per-file quantities copied per event. The relay-log name is already reachable as qev->ir->name, whose lifetime is held for the envelope's duration by the refcount that rpl_parallel::do_event() takes on the inuse_relaylog.
4. Consider partial release of rgi instances during runtime, so that resident memory decays instead of standing at the high-water mark until STOP SLAVE. handle_rpl_parallel_thread() already holds LOCK_rpl_thread at its batch_free() call on every path to idleness.
5. Add missing instrumentation: PSI keys for queued_event, m_rows_buf, rpl_group_info and the Log_event objects; per-worker queued_size and in-flight rgi count exposed through P_S.