Corrupt Heartbeat Log Event can Crash Slave

A replication source can terminate its replica by sending a single malformed heartbeat event.

The Heartbeat_log_event parser derives the length of the binlog file name by subtracting the common header length from the event length, and it does so without first establishing that the event is long enough for the subtraction to mean anything

  uint8 header_size= description_event->common_header_len;

  ...

  log_ident= buf + header_size;

  ident_len = event_len - header_size;              /* underflows */

We do catch the underflow. is_valid() requires ident_len <= FN_REFLEN-1, so the event is rejected and never applied.

The rejection itself is where the length gets used, before anything has bounded it (sql/slave.cc:6831):

    if (!hb.is_valid())

    {

      error= ER_SLAVE_HEARTBEAT_FAILURE;

      ...

      error_msg.append((char*) hb.get_log_ident(), (uint) hb.get_ident_len());

error_msg is a StringBuffer<1024>, so the buffer that String::append is being asked to grow by roughly 4 GB lives in queue_event()'s own stack frame.

The outcome follows from the gap by which header_size exceeds event_len. More specifically, alloc_length resolves to ident_len plus 197, being the 69 bytes of message literals already appended plus the 128 byte extra_alloc that realloc_with_extra() adds, and it is that figure the allocator guards compare against.

• Debug builds abort. DBUG_ASSERT(len > alloc_length) in Binary_string::realloc_raw (sql/sql_string.cc:93) fires for any gap at all. This is the outcome the reproduction below produces.
• 64-bit release builds survive a gap below 206 and crash at or above it. Below 206, ALIGN_SIZE(alloc_length+1) truncates when it is stored into a uint32, the following if (len <= alloc_length) return TRUE; notices, and the append fails cleanly, leaving the IO thread to stop with an error message that is merely missing its file name. At 206 and above that truncation no longer occurs, realloc_raw requests roughly 4 GB, and on success q_append copies that much out of a 36 byte packet. Note that reaching a gap this large calls for the Format_description route, which means a common_header_len of 242 or more against a heartbeat of the usual 36 bytes.
• 32-bit builds corrupt the stack. For any gap up to 69, str_length + size wraps to a value below Alloced_length, realloc_with_extra_if_needed short circuits on it, and q_append writes past the 1024 byte buffer in queue_event()'s frame.

Reproduction (note that we use the debug mechanism to alter the format descriptor event's stored length of the heartbeat header):

--- a/sql/sql_repl.cc

+++ b/sql/sql_repl.cc

@@ -866,7 +866,13 @@ static int send_heartbeat_event(binlog_send_info *info,

   int4store(header + EVENT_LEN_OFFSET, event_len);

   int2store(header + FLAGS_OFFSET, 0);

-  packet->append(header, sizeof(header));

+  size_t hb_header_len= sizeof(header);

+  DBUG_EXECUTE_IF("binlog_sender_truncated_heartbeat",

+                  hb_header_len= 17; ident_len= 0;);

+

+  packet->append(header, hb_header_len);

   if (sub_header_in_use)

     packet->append(sub_header_buf, sizeof(sub_header_buf));

   packet->append(p, ident_len);                    // log_file_name

@@ -874,7 +880,7 @@ static int send_heartbeat_event(binlog_send_info *info,

   if (do_checksum)

   {

     char b[BINLOG_CHECKSUM_LEN];

-    ha_checksum crc= my_checksum(0, (uchar*) header, sizeof(header));

+    ha_checksum crc= my_checksum(0, (uchar*) header, hb_header_len);

     if (sub_header_in_use)

       crc= my_checksum(crc, (uchar*) sub_header_buf, sizeof(sub_header_buf));

     crc= my_checksum(crc, (uchar*) p, ident_len);

Test:

--source include/have_debug.inc

--source include/master-slave.inc

# Set on the source before the replica reconnects below, so the new dump

# thread inherits it.

--connection master

set @@global.debug_dbug= "+d,binlog_sender_truncated_heartbeat";

# The replica is caught up and idle, so the dump thread waits for new events

# and emits an artificial heartbeat once the period elapses. The default

# period is slave_net_timeout/2, so shorten it.

--connection slave

--source include/stop_slave.inc

CHANGE MASTER TO MASTER_HEARTBEAT_PERIOD= 0.5;

--source include/start_slave.inc

# The heartbeat arrives 17 bytes long, so ident_len underflows to 2^32-2 and

# queue_event() hands that to String::append on the rejection path. The wait

# below only holds the test open: the replica dies before the IO thread ever

# records an error.

--let $slave_io_errno= 1595

--source include/wait_for_slave_io_error.inc

--source include/rpl_end.inc

As a disclaimer, this reproduces a gap of 2, which is enough to demonstrate the underflow and to exercise a fix, but it stops at the first of the three outcomes described above. Reaching the 4 GB allocation calls for the Format_description route instead, and that route applies its inflated common_header_len to every subsequent event, which stops the IO thread before a heartbeat ever arrives.