heap_update() key-update loop does not roll back on non-duplicate-key failures, and hp_rb_write_key() masks allocation failure as HA_ERR_FOUND_DUPP_KEY

Summary

heap_update() (storage/heap/hp_update.c) moves every changed key entry
to its new value (delete old + write new) before the record itself is
updated:

  for (keydef= share->keydef, end= keydef + share->keys; keydef < end; keydef++)

  {

    if (hp_rec_key_cmp(keydef, heap_new, old, NULL))

    {

      if ((*keydef->delete_key)(info, keydef, old, pos, keydef == p_lastinx) ||

          (*keydef->write_key)(info, keydef, heap_new, pos))

        goto err;

      ...

The recovery at the err: label undoes the already-moved keys only when
my_errno == HA_ERR_FOUND_DUPP_KEY:

 err:

  if (my_errno == HA_ERR_FOUND_DUPP_KEY)

  {

    /* rollback loop: delete_key(new) + write_key(old) ... */

Any other error reaching goto err from the key loop skips the rollback
entirely: keydefs processed before the failing one keep entries keyed on the
new column values while the record still holds the old values. Subsequent
index lookups by the real column value silently miss the row, CHECK TABLE
reports the table corrupt, and on debug builds the heap consistency check in
ha_heap::external_lock() raises an error into an already-set diagnostics
area, firing a Diagnostics_area assertion on a following statement. This
is the same corruption class fixed for the blob-chain write path in
MDEV-40378, but triggered from the key loop itself.

Failure modes that reach the unguarded path

1. HASH delete_key -> HA_ERR_CRASHED. hp_delete_key() returns
   HA_ERR_CRASHED when the entry for the old key value is not found in the
   hash chain. This only happens when the index is already inconsistent, but the
   missing rollback then compounds the corruption: keydefs already moved to new
   values are abandoned mid-flight, so one corrupt index degrades the others.
2. BTREE write_key allocation failure – currently misclassified.
   hp_rb_write_key() maps every tree_insert() failure to a duplicate:

   if (!tree_insert(&keyinfo->rb_tree, (void*)info->recbuf,

   custom_arg.key_length, &custom_arg))

   {

   my_errno= HA_ERR_FOUND_DUPP_KEY;

   return 1;

   }

   tree_insert() returns NULL both for a duplicate in a TREE_NO_DUPS
   tree and for a node allocation failure (init_tree() is called with
   memory_limit=0 in hp_create(), so the only allocation failure source
   is genuine malloc failure). Consequences of the conflation:

   • An out-of-memory during UPDATE of a BTREE-indexed HEAP table is reported
     to the user as ER_DUP_ENTRY ("Duplicate entry ... for key ...") – with a
     fabricated duplicate value, and possibly naming a non-unique index, on
     which a duplicate error is nonsensical.
   • The bogus DUPP_KEY classification accidentally routes the failure into
     the rollback branch, so today the indexes are usually repaired. The two
     defects interlock: correcting the error code (e.g. to
     HA_ERR_OUT_OF_MEM) without also generalizing the err: rollback would
     trade the wrong error message for silent index corruption.
3. Rollback itself failing under sustained memory pressure. Inside the
   DUPP_KEY branch, the BTREE special case re-inserts the old key; if that
   re-insert also fails (same OOM), heap_update() returns immediately
   without rolling back the keydefs processed earlier – again leaving mixed
   old/new index state, silently.
4. Future failure modes. The gate is not future-proof: any new error source
   added to a delete_key/write_key implementation lands on the unguarded
   path with no rollback. The blob-chain writes introduced in the 13.1 preview
   did exactly this (MDEV-40378).

Note: HASH write_key cannot fail with anything but a duplicate in this
loop today – hp_find_free_hash() reuses the slot freed by the preceding
delete_key (records < block->last_allocated), so no block allocation
occurs.

Reachability

Without pre-existing corruption, the trigger is allocation failure at a
precise moment (fault-injection territory), so this is not expected in
healthy production workloads. It is still index corruption without any
crashed-table signalling, and item 2 (wrong user error on OOM) is reachable
on any OOM during a BTREE-indexed UPDATE.

Suggested fix

• In hp_rb_write_key(), distinguish duplicate from allocation failure
  (e.g. classify via the errno set by the failed allocation) and return
  HA_ERR_OUT_OF_MEM for the latter, so the SQL layer reports the true
  error.
• Generalize the err: recovery in heap_update() to roll back all
  fully-moved keydefs for any failure, handling the partially-processed
  failing keydef per algorithm (BTREE: nothing was inserted, re-insert old;
  HASH duplicate: the new entry is present and must be deleted), mirroring the
  structure of the MDEV-40378 err_blob: recovery. Both changes must land
  together (see interlock above).

Affected versions

The DUPP_KEY-only rollback gate and the hp_rb_write_key() error
conflation are inherited from the original MySQL HEAP engine code and are
present in all maintained MariaDB versions. The related blob-path variant
affects only the 13.1 preview and is fixed under MDEV-40378.