In maria-5.3-handler, HANDLER + Memory tables is broken. It also appears that the handler.heap.test was --record-ed improperly, so it does not catch this critical breakage.

To repeat:

create table t1 (f1 integer not null, key (f1)) engine=Memory;
insert into t1 values (1);
HANDLER t1 OPEN;

MariaDB [test]> HANDLER t1 READ f1 NEXT;
ERROR 1031 (HY000): Table storage engine for 't1' doesn't have this option
but should have returned 1 row

MariaDB [test]> HANDLER t1 READ f1 NEXT;
ERROR 1020 (HY000): Record has changed since last read in table 't1'
should have returned an empty row