Valgrind exposed a problem with the join_cache for hash joins:
=25636== Conditional jump or move depends on uninitialised value(s)
==25636== at 0xA8FF4E: JOIN_CACHE_HASHED::init_hash_table() (sql_join_cache.cc:2901)
==25636== by 0xA8FCEA: JOIN_CACHE_HASHED::init(bool) (sql_join_cache.cc:2841)
==25636== by 0xA91381: JOIN_CACHE_BNLH::init(bool) (sql_join_cache.cc:3940)
==25636== by 0x8FD85A: JOIN::init_join_caches() (sql_select.cc:1878)
==25636== by 0x902058: JOIN::optimize_stage2() (sql_select.cc:3111)
==25636== by 0x8FF7F0: JOIN::optimize_inner() (sql_select.cc:2394)
==25636== by 0x8FCF7D: JOIN::optimize() (sql_select.cc:1711)

The reason for this was that avg_record_length has a random value if one used
SET optimizer_switch='optimize_join_buffer_size=off';

This causes either 'random size' memory to be allocated (up to join_buffer_size) which can
increase memory usage or if avg_record_length is less than the row size, memory overwrites in thd->mem_root, which is bad.

Fixed by setting avg_record_length in JOIN_CACHE_HASHED::init() before it's used.