If I trace execution in MariaDB, I see that

• rand()*100 is evaluated when writing to the temporary table
• "r<20" is checked after reading from the temporary table. The value of r is obtained from the temporary table field.

Code-wise, it is Item_func_lt::val_int(), which has an argument of Item_ref, which points to Item_field, which is a field in the temporary table.

If I trace execution in MySQL 5.6, I see that "r < 20" is checked before writing to the temporary table. It seems, the problem is that rand() is evaluated two times for each group:
1. first time when preparing column for the temporary table
2. second time when evaluating HAVING.

In other words, the following happens:
S1. rand() returns a value that doesn't satisfy HAVING. The value is put into temp. table row buffer.

S2. The second call to rand() call returns value that does satisfy HAVING. We write tem. table row buffer into the temp. table

Possible solutions:

== Evaluate the "late" form of HAVING clause ==
The "early" form of HAVING uses "rand() * 100"
The "late" form of HAVING uses "temptable.r"

if we used "late" form, we would have avoided re-calculation of rand().

== Change rand() to have next_value() method ==

Make rand() return the same value until somebody calls next_value() or something like that. This is a rather big change, because we will need to define when it's time to use the "next value" for all possible cases.

== Don't use early-HAVING-evaluation for rand() ==

end_write_group() has the code that checks HAVING clause (join->having) in MariaDB 5.5

However, when tracing the query from the bug, I see that join->having==NULL.
Apparently, MariaDB 5.5

• generally, supports early HAVING evaluation
• but does not use it in this case.

Perhaps, we should make MySQL 5.6 not to use early HAVING evaluation for conditions with RAND_TABLE_BIT?

The optimizer switches between "early" and "late" form of WHERE by changing "ref array".

Here is how the changes are made:
== mariadb 5.5 ==
Hardware watchpoint 27: **$a
Old value = (Item_func_mul *) 0x7fff94006718
New value = (Item_field *) 0x7fff94013bc8

curr_join->set_items_ref_array(items1);

== mysql 5.6 ==

Hardware watchpoint 16: **$a
Old value = (Item_func_mul *) 0x7fffc40071c8
New value = (Item_field *) 0x7fffc400f898

In JOIN::make_tmp_tables_info():

// Need to set them now for correct group_fields setup, reset at the end.
set_items_ref_array(items1);

Hardware watchpoint 16: **$a
Old value = (Item_field *) 0x7fffc400f898
New value = (Item_func_mul *) 0x7fffc40071c8
in JOIN::make_tmp_tables_info:

// Reset before execution
set_items_ref_array(items0);

JOIN::exec() ...
end_write_group()
if (!join_tab->having || join_tab->having->val_int())

^^^^^^^^ this is where the problem happens

....
Hardware watchpoint 16: **$a
Old value = (Item_func_mul *) 0x7fffc40071c8
New value = (Item_field *) 0x7fffc400f898

in QEP_tmp_table::end_send ():

// Update ref array
join_tab->join->set_items_ref_array(*join_tab->ref_array);
table->reginfo.lock_type= TL_UNLOCK;

Hardware watchpoint 16: **$a
Old value = (Item_field *) 0x7fffc400f898
New value = (Item_func_mul *) 0x7fffc40071c8

in JOIN::cleanup (this=0x7fffc4007f30, full=true):

/* Restore ref array to original state */
if (current_ref_ptrs != items0)

The fix is in the last comment.