It is most likely a regression introduced (or made visible) by the following revision:
I cannot verify it with 100% certainty because the provided test case causes a crash on revisions 3628..3634; but it produces a correct result on maria/5.3 up to revno 3627 and a wrong result starting from revno 3635 and up to (and including) the current 3646.
Also reproducible on current maria/5.5.
Expected result - empty set:
Inconsistency of the actual result can be confirmed by executing the second part of AND separately:
It is a correct result, and it doesn't include the row with a=5, so the result set for the bigger query cannot include it either (but currently it does).
Reproducible with the default optimizer_switch as well as with all OFF values.