I believe there is an issue with the query optimizer in MariaDB v5.5.30 and v10. Zip file contains SQL dump of test database structure and data as well as a SQL file of the offending query (copied below). I can replicate this issue on MariaDB v5.5.30 and v10. I CANNOT replicate this error on MariaDB v5.5.29 nor MySQL. I cannot be certain but based on the commit descriptions, it may be related to 2502.567.70, 2502.567.71, or 2502.567.72.
When the ISNULL() expression is present in the query below, mysqld seizes 100% of available physical memory and continues to allocate further until all available virtual memory is consumed as well, thereby hanging the machine:
1. Running the query as below, with or without EXPLAIN, causes an out-of-memory condition.
2. Commenting out the ISNULL(t4.c11) lets the query succeed.
3. Running SELECT * FROM t4 WHERE ISNULL(t4.c11) succeeds.
4. Removing various parts of the query suggest that this issue is only present with multiple joins and multiple WHERE conditions.
Further iterations suggest that all works until the ISNULL expression is added back to the query. I cannot pull the query plan due to the aforementioned out-of-memory condition.