Thanks for the report.

The problem was introduced by this commit:

commit 09b87d6293b4b41321ba98366d5d7ade9ad681d3

Author: Alexander Barkov <bar@mariadb.org>

Date:   Wed Sep 30 10:05:16 2015 +0400

    MDEV-8871 Wrong result for CREATE TABLE .. SELECT LEAST(unsigned_column,unsigned_column)

Note, the same problem is repeatable in 10.0 with other hybrid type functions:

SET SESSION wait_timeout = COALESCE(28000, @@wait_timeout);

SET SESSION wait_timeout = IFNULL(28000, @@wait_timeout);

SET SESSION wait_timeout = CASE WHEN TRUE THEN 28000 ELSE @@wait_timeout END;

So the patch for MDEV-8871 only made LEAST detect its data type consistently with COALESCE, IFNULL, CASE.
The problem was not really "introduced" by MDEV-8871.
It exists for a long time and should be fixed for all hybrid functions at once.

Possible fixes:
1. Fix hybrid functions to fallback to INT or BIGINT if the result data type was detected as DECIMAL(N,0) with a small N.
Note, fallback should work only if all arguments are of INT-alike types actually (and should keep DECIMAL if any of the arguments was DECIMAL).
2. Fix int-type variables
a. Accept assignment of a DECIMAL(N,0) expression with a small N.

• Allow integer type variables to accept expressions of the DECIMAL(N, 0) type, providing that N fits into the underlying INT data type
• Fix data type of @@wait_timeout to return INT(11), instead of BIGINT(21), otherwise the "N fits" requirement will be FALSE.

b. Another option is just to accept all DECIMAL(N, 0), even if N is big enough, and produce a warning on overflow.

c. Another option is to accept all DECIMAL(N,M), even if M>0, and produce a warning on fraction truncation.

During a discussion, Serg and Bar decided to accept DECIMAL(N,0), no matter if N is big enough, and produce an error if the value does not fit.

ok to push