0 - x returns -9223372036854775808 for x = -9223372036854775808 instead of ERROR 1690: the overflow check misses a left operand of 0

Subtracting -9223372036854775808 from a BIGINT zero overflows: the result, 2^63, is not a BIGINT. MariaDB raises ERROR 1690 for 1 - x, but for 0 - x it returns -9223372036854775808, the wrong sign, with no error or warning. Any left operand whose value is 0 does this: a literal, a column holding 0, or a computed 0.

CREATE TABLE t1 (x BIGINT, z BIGINT, o BIGINT);

INSERT INTO t1 VALUES (-9223372036854775808, 0, 1);

SELECT 0 - x FROM t1;

SELECT z - x FROM t1;

SELECT (7 DIV x) - x FROM t1;

SELECT AVG(0 - x) FROM t1;

SELECT 0 - CAST(-9223372036854775808 AS SIGNED);

SELECT o - x FROM t1;

SELECT -x FROM t1;

SELECT -1 * x FROM t1;

SELECT x DIV -1 FROM t1;

DROP TABLE t1;

Leads to, on 12.3.3 and 13.0.2 alike:

0 - x                                      -9223372036854775808

z - x                                      -9223372036854775808

(7 DIV x) - x                              -9223372036854775808

AVG(0 - x)                                 -9223372036854775808.0000

0 - CAST(-9223372036854775808 AS SIGNED)   -9223372036854775808

o - x      ERROR 1690 (22003): BIGINT value is out of range in '`test`.`t1`.`o` - `test`.`t1`.`x`'

-x         ERROR 1690 (22003): BIGINT value is out of range in '-`test`.`t1`.`x`'

-1 * x     ERROR 1690 (22003): BIGINT value is out of range in '-1 * `test`.`t1`.`x`'

x DIV -1   ERROR 1690 (22003): BIGINT value is out of range in '`test`.`t1`.`x` DIV -1'

Expected: ERROR 1690 for the first five statements too, as for o - x. MySQL 8.4.13 raises ERROR 1690 for 0 - x, z - x, (7 DIV x) - x and 0 - CAST(-9223372036854775808 AS SIGNED), and PostgreSQL 14.24 raises bigint out of range. The result is the same under sql_mode='STRICT_ALL_TABLES,ONLY_FULL_GROUP_BY'.

From the code, the cause seems to be this test in Item_func_minus::int_op() for two signed operands (sql/item_func.cc, line 1404 at mariadb-12.3.3, unchanged on main):

      if (val0 > 0 && val1 < 0)

        res_unsigned= TRUE;

      else if (val0 < 0 && val1 > 0 && val0 < (LONGLONG_MIN + val1))

        goto err;

With val0 == 0 and val1 == LONGLONG_MIN neither branch is taken, so res= val0 - val1 is a signed subtraction that overflows (undefined behaviour; LONGLONG_MIN in practice), and check_integer_overflow(res, false) accepts it. With val0 >= 0 the result would be the unsigned 2^63, which check_integer_overflow() refuses, as it does for val0 == 1. MDEV-35329 reports the undefined behaviour in the same function; the patch proposed there (computing in ulonglong) would still return LONGLONG_MIN here, because res_unsigned stays FALSE.