UBSAN: signed integer overflow in my_strntoll_8bit() during BLOB-to-integer conversion

SET sql_mode='';

CREATE TABLE t1 (c BLOB);

INSERT INTO t1 VALUES ('-9223372036854775808');

SELECT * FROM t1 WHERE c IN (VALUES(1));

Version: '10.11.17-MariaDB-asan-debug'  socket: '/test/UBASAN_MD100526-mariadb-10.11.17-linux-x86_64-dbg/socket.sock'  port: 11916  MariaDB Server

/test/10.11_dbg_san/strings/ctype-simple.c:673:22: runtime error: negation of -9223372036854775808 cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself

    #0 0x6288d9a121e0 in my_strntoll_8bit /test/10.11_dbg_san/strings/ctype-simple.c:673:22

    #1 0x6288d6e55296 in charset_info_st::strntoll(char const*, unsigned long, int, char**, int*) const /test/10.11_dbg_san/include/m_ctype.h:918:12

    #2 0x6288d6e55296 in Value_source::Converter_strntoll::Converter_strntoll(charset_info_st const*, char const*, unsigned long) /test/10.11_dbg_san/sql/field.h:239:21

    #3 0x6288d6e55296 in Value_source::Converter_strntoll_with_warn::Converter_strntoll_with_warn(THD*, Value_source::Warn_filter, charset_info_st const*, char const*, unsigned long) /test/10.11_dbg_san/sql/field.h:302:8

    #4 0x6288d6e1fcf7 in Field_blob::val_int() /test/10.11_dbg_san/sql/field.cc:8934:10

    #5 0x6288d715bd87 in Field::do_field_int(Copy_field*) /test/10.11_dbg_san/sql/field_conv.cc:404:37

    #6 0x6288d803b5f0 in store_key_field::copy_inner() /test/10.11_dbg_san/sql/sql_select.h:2045:5

    #7 0x6288d744ef91 in store_key::copy(THD*) /test/10.11_dbg_san/sql/sql_select.h:1970:30

    #8 0x6288d7fe2c6e in cp_buffer_from_ref(THD*, TABLE*, st_table_ref*) /test/10.11_dbg_san/sql/sql_select.cc:27142:18

    #9 0x6288d7fdd305 in join_read_always_key(st_join_table*) /test/10.11_dbg_san/sql/sql_select.cc:23848:7

    #10 0x6288d7eafdbf in sub_select(JOIN*, st_join_table*, bool) /test/10.11_dbg_san/sql/sql_select.cc:23074:12

    #11 0x6288d7fd37f8 in evaluate_join_record(JOIN*, st_join_table*, int) /test/10.11_dbg_san/sql/sql_select.cc:23307:11

    #12 0x6288d7eaff1f in sub_select(JOIN*, st_join_table*, bool) /test/10.11_dbg_san/sql/sql_select.cc:23077:9

    #13 0x6288d7f3dae6 in do_select(JOIN*, Procedure*) /test/10.11_dbg_san/sql/sql_select.cc:22600:14

    #14 0x6288d7f3a46a in JOIN::exec_inner() /test/10.11_dbg_san/sql/sql_select.cc:5023:50

    #15 0x6288d7f37f62 in JOIN::exec() /test/10.11_dbg_san/sql/sql_select.cc:4807:3

    #16 0x6288d7eb39ff in mysql_select(THD*, TABLE_LIST*, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) /test/10.11_dbg_san/sql/sql_select.cc:5285:9

    #17 0x6288d7eb24ed in handle_select(THD*, LEX*, select_result*, unsigned long long) /test/10.11_dbg_san/sql/sql_select.cc:601:10

    #18 0x6288d7d7e207 in execute_sqlcom_select(THD*, TABLE_LIST*) /test/10.11_dbg_san/sql/sql_parse.cc:6463:12

    #19 0x6288d7d65479 in mysql_execute_command(THD*, bool) /test/10.11_dbg_san/sql/sql_parse.cc:4042:12

    #20 0x6288d7d36634 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/10.11_dbg_san/sql/sql_parse.cc:8223:18

    #21 0x6288d7d2f6d5 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/10.11_dbg_san/sql/sql_parse.cc:1924:7

    #22 0x6288d7d37dc8 in do_command(THD*, bool) /test/10.11_dbg_san/sql/sql_parse.cc:1434:17

    #23 0x6288d848576c in do_handle_one_connection(CONNECT*, bool) /test/10.11_dbg_san/sql/sql_connect.cc:1475:11

    #24 0x6288d8485041 in handle_one_connection /test/10.11_dbg_san/sql/sql_connect.cc:1387:5

    #25 0x6288d6bf237c in asan_thread_start(void*) crtstuff.c

    #26 0x7bf3c569caa3 in start_thread nptl/pthread_create.c:447:8

    #27 0x7bf3c5729c6b in clone3 misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

SUMMARY: UndefinedBehaviorSanitizer: signed-integer-overflow /test/10.11_dbg_san/strings/ctype-simple.c:673:22 

Compiled with a recent version of Clang and LLVM. Ubuntu instructions for Clang/LLVM 18:

  # Note: It is strongly recommended to uninstall all old Clang & LLVM packages (ref  dpkg --list | grep -iE 'clang|llvm'  and use  apt purge  and  dpkg --purge  to remove the packages), before installing Clang/LLVM 18

     sudo apt install clang llvm-18 llvm-18-linker-tools llvm-18-runtime llvm-18-tools llvm-18-dev libstdc++-14-dev llvm-dev lld-18

Compiled with: "-DCMAKE_C_COMPILER=/usr/bin/clang -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DCMAKE_C{,XX}_FLAGS='-march=native -mtune=native'" and:

    -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON -DWITH_UBSAN=ON -DWSREP_LIB_WITH_ASAN=ON

Set before execution:

    export UBSAN_OPTIONS=print_stacktrace=1:report_error_type=1   # And you may also want to supress UBSAN startup issues using 'suppressions=UBSAN.filter' in UBSAN_OPTIONS. For an example of UBSAN.filter, which includes current startup issues see: https://github.com/mariadb-corporation/mariadb-qa/blob/master/UBSAN.filter

    Rel    o/d  Build   Commit                                    UniqueID observed             

CS  10.6   dbg  100526  d37e50c6d04c7c27362f1668ae86fa592b94fb23  UBSAN|negation of -X cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself|strings/ctype-simple.c|my_strntoll_8bit|charset_info_st::strntoll|Value_source::Converter_strntoll::Converter_strntoll|Value_source::Converter_strntoll_with_warn::Converter_strntoll_with_warn

CS  10.6   opt  100526  d37e50c6d04c7c27362f1668ae86fa592b94fb23  UBSAN|negation of -X cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself|strings/ctype-simple.c|my_strntoll_8bit|charset_info_st::strntoll|Value_source::Converter_strntoll::Converter_strntoll|Value_source::Converter_strntoll_with_warn::Converter_strntoll_with_warn

CS  10.11  dbg  100526  8721a00dd38dc0aa1514a3b5ca8c95c6e94af1c9  UBSAN|negation of -X cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself|strings/ctype-simple.c|my_strntoll_8bit|charset_info_st::strntoll|Value_source::Converter_strntoll::Converter_strntoll|Value_source::Converter_strntoll_with_warn::Converter_strntoll_with_warn

CS  10.11  opt  100526  8721a00dd38dc0aa1514a3b5ca8c95c6e94af1c9  UBSAN|negation of -X cannot be represented in type 'longlong' (aka 'long long'); cast to an unsigned type to negate this value to itself|strings/ctype-simple.c|my_strntoll_8bit|charset_info_st::strntoll|Value_source::Converter_strntoll::Converter_strntoll|Value_source::Converter_strntoll_with_warn::Converter_strntoll_with_warn

CS  11.4   dbg  100526  f279551013d1319f27344080e2c0758f3959cebf  No bug found                  

CS  11.4   opt  100526  f279551013d1319f27344080e2c0758f3959cebf  No bug found                  

CS  11.8   dbg  100526  04e09010773caf0b302b2933fff3fe95381a5e13  No bug found                  

CS  11.8   opt  100526  04e09010773caf0b302b2933fff3fe95381a5e13  No bug found                  

CS  12.3   dbg  100526  4c371e30f003b601e7485533476208ae27d51937  No bug found                  

CS  12.3   opt  100526  4c371e30f003b601e7485533476208ae27d51937  No bug found                  

CS  13.0   dbg  100526  96b3dd0c34427e9338dda1375575a0e05a7cd267  No bug found                  

CS  13.0   opt  100526  96b3dd0c34427e9338dda1375575a0e05a7cd267  No bug found                  

ES  10.6   dbg  100526  55cfada6c54d1b08f2372adc1369a5e5e76f472d  No bug found                  

ES  10.6   opt  100526  55cfada6c54d1b08f2372adc1369a5e5e76f472d  No bug found                  

ES  11.4   dbg  100526  90f707057d44f1b5c013a0c3672fd12f32ea7085  No bug found                  

ES  11.4   opt  100526  90f707057d44f1b5c013a0c3672fd12f32ea7085  No bug found                  

ES  11.8   dbg  100526  d4fbd664a4514441bb3d9042c0089842ee6fc3c8  No bug found                  

ES  11.8   opt  100526  d4fbd664a4514441bb3d9042c0089842ee6fc3c8  No bug found                  

ES  12.3   dbg  100526  4063148254974421994024b7cc94f6f2a850177d  No bug found                  

ES  12.3   opt  100526  4063148254974421994024b7cc94f6f2a850177d  No bug found