[MDEV-31117] Spider UBSAN runtime error: applying non-zero offset x to null pointer in st_spider_param_string_parse::restore_delims Created: 2023-04-24  Updated: 2023-10-22  Resolved: 2023-08-23

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11, 11.0, 11.1
Fix Version/s: 10.4.32, 10.5.23, 10.6.16, 10.10.7, 10.11.6, 11.0.4, 11.1.3, 11.2.2

Type: Bug Priority: Critical
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Fixed Votes: 0
Labels: UBSAN

Issue Links:
Blocks
blocks MDEV-28856 Spider: Implement more engine-defined... Closed
Relates
relates to MDEV-30576 Include spider error codes in extra/p... Open

 Description    

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
 
CREATE TABLE t (c INT) ENGINE=Spider COMMENT='abc';

Leads to:

11.0.2 8e55d7ea4a2f94ae3f38fdd8785778612d4b1203 (Debug)

 
/test/11.0_dbg_san/storage/spider/spd_table.h:290:9: runtime error: applying non-zero offset 18446744073709551615 to null pointer

11.0.2 8e55d7ea4a2f94ae3f38fdd8785778612d4b1203 (Debug)

 
    #0 0x14bb722376e9 in st_spider_param_string_parse::restore_delims() /test/11.0_dbg_san/storage/spider/spd_table.h:290
 
    #1 0x14bb722376e9 in st_spider_param_string_parse::print_param_error() /test/11.0_dbg_san/storage/spider/spd_table.cc:1681
 
    #2 0x14bb722b12cc in st_spider_param_string_parse::get_next_parameter_head(char*, char**) /test/11.0_dbg_san/storage/spider/spd_table.h:190
 
    #3 0x14bb72252a8a in spider_parse_connect_info(st_spider_share*, TABLE_SHARE*, partition_info*, unsigned int) /test/11.0_dbg_san/storage/spider/spd_table.cc:2203
 
    #4 0x14bb723264b3 in ha_spider::create(char const*, TABLE*, HA_CREATE_INFO*) /test/11.0_dbg_san/storage/spider/ha_spider.cc:8666
 
    #5 0x556d0faa3214 in handler::ha_create(char const*, TABLE*, HA_CREATE_INFO*) /test/11.0_dbg_san/sql/handler.cc:5656
 
    #6 0x556d0faa8a0a in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, st_mysql_const_unsigned_lex_string*, bool) /test/11.0_dbg_san/sql/handler.cc:6125
 
    #7 0x556d0ea04861 in create_table_impl /test/11.0_dbg_san/sql/sql_table.cc:4671
 
    #8 0x556d0ea05ce3 in mysql_create_table_no_lock(THD*, st_ddl_log_state*, st_ddl_log_state*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*) /test/11.0_dbg_san/sql/sql_table.cc:4772
 
    #9 0x556d0ea11551 in mysql_create_table /test/11.0_dbg_san/sql/sql_table.cc:4888
 
    #10 0x556d0ea11551 in Sql_cmd_create_table_like::execute(THD*) /test/11.0_dbg_san/sql/sql_table.cc:12479
 
    #11 0x556d0e38f189 in mysql_execute_command(THD*, bool) /test/11.0_dbg_san/sql/sql_parse.cc:6003
 
    #12 0x556d0e398aa8 in mysql_parse(THD*, char*, unsigned int, Parser_state*) /test/11.0_dbg_san/sql/sql_parse.cc:7999
 
    #13 0x556d0e3a883c in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) /test/11.0_dbg_san/sql/sql_parse.cc:1894
 
    #14 0x556d0e3b6641 in do_command(THD*, bool) /test/11.0_dbg_san/sql/sql_parse.cc:1407
 
    #15 0x556d0ed7a91b in do_handle_one_connection(CONNECT*, bool) /test/11.0_dbg_san/sql/sql_connect.cc:1416
 
    #16 0x556d0ed7be36 in handle_one_connection /test/11.0_dbg_san/sql/sql_connect.cc:1318
 
    #17 0x14bb957fbb42 in start_thread nptl/pthread_create.c:442
 
    #18 0x14bb9588d9ff  (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff)

Setup:

Compiled with GCC >=7.5.0 (I use GCC 11.3.0) and:
 
    -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON -DWITH_UBSAN=ON -DWITH_RAPID=OFF -DWSREP_LIB_WITH_ASAN=ON
 
Set before execution:
 
    export UBSAN_OPTIONS=print_stacktrace=1

Bug confirmed present in:
MariaDB: 10.4.29 (dbg), 10.4.29 (opt), 10.5.20 (dbg), 10.5.20 (opt), 10.6.13 (dbg), 10.6.13 (opt), 10.7.8 (dbg), 10.7.8 (opt), 10.8.8 (dbg), 10.8.8 (opt), 10.9.6 (dbg), 10.9.6 (opt), 10.10.4 (dbg), 10.10.4 (opt), 10.11.3 (dbg), 10.11.3 (opt), 11.0.2 (dbg), 11.0.2 (opt), 11.1.0 (dbg), 11.1.0 (opt)

 Comments   
Comment by Roel Van de Paar [ 2023-04-24 ]

Interestingly, in 10.4 only, the debug and optimized stacks are missing function names in the leading frames:

10.4.29 ed2adc8c6f986f7e9c81d7a99f85cad0e2d46d80 (Optimized, UBASAN)

 
/test/10.4_opt_san/storage/spider/spd_table.h:290:9: runtime error: applying non-zero offset 18446744073709551615 to null pointer
 
    #0 0x14d5c97a8b0f  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-opt/lib/plugin/ha_spider.so+0x76ab0f)
 
    #1 0x14d5c97c7eab  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-opt/lib/plugin/ha_spider.so+0x789eab)
 
    #2 0x14d5c9897153  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-opt/lib/plugin/ha_spider.so+0x859153)
 
    #3 0x55a1c25f3b7e in handler::ha_create(char const*, TABLE*, HA_CREATE_INFO*) /test/10.4_opt_san/sql/handler.cc:4840
 
    #4 0x55a1c25f8487 in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, st_mysql_const_unsigned_lex_string*) /test/10.4_opt_san/sql/handler.cc:5308
 
    #5 0x55a1c190445f in create_table_impl /test/10.4_opt_san/sql/sql_table.cc:5183
 
    #6 0x55a1c1906f39 in mysql_create_table_no_lock(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*) /test/10.4_opt_san/sql/sql_table.cc:5267
 
    #7 0x55a1c1907f31 in mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*) /test/10.4_opt_san/sql/sql_table.cc:5362
 
    #8 0x55a1c19247f5 in Sql_cmd_create_table_like::execute(THD*) /test/10.4_opt_san/sql/sql_table.cc:11747
 
    #9 0x55a1c13ffb83 in mysql_execute_command(THD*) /test/10.4_opt_san/sql/sql_parse.cc:6206
 
    #10 0x55a1c142ea26 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /test/10.4_opt_san/sql/sql_parse.cc:7998
 
    #11 0x55a1c143b302 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /test/10.4_opt_san/sql/sql_parse.cc:1857
 
    #12 0x55a1c1448949 in do_command(THD*) /test/10.4_opt_san/sql/sql_parse.cc:1378
 
    #13 0x55a1c1bba0b9 in do_handle_one_connection(CONNECT*) /test/10.4_opt_san/sql/sql_connect.cc:1420
 
    #14 0x55a1c1bba97c in handle_one_connection /test/10.4_opt_san/sql/sql_connect.cc:1324
 
    #15 0x14d60031fb42 in start_thread nptl/pthread_create.c:442
 
    #16 0x14d6003b19ff  (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff)

10.4.29 ed2adc8c6f986f7e9c81d7a99f85cad0e2d46d80 (Debug, UBASAN)

 
/test/10.4_dbg_san/storage/spider/spd_table.h:290:9: runtime error: applying non-zero offset 18446744073709551615 to null pointer
 
    #0 0x145f57b2c64f  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-dbg/lib/plugin/ha_spider.so+0x78764f)
 
    #1 0x145f57ba0d4e  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-dbg/lib/plugin/ha_spider.so+0x7fbd4e)
 
    #2 0x145f57b4603e  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-dbg/lib/plugin/ha_spider.so+0x7a103e)
 
    #3 0x145f57c13f9e  (/test/UBASAN_MD070423-mariadb-10.4.29-linux-x86_64-dbg/lib/plugin/ha_spider.so+0x86ef9e)
 
    #4 0x5599cc7559a6 in handler::ha_create(char const*, TABLE*, HA_CREATE_INFO*) /test/10.4_dbg_san/sql/handler.cc:4840
 
    #5 0x5599cc75b0f2 in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, st_mysql_const_unsigned_lex_string*) /test/10.4_dbg_san/sql/handler.cc:5308
 
    #6 0x5599cb8fa067 in create_table_impl /test/10.4_dbg_san/sql/sql_table.cc:5183
 
    #7 0x5599cb8fb284 in mysql_create_table_no_lock(THD*, st_mysql_const_lex_string const*, st_mysql_const_lex_string const*, Table_specification_st*, Alter_info*, bool*, int, TABLE_LIST*) /test/10.4_dbg_san/sql/sql_table.cc:5267
 
    #8 0x5599cb8fc757 in mysql_create_table(THD*, TABLE_LIST*, Table_specification_st*, Alter_info*) /test/10.4_dbg_san/sql/sql_table.cc:5362
 
    #9 0x5599cb9073e3 in Sql_cmd_create_table_like::execute(THD*) /test/10.4_dbg_san/sql/sql_table.cc:11747
 
    #10 0x5599cb38254e in mysql_execute_command(THD*) /test/10.4_dbg_san/sql/sql_parse.cc:6206
 
    #11 0x5599cb390367 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /test/10.4_dbg_san/sql/sql_parse.cc:7998
 
    #12 0x5599cb3a00aa in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /test/10.4_dbg_san/sql/sql_parse.cc:1857
 
    #13 0x5599cb3ae350 in do_command(THD*) /test/10.4_dbg_san/sql/sql_parse.cc:1378
 
    #14 0x5599cbbeb0e0 in do_handle_one_connection(CONNECT*) /test/10.4_dbg_san/sql/sql_connect.cc:1420
 
    #15 0x5599cbbeb6b0 in handle_one_connection /test/10.4_dbg_san/sql/sql_connect.cc:1324
 
    #16 0x145f8da15b42 in start_thread nptl/pthread_create.c:442
 
    #17 0x145f8daa79ff  (/lib/x86_64-linux-gnu/libc.so.6+0x1269ff)

Comment by Roel Van de Paar [ 2023-05-20 ]

Similar outcome with this testcase

INSTALL PLUGIN Spider SONAME 'ha_spider.so';
 
ALTER TABLE mysql.help_topic ENGINE=Spider;

And interesting:

11.0.2 368dd22a816f3b437bccd0b9ff28b9de9b1abf0a (Debug, UBASAN)

 
11.0.2-dbg>ALTER TABLE mysql.help_topic ENGINE=Spider;
 
ERROR 12501 (HY000): The connect info 'help topics' is invalid

Comment by Yuchen Pei [ 2023-05-25 ]

With the testcase in the description, at 10.4 9f909e546e14ed9b529cddc2c1a5c2aa61d57e56 and 11.0 8e55d7ea4a2f94ae3f38fdd8785778612d4b1203, I get

mysqltest: At line 17: query 'CREATE TABLE t (c INT) ENGINE=Spider COMMENT='abc'' failed: 12501: The connect info 'abc' is invalid

With the other testcase that has ALTER TABLE mysql.help_topic ENGINE=Spider;, at 11.0 8e55d7ea4a2f94ae3f38fdd8785778612d4b1203 I get 

mysqltest: At line 18: query 'ALTER TABLE mysql.help_topic ENGINE=Spider' failed: <Unknown> (12501): The connect info 'help topics' is invalid

Comment by Roel Van de Paar [ 2023-05-27 ]

Yes, these are the outcomes in the CLI (or MTR without changes).
To get the UBSAN error in MTR we need to "cater" for the error, as follows:

IF (`SELECT NOT(COUNT(*)) FROM information_schema.system_variables WHERE variable_name='have_sanitizer' AND global_value LIKE "%UBSAN%"`)
 
{
 
--skip Test needs to be run with an UBSAN build
 
}
 
INSTALL PLUGIN Spider SONAME 'ha_spider.so';
 
--error 12501
 
CREATE TABLE t (c INT) ENGINE=Spider COMMENT='abc';

Leads to:

11.0.2 368dd22a816f3b437bccd0b9ff28b9de9b1abf0a (Debug, UBASAN)

 
main.test                                [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2023-05-27 14:42:36
 
line
 
/test/11.0_dbg_san/storage/spider/spd_table.h:290:9: runtime error: applying non-zero offset 18446744073709551615 to null pointer
 
^ Found warnings in /test/UBASAN_MD270523-mariadb-11.0.2-linux-x86_64-dbg/mysql-test/var/log/mysqld.1.err

Interesting here is that the error is "Unknown" (so I had to use the numerical to get it to work, this is the original output before the fix):

11.0.2 368dd22a816f3b437bccd0b9ff28b9de9b1abf0a (Debug, UBASAN)

 
mysqltest: At line 2: query 'CREATE TABLE t (c INT) ENGINE=Spider COMMENT='abc'' failed: <Unknown> (12501): The connect info 'abc' is invalid

It would be good to resolve the "Unknown" as par of the fix. I assume this is another example of MDEV-30576, as:

11.0.2 368dd22a816f3b437bccd0b9ff28b9de9b1abf0a (Debug, UBASAN)

 
$ ./bin/perror 12501
 
Illegal error code: 12501

The other testcase:

IF (`SELECT NOT(COUNT(*)) FROM information_schema.system_variables WHERE variable_name='have_sanitizer' AND global_value LIKE "%UBSAN%"`)
 
{
 
--skip Test needs to be run with an UBSAN build
 
}
 
INSTALL PLUGIN Spider SONAME 'ha_spider.so';
 
--error 12501
 
ALTER TABLE mysql.help_topic ENGINE=Spider;

Same outcome.

Comment by Yuchen Pei [ 2023-06-16 ]

A crude fix: https://github.com/MariaDB/server/commit/025ad07aae1

But the spider connection info parser itself does not look very healthy. It is poorly written and documented, and should be cleaned up.

Comment by Yuchen Pei [ 2023-06-19 ]

So the invalid connection string "help topics" comes from the comment
of the system table mysql.help_topic, which is created in a bootstrap
script scripts/mysql_system_tables.sql, as are other system
tables. Just shows you how confusing having the options in the comment
can be... A reason why we need something like MDEV-31146

Comment by Roel Van de Paar [ 2023-06-26 ]

An additional testcase with a different stack

SET sql_mode='';
 
INSTALL PLUGIN Spider SONAME 'ha_spider.so';
 
SET SESSION enforce_storage_engine=Spider;
 
CREATE TABLE t (c BINARY KEY) COMMENT='ENGINE "Spider"';

Leads to UniqueID/stack:

UBSAN|applying non-zero offset X to null pointer|storage/spider/spd_table.h|st_spider_param_string_parse::restore_delims|st_spider_param_string_parse::print_param_error|spider_parse_connect_info|ha_spider::create

Comment by Yuchen Pei [ 2023-07-05 ]

Hi holyfoot, ptal thanks:

https://github.com/MariaDB/server/commit/1c997a3d07fda1a81baf39df89ef011436a71740

The parsing logic that caused the issue in this ticket was a mess. So my fix involves a cleanup of the parser. In the future we may remove the spider parsing mechanism altogether in favour of engine-defined options (MDEV-28856 etc.), but it will probably be a long time before that happens, and it will probably be for higher versions rather than 10.4+ in this ticket.

This above commit is based on 11.0. I also have a commit based on 10.4[1] which is not qualitatively different, and a review of the 11.0 version should be sufficient.

[1] https://github.com/MariaDB/server/commit/e53ad7275ed

Comment by Alexey Botchkov [ 2023-08-16 ]

ok to push.

Comment by Yuchen Pei [ 2023-08-23 ]

Thanks for the review.

Pushed e9f3ca612528c5f917e27ef6113fd1deda2aef26 to 10.4

before pushing: tested on 10.4-6,10.10-11.2,ES-10.4,10.6,23.06

after merge changes:

  • 10.4->10.5 bcb5c31f0c7ee0b83b712f55c208f768e4ee6dac
  • 10.4->ES-10.4 e5952cf057c7712df0587121f977b1529d086d3a
  • 10.6->10.10 dc5be8988afa88ca39a36cd9d2219e0eac9e63f3
Generated at Thu Feb 08 10:21:25 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.