[MDEV-26541] Undefined symbol: _ZTI12ha_partition when attempting to use ha_spider.so in UBSAN builds Created: 2021-09-05  Updated: 2023-07-04  Resolved: 2023-01-23

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - Spider
Affects Version/s: 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.10, 10.11
Fix Version/s: 10.11.2, 11.0.1, 10.4.28, 10.5.19, 10.6.12, 10.7.8, 10.8.7, 10.9.5, 10.10.3

Type: Bug Priority: Major
Reporter: Roel Van de Paar Assignee: Yuchen Pei
Resolution: Fixed Votes: 0
Labels: UBSAN, affects-tests, not-10.3

Issue Links:
Blocks
Problem/Incident
causes MDEV-31421 spider/bugfix.mdev_26541 fails post-t... Closed
Relates
relates to MDEV-30191 SIGSEGV & heap-use-after-free in spid... Closed

 Description   

This bug blocks ASAN/UBSAN testing of the Spider Engine.

Steps to reproduce:
Build server as ASAN/UBSAN build. Attempt to load Spider engine.

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Debug)

 
10.7.0-dbg>INSTALL PLUGIN spider SONAME 'ha_spider.so';
 
ERROR 1126 (HY000): Can't open shared library '/test/UBASAN_MD060921-mariadb-10.7.0-linux-x86_64-dbg/lib/plugin/ha_spider.so' (errno: 2, undefined symbol: _ZTI12ha_partition)

The file is there:

$ ls -lh /test/UBASAN_MD060921-mariadb-10.7.0-linux-x86_64-dbg/lib/plugin/ha_spider.so
 
-rw-r--r-- 1 roel roel 39M Sep  6 06:00 /test/UBASAN_MD060921-mariadb-10.7.0-linux-x86_64-dbg/lib/plugin/ha_spider.so


$ cat BUILD_CMD_CMAKE 
cmake . -DWITH_SSL=bundled -DCMAKE_BUILD_TYPE=Debug -DBUILD_CONFIG=mysql_release -DWITH_TOKUDB=0 -DWITH_JEMALLOC=no -DFEATURE_SET=community -DDEBUG_EXTNAME=OFF -DWITH_EMBEDDED_SERVER=0 -DENABLE_DOWNLOADS=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=/tmp/boost_805116 -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DWITH_PERFSCHEMA_STORAGE_ENGINE=1 -DWITH_ZLIB=bundled -DWITH_ROCKSDB=1 -DWITH_PAM=ON -DFORCE_INSOURCE_BUILD=1 -DWITH_ASAN=ON -DWITH_ASAN_SCOPE=ON -DWITH_UBSAN=ON -DWSREP_LIB_WITH_ASAN=ON -DCMAKE_CXX_FLAGS=-static-libasan -DMYSQL_MAINTAINER_MODE=OFF

Present in 10.5 and 10.7. Likely same in older versions also.

 Comments   
Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-07 ]

I confirm that the bug is reproducible on 10.4 and10.5 but not on 10.3.

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-07 ]

This seems to be a known issue because BUILD/compile-pentium64-ubsan says as the following:

We also have to compile without Spider as linking with Spider library does
not work. (errno: 11, undefined symbol: _ZTI12ha_partition)

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-09 ]
  • No error
    • cmake .. -DWITH_ASAN=ON
  • Error
    • cmake .. -DWITH_ASAN=ON -DWITH_UBSAN=ON
Comment by Roel Van de Paar [ 2021-09-09 ]

nayuta-yanagisawa That is interesting. Standard MariaDB builds allow both flags/building with both enabled. I will check on my end to confirm your findings.

Comment by Roel Van de Paar [ 2021-09-10 ]

nayuta-yanagisawa Confirming that the workaround works.

I also tested the reverse: -DWITH_UBSAN=ON only (without any ASAN options), and the issue on that one is the same:

10.7.0 1bc82aaf0a7746c0921a94034aff2d51f0d75cd0 (Debug)

 
10.7.0-dbg>INSTALL PLUGIN spider SONAME 'ha_spider.so';
 
ERROR 1126 (HY000): Can't open shared library '/test/UBASAN_MD100921-mariadb-10.7.0-linux-x86_64-dbg/lib/plugin/ha_spider.so' (errno: 2, undefined symbol: _ZTI12ha_partition)

The issue is thus limited to UBSAN only. Updated title.

Comment by Roel Van de Paar [ 2021-09-10 ]

So thinking about this, perhaps an #include is missing or similar? Specifically partitioning (given _ZTI12ha_partition) in Spider (ha_spider)?

Comment by Nayuta Yanagisawa (Inactive) [ 2021-09-10 ]

Roel Thank you for your confirmation and insight. I'm still investigating why WITH_UNSAN does not work.

Comment by Roel Van de Paar [ 2022-12-10 ]

I increased prio as the issue blocks a fair bit of Spider testing (UBSAN continues to fail working):

10.5.19 851816532b39b4bf04b1d352cf3c28929ec99cf1 (Debug, UBSAN)

 
ERROR 1126 (HY000) at line 2 in file: 't11': Can't open shared library '/test/UBSAN_SPIDER_MD101222-mariadb-10.5.19-linux-x86_64-dbg/lib/plugin/ha_spider.so' (errno: 2, undefined symbol: _ZTI12ha_partition)

serg Do you have any ideas/insights on perhaps missing lib/header? Thank you

Comment by Nayuta Yanagisawa (Inactive) [ 2022-12-19 ]

I believe that this has been started from https://github.com/MariaDB/server/commit/2400e06946bb5856684fd27ee86f610267bf4dc3.

Comment by Nayuta Yanagisawa (Inactive) [ 2022-12-19 ]

This is likely caused by the mixture of -fno-rtti and -frtti binaries. In fact, the following patch fixes the problem.

diff --git a/storage/spider/CMakeLists.txt b/storage/spider/CMakeLists.txt
 
index 706b11ac141..0eb2140fe4b 100644
 
--- a/storage/spider/CMakeLists.txt
 
+++ b/storage/spider/CMakeLists.txt
 
@@ -78,3 +78,7 @@ IF(MSVC)
 
                        COMMAND copy Debug\\ha_spider.dll ..\\..\\sql\\lib\\plugin\\ha_spider.dll)
 
   ENDIF()
 
 ENDIF()
 
+
 
+if(WITH_UBSAN)
 
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-rtti")
 
+endif()

I've not checked the detail, but WITH_UBSAN might magically disable RTTI (??).

Comment by Roel Van de Paar [ 2022-12-19 ]

nayuta-yanagisawa Thank you

Comment by Yuchen Pei [ 2022-12-21 ]

mtr testcase:

--echo #
 
--echo # MDEV-26541 Undefined symbol: _ZTI12ha_partition when attempting to use ha_spider.so in UBSAN builds
 
--echo #
 
 
 
if (`select not(count(*)) from information_schema.system_variables where variable_name='have_sanitizer' and global_value="UBSAN"`)
 
{
 
--skip test needs to be run with UBSAN
 
}
 
 
 
# init spider
 
 
 
INSTALL PLUGIN spider SONAME 'ha_spider.so';
 
 
 
let $PLUGIN_NAME= spider_flush_table_mon_cache;
 
let $PLUGIN_EXIST=
 
  `SELECT COUNT(*) FROM mysql.func WHERE name = '$PLUGIN_NAME'`;
 
while (!$PLUGIN_EXIST)
 
{
 
  let $PLUGIN_EXIST=
 
    `SELECT COUNT(*) FROM mysql.func WHERE name = '$PLUGIN_NAME'`;
 
}
 
 
 
# deinit spider
 
 
 
DROP FUNCTION spider_flush_table_mon_cache;
 
DROP FUNCTION spider_copy_tables;
 
DROP FUNCTION spider_ping_table;
 
DROP FUNCTION spider_bg_direct_sql;
 
DROP FUNCTION spider_direct_sql;
 
UNINSTALL PLUGIN spider_alloc_mem;
 
UNINSTALL PLUGIN spider;
 
DROP TABLE IF EXISTS mysql.spider_xa;
 
DROP TABLE IF EXISTS mysql.spider_xa_member;
 
DROP TABLE IF EXISTS mysql.spider_xa_failed_log;
 
DROP TABLE IF EXISTS mysql.spider_tables;
 
DROP TABLE IF EXISTS mysql.spider_link_mon_servers;
 
DROP TABLE IF EXISTS mysql.spider_link_failed_log;
 
DROP TABLE IF EXISTS mysql.spider_table_position_for_recovery;
 
DROP TABLE IF EXISTS mysql.spider_table_sts;
 
DROP TABLE IF EXISTS mysql.spider_table_crd;

Comment by Yuchen Pei [ 2022-12-22 ]

It seems this issue cannot be directly analysed by a debugger, as the traces are identical with or without ubsan.

The server binary is built with rtti, which can be verified using the method mentioned in <https://stackoverflow.com/questions/22150806/how-can-i-check-if-a-library-was-compiled-with-fno-rtti>.

Without -fno-rtti for spider, we get 

$ nm /home/ycp/source/mariadb-server/mdev-26541/build/mysql-test/var/plugins/ha_spider.so
 
...
 
0000000000b4a488 V _ZTI10Item_const
 
                 U _ZTI10Item_field
 
                 U _ZTI10Item_ident
 
                 U _ZTI11Item_string
 
                 U _ZTI11Query_arena
 
                 U _ZTI12ha_partition
 
...
 
$ nm -C /home/ycp/source/mariadb-server/mdev-26541/build/mysql-test/var/plugins/ha_spider.so
 
...
 
0000000000b4a488 V typeinfo for Item_const
 
                 U typeinfo for Item_field
 
                 U typeinfo for Item_ident
 
                 U typeinfo for Item_string
 
                 U typeinfo for Query_arena
 
                 U typeinfo for ha_partition
 
...
 
$ nm -C /home/ycp/source/mariadb-server/mdev-26541/build/sql/mysqld
 
...
 
0000000007beb3d8 d typeinfo for ha_partition
 
...
 
$ nm sql/mysqld | grep 0000000007beb3d8
 
0000000007beb3d8 d _ZTI12ha_partition

With -fno-rtti for spider, as with the stackoverflow post, there's no typeinfo for ha_partition, or anything for ha_partition:

$ nm /home/ycp/source/mariadb-server/mdev-26541/build/mysql-test/var/plugins/ha_spider.so | grep ha_partition
 
0000000000782488 W _ZN12ha_partition18get_child_handlersEv
 
 
 
$ nm -C /home/ycp/source/mariadb-server/mdev-26541/build/mysql-test/var/plugins/ha_spider.so | grep ha_partition
 
0000000000782488 W ha_partition::get_child_handlers()

Furthermore, with the -fno-rtti and ubsan build for spider, we get test failures which
disappears with the --nowarnings mtr flag:

$ ./mysql-test/mtr spider.basic_sql
 
...
 
worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
 
spider.basic_sql                         [ fail ]  Found warnings/errors in server log file!
 
        Test ended at 2022-12-21 18:56:23
 
line
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/handler.cc:320:17: runtime error: member call on address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/handler.h:3168:36: runtime error: member call on address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/handler.h:3168:5: runtime error: member access within address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/sql_table.cc:3463:39: runtime error: member call on address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/handler.h:3922:12: runtime error: member call on address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
/home/ycp/source/mariadb-server/mdev-26541/src/sql/sql_table.cc:3497:61: runtime error: member call on address 0x7f54e8016ce8 which does not point to an object of type 'handler'
 
...

These seem to be errors produced by ubsan, cf <https://jira.mariadb.org/browse/MDEV-20923>. If we use Nayuta's patch, we will need to fix these errors. OTOH I wonder whether the tests will pass if we can find a fix for this issue without disabling rtti.

Comment by Marko Mäkelä [ 2023-01-11 ]

The following change allows the _ZTI12ha_partition to be exported as a global symbol in mariadbd:

diff --git a/cmake/libutils.cmake b/cmake/libutils.cmake
 
index 74853c36a74..a6d2628fffc 100644
 
--- a/cmake/libutils.cmake
 
+++ b/cmake/libutils.cmake
 
@@ -344,11 +344,6 @@ SET(VISIBILITY_HIDDEN_FLAG)
 
 
 IF(CMAKE_C_COMPILER_ID MATCHES "SunPro")
 
   SET(VISIBILITY_HIDDEN_FLAG "-xldscope=hidden")
 
-ELSEIF(UNIX)
 
-  CHECK_C_COMPILER_FLAG("-fvisibility=hidden" HAVE_VISIBILITY_HIDDEN)
 
-  IF(HAVE_VISIBILITY_HIDDEN)
 
-    SET(VISIBILITY_HIDDEN_FLAG "-fvisibility=hidden")
 
-  ENDIF()
 
 ENDIF()
 
 
 # We try to hide the symbols in bundled libraries to avoid name clashes with

The proper way to fix this would be to apply something like the

  IF(VISIBILITY_HIDDEN_FLAG AND TARGET wsrep)

tweak in sql/CMakeLists.txt to other files.

With that sorted out, several Spider tests would fail due to one error:

diff --git a/storage/spider/spd_conn.cc b/storage/spider/spd_conn.cc
 
index 80d2489185b..a0b36dbe0b6 100644
 
--- a/storage/spider/spd_conn.cc
 
+++ b/storage/spider/spd_conn.cc
 
@@ -648,8 +648,8 @@ SPIDER_CONN *spider_create_conn(
 
       share->tgt_usernames_lengths[link_idx]);
 
     conn->tgt_password_length = share->tgt_passwords_lengths[link_idx];
 
     conn->tgt_password = tmp_password;
 
-    memcpy(conn->tgt_password, share->tgt_passwords[link_idx],
 
-      share->tgt_passwords_lengths[link_idx]);
 
+    if (size_t s= share->tgt_passwords_lengths[link_idx])
 
+      memcpy(conn->tgt_password, share->tgt_passwords[link_idx], s);
 
     conn->tgt_socket_length = share->tgt_sockets_lengths[link_idx];
 
     conn->tgt_socket = tmp_socket;
 
     memcpy(conn->tgt_socket, share->tgt_sockets[link_idx],

A number of tests still fail with that patch due to other nonnull violations in storage/spider/spd_trx.cc:

10.6 56c9b0bca0576985c31f20b46dcb060a01e81a2b with the above patch

 
Completed: Failed 3/40 tests, 92.50% were successful.
 
 
 
Failing test(s): spider.ha_part spider.auto_increment spider.ha

Comment by Yuchen Pei [ 2023-01-12 ]

Thanks marko very much for unblocking this ticket.

holyfoot: I've created a patch at https://github.com/MariaDB/server/commit/b4d5a4f46c2, PTAL thanks.

Comment by Yuchen Pei [ 2023-01-13 ]

marko holyfoot updated patch incorporating marko's comment at https://github.com/MariaDB/server/commit/af2de138303

Comment by Alexey Botchkov [ 2023-01-19 ]

Ok to push.
Minor comment on Slack.

Generated at Thu Feb 08 09:46:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.