Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
5.5(EOL), 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5
-
Debian GNU/Linux unstable, clang 8.0.1
Description
MemorySanitizer is a compile-time instrumentation layer in clang but not GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
No patches are necessary since 10.5 94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157 (see the commit message for instructions); cmake -DWITH_MSAN=ON is supposed to work ‘out of the box’.
This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems.
How to instrumented libraries
These instructions are for clang-10. The script build-msan2.sh was developed to resolve
MDEV-22083 a.k.a. MDEV-26758.
mkdir /tmp/build
|
cd /tmp/build
|
mkdir "$HOME/msan-libs"
|
CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh
|
Note: to use different clang (tested with clang-8, clang-9, clang-11, clang-13), just replace 10 with the major version of the compiler above.
For clang-15, the procedure is a little different:
mkdir /tmp/build
|
cd /tmp/build
|
mkdir "$HOME/msan-libs"
|
CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh
|
How to build MariaDB Server 10.5 or later with the instrumented libraries
cd /mariadb/10.5 |
mkdir build |
cd build |
cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \
|
-DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ |
-DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ |
-DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \
|
-DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \
|
-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
|
-DWITH_SAFEMALLOC=OFF \
|
-DWITH_{ZLIB,SSL,PCRE}=bundled \
|
-DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \
|
-DWITH_MSAN=ON \
|
-G Ninja ..
|
ninja
|
Note: -march=native -mtune=native is optional since the second fix of MDEV-20386
How to build with minimal cmake arguments
cd /mariadb/10.5 |
mkdir build |
cd build |
cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ |
-DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ |
-DSECURITY_HARDENED=OFF \
|
-DPLUGIN_{CONNECT,SPIDER}=NO \
|
-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \
|
-DWITH_{ZLIB,SSL,PCRE}=bundled \
|
-DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \
|
-DWITH_MSAN=ON -G Ninja ..
|
cmake --build .
|
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with -fsanitize=memory. Connect and Spider are disabled due to test failures that were not investigated yet. MDEV-34921 was tested in this way.
Note the -DSECURITY_HARDENED=OFF; it is enabled by default and seems to break operations like memcpy() with RelWithDebInfo but not Debug.
How to run tests
cd mysql-test
|
LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st
|
LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file
|
Note: It may be wise to omit MSAN_OPTIONS=abort_on_error=1 except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages.
Note: The llvm-symbolizer in clang 14 or later will refuse to load if LD_LIBRARY_PATH includes an MSAN-instrumented libgmp.so. To get nice resolved stack traces, you must point the environment variable MSAN_SYMBOLIZER_PATH to a script like the following. The script name had better start with llvm-symbolizer- in order to avoid a warning:
#!/bin/sh
|
unset LD_LIBRARY_PATH
|
exec llvm-symbolizer-15 "$@"
|
The MSAN_OPTIONS=poison_in_dtor=0 (to work around MDEV-30936, MDEV-30942) is an old option that was enabled by default in clang 15.
Attachments
Issue Links
- blocks
-
MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively
-
- Closed
-
- includes
-
CONC-762 mysql_stmt_fetch in cursor mode creates bind.length unitialized (MSAN)
-
- Open
-
-
MDBF-136 MSAN - create instrumented libcrack
-
- Closed
-
-
MDEV-20386 Replace inline asm with compiler-builtin intrinsic functions
-
- Closed
-
-
MDEV-20388 Allegedly uninitialized values left by MyCTX_nopad
-
- Closed
-
-
MDEV-22059 MSAN report at replicate_ignore_table_grant
-
- Closed
-
- is blocked by
-
MDEV-22688 MSAN use-of-uninitialized-value in READ_INFO::read_xml()
-
- Closed
-
-
MDEV-22689 MSAN use-of-uninitialized-value in decode_bytes()
-
- Closed
-
-
MDEV-22690 MSAN use-of-uninitialized-value in optimizer_trace
-
- Closed
-
-
MDEV-22691 MSAN use-of-uninitialized-value in test maria.maria-recovery2
-
- Closed
-
-
MDEV-22692 MSAN use-of-uninitialized-value in pagecache_fwrite()
-
- Closed
-
- relates to
-
CONC-483 Workaround for MSAN bug needed
-
- Closed
-
-
MDEV-20309 InnoDB encryption accesses memory outside of allocated block
-
- Closed
-
-
MDEV-22070 MSAN use-of-uninitialized-value in encryption.innodb-redo-badkey
-
- Closed
-
-
MDEV-22071 MSAN use-of-uninitialized-value in build_frm_image()
-
- Closed
-
-
MDEV-22072 MSAN use-of-uninitialized-value in wf_incremental_finalize
-
- Open
-
-
MDEV-22073 MSAN use-of-uninitialized-value in collect_statistics_for_table()
-
- Closed
-
-
MDEV-22244 Field::error_generated_column_function_is_not_allowed: Conditional jump or move depends on uninitialised value
-
- Open
-
-
MDEV-24514 WITH_MSAN is disabling WOLFSSL_AESNI acceleration
-
- Closed
-
-
MDEV-26758 Make libmariadb run under MSAN
-
- Closed
-
-
MDEV-27964 The function ENCRYPT() causes SIGSEGV in WITH_MSAN builds
-
- Open
-
-
MDEV-30942 MSAN_OPTIONS=poison_in_dtor=1 causes failures in free_root()
-
- Open
-
-
MDEV-31790 Extremely slow tests rpl.rpl_non_direct_mixed_mixing_engines and rpl.rpl_stm_mixing_engines
-
- Open
-
-
MDEV-34921 MemorySanitizer reports errors for non-debug builds
-
- Closed
-
-
MDEV-34933 The test plugins.rpl_auth uses not_msan.inc without a good reason
-
- Open
-
-
MDEV-36482 cmake -DWITH_MSAN=ON does not work with asynchronous I/O
-
- In Review
-
-
MDEV-14024 PCRE2
-
- Closed
-
-
MDEV-20310 valgrind bugs found in 10.5
-
- Closed
-
-
MDEV-20383 Use of uninitialized value in Datafile::find_space_id() for ROW_FORMAT=COMPRESSED
-
- Closed
-
-
MDEV-22060 MSAN use-of-uninitialized-value in main.query_cache_innodb
-
- Closed
-
-
MDEV-24841 Build error with MSAN use-of-uninitialized-value comp_err.c:734:10
-
- Closed
-
-
MDEV-34996 Buildbot MSAN options in buildbot rather than server
-
- Closed
-
Activity
Field | Original Value | New Value |
---|---|---|
Link |
This issue relates to |
Link |
This issue relates to |
Link |
This issue relates to |
Link |
This issue includes |
Link |
This issue includes |
Link |
This issue relates to |
Attachment | 10.5-msan.patch [ 50302 ] |
Fix Version/s | 10.5 [ 23123 ] |
Link | This issue blocks MENT-314 [ MENT-314 ] |
Link | This issue relates to MENT-210 [ MENT-210 ] |
Link |
This issue includes |
Link |
This issue relates to |
Link |
This issue relates to |
Link | This issue relates to MDEV-22072 [ MDEV-22072 ] |
Link |
This issue relates to |
Link |
This issue is blocked by |
Link | This issue relates to MDEV-22244 [ MDEV-22244 ] |
Fix Version/s | 10.5 [ 23123 ] |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Link |
This issue is blocked by |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
h2. Creating MSAN-instrumented libc++ {code:sh} cd /mariadb sudo apt source libc++-8-dev cd llvm-toolchain-8-8.0.1 mkdir libc++msan; cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-8 -DCMAKE_CXX_COMPILER=clang++-8 {code} h2. Introduce an option WITH_MSAN {code:diff} patch -p1<<'EOF' diff --git a/CMakeLists.txt b/CMakeLists.txt index 33b69a9a1e3..ed61853bbc5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -236,6 +236,11 @@ IF (WITH_UBSAN) MY_CHECK_AND_SET_COMPILER_FLAG("-fsanitize=undefined -fno-sanitize=alignment -U_FORTIFY_SOURCE" DEBUG RELWITHDEBINFO) ENDIF() +OPTION(WITH_MSAN "Enable memory sanitizer" OFF) +IF (WITH_MSAN) + MY_CHECK_AND_SET_COMPILER_FLAG("-fsanitize=memory -U_FORTIFY_SOURCE" DEBUG RELWITHDEBINFO) +ENDIF() + IF(NOT WITH_TSAN) # enable security hardening features, like most distributions do # in our benchmarks that costs about ~1% of performance, depending on the load diff --git a/cmake/plugin.cmake b/cmake/plugin.cmake index 89dfdbb306b..3582f8ac005 100644 --- a/cmake/plugin.cmake +++ b/cmake/plugin.cmake @@ -223,7 +223,7 @@ MACRO(MYSQL_ADD_PLUGIN) ELSEIF(NOT CMAKE_SYSTEM_NAME STREQUAL "Linux") TARGET_LINK_LIBRARIES (${target} mysqld) ENDIF() - ELSEIF(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND NOT WITH_ASAN AND NOT WITH_TSAN AND NOT WITH_UBSAN) + ELSEIF(CMAKE_SYSTEM_NAME STREQUAL "Linux" AND NOT WITH_ASAN AND NOT WITH_TSAN AND NOT WITH_UBSAN AND NOT WITH_MSAN) TARGET_LINK_LIBRARIES (${target} "-Wl,--no-undefined") ENDIF() diff --git a/libmariadb/libmariadb/CMakeLists.txt b/libmariadb/libmariadb/CMakeLists.txt index 9581461..8ba9c32 100644 --- a/libmariadb/libmariadb/CMakeLists.txt +++ b/libmariadb/libmariadb/CMakeLists.txt @@ -412,7 +412,7 @@ ENDIF() IF(CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "GNU") - IF (NOT WITH_ASAN AND NOT WITH_TSAN) + IF (NOT WITH_ASAN AND NOT WITH_TSAN AND NOT WITH_MSAN) TARGET_LINK_LIBRARIES (libmariadb "-Wl,--no-undefined") ENDIF() SET_TARGET_PROPERTIES(libmariadb PROPERTIES LINK_FLAGS "${CC_BINARY_DIR}/libmariadb/mariadbclient.def") EOF {code} h2. Compile with libc++ (instead of libstdc++) and bundled libraries {code:sh} mkdir build; cd build cmake -DWITH_MSAN=ON -DWITH_SSL=bundled -DWITH_ZLIB=bundled -DCMAKE_CXX_FLAGS='-stdlib=libc++' .. make -j$(nproc) {code} h2. Run tests with the instrumented libc++ {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-8-8.0.1/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} h2. Problems found so far For some reason, {{getservbyname()}} claims that the buffer is uninitialized. Maybe we need an instrumented library? Workaround: comment out the calls: {code:diff} diff --git a/sql/mysqld.cc b/sql/mysqld.cc index 695616f9269..f1df81c9095 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -2171,7 +2171,7 @@ static void set_ports() line options. */ -#if MYSQL_PORT_DEFAULT == 0 +#if 0 // MYSQL_PORT_DEFAULT == 0 struct servent *serv_ptr; if ((serv_ptr= getservbyname("mysql", "tcp"))) SYSVAR_AUTOSIZE(mysqld_port, ntohs((u_short) serv_ptr->s_port)); diff --git a/libmariadb/mariadb_lib.c b/libmariadb/mariadb_lib.c index d43b68c..fb6236f 100644 --- a/libmariadb/mariadb_lib.c +++ b/libmariadb/mariadb_lib.c @@ -3539,12 +3539,16 @@ static void mysql_once_init() } if (!mysql_port) { +#if 0 struct servent *serv_ptr; +#endif char *env; mysql_port = MARIADB_PORT; +#if 0 if ((serv_ptr = getservbyname("mysql", "tcp"))) mysql_port = (uint)ntohs((ushort)serv_ptr->s_port); +#endif if ((env = getenv("MYSQL_TCP_PORT"))) mysql_port =(uint)atoi(env); } {code} Inline assembler code leads to bogus claims about uninitialized memory. Compiler built-ins or intrinsic functions seem to work correctly. Alas, we will have to add some compile-time options to allow these instructions to be emitted. I used {{-march=native}} as a quick hack: {code:diff} diff --git a/extra/wolfssl/wolfssl/wolfcrypt/src/random.c b/extra/wolfssl/wolfssl/wolfcrypt/src/random.c index 6b0d5dafc..817619d16 100644 --- a/extra/wolfssl/wolfssl/wolfcrypt/src/random.c +++ b/extra/wolfssl/wolfssl/wolfcrypt/src/random.c @@ -173,7 +173,7 @@ int wc_RNG_GenerateByte(WC_RNG* rng, byte* b) static int wc_GenerateRand_IntelRD(OS_Seed* os, byte* output, word32 sz); #endif -#ifdef USE_WINDOWS_API +#if 1 /*def USE_WINDOWS_API */ #include <immintrin.h> #endif /* USE_WINDOWS_API */ #endif @@ -1282,7 +1282,7 @@ int wc_FreeNetRandom(void) #ifdef HAVE_INTEL_RDSEED -#ifndef USE_WINDOWS_API +#if 0/*ndef USE_WINDOWS_API*/ /* return 0 on success */ static WC_INLINE int IntelRDseed64(word64* seed) @@ -1302,7 +1302,7 @@ int wc_FreeNetRandom(void) { int ok; - ok = _rdseed64_step(seed); + ok = _rdseed64_step((unsigned long long*) seed); return (ok) ? 0 : -1; } diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ut0crc32.cc index 4a6447c1dcf..58273e9058e 100644 --- a/storage/innobase/ut/ut0crc32.cc +++ b/storage/innobase/ut/ut0crc32.cc @@ -219,6 +219,8 @@ ut_crc32_8_hw( { #ifdef _MSC_VER *crc = _mm_crc32_u8(*crc, (*data)[0]); +#elif 1 + *crc = __builtin_ia32_crc32qi(*crc, (*data)[0]); #else asm("crc32b %1, %0" /* output operands */ @@ -251,6 +253,8 @@ ut_crc32_64_low_hw( #else #error Not Supported processors type. #endif +#elif 1 + crc_64bit = __builtin_ia32_crc32di(crc_64bit, data); #else asm("crc32q %1, %0" /* output operands */ {code} h2. Errors in {{mysqltest}}: {code:diff} diff --git a/mysql-test/lib/My/SafeProcess/safe_process.cc b/mysql-test/lib/My/SafeProcess/safe_process.cc index 9b544a25178..84003288dab 100644 --- a/mysql-test/lib/My/SafeProcess/safe_process.cc +++ b/mysql-test/lib/My/SafeProcess/safe_process.cc @@ -219,6 +219,7 @@ int main(int argc, char* const argv[] ) sigemptyset(&sa.sa_mask); sa_abort.sa_handler= handle_abort; + sa_abort.sa_flags= 0; sigemptyset(&sa_abort.sa_mask); /* Install signal handlers */ sigaction(SIGTERM, &sa,NULL); diff --git a/client/mysqltest.cc b/client/mysqltest.cc index 5f28bf422e1..0ba9cdc13c4 100644 --- a/client/mysqltest.cc +++ b/client/mysqltest.cc @@ -1687,6 +1687,7 @@ void abort_not_supported_test(const char *fmt, ...) cur_file->file_name, cur_file->lineno); char buff[DIE_BUFF_SIZE]; + buff[0] = '\0'; print_file_stack(buff, buff + sizeof(buff)); fprintf(stderr, "%s", buff); @@ -10363,6 +10364,7 @@ void free_replace_regex() string - the string to perform substitutions in icase - flag, if set to 1 the match is case insensitive */ +__attribute__((no_sanitize("memory"))) int reg_replace(char** buf_p, int* buf_len_p, char *pattern, char *replace, char *string, int icase) { {code} Note: the possible bug in {{reg_replace()}} should be investigated deeper, and not simply suppressed. h2. SSL-related problem {code:diff} diff --git a/mysys_ssl/my_crypt.cc b/mysys_ssl/my_crypt.cc index 02770644259..cb9e60e4dc7 100644 --- a/mysys_ssl/my_crypt.cc +++ b/mysys_ssl/my_crypt.cc @@ -94,6 +94,8 @@ class MyCTX } }; +#include <sanitizer/msan_interface.h> + class MyCTX_nopad : public MyCTX { public: @@ -143,6 +145,7 @@ class MyCTX_nopad : public MyCTX of this class too. */ uchar mask[MY_AES_BLOCK_SIZE]; + __msan_unpoison(mask, sizeof mask); uint mlen; int rc= my_aes_crypt(MY_AES_ECB, ENCRYPTION_FLAG_ENCRYPT | ENCRYPTION_FLAG_NOPAD, {code} The above is only a work-around. Without the above, the last 4 payload bytes in an encrypted InnoDB redo log block (at offset 512-8) would be claimed to uninitialized on {{pwrite64()}}. This needs to be investigated properly. h2. Some remaining problems (blocking further tests): {{innodb.innodb-page_compression_lz4}} alleges uninitialized value in buf_page_encrypt() when computing ut_crc32_8_hw(); maybe we simply need an instrumented liblz4, or should avoid linking with the uninstrumented library? This one occurs at least in {{innodb.innodb-wl5980-alter}} and {{innodb.innodb-alter}}: {noformat:title=10.5 da53fb6d7de906fd8bd73d5f244bac4d77b687aa} ==16796==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x4b9edc in var_get(char const*, char const**, char, char) /mariadb/10.5/client/mysqltest.cc:2498:12 #1 0x51b995 in append_replace_regex(char*, char*, st_replace_regex*, char**) /mariadb/10.5/client/mysqltest.cc:10205:17 #2 0x5139f2 in init_replace_regex(char*) /mariadb/10.5/client/mysqltest.cc:10163:3 #3 0x5139f2 in do_get_replace_regex(st_command*) /mariadb/10.5/client/mysqltest.cc:10324 #4 0x50f060 in main /mariadb/10.5/client/mysqltest.cc:9608:9 #5 0x7f8847a7409a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a) #6 0x438b19 in _start (/dev/shm/10.5/client/mysqltest+0x438b19) Uninitialized value was created by an allocation of 'v_end' in the stack frame of function '_Z20append_replace_regexPcS_P16st_replace_regexPS_' #0 0x51b630 in append_replace_regex(char*, char*, st_replace_regex*, char**) /mariadb/10.5/client/mysqltest.cc:10176 {noformat} {noformat:title=10.5 da53fb6d7de906fd8bd73d5f244bac4d77b687aa} CURRENT_TEST: innodb.innodb-replace-debug mysqltest: At line 13: query 'replace into t1 values (14, 25, 34)' failed: 2013: Lost connection to MySQL server during query … ha_commit_trans: info: is_real_trans: 1 rw_trans: 1 rw_ha_count: 1 MDL_context::acquire_lock: enter: ==20608==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x397a8df in dtoa /mariadb/10.5/strings/dtoa.c:2227:7 #1 0x3970d61 in my_fcvt /mariadb/10.5/strings/dtoa.c:96:8 #2 0x398f8bb in process_dbl_arg /mariadb/10.5/strings/my_vsnprintf.c:246:10 #3 0x398f8bb in my_vsnprintf_ex /mariadb/10.5/strings/my_vsnprintf.c:638 #4 0x385db9f in DbugVfprintf /mariadb/10.5/dbug/dbug.c:1329:10 #5 0x385db9f in _db_doprnt_ /mariadb/10.5/dbug/dbug.c:1309 #6 0x120342f in MDL_context::acquire_lock(MDL_request*, double) /mariadb/10.5/sql/mdl.cc:2242:3 #7 0x17d8de3 in ha_commit_trans(THD*, bool) /mariadb/10.5/sql/handler.cc:1470:24 #8 0x1228d0a in trans_commit_stmt(THD*) /mariadb/10.5/sql/transaction.cc:436:10 #9 0xc28e14 in mysql_execute_command(THD*) /mariadb/10.5/sql/sql_parse.cc:6134:7 #10 0xbfcce2 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /mariadb/10.5/sql/sql_parse.cc:7884:18 #11 0xbed4e5 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /mariadb/10.5/sql/sql_parse.cc:1842:7 #12 0xbfee52 in do_command(THD*) /mariadb/10.5/sql/sql_parse.cc:1359:17 #13 0x11cf88f in do_handle_one_connection(CONNECT*, bool) /mariadb/10.5/sql/sql_connect.cc:1414:11 #14 0x11ce99c in handle_one_connection /mariadb/10.5/sql/sql_connect.cc:1309:5 #15 0x2533fd8 in pfs_spawn_thread /mariadb/10.5/storage/perfschema/pfs.cc:1862:3 #16 0x7ff07e856fa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2) #17 0x7ff07de3e4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce) Uninitialized value was created by an allocation of 'cvtbuf.i' in the stack frame of function '_db_doprnt_' #0 0x385d4d0 in _db_doprnt_ /mariadb/10.5/dbug/dbug.c:1288 {noformat} This one was reported by Valgrind on some (not all) platforms. Here it is for {{innodb.doublewrite}}: {noformat:title=10.5 da53fb6d7de906fd8bd73d5f244bac4d77b687aa} ==21866==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x3093dcc in buf_page_is_corrupted(bool, unsigned char const*, unsigned long) /mariadb/10.5/storage/innobase/buf/buf0buf.cc:1037:14 #1 0x338b7cb in Datafile::find_space_id() /mariadb/10.5/storage/innobase/fsp/fsp0file.cc:711:22 #2 0x3389af5 in Datafile::validate_for_recovery() /mariadb/10.5/storage/innobase/fsp/fsp0file.cc:461:9 #3 0x33123ed in fil_ibd_load(unsigned long, char const*, fil_space_t*&) /mariadb/10.5/storage/innobase/fil/fil0fil.cc:3746:15 {noformat} |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for the currently latest compiler, {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use an older {{clang}} (I have successfully used {{clang-8}} and {{clang-9}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -march=native -mtune=native -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native -mtune=native -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
Summary | Enable MemorySanitizer | Make WITH_MSAN more usable |
issue.field.resolutiondate | 2020-07-01 14:35:27.0 | 2020-07-01 14:35:27.911 |
Fix Version/s | 10.2.33 [ 24307 ] | |
Fix Version/s | 10.3.24 [ 24306 ] | |
Fix Version/s | 10.4.14 [ 24305 ] | |
Fix Version/s | 10.5.5 [ 24423 ] | |
Fix Version/s | 10.2 [ 14601 ] | |
Fix Version/s | 5.5 [ 15800 ] | |
Fix Version/s | 10.1 [ 16100 ] | |
Fix Version/s | 10.3 [ 22126 ] | |
Fix Version/s | 10.4 [ 22408 ] | |
Fix Version/s | 10.5 [ 23123 ] | |
Resolution | Fixed [ 1 ] | |
Status | Open [ 1 ] | Closed [ 6 ] |
Link |
This issue is blocked by |
Link |
This issue relates to |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for the currently latest compiler, {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use an older {{clang}} (I have successfully used {{clang-8}} and {{clang-9}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -march=native -mtune=native -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native -mtune=native -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for the currently latest compiler, {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use an older {{clang}} (I have successfully used {{clang-8}} and {{clang-9}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
Link | This issue blocks TODO-2606 [ TODO-2606 ] |
Link |
This issue relates to |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for the currently latest compiler, {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use an older {{clang}} (I have successfully used {{clang-8}} and {{clang-9}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
Attachment | build-msan.sh [ 59613 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to build an instrumented C++ runtime library These instructions are for {{clang-10}}. {code:sh} sudo apt install clang-10 clang++-10 libc++-10-dev libc++abi-10-dev cd /mariadb apt source libc++-10-dev cd llvm-toolchain-10-10.0.0 mkdir libc++msan cd libc++msan cmake ../libcxx -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Memory -DCMAKE_C_COMPILER=clang-10 -DCMAKE_CXX_COMPILER=clang++-10 make -j$(nproc) {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib ./mtr main.1st LD_LIBRARY_PATH=/mariadb/llvm-toolchain-10-10.0.0/libc++msan/lib MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
Attachment | build-msan.sh [ 59614 ] |
Attachment | build-msan.sh [ 59615 ] |
Attachment | build-msan.sh [ 59614 ] |
Attachment | build-msan.sh [ 59613 ] |
Link |
This issue is duplicated by |
Link |
This issue relates to |
Link |
This issue is duplicated by |
Workflow | MariaDB v3 [ 99008 ] | MariaDB v4 [ 156645 ] |
Link | This issue relates to MDEV-27964 [ MDEV-27964 ] |
Attachment | build-msan-1.sh [ 63753 ] |
Attachment | build-msan-1.sh [ 63753 ] |
Attachment | build-msan2.sh [ 63754 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
Attachment | build-msan15.sh [ 67035 ] |
Link |
This issue blocks |
Link | This issue relates to MDEV-30942 [ MDEV-30942 ] |
Attachment | build-msan15.sh [ 67035 ] |
Attachment | build-msan15.sh [ 69172 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{poison_in_dtor=1}} ( |
Link | This issue relates to MDEV-31790 [ MDEV-31790 ] |
Attachment | build-msan16.sh [ 72069 ] |
Attachment | build-msan18.sh [ 73251 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{poison_in_dtor=1}} ( |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{poison_in_dtor=1}} ( Also make sure you have {{lldb}} installed in order to get meaningful stack traces. |
Attachment | build-msan19.sh [ 73975 ] |
Attachment | build-msan19-1.sh [ 73976 ] |
Attachment | build-msan19-1.sh [ 73976 ] |
Attachment | build-msan19.sh [ 73975 ] |
Attachment | build-msan19.sh [ 73977 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=1 ./mtr --big-test --parallel=auto --force --retry=0 {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{poison_in_dtor=1}} ( Also make sure you have {{lldb}} installed in order to get meaningful stack traces. |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
Attachment | build-msan19.sh [ 73977 ] |
Attachment | build-msan19.sh [ 74029 ] |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang and GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang *but not* GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to build with minimal {{cmake}} arguments {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ -DSECURITY_HARDENED=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON -G Ninja .. cmake --build . {code} {{cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo}} is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with {{-fsanitize=memory}}, or building some storage engines. Note the {{-DSECURITY_HARDENED=OFF}}; it is enabled by default and seems to break operations like {{memcpy()}} with {{RelWithDebInfo}} but not {{Default}}. h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
Link |
This issue relates to |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang *but not* GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to build with minimal {{cmake}} arguments {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ -DSECURITY_HARDENED=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON -G Ninja .. cmake --build . {code} {{cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo}} is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with {{-fsanitize=memory}}, or building some storage engines. Note the {{-DSECURITY_HARDENED=OFF}}; it is enabled by default and seems to break operations like {{memcpy()}} with {{RelWithDebInfo}} but not {{Default}}. h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang *but not* GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to build with minimal {{cmake}} arguments {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ -DSECURITY_HARDENED=OFF \ -DPLUGIN_{CONNECT,SPIDER}=NO \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON -G Ninja .. cmake --build . {code} {{cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo}} is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with {{-fsanitize=memory}}, and some tests for Connect and all tests for Spider would fail. Note the {{-DSECURITY_HARDENED=OFF}}; it is enabled by default and seems to break operations like {{memcpy()}} with {{RelWithDebInfo}} but not {{Default}}. h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
Description |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang *but not* GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to build with minimal {{cmake}} arguments {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ -DSECURITY_HARDENED=OFF \ -DPLUGIN_{CONNECT,SPIDER}=NO \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON -G Ninja .. cmake --build . {code} {{cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo}} is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with {{-fsanitize=memory}}, and some tests for Connect and all tests for Spider would fail. Note the {{-DSECURITY_HARDENED=OFF}}; it is enabled by default and seems to break operations like {{memcpy()}} with {{RelWithDebInfo}} but not {{Default}}. h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
[MemorySanitizer|https://github.com/google/sanitizers/wiki/MemorySanitizerLibcxxHowTo] is a compile-time instrumentation layer in clang *but not* GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.
*No patches are necessary* since 10.5 [94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157|https://github.com/MariaDB/server/commit/94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157] (see the commit message for instructions); {{cmake -DWITH_MSAN=ON}} is supposed to work ‘out of the box’. This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems. h2. How to instrumented libraries These instructions are for {{clang-10}}. The script [^build-msan2.sh] was developed to resolve {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh {code} Note: to use different {{clang}} (tested with {{clang-8}}, {{clang-9}}, {{clang-11}}, {{clang-13}}), just replace {{10}} with the major version of the compiler above. For {{clang-15}}, the procedure is a little different: {code:sh} mkdir /tmp/build cd /tmp/build mkdir "$HOME/msan-libs" CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh {code} h2. How to build MariaDB Server 10.5 or later with the instrumented libraries {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \ -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro' \ -DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \ -DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_SAFEMALLOC=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON \ -G Ninja .. ninja {code} Note: {{-march=native -mtune=native}} is optional since [the second fix|https://github.com/MariaDB/server/commit/31e6c96b0449761dc15f548c28ded671d1b7219b] of h2. How to build with minimal {{cmake}} arguments {code:bash} cd /mariadb/10.5 mkdir build cd build cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \ -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native' \ -DSECURITY_HARDENED=OFF \ -DPLUGIN_{CONNECT,SPIDER}=NO \ -DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \ -DWITH_{ZLIB,SSL,PCRE}=bundled \ -DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \ -DWITH_MSAN=ON -G Ninja .. cmake --build . {code} {{cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo}} is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with {{-fsanitize=memory}}. Connect and Spider are disabled due to test failures that were not investigated yet. Note the {{-DSECURITY_HARDENED=OFF}}; it is enabled by default and seems to break operations like {{memcpy()}} with {{RelWithDebInfo}} but not {{Debug}}. h2. How to run tests {code:sh} cd mysql-test LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file {code} Note: It may be wise to omit {{MSAN_OPTIONS=abort_on_error=1}} except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages. Note: The {{llvm-symbolizer}} in clang 14 or later will refuse to load if {{LD_LIBRARY_PATH}} includes an MSAN-instrumented {{libgmp.so}}. To get nice resolved stack traces, you must point the environment variable {{MSAN_SYMBOLIZER_PATH}} to a script like the following. The script name had better start with {{llvm-symbolizer-}} in order to avoid a warning: {code:sh} #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" {code} The {{MSAN_OPTIONS=poison_in_dtor=0}} (to work around |
Link | This issue relates to MDEV-34933 [ MDEV-34933 ] |
Link |
This issue relates to |
Link | This issue relates to MDEV-36482 [ MDEV-36482 ] |
I have now pushed WITH_MSAN to 10.2. At least the following issues will have to be resolved for this to be useful for mysql-test-run:
MDEV-20388Allegedly uninitialized values left by MyCTX_nopadMDEV-20386Replace inline asm with compiler-builtin intrinsic functionsMDEV-20309InnoDB encryption accesses memory outside of allocated block