[MDEV-20377] Make WITH_MSAN more usable - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Affects Version/s: 5.5(EOL), 10.1(EOL), 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5
Fix Version/s: 10.2.33, 10.3.24, 10.4.14, 10.5.5
Component/s: Tests
Labels:
- Sanity_Checking
- valgrind
Environment:
Debian GNU/Linux unstable, clang 8.0.1

Description

MemorySanitizer is a compile-time instrumentation layer in clang but not GCC. Together with AddressSanitizer mostly makes the run-time instrumentation of Valgrind redundant. It is a little more tricky to set up, because running with uninstrumented libraries will lead into false positives.

No patches are necessary since 10.5 94d0bb4dbeb28a94d1f87fdd55f4297ff3df0157 (see the commit message for instructions); cmake -DWITH_MSAN=ON is supposed to work ‘out of the box’.

This task will be kept open until all tests pass and MemorySanitizer can be enabled on CI systems.

How to instrumented libraries

These instructions are for clang-10. The script build-msan2.sh was developed to resolve ~~MDEV-22083~~ a.k.a. ~~MDEV-26758~~.

mkdir /tmp/build

cd /tmp/build

mkdir "$HOME/msan-libs"

CLANG=10 MSAN_LIBDIR="$HOME/msan-libs" build-msan.sh

Note: to use different clang (tested with clang-8, clang-9, clang-11, clang-13), just replace 10 with the major version of the compiler above.

For clang-15, the procedure is a little different:

mkdir /tmp/build

cd /tmp/build

mkdir "$HOME/msan-libs"

CLANG=15 MSAN_LIBDIR="$HOME/msan-libs" build-msan15.sh

How to build MariaDB Server 10.5 or later with the instrumented libraries

cd /mariadb/10.5

mkdir build

cd build

cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-10 \

-DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro' \

-DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro'  \

-DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug \

-DWITH_DBUG_TRACE=OFF -DWITH_SAFEMALLOC=OFF \

-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \

-DWITH_SAFEMALLOC=OFF \

-DWITH_{ZLIB,SSL,PCRE}=bundled \

-DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \

-DWITH_MSAN=ON \

-G Ninja ..

ninja

Note: -march=native -mtune=native is optional since the second fix of ~~MDEV-20386~~

How to build with minimal `cmake` arguments

cd /mariadb/10.5

mkdir build

cd build

cmake -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++}-19 -DCMAKE_C_FLAGS='-O2 -march=native' \

-DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -march=native'  \

-DSECURITY_HARDENED=OFF \

-DPLUGIN_{CONNECT,SPIDER}=NO \

-DWITH_INNODB_{BZIP2,LZ4,LZMA,LZO,SNAPPY}=OFF \

-DWITH_{ZLIB,SSL,PCRE}=bundled \

-DHAVE_LIBAIO_H=0 -DCMAKE_DISABLE_FIND_PACKAGE_{URING,LIBAIO}=1 \

-DWITH_MSAN=ON -G Ninja ..

cmake --build .

cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo is implied. OK, this is almost minimal. I wanted to save the trouble of building numerous compression libraries with -fsanitize=memory. Connect and Spider are disabled due to test failures that were not investigated yet. ~~MDEV-34921~~ was tested in this way.

Note the -DSECURITY_HARDENED=OFF; it is enabled by default and seems to break operations like memcpy() with RelWithDebInfo but not Debug.

How to run tests

cd mysql-test

LD_LIBRARY_PATH="$HOME"/msan-libs ./mtr main.1st

LD_LIBRARY_PATH="$HOME"/msan-libs MSAN_OPTIONS=abort_on_error=1:poison_in_dtor=0 ./mtr --big-test --parallel=auto --force --retry=0 --skip-stack-trace --skip-core-file

Note: It may be wise to omit MSAN_OPTIONS=abort_on_error=1 except when running code in a debugger. On some occasions, it may cause truncation of the diagnostic messages.

Note: The llvm-symbolizer in clang 14 or later will refuse to load if LD_LIBRARY_PATH includes an MSAN-instrumented libgmp.so. To get nice resolved stack traces, you must point the environment variable MSAN_SYMBOLIZER_PATH to a script like the following. The script name had better start with llvm-symbolizer- in order to avoid a warning:

#!/bin/sh

unset LD_LIBRARY_PATH

exec llvm-symbolizer-15 "$@"

The MSAN_OPTIONS=poison_in_dtor=0 (to work around ~~MDEV-30936~~, MDEV-30942) is an old option that was enabled by default in clang 15.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

10.5-msan.patch
3 kB
2020-01-23 13:14
build-msan.sh
1 kB
2021-10-05 09:58
build-msan15.sh
3 kB
2023-03-28 12:39
build-msan16.sh
3 kB
2023-09-22 13:00
build-msan18.sh
3 kB
2024-03-08 06:46
build-msan19.sh
3 kB
2024-09-11 09:31
build-msan2.sh
3 kB
2022-05-17 11:56

Issue Links

blocks

MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively

Closed

includes

CONC-762 mysql_stmt_fetch in cursor mode creates bind.length unitialized (MSAN)

Open

MDBF-136 MSAN - create instrumented libcrack

Closed

MDEV-20386 Replace inline asm with compiler-builtin intrinsic functions

Closed

MDEV-20388 Allegedly uninitialized values left by MyCTX_nopad

Closed

MDEV-22059 MSAN report at replicate_ignore_table_grant

Closed

is blocked by

MDEV-22688 MSAN use-of-uninitialized-value in READ_INFO::read_xml()

Closed

MDEV-22689 MSAN use-of-uninitialized-value in decode_bytes()

Closed

MDEV-22690 MSAN use-of-uninitialized-value in optimizer_trace

Closed

MDEV-22691 MSAN use-of-uninitialized-value in test maria.maria-recovery2

Closed

MDEV-22692 MSAN use-of-uninitialized-value in pagecache_fwrite()

Closed

relates to

CONC-483 Workaround for MSAN bug needed

Closed

MDEV-20309 InnoDB encryption accesses memory outside of allocated block

Closed

MDEV-22070 MSAN use-of-uninitialized-value in encryption.innodb-redo-badkey

Closed

MDEV-22071 MSAN use-of-uninitialized-value in build_frm_image()

Closed

MDEV-22072 MSAN use-of-uninitialized-value in wf_incremental_finalize

Open

MDEV-22073 MSAN use-of-uninitialized-value in collect_statistics_for_table()

Closed

MDEV-22244 Field::error_generated_column_function_is_not_allowed: Conditional jump or move depends on uninitialised value

Open

MDEV-24514 WITH_MSAN is disabling WOLFSSL_AESNI acceleration

Closed

MDEV-26758 Make libmariadb run under MSAN

Closed

MDEV-27964 The function ENCRYPT() causes SIGSEGV in WITH_MSAN builds

Open

MDEV-30942 MSAN_OPTIONS=poison_in_dtor=1 causes failures in free_root()

Open

MDEV-31790 Extremely slow tests rpl.rpl_non_direct_mixed_mixing_engines and rpl.rpl_stm_mixing_engines

Open

MDEV-34921 MemorySanitizer reports errors for non-debug builds

Closed

MDEV-34933 The test plugins.rpl_auth uses not_msan.inc without a good reason

Open

MDEV-14024 PCRE2

Closed

MDEV-20310 valgrind bugs found in 10.5

Closed

MDEV-20383 Use of uninitialized value in Datafile::find_space_id() for ROW_FORMAT=COMPRESSED

Closed

MDEV-22060 MSAN use-of-uninitialized-value in main.query_cache_innodb

Closed

MDEV-24841 Build error with MSAN use-of-uninitialized-value comp_err.c:734:10

Closed

MDEV-34996 Buildbot MSAN options in buildbot rather than server

Closed

(1 includes, 5 is blocked by, 20 relates to)

Activity

Ascending order - Click to sort in descending order

View 19 older comments

Marko Mäkelä added a comment - 2024-03-08 06:50

build-msan18.sh is almost like build-msan16.sh:

--- build-msan16.sh	2023-09-22 13:34:39.607121198 +0300

+++ build-msan18.sh	2024-03-08 08:34:21.091289335 +0200

@@ -1,6 +1,6 @@

 #!/bin/sh

 set -eux

-: ${CLANG=16}

+: ${CLANG=18}

 : ${MSAN_LIBDIR=..}

 : ${PARALLEL=$(nproc)}

@@ -22,7 +22,7 @@

 mkdir -p ll-build

 cd ll-build

 cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \

-      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \

+      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \

       -DCMAKE_C_COMPILER=clang-$CLANG \

       -DCMAKE_CXX_COMPILER=clang++-$CLANG \

       -DLLVM_USE_SANITIZER=MemoryWithOrigins

I checked that it still catches ~~MDEV-33478~~, just like anything after clang-15 does.

Marko Mäkelä added a comment - 2024-03-08 06:50 build-msan18.sh is almost like build-msan16.sh : --- build-msan16.sh 2023-09-22 13:34:39.607121198 +0300 +++ build-msan18.sh 2024-03-08 08:34:21.091289335 +0200 @@ -1,6 +1,6 @@ #!/bin/sh set -eux -: ${CLANG=16} +: ${CLANG=18} : ${MSAN_LIBDIR=..} : ${PARALLEL=$(nproc)} @@ -22,7 +22,7 @@ mkdir -p ll-build cd ll-build cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi" \ + -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \ -DCMAKE_C_COMPILER=clang-$CLANG \ -DCMAKE_CXX_COMPILER=clang++-$CLANG \ -DLLVM_USE_SANITIZER=MemoryWithOrigins I checked that it still catches MDEV-33478 , just like anything after clang-15 does.

Marko Mäkelä added a comment - 2024-08-26 06:03 - edited

build-msan19.sh changed a little bit again:

--- build-msan18.sh	2024-03-08 08:34:21.091289335 +0200

+++ build-msan19.sh	2024-08-26 11:47:21.256980501 +0300

@@ -1,6 +1,6 @@

 #!/bin/sh

 set -eux

-: ${CLANG=18}

+: ${CLANG=19}

 : ${MSAN_LIBDIR=..}

 : ${PARALLEL=$(nproc)}

@@ -15,14 +15,17 @@

fi

 sudo apt -o APT::Get::Assume-Yes=true install \

-    clang-$CLANG libc++-$CLANG-dev libc++abi-$CLANG-dev automake

+    clang-$CLANG libc++-$CLANG-dev libc++abi-$CLANG-dev \

+    libclang-$CLANG-dev libllvmlibc-$CLANG-dev automake

 apt -o APT::Get::Assume-Yes=true source \

-    llvm-toolchain-$CLANG libgnutls28-dev libnettle8 libidn2 libgmp10

+    llvm-toolchain-$CLANG libgnutls28-dev libnettle8t64 libidn2 libgmp10

 mkdir -p ll-build

 cd ll-build

 cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \

-      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \

+      -DLLVM_ENABLE_RUNTIMES="libc;libcxx;libcxxabi;libunwind" \

+      -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_DOCS=OFF \

+      -DLLVM_ENABLE_SPHINX=OFF \

       -DCMAKE_C_COMPILER=clang-$CLANG \

       -DCMAKE_CXX_COMPILER=clang++-$CLANG \

       -DLLVM_USE_SANITIZER=MemoryWithOrigins

This will include https://reviews.llvm.org/D158943, which adds instrumentation for some recent ISO/IEC 9899:2023 compliant functions in GNU libc.

In the current Debian Sid, apt source libnettle8 would fetch Nettle 3.9, while libnettle8t64 includes Nettle 3.10, which the libgnutls would be built against.

Note: If Valgrind is installed, the configure script for Nettle 3.10 build may hit Valgrind bug 492255 (hang when trying to execute valgrind on an empty MemorySanitizer compiled program). You can send SIGKILL to the memcheck (or similar) process to work around that, or you can uninstall Valgrind before executing the build script.

Marko Mäkelä added a comment - 2024-08-26 06:03 - edited build-msan19.sh changed a little bit again: --- build-msan18.sh 2024-03-08 08:34:21.091289335 +0200 +++ build-msan19.sh 2024-08-26 11:47:21.256980501 +0300 @@ -1,6 +1,6 @@ #!/bin/sh set -eux -: ${CLANG=18} +: ${CLANG=19} : ${MSAN_LIBDIR=..} : ${PARALLEL=$(nproc)} @@ -15,14 +15,17 @@ fi sudo apt -o APT::Get::Assume-Yes=true install \ - clang-$CLANG libc++-$CLANG-dev libc++abi-$CLANG-dev automake + clang-$CLANG libc++-$CLANG-dev libc++abi-$CLANG-dev \ + libclang-$CLANG-dev libllvmlibc-$CLANG-dev automake apt -o APT::Get::Assume-Yes=true source \ - llvm-toolchain-$CLANG libgnutls28-dev libnettle8 libidn2 libgmp10 + llvm-toolchain-$CLANG libgnutls28-dev libnettle8t64 libidn2 libgmp10 mkdir -p ll-build cd ll-build cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \ - -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \ + -DLLVM_ENABLE_RUNTIMES="libc;libcxx;libcxxabi;libunwind" \ + -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_DOCS=OFF \ + -DLLVM_ENABLE_SPHINX=OFF \ -DCMAKE_C_COMPILER=clang-$CLANG \ -DCMAKE_CXX_COMPILER=clang++-$CLANG \ -DLLVM_USE_SANITIZER=MemoryWithOrigins This will include https://reviews.llvm.org/D158943 , which adds instrumentation for some recent ISO/IEC 9899:2023 compliant functions in GNU libc. In the current Debian Sid, apt source libnettle8 would fetch Nettle 3.9, while libnettle8t64 includes Nettle 3.10, which the libgnutls would be built against. Note: If Valgrind is installed, the configure script for Nettle 3.10 build may hit Valgrind bug 492255 (hang when trying to execute valgrind on an empty MemorySanitizer compiled program). You can send SIGKILL to the memcheck (or similar) process to work around that, or you can uninstall Valgrind before executing the build script.

Marko Mäkelä added a comment - 2024-09-11 07:48 - edited

The packaging for Ubuntu is a little different from Debian, at least for Ubuntu Noble (24.04). You have to install dpkg-dev and pkg-config and add deb-src to /etc/apt/sources.list.d/ubuntu.sources.

In https://apt.llvm.org/ you should note that the repository for stable Debian or Ubuntu release includes the name of the release, such as llvm-toolchain-noble-19 instead of llvm-toolchain-19, like it is for Debian Sid.

I would like to emphasize that gnutls needs pkg-config to find nettle-dev. With these tweaks, I was able to build MSAN instrumented libraries for Ubuntu.

Marko Mäkelä added a comment - 2024-09-11 07:48 - edited The packaging for Ubuntu is a little different from Debian, at least for Ubuntu Noble (24.04). You have to install dpkg-dev and pkg-config and add deb-src to /etc/apt/sources.list.d/ubuntu.sources . In https://apt.llvm.org/ you should note that the repository for stable Debian or Ubuntu release includes the name of the release, such as llvm-toolchain-noble-19 instead of llvm-toolchain-19 , like it is for Debian Sid. I would like to emphasize that gnutls needs pkg-config to find nettle-dev . With these tweaks, I was able to build MSAN instrumented libraries for Ubuntu.

Marko Mäkelä added a comment - 2024-09-11 09:33

I had included an accidental change in build-msan19.sh. The file has now been replaced with the corrected one:

@@ -23,7 +23,7 @@

 mkdir -p ll-build

 cd ll-build

 cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \

-      -DLLVM_ENABLE_RUNTIMES="libc;libcxx;libcxxabi;libunwind" \

+      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \

       -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_DOCS=OFF \

       -DLLVM_ENABLE_SPHINX=OFF \

       -DCMAKE_C_COMPILER=clang-$CLANG \

There is no need to build any instrumented libc, because clang should include "interceptors" for all functions for the system libc.so.

Marko Mäkelä added a comment - 2024-09-11 09:33 I had included an accidental change in build-msan19.sh . The file has now been replaced with the corrected one: @@ -23,7 +23,7 @@ mkdir -p ll-build cd ll-build cmake ../llvm-toolchain-$CLANG-$CLANG.*/runtimes -DCMAKE_BUILD_TYPE=Release \ - -DLLVM_ENABLE_RUNTIMES="libc;libcxx;libcxxabi;libunwind" \ + -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \ -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_DOCS=OFF \ -DLLVM_ENABLE_SPHINX=OFF \ -DCMAKE_C_COMPILER=clang-$CLANG \ There is no need to build any instrumented libc , because clang should include "interceptors" for all functions for the system libc.so .

Marko Mäkelä added a comment - 2024-09-13 09:43 - edited

The following patch is necessary for avoiding a bogus-looking stack overflow when building with CMAKE_BUILD_TYPE=RelWithDebInfo:

diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc

index 4d28c0e843c..0a0b8e78b71 100644

--- a/sql/sql_insert.cc

+++ b/sql/sql_insert.cc

@@ -3512,7 +3512,6 @@ pthread_handler_t handle_delayed_insert(void *arg)

     DBUG_LEAVE;

   my_thread_end();

-  pthread_exit(0);

   return 0;

That code seems to be redundant.

Edit: ~~MDEV-34921~~ will cover this and a few other small fixes.

Marko Mäkelä added a comment - 2024-09-13 09:43 - edited The following patch is necessary for avoiding a bogus-looking stack overflow when building with CMAKE_BUILD_TYPE=RelWithDebInfo : diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc index 4d28c0e843c..0a0b8e78b71 100644 --- a/sql/sql_insert.cc +++ b/sql/sql_insert.cc @@ -3512,7 +3512,6 @@ pthread_handler_t handle_delayed_insert(void *arg) DBUG_LEAVE; } my_thread_end(); - pthread_exit(0); return 0; } That code seems to be redundant. Edit: MDEV-34921 will cover this and a few other small fixes.

People

Assignee:: Marko Mäkelä

Reporter:: Marko Mäkelä

Votes:: 1 Vote for this issue

Watchers:: 9 Start watching this issue

Dates

Created:: 2019-08-19 08:48

Updated:: 3 days ago 00:26

Resolved:: 2020-07-01 14:35

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

How to instrumented libraries

How to build MariaDB Server 10.5 or later with the instrumented libraries

How to build with minimal cmake arguments

How to run tests

Attachments

Attachments

Issue Links

Activity

People

Dates

Git Integration

How to build with minimal `cmake` arguments