[MDEV-24841] Build error with MSAN use-of-uninitialized-value comp_err.c:734:10 Created: 2020-12-02  Updated: 2022-03-15  Resolved: 2022-03-14

Status: Closed
Project: MariaDB Server
Component/s: Compiling, Configuration
Affects Version/s: 10.2, 10.3, 10.4, 10.5, 10.6
Fix Version/s: 10.9.0, 10.5.16, 10.6.8, 10.7.4, 10.8.3

Type: Bug Priority: Critical
Reporter: Alexey Bychko (Inactive) Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: MSAN, affects-tests

Attachments: File MDEV-24841_Ubuntu_hirsute_impish_and_arch_linux_MSan_build.patch    
Issue Links:
Blocks
Relates
relates to MDEV-20377 Make WITH_MSAN more usable Closed

 Description   

clang version 10.0.1, 11.0.0
configured with cmake . -DBUILD_CONFIG=enterprise -DWITH_MSAN=ON

Uninitialized bytes in __interceptor_strcmp at offset 3 inside [0x702000000b78, 4)
 
==22890==WARNING: MemorySanitizer: use-of-uninitialized-value
 
    #0 0x5584db in find_message /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:734:10
 
    #1 0x5584db in parse_input_file /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:585:11
 
    #2 0x5584db in main /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:183:22
 
    #3 0x7f29e377ee8a in __libc_start_main (/lib64/libc.so.6+0x23e8a)
 
    #4 0x4daa59 in _start (/home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err+0x4daa59)
 
 
 
  Uninitialized value was created by a heap allocation
 
    #0 0x5088cd in malloc (/home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err+0x5088cd)
 
    #1 0x5674d7 in my_malloc /home/abychko/GitHub/MariaDB/MariaDBEnterprise/mysys/my_malloc.c:88:29
 
    #2 0x56782b in my_strndup /home/abychko/GitHub/MariaDB/MariaDBEnterprise/mysys/my_malloc.c:242:21
 
    #3 0x55803a in parse_message_string /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:927:2
 
    #4 0x55803a in parse_input_file /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:579:12
 
    #5 0x55803a in main /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:183:22
 
    #6 0x7f29e377ee8a in __libc_start_main (/lib64/libc.so.6+0x23e8a)
 
 
 
SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/abychko/GitHub/MariaDB/MariaDBEnterprise/extra/comp_err.c:734:10 in find_message
 
Exiting
 
make[2]: *** [extra/CMakeFiles/GenError.dir/build.make:82: include/mysqld_error.h.tmp] Error 77
 
make[1]: *** [CMakeFiles/Makefile2:10565: extra/CMakeFiles/GenError.dir/all] Error 2
 
make[1]: *** Waiting for unfinished jobs....



 Comments   
Comment by Roel Van de Paar [ 2020-12-02 ]

marko kevg Any ideas?

Comment by Alexey Bychko (Inactive) [ 2020-12-02 ]

it blocks MSAN builds for ES in Jenkins

Comment by Roel Van de Paar [ 2021-02-04 ]

abychko does this ticket block all MSAN builds? For CS and ES?

Comment by Nikita Malyavin [ 2021-10-11 ]

I see another error instead:

cd /home/nik/mariadb/bld/extra && /home/nik/mariadb/bld/extra/comp_err --charset=/home/nik/mariadb/sql/share/charsets --out-dir=/home/nik/mariadb/bld/sql/share/ --header_file=/home/nik/mariadb/bld/include/mysqld_error.h.tmp --name_file=/home/nik/mariadb/bld/include/mysqld_ername.h.tmp --state_file=/home/nik/mariadb/bld/include/sql_state.h.tmp --in_file=/home/nik/mariadb/sql/share/errmsg-utf8.txt && /usr/bin/cmake -E copy_if_different /home/nik/mariadb/bld/include/mysqld_error.h.tmp /home/nik/mariadb/bld/include/mysqld_error.h && /usr/bin/cmake -E copy_if_different /home/nik/mariadb/bld/include/mysqld_ername.h.tmp /home/nik/mariadb/bld/include/mysqld_ername.h && /usr/bin/cmake -E copy_if_different /home/nik/mariadb/bld/include/sql_state.h.tmp /home/nik/mariadb/bld/include/sql_state.h
 
==70418==WARNING: MemorySanitizer: use-of-uninitialized-value
 
    #0 0x558dc1afe001 in my_read_charset_file /home/nik/mariadb/mysys/charset.c:507:62
 
    #1 0x558dc1af509d in init_available_charsets /home/nik/mariadb/mysys/charset.c:641:3
 
    #2 0x7f997a278c26 in __pthread_once_slow (/usr/lib/libpthread.so.0+0x10c26)
 
    #3 0x558dc1af51dd in get_charset_number /home/nik/mariadb/mysys/charset.c:705:3
 
    #4 0x558dc1ae9f8c in create_sys_files /home/nik/mariadb/extra/comp_err.c:350:18
 
    #5 0x558dc1ae9f8c in main /home/nik/mariadb/extra/comp_err.c:200:9
 
    #6 0x7f9979d20b24 in __libc_start_main (/usr/lib/libc.so.6+0x27b24)
 
    #7 0x558dc1a6043d in _start (/home/nik/mariadb/bld/extra/comp_err+0x4243d)
 
 
 
  Uninitialized value was created by an allocation of 'stat_info' in the stack frame of function 'my_read_charset_file'
 
    #0 0x558dc1afd1e0 in my_read_charset_file /home/nik/mariadb/mysys/charset.c:500
 
 
 
SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/nik/mariadb/mysys/charset.c:507:62 in my_read_charset_file

It turns out that MSAN wrongly supposes stat argument uninitialized, though POSIX stat call returns success.

I filed the bug upstream: https://github.com/google/sanitizers/issues/1452

The sanitizer interceptor exists, though it's somehow ignored, even in a minimally feasible program (see link). Linking with sanitizer-friendly libcxx version also gave no success.

The solution should be just to force stat output defined for sanitizers through MEM_MAKE_DEFINED call.

Comment by Nikita Malyavin [ 2021-10-14 ]

Roel abychko can someone of you reconfirm the issue? It seems that I have a different problem on my machine.

Comment by Timofey Turenko [ 2021-10-18 ]

I can build both CS and ES, cmake options:

-DWITH_EMBEDDED_SERVER=OFF -DWITH_UNIT_TESTS=OFF -DCMAKE_BUILD_TYPE=Debug -DWITH_INNODB_BZIP2=OFF -DWITH_INNODB_LZ4=OFF -DWITH_INNODB_LZMA=OFF -DWITH_INNODB_LZO=OFF -DWITH_INNODB_SNAPPY=OFF -DPLUGIN_ARCHIVE=NO -DPLUGIN_TOKUDB=NO -DPLUGIN_MROONGA=NO -DPLUGIN_OQGRAPH=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_CONNECT=NO -DPLUGIN_SPIDER=NO -DWITH_SAFEMALLOC=OFF -DWITH_ZLIB=bundled -DWITH_SSL=bundled -DWITH_PCRE=bundled -DWITH_MSAN=ON -DCMAKE_C_FLAGS='-O2 -Wno-unused-command-line-argument -fdebug-macro -msse4.2' -DCMAKE_CXX_FLAGS='-stdlib=libc++ -O2 -Wno-unused-command-line-argument -fdebug-macro -msse4.2'

but I can not run tests:

sudo su - mysql -s /bin/bash -c ' export LD_LIBRARY_PATH=/usr/local/lib ; export CC=clang-10; export CXX=clang++-10;  export WSREP_PROVIDER=/usr/lib/libgalera_smm.so;  cd /home/timofey_turenko_mariadb_com/MariaDBEnterprise/mysql-test && perl mysql-test-run.pl --max-test-fail=0 --retry=3 --parallel=auto --max-save-core=0 --max-save-datadir=1 --force --vardir=/var/tmp/mtr  --xml-report=mtr.xml '
 
su: warning: cannot change directory to /home/mysql: No such file or directory
 
Logging: mysql-test-run.pl  --max-test-fail=0 --retry=3 --parallel=auto --max-save-core=0 --max-save-datadir=1 --force --vardir=/var/tmp/mtr --xml-report=mtr.xml
 
vardir: /var/tmp/mtr
 
Removing old var directory...
 
Creating var directory '/var/tmp/mtr'...
 
Checking supported features...
 
MariaDB Version 10.5.13-MariaDB-debug
 
 - SSL connections supported
 
 - binaries are debug compiled
 
 - binaries built with wsrep patch
 
Using suites: main-,archive-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_test,user_variables,wsrep_info
 
Collecting tests...
 
Installing system database...
 
mysql-test-run: *** ERROR: Error executing mysqld --bootstrap
 
Could not install system database from /var/tmp/mtr/log/bootstrap.sql
 
The /var/tmp/mtr/log/bootstrap.log file contains:
 
/home/timofey_turenko_mariadb_com/MariaDBEnterprise/bin/mariadbd --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/home/timofey_turenko_mariadb_com/MariaDBEnterprise --datadir=/var/tmp/mtr/install.db --plugin-dir=/home/timofey_turenko_mariadb_com/MariaDBEnterprise/lib/plugin --default-storage-engine=myisam --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-lock-waits --loose-skip-plugin-innodb-locks --loose-skip-plugin-innodb-metrics --loose-skip-plugin-innodb-mutexes --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-datafiles --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-semaphore-waits --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-innodb-trx --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-thread-pool-groups --loose-skip-plugin-thread-pool-queues --loose-skip-plugin-thread-pool-stats --loose-skip-plugin-thread-pool-waits --loose-skip-plugin-unix-socket --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=10M --disable-sync-frm --tmpdir=/var/tmp/mtr/tmp/ --core-file --console --character-set-server=latin1 --lc-messages-dir=/home/timofey_turenko_mariadb_com/MariaDBEnterprise/share/ --character-sets-dir=/home/timofey_turenko_mariadb_com/MariaDBEnterprise/share/charsets
 
2021-10-15 22:45:06 0 [Note] /home/timofey_turenko_mariadb_com/MariaDBEnterprise/bin/mariadbd (mysqld 10.5.13-MariaDB-debug) starting as process 79871 ...
 
2021-10-15 22:45:06 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32190)
 
2021-10-15 22:45:06 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
 
2021-10-15 22:45:06 0 [Note] Plugin 'partition' is disabled.
 
2021-10-15 22:45:06 0 [Note] Plugin 'SEQUENCE' is disabled.
 
2021-10-15 22:45:06 0 [Note] InnoDB: The first innodb_system data file 'ibdata1' did not exist. A new tablespace will be created!
 
2021-10-15 22:45:06 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
 
2021-10-15 22:45:06 0 [Note] InnoDB: Uses event mutexes
 
2021-10-15 22:45:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
 
2021-10-15 22:45:06 0 [Note] InnoDB: Number of pools: 1
 
2021-10-15 22:45:06 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
 
2021-10-15 22:45:06 0 [Note] InnoDB: Using Linux native AIO
 
2021-10-15 22:45:06 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
 
2021-10-15 22:45:07 0 [Note] InnoDB: Completed initialization of buffer pool
 
2021-10-15 22:45:07 0 [Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ...
 
2021-10-15 22:45:07 0 [Note] InnoDB: File './ibdata1' size is now 12 MB.
 
2021-10-15 22:45:07 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 10485760 bytes
 
==79871==WARNING: MemorySanitizer: use-of-uninitialized-value
 
    #0 0x5601e5414ec4 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*) /home/timofey_turenko_mariadb_com/server-long-build-directory-name-to-workaround-rpm-bug/tpool/aio_linux.cc:127:24
 
    #1 0x5601e5415d41 in decltype(std::__1::forward<void (*)(tpool::aio_linux*)>(fp)(std::__1::forward<tpool::aio_linux*>(fp0))) std::__1::__invoke<void (*)(tpool::aio_linux*), tpool::aio_linux*>(void (*&&)(tpool::aio_linux*), tpool::aio_linux*&&) /usr/lib/llvm-10/bin/../include/c++/v1/type_traits:3539:1
 
    #2 0x5601e5415d41 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*>&, std::__1::__tuple_indices<2ul>) /usr/lib/llvm-10/bin/../include/c++/v1/thread:273:5
 
    #3 0x5601e5415d41 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*> >(void*) /usr/lib/llvm-10/bin/../include/c++/v1/thread:284:5
 
    #4 0x7f209be01608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477:8
 
    #5 0x7f209b924292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
  Uninitialized value was created by an allocation of 'events' in the stack frame of function '_ZN5tpool9aio_linux23getevent_thread_routineEPS0_'
 
    #0 0x5601e5414980 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*) /home/timofey_turenko_mariadb_com/server-long-build-directory-name-to-workaround-rpm-bug/tpool/aio_linux.cc:97
 
SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/timofey_turenko_mariadb_com/server-long-build-directory-name-to-workaround-rpm-bug/tpool/aio_linux.cc:127:24 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)
 
Exiting

Comment by Nikita Malyavin [ 2021-10-19 ]

Since 2.33 GLIBC release, stat is moved to a dynamic library (it was in libc_noshared.a before, transferring calls to __xstat). So every release featuring this version will have msan broken. In particular, two last ubuntu's and arch linux are affected. Debian sid is on 2.32 yet.

I have created the review request for llvm: https://reviews.llvm.org/D111984

Comment by Nikita Malyavin [ 2021-10-20 ]

While the LLVM review is not passed, and we have no immediate need of this hotfix, since no build machines run Ubuntu hirsute or impish, I will not patch the code base.

However, here is the patch for a local use if someone will experience this problem on a local machine. Once new clang is released, the problem should be gone.

MDEV-24841_Ubuntu_hirsute_impish_and_arch_linux_MSan_build.patch

For now I will close the issue as "Won't Fix": the original problem does not reproduce, and the discovered problem is a compiler issue that is to be fixed soon.

Comment by Marko Mäkelä [ 2022-03-14 ]

It turns out that nikitamalyavin’s fix to clang was incomplete and did not cover the variants of stat() functions where the file offset is explicitly 64 bits.

Comment by Marko Mäkelä [ 2022-03-14 ]

To be able to compile the code with clang version 13 or 14 as noted in MDEV-20377, I added work-arounds to declare the memory returned by a successful stat(), lstat(), or fstat() as initialized.

Even with these changes, most 10.5 tests failed with SIGSEGV due to something related to pthread_exit():

10.5 258c34f17cd5a06e29888498064bb46d019dc58f

 
    #0 0x7f1a1e81fbe7 in unw_get_proc_info (/usr/lib/x86_64-linux-gnu/libunwind.so.1+0x1be7) (BuildId: 1fbb529fd34f80574daa43bf41c44876b1dfae42)
 
    #1 0x7f1a1e8238cb in _Unwind_GetLanguageSpecificData (/usr/lib/x86_64-linux-gnu/libunwind.so.1+0x58cb) (BuildId: 1fbb529fd34f80574daa43bf41c44876b1dfae42)
 
    #2 0x7f1a1e80dfcc in __gxx_personality_v0 (/usr/lib/x86_64-linux-gnu/libc++abi.so.1+0x27fcc) (BuildId: 4bd847b1f8d3dcd40106e2f5dd846f77632085e3)
 
    #3 0x7f1a1e7d0ac5  (/lib/x86_64-linux-gnu/libgcc_s.so.1+0x16ac5) (BuildId: 57a2071bc064a943a1095dda6dd4963ea031782b)
 
    #4 0x7f1a1e7d11bf in _Unwind_ForcedUnwind (/lib/x86_64-linux-gnu/libgcc_s.so.1+0x171bf) (BuildId: 57a2071bc064a943a1095dda6dd4963ea031782b)
 
    #5 0x7f1a1ea35d1f in __pthread_unwind nptl/unwind.c:131:3
 
    #6 0x7f1a1ea2e04b in __do_cancel nptl/pthreadP.h:306:3
 
    #7 0x7f1a1ea2e04b in pthread_exit nptl/pthread_exit.c:28:3
 
    #8 0x55719563c568 in os_thread_exit() /mariadb/10.5m/storage/innobase/os/os0thread.cc:103:2
 
    #9 0x557195b2d19e in trx_rollback_all_recovered /mariadb/10.5m/storage/innobase/trx/trx0roll.cc:848:2
 
    #10 0x7f1a1ea2cd7f in start_thread nptl/pthread_create.c:481:8
 
    #11 0x7f1a1e6db76e in __clone misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
 
 
 
MemorySanitizer can not provide additional info.
 
SUMMARY: MemorySanitizer: SEGV (/usr/lib/x86_64-linux-gnu/libunwind.so.1+0x1be7) (BuildId: 1fbb529fd34f80574daa43bf41c44876b1dfae42) in unw_get_proc_info

On 10.6, this seriously affects replication tests, but not so much --suite=innodb.

Comment by Marko Mäkelä [ 2022-03-15 ]

A follow-up fix to clang (D121652) has been submitted.

Generated at Thu Feb 08 09:33:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.