Marko Mäkelä added a comment - 2023-03-28 06:48 Here is a reason for some remaining failures: 10.6 216d99bb395c4fda43b4e3583672ef925103fae5 #1 0x000056129ea96227 in __msan::PoisonMemory(void const*, unsigned long, __sanitizer::StackTrace*) () #2 0x000056129ea43fd6 in __sanitizer_dtor_callback () #3 0x000056129f407ba1 in Json_writer_struct::~Json_writer_struct (this=0x7f2a766d20c0) at /mariadb/10.6/sql/my_json_writer.h:402 #4 Json_writer_object::~Json_writer_object (this=0x7f2a766d20c0) at /mariadb/10.6/sql/my_json_writer.h:460 #5 sp_lex_keeper::reset_lex_and_exec_core (this=this@entry=0x7100000140d0, thd=thd@entry=0x72b00004d018, nextp=nextp@entry=0x7f2a766d225c, open_tables=false, instr=instr@entry=0x707000007ab0) at /mariadb/10.6/sql/sp_head.cc:3586 #6 0x000056129f4153c3 in sp_lex_keeper::cursor_reset_lex_and_exec_core (this=0x7100000140d0, thd=0x72b00004d018, nextp=0x7f2a766d225c, open_tables=false, instr=0x707000007ab0) at /mariadb/10.6/sql/sp_head.cc:3601 #7 sp_instr_copen::execute (this=0x707000007ab0, thd=0x72b00004d018, nextp=0x7f2a766d225c) at /mariadb/10.6/sql/sp_head.cc:4511 #8 0x000056129f3e7ddf in sp_head::execute (this=this@entry=0x71e000015030, thd=thd@entry=0x72b00004d018, merge_da_on_success=true) at /mariadb/10.6/sql/sp_head.cc:1438 #9 0x000056129f3f0c9f in sp_head::execute_procedure (this=0x71e000015030, thd=0x72b00004d018, args=0x72b0000522a8) at /mariadb/10.6/sql/sp_head.cc:2452 #10 0x000056129f946b14 in do_execute_sp (thd=0x72b00004d018, sp=sp@entry=0x71e000015030) at /mariadb/10.6/sql/sql_parse.cc:3026 #11 0x000056129f945c84 in Sql_cmd_call::execute (this=<optimized out>, thd=0x7f2a7668d000) at /mariadb/10.6/sql/sql_parse.cc:3272 #12 0x000056129f94c7b5 in mysql_execute_command (thd=thd@entry=0x72b00004d018, is_called_from_prepared_stmt=false) at /mariadb/10.6/sql/sql_parse.cc:6002 #13 0x000056129f935a52 in mysql_parse (thd=thd@entry=0x72b00004d018, rawbuf=0x70a00001b5b0 "call cur1()", length=11, parser_state=parser_state@entry=0x7f2a766d42a0) at /mariadb/10.6/sql/sql_parse.cc:8021 This would cause an error in free_root() for comparing root->used to nullptr . I am not familiar enough with the code to say how this should be best. A possible work-around might be as follows, but it feels like an overkill: diff --git a/mysys/my_alloc.c b/mysys/my_alloc.c index aa0182c755e..b9071ad7eee 100644 --- a/mysys/my_alloc.c +++ b/mysys/my_alloc.c @@ -415,14 +415,26 @@ void free_root(MEM_ROOT *root, myf MyFlags) if (!(MyFlags & MY_KEEP_PREALLOC)) root->pre_alloc=0; +#if __has_feature(memory_sanitizer) + /* Work around MSAN_OPTIONS=poison_in_dtor=1 */ + MEM_MAKE_DEFINED(&root->used, sizeof root->used); + MEM_MAKE_DEFINED(&root->free, sizeof root->free); +#endif + for (next=root->used; next ;) { +#if __has_feature(memory_sanitizer) + MEM_MAKE_DEFINED(&next->next, sizeof next->next); +#endif old=next; next= next->next ; if (old != root->pre_alloc) my_free(old); } for (next=root->free ; next ;) { +#if __has_feature(memory_sanitizer) + MEM_MAKE_DEFINED(&next->next, sizeof next->next); +#endif old=next; next= next->next; if (old != root->pre_alloc) my_free(old); Another work-around might be to explicitly set MSAN_OPTIONS=poison_in_dtor=0 to revert the change of the default value .

Marko Mäkelä added a comment - 2023-03-28 11:59 The mystery llvm-symbolizer failures actually started with llvm-symbolizer-14 , which was the first one to link dynamically to libgmp . Because we are passing an MSAN-instrumented libgmp to all binaries, also the newer llvm-symbolizer will fail to load, because we are not using a MSAN-instrumented llvm-symbolizer . In other words, the following errors indicate that the loading of dynamic libraries while starting up llvm-symbolizer failed: /usr/bin/llvm-symbolizer-15: symbol lookup error: /home/marko/libmsan-15/libgmp.so.10: undefined symbol: __msan_va_arg_overflow_size_tls ==3567352==WARNING: Can't read from symbolizer at fd 84 ==3567352==WARNING: Can't write to symbolizer at fd 87 The subsequent warnings are for something related to the failed popen() or whatever the instrumented code used to invoke llvm-symbolizer . The fix to this is to write a simple wrapper script that discards the environment variable LD_LIBRARY_PATH : #!/bin/sh unset LD_LIBRARY_PATH exec llvm-symbolizer-15 "$@" This script can be pointed to by the environment variable MSAN_SYMBOLIZER_PATH , to get nice stack traces for MSAN failures. I will update also MDEV-20377 and MDEV-26758 with this information.