[MDEV-25260] MSAN error in tpool::aio_linux::getevent_thread_routine Created: 2021-03-25  Updated: 2024-01-05  Resolved: 2024-01-05

Status: Closed
Project: MariaDB Server
Component/s: Compiling
Affects Version/s: 10.6
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Daniel Black Assignee: Marko Mäkelä
Resolution: Not a Bug Votes: 0
Labels: None


 Description   

https://buildbot.mariadb.org/#/builders/195/builds/739/steps/7/logs/stdio

Logging: ./mtr  --big-test --force --retry=0 --max-test-fail=40 --parallel=10
vardir: /buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var
Removing old var directory...
Creating var directory '/buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var'...
Checking supported features...
MariaDB Version 10.6.0-MariaDB-debug
 - SSL connections supported
 - binaries are debug compiled
 - binaries built with wsrep patch
Using suites: main-,archive-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,sequence,sql_discovery,disks,func_test,metadata_lock_info,query_response_time,type_inet,type_test,user_variables
Collecting tests...
Installing system database...
mysql-test-run: *** ERROR: Error executing mysqld --bootstrap
Could not install system database from /buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/log/bootstrap.sql
The /buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/log/bootstrap.log file contains:
/buildbot/amd64-ubuntu-1804-msan/build/sql/mariadbd --no-defaults --disable-getopt-prefix-matching --bootstrap --basedir=/buildbot/amd64-ubuntu-1804-msan/build --datadir=/buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/install.db --plugin-dir=/buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/plugins --default-storage-engine=myisam --loose-skip-plugin-feedback --loose-skip-plugin-innodb --loose-skip-plugin-innodb-buffer-page-lru --loose-skip-plugin-innodb-buffer-pool-stats --loose-skip-plugin-innodb-cmp --loose-skip-plugin-innodb-cmp-per-index --loose-skip-plugin-innodb-cmp-per-index-reset --loose-skip-plugin-innodb-cmp-reset --loose-skip-plugin-innodb-cmpmem --loose-skip-plugin-innodb-cmpmem-reset --loose-skip-plugin-innodb-ft-being-deleted --loose-skip-plugin-innodb-ft-config --loose-skip-plugin-innodb-ft-default-stopword --loose-skip-plugin-innodb-ft-deleted --loose-skip-plugin-innodb-ft-index-cache --loose-skip-plugin-innodb-ft-index-table --loose-skip-plugin-innodb-sys-columns --loose-skip-plugin-innodb-sys-fields --loose-skip-plugin-innodb-sys-foreign --loose-skip-plugin-innodb-sys-foreign-cols --loose-skip-plugin-innodb-sys-indexes --loose-skip-plugin-innodb-sys-tables --loose-skip-plugin-innodb-sys-tablespaces --loose-skip-plugin-innodb-sys-tablestats --loose-skip-plugin-innodb-sys-virtual --loose-skip-plugin-innodb-tablespaces-encryption --loose-skip-plugin-partition --loose-skip-plugin-sequence --loose-skip-plugin-thread-pool-groups --loose-skip-plugin-thread-pool-queues --loose-skip-plugin-thread-pool-stats --loose-skip-plugin-thread-pool-waits --loose-skip-plugin-unix-socket --loose-skip-plugin-user-variables --loose-innodb --loose-innodb-log-file-size=10M --disable-sync-frm --tmpdir=/buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/tmp/ --core-file --console --character-set-server=latin1 --lc-messages-dir=/buildbot/amd64-ubuntu-1804-msan/build/sql/share/ --character-sets-dir=/buildbot/amd64-ubuntu-1804-msan/build/sql/share/charsets
2021-03-24 18:17:55 0 [Note] /buildbot/amd64-ubuntu-1804-msan/build/sql/mariadbd (mysqld 10.6.0-MariaDB-debug) starting as process 15007 ...
2021-03-24 18:17:55 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 32206)
2021-03-24 18:17:55 0 [Warning] Changed limits: max_open_files: 1024  max_connections: 151 (was 151)  table_cache: 421 (was 2000)
2021-03-24 18:17:55 0 [Note] Plugin 'partition' is disabled.
2021-03-24 18:17:55 0 [Note] Plugin 'SEQUENCE' is disabled.
2021-03-24 18:17:55 0 [Note] InnoDB: The first innodb_system data file 'ibdata1' did not exist. A new tablespace will be created!
2021-03-24 18:17:55 0 [Note] InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2021-03-24 18:17:55 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-03-24 18:17:55 0 [Note] InnoDB: Number of pools: 1
2021-03-24 18:17:55 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-03-24 18:17:55 0 [Note] mariadbd: O_TMPFILE is not supported on /buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/tmp/ (disabling future attempts)
2021-03-24 18:17:55 0 [Note] InnoDB: Using Linux native AIO
2021-03-24 18:17:55 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-03-24 18:17:55 0 [Note] InnoDB: Completed initialization of buffer pool
2021-03-24 18:17:55 0 [Note] InnoDB: Setting file './ibdata1' size to 12 MB. Physically writing the file full; Please wait ...
2021-03-24 18:17:55 0 [Note] InnoDB: File './ibdata1' size is now 12 MB.
2021-03-24 18:17:55 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-03-24 18:17:55 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 10485760 bytes
==15007==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55e5552b8b95 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*) /buildbot/amd64-ubuntu-1804-msan/build/tpool/aio_linux.cc:124:24
    #1 0x55e5552b9a01 in decltype(std::__1::forward<void (*)(tpool::aio_linux*)>(fp)(std::__1::forward<tpool::aio_linux*>(fp0))) std::__1::__invoke<void (*)(tpool::aio_linux*), tpool::aio_linux*>(void (*&&)(tpool::aio_linux*), tpool::aio_linux*&&) /usr/lib/llvm-10/bin/../include/c++/v1/type_traits:3539:1
    #2 0x55e5552b9a01 in void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*>&, std::__1::__tuple_indices<2ul>) /usr/lib/llvm-10/bin/../include/c++/v1/thread:273:5
    #3 0x55e5552b9a01 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*> >(void*) /usr/lib/llvm-10/bin/../include/c++/v1/thread:284:5
    #4 0x7f88b2c546da in start_thread /build/glibc-2ORdQG/glibc-2.27/nptl/pthread_create.c:463
    #5 0x7f88b1111a3e in clone /build/glibc-2ORdQG/glibc-2.27/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
  Uninitialized value was created by an allocation of 'events' in the stack frame of function '_ZN5tpool9aio_linux23getevent_thread_routineEPS0_'
    #0 0x55e5552b8640 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*) /buildbot/amd64-ubuntu-1804-msan/build/tpool/aio_linux.cc:94
SUMMARY: MemorySanitizer: use-of-uninitialized-value /buildbot/amd64-ubuntu-1804-msan/build/tpool/aio_linux.cc:124:24 in tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*)
Exiting
210324 18:17:55 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
Server version: 10.6.0-MariaDB-debug
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 468082 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
??:0(__interceptor_backtrace)[0x55e552038290]
??:0(my_print_stacktrace)[0x55e555553e51]
??:0(handle_fatal_signal)[0x55e5531e0985]
msan_interceptors.cpp:0(SignalHandler(int))[0x55e5520623d9]
??:0(__restore_rt)[0x7f88b2c5f8a0]
linux/raise.c:51(__GI_raise)[0x7f88b102ef47]
stdlib/abort.c:81(__GI_abort)[0x7f88b10308b1]
:0(__sanitizer::Abort())[0x55e55207ffa7]
:0(__sanitizer::Die())[0x55e55207e9d1]
??:0(__msan_warning_noreturn)[0x55e5520165b9]
tpool/aio_linux.cc:135(tpool::aio_linux::getevent_thread_routine(tpool::aio_linux*))[0x55e5552b8b96]
v1/memory:2615(void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)(tpool::aio_linux*), tpool::aio_linux*> >(void*))[0x55e5552b9a02]
nptl/pthread_create.c:463(start_thread)[0x7f88b2c546db]
x86_64/clone.S:97(clone)[0x7f88b1111a3f]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /buildbot/amd64-ubuntu-1804-msan/build/mysql-test/var/install.db
Resource Limits:
Fatal signal 11 while backtracing



 Comments   
Comment by Marko Mäkelä [ 2024-01-05 ]

This looks like the result of misconfiguration. As I have noted in MDEV-20377, it is easiest to disable the asynchronous I/O system calls when compiling with MemorySanitizer instrumentation. The scripts that are posted there do that both for the old libaio (io_setup, io_submit, io_getevents) and the more modern io_uring.

For what it is worth, I think that the Valgrind builds disable libaio as well.

Generated at Thu Feb 08 09:36:21 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.