Marko Mäkelä added a comment - 2019-08-19 10:47 I can repeat this with MSAN (enabled as described in MDEV-20377 ): 10.5 da53fb6d7de906fd8bd73d5f244bac4d77b687aa ==11249==WARNING: MemorySanitizer: use-of-uninitialized-value 2019-08-19 13:15:46 0 [Note] Server socket created on IP: '127.0.0.1'. 2019-08-19 13:15:46 0 [Note] Reading of all Master_info entries succeeded 2019-08-19 13:15:46 0 [Note] Added new Master_info '' to hash table 2019-08-19 13:15:46 0 [Note] /dev/shm/10.5/sql/mysqld: ready for connections. Version: '10.5.0-MariaDB-debug-log' socket: '/dev/shm/10.5/mysql-test/var/tmp/2/mysqld.1.sock' port: 16020 Source distribution #0 0x33350be in fil_space_encrypt(fil_space_t const*, unsigned long, unsigned long, unsigned char*, unsigned char*) /mariadb/10.5/storage/innobase/fil/fil0crypt.cc:756:4 #1 0x30f35c6 in buf_page_encrypt(fil_space_t*, buf_page_t*, unsigned char*) /mariadb/10.5/storage/innobase/buf/buf0buf.cc:7547:10 #2 0x312f71d in buf_flush_write_block_low(buf_page_t*, buf_flush_t, bool) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1026:11 #3 0x312f71d in buf_flush_page(buf_pool_t*, buf_page_t*, buf_flush_t, bool) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1211 #4 0x3136b54 in buf_flush_try_neighbors(page_id_t, buf_flush_t, unsigned long, unsigned long) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1437:9 #5 0x3136b54 in buf_flush_page_and_try_neighbors(buf_page_t*, buf_flush_t, unsigned long, unsigned long*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1511 #6 0x3139f7c in buf_do_flush_list_batch(buf_pool_t*, unsigned long, unsigned long) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1769:3 #7 0x3139f7c in buf_flush_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1840 #8 0x3139f7c in buf_flush_do_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1998 #9 0x313d030 in buf_flush_lists(unsigned long, unsigned long, unsigned long*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:2100:8 #10 0x33413ee in fil_crypt_flush_space(rotate_thread_t*) /mariadb/10.5/storage/innobase/fil/fil0crypt.cc:2179:14 #11 0x33413ee in fil_crypt_complete_rotate_space(rotate_thread_t*) /mariadb/10.5/storage/innobase/fil/fil0crypt.cc:2290 #12 0x333e598 in fil_crypt_thread /mariadb/10.5/storage/innobase/fil/fil0crypt.cc:2393:5 The error is reported for the following comparison: ut_ad(! memcmp (src, tmp_mem, (size - FIL_PAGE_FCRC32_CHECKSUM))); We have size=16384 and FIL_PAGE_FCRC32_CHECKSUM=4 . According to the following, the entire src is initialized, but tmp_mem is only partly initialized: call (void)__msan_print_shadow(src,16384) call (void)__msan_print_shadow(tmp_mem,16384) Only the first 26 and last 12 bytes of tmp_mem are initialized.

Marko Mäkelä added a comment - 2019-08-19 11:22 - edited The bulk of tmp_mem should be initialized by the decryption call: ut_ad(fil_space_decrypt(space->id, crypt_data, tmp_mem, size, space->flags, tmp, &err)); Also tmp is fully initialized. For me, the error is reported for page 1 in the system tablespace. We have innodb_checksum_algorithm=full_crc32 in this case (it is the default on 10.5). The function fil_space_decrypt_full_crc32() does get invoked, and as far as I can tell, it is invoking encryption_scheme_decrypt() a.k.a. do_crypt() correctly. (If it were not, that assertion on memcmp() would likely fail.) It looks like the issue is an allegedly uninitialized key.key in do_crypt() . call (void)__msan_print_shadow(&key,sizeof key) produces the following: Shadow map of [0x2fffe06edd18, 0x2fffe06edd2c), 20 bytes: 0x2fffe06edd18: 00000000 ffffffff ffffffff ffffffff |. A A A| 0x2fffe06edd28: ffffffff ........ ........ ........ |A . . .| The following work-around makes the report go away: diff --git a/sql/encryption.cc b/sql/encryption.cc index fc07623883e..ca31569de38 100644 --- a/sql/encryption.cc +++ b/sql/encryption.cc @@ -186,6 +186,7 @@ static uint scheme_get_key(st_encryption_scheme *scheme, return rc; } +#include <sanitizer/msan_interface.h> int do_crypt(const unsigned char* src, unsigned int slen, unsigned char* dst, unsigned int* dlen, struct st_encryption_scheme *scheme, @@ -213,14 +214,18 @@ int do_crypt(const unsigned char* src, unsigned int slen, uint rc= scheme_get_key(scheme, &key); if (rc) return (int)rc; + __msan_unpoison(key.key, sizeof key.key); unsigned char iv[4 + 4 + 8]; int4store(iv + 0, i32_1); int4store(iv + 4, i32_2); int8store(iv + 8, i64); - return encryption_crypt(src, slen, dst, dlen, key.key, sizeof(key.key), + int r= encryption_crypt(src, slen, dst, dlen, key.key, sizeof(key.key), iv, sizeof(iv), flag, scheme->key_id, key_version); + __msan_check_mem_is_initialized(src, slen); + __msan_unpoison(dst, *dlen); + return r; } int encryption_scheme_encrypt(const unsigned char* src, unsigned int slen, I think that wlad should study this code and find out a proper way to silence the error. (As noted in MDEV-20377 , I was using WITH_SSL=bundled , hence WolfSSL.) With the above work-around in place, and compiled with -O2 , the test will pass. Without -O2 I get a stack overflow in MYSQLparse() during parsing SHOW SLAVE STATUS . I still get crashes on shutdown due to another (possibly related) error: Note: this is with the __builtin_ia32_crc32qi() patch of MDEV-20377 that avoids false MSAN positives in ut_crc32_hw : 10.5 da53fb6d7de906fd8bd73d5f244bac4d77b687aa ==8278==WARNING: MemorySanitizer: use-of-uninitialized-value #0 0x2eed1bc in ut_crc32_8_hw(unsigned int*, unsigned char const**, unsigned long*) /mariadb/10.5/storage/innobase/ut/ut0crc32.cc:223:9 #1 0x2eed1bc in ut_crc32_hw(unsigned char const*, unsigned long) /mariadb/10.5/storage/innobase/ut/ut0crc32.cc:381 #2 0x293d940 in log_block_calc_checksum_crc32(unsigned char const*) /mariadb/10.5/storage/innobase/include/log0log.ic:227:9 #3 0x293d940 in log_block_store_checksum(unsigned char*) /mariadb/10.5/storage/innobase/log/log0log.cc:609 #4 0x293d940 in log_write_buf(unsigned char*, unsigned long, unsigned long, unsigned long, unsigned long) /mariadb/10.5/storage/innobase/log/log0log.cc:751 #5 0x293d940 in log_write_up_to(unsigned long, bool, bool) /mariadb/10.5/storage/innobase/log/log0log.cc:1012 #6 0x312f640 in buf_flush_write_block_low(buf_page_t*, buf_flush_t, bool) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:997:3 #7 0x312f640 in buf_flush_page(buf_pool_t*, buf_page_t*, buf_flush_t, bool) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1211 #8 0x3136c04 in buf_flush_try_neighbors(page_id_t, buf_flush_t, unsigned long, unsigned long) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1437:9 #9 0x3136c04 in buf_flush_page_and_try_neighbors(buf_page_t*, buf_flush_t, unsigned long, unsigned long*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1511 #10 0x313a02c in buf_do_flush_list_batch(buf_pool_t*, unsigned long, unsigned long) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1769:3 #11 0x313a02c in buf_flush_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1840 #12 0x313a02c in buf_flush_do_batch(buf_pool_t*, buf_flush_t, unsigned long, unsigned long, flush_counters_t*) /mariadb/10.5/storage/innobase/buf/buf0flu.cc:1998 #13 0x314af8f in pc_flush_slot() /mariadb/10.5/storage/innobase/buf/buf0flu.cc:2797:27 #14 0x314795c in buf_flush_page_cleaner_coordinator /mariadb/10.5/storage/innobase/buf/buf0flu.cc:3301:10 #15 0x7ff72bb05fa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2) #16 0x7ff72b0ed4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)

Marko Mäkelä added a comment - 2019-08-19 12:07 Note: What I repeated with MSAN is something else: alleged use of uninitialized data. Valgrind complained about accessing unaddressible memory, not about the use of uninitialized data. Unaddressible memory should be caught with ASAN, but it does not report anything for me.

Marko Mäkelä added a comment - 2020-01-24 10:40 For me, WITH_ASAN on a recent 10.4 , I cannot see any encryption trouble. monty , it seems possible that this problem was fixed in an update of WolfSSL in MariaDB 10.4.9 (and 10.5.0). I also tried WITH_MSAN on 10.5 with the recipe in MDEV-20377 . No matter if I include the above work-around to do_crypt() , it looks like any tests that use encryption would fail.

Vladislav Vaintroub added a comment - 2020-01-24 13:19 I reported it back in the day as https://github.com/wolfSSL/wolfssl/issues/2264 , with self-contained test-case, and a analysis. I tested their fix, too