Details
-
Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
Description
As noted in MDEV-20377, MemorySanitizer (cmake -DWITH_MSAN=ON) requires that all libraries be instrumented, with the possible exception of libc. Currently, libmariadb insists on linking with GNUTLS instead of WolfSSL, which we can easily build ourselves by specifying -DWITH_SSL=bundled.
An instrumented library is a must-have for WITH_MSAN builds, and it would also improve the coverage of other sanitizers (WITH_ASAN, WITH_UBSAN, WITH_TSAN).
Because libmariadb is normally licensed under the GNU LGPL version 2, which WolfSSL is believed to be incompatible with, we could consider introducing some configuration parameter, similar to the existing cmake flag NOT_FOR_DISTRIBUTION. That flag is already being used in cmake/readline.cmake and cmake/aws_sdk.cmake. However, its purpose appears to allow something GPLv2 incompatible to be built, while here the result should still be GPLv2 compatible while being LGPLv2 incompatible. Perhaps the option should be called NOT_LGPL or similar, and it should apply to Connector/C only, not to the MariaDB server?
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
MDEV-21835 Implement option 'system-wolfssl' in build flag WITH_SSL
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
Activity
Some genuine errors were caught in mysql_client_test.c. This broke the tests main.mysql_client_test_nonblock main.mysql_client_test main.mysql_client_test_comp.
We still have the following failures:
|
10.5 ead38354e60e0fb241de8abe0ed6a57c14dfb820 |
main.truncate_notembedded w3 [ fail ]
|
Test ended at 2021-10-04 19:52:02
|
|
|
CURRENT_TEST: main.truncate_notembedded
|
mysqltest: At line 21: query 'reap' failed: 1290: The MariaDB server is running with the --max-thread-mem-used=45500 option so it cannot execute this statement
|
|
|
The result from queries just before the failure was:
|
#
|
# MDEV-23365: Assertion `!is_set() || (m_status == DA_OK_BULK &&
|
# is_bulk_op())' failed upon killed TRUNCATE
|
#
|
CREATE TABLE t1 (a INT) ENGINE=MyISAM;
|
LOCK TABLE t1 READ;
|
connect con1,localhost,root,,test;
|
SET SESSION max_session_mem_used= 45500;
|
LOCK TABLE t1 WRITE;cause
|
connection default;
|
SELECT * FROM t1;
|
a
|
UNLOCK TABLES;
|
connection con1;
|
|
|
- saving '/dev/shm/10.5msan/mysql-test/var/3/log/main.truncate_notembedded/' to '/dev/shm/10.5msan/mysql-test/var/log/main.truncate_notembedded/'
|
main.sp w2 [ fail ]
|
Test ended at 2021-10-04 19:52:15
|
|
|
CURRENT_TEST: main.sp
|
/dev/shm/10.5msan/client/mysqltest: Error on delete of '/dev/shm/10.5msan/mysql-test/var/2/tmp//t1.frm' (Errcode: 2 "No such file or directory")
|
/dev/shm/10.5msan/client/mysqltest: Error on delete of '/dev/shm/10.5msan/mysql-test/var/2/tmp//t1.MYD' (Errcode: 2 "No such file or directory")
|
--- /mariadb/10.5m/mysql-test/main/sp.result 2021-10-04 12:13:51.991511056 +0300
|
+++ /mariadb/10.5m/mysql-test/main/sp.reject 2021-10-04 19:52:15.586545147 +0300
|
@@ -8889,7 +8889,7 @@
|
SELECT 1 latin1 latin1_swedish_ci latin1_swedish_ci
|
SELECT VARIABLE_VALUE-@local_mem_used FROM INFORMATION_SCHEMA.SESSION_STATUS WHERE VARIABLE_NAME='MEMORY_USED';
|
VARIABLE_VALUE-@local_mem_used
|
-0
|
+640
|
DROP PROCEDURE sp1;
|
# End of 10.3 tests
|
#
|
|
|
mysqltest: Result length mismatch
|
|
|
- saving '/dev/shm/10.5msan/mysql-test/var/2/log/main.sp/' to '/dev/shm/10.5msan/mysql-test/var/log/main.sp/'
|
Furthermore, the test innodb.alter_large_dml tends to trigger DEBUG_SYNC timeout.
Here is proof that the test innodb.alter_large_dml sometimes passes with MemorySanitizer. This run was only with --parallel=3. Normally, with much more concurrent tests, the test would almost always timeout.
|
10.5 ead38354e60e0fb241de8abe0ed6a57c14dfb820 |
innodb.alter_large_dml 'innodb' w1 [ pass ] 209421
|
The server error log would contain the following:
2021-10-04 19:52:01 0 [Note] InnoDB: 10.5.13 started; log sequence number 47822; transaction id 27
|
…
|
2021-10-04 19:55:32 0 [Note] InnoDB: Shutdown completed; log sequence number 785123071; transaction id 57
|
2021-10-04 19:55:32 0 [Note] Debug sync points hit: 2568158
|
2021-10-04 19:55:32 0 [Note] Debug sync points executed: 8
|
2021-10-04 19:55:32 0 [Note] Debug sync points max active per thread: 3
|
2021-10-04 19:55:32 0 [Note] /dev/shm/10.5msan/sql/mariadbd: Shutdown complete
|
I attached a script build-msan.sh in MDEV-20377 for building all dependencies using the specified version of clang. I tested it on clang version 13 today. Previously I was using version 11 for MemorySanitizer builds.
Starting with clang-14, llvm-symbolizer depends on libgmp. Therefore, it will fail to work if LD_LIBRARY_PATH points to something that includes an MSAN instrumented libgmp. As noted in MDEV-30936, this can be resolved by making the environment variable MSAN_SYMBOLIZER_PATH point to a simple wrapper script:
#!/bin/sh
|
unset LD_LIBRARY_PATH
|
exec llvm-symbolizer-15 "$@"
|
For clang-15, there is another script build-msan15.sh attached to MDEV-20377.
The following patch made the trick:
--- config.h.in 2021-06-06 23:06:29.000000000 +0300+++ config.h.in 2021-10-04 16:09:18.636629855 +0300@@ -45,9 +45,6 @@/* Define to 1 if you have the `gmp' library (-lgmp). */#undef HAVE_LIBGMP-/* Define if compiler and linker supports __attribute__ ifunc */-#undef HAVE_LINK_IFUNC-/* Define to 1 if you have the <malloc.h> header file. */#undef HAVE_MALLOC_HAnd we got the next error:
CURRENT_TEST: main.userstat==929302==WARNING: MemorySanitizer: use-of-uninitialized-value#0 0x7fb162bd545a in decrypt_packet_tls13 /mariadb/gnutls28-3.7.2/lib/cipher.c:899:27#1 0x7fb162bd545a in _gnutls_decrypt /mariadb/gnutls28-3.7.2/lib/cipher.c:157:7It looks like we have to rebuild gnutls28 once more:
cd gnutls28-3.7.2CC=clang-11 CXX=clang++-11 ./configure C{,XX}FLAGS="-fno-omit-frame-pointer -O2 -g -fsanitize=memory" LDFLAGS="-fsanitize=memory" --with-included-libtasn1 --with-included-unistring --without-p11-kit --disable-hardware-accelerationautomake --add-missingmake cleanmake -j$(nproc)..And that was it, for this test:
main.userstat 'innodb' [ pass ] 1345