[#MDEV-11708] cmake -DWITH_ASAN no longer works

The cmake option -DWITH_ASAN does not work in MariaDB Server 10.2. In 10.1 it does work, and it can catch a class of bugs that is not noticed by Valgrind, such as ~~MDEV-11601~~.

Currently, an attempt to configure 10.2 -DWITH_ASAN results in a message that incorrectly claims that we do not know how to enable ASAN. The patch below fixes that:

diff --git a/CMakeLists.txt b/CMakeLists.txt

index 731afdde3d8..fb99908404c 100644

--- a/CMakeLists.txt

+++ b/CMakeLists.txt

@@ -180,27 +180,33 @@ OPTION(NOT_FOR_DISTRIBUTION "Allow linking with GPLv2-incompatible system librar

 INCLUDE(check_compiler_flag)

-OPTION(WITH_ASAN "Enable address sanitizer" OFF)

-IF (WITH_ASAN)

-  # gcc 4.8.1 and new versions of clang

-  MY_CHECK_AND_SET_COMPILER_FLAG("-fsanitize=address -O1 -Wno-error -fPIC"

-    DEBUG RELWITHDEBINFO)

-  SET(HAVE_C_FSANITIZE ${HAVE_C__fsanitize_address__O1__Wno_error__fPIC})

-  SET(HAVE_CXX_FSANITIZE ${HAVE_CXX__fsanitize_address__O1__Wno_error__fPIC})

-  IF(HAVE_C_FSANITIZE AND HAVE_CXX_FSANITIZE)

-    SET(WITH_ASAN_OK 1)

+MACRO(MY_SANITIZER_CHECK SAN_OPT RESULT)

+  MY_CHECK_C_COMPILER_FLAG("${SAN_OPT}")

+  SET(${RESULT} result)

+  MY_CHECK_CXX_COMPILER_FLAG("${SAN_OPT}")

+  IF(${RESULT} AND result)

+    # We switch on basic optimization also for debug builds.

+    # With optimization we may get some warnings, so we switch off -Werror

+    SET(CMAKE_C_FLAGS_DEBUG

+      "${CMAKE_C_FLAGS_DEBUG} ${SAN_OPT} -O1 -Wno-error -fPIC")

+    SET(CMAKE_C_FLAGS_RELWITHDEBINFO

+      "${CMAKE_C_FLAGS_RELWITHDEBINFO} ${SAN_OPT} -fPIC")

+    SET(CMAKE_CXX_FLAGS_DEBUG

+      "${CMAKE_CXX_FLAGS_DEBUG} ${SAN_OPT} -O1 -Wno-error -fPIC")

+    SET(CMAKE_CXX_FLAGS_RELWITHDEBINFO

+      "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} ${SAN_OPT} -fPIC")

+    SET(${RESULT} 1)

   ELSE()

-    # older versions of clang

-    MY_CHECK_AND_SET_COMPILER_FLAG("-faddress-sanitizer -O1 -fPIC"

-      DEBUG RELWITHDEBINFO)

-    SET(HAVE_C_FADDRESS ${HAVE_C__faddress_sanitizer__O1__fPIC})

-    SET(HAVE_CXX_FADDRESS ${HAVE_CXX__faddress_sanitizer__O1__fPIC})

-    IF(HAVE_C_FADDRESS AND HAVE_CXX_FADDRESS)

-      SET(WITH_ASAN_OK 1)

-    ENDIF()

+    SET(${RESULT} 0)

   ENDIF()

+ENDMACRO()

-  IF(NOT WITH_ASAN_OK)

+OPTION(WITH_ASAN "Enable address sanitizer" OFF)

+IF(WITH_ASAN)

+  MY_SANITIZER_CHECK("-fsanitize=address" WITH_ASAN_OK)

+  IF(WITH_ASAN_OK)

+    SET(HAVE_ASAN 1)

+  ELSE()

     MESSAGE(FATAL_ERROR "Do not know how to enable address sanitizer")

   ENDIF()

 ENDIF()

Alas, with this patch, the linking of libmariadb will fail due to unresolved references to ASAN symbols, presumably because the above compiler flags are not being used when linking (possible also when compiling) libmariadb.

Note: I think that the sanitizer flags must be set differently for debug and release. I do not want -Wno-error or -O1 for release builds, because release builds typically specify 'stronger' flags (such as -Werror -O3). I guess we want -O1 in debug builds, so that ASAN will not slow down things too much. I think that the -Wno-error is optional (probably needed because the internal continuous integration system at Oracle is actually testing code with ASAN, and some debug code could generate warnings).

[MDEV-11708] cmake -DWITH_ASAN no longer works Created: 2017-01-03 Updated: 2017-02-15 Resolved: 2017-02-15
Status:	Closed
Project:	MariaDB Server
Component/s:	Server
Affects Version/s:	10.2.0
Fix Version/s:	10.2.4

[MDEV-11708] cmake -DWITH_ASAN no longer works Created: 2017-01-03 Updated: 2017-02-15 Resolved: 2017-02-15