Make testsuite independent

Hello, Fedora maintainer here.
I'd like to run the CONC/C testuite, but I can't use it as you do in e.g. Travis CI.

In Fedora, the testing and package build doesn't usually happen at once. I have a package sources and a SPECfile, and the package is built in minimalistic clean environment with only the software needed for the build.
Thus installing, setting up and running a database server for testing inside of the build environment is the opposite of what we are trying to do.
Once the package is built, the time for testing comes, and how heavy the test environment will be or how long the tests will run (e.g. a day or two with full MariaDB testsuite on slow armv7hl machines)

Another problem is, that since in Fedora we use CONC/C as the MariaDB client library implementation, the MariaDB-server pulls it in. And It may not be ideal to have a CONC/C package in the build environment of CONC/C package.

I made a proof of concept: taking the testsuite (compiled test binaries + libcctap.so + CTestTestfile.cmake file) and pack it as a standalone subpackage.
Then it could be installed on any system and ran just by calling 'ctest ./' in the directory, if we assume the MariaDB database is running and is properly configured (the "test" DB exists)

However before it could go to production, I need to understand and solve several issues I met on the way.

I started with a short patch:

--- mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt	2019-07-25 10:03:48.000000000 -0400

+++ mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt_patched	2019-08-19 17:02:26.317247605 -0400

@@ -71,3 +71,7 @@ FOREACH(API_TEST ${MANUAL_TESTS})

   ADD_EXECUTABLE(${API_TEST} ${API_TEST}.c)

   TARGET_LINK_LIBRARIES(${API_TEST} cctap ma_getopt mariadbclient)

 ENDFOREACH()

+

+INSTALL(TARGETS ${API_TESTS} DESTINATION ${SHAREDIR}/mariadb-connector-c/tests COMPONENT Tests)

+INSTALL(FILES ${CC_SOURCE_DIR}/unittest/mytap/libcctap.so PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ WORLD_EXECUTE DESTINATION ${INSTALL_LIBDIR} COMPONENT Tests)

+INSTALL(FILES ${CC_SOURCE_DIR}/unittest/libmariadb/CTestTestfile.cmake DESTINATION ${SHAREDIR}/mariadb-connector-c/tests COMPONENT Tests)

This simple patch alone is sufficient to be able to pack them to the RPM "mariadb-connector-c-tests" sub-package.

1)
In "mariadb-connector-c-3.1.3-src/CMakeLists.txt" on line 456 starts a list of "SET(CPACK_SOURCE_IGNORE_FILES ...)".
The lis includes both "unittest" and "/CTestTestfile.cmake", but the patch above seems to either ignore or override this "CPACK_SOURCE_IGNORE_FILES" list.
Is that correct?

2)
In "mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt" on line 59 starts a block of code which should link the created binaries against "ma_getopt" library.

ADD_LIBRARY(ma_getopt ma_getopt.c)

FOREACH(API_TEST ${API_TESTS})

  IF (NOT TARGET ${API_TEST})

    ADD_EXECUTABLE(${API_TEST} ${API_TEST}.c)

  ENDIF()

  TARGET_LINK_LIBRARIES(${API_TEST} cctap ma_getopt mariadbclient)

  ADD_TEST(${API_TEST} ${EXECUTABLE_OUTPUT_PATH}/${API_TEST})

  SET_TESTS_PROPERTIES(${API_TEST} PROPERTIES TIMEOUT 180)

ENDFOREACH(API_TEST)

FOREACH(API_TEST ${MANUAL_TESTS})

  ADD_EXECUTABLE(${API_TEST} ${API_TEST}.c)

  TARGET_LINK_LIBRARIES(${API_TEST} cctap ma_getopt mariadbclient)

ENDFOREACH()

However when you run "ldd" on all of the created binaries, the "ma_getopt" is not used anywhere (only the "libcctap.so")
Is that correct?
If yes, what was the original purpose of the "ma_getopt" library? Shouldn't it be entirely removed when not used anymore?

3)
The generated "CTestTestfile.cmake" file is somehow dependent on the environment where it was created.

# CMake generated Testfile for 

# Source directory: /builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb

# Build directory: /builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb

# 

# This file includes the relevant testing commands required for 

# testing this directory and lists subdirectories to be tested as well.

add_test(conc336 "/conc336")

set_tests_properties(conc336 PROPERTIES  TIMEOUT "180" _BACKTRACE_TRIPLES "/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;66;ADD_TEST;/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;0;")

add_test(bulk1 "/bulk1")

set_tests_properties(bulk1 PROPERTIES  TIMEOUT "180" _BACKTRACE_TRIPLES "/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;66;ADD_TEST;/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;0;")

add_test(performance "/performance")

set_tests_properties(performance PROPERTIES  TIMEOUT "180" _BACKTRACE_TRIPLES "/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;66;ADD_TEST;/builddir/build/BUILD/mariadb-connector-c-3.1.3-src/unittest/libmariadb/CMakeLists.txt;0;")

...

I'm not sure what the _BACKTRACE_TRIPLES "<path>/CMakeLists.txt;66;ADD_TEST;<path>/CMakeLists.txt;0;" is for. But since it points to / needs the CMake file, it won't be usefull outside the build environment, so it might be nice to not generate that part, when making a portable testsuite.

4)
It might be difficult to find correct location for those files.
Since the library has a very specific purpose, serving only the CONC/C package, it should be in %{_libdir}/mariadb/tests rather than directly in %{_libdir}. However since the binaries looks for the library in the standard locations, this location has to be specified in some ldconfig file.
Since the binaries are in fact tests that are not expected to be executed by an user, but by the CTest program, I'd favor other location than %{_bindir}. The MariaDB e.g. has the testsuite in /usr/share, but such location is originally reserved for architecture independent files, as per FSH, which binaries certainly aren't.
The "CTestTestfile.cmake" is however a good candidate for /usr/share location, but it first must not contain the paths listed in (3) and it has to conatin correct paths to the test binaries.

Which paths should be correct or best for Fedora is a subject of discussion on the Fedora developers mailing list. What I wanted to highlight here is the fast that many parts has to be made configurable by CMake arguments (but with default values).

I'd like to know what you think about this vision of portable testsuite, which can also be ran on the target system, instead of the build system only.
It doesn't seem too hard to implement this feature and I'm willing to help or to try to implement it, however I will need explanations and some guidenance.
(Because issues like "INSTALL(TARGETS...)" can be only used to binaries, libraries and modules are very hard to find and the CMake itself is not really helpful and the CMake IRC is not very responsive)