|
Contents of the change that makes s390x start failing:
|
|
The package mariadb-10.4 has now entered Debian unstable and the same s390x build is failing there: https://buildd.debian.org/status/package.php?p=mariadb-10.4&suite=experimental
See full log at https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.4&arch=s390x&ver=1%3A10.4.12-1%7Eexp1&stamp=1582486640&raw=0
The reason is the same: mysqld fails to start and thus all tests fail.
This would need to be fixed for the package to enter Debian and Ubuntu fully, since both distros have s390x as one of the official architectures.
|
|
Could maybe cvicentiu or marko help out with this?
|
|
Starting with MDEV-18531 in MariaDB 10.4.6, WITH_SSL=bundled includes WolfSSL instead of YaSSL. Maybe some low-level tricks in WolfSSL are not compatible with the s390x architecture?
|
|
Thanks for the info marko! WolfSSL has been in Debian for a while already: https://packages.debian.org/search?keywords=wolfssl&searchon=names&suite=all§ion=all – https://tracker.debian.org/pkg/wolfssl
If builds on all platforms (https://buildd.debian.org/status/package.php?p=wolfssl)
There are no signs from WolfSSL side that it would have any quirks with s390x.
|
|
I removed from debian/rules and installed system WolfSSL, but now the build fails with:
mariadb-10.4$ git show
|
commit 7f5ce22d3fb9fdea5bcbe7eac83f1d57b4a0a3fd (HEAD -> feature/system-wolfssl)
|
Author: Otto Kekäläinen <otto@debian.org>
|
Date: Mon Feb 24 18:42:56 2020 +0200
|
|
WIP: Build using system WolfSSL library
|
|
diff --git a/debian/rules b/debian/rules
|
index 99bb92e27..22eb517df 100755
|
--- a/debian/rules
|
+++ b/debian/rules
|
@@ -101,7 +101,6 @@ endif
|
-DINSTALL_PLUGINDIR=lib/$(DEB_HOST_MULTIARCH)/mariadb19/plugin \
|
-DINSTALL_MYSQLTESTDIR=share/mysql/mysql-test \
|
-DPLUGIN_AUTH_SOCKET=STATIC \
|
- -DWITH_SSL=bundled \
|
-DPLUGIN_AWS_KEY_MANAGEMENT=NO \
|
-DWITH_INNODB_SNAPPY=ON \
|
-DDEB=$(DEB_VENDOR) ..'
|
|
mariadb-10.4$ apt install libwolfssl-dev
|
...
|
Setting up libwolfssl24:amd64 (4.3.0+dfsg-2) ...
|
Setting up libwolfssl-dev:amd64 (4.3.0+dfsg-2) ...
|
...
|
mariadb-10.4$ gbp buildpackage
|
...
|
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)
|
CMake Error at cmake/ssl.cmake:155 (MESSAGE):
|
Cannot find appropriate system libraries for SSL. Use WITH_SSL=bundled to
|
enable SSL support
|
Call Stack (most recent call first):
|
CMakeLists.txt:364 (MYSQL_CHECK_SSL)
|
Seems the build code is not compatible with using system WolfSSL libraries. Suggestions what needs to be changed to build against them?
|
|
Seems libmariadb3 client library is built with libgnutls28-dev. Apparently that would not work for the server? Confusing with so many TLS libraries in use / supported.
|
|
I tried to add support for system WolfSSL in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/compare/master...feature%2Fsystem-wolfssl but build fails now with:
cd /tmp/build/source/builddir/strings && /usr/bin/cmake -P CMakeFiles/strings.dir/cmake_clean_target.cmake
|
cd /tmp/build/source/builddir/strings && /usr/bin/cmake -E cmake_link_script CMakeFiles/strings.dir/link.txt --verbose=1
|
/usr/bin/ar qc libstrings.a CMakeFiles/strings.dir/bchange.c.o CMakeFiles/strings.dir/bmove_upp.c.o CMakeFiles/strings.dir/ctype-big5.c.o CMakeFiles/strings.dir/ctype-bin.c.o CMakeFiles/strings.dir/ctype-cp932.c.o CMakeFiles/strings.dir/ctype-czech.c.o CMakeFiles/strings.dir/ctype-euc_kr.c.o CMakeFiles/strings.dir/ctype-eucjpms.c.o CMakeFiles/strings.dir/ctype-extra.c.o CMakeFiles/strings.dir/ctype-gb2312.c.o CMakeFiles/strings.dir/ctype-gbk.c.o CMakeFiles/strings.dir/ctype-latin1.c.o CMakeFiles/strings.dir/ctype-mb.c.o CMakeFiles/strings.dir/ctype-simple.c.o CMakeFiles/strings.dir/ctype-sjis.c.o CMakeFiles/strings.dir/ctype-tis620.c.o CMakeFiles/strings.dir/ctype-uca.c.o CMakeFiles/strings.dir/ctype-ucs2.c.o CMakeFiles/strings.dir/ctype-ujis.c.o CMakeFiles/strings.dir/ctype-utf8.c.o CMakeFiles/strings.dir/ctype-win1250ch.c.o CMakeFiles/strings.dir/ctype.c.o CMakeFiles/strings.dir/decimal.c.o CMakeFiles/strings.dir/dtoa.c.o CMakeFiles/strings.dir/int2str.c.o CMakeFiles/strings.dir/is_prefix.c.o CMakeFiles/strings.dir/llstr.c.o CMakeFiles/strings.dir/longlong2str.c.o CMakeFiles/strings.dir/my_strtoll10.c.o CMakeFiles/strings.dir/my_vsnprintf.c.o CMakeFiles/strings.dir/str2int.c.o CMakeFiles/strings.dir/strcend.c.o CMakeFiles/strings.dir/strend.c.o CMakeFiles/strings.dir/strfill.c.o CMakeFiles/strings.dir/strmake.c.o CMakeFiles/strings.dir/strmov.c.o CMakeFiles/strings.dir/strnmov.c.o CMakeFiles/strings.dir/strxmov.c.o CMakeFiles/strings.dir/strxnmov.c.o CMakeFiles/strings.dir/xml.c.o CMakeFiles/strings.dir/strmov_overlapp.c.o CMakeFiles/strings.dir/my_strchr.c.o CMakeFiles/strings.dir/strcont.c.o CMakeFiles/strings.dir/strappend.c.o CMakeFiles/strings.dir/json_lib.c.o
|
/usr/bin/ranlib libstrings.a
|
make[4]: Leaving directory '/tmp/build/source/builddir'
|
[ 5%] Built target strings
|
/tmp/build/source/sql/sql_yacc.yy: warning: fix-its can be applied. Rerun with option '--update'. [-Wother]
|
cd /tmp/build/source/builddir && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/build/source /tmp/build/source/sql /tmp/build/source/builddir /tmp/build/source/builddir/sql /tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/DependInfo.cmake --color=
|
Dependee "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/DependInfo.cmake" is newer than depender "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/depend.internal".
|
Dependee "/tmp/build/source/builddir/sql/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/depend.internal".
|
Scanning dependencies of target gen_lex_token
|
make[4]: Leaving directory '/tmp/build/source/builddir'
|
/usr/bin/make -f sql/CMakeFiles/gen_lex_token.dir/build.make sql/CMakeFiles/gen_lex_token.dir/build
|
make[4]: Entering directory '/tmp/build/source/builddir'
|
[ 5%] Building CXX object sql/CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o
|
cd /tmp/build/source/builddir/sql && /usr/lib/ccache/x86_64-linux-gnu-g++ -DHAVE_CONFIG_H -DHAVE_EVENT_SCHEDULER -DHAVE_POOL_OF_THREADS -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -I/tmp/build/source/wsrep-lib/include -I/tmp/build/source/wsrep-lib/wsrep-API/v26 -I/tmp/build/source/builddir/include -I/tmp/build/source/include -I/tmp/build/source/sql -I/tmp/build/source/builddir/sql -I/usr/include/wolfssl -I/usr/include/wolfssl/wolfssl -g -O2 -fdebug-prefix-map=/tmp/build/source=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_OPENSSL -DHAVE_WOLFSSL -DWOLFSSL_USER_SETTINGS -std=gnu++11 -o CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o -c /tmp/build/source/sql/gen_lex_token.cc
|
[ 5%] Linking CXX executable gen_lex_token
|
cd /tmp/build/source/builddir/sql && /usr/bin/cmake -E cmake_link_script CMakeFiles/gen_lex_token.dir/link.txt --verbose=1
|
/usr/lib/ccache/x86_64-linux-gnu-g++ -g -O2 -fdebug-prefix-map=/tmp/build/source=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wl,-z,relro -Wl,-z,now CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o -o gen_lex_token -lpthread
|
make[4]: Leaving directory '/tmp/build/source/builddir'
|
[ 5%] Built target gen_lex_token
|
make[3]: Leaving directory '/tmp/build/source/builddir'
|
make[2]: *** [Makefile:163: all] Error 2
|
Please help me either test building with system WolfSSL or figure out the original real cause for why mysqld fails to start on s390x.
|
|
Build log at https://salsa.debian.org/mariadb-team/mariadb-10.4/-/jobs/579122 has better output in the sense that the last lines have the actual error:
cd /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/builddir/mysys_ssl && /usr/lib/ccache/x86_64-linux-gnu-g++ -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/wsrep-lib/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/wsrep-lib/wsrep-API/v26 -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/builddir/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl -I/usr/include/wolfssl -I/usr/include/wolfssl/wolfssl -g -O2 -fdebug-prefix-map=/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_OPENSSL -DHAVE_WOLFSSL -DWOLFSSL_USER_SETTINGS -fPIC -std=gnu++11 -o CMakeFiles/mysys_ssl.dir/my_sha1.cc.o -c /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha1.cc
|
In file included from /usr/include/wolfssl/wolfcrypt/types.h:29,
|
from /usr/include/wolfssl/wolfcrypt/sha.h:30,
|
from /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha.ic:35,
|
from /builds/mariadb-team/mari+ cleanup
|
+ docker rm -f f3481697e091d0ee2e087453dde6e38ce8aec0745fc06c27eaaaf85020764057
|
gbp:error: 'docker-build.sh registry.salsa.debian.org/salsa-ci-team/pipeline/dockerbuilder:sid' failed: it exited with 1
|
adb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha1.cc:18:
|
/usr/include/wolfssl/wolfcrypt/settings.h:208:14: fatal error: user_settings.h: No such file or directory
|
208 | #include "user_settings.h"
|
| ^~~~~~~~~~~~~~~~~
|
compilation terminated.
|
System WolfSSL in Debian unstable is 4.3.0+dfsg-2 (https://tracker.debian.org/pkg/wolfssl).
|
|
marko Reported that user_settings.h is missing: https://packages.debian.org/sid/amd64/libwolfssl-dev/filelist
The embedded WolfSSL library has this.
|
|
FIled issue in Debian about missing user_settings.h, but it seems it is an optional file. Build should not stop on this?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952538
|
|
I tried to do a debug build in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/30f27ad426f298e3a4c3eae9130f37e0c66c85e1
..but now compilation fails on all platforms:
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.4/+builds?build_text=&build_state=all
The error is unclear due to parallel build.
One error on s390x is at least:
https://launchpadlibrarian.net/466640649/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582701031.30f27ad42_BUILDING.txt.gz
/<<PKGBUILDDIR>>/plugin/auth_pam/auth_pam_tool.c: In function ‘main’:
|
/<<PKGBUILDDIR>>/plugin/auth_pam/auth_pam_tool.c:73:10: error: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Werror=unused-result]
|
73 | (void) setreuid(0, 0);
|
| ^~~~~~~~~~~~~~
|
On armhf the error is maybe:
https://launchpadlibrarian.net/466643539/buildlog_ubuntu-focal-armhf.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582701031.30f27ad42_BUILDING.txt.gz
cc1: all warnings being treated as errors
|
make[4]: *** [libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/build.make:336: libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/mariadb_stmt.c.o] Error 1
|
make[4]: *** Waiting for unfinished jobs....
|
On other platforms last lines in build log are about RocksDB.
|
|
Unfortunately, CMAKE_BUILD_TYPE=Debug makes the default setting MYSQL_MAINTAINER_MODE=AUTO add -Werror to the compilation flags. You might want to remove the flag from cmake/maintainer.cmake to work around this (and to still have useful warnings enabled), or you could set MYSQL_MAINTAINER_MODE=OFF.
|
|
OK, finally got some debug data. See section starting with
Server version: 10.4.12-MariaDB-1~ubuntu20.04.1~1582728811.ab0850b0a-log
|
in log https://launchpadlibrarian.net/466692682/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582728811.ab0850b0a_BUILDING.txt.gz
stack_bottom = 0x0 thread_stack 0x49000
|
/<<PKGBUILDDIR>>/builddir/sql/mysqld(my_print_stacktrace+0x2a)[0x12afc293a]
|
mysys/stacktrace.c:270(my_print_stacktrace)[0x12a9c7ca4]
|
[0x3ffed0f3f0e]
|
sql/signal_handler.cc:209(handle_fatal_signal)[0x12af9dfce]
|
src/tfm.c:3642(fp_copy)[0x12af9fa8c]
|
src/tfm.c:1601(_fp_exptmod)[0x12afa044e]
|
src/tfm.c:2277(fp_exptmod)[0x12afa0aa0]
|
src/tfm.c:4021(fp_prime_miller_rabin_ex)[0x12afa10f6]
|
src/tfm.c:4283(mp_prime_is_prime_ex)[0x12af8393a]
|
src/dh.c:2108(_DhSetKey)[0x12af844b6]
|
src/dh.c:2154(wc_DhSetCheckKey)[0x12af2dade]
|
src/ssl.c:1744(wolfSSL_CTX_SetTmpDH)[0x12af5083e]
|
src/ssl.c:38473(wolfSSL_CTX_set_tmp_dh)[0x12b025eba]
|
vio/viosslfactories.c:340(new_VioSSLFd)[0x12b0260cc]
|
vio/viosslfactories.c:413(new_VioSSLAcceptorFd)[0x12a6cab58]
|
/lib/s390x-linux-gnu/libc.so.6(__libc_start_main+0x10a)[0x3ff88f2a90a]
|
/<<PKGBUILDDIR>>/builddir/sql/mysqld(+0x5bcae4)[0x12a6bcae4]
|
[0x0]
|
|
|
otto, it seems to me that the pointer that _fp_exptmod() is passing as the first parameter of fp_copy() is bad on this architecture for some reason. It seems to me that adding -DWC_NO_CACHE_RESISTANT to the CMAKE_C_FLAGS on this architecture could address the problem.
|
|
Fixed downstream in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/d92c7b9c0f0ac2c1bf098f8da4ab6bf4bb6e678f (and Marko's debug tips documented in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/1a0a2abcdf313de6746bb835547d295bf2abaaf6).
I also filed upstream WolfSSL an issue about this: https://github.com/wolfSSL/wolfssl/issues/2828
Based on discussions with serg, it would make sense to also support WITH_SSL=system-wolfssl so I've kept my stub at that public and filed another issue to track if that gets finished: https://jira.mariadb.org/browse/MDEV-21835
Since this is mitigated downstream and upstream is expected to make the final fix, I am closing this issue in MariaDB now.
|
|
Tested with MariaDB 10.5.5 with the temporary hack disabled and the build on s390x was OK including the main test suite (736 tests, mysqld did not crash).
--- a/debian/rules
|
+++ b/debian/rules
|
@@ -52,15 +52,6 @@ ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
|
endif
|
endif
|
|
-# Add extra flag to avoid WolfSSL code crashing the entire mariadbd on s390x. This
|
-# can be removed once upstream has made the code s390x compatible, see
|
-# https://jira.mariadb.org/browse/MDEV-21705 and
|
-# https://github.com/wolfSSL/wolfssl/issues/2828
|
-#ifeq ($(DEB_HOST_ARCH),s390x)
|
-# CFLAGS += -DWC_NO_CACHE_RESISTANT
|
-#endif
|
--> https://launchpadlibrarian.net/501100344/buildlog_ubuntu-groovy-s390x.mariadb-10.5_1%3A10.5.5-3~exp2~ubuntu20.10.1~1602100063.bde2cf481.master_BUILDING.txt.gz
|
|
I removed via https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/346b704f83c5d21cb734d98989bf6ee771ec5387 the workaround described above and at least WolfSSL in MariaDB 10.5.5 in Debian seemed to build fine now without it. Build logs at https://buildd.debian.org/status/package.php?p=mariadb-10.5
|
|
Debian is now using openssl, however as a test - https://buildbot.mariadb.org/#/builders/309/builds/534/steps/6/logs/stdio passes on s390x, so WolfSSL folks have indeed fixed this.
|