Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21705

Source embedded WolfSSL crashes mysqld on s390x

Details

    Description

      While updating the official Debian packaging and working on release 10.4.12 I noticed that the mtr testsuite in the s390x build on Launchpad failed on all tests, indicating that mysqld completely crashes.

      This was working before, so I did many builds and managed to narrow it down to a SSL/GnuTLS issue.

      Build with external libssl-dev passes and tests run OK:
      https://launchpadlibrarian.net/464501349/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1581415272.6e13abad1_BUILDING.txt.gz

      However after activating the source embedded GnuTLS the mysqld starts to crash on all tests:

      worker[3] mysql-test-run: WARNING: Process [mysqld.1 - pid: 27108, winpid: 27108, exit: 256] died after mysql-test-run waited 0.9 seconds for /<<PKGBUILDDIR>>/builddir/mysql-test/var/3/run/mysqld.1.pid to be created.
       
      main.ipv4_as_ipv6                        w3 [ fail ]
              Test ended at 2020-02-11 11:20:41
       
      CURRENT_TEST: main.ipv4_as_ipv6
       
       
      Failed to start mysqld.1
      mysqltest failed but provided no output
      ...
      worker[2] mysql-test-run: WARNING: Process [mysqld.1 - pid: 27114, winpid: 27114, exit: 256] died after mysql-test-run waited 1.1 seconds for /<<PKGBUILDDIR>>/builddir/mysql-test/var/2/run/mysqld.1.pid to be created.
      main.stat_tables_rbr 'innodb'            w2 [ fail ]
              Test ended at 2020-02-11 11:20:41
       
      CURRENT_TEST: main.stat_tables_rbr
       
       
      Failed to start mysqld.1
      mysqltest failed but provided no output
      

      https://launchpadlibrarian.net/464506664/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1581419236.0790a802b_BUILDING.txt.gz

      The change between those builds was:
      https://salsa.debian.org/mariadb-team/mariadb-10.4/commit/0790a802b55adcf95e41e695185e50345ac9339f

      Switching to OpenSSL in official Debian is not an option due to licensing views. MariaDB has been with GnuTLS and fully working in latest 10.3.22, 10.1, 10.0 etc in Debian.

      This is now a regression in 10.4 series. Please help me figure out what it is.

      Attachments

        Issue Links

          Activity

            otto, it seems to me that the pointer that _fp_exptmod() is passing as the first parameter of fp_copy() is bad on this architecture for some reason. It seems to me that adding -DWC_NO_CACHE_RESISTANT to the CMAKE_C_FLAGS on this architecture could address the problem.

            marko Marko Mäkelä added a comment - otto , it seems to me that the pointer that _fp_exptmod() is passing as the first parameter of fp_copy() is bad on this architecture for some reason. It seems to me that adding -DWC_NO_CACHE_RESISTANT to the CMAKE_C_FLAGS on this architecture could address the problem.

            Fixed downstream in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/d92c7b9c0f0ac2c1bf098f8da4ab6bf4bb6e678f (and Marko's debug tips documented in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/1a0a2abcdf313de6746bb835547d295bf2abaaf6).

            I also filed upstream WolfSSL an issue about this: https://github.com/wolfSSL/wolfssl/issues/2828

            Based on discussions with serg, it would make sense to also support WITH_SSL=system-wolfssl so I've kept my stub at that public and filed another issue to track if that gets finished: https://jira.mariadb.org/browse/MDEV-21835

            Since this is mitigated downstream and upstream is expected to make the final fix, I am closing this issue in MariaDB now.

            otto Otto Kekäläinen added a comment - Fixed downstream in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/d92c7b9c0f0ac2c1bf098f8da4ab6bf4bb6e678f (and Marko's debug tips documented in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/1a0a2abcdf313de6746bb835547d295bf2abaaf6 ). I also filed upstream WolfSSL an issue about this: https://github.com/wolfSSL/wolfssl/issues/2828 Based on discussions with serg , it would make sense to also support WITH_SSL=system-wolfssl so I've kept my stub at that public and filed another issue to track if that gets finished: https://jira.mariadb.org/browse/MDEV-21835 Since this is mitigated downstream and upstream is expected to make the final fix, I am closing this issue in MariaDB now.

            Tested with MariaDB 10.5.5 with the temporary hack disabled and the build on s390x was OK including the main test suite (736 tests, mysqld did not crash).

            --- a/debian/rules
            +++ b/debian/rules
            @@ -52,15 +52,6 @@ ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
                 endif
             endif
             
            -# Add extra flag to avoid WolfSSL code crashing the entire mariadbd on s390x. This
            -# can be removed once upstream has made the code s390x compatible, see
            -# https://jira.mariadb.org/browse/MDEV-21705 and
            -# https://github.com/wolfSSL/wolfssl/issues/2828
            -#ifeq ($(DEB_HOST_ARCH),s390x)
            -#    CFLAGS += -DWC_NO_CACHE_RESISTANT
            -#endif
            

            --> https://launchpadlibrarian.net/501100344/buildlog_ubuntu-groovy-s390x.mariadb-10.5_1%3A10.5.5-3~exp2~ubuntu20.10.1~1602100063.bde2cf481.master_BUILDING.txt.gz

            otto Otto Kekäläinen added a comment - Tested with MariaDB 10.5.5 with the temporary hack disabled and the build on s390x was OK including the main test suite (736 tests, mysqld did not crash). --- a/debian/rules +++ b/debian/rules @@ -52,15 +52,6 @@ ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH)) endif endif -# Add extra flag to avoid WolfSSL code crashing the entire mariadbd on s390x. This -# can be removed once upstream has made the code s390x compatible, see -# https://jira.mariadb.org/browse/MDEV-21705 and -# https://github.com/wolfSSL/wolfssl/issues/2828 -#ifeq ($(DEB_HOST_ARCH),s390x) -# CFLAGS += -DWC_NO_CACHE_RESISTANT -#endif --> https://launchpadlibrarian.net/501100344/buildlog_ubuntu-groovy-s390x.mariadb-10.5_1%3A10.5.5-3~exp2~ubuntu20.10.1~1602100063.bde2cf481.master_BUILDING.txt.gz

            I removed via https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/346b704f83c5d21cb734d98989bf6ee771ec5387 the workaround described above and at least WolfSSL in MariaDB 10.5.5 in Debian seemed to build fine now without it. Build logs at https://buildd.debian.org/status/package.php?p=mariadb-10.5

            otto Otto Kekäläinen added a comment - I removed via https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/346b704f83c5d21cb734d98989bf6ee771ec5387 the workaround described above and at least WolfSSL in MariaDB 10.5.5 in Debian seemed to build fine now without it. Build logs at https://buildd.debian.org/status/package.php?p=mariadb-10.5
            danblack Daniel Black added a comment -

            Debian is now using openssl, however as a test - https://buildbot.mariadb.org/#/builders/309/builds/534/steps/6/logs/stdio passes on s390x, so WolfSSL folks have indeed fixed this.

            danblack Daniel Black added a comment - Debian is now using openssl, however as a test - https://buildbot.mariadb.org/#/builders/309/builds/534/steps/6/logs/stdio passes on s390x, so WolfSSL folks have indeed fixed this.

            People

              marko Marko Mäkelä
              otto Otto Kekäläinen
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.