[MDEV-21705] Source embedded WolfSSL crashes mysqld on s390x Created: 2020-02-11  Updated: 2021-08-12  Resolved: 2020-02-27

Status: Closed
Project: MariaDB Server
Component/s: Packaging, Platform Debian, SSL
Affects Version/s: 10.4.12
Fix Version/s: N/A

Type: Bug Priority: Minor
Reporter: Otto Kekäläinen Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: None

Attachments: PNG File commit 0790a802b in Debian repos.png    
Issue Links:
Relates
relates to MDEV-18531 Use WolfSSL instead of YaSSL as "bund... Closed

 Description   

While updating the official Debian packaging and working on release 10.4.12 I noticed that the mtr testsuite in the s390x build on Launchpad failed on all tests, indicating that mysqld completely crashes.

This was working before, so I did many builds and managed to narrow it down to a SSL/GnuTLS issue.

Build with external libssl-dev passes and tests run OK:
https://launchpadlibrarian.net/464501349/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1581415272.6e13abad1_BUILDING.txt.gz

However after activating the source embedded GnuTLS the mysqld starts to crash on all tests:

worker[3] mysql-test-run: WARNING: Process [mysqld.1 - pid: 27108, winpid: 27108, exit: 256] died after mysql-test-run waited 0.9 seconds for /<<PKGBUILDDIR>>/builddir/mysql-test/var/3/run/mysqld.1.pid to be created.
 
main.ipv4_as_ipv6                        w3 [ fail ]
        Test ended at 2020-02-11 11:20:41
 
CURRENT_TEST: main.ipv4_as_ipv6
 
 
Failed to start mysqld.1
mysqltest failed but provided no output
...
worker[2] mysql-test-run: WARNING: Process [mysqld.1 - pid: 27114, winpid: 27114, exit: 256] died after mysql-test-run waited 1.1 seconds for /<<PKGBUILDDIR>>/builddir/mysql-test/var/2/run/mysqld.1.pid to be created.
main.stat_tables_rbr 'innodb'            w2 [ fail ]
        Test ended at 2020-02-11 11:20:41
 
CURRENT_TEST: main.stat_tables_rbr
 
 
Failed to start mysqld.1
mysqltest failed but provided no output

https://launchpadlibrarian.net/464506664/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1581419236.0790a802b_BUILDING.txt.gz

The change between those builds was:
https://salsa.debian.org/mariadb-team/mariadb-10.4/commit/0790a802b55adcf95e41e695185e50345ac9339f

Switching to OpenSSL in official Debian is not an option due to licensing views. MariaDB has been with GnuTLS and fully working in latest 10.3.22, 10.1, 10.0 etc in Debian.

This is now a regression in 10.4 series. Please help me figure out what it is.



 Comments   
Comment by Otto Kekäläinen [ 2020-02-11 ]

Contents of the change that makes s390x start failing:

Comment by Otto Kekäläinen [ 2020-02-24 ]

The package mariadb-10.4 has now entered Debian unstable and the same s390x build is failing there: https://buildd.debian.org/status/package.php?p=mariadb-10.4&suite=experimental
See full log at https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.4&arch=s390x&ver=1%3A10.4.12-1%7Eexp1&stamp=1582486640&raw=0

The reason is the same: mysqld fails to start and thus all tests fail.

This would need to be fixed for the package to enter Debian and Ubuntu fully, since both distros have s390x as one of the official architectures.

Comment by Otto Kekäläinen [ 2020-02-24 ]

Could maybe cvicentiu or marko help out with this?

Comment by Marko Mäkelä [ 2020-02-24 ]

Starting with MDEV-18531 in MariaDB 10.4.6, WITH_SSL=bundled includes WolfSSL instead of YaSSL. Maybe some low-level tricks in WolfSSL are not compatible with the s390x architecture?

Comment by Otto Kekäläinen [ 2020-02-24 ]

Thanks for the info marko! WolfSSL has been in Debian for a while already: https://packages.debian.org/search?keywords=wolfssl&searchon=names&suite=all&section=allhttps://tracker.debian.org/pkg/wolfssl

If builds on all platforms (https://buildd.debian.org/status/package.php?p=wolfssl)

There are no signs from WolfSSL side that it would have any quirks with s390x.

Comment by Otto Kekäläinen [ 2020-02-24 ]

I removed from debian/rules and installed system WolfSSL, but now the build fails with:

mariadb-10.4$ git show
commit 7f5ce22d3fb9fdea5bcbe7eac83f1d57b4a0a3fd (HEAD -> feature/system-wolfssl)
Author: Otto Kekäläinen <otto@debian.org>
Date:   Mon Feb 24 18:42:56 2020 +0200
 
    WIP: Build using system WolfSSL library
 
diff --git a/debian/rules b/debian/rules
index 99bb92e27..22eb517df 100755
--- a/debian/rules
+++ b/debian/rules
@@ -101,7 +101,6 @@ endif
            -DINSTALL_PLUGINDIR=lib/$(DEB_HOST_MULTIARCH)/mariadb19/plugin \
            -DINSTALL_MYSQLTESTDIR=share/mysql/mysql-test \
            -DPLUGIN_AUTH_SOCKET=STATIC \
-                       -DWITH_SSL=bundled \
            -DPLUGIN_AWS_KEY_MANAGEMENT=NO \
            -DWITH_INNODB_SNAPPY=ON \
            -DDEB=$(DEB_VENDOR) ..'
 
mariadb-10.4$ apt install libwolfssl-dev
...
Setting up libwolfssl24:amd64 (4.3.0+dfsg-2) ...
Setting up libwolfssl-dev:amd64 (4.3.0+dfsg-2) ...
...
mariadb-10.4$ gbp buildpackage
...
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR) 
CMake Error at cmake/ssl.cmake:155 (MESSAGE):
  Cannot find appropriate system libraries for SSL.  Use WITH_SSL=bundled to
  enable SSL support
Call Stack (most recent call first):
  CMakeLists.txt:364 (MYSQL_CHECK_SSL)

Seems the build code is not compatible with using system WolfSSL libraries. Suggestions what needs to be changed to build against them?

Comment by Otto Kekäläinen [ 2020-02-24 ]

Seems libmariadb3 client library is built with libgnutls28-dev. Apparently that would not work for the server? Confusing with so many TLS libraries in use / supported.

Comment by Otto Kekäläinen [ 2020-02-24 ]

I tried to add support for system WolfSSL in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/compare/master...feature%2Fsystem-wolfssl but build fails now with:

cd /tmp/build/source/builddir/strings && /usr/bin/cmake -P CMakeFiles/strings.dir/cmake_clean_target.cmake
cd /tmp/build/source/builddir/strings && /usr/bin/cmake -E cmake_link_script CMakeFiles/strings.dir/link.txt --verbose=1
/usr/bin/ar qc libstrings.a  CMakeFiles/strings.dir/bchange.c.o CMakeFiles/strings.dir/bmove_upp.c.o CMakeFiles/strings.dir/ctype-big5.c.o CMakeFiles/strings.dir/ctype-bin.c.o CMakeFiles/strings.dir/ctype-cp932.c.o CMakeFiles/strings.dir/ctype-czech.c.o CMakeFiles/strings.dir/ctype-euc_kr.c.o CMakeFiles/strings.dir/ctype-eucjpms.c.o CMakeFiles/strings.dir/ctype-extra.c.o CMakeFiles/strings.dir/ctype-gb2312.c.o CMakeFiles/strings.dir/ctype-gbk.c.o CMakeFiles/strings.dir/ctype-latin1.c.o CMakeFiles/strings.dir/ctype-mb.c.o CMakeFiles/strings.dir/ctype-simple.c.o CMakeFiles/strings.dir/ctype-sjis.c.o CMakeFiles/strings.dir/ctype-tis620.c.o CMakeFiles/strings.dir/ctype-uca.c.o CMakeFiles/strings.dir/ctype-ucs2.c.o CMakeFiles/strings.dir/ctype-ujis.c.o CMakeFiles/strings.dir/ctype-utf8.c.o CMakeFiles/strings.dir/ctype-win1250ch.c.o CMakeFiles/strings.dir/ctype.c.o CMakeFiles/strings.dir/decimal.c.o CMakeFiles/strings.dir/dtoa.c.o CMakeFiles/strings.dir/int2str.c.o CMakeFiles/strings.dir/is_prefix.c.o CMakeFiles/strings.dir/llstr.c.o CMakeFiles/strings.dir/longlong2str.c.o CMakeFiles/strings.dir/my_strtoll10.c.o CMakeFiles/strings.dir/my_vsnprintf.c.o CMakeFiles/strings.dir/str2int.c.o CMakeFiles/strings.dir/strcend.c.o CMakeFiles/strings.dir/strend.c.o CMakeFiles/strings.dir/strfill.c.o CMakeFiles/strings.dir/strmake.c.o CMakeFiles/strings.dir/strmov.c.o CMakeFiles/strings.dir/strnmov.c.o CMakeFiles/strings.dir/strxmov.c.o CMakeFiles/strings.dir/strxnmov.c.o CMakeFiles/strings.dir/xml.c.o CMakeFiles/strings.dir/strmov_overlapp.c.o CMakeFiles/strings.dir/my_strchr.c.o CMakeFiles/strings.dir/strcont.c.o CMakeFiles/strings.dir/strappend.c.o CMakeFiles/strings.dir/json_lib.c.o
/usr/bin/ranlib libstrings.a
make[4]: Leaving directory '/tmp/build/source/builddir'
[  5%] Built target strings
/tmp/build/source/sql/sql_yacc.yy: warning: fix-its can be applied.  Rerun with option '--update'. [-Wother]
cd /tmp/build/source/builddir && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/build/source /tmp/build/source/sql /tmp/build/source/builddir /tmp/build/source/builddir/sql /tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/DependInfo.cmake --color=
Dependee "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/DependInfo.cmake" is newer than depender "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/depend.internal".
Dependee "/tmp/build/source/builddir/sql/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/tmp/build/source/builddir/sql/CMakeFiles/gen_lex_token.dir/depend.internal".
Scanning dependencies of target gen_lex_token
make[4]: Leaving directory '/tmp/build/source/builddir'
/usr/bin/make -f sql/CMakeFiles/gen_lex_token.dir/build.make sql/CMakeFiles/gen_lex_token.dir/build
make[4]: Entering directory '/tmp/build/source/builddir'
[  5%] Building CXX object sql/CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o
cd /tmp/build/source/builddir/sql && /usr/lib/ccache/x86_64-linux-gnu-g++  -DHAVE_CONFIG_H -DHAVE_EVENT_SCHEDULER -DHAVE_POOL_OF_THREADS -DMYSQL_SERVER -D_FILE_OFFSET_BITS=64 -I/tmp/build/source/wsrep-lib/include -I/tmp/build/source/wsrep-lib/wsrep-API/v26 -I/tmp/build/source/builddir/include -I/tmp/build/source/include -I/tmp/build/source/sql -I/tmp/build/source/builddir/sql -I/usr/include/wolfssl -I/usr/include/wolfssl/wolfssl  -g -O2 -fdebug-prefix-map=/tmp/build/source=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF   -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_OPENSSL -DHAVE_WOLFSSL  -DWOLFSSL_USER_SETTINGS -std=gnu++11 -o CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o -c /tmp/build/source/sql/gen_lex_token.cc
[  5%] Linking CXX executable gen_lex_token
cd /tmp/build/source/builddir/sql && /usr/bin/cmake -E cmake_link_script CMakeFiles/gen_lex_token.dir/link.txt --verbose=1
/usr/lib/ccache/x86_64-linux-gnu-g++  -g -O2 -fdebug-prefix-map=/tmp/build/source=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF  -Wl,-z,relro -Wl,-z,now CMakeFiles/gen_lex_token.dir/gen_lex_token.cc.o  -o gen_lex_token  -lpthread 
make[4]: Leaving directory '/tmp/build/source/builddir'
[  5%] Built target gen_lex_token
make[3]: Leaving directory '/tmp/build/source/builddir'
make[2]: *** [Makefile:163: all] Error 2

Please help me either test building with system WolfSSL or figure out the original real cause for why mysqld fails to start on s390x.

Comment by Otto Kekäläinen [ 2020-02-25 ]

Build log at https://salsa.debian.org/mariadb-team/mariadb-10.4/-/jobs/579122 has better output in the sense that the last lines have the actual error:

 cd /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/builddir/mysys_ssl && /usr/lib/ccache/x86_64-linux-gnu-g++  -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/wsrep-lib/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/wsrep-lib/wsrep-API/v26 -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/builddir/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/include -I/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl -I/usr/include/wolfssl -I/usr/include/wolfssl/wolfssl  -g -O2 -fdebug-prefix-map=/builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -O2 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF   -Wdate-time -D_FORTIFY_SOURCE=2 -DHAVE_OPENSSL -DHAVE_WOLFSSL  -DWOLFSSL_USER_SETTINGS -fPIC -std=gnu++11 -o CMakeFiles/mysys_ssl.dir/my_sha1.cc.o -c /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha1.cc
 In file included from /usr/include/wolfssl/wolfcrypt/types.h:29,
                  from /usr/include/wolfssl/wolfcrypt/sha.h:30,
                  from /builds/mariadb-team/mariadb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha.ic:35,
                  from /builds/mariadb-team/mari+ cleanup
 + docker rm -f f3481697e091d0ee2e087453dde6e38ce8aec0745fc06c27eaaaf85020764057
 gbp:error: 'docker-build.sh registry.salsa.debian.org/salsa-ci-team/pipeline/dockerbuilder:sid' failed: it exited with 1
 adb-10.4/debian/output/mariadb-10.4-10.4.12/mysys_ssl/my_sha1.cc:18:
 /usr/include/wolfssl/wolfcrypt/settings.h:208:14: fatal error: user_settings.h: No such file or directory
   208 |     #include "user_settings.h"
       |              ^~~~~~~~~~~~~~~~~
 compilation terminated.

System WolfSSL in Debian unstable is 4.3.0+dfsg-2 (https://tracker.debian.org/pkg/wolfssl).

Comment by Otto Kekäläinen [ 2020-02-25 ]

marko Reported that user_settings.h is missing: https://packages.debian.org/sid/amd64/libwolfssl-dev/filelist
The embedded WolfSSL library has this.

Comment by Otto Kekäläinen [ 2020-02-25 ]

FIled issue in Debian about missing user_settings.h, but it seems it is an optional file. Build should not stop on this?
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=952538

Comment by Otto Kekäläinen [ 2020-02-26 ]

I tried to do a debug build in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/30f27ad426f298e3a4c3eae9130f37e0c66c85e1

..but now compilation fails on all platforms:
https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.4/+builds?build_text=&build_state=all

The error is unclear due to parallel build.

One error on s390x is at least:
https://launchpadlibrarian.net/466640649/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582701031.30f27ad42_BUILDING.txt.gz

/<<PKGBUILDDIR>>/plugin/auth_pam/auth_pam_tool.c: In function ‘main’:
/<<PKGBUILDDIR>>/plugin/auth_pam/auth_pam_tool.c:73:10: error: ignoring return value of ‘setreuid’, declared with attribute warn_unused_result [-Werror=unused-result]
   73 |   (void) setreuid(0, 0);
      |          ^~~~~~~~~~~~~~

On armhf the error is maybe:
https://launchpadlibrarian.net/466643539/buildlog_ubuntu-focal-armhf.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582701031.30f27ad42_BUILDING.txt.gz

cc1: all warnings being treated as errors
make[4]: *** [libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/build.make:336: libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/mariadb_stmt.c.o] Error 1
make[4]: *** Waiting for unfinished jobs....

On other platforms last lines in build log are about RocksDB.

Comment by Marko Mäkelä [ 2020-02-26 ]

Unfortunately, CMAKE_BUILD_TYPE=Debug makes the default setting MYSQL_MAINTAINER_MODE=AUTO add -Werror to the compilation flags. You might want to remove the flag from cmake/maintainer.cmake to work around this (and to still have useful warnings enabled), or you could set MYSQL_MAINTAINER_MODE=OFF.

Comment by Otto Kekäläinen [ 2020-02-26 ]

OK, finally got some debug data. See section starting with

Server version: 10.4.12-MariaDB-1~ubuntu20.04.1~1582728811.ab0850b0a-log

in log https://launchpadlibrarian.net/466692682/buildlog_ubuntu-focal-s390x.mariadb-10.4_1%3A10.4.12-1~ubuntu20.04.1~1582728811.ab0850b0a_BUILDING.txt.gz

stack_bottom = 0x0 thread_stack 0x49000
/<<PKGBUILDDIR>>/builddir/sql/mysqld(my_print_stacktrace+0x2a)[0x12afc293a]
mysys/stacktrace.c:270(my_print_stacktrace)[0x12a9c7ca4]
[0x3ffed0f3f0e]
sql/signal_handler.cc:209(handle_fatal_signal)[0x12af9dfce]
src/tfm.c:3642(fp_copy)[0x12af9fa8c]
src/tfm.c:1601(_fp_exptmod)[0x12afa044e]
src/tfm.c:2277(fp_exptmod)[0x12afa0aa0]
src/tfm.c:4021(fp_prime_miller_rabin_ex)[0x12afa10f6]
src/tfm.c:4283(mp_prime_is_prime_ex)[0x12af8393a]
src/dh.c:2108(_DhSetKey)[0x12af844b6]
src/dh.c:2154(wc_DhSetCheckKey)[0x12af2dade]
src/ssl.c:1744(wolfSSL_CTX_SetTmpDH)[0x12af5083e]
src/ssl.c:38473(wolfSSL_CTX_set_tmp_dh)[0x12b025eba]
vio/viosslfactories.c:340(new_VioSSLFd)[0x12b0260cc]
vio/viosslfactories.c:413(new_VioSSLAcceptorFd)[0x12a6cab58]
/lib/s390x-linux-gnu/libc.so.6(__libc_start_main+0x10a)[0x3ff88f2a90a]
/<<PKGBUILDDIR>>/builddir/sql/mysqld(+0x5bcae4)[0x12a6bcae4]
[0x0]

Comment by Marko Mäkelä [ 2020-02-27 ]

otto, it seems to me that the pointer that _fp_exptmod() is passing as the first parameter of fp_copy() is bad on this architecture for some reason. It seems to me that adding -DWC_NO_CACHE_RESISTANT to the CMAKE_C_FLAGS on this architecture could address the problem.

Comment by Otto Kekäläinen [ 2020-02-27 ]

Fixed downstream in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/d92c7b9c0f0ac2c1bf098f8da4ab6bf4bb6e678f (and Marko's debug tips documented in https://salsa.debian.org/mariadb-team/mariadb-10.4/-/commit/1a0a2abcdf313de6746bb835547d295bf2abaaf6).

I also filed upstream WolfSSL an issue about this: https://github.com/wolfSSL/wolfssl/issues/2828

Based on discussions with serg, it would make sense to also support WITH_SSL=system-wolfssl so I've kept my stub at that public and filed another issue to track if that gets finished: https://jira.mariadb.org/browse/MDEV-21835

Since this is mitigated downstream and upstream is expected to make the final fix, I am closing this issue in MariaDB now.

Comment by Otto Kekäläinen [ 2020-10-08 ]

Tested with MariaDB 10.5.5 with the temporary hack disabled and the build on s390x was OK including the main test suite (736 tests, mysqld did not crash).

--- a/debian/rules
+++ b/debian/rules
@@ -52,15 +52,6 @@ ifneq ($(DEB_BUILD_ARCH),$(DEB_HOST_ARCH))
     endif
 endif
 
-# Add extra flag to avoid WolfSSL code crashing the entire mariadbd on s390x. This
-# can be removed once upstream has made the code s390x compatible, see
-# https://jira.mariadb.org/browse/MDEV-21705 and
-# https://github.com/wolfSSL/wolfssl/issues/2828
-#ifeq ($(DEB_HOST_ARCH),s390x)
-#    CFLAGS += -DWC_NO_CACHE_RESISTANT
-#endif

--> https://launchpadlibrarian.net/501100344/buildlog_ubuntu-groovy-s390x.mariadb-10.5_1%3A10.5.5-3~exp2~ubuntu20.10.1~1602100063.bde2cf481.master_BUILDING.txt.gz

Comment by Otto Kekäläinen [ 2020-10-11 ]

I removed via https://salsa.debian.org/mariadb-team/mariadb-10.5/-/commit/346b704f83c5d21cb734d98989bf6ee771ec5387 the workaround described above and at least WolfSSL in MariaDB 10.5.5 in Debian seemed to build fine now without it. Build logs at https://buildd.debian.org/status/package.php?p=mariadb-10.5

Comment by Daniel Black [ 2021-08-12 ]

Debian is now using openssl, however as a test - https://buildbot.mariadb.org/#/builders/309/builds/534/steps/6/logs/stdio passes on s390x, so WolfSSL folks have indeed fixed this.

Generated at Thu Feb 08 09:09:11 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.