[MDEV-15587] AES test fails, segfaults in EVP_CipherInit_ex Created: 2018-03-16  Updated: 2020-01-29  Resolved: 2019-03-29

Status: Closed
Project: MariaDB Server
Component/s: Encryption, SSL, Tests
Affects Version/s: 10.2.13, 10.4.0, 10.3
Fix Version/s: 10.2.24, 10.3.14, 10.4.4

Type: Bug Priority: Major
Reporter: Nicolas G. Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: crash, yassl
Environment:

Debian 8.10, openssl 1.0.2k


Issue Links:
Relates
relates to MDEV-15584 Linux - use O_TMPFILE for create_temp... Closed

 Description   

The aes test always fails:

gdb /home/mariadb/mariadb-10.2.13/unittest/mysys/aes-t
GNU gdb (Debian 7.11.1-2~bpo8+1) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/mariadb/mariadb-10.2.13/unittest/mysys/aes-t...done.
(gdb) r
Starting program: /home/mariadb/mariadb-10.2.13/unittest/mysys/aes-t 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
1..87
ok 1 - encrypt MY_AES_ECB 200 pad
ok 2 - my_aes_get_size
ok 3 - md5
ok 4 - decrypt MY_AES_ECB 208
ok 5 - memcmp
ok 6 - encrypt MY_AES_ECB 128 pad
ok 7 - my_aes_get_size
ok 8 - md5
ok 9 - decrypt MY_AES_ECB 144
ok 10 - memcmp
ok 11 - encrypt MY_AES_CBC 159 pad
ok 12 - my_aes_get_size
ok 13 - md5
ok 14 - decrypt MY_AES_CBC 160
ok 15 - memcmp
ok 16 - encrypt MY_AES_CBC 192 pad
ok 17 - my_aes_get_size
ok 18 - md5
ok 19 - decrypt MY_AES_CBC 208
ok 20 - memcmp
 
Program received signal SIGSEGV, Segmentation fault.
EVP_CipherInit_ex (enc=1, iv=0x0, key=<optimized out>, cipher=0x555555600d04 <EVP_aes_128_ecb()::c>, ctx=0x7fffffffd208) at /home/mariadb/mariadb-10.2.13/mysys_ssl/yassl.cc:100
100         TAO(ctx)->SetIV(iv);
(gdb) quit



 Comments   
Comment by Daniel Black [ 2018-03-17 ]

Ok, odd questions first, how is it gdb has reversed the arguments of EVP_CipherInit_ex from right to left?

And the yassl.c line 99 is if (iv) so how did line 100 get executed when iv=0?

What compiler and cflags have been used in the build?

Comment by Nicolas G. [ 2018-03-19 ]

I can't answer your first two questions. I've downloaded the standard 10.2.13 tarball, didn't modify any file, just ran "cmake && make" (using GCC 4.9.2, which is the default version under Debian Jessie).

Here are the compilation flags:

[ 28%] Building C object unittest/mysys/CMakeFiles/aes-t.dir/aes-t.c.o
cd /home/mariadb/mariadb-10.2.13/unittest/mysys && /usr/bin/cc -DHAVE_CONFIG_H -DHAVE_SYSTEMD -D_FILE_OFFSET_BITS=64 -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -DWITH_INNODB_DISALLOW_WRITES -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -I/home/mariadb/mariadb-10.2.13/include -I/home/mariadb/mariadb-10.2.13/unittest/mytap -DHAVE_YASSL -DYASSL_PREFIX -DHAVE_OPENSSL -DMULTI_THREADED -o CMakeFiles/aes-t.dir/aes-t.c.o -c /home/mariadb/mariadb-10.2.13/unittest/mysys/aes-t.c
Linking CXX executable aes-t
cd /home/mariadb/mariadb-10.2.13/unittest/mysys && /usr/bin/cmake -E cmake_link_script CMakeFiles/aes-t.dir/link.txt --verbose=1
/usr/bin/c++ -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -DWITH_INNODB_DISALLOW_WRITES -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF CMakeFiles/aes-t.dir/aes-t.c.o -o aes-t -lpthread ../mytap/libmytap.a ../../mysys/libmysys.a ../../dbug/libdbug.a ../../mysys_ssl/libmysys_ssl.a ../../mysys/libmysys.a ../../dbug/libdbug.a ../../mysys_ssl/libmysys_ssl.a ../../zlib/libzlib.a -lm -ldl ../../strings/libstrings.a ../../extra/yassl/libyassl.a ../../extra/yassl/taocrypt/libtaocrypt.a -lpthread
make[2]: Leaving directory '/home/mariadb/mariadb-10.2.13'
/usr/bin/cmake -E cmake_progress_report /home/mariadb/mariadb-10.2.13/CMakeFiles
[ 28%] Built target aes-t

Comment by Daniel Black [ 2018-04-30 ]

FWIW- managed to reproduce on FC28. Fails all mtr tests with enabled binlog

cmake -DWITH_SSL=bundled  ../mariadb-server-10.3/
 
Core was generated by `/home/dan/repos/build-mariadb-server-10.3/sql/mysqld --defaults-group-suffix=.1'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
56	  return (INTERNAL_SYSCALL_ERROR_P (val, err)
[Current thread is 1 (Thread 0x7fd95fa28740 (LWP 15326))]
(gdb) bt
#0  __pthread_kill (threadid=<optimized out>, signo=signo@entry=11) at ../sysdeps/unix/sysv/linux/pthread_kill.c:56
#1  0x000055eebce1c0aa in my_write_core (sig=sig@entry=11) at /home/dan/repos/mariadb-server-10.3/mysys/stacktrace.c:481
#2  0x000055eebc9b2374 in handle_fatal_signal (sig=11) at /home/dan/repos/mariadb-server-10.3/sql/signal_handler.cc:305
#3  <signal handler called>
#4  EVP_CipherInit_ex (cipher=0x55eebd012858 <EVP_aes_128_ecb()::c>, cipher=0x55eebd012858 <EVP_aes_128_ecb()::c>, enc=1, iv=0x0, 
    key=0x55eebd80abf8 <mysql_bin_log+3064> "w\n\212e\332\025m$\356*\t2wS\001B", ctx=<optimized out>) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/yassl.cc:101
#5  MyCTX::init (ivlen=0, iv=0x0, klen=<optimized out>, key=0x55eebd80abf8 <mysql_bin_log+3064> "w\n\212e\332\025m$\356*\t2wS\001B", encrypt=1, 
    cipher=0x55eebd012858 <EVP_aes_128_ecb()::c>, this=0x7ffe52f201a0) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:57
#6  MyCTX_nopad::init (this=0x7ffe52f201a0, cipher=0x55eebd012858 <EVP_aes_128_ecb()::c>, encrypt=1, 
    key=0x55eebd80abf8 <mysql_bin_log+3064> "w\n\212e\332\025m$\356*\t2wS\001B", klen=<optimized out>, iv=0x0, ivlen=0)
    at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:99
#7  0x000055eebce30dd0 in my_aes_crypt (mode=mode@entry=MY_AES_ECB, flags=flags@entry=3, src=src@entry=0x7ffe52f20768 "\033\333\365M\335r\025n\263\362\035T(\001", 
    slen=slen@entry=16, dst=dst@entry=0x7ffe52f20440 "\004", dlen=dlen@entry=0x7ffe52f2043c, 
    key=0x55eebd80abf8 <mysql_bin_log+3064> "w\n\212e\332\025m$\356*\t2wS\001B", klen=16, iv=0x0, ivlen=0)
    at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:289
#8  0x000055eebce31553 in MyCTX_nopad::finish (this=0x7ffe52f205d0, dst=0x7ffe52f204c0 "F\230\241B\020", dlen=0x7ffe52f204bc)
    at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:129
#9  0x000055eebce30d4f in my_aes_crypt_finish (ctx=0x7ffe52f205d0, dst=<optimized out>, dlen=<optimized out>)
    at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:277
#10 0x000055eebca85a42 in Log_event_writer::write_footer (this=0x7ffe52f207e0) at /home/dan/repos/mariadb-server-10.3/sql/log_event.cc:1691
#11 0x000055eebca8c4de in Log_event::write_footer (this=0x7ffe52f20920) at /home/dan/repos/mariadb-server-10.3/sql/log_event.h:1373
#12 Gtid_list_log_event::write() () at /home/dan/repos/mariadb-server-10.3/sql/log_event.cc:8361
#13 0x000055eebca6b516 in Log_event_writer::write (ev=0x7ffe52f20920, this=0x7ffe52f207e0) at /home/dan/repos/mariadb-server-10.3/sql/log_event.h:5180
#14 MYSQL_BIN_LOG::write_event (this=this@entry=0x55eebd80a000 <mysql_bin_log>, ev=ev@entry=0x7ffe52f20920, cache_data=cache_data@entry=0x0, 
    file=file@entry=0x55eebd80a318 <mysql_bin_log+792>) at /home/dan/repos/mariadb-server-10.3/sql/log.cc:5305
#15 0x000055eebca76158 in MYSQL_BIN_LOG::write_event (ev=0x7ffe52f20920, this=0x55eebd80a000 <mysql_bin_log>) at /home/dan/repos/mariadb-server-10.3/sql/log.h:813
#16 MYSQL_BIN_LOG::open(char const*, enum_log_type, char const*, unsigned long, cache_type, unsigned long, bool, bool) ()
    at /home/dan/repos/mariadb-server-10.3/sql/log.cc:3611
#17 0x000055eebc7540a9 in init_server_components () at /home/dan/repos/mariadb-server-10.3/sql/mysqld.cc:5551
#18 0x000055eebc75a0fd in mysqld_main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.3/sql/mysqld.cc:5979
#19 0x00007fd95dcbf1bb in __libc_start_main (main=0x55eebc7395d0 <main(int, char**)>, argc=25, argv=0x7ffe52f21218, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffe52f21208) at ../csu/libc-start.c:308
#20 0x000055eebc74de3a in _start () at /home/dan/repos/mariadb-server-10.3/sql/sql_list.h:142

$  /usr/lib64/ccache/c++  --version
c++ (GCC) 8.0.1 20180324 (Red Hat 8.0.1-0.20)
 
/usr/lib64/ccache/c++   -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -fno-rtti -O2 -g -DNDEBUG -D_FORTIFY_SOURCE=2 -DDBUG_OFF   
 
$ gdb unittest/mysys/aes-t 
GNU gdb (GDB) Fedora 8.1-11.fc28
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from unittest/mysys/aes-t...done.
(gdb) run
Starting program: /home/dan/repos/build-mariadb-server-10.3/unittest/mysys/aes-t 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
1..87
ok 1 - encrypt MY_AES_ECB 200 pad
ok 2 - my_aes_get_size
ok 3 - md5
ok 4 - decrypt MY_AES_ECB 208
ok 5 - memcmp
ok 6 - encrypt MY_AES_ECB 128 pad
ok 7 - my_aes_get_size
ok 8 - md5
ok 9 - decrypt MY_AES_ECB 144
ok 10 - memcmp
ok 11 - encrypt MY_AES_CBC 159 pad
ok 12 - my_aes_get_size
ok 13 - md5
ok 14 - decrypt MY_AES_CBC 160
ok 15 - memcmp
ok 16 - encrypt MY_AES_CBC 192 pad
ok 17 - my_aes_get_size
ok 18 - md5
ok 19 - decrypt MY_AES_CBC 208
ok 20 - memcmp
 
Program received signal SIGSEGV, Segmentation fault.
EVP_CipherInit_ex (cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, enc=1, iv=0x0, 
    key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", ctx=<optimized out>) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/yassl.cc:101
101	  ctx->encrypt= enc;
(gdb) bt
#0  EVP_CipherInit_ex (cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, enc=1, iv=0x0, 
    key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", ctx=<optimized out>) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/yassl.cc:101
#1  MyCTX::init (ivlen=0, iv=0x0, klen=<optimized out>, key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", encrypt=1, cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, 
    this=0x7fffffffcb80) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:57
#2  MyCTX_nopad::init (this=0x7fffffffcb80, cipher=0x5555555dfd20 <EVP_aes_128_ecb()::c>, encrypt=1, key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", 
    klen=<optimized out>, iv=0x0, ivlen=0) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:99
#3  0x0000555555586370 in my_aes_crypt (mode=mode@entry=MY_AES_ECB, flags=flags@entry=3, src=src@entry=0x7fffffffd028 "\002\003\004\005\006\a\b\t", 
    slen=slen@entry=16, dst=dst@entry=0x7fffffffce20 "\300", dlen=dlen@entry=0x7fffffffce1c, key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", klen=16, iv=0x0, 
    ivlen=0) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:289
#4  0x0000555555586af3 in MyCTX_nopad::finish (this=0x7fffffffce90, dst=0x7fffffffda20 "\313Tc\272\366\035\365N\366#\334\314\353\201\v\031\060\336\377\377\377\177", 
    dlen=0x7fffffffd0b4) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:129
#5  0x00005555555862ef in my_aes_crypt_finish (ctx=ctx@entry=0x7fffffffce90, dst=<optimized out>, dlen=dlen@entry=0x7fffffffd0b4)
    at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:277
#6  0x00005555555863ca in my_aes_crypt (mode=<optimized out>, flags=<optimized out>, src=0x7fffffffd180 '.' <repeats 200 times>..., slen=200, 
    dst=0x7fffffffd960 "\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\276\006c\"\226\066\f\252Zu`\206\372i\200\373\313Tc\272\366\035\365N"..., dlen=0x7fffffffd148, key=0x7fffffffd150 "\001\002\003\004\005\006\a\b\t", 
    klen=16, iv=0x7fffffffd160 "\002\003\004\005\006\a\b\t", ivlen=16) at /home/dan/repos/mariadb-server-10.3/mysys_ssl/my_crypt.cc:292
#7  0x000055555557bb27 in main (argc=<optimized out>, argv=<optimized out>) at /home/dan/repos/mariadb-server-10.3/unittest/mysys/aes-t.c:79

Comment by Cesar Hernandez [ 2018-06-01 ]

Hi

Had the same issue running my compiled mariadb.
After doing some debug on my_crypt.cc and trying different versions of mariadb, I found the best was to use openssl, instead of the integrated yassl.
I solved it rebuilding mariadb adding the following setting to cmake:

DWITH_SSL=/usr/local/openssl/

Versions:

debian 8.4
mariadb-10.2.14
openssl-1.0.2o

Cesar Hernandez

Comment by Marko Mäkelä [ 2018-10-31 ]

I get various crashes in encryption-related code when building with GCC 8.2.0 and with at least -O1. Both for Debug (where I have to explicitly specify -O1) and RelWithDebInfo (which defaults to (-O2).
Here is an example of a server crash during startup, while running ./mtr --suite=encryption:

10.2 dc91ea5bb762f8ef9712babd8661ed6ae31bf2d6

#2  0x0000555c0ab55ae4 in handle_fatal_signal (sig=11) at /mariadb/10.2/sql/signal_handler.cc:305
#3  <signal handler called>
#4  EVP_CipherInit_ex (cipher=0x555c0b0e37d8 <EVP_aes_128_ecb()::c>, cipher=0x555c0b0e37d8 <EVP_aes_128_ecb()::c>, enc=1, iv=0x0, key=0x7ffce8f1d3f0 "\347\260\264\335\221~\252\204\252\310j\200\314*\255\265\n\356%\206?mRJ\244\217uo\321\026\337\360\357\374L|kC\031CP\306r\224\365\334y\270p\337\245\316\343x,\030]\362f\252\210\027\334\210\001\202\300\360\025\345\311\066\224\304\310GiD\254\002\303Tc\037\242~\305\365\277\266\204\302W", ctx=<optimized out>) at /mariadb/10.2/mysys_ssl/yassl.cc:101
#5  MyCTX::init (ivlen=0, iv=0x0, klen=<optimized out>, key=0x7ffce8f1d3f0 "\347\260\264\335\221~\252\204\252\310j\200\314*\255\265\n\356%\206?mRJ\244\217uo\321\026\337\360\357\374L|kC\031CP\306r\224\365\334y\270p\337\245\316\343x,\030]\362f\252\210\027\334\210\001\202\300\360\025\345\311\066\224\304\310GiD\254\002\303Tc\037\242~\305\365\277\266\204\302W", encrypt=1, cipher=0x555c0b0e37d8 <EVP_aes_128_ecb()::c>, this=0x7ffce8f1d150) at /mariadb/10.2/mysys_ssl/my_crypt.cc:57
#6  MyCTX_nopad::init (this=0x7ffce8f1d150, cipher=0x555c0b0e37d8 <EVP_aes_128_ecb()::c>, encrypt=1, key=0x7ffce8f1d3f0 "\347\260\264\335\221~\252\204\252\310j\200\314*\255\265\n\356%\206?mRJ\244\217uo\321\026\337\360\357\374L|kC\031CP\306r\224\365\334y\270p\337\245\316\343x,\030]\362f\252\210\027\334\210\001\202\300\360\025\345\311\066\224\304\310GiD\254\002\303Tc\037\242~\305\365\277\266\204\302W", klen=<optimized out>, iv=0x0, ivlen=0) at /mariadb/10.2/mysys_ssl/my_crypt.cc:99
#7  0x0000555c0af840e0 in my_aes_crypt (mode=mode@entry=MY_AES_ECB, flags=flags@entry=3, src=src@entry=0x555c0b699d8c <info+12> "\240\202\314\036OD\030\342\244\324v;\206k\035\f", slen=slen@entry=16, dst=dst@entry=0x555c0b699d9c <info+28> "", dlen=dlen@entry=0x7ffce8f1d3ec, key=0x7ffce8f1d3f0 "\347\260\264\335\221~\252\204\252\310j\200\314*\255\265\n\356%\206?mRJ\244\217uo\321\026\337\360\357\374L|kC\031CP\306r\224\365\334y\270p\337\245\316\343x,\030]\362f\252\210\027\334\210\001\202\300\360\025\345\311\066\224\304\310GiD\254\002\303Tc\037\242~\305\365\277\266\204\302W", klen=16, iv=0x0, ivlen=0) at /mariadb/10.2/mysys_ssl/my_crypt.cc:289
#8  0x0000555c0ad0ec72 in init_crypt_key(crypt_info_t*, bool) () at /mariadb/10.2/storage/innobase/log/log0crypt.cc:173
#9  0x0000555c0ad0eef9 in log_crypt_init() () at /mariadb/10.2/storage/innobase/log/log0crypt.cc:218
#10 0x0000555c0a92cf71 in create_log_files (logfilename=logfilename@entry=0x7ffce8f21050 "./ib_logfile1", dirnamelen=dirnamelen@entry=2, lsn=1633879, logfile0=@0x7ffce8f1d998: 0x555c0cecd670 "./ib_logfile101") at /mariadb/10.2/storage/innobase/include/ib0mutex.h:603
#11 0x0000555c0a92f2cb in innobase_start_or_create_for_mysql () at /mariadb/10.2/storage/innobase/srv/srv0start.cc:2428
#12 0x0000555c0acaf457 in innobase_init(void*) () at /mariadb/10.2/storage/innobase/handler/ha_innodb.cc:4361
#13 0x0000555c0ab57815 in ha_initialize_handlerton (plugin=0x555c0cd60530) at /mariadb/10.2/sql/handler.cc:521

I was unable to repeat this on MariaDB Server 10.1, so it must be something specific to 10.2 and later.

Possibly related to this, I remember seeing a number of warnings for incompatible assignments to function pointers, in encryption code, either from GCC 8 or clang 7.

With -O0 or with clang, the code does not crash for me.

Comment by Marko Mäkelä [ 2018-11-05 ]

The above crash was with amd64. I also see it with i686. Here is a different stack trace from binlog_encryption.rpl_relayrotate:

#3  0x56b2185f in handle_fatal_signal (sig=11) at /mariadb/10.2/sql/signal_handler.cc:305
#4  <signal handler called>
#5  0x5706bf59 in memcpy (__len=16, __src=0x0, __dest=<optimized out>) at /usr/include/i386-linux-gnu/bits/string_fortified.h:34
#6  TaoCrypt::AES::SetIV (iv=0x0, this=<optimized out>) at /mariadb/10.2/extra/yassl/taocrypt/include/aes.hpp:54
#7  EVP_CipherInit_ex (cipher=0x57202574 <EVP_aes_128_ecb()::c>, cipher=0x57202574 <EVP_aes_128_ecb()::c>, enc=1, iv=0x0, key=0x5770e394 <mysql_bin_log+2580> "w\n\212e\332\025m$\356*\t2wS\001B", ctx=<optimized out>) at /mariadb/10.2/mysys_ssl/yassl.cc:100
#8  MyCTX::init (ivlen=<optimized out>, iv=0x0, klen=<optimized out>, key=0x5770e394 <mysql_bin_log+2580> "w\n\212e\332\025m$\356*\t2wS\001B", encrypt=1, cipher=0x57202574 <EVP_aes_128_ecb()::c>, this=0xffab49d0) at /mariadb/10.2/mysys_ssl/my_crypt.cc:57
#9  MyCTX_nopad::init (this=<optimized out>, cipher=<optimized out>, encrypt=<optimized out>, key=<optimized out>, klen=<optimized out>, iv=<optimized out>, ivlen=<optimized out>) at /mariadb/10.2/mysys_ssl/my_crypt.cc:99
#10 0x5706b8e2 in my_aes_crypt (mode=mode@entry=MY_AES_ECB, flags=flags@entry=3, src=src@entry=0xffab4f74 "\232\021l\355\264\350/\334\rpP\336(\001", slen=slen@entry=16, dst=dst@entry=0xffab4c6c "\030\330\300V\001", dlen=dlen@entry=0xffab4c68, key=0x5770e394 <mysql_bin_log+2580> "w\n\212e\332\025m$\356*\t2wS\001B", klen=16, iv=iv@entry=0x0, ivlen=ivlen@entry=0) at /mariadb/10.2/mysys_ssl/my_crypt.cc:289
#11 0x5706c468 in MyCTX_nopad::finish (this=0xffab4e00, dst=0xffab4cec "\021\300J\357\034P\253\377\\M\253\377\006", dlen=0xffab4ce8) at /mariadb/10.2/mysys_ssl/my_crypt.cc:129
#12 0x5706b850 in my_aes_crypt_finish (ctx=0xffab4e00, dst=0xffab4cec "\021\300J\357\034P\253\377\\M\253\377\006", dlen=0xffab4ce8) at /mariadb/10.2/mysys_ssl/my_crypt.cc:277
#13 0x56c0e2b8 in Log_event_writer::write_footer (this=0xffab501c) at /mariadb/10.2/sql/log_event.cc:1688
#14 0x56c15f6b in Log_event::write_footer (this=0xffab50ec) at /mariadb/10.2/sql/log_event.h:1345
#15 Gtid_list_log_event::write() () at /mariadb/10.2/sql/log_event.cc:7951
#16 0x56bf003f in Log_event_writer::write (ev=<optimized out>, this=0xffab501c) at /mariadb/10.2/sql/log_event.h:5164
#17 MYSQL_BIN_LOG::write_event (this=<optimized out>, this@entry=0x5770d980 <mysql_bin_log>, ev=<optimized out>, ev@entry=0xffab50ec, file=<optimized out>, file@entry=0x5770dc7c <mysql_bin_log+764>) at /mariadb/10.2/sql/log.cc:5250
#18 0x56bfc407 in MYSQL_BIN_LOG::write_event (ev=0xffab50ec, this=0x5770d980 <mysql_bin_log>) at /mariadb/10.2/sql/log.h:754
#19 MYSQL_BIN_LOG::open(char const*, enum_log_type, char const*, unsigned long, cache_type, unsigned long, bool, bool) () at /mariadb/10.2/sql/log.cc:3559
#20 0x568e200f in init_server_components () at /mariadb/10.2/sql/mysqld.cc:5458
#21 mysqld_main (argc=<optimized out>, argv=<optimized out>) at /mariadb/10.2/sql/mysqld.cc:5891
#22 0x568bc627 in main (argc=28, argv=0xffab57e4) at /mariadb/10.2/sql/main.cc:25

In both stack traces, the problem could be iv=0x0.
If I remember correctly from a debugging session a couple of months ago, the parameter should have been passed as an address of a stack-local variable. Maybe some function attribute could be provoking wrong behaviour from GCC? This would not be the first time: I encountered something similar in GCC 4.6.3, 4.7.2, 4.9.1 years ago.

Comment by Marko Mäkelä [ 2019-03-12 ]

For me, the problem goes away if I ensure that CMAKE_CXX_FLAGS contain -O0 instead of -O1 or -O2. The CMAKE_C_FLAGS can be anything. I also tested 10.2 with CMAKE_CXX_FLAGS=-O2 -fno-lifetime-dse, but disabling that optimization did not fix the issue.

Strangely, 10.1 is not affected, even though I build it with the same options, and InnoDB uses encryption in both.

Comment by Eugene Kosov (Inactive) [ 2019-03-29 ]

#5  0x5706bf59 in memcpy (__len=16, __src=0x0, __dest=<optimized out>) at /usr/include/i386-linux-gnu/bits/string_fortified.h:34

Argument order here looks unusual.

Comment by Marko Mäkelä [ 2019-03-29 ]

I believe that the problem is that in EVP_CipherInit_ex(), a condition on iv is wrongly being optimized away.
I see that there are several callers that invoke my_aes_crypt(…, iv=NULL, ivlen=0).
The problematic code is here:

static int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *cipher,
                             void *, const uchar *key, const uchar *iv, int enc)
{
  new (ctx->tao_buf) TaoCrypt::AES(enc ? TaoCrypt::ENCRYPTION
                                       : TaoCrypt::DECRYPTION, cipher->mode);
  TAO(ctx)->SetKey(key, cipher->key_len);
  if (iv)
    TAO(ctx)->SetIV(iv);

The following patch works around the issue for me:

diff --git a/mysys_ssl/yassl.cc b/mysys_ssl/yassl.cc
--- a/mysys_ssl/yassl.cc
+++ b/mysys_ssl/yassl.cc
@@ -90,6 +90,7 @@ static int EVP_CIPHER_CTX_set_padding(EVP_CIPHER_CTX *ctx, int pad)
  return 1;@@ -90,6 +90,7 @@ static int EVP_CIPHER_CTX_set_padding(EVP_CIPHER_CTX *ctx, int pad)
   return 1;
 }
 
+__attribute__((noinline))
 static int EVP_CipherInit_ex(EVP_CIPHER_CTX *ctx, const EVP_CIPHER *cipher,
                              void *, const uchar *key, const uchar *iv, int enc)
 {

Comment by Marko Mäkelä [ 2019-03-29 ]

It looks like the call memcpy(oiv, iv, ivlen=0) allowed GCC to infer that both pointers must be __attribute__((nonnull)).

Generated at Thu Feb 08 08:22:31 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.