Marko Mäkelä
added a comment - I can still repeat the problem.
10.2 b0ef1b388bff28b93bfdf83a97723a56c869ca34, libmariadb v3.0.6
main.connect_debug [ fail ]
Test ended at 2018-08-22 14:25:22
CURRENT_TEST: main.connect_debug
=================================================================
==24535==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x00000059f752 bp 0x7ffc5fe6b320 sp 0x7ffc5fe6b318
READ of size 1 at 0x629000005200 thread T0
#0 0x59f751 in client_mpvio_read_packet /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18
#1 0x5a0fb0 in auth_old_password /mariadb/10.2/libmariadb/plugins/auth/old_password.c:91:19
#2 0x59ed07 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:548:8
#3 0x557786 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1501:7
#4 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
#5 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
#6 0x547035 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
#7 0x5394a5 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
#8 0x5386ea in main /mariadb/10.2/client/mysql.cc:1209:7
#9 0x7fdf660edb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
#10 0x43be19 in _start (/dev/shm/10.2/client/mysql+0x43be19)
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
freed by thread T0 here:
#0 0x500a20 in __interceptor_free.localalias.0 (/dev/shm/10.2/client/mysql+0x500a20)
#1 0x5a9612 in ma_net_end /mariadb/10.2/libmariadb/libmariadb/ma_net.c:114:3
previously allocated by thread T0 here:
#0 0x500bf0 in __interceptor_malloc (/dev/shm/10.2/client/mysql+0x500bf0)
#1 0x5a91f3 in ma_net_init /mariadb/10.2/libmariadb/libmariadb/ma_net.c:83:28
#2 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
#3 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18 in client_mpvio_read_packet
Shadow bytes around the buggy address:
0x0c527fff89f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c527fff8a40:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==24535==ABORTING
Aborted
mysqltest: At line 10: command "$MYSQL --default-auth=mysql_old_password --user=bad --password=worse" failed with wrong error: 134
Note: I had to edit cmake/submodules.cmake to force the use of libmariadb v3.0.6:
(cd libmariadb; git checkout v3.0.6)
$EDITOR cmake/submodules.cmake
mkdir build
cd build
CC=clang-6.0 CXX=clang++-6.0 CFLAGS=-O1 CXXFLAGS=-O1 cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON ..
cd mysql-test
ASAN_OPTIONS=abort_on_error=1,detect_leaks=0 ./mtr main.connect_debug
#4 0x553620 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
#5 0x53ac33 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
#6 0x5396c6 in main /mariadb/10.2/client/mysql.cc:1209:7
#7 0x7fcb9459fb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
#8 0x43be49 in _start (/dev/shm/10.2/client/mysql+0x43be49)
The source line is:
if (res > CR_OK && mysql->net.read_pos[0] != 254)
For some reason, res==0 there, so the condition res > -1 will hold.
Marko Mäkelä
added a comment - Note: When compiling with -O1 I got a wrong value pkt_len=0 displayed by gdb. With -O0 it made more sense:
(gdb) i lo
mpvio = 0x7fffffffc840
mysql = 0x19ccc60 <mysql>
pkt_len = 4294967295
(gdb) p *mpvio
$1 = {read_packet = 0x606970 <client_mpvio_read_packet>,
write_packet = 0x606f00 <client_mpvio_write_packet>, info = 0x6072e0 <client_mpvio_info>,
mysql = 0x19ccc60 <mysql>, plugin = 0xa754e0 <mysql_old_password_client_plugin>, db = 0x0,
cached_server_reply = {pkt = 0x0, pkt_len = 0}, packets_read = 0, packets_written = 1,
mysql_change_user = 0 '\000', last_read_packet_len = -1}
With the following fix, it fails a little later:
diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c
index 7c84c78..2d4b39c 100644
--- a/plugins/auth/my_auth.c
+++ b/plugins/auth/my_auth.c
@@ -357,6 +357,8 @@ static int client_mpvio_read_packet(struct st_plugin_vio *mpv, uchar **buf)
mpvio->last_read_packet_len= pkt_len;
*buf= mysql->net.read_pos;
+ if (pkt_len == packet_error)
+ return (int)packet_error;
/* was it a request to change plugins ? */
if (pkt_len && **buf == 254)
return (int)packet_error; /* if yes, this plugin shan't continue */
It still fails on the same address as *buf :
==18127==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x000000605d44 bp 0x7ffcc69446b0 sp 0x7ffcc69446a8
READ of size 1 at 0x629000005200 thread T0
#0 0x605d43 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:552:22
#1 0x5759f7 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1501:7
#2 0x57223b in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
#3 0x53ce06 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
#4 0x553620 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
#5 0x53ac33 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
#6 0x5396c6 in main /mariadb/10.2/client/mysql.cc:1209:7
#7 0x7fcb9459fb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
#8 0x43be49 in _start (/dev/shm/10.2/client/mysql+0x43be49)
The source line is:
if (res > CR_OK && mysql->net.read_pos[0] != 254)
For some reason, res==0 there, so the condition res > -1 will hold.
I do not see this failing with newer MariaDB server versions, but I do see the heap-use-after-free when testing the newest Connector/C library against the old MariaDB 10.2 server that I mentioned earlier:
Maybe, when run against a newer server, the handshake fails earlier, and the problematic code in Connector/C does not get executed. The problem appears to affect connections to older servers only. Here is the result from the above:
Marko Mäkelä
added a comment - I do not see this failing with newer MariaDB server versions, but I do see the heap-use-after-free when testing the newest Connector/C library against the old MariaDB 10.2 server that I mentioned earlier:
git checkout b0ef1b388bff28b93bfdf83a97723a56c869ca34
(cd libmariadb; git checkout v3.0.8-release)
echo 'RETURN()' > cmake/submodules.cmake
cmake -DWITH_ASAN=ON -DWITH_SAFEMALLOC=OFF …
ASAN_OPTIONS=abort_on_error=1 ./mtr main.connect_debug
Maybe, when run against a newer server, the handshake fails earlier, and the problematic code in Connector/C does not get executed. The problem appears to affect connections to older servers only. Here is the result from the above:
CURRENT_TEST: main.connect_debug
=================================================================
==27725==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x000000596342 bp 0x7fffb2a14380 sp 0x7fffb2a14378
READ of size 1 at 0x629000005200 thread T0
#0 0x596341 in client_mpvio_read_packet /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18
#1 0x597cb0 in auth_old_password /mariadb/10.2/libmariadb/plugins/auth/old_password.c:91:19
#2 0x5958f7 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:548:8
#3 0x53f485 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1502:7
#4 0x53daa2 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1187:10
#5 0x520799 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
#6 0x519642 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
#7 0x519642 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754
#8 0x518843 in main /mariadb/10.2/client/mysql.cc:1209:7
#9 0x7f46d837409a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
#10 0x43cdb9 in _start (/dev/shm/10.2/client/mysql+0x43cdb9)
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
freed by thread T0 here:
#0 0x4e4882 in free (/dev/shm/10.2/client/mysql+0x4e4882)
#1 0x5a3672 in ma_net_end /mariadb/10.2/libmariadb/libmariadb/ma_net.c:114:3
previously allocated by thread T0 here:
#0 0x4e4c03 in __interceptor_malloc (/dev/shm/10.2/client/mysql+0x4e4c03)
#1 0x5a3253 in ma_net_init /mariadb/10.2/libmariadb/libmariadb/ma_net.c:83:28
#2 0x53daa2 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1187:10
#3 0x520799 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
I can still repeat the problem.
10.2 b0ef1b388bff28b93bfdf83a97723a56c869ca34, libmariadb v3.0.6
main.connect_debug [ fail ]
Test ended at 2018-08-22 14:25:22
CURRENT_TEST: main.connect_debug
=================================================================
==24535==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x00000059f752 bp 0x7ffc5fe6b320 sp 0x7ffc5fe6b318
READ of size 1 at 0x629000005200 thread T0
#0 0x59f751 in client_mpvio_read_packet /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18
#1 0x5a0fb0 in auth_old_password /mariadb/10.2/libmariadb/plugins/auth/old_password.c:91:19
#2 0x59ed07 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:548:8
#3 0x557786 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1501:7
#4 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
#5 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
#6 0x547035 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
#7 0x5394a5 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
#8 0x5386ea in main /mariadb/10.2/client/mysql.cc:1209:7
#9 0x7fdf660edb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
#10 0x43be19 in _start (/dev/shm/10.2/client/mysql+0x43be19)
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
freed by thread T0 here:
#0 0x500a20 in __interceptor_free.localalias.0 (/dev/shm/10.2/client/mysql+0x500a20)
#1 0x5a9612 in ma_net_end /mariadb/10.2/libmariadb/libmariadb/ma_net.c:114:3
previously allocated by thread T0 here:
#0 0x500bf0 in __interceptor_malloc (/dev/shm/10.2/client/mysql+0x500bf0)
#1 0x5a91f3 in ma_net_init /mariadb/10.2/libmariadb/libmariadb/ma_net.c:83:28
#2 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
#3 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18 in client_mpvio_read_packet
Shadow bytes around the buggy address:
0x0c527fff89f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c527fff8a30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c527fff8a40:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c527fff8a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==24535==ABORTING
Aborted
mysqltest: At line 10: command "$MYSQL --default-auth=mysql_old_password --user=bad --password=worse" failed with wrong error: 134
Note: I had to edit cmake/submodules.cmake to force the use of libmariadb v3.0.6:
(cd libmariadb; git checkout v3.0.6)
$EDITOR cmake/submodules.cmake
mkdir build
cd build
CC=clang-6.0 CXX=clang++-6.0 CFLAGS=-O1 CXXFLAGS=-O1 cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON ..
cd mysql-test
ASAN_OPTIONS=abort_on_error=1,detect_leaks=0 ./mtr main.connect_debug