[CONC-345] heap-use-after-free in client_mpvio_read_packet Created: 2018-07-02  Updated: 2021-05-07  Resolved: 2021-05-07

Status: Closed
Project: MariaDB Connector/C
Component/s: None
Affects Version/s: None
Fix Version/s: 3.0.7

Type: Bug Priority: Major
Reporter: Marko Mäkelä Assignee: Georg Richter
Resolution: Fixed Votes: 0
Labels: ASAN, corruption
Environment:

Debian GNU/Linux unstable

Issue Links:
Relates
relates to MDEV-12361 The MariaDB Server never returns to c... Closed

 Description   

When compiling the code with clang 6.0 and cmake -DWITH_ASAN:BOOL=ON I got this error when running tests with

ASAN_OPTIONS=abort_on_error=1,disable_coredump=0,detect_leaks=0 ./mtr --parallel=auto --force --retry=0 --max-test-fail=0

10.3 71144afa966a85d08053eb616a1021fd339102d1, libmariadb a12a0b8362fe8c92ec7252c8da19c14d22e289fc

 
CURRENT_TEST: main.connect_debug
 
=================================================================
 
==7822==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x0000005a18b5 bp 0x7fff77936f60 sp 0x7fff77936f58
 
READ of size 1 at 0x629000005200 thread T0
 
    #0 0x5a18b4 in client_mpvio_read_packet /mariadb/10.3m/libmariadb/plugins/auth/my_auth.c:360:7
 
    #1 0x5a3120 in auth_old_password /mariadb/10.3m/libmariadb/plugins/auth/old_password.c:91:19
 
    #2 0x5a0e94 in run_plugin_auth /mariadb/10.3m/libmariadb/plugins/auth/my_auth.c:547:8
 
    #3 0x55a14f in mthd_my_real_connect /mariadb/10.3m/libmariadb/libmariadb/mariadb_lib.c:1499:7
 
    #4 0x558ba2 in mysql_real_connect /mariadb/10.3m/libmariadb/libmariadb/mariadb_lib.c:1183:10
 
    #5 0x53cc09 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.3m/client/mysql.cc:1389:10
 
    #6 0x5490e5 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.3m/client/mysql.cc:4702:8
 
    #7 0x53b6b5 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.3m/client/mysql.cc:4750:16
 
    #8 0x53a8f2 in main /mariadb/10.3m/client/mysql.cc:1207:7
 
    #9 0x7f87d58e0a86 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21a86)
 
    #10 0x43ce59 in _start (/dev/shm/10.3d/client/mysql+0x43ce59)
 
 
 
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
 
freed by thread T0 here:
 
    #0 0x4fcb40 in __interceptor_free.localalias.0 (/dev/shm/10.3d/client/mysql+0x4fcb40)
 
    #1 0x5ab7c2 in ma_net_end /mariadb/10.3m/libmariadb/libmariadb/ma_net.c:114:3
 
 
 
previously allocated by thread T0 here:
 
    #0 0x4fcd10 in __interceptor_malloc (/dev/shm/10.3d/client/mysql+0x4fcd10)
 
    #1 0x5ab3a3 in ma_net_init /mariadb/10.3m/libmariadb/libmariadb/ma_net.c:83:28
 
    #2 0x558ba2 in mysql_real_connect /mariadb/10.3m/libmariadb/libmariadb/mariadb_lib.c:1183:10
 
    #3 0x53cc09 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.3m/client/mysql.cc:1389:10
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.3m/libmariadb/plugins/auth/my_auth.c:360:7 in client_mpvio_read_packet
 
Shadow bytes around the buggy address:
 
  0x0c527fff89f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
=>0x0c527fff8a40:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
==7822==ABORTING
 
Aborted
 
mysqltest: At line 10: command "$MYSQL --default-auth=mysql_old_password --user=bad --password=worse" failed with wrong error: 134

It looks like some error handling is wrong in Connector/C. The test is trying to misauthenticate:

source include/have_debug.inc;
 
set @old_dbug=@@global.debug_dbug;
 
 
 
#
 
# use after free if need plugin change and auth aborted
 
#
 
set global debug_dbug='+d,auth_disconnect';
 
create user 'bad' identified by 'worse';
 
--error 1
 
--exec $MYSQL --default-auth=mysql_old_password --user=bad --password=worse
 
set global debug_dbug=@old_dbug;
 
drop user bad;

It is the exec statement that fails. I wonder if this could explain MDEV-12361.

 Comments   
Comment by Marko Mäkelä [ 2018-08-22 ]

I can still repeat the problem.

10.2 b0ef1b388bff28b93bfdf83a97723a56c869ca34, libmariadb v3.0.6

 
main.connect_debug                       [ fail ]
 
        Test ended at 2018-08-22 14:25:22
 
 
 
CURRENT_TEST: main.connect_debug
 
=================================================================
 
==24535==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x00000059f752 bp 0x7ffc5fe6b320 sp 0x7ffc5fe6b318
 
READ of size 1 at 0x629000005200 thread T0
 
    #0 0x59f751 in client_mpvio_read_packet /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18
 
    #1 0x5a0fb0 in auth_old_password /mariadb/10.2/libmariadb/plugins/auth/old_password.c:91:19
 
    #2 0x59ed07 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:548:8
 
    #3 0x557786 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1501:7
 
    #4 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
 
    #5 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
 
    #6 0x547035 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
 
    #7 0x5394a5 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
 
    #8 0x5386ea in main /mariadb/10.2/client/mysql.cc:1209:7
 
    #9 0x7fdf660edb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
 
    #10 0x43be19 in _start (/dev/shm/10.2/client/mysql+0x43be19)
 
 
 
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
 
freed by thread T0 here:
 
    #0 0x500a20 in __interceptor_free.localalias.0 (/dev/shm/10.2/client/mysql+0x500a20)
 
    #1 0x5a9612 in ma_net_end /mariadb/10.2/libmariadb/libmariadb/ma_net.c:114:3
 
 
 
previously allocated by thread T0 here:
 
    #0 0x500bf0 in __interceptor_malloc (/dev/shm/10.2/client/mysql+0x500bf0)
 
    #1 0x5a91f3 in ma_net_init /mariadb/10.2/libmariadb/libmariadb/ma_net.c:83:28
 
    #2 0x556232 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
 
    #3 0x53a9f9 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
 
 
 
SUMMARY: AddressSanitizer: heap-use-after-free /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18 in client_mpvio_read_packet
 
Shadow bytes around the buggy address:
 
  0x0c527fff89f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a10: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a20: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
  0x0c527fff8a30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
 
=>0x0c527fff8a40:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a60: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
  0x0c527fff8a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
 
Shadow byte legend (one shadow byte represents 8 application bytes):
 
  Addressable:           00
 
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
 
  Freed heap region:       fd
 
  Stack left redzone:      f1
 
  Stack mid redzone:       f2
 
  Stack right redzone:     f3
 
  Stack after return:      f5
 
  Stack use after scope:   f8
 
  Global redzone:          f9
 
  Global init order:       f6
 
  Poisoned by user:        f7
 
  Container overflow:      fc
 
  Array cookie:            ac
 
  Intra object redzone:    bb
 
  ASan internal:           fe
 
  Left alloca redzone:     ca
 
  Right alloca redzone:    cb
 
==24535==ABORTING
 
Aborted
 
mysqltest: At line 10: command "$MYSQL --default-auth=mysql_old_password --user=bad --password=worse" failed with wrong error: 134

Note: I had to edit cmake/submodules.cmake to force the use of libmariadb v3.0.6:

(cd libmariadb; git checkout v3.0.6)
 
$EDITOR cmake/submodules.cmake
 
mkdir build
 
cd build
 
CC=clang-6.0 CXX=clang++-6.0 CFLAGS=-O1 CXXFLAGS=-O1 cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON ..
 
cd mysql-test
 
ASAN_OPTIONS=abort_on_error=1,detect_leaks=0 ./mtr main.connect_debug

Comment by Marko Mäkelä [ 2018-08-22 ]

Note: When compiling with -O1 I got a wrong value pkt_len=0 displayed by gdb. With -O0 it made more sense:

(gdb) i lo
 
mpvio = 0x7fffffffc840
 
mysql = 0x19ccc60 <mysql>
 
pkt_len = 4294967295
 
(gdb) p *mpvio
 
$1 = {read_packet = 0x606970 <client_mpvio_read_packet>, 
  write_packet = 0x606f00 <client_mpvio_write_packet>, info = 0x6072e0 <client_mpvio_info>, 
  mysql = 0x19ccc60 <mysql>, plugin = 0xa754e0 <mysql_old_password_client_plugin>, db = 0x0, 
  cached_server_reply = {pkt = 0x0, pkt_len = 0}, packets_read = 0, packets_written = 1, 
  mysql_change_user = 0 '\000', last_read_packet_len = -1}

With the following fix, it fails a little later:

diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c
 
index 7c84c78..2d4b39c 100644
 
--- a/plugins/auth/my_auth.c
 
+++ b/plugins/auth/my_auth.c
 
@@ -357,6 +357,8 @@ static int client_mpvio_read_packet(struct st_plugin_vio *mpv, uchar **buf)
 
   mpvio->last_read_packet_len= pkt_len;
 
   *buf= mysql->net.read_pos;
 
 
+  if (pkt_len == packet_error)
 
+    return (int)packet_error;
 
   /* was it a request to change plugins ? */
 
   if (pkt_len && **buf == 254)
 
     return (int)packet_error; /* if yes, this plugin shan't continue */

It still fails on the same address as *buf:

==18127==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x000000605d44 bp 0x7ffcc69446b0 sp 0x7ffcc69446a8
 
READ of size 1 at 0x629000005200 thread T0
 
    #0 0x605d43 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:552:22
 
    #1 0x5759f7 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1501:7
 
    #2 0x57223b in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1186:10
 
    #3 0x53ce06 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
 
    #4 0x553620 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
 
    #5 0x53ac33 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754:16
 
    #6 0x5396c6 in main /mariadb/10.2/client/mysql.cc:1209:7
 
    #7 0x7fcb9459fb16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16)
 
    #8 0x43be49 in _start (/dev/shm/10.2/client/mysql+0x43be49)

The source line is:

  if (res > CR_OK && mysql->net.read_pos[0] != 254)

For some reason, res==0 there, so the condition res > -1 will hold.

Comment by Marko Mäkelä [ 2019-01-25 ]

I do not see this failing with newer MariaDB server versions, but I do see the heap-use-after-free when testing the newest Connector/C library against the old MariaDB 10.2 server that I mentioned earlier:

git checkout b0ef1b388bff28b93bfdf83a97723a56c869ca34
 
(cd libmariadb; git checkout v3.0.8-release)
 
echo 'RETURN()' > cmake/submodules.cmake
 
cmake -DWITH_ASAN=ON -DWITH_SAFEMALLOC=OFF …
 
ASAN_OPTIONS=abort_on_error=1 ./mtr main.connect_debug

Maybe, when run against a newer server, the handshake fails earlier, and the problematic code in Connector/C does not get executed. The problem appears to affect connections to older servers only. Here is the result from the above:

CURRENT_TEST: main.connect_debug
 
=================================================================
 
==27725==ERROR: AddressSanitizer: heap-use-after-free on address 0x629000005200 at pc 0x000000596342 bp 0x7fffb2a14380 sp 0x7fffb2a14378
 
READ of size 1 at 0x629000005200 thread T0
 
    #0 0x596341 in client_mpvio_read_packet /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:361:18
 
    #1 0x597cb0 in auth_old_password /mariadb/10.2/libmariadb/plugins/auth/old_password.c:91:19
 
    #2 0x5958f7 in run_plugin_auth /mariadb/10.2/libmariadb/plugins/auth/my_auth.c:548:8
 
    #3 0x53f485 in mthd_my_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1502:7
 
    #4 0x53daa2 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1187:10
 
    #5 0x520799 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10
 
    #6 0x519642 in sql_real_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4706:8
 
    #7 0x519642 in sql_connect(char*, char*, char*, char*, unsigned int) /mariadb/10.2/client/mysql.cc:4754
 
    #8 0x518843 in main /mariadb/10.2/client/mysql.cc:1209:7
 
    #9 0x7f46d837409a in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2409a)
 
    #10 0x43cdb9 in _start (/dev/shm/10.2/client/mysql+0x43cdb9)
 
 
 
0x629000005200 is located 0 bytes inside of 16384-byte region [0x629000005200,0x629000009200)
 
freed by thread T0 here:
 
    #0 0x4e4882 in free (/dev/shm/10.2/client/mysql+0x4e4882)
 
    #1 0x5a3672 in ma_net_end /mariadb/10.2/libmariadb/libmariadb/ma_net.c:114:3
 
 
 
previously allocated by thread T0 here:
 
    #0 0x4e4c03 in __interceptor_malloc (/dev/shm/10.2/client/mysql+0x4e4c03)
 
    #1 0x5a3253 in ma_net_init /mariadb/10.2/libmariadb/libmariadb/ma_net.c:83:28
 
    #2 0x53daa2 in mysql_real_connect /mariadb/10.2/libmariadb/libmariadb/mariadb_lib.c:1187:10
 
    #3 0x520799 in do_connect(st_mysql*, char const*, char const*, char const*, char const*, unsigned long) /mariadb/10.2/client/mysql.cc:1391:10

Comment by Kentoku Shiba (Inactive) [ 2019-07-02 ]

I assume this can be fixed like this. These error handling looks logically required.

diff --git a/plugins/auth/my_auth.c b/plugins/auth/my_auth.c
 
index 7c84c78..0a455dc 100644
 
--- a/plugins/auth/my_auth.c
 
+++ b/plugins/auth/my_auth.c
 
@@ -354,6 +354,9 @@ static int client_mpvio_read_packet(struct st_plugin_vio *mp
 
 
 
   /* otherwise read the data */
 
   pkt_len= ma_net_safe_read(mysql);
 
+  if (pkt_len == packet_error)
 
+    return (int)packet_error;
 
+
 
   mpvio->last_read_packet_len= pkt_len;
 
   *buf= mysql->net.read_pos;
 
 
 
@@ -547,7 +550,8 @@ int run_plugin_auth(MYSQL *mysql, char *data, uint data_len,
 
 
 
   res= auth_plugin->authenticate_user((struct st_plugin_vio *)&mpvio, mysql);
 
 
 
-  if (res > CR_OK && mysql->net.read_pos[0] != 254)
 
+  if ((res == CR_ERROR && !mysql->net.buff) ||
 
+    (res > CR_OK && mysql->net.read_pos[0] != 254))
 
   {
 
     /*
 
       the plugin returned an error. write it down in mysql,

Comment by Georg Richter [ 2021-05-07 ]

this was already fixed by rev. 0f4891359a2f336698b4da1d2442a2a5e811435d

Generated at Thu Feb 08 03:04:39 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.