Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22035

Memory leak in main.mysqltest

    XMLWordPrintable

    Details

      Description

      I debugged the main.mysqltest failure that reliably occurs on kvm-asan. It looks like a bug in GCC 5.4.0’s ASAN implementation to me:

      10.5 6fb59d525b7047c82e350d2b721bffc50431eb12

      mysqltest: At line 1: Error parsing replace_regex "a"
       
      Program received signal SIGABRT, Aborted.
      0x00007ffff534a428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) bt
      #0  0x00007ffff534a428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
      #1  0x00007ffff534c02a in abort () from /lib/x86_64-linux-gnu/libc.so.6
      #2  0x00007ffff6f17dc9 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #3  0x00007ffff6f0a769 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #4  0x00007ffff6f0f5d2 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #5  0x00007ffff6f17558 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #6  0x00007ffff6f18a38 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #7  0x00007ffff6f18bbe in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #8  0x00007ffff6f225a6 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #9  0x00007ffff534f36a in __cxa_finalize ()
         from /lib/x86_64-linux-gnu/libc.so.6
      #10 0x00007ffff6e875a3 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #11 0x00007ffffffbdae0 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
      

      Even if I set a breakpoint on __cxa_finalize () and rerun, I will not get more information. There is only that one thread:

      Starting program: /home/buildbot/server/build/client/mysqltest --defaults-file=/home/buildbot/server/build/mysql-test/var/my.cnf --silent --tmpdir=/home/buildbot/server/build/mysql-test/var/tmp --character-sets-dir=/home/buildbot/server/sql/share/charsets --logdir=/home/buildbot/server/build/mysql-test/var/log --database=test --timer-file=/home/buildbot/server/build/mysql-test/var/log/timer < foo
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      mysqltest: At line 1: Error parsing replace_regex "a"
       
      Breakpoint 1, 0x00007ffff534f2d0 in __cxa_finalize ()
         from /lib/x86_64-linux-gnu/libc.so.6
      (gdb) thr a a bt
       
      Thread 1 (Thread 0x7ffff7fe0780 (LWP 21098)):
      #0  0x00007ffff534f2d0 in __cxa_finalize ()
         from /lib/x86_64-linux-gnu/libc.so.6
      #1  0x00007ffff6e875a3 in ?? () from /usr/lib/x86_64-linux-gnu/libasan.so.2
      #2  0x00007ffffffbdae0 in ?? ()
      Backtrace stopped: previous frame identical to this frame (corrupt stack?)
      

      Test setup:

      cd build/mysql-test
      echo 'echo $MYSQL_TEST;' > ../../mysql-test/main/m.test
      echo --replace_regex a > foo
      ./mtr --manual-gdb main.m
      gdb -cd /home/buildbot/server/mysql-test -x /home/buildbot/server/build/mysql-test/var/tmp/gdbinit.mysqld.1 /home/buildbot/server/build/sql/mysqld
      

      Then, in gdb, type

      run
      # (hit ctrl-z to suspend the server process)
      detach
      quit
      

      and finally start the mysqltest client in another gdb:

      run /home/buildbot/server/build/client/mysqltest --defaults-file=/home/buildbot/server/build/mysql-test/var/my.cnf --silent --tmpdir=/home/buildbot/server/build/mysql-test/var/tmp --character-sets-dir=/home/buildbot/server/sql/share/charsets --logdir=/home/buildbot/server/build/mysql-test/var/log --database=test --timer-file=/home/buildbot/server/build/mysql-test/var/log/timer < foo
      

      The arguments are from the $MYSQL_TEST, except for the < foo redirection.

      Note: If I try to invoke ./mtr main.n on the following n.test file directly, then there is no SIGABRT reported:

      --replace_regex a
      

      It will only report a failure like this:

      main.n                                   [ fail ]
              Test ended at 2020-03-21 08:19:40
       
      CURRENT_TEST: main.n
      mysqltest: At line 1: Error parsing replace_regex "a"
      

      There is nothing in any log files either:

      grep -ri sanitizer var

      var/tmp/check-mysqld_1.result:HAVE_SANITIZER	ASAN
      var/tmp/check-mysqld_1.log:HAVE_SANITIZER	ASAN
      var/log/main.n/tmp/check-mysqld_1.result:HAVE_SANITIZER	ASAN
      var/log/main.n/tmp/check-mysqld_1.log:HAVE_SANITIZER	ASAN
      

      On my local system, typically using clang version 7, 8, 9, or 10, I have experienced a hang of main.mysqltest since June 2019, on 10.5 and possibly also on 10.4. I finally decided to debug it, and it looks like it could be the same problem. Only, for some reason, the mysqltest process is hanging instead of crashing:

      10.5 9a7d284e202d8862a7b34aaf2253d4627177a90e

      gdb /dev/shm/10.5a/client/mysqltest
      GNU gdb (Debian 9.1-2) 9.1
      Copyright (C) 2020 Free Software Foundation, Inc.
      License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
      Type "show copying" and "show warranty" for details.
      This GDB was configured as "x86_64-linux-gnu".
      Type "show configuration" for configuration details.
      For bug reporting instructions, please see:
      <http://www.gnu.org/software/gdb/bugs/>.
      Find the GDB manual and other documentation resources online at:
          <http://www.gnu.org/software/gdb/documentation/>.
       
      For help, type "help".
      Type "apropos word" to search for commands related to "word"...
      Reading symbols from /dev/shm/10.5a/client/mysqltest...
      (gdb) set env ASAN_OPTIONS=abort_on_error=1
      (gdb) run --defaults-file=/dev/shm/10.5a/mysql-test/var/my.cnf --silent --tmpdir=/dev/shm/10.5a/mysql-test/var/tmp --character-sets-dir=/mariadb/10.5m/sql/share/charsets --logdir=/dev/shm/10.5a/mysql-test/var/log --database=test --timer-file=/dev/shm/10.5a/mysql-test/var/log/timer < /mariadb/10.5m/mysql-test/main/n.test 
      Starting program: /dev/shm/10.5a/client/mysqltest --defaults-file=/dev/shm/10.5a/mysql-test/var/my.cnf --silent --tmpdir=/dev/shm/10.5a/mysql-test/var/tmp --character-sets-dir=/mariadb/10.5m/sql/share/charsets --logdir=/dev/shm/10.5a/mysql-test/var/log --database=test --timer-file=/dev/shm/10.5a/mysql-test/var/log/timer < /mariadb/10.5m/mysql-test/main/n.test
      [Thread debugging using libthread_db enabled]
      Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
      mysqltest: At line 1: Error parsing replace_regex "a"
       
      Program received signal SIGABRT, Aborted.
      __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      50	../sysdeps/unix/sysv/linux/raise.c: Tiedostoa tai hakemistoa ei ole.
      (gdb) bt
      #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
      #1  0x00007ffff770f55b in __GI_abort () at abort.c:79
      #2  0x00000000004d1f37 in __sanitizer::Abort() ()
          at /mariadb/10.5m/client/mysqltest.cc:829
      #3  0x00000000004d0961 in __sanitizer::Die() ()
          at /mariadb/10.5m/client/mysqltest.cc:829
      #4  0x00000000004db3d4 in __lsan::CheckForLeaks() ()
          at /mariadb/10.5m/client/mysqltest.cc:829
      #5  0x00000000004db0a2 in __lsan::DoLeakCheck() ()
          at /mariadb/10.5m/client/mysqltest.cc:829
      #6  0x00007ffff7727e27 in __run_exit_handlers (status=1, 
          listp=0x7ffff78a6718 <__exit_funcs>, 
          run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
          at exit.c:108
      #7  0x00007ffff7727fda in __GI_exit (status=<optimized out>) at exit.c:139
      #8  0x00000000004ed2dd in cleanup_and_exit (exit_code=1)
          at /mariadb/10.5m/client/mysqltest.cc:1567
      #9  0x00000000004ed0c7 in really_die (msg=<optimized out>)
          at /mariadb/10.5m/client/mysqltest.cc:1276
      #10 0x00000000004eae92 in die (fmt=<optimized out>)
          at /mariadb/10.5m/client/mysqltest.cc:1615
      #11 0x000000000051f88a in append_replace_regex (expr=<optimized out>, 
          expr_end=<optimized out>, res=<optimized out>, buf_p=<optimized out>)
          at /mariadb/10.5m/client/mysqltest.cc:10269
      #12 0x000000000051a268 in init_replace_regex (expr=0x604000000e36 "a")
          at /mariadb/10.5m/client/mysqltest.cc:10188
      #13 do_get_replace_regex (command=0x612000000058)
          at /mariadb/10.5m/client/mysqltest.cc:10349
      #14 0x0000000000517cca in main (argc=<optimized out>, argv=0x7fffffffdfe0)
          at /mariadb/10.5m/client/mysqltest.cc:9633
      

      This particular one was built cmake -DWITH_PCRE2=bundled. I have seen (but not debugged) the hang also with the system-provided library.

      Workaround: Set ASAN_OPTIONS=detect_leaks=0 for the mysqltest invocation, and it will not crash or hang.
      I can run the entire test without problems, using

      ASAN_OPTIONS=detect_leaks=0 ./mtr main.mysqltest
      

        Attachments

          Activity

            People

            Assignee:
            marko Marko Mäkelä
            Reporter:
            marko Marko Mäkelä
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: