Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27955

main.func_json_notembedded test fails on out-of-memory on MariaDB 10.6.7

Details

    Description

      I notice that builds of MariaDB 10.6.7 in Debian was failing due to multiple reasons. One failure not reported before was main.func_json_notembedded on arch mipsel/s390x/powerpc:

      main.func_json_notembedded               w3 [ fail ]
              Test ended at 2022-02-25 14:49:46
       
      CURRENT_TEST: main.func_json_notembedded
      mysqltest: At line 12: query 'set @obj=concat_ws('','{', repeat('"a":"b",', 125000000/2), '"c":"d"}')' failed: <Unknown> (5): Out of memory (Needed 500000016 bytes)
       
      The result from queries just before the failure was:
      set global max_allowed_packet=1073741824;
      connect u,localhost,root;
      #
      # MDEV-24909 JSON functions don't respect KILL QUERY / max_statement_time limit
      #
      set group_concat_max_len= 4294967295;
      set @obj=concat_ws('','{', repeat('"a":"b",', 125000000/2), '"c":"d"}');
      

      Full logs at

      Attachments

        Issue Links

          Activity

            danblack Daniel Black added a comment -

            8 bytes repeated 62.5 million times seems to need more than 500M. So its not an overallocation, its a normal lack of resources.

            I had noticed it failing on bb on s390x, and its not marked as a big test, so mark it as such.

            Thanks for identifying the OOM cause.

            danblack Daniel Black added a comment - 8 bytes repeated 62.5 million times seems to need more than 500M. So its not an overallocation, its a normal lack of resources. I had noticed it failing on bb on s390x, and its not marked as a big test, so mark it as such. Thanks for identifying the OOM cause.

            The idea with the test is to check if json_array_append(@arr, '$[0]', 1) can be interrupted.
            Making the strings to small will defeat the purpose of the test, which is to ensure that commands that takes more than a second are interrupted
            I did some checking on my desktop (new,reasonable powerful cpu) and one can on that one decrease the repeats down to 1250 and still get
            the statement interrupted.

            That said, I am fine with the suggested patch, but better to mark the test 'big' too, to ensure it is not run on machines with little resources.

            monty Michael Widenius added a comment - The idea with the test is to check if json_array_append(@arr, '$ [0] ', 1) can be interrupted. Making the strings to small will defeat the purpose of the test, which is to ensure that commands that takes more than a second are interrupted I did some checking on my desktop (new,reasonable powerful cpu) and one can on that one decrease the repeats down to 1250 and still get the statement interrupted. That said, I am fine with the suggested patch, but better to mark the test 'big' too, to ensure it is not run on machines with little resources.

            Ok to push

            monty Michael Widenius added a comment - Ok to push

            main.func_json_notembedded still failed in 10.6.11 and 10.11.1 in Debian, see e.g. https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.6&arch=ppc64&ver=1%3A10.6.11-2&stamp=1672785257&raw=0

            However the test does not run out of memory now but crashes:

            mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/<<PKGBUILDDIR>>/builddir/mysql-t' (111)
             
             
             - found 'core' (1/5)
            Core generated by '/<<PKGBUILDDIR>>/builddir/sql/mariadbd'
            Output from gdb follows. The first stack trace is from the failing thread.
            The following stack traces are from all threads (so the failing one is
            duplicated).
            --------------------------
            [New LWP 645951]
            [New LWP 645958]
            [New LWP 645971]
            [New LWP 645960]
            [New LWP 645961]
            [New LWP 645968]
            [New LWP 645959]
            [Thread debugging using libthread_db enabled]
            Using host libthread_db library "/lib/powerpc64-linux-gnu/libthread_db.so.1".
            Core was generated by `/<<PKGBUILDDIR>>/builddir/sql/mariadbd --default'.
            Program terminated with signal SIGABRT, Aborted.
            #0  0x00007fff8e95da5c in ?? () from /lib/powerpc64-linux-gnu/libc.so.6
            [Current thread is 1 (Thread 0x7fff8fa7b900 (LWP 645951))]
            #0  0x00007fff8e95da5c in ?? () from /lib/powerpc64-linux-gnu/libc.so.6
            #1  0x0000000134204998 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:424
            #2  0x0000000133b8c1e8 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:355
            #3  <signal handler called>
            #4  0x00007fff8e9edaf0 in .__poll () from /lib/powerpc64-linux-gnu/libc.so.6
            #5  0x00000001337bc5dc in poll (__timeout=-1, __nfds=<optimized out>, __fds=0x1000725e508) at /usr/include/powerpc64-linux-gnu/bits/poll2.h:39
            #6  handle_connections_sockets () at ./sql/mysqld.cc:6182
            #7  0x00000001337be01c in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:5887
            #8  0x0000000133765438 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:34
            

            otto Otto Kekäläinen added a comment - main.func_json_notembedded still failed in 10.6.11 and 10.11.1 in Debian, see e.g. https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.6&arch=ppc64&ver=1%3A10.6.11-2&stamp=1672785257&raw=0 However the test does not run out of memory now but crashes: mysqltest: Could not open connection 'default' after 500 attempts: 2002 Can't connect to local server through socket '/<<PKGBUILDDIR>>/builddir/mysql-t' (111)     - found 'core' (1/5) Core generated by '/<<PKGBUILDDIR>>/builddir/sql/mariadbd' Output from gdb follows. The first stack trace is from the failing thread. The following stack traces are from all threads (so the failing one is duplicated). -------------------------- [New LWP 645951] [New LWP 645958] [New LWP 645971] [New LWP 645960] [New LWP 645961] [New LWP 645968] [New LWP 645959] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/powerpc64-linux-gnu/libthread_db.so.1". Core was generated by `/<<PKGBUILDDIR>>/builddir/sql/mariadbd --default'. Program terminated with signal SIGABRT, Aborted. #0 0x00007fff8e95da5c in ?? () from /lib/powerpc64-linux-gnu/libc.so.6 [Current thread is 1 (Thread 0x7fff8fa7b900 (LWP 645951))] #0 0x00007fff8e95da5c in ?? () from /lib/powerpc64-linux-gnu/libc.so.6 #1 0x0000000134204998 in my_write_core (sig=<optimized out>) at ./mysys/stacktrace.c:424 #2 0x0000000133b8c1e8 in handle_fatal_signal (sig=<optimized out>) at ./sql/signal_handler.cc:355 #3 <signal handler called> #4 0x00007fff8e9edaf0 in .__poll () from /lib/powerpc64-linux-gnu/libc.so.6 #5 0x00000001337bc5dc in poll (__timeout=-1, __nfds=<optimized out>, __fds=0x1000725e508) at /usr/include/powerpc64-linux-gnu/bits/poll2.h:39 #6 handle_connections_sockets () at ./sql/mysqld.cc:6182 #7 0x00000001337be01c in mysqld_main (argc=<optimized out>, argv=<optimized out>) at ./sql/mysqld.cc:5887 #8 0x0000000133765438 in main (argc=<optimized out>, argv=<optimized out>) at ./sql/main.cc:34
            danblack Daniel Black added a comment -

            Abort is from MTR as its took too long. There was some guidance somewhere on improving the test.

            danblack Daniel Black added a comment - Abort is from MTR as its took too long. There was some guidance somewhere on improving the test.

            People

              danblack Daniel Black
              otto Otto Kekäläinen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.