Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-714

LP:1020645 - crash (sig 11) with union query

Details

    Description

      Hello and thanks for mariadb-5.3.7-Linux-x86_64.tar.gz,

      Running on 2.6.18-274.7.1.el5 x86_64 x86_64 x86_64 GNU/Linux RedHat EL.

      We recently upgraded some mysqld instances in a pool from MySQL 5.1.X to mariadb 5.3.7, we have had two crashes (on different instances on different hosts) with the same backtrace and very similar queries, the pool is a production pool so the priority has been restoration of service and a rollback, the query from the first backtrace did not cause a crash when run in isolation on our development and staging instances (I have not tested the 2nd yet), the development and staging instances do not have identical configurations, data or queries running, but are the same version (MariaDB 5.3.7) on the same OS on the same architecture. Oh, also both mariadb instances ran for several days days in production before crashing.

      Here are the contents of the error log for the first crash (the backtrace is the same for the second crash and the query is very similar). I have removed the actual query reported and have included a representation of the query, I can send the actual query and table definitions privately.

      I am wondering if (and hoping that) the backtrace looks familiar.

      120627  5:20:14 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 5.3.7-MariaDB-log
      key_buffer_size=33554432
      read_buffer_size=2097152
      max_used_connections=319
      max_threads=3001
      thread_count=29
      connection_count=29
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18510029 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x2ab7fc3499b0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x512fc0f8 thread_stack 0x48000
      ./bin/mysqld(my_print_stacktrace+0x2e) [0xa2b62e]
      ./bin/mysqld(handle_fatal_signal+0x3f9) [0x7627f9]
      /lib64/libpthread.so.0 [0x341f00eb70]
      ./bin/mysqld [0x6be068]
      ./bin/mysqld(JOIN::exec()+0x852) [0x6cfc52]
      ./bin/mysqld(st_select_lex_unit::exec()+0x184) [0x7ab904]
      ./bin/mysqld(mysql_union(THD*, st_lex*, select_result*, st_select_lex_unit*, unsigned long)+0x2e) [0x7add0e]
      ./bin/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x82) [0x6d2632]
      ./bin/mysqld [0x647e7e]
      ./bin/mysqld(mysql_execute_command(THD*)+0x3a58) [0x64da78]
      ./bin/mysqld(mysql_parse(THD*, char*, unsigned int, char const**)+0x299) [0x650859]
      ./bin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xa9b) [0x65174b]
      ./bin/mysqld(do_command(THD*)+0x101) [0x6522a1]
      ./bin/mysqld(handle_one_connection+0xfd) [0x6436ad]
      /lib64/libpthread.so.0 [0x341f00673d]
      /lib64/libc.so.6(clone+0x6d) [0x341e4d44bd]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x2ab7deb1abf8): 
      (select many fields from a few tables with joins and inner joins group by sort by limit 5000)
      UNION ALL 
      (select many fields from a few tables with joins and inner joins group by sort by limit 5000) 
      order by limit 5000
      Connection ID (thread ID): 30483258
      Status: KILL_CONNECTION
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on

      Thank you.

      Attachments

        Issue Links

          Activity

            elenst, I've tried your rqg grammar for both 5.3 and 5.5, debug and optimized builds. No crashes.

            serg Sergei Golubchik added a comment - elenst , I've tried your rqg grammar for both 5.3 and 5.5, debug and optimized builds. No crashes.

            Still crashes for me (tried current 5.5 tree).
            I'll set up the test on perro.

            elenst Elena Stepanova added a comment - Still crashes for me (tried current 5.5 tree). I'll set up the test on perro.

            I've set it up on perro, for 5.3 and 5.5, current trees, debug builds.

            • log in as usual;
              cd mdev714
              . ./run53
              or
              . ./run55

            Logs, including coredumps, are in ./vardir53 and ./vardir55, correspondingly.

            elenst Elena Stepanova added a comment - I've set it up on perro, for 5.3 and 5.5, current trees, debug builds. log in as usual; cd mdev714 . ./run53 or . ./run55 Logs, including coredumps, are in ./vardir53 and ./vardir55, correspondingly.

            Thank you.
            We are still running 5.3.12 and waiting for 10 to be GA before starting on a roll out.
            With this fix and other fixes made in 5.3.13 since 5.3.12 was released, I am keen to go to 5.3.13 before starting on 10.
            I was wondering if any of the fixes in 5.3.13 would result in a 5.3.13 release before April this year, or if I should go with the latest revision in the 5.3 branch that is successful in the buildbot.
            Thanks again.

            thatsafunnyname Peter (Stig) Edwards added a comment - Thank you. We are still running 5.3.12 and waiting for 10 to be GA before starting on a roll out. With this fix and other fixes made in 5.3.13 since 5.3.12 was released, I am keen to go to 5.3.13 before starting on 10. I was wondering if any of the fixes in 5.3.13 would result in a 5.3.13 release before April this year, or if I should go with the latest revision in the 5.3 branch that is successful in the buildbot. Thanks again.

            If you absolutely want to stay with 5.3, I'd suggest to grab the latest revision from buildbot.
            But really, you should consider upgrading to 5.5. It may be problematic to upgrade directly to 10.0.

            See, for example, MDEV-4860 — you cannot upgrade from InnoDB-5.1 to InnoDB-5.6 without upgrading to InnoDB-5.5 first. And MariaDB-5.3 uses InnoDB-5.1, MariaDB-10.0 uses InnoDB-5.6.

            serg Sergei Golubchik added a comment - If you absolutely want to stay with 5.3, I'd suggest to grab the latest revision from buildbot. But really, you should consider upgrading to 5.5. It may be problematic to upgrade directly to 10.0. See, for example, MDEV-4860 — you cannot upgrade from InnoDB-5.1 to InnoDB-5.6 without upgrading to InnoDB-5.5 first. And MariaDB-5.3 uses InnoDB-5.1, MariaDB-10.0 uses InnoDB-5.6.

            People

              serg Sergei Golubchik
              peterstigedwards Peter (Stig) Edwards (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.