Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-5147

crash with "is an invalid pointer" in mariadb since 5.5.32 (but not in any oracle version)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 10.0.4, 5.3.12, 5.5.33a
    • Fix Version/s: 10.0.5, 5.5.34, 5.3.13
    • Component/s: None
    • Labels:
      None
    • Environment:
      centos x64

      Description

      I've recently upgraded from mariadb 5.5.29 to 5.5.33a on many servers. There are no problem except one server (out of ~15) that keeps crashing (every 5-20 minutes). I've tested on oracle mysql 5.5.34 (with same server) and it does not crash with it. It is server provided to hosting users with about 75GB od data on innodb and myisam. Also server stopps crashing when I switch back to mariadb 5.5.29. I use precompiled binaries of mariadb but I also compiled 5.5.33a on my system and same issue was occuring there.

      131016 18:23:01 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
       
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
       
      Server version: 5.5.33a-MariaDB
      key_buffer_size=251658240
      read_buffer_size=524288
      max_used_connections=9
      max_threads=2050
      thread_count=10
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5530353 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x0x7f3cb39cb000
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x411ec090 thread_stack 0x80000
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(my_print_stacktrace+0x2e)[0xb0bc9e]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(handle_fatal_signal+0x422)[0x6eea32]
      /lib64/libpthread.so.0[0x7f3d358c9b10]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZNK10Item_field11result_typeEv+0xb)[0x71a0fb]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld[0x5c0d8f]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld[0x5c1aa7]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld[0x5c1e1c]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld[0x5c2356]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld[0x5c34f1]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN4JOIN8optimizeEv+0x456)[0x5d9596]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z22mysql_derived_optimizeP3THDP3LEXP10TABLE_LIST+0xfd)[0x56fc1d]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z27mysql_handle_single_derivedP3LEXP10TABLE_LISTj+0xb8)[0x56fd58]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN10TABLE_LIST14handle_derivedEP3LEXj+0x7a)[0x62663a]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13st_select_lex14handle_derivedEP3LEXj+0x40)[0x581ac0]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN10TABLE_LIST14handle_derivedEP3LEXj+0x5a)[0x62661a]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13st_select_lex14handle_derivedEP3LEXj+0x40)[0x581ac0]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN10TABLE_LIST14handle_derivedEP3LEXj+0x5a)[0x62661a]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13st_select_lex14handle_derivedEP3LEXj+0x40)[0x581ac0]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN4JOIN8optimizeEv+0x134)[0x5d9274]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex
      _unitP13st_select_lex+0xd1)[0x5e48b1]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_sele
      ct_lexPP12multi_update+0x1cf)[0x61cd3f]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0x5a33)[0x596bd3]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1e)[0x80caee]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x210)[0x80ce00]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x6a9)[0x8141b9]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN7sp_head7executeEP3THDb+0x51f)[0x810dcf]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x675)[0x812635]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3b43)[0x594ce3]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x1e)[0x80caee]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x210)[0x80ce00]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x6a9)[0x8141b9]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN7sp_head7executeEP3THDb+0x51f)[0x810dcf]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x675)[0x812635]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0x3b43)[0x594ce3]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1b0)[0x598490]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x172e)[0x599bce]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z10do_commandP3THD+0xdb)[0x59a13b]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x144)[0x6558d4]
      /var/mariadb-5.5.33a-linux-x86_64/bin/mysqld(handle_one_connection+0x4c)[0x655a0c]
      /lib64/libpthread.so.0[0x7f3d358c173d]
      /lib64/libc.so.6(clone+0x6d)[0x7f3d34a6f4bd]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7f3cb5790490): is an invalid pointer
      Connection ID (thread ID): 2250
      Status: NOT_KILLED
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersect
      ion=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,mat
      erialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cos
      t_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cach
      e_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off
       
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.

      Now i compared that "Connection ID (thread ID): 2250" with general query log and server crashes always after this sequence of queries:

      		 2250 Query	SET NAMES utf8
      		 2250 Init DB	user_user
      		 2250 Query	CALL get_round_table_09(125,1)
      		 2250 Query	DROP TEMPORARY TABLE IF EXISTS temp_round_table, temp_positions
      		 2250 Query	CREATE TEMPORARY TABLE temp_round_table (INDEX(league_id), INDEX(team_id)) AS
              SELECT l.league_id, 0 AS position, t.team_id AS team_id, t.name AS team_name, ntz(played) AS played, ntz(won) AS won, ntz(drew) AS drew, ntz(lost) AS lost, ntz(scored) AS scored, ntz(lossed) AS lossed, ntz(points) AS points, 0 AS small_points, 0 AS small_ballance, 0 AS small_scored, history, if(isnull(te.team_id),0,1) AS exclude
              FROM leagues l
                JOIN league_squads ls ON l.league_id=ls.league_id
                JOIN teams t ON ls.team_id=t.team_id
                LEFT JOIN
                  (SELECT rt.team_id, count(*) AS played, sum(if(rt.general_result='w',1,0)) AS won, sum(if(rt.general_result='r',1,0)) AS drew, sum(if(rt.general_result='p',1,0)) AS lost, sum(rt.score) AS scored, sum(rop.score) AS lossed, sum(rt.points) AS points, group_concat(rt.general_result ORDER BY m.date, m.time SEPARATOR '') AS history
                   FROM rounds r
                     JOIN matches m ON m.round_id=r.round_id AND m.finished AND NOT m.cancelled
                     JOIN results rt ON m.match_id=rt.match_id
                     JOIN results rop ON m.match_id=rop.match_id AND rop.team_id<>rt.team_id
                     JOIN leagues lt ON lt.round_id= NAME_CONST('var_round_id',125)
                     JOIN league_squads lst ON lst.league_id=lt.league_id AND lst.team_id=rt.team_id
                     JOIN leagues lop ON lop.round_id= NAME_CONST('var_round_id',125) AND lt.league_id=lop.league_id
                   WHERE r.round_id= NAME_CONST('var_round_id',125)
                   GROUP BY rt.team_id)
                m1 ON t.team_id=m1.team_id
                LEFT JOIN team_excludies te ON t.team_id=te.team_id AND te.competition_id= NAME_CONST('var_competition_id',87)
              WHERE l.round_id= NAME_CONST('var_round_id',125)
      		 2250 Query	DROP TEMPORARY TABLE IF EXISTS temp_rx
      		 2250 Query	CREATE TEMPORARY TABLE temp_rx AS
          SELECT league_id, team_id, points
          FROM temp_round_table
      		 2250 Query	UPDATE temp_round_table trt,
               (SELECT r.team_id, sum(r.points) AS points, sum(r.score)-sum(rl.score) AS ballance, sum(r.score) AS scored
                FROM
                   (SELECT mb.match_id
                    FROM
                     (SELECT m.match_id
                        FROM
                           (SELECT team_id
                            FROM temp_rx
                            WHERE league_id= NAME_CONST('fetched_league_id',138) AND points= NAME_CONST('fetched_points',3)) tx
                          JOIN matches m ON m.round_id= NAME_CONST('var_round_id',125) AND m.finished AND NOT m.cancelled
                          JOIN results r ON r.match_id=m.match_id AND r.team_id=tx.team_id
                      WHERE true) mb
                    GROUP BY mb.match_id
                    HAVING count(*)>1) mx
                  JOIN results r ON r.match_id=mx.match_id
                  JOIN results rl ON rl.match_id=mx.match_id AND rl.team_id<>r.team_id
                GROUP BY r.team_id) stb
              SET trt.small_points=stb.points, trt.small_scored=stb.scored, trt.small_ballance=stb.ballance
              WHERE trt.team_id=stb.team_id
      		 2250 Query	UPDATE temp_round_table trt,
               (SELECT r.team_id, sum(r.points) AS points, sum(r.score)-sum(rl.score) AS ballance, sum(r.score) AS scored
                FROM
                   (SELECT mb.match_id
                    FROM
                     (SELECT m.match_id
                        FROM
                           (SELECT team_id
                            FROM temp_rx
                            WHERE league_id= NAME_CONST('fetched_league_id',138) AND points= NAME_CONST('fetched_points',4)) tx
                          JOIN matches m ON m.round_id= NAME_CONST('var_round_id',125) AND m.finished AND NOT m.cancelled
                          JOIN results r ON r.match_id=m.match_id AND r.team_id=tx.team_id
                      WHERE true) mb
                    GROUP BY mb.match_id
                    HAVING count(*)>1) mx
                  JOIN results r ON r.match_id=mx.match_id
                  JOIN results rl ON rl.match_id=mx.match_id AND rl.team_id<>r.team_id
                GROUP BY r.team_id) stb
              SET trt.small_points=stb.points, trt.small_scored=stb.scored, trt.small_ballance=stb.ballance
              WHERE trt.team_id=stb.team_id

        Attachments

          Activity

            People

            Assignee:
            sanja Oleksandr Byelkin
            Reporter:
            naox naox
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: