Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35341

Crash with dump file in best_access_path()

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.11.10
    • N/A
    • None
    • Windows 11 Pro x64 / 23H2

    Description

      Crash as log reported.
      The minidump uploaded in the ftp/private.

      Server version: 10.11.10-MariaDB-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
      key_buffer_size=16777216
      read_buffer_size=67108864
      max_used_connections=66
      max_threads=103
      thread_count=48
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 13519083 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x1422d000f18
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      server.dll!best_access_path()[sql_select.cc:9031]
      server.dll!get_costs_for_tables()[sql_select.cc:10667]
      server.dll!best_extension_by_limited_search()[sql_select.cc:11250]
      server.dll!greedy_search()[sql_select.cc:9914]
      server.dll!choose_plan()[sql_select.cc:9467]
      server.dll!st_join_table::choose_best_splitting()[opt_split.cc:1083]
      server.dll!best_access_path()[sql_select.cc:8249]
      server.dll!get_costs_for_tables()[sql_select.cc:10667]
      server.dll!best_extension_by_limited_search()[sql_select.cc:11250]
      server.dll!best_extension_by_limited_search()[sql_select.cc:11449]
      server.dll!greedy_search()[sql_select.cc:9914]
      server.dll!choose_plan()[sql_select.cc:9467]
      server.dll!make_join_statistics()[sql_select.cc:6175]
      server.dll!JOIN::optimize_inner()[sql_select.cc:2641]
      server.dll!JOIN::optimize()[sql_select.cc:1956]
      server.dll!mysql_derived_optimize()[sql_derived.cc:1045]
      server.dll!mysql_handle_single_derived()[sql_derived.cc:200]
      server.dll!TABLE_LIST::handle_derived()[table.cc:9686]
      server.dll!st_select_lex::handle_derived()[sql_lex.cc:5066]
      server.dll!TABLE_LIST::handle_derived()[table.cc:9683]
      server.dll!st_select_lex::handle_derived()[sql_lex.cc:5066]
      server.dll!JOIN::optimize_inner()[sql_select.cc:2464]
      server.dll!mysql_select()[sql_select.cc:5208]
      server.dll!handle_select()[sql_select.cc:600]
      server.dll!execute_sqlcom_select()[sql_parse.cc:6412]
      server.dll!mysql_execute_command()[sql_parse.cc:3999]
      server.dll!sp_instr_stmt::exec_core()[sp_head.cc:3926]
      server.dll!sp_lex_keeper::reset_lex_and_exec_core()[sp_head.cc:3646]
      server.dll!sp_instr_stmt::execute()[sp_head.cc:3827]
      server.dll!sp_head::execute()[sp_head.cc:1463]
      server.dll!sp_head::execute_procedure()[sp_head.cc:2479]
      server.dll!do_execute_sp()[sql_parse.cc:3079]
      server.dll!Sql_cmd_call::execute()[sql_parse.cc:3321]
      server.dll!mysql_execute_command()[sql_parse.cc:6165]
      server.dll!sp_instr_stmt::exec_core()[sp_head.cc:3926]
      server.dll!sp_lex_keeper::reset_lex_and_exec_core()[sp_head.cc:3646]
      server.dll!sp_instr_stmt::execute()[sp_head.cc:3827]
      server.dll!sp_head::execute()[sp_head.cc:1463]
      server.dll!sp_head::execute_trigger()[sp_head.cc:1973]
      server.dll!Table_triggers_list::process_triggers()[sql_trigger.cc:2486]
      server.dll!write_record()[sql_insert.cc:2147]
      server.dll!mysql_insert()[sql_insert.cc:1174]
      server.dll!mysql_execute_command()[sql_parse.cc:4619]
      server.dll!sp_instr_stmt::exec_core()[sp_head.cc:3926]
      server.dll!sp_lex_keeper::reset_lex_and_exec_core()[sp_head.cc:3646]
      server.dll!sp_instr_stmt::execute()[sp_head.cc:3827]
      server.dll!sp_head::execute()[sp_head.cc:1463]
      server.dll!sp_head::execute_procedure()[sp_head.cc:2479]
      server.dll!do_execute_sp()[sql_parse.cc:3079]
      server.dll!Sql_cmd_call::execute()[sql_parse.cc:3321]
      server.dll!mysql_execute_command()[sql_parse.cc:6165]
      server.dll!mysql_parse()[sql_parse.cc:8178]
      server.dll!dispatch_command()[sql_parse.cc:1908]
      server.dll!do_command()[sql_parse.cc:1419]
      server.dll!do_handle_one_connection()[sql_connect.cc:1407]
      server.dll!handle_one_connection()[sql_connect.cc:1325]
      server.dll!pfs_spawn_thread()[pfs.cc:2204]
      server.dll!pthread_start()[my_winthread.c:62]
      ucrtbase.dll!_recalloc()
      KERNEL32.DLL!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      

      Attachments

        Activity

          Could you show what stored procedure caused this, affected tables, and the trigger?

          serg Sergei Golubchik added a comment - Could you show what stored procedure caused this, affected tables, and the trigger?
          npdmailing@gmail.com Mitchell Lee added a comment -

          @Sergei Golubchik
          Are you asking a specific name of object? or...kind of overall detail?

          npdmailing@gmail.com Mitchell Lee added a comment - @Sergei Golubchik Are you asking a specific name of object? or...kind of overall detail?

          A stack track shows that it's a CALL statement of some stored procedure, which does INSERT statement, which invokes a trigger, which runs what appears to be a SELECT, which crashes in the optimizer.

          If you happen to know what stored procedure it was and what table and what trigger — that's what I asked for. If you don't know, and there is nothing the error log that could help to find it out — then we'll think of something else to ask, I certainly did not mean to ask for all your stored routines and tables and triggers.

          serg Sergei Golubchik added a comment - A stack track shows that it's a CALL statement of some stored procedure, which does INSERT statement, which invokes a trigger, which runs what appears to be a SELECT , which crashes in the optimizer. If you happen to know what stored procedure it was and what table and what trigger — that's what I asked for. If you don't know, and there is nothing the error log that could help to find it out — then we'll think of something else to ask, I certainly did not mean to ask for all your stored routines and tables and triggers.
          npdmailing@gmail.com Mitchell Lee added a comment - - edited

          Before I try to let you know the sequance of the calling object name,
          I figure it out a hint from your comment what we actually need to track.
          -----------
          Seems,
          Step 0. DB started. and do normal batch job. Wait exernal heavy IO intensive input.
          Step 1. An external script calling a stored procedure to insert/update data into the DB
          Step 2. A record inserted and a trigger fired to insert another tables
          Step 3. the trigger calls another mutiple stored procedure which also contains mutiple child sotred procedure
          Step 4. one of a child stored procedure in the mutiple stored procedure calling chain, use option
          "SET SESSION optimizer_switch='derived_merge=off,condition_pushdown_for_derived=off,condition_pushdown_for_subquery=off,condition_pushdown_from_having=off';" at the begining of a stored procedure.
          And never turn back to default.
          Step 5. Finish this calling chain somehow in the end.
          Step 6. Start over 'step 1' with the reused calling session.

          What I'm trying to tell you is,
          Once changed the 'session optimizer' option never turned back as default. Seems it being a cause of the crash somehow at the certain point of time after several thousand times of reused same CRUD session process. (FYI, I suspect 'xxx_pushdown_' series option making the problem rather then 'derived_merge' since its added yesterday)
          (It happend today again, that same pattern of crash. so I upload the another crashdump(as 'next day') in the ftp/private)

          Isn't it the more right answer for the issue...?
          I will try to the optimizer option back to the default when right after it uses for the time being as an workaround. Meanwhile hope you guys figure it out the solution.

          npdmailing@gmail.com Mitchell Lee added a comment - - edited Before I try to let you know the sequance of the calling object name, I figure it out a hint from your comment what we actually need to track. ----------- Seems, Step 0. DB started. and do normal batch job. Wait exernal heavy IO intensive input. Step 1. An external script calling a stored procedure to insert/update data into the DB Step 2. A record inserted and a trigger fired to insert another tables Step 3. the trigger calls another mutiple stored procedure which also contains mutiple child sotred procedure Step 4. one of a child stored procedure in the mutiple stored procedure calling chain, use option "SET SESSION optimizer_switch='derived_merge=off,condition_pushdown_for_derived=off,condition_pushdown_for_subquery=off,condition_pushdown_from_having=off';" at the begining of a stored procedure. And never turn back to default. Step 5. Finish this calling chain somehow in the end. Step 6. Start over 'step 1' with the reused calling session. What I'm trying to tell you is, Once changed the 'session optimizer' option never turned back as default. Seems it being a cause of the crash somehow at the certain point of time after several thousand times of reused same CRUD session process. (FYI, I suspect 'xxx_pushdown_' series option making the problem rather then 'derived_merge' since its added yesterday) (It happend today again, that same pattern of crash. so I upload the another crashdump(as 'next day') in the ftp/private) Isn't it the more right answer for the issue...? I will try to the optimizer option back to the default when right after it uses for the time being as an workaround. Meanwhile hope you guys figure it out the solution.

          npdmailing@gmail.com Hi! At the end of the stack trace in your error log, directly under that last line ("Some pointers may be invalid and cause the dump to abort.") in the original description, did it have the failing query? It would look something like this:

          Trying to get some variables.
          Some pointers may be invalid and cause the dump to abort.
          Query (0x52b000085238):  some... query...
          

          If there is (which is not always the case), it may help to troubleshoot the issue further.

          Roel Roel Van de Paar added a comment - npdmailing@gmail.com Hi! At the end of the stack trace in your error log, directly under that last line ("Some pointers may be invalid and cause the dump to abort.") in the original description, did it have the failing query? It would look something like this: Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x52b000085238): some... query... If there is (which is not always the case), it may help to troubleshoot the issue further.
          npdmailing@gmail.com Mitchell Lee added a comment -

          @Roel Van de Paar
          Well....the crash make the DB service daemon stop as every single crash does. so need to the DB service restart manually.
          It likely , the crash dump may generated while the DB service down.
          Means, as I assume/of course, it cause of the query failing since the DB system going crash shutdown. and no more RAM memory information exist.

          npdmailing@gmail.com Mitchell Lee added a comment - @Roel Van de Paar Well....the crash make the DB service daemon stop as every single crash does. so need to the DB service restart manually. It likely , the crash dump may generated while the DB service down. Means, as I assume/of course, it cause of the query failing since the DB system going crash shutdown. and no more RAM memory information exist.
          Roel Roel Van de Paar added a comment - - edited

          npdmailing@gmail.com Thank you for your comment. What I meant is - is the full error log (in your MariaDB client you can use

          SHOW GLOBAL VARIABLES LIKE 'log_error';
          

          to see the location of the error log) still available? If so, it may contain additional information, like the failing query, besides the stack trace as pasted in the original description. Thank you!

          Roel Roel Van de Paar added a comment - - edited npdmailing@gmail.com Thank you for your comment. What I meant is - is the full error log (in your MariaDB client you can use SHOW GLOBAL VARIABLES LIKE 'log_error' ; to see the location of the error log) still available? If so, it may contain additional information, like the failing query, besides the stack trace as pasted in the original description. Thank you!

          wlad Kindly checked the minidump for us and regrettably there is not much information within it other than

          Unhandled exception at 0x00007FF804EFF3E4 (server.dll) in MDEV-35341--mysqld.dmp: 0xC0000005: Access violation reading location 0x0000000000000040.
          

          And that the length of the query packet is 849 bytes.

          npdmailing@gmail.com Looking forward to the error log (and schema if possible), if you have them.

          Roel Roel Van de Paar added a comment - wlad Kindly checked the minidump for us and regrettably there is not much information within it other than Unhandled exception at 0x00007FF804EFF3E4 (server.dll) in MDEV-35341--mysqld.dmp: 0xC0000005: Access violation reading location 0x0000000000000040. And that the length of the query packet is 849 bytes. npdmailing@gmail.com Looking forward to the error log (and schema if possible), if you have them.

          People

            Unassigned Unassigned
            npdmailing@gmail.com Mitchell Lee
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.