Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8540

Crash on server shutdown since 10.0.16

Details

    Description

      This problem was first found in 10.0.20, and I bisected it back to 10.0.16. Every time the server is shut down, it crashes. I am able to reproduce this with an empty data directory.

      150724 13:08:00 [Note] /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld: Normal shutdown
      		 
       
      		 
      150724 13:08:00 [Note] Event Scheduler: Purging the queue. 0 events
      		 
      150724 13:08:00 [ERROR] mysqld got signal 11 ;
      		 
      This could be because you hit a bug. It is also possible that this binary
      		 
      or one of the libraries it was linked against is corrupt, improperly built,
      		 
      or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
       
      		 
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
      		 
       
      		 
      We will try our best to scrape up some info that will hopefully help
      		 
      diagnose the problem, but since we have already crashed,
      		 
      something is definitely wrong and this may fail.
      		 
       
      		 
      Server version: 10.0.16-MariaDB-log
      		 
      key_buffer_size=134217728
      		 
      read_buffer_size=131072
      		 
      max_used_connections=1
      		 
      max_threads=1026
      		 
      thread_count=0
      		 
      It is possible that mysqld could use up to
      		 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2384868 K  bytes of memory
      		 
      Hope that's ok; if not, decrease some variables in the equation.
      		 
       
      		 
      Thread pointer: 0x0x0
      		 
      Attempting backtrace. You can use the following information to find out
      		 
      where mysqld died. If you see no messages after this, something went
      		 
      terribly wrong...
      		 
      stack_bottom = 0x0 thread_stack 0x48000
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(my_print_stacktrace+0x29)[0xb75919]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(handle_fatal_signal+0x398)[0x721868]
      		 
      /nix/store/la5imi1602jxhpds9675n2n2d0683lbq-glibc-2.20/lib/libpthread.so.0(+0x10020)[0x7f8852c59020]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(_ZN13MYSQL_BIN_LOG5closeEj+0x2cf)[0x7da30f]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(_ZN13MYSQL_BIN_LOG7cleanupEv+0x52)[0x7da422]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld[0x53402d]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(_Z10unireg_endv+0x2d)[0x535fed]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld[0x538ab9]
      		 
      /nix/store/w2f13ydy5nyp53vsi17k6y0igkw99d7l-mariadb-10.0.16/bin/mysqld(kill_server_thread+0xe)[0x538b8e]
      		 
      /nix/store/la5imi1602jxhpds9675n2n2d0683lbq-glibc-2.20/lib/libpthread.so.0(+0x8374)[0x7f8852c51374]
      		 
      /nix/store/la5imi1602jxhpds9675n2n2d0683lbq-glibc-2.20/lib/libc.so.6(clone+0x6d)[0x7f88514c9f9d]
      		 
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      		 
      information that should help you find out what is causing the crash.

      I attached the config file in use. MariaDB is built from source. The OS is NixOS.

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          View 16 older comments
          phunehehe Hoang Xuan Phu added a comment -

          elenst that was fantastic!

          phunehehe Hoang Xuan Phu added a comment - elenst that was fantastic!
          svoj Sergey Vojtovich added a comment -

          There're two problems here.

          Crash itself happens because we're trying to access current_thd->variables.lc_messages->errmsgs->errmsgs, whereas current_thd is NULL, when we attempt to report an error. This affects 5.5+.

          But the fact that we come across this error is another bug. Since 10.0 we do fclose(stdin) after binlog index file has been opened. If we were started with stdin closed, binlog index file gets 0 fd and then it gets closed. On shutdown we attempt to close binlog index file again and second close fails.

          svoj Sergey Vojtovich added a comment - There're two problems here. Crash itself happens because we're trying to access current_thd->variables.lc_messages->errmsgs->errmsgs, whereas current_thd is NULL, when we attempt to report an error. This affects 5.5+. But the fact that we come across this error is another bug. Since 10.0 we do fclose(stdin) after binlog index file has been opened. If we were started with stdin closed, binlog index file gets 0 fd and then it gets closed. On shutdown we attempt to close binlog index file again and second close fails.
          svoj Sergey Vojtovich added a comment -

          In 10.1 things are got even more complicated with the following patch:

          commit 9f00950d2ad9be5533e66a5fa87a795a9574585f
          		 
          Author: Nirbhay Choubey <nirbhay@mariadb.com>
          		 
          Date:   Wed Jun 24 23:25:22 2015 -0400
          		 
           
          		 
              MDEV-7631 : Invalid WSREP_SST rows appear in mysqld-bin.index file
          		 
           
          		 
              Try not to release 0 (STDIN_FILENO) file descriptor as it can
          		 
              incorrectly get reused by streams opened later.
          		 
           
          		 
          diff --git a/sql/mysqld.cc b/sql/mysqld.cc
          		 
          index c297367..a2fe4ad 100644
          		 
          --- a/sql/mysqld.cc
          		 
          +++ b/sql/mysqld.cc
          		 
          @@ -6278,7 +6278,14 @@ int mysqld_main(int argc, char **argv)
          		 
                                    (char*) "" : mysqld_unix_port),
          		 
                                    mysqld_port,
          		 
                                    MYSQL_COMPILATION_COMMENT);
          		 
          -  fclose(stdin);
          		 
          +
          		 
          +  // try to keep fd=0 busy
          		 
          +  if (!freopen(IF_WIN("NUL","/dev/null"), "r", stdin))
          		 
          +  {
          		 
          +    // fall back on failure
          		 
          +    fclose(stdin);
          		 
          +  }
          		 
          +
          		 
           #if defined(_WIN32) && !defined(EMBEDDED_LIBRARY)
          		 
             Service.SetRunning();
          		 
           #endif

          Looks like an incorrect attempt to workaround this problem.

          svoj Sergey Vojtovich added a comment - In 10.1 things are got even more complicated with the following patch: commit 9f00950d2ad9be5533e66a5fa87a795a9574585f Author: Nirbhay Choubey <nirbhay@mariadb.com> Date: Wed Jun 24 23:25:22 2015 -0400   MDEV-7631 : Invalid WSREP_SST rows appear in mysqld-bin.index file   Try not to release 0 (STDIN_FILENO) file descriptor as it can incorrectly get reused by streams opened later.   diff --git a/sql/mysqld.cc b/sql/mysqld.cc index c297367..a2fe4ad 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -6278,7 +6278,14 @@ int mysqld_main(int argc, char **argv) (char*) "" : mysqld_unix_port), mysqld_port, MYSQL_COMPILATION_COMMENT); - fclose(stdin); + + // try to keep fd=0 busy + if (!freopen(IF_WIN("NUL","/dev/null"), "r", stdin)) + { + // fall back on failure + fclose(stdin); + } + #if defined(_WIN32) && !defined(EMBEDDED_LIBRARY) Service.SetRunning(); #endif Looks like an incorrect attempt to workaround this problem.
          svoj Sergey Vojtovich added a comment -

          sql_print_error() bug fixed in 5.5+.

          fclose() bug fixed in 10.0+.

          svoj Sergey Vojtovich added a comment - sql_print_error() bug fixed in 5.5+. fclose() bug fixed in 10.0+.
          phunehehe Hoang Xuan Phu added a comment -

          <3

          phunehehe Hoang Xuan Phu added a comment - <3

          People

            svoj Sergey Vojtovich
            phunehehe Hoang Xuan Phu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.