Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-9088

Server crashes on shutdown after the second post of feedback report

Details

    • 10.0.23

    Description

      See also MDEV-9087 about debug build – there it fails right after the second post, while here it crashes on server shutdown.
      Reproducible with the dynamic plugin.
      cmake . && make -j3

      151105 19:51:26 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent
      		 
      151105 19:51:27 [Note] feedback plugin: server replied 'ok'
      		 
      151105 19:54:29 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent
      		 
      151105 19:54:35 [Note] feedback plugin: server replied 'ok'
      		 
      151105 20:08:27 [Note] /home/elenst/git/10.0-bug2/sql/mysqld: Normal shutdown
      		 
       
      		 
      151105 20:08:27 [Note] Event Scheduler: Purging the queue. 0 events
      		 
      151105 20:08:28 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent
      		 
      151105 20:08:28 [ERROR] mysqld got signal 11 ;
      		 

      Thread 1 (Thread 0x7f894eff9700 (LWP 6851)):
      		 
      #0  0x00007f8977af9f8c in pthread_kill () from /lib/x86_64-linux-gnu/libpthread.so.0
      		 
      #1  0x00000000006ec3f2 in handle_fatal_signal (sig=11) at sql/signal_handler.cc:262
      		 
      #2  <signal handler called>
      		 
      #3  0x000000000058f7b5 in thd_wait_begin (thd=0x7f894d0c4008, wait_type=<optimised out>) at sql/sql_class.cc:4557
      		 
      #4  0x0000000000af209f in vio_io_wait (vio=vio@entry=0x7f894d041208, event=VIO_IO_EVENT_READ, timeout=60000) at vio/viosocket.c:967
      		 
      #5  0x0000000000af2148 in vio_socket_io_wait (vio=vio@entry=0x7f894d041208, event=<optimised out>) at vio/viosocket.c:108
      		 
      #6  0x0000000000af2c51 in vio_ssl_read (vio=0x7f894d041208, buf=0x7f894eff7d30 "POST /feedback_plugin/post HTTP/1.0\r\nUser-Agent: MariaDB User Feedback Plugin\r\nHost: mariadb.org:443\r\nAccept: */*\r\nContent-Length: 280\r\nContent-Type: multipart/form-data; boundary=", '-' <repeats 20 times>..., size=<optimised out>) at vio/viossl.c:155
      		 
      #7  0x00007f89764d55ec in feedback::Url_http::send (this=0x7f8975cd6d30, data=0x7f894d012008 "FEEDBACK_SERVER_UID\t/zAqDR9PHkV223/Obu2tCd/0E4E=\nFEEDBACK_WHEN\tshutdown\nFEEDBACK_USER_INFO\t\n\ntCd/0E4E=\nPERFORMANCE_SCHEMA_EVENTS_WAITS_HISTORY_LONG_SIZE\t-1\nLARGE_PAGES\tOFF\nINNODB_FLUSH_LOG_AT_TIMEOUT\t"..., data_length=93) at plugin/feedback/url_http.cc:269
      		 
      #8  0x00007f89764d3423 in feedback::send_report (when=when@entry=0x7f89764d6991 "shutdown") at plugin/feedback/sender_thread.cc:242
      		 
      #9  0x00007f89764d5069 in feedback::background_thread (arg=<optimised out>) at plugin/feedback/sender_thread.cc:296
      		 
      #10 0x00007f8977af4e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
      		 
      #11 0x00007f89769e4cbd in clone () from /lib/x86_64-linux-gnu/libc.so.6

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9087 Assertion `(longlong) thd->status_var.memory_used >= 0' fails in my_malloc_size_cb_func on second post of feedback report

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-9051 mysqld got signal 11, after upgrade to 10.1.8

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            elenst Elena Stepanova added a comment - - edited

            Still reproducible on 10.0 commit 90ea0145856338221803ebb9b446ed2a6e082412 (current pre-10.0.23 tree of December 16).
            I've set it up on perro.

            • log in as usual 

              cd mdev9088
              		 
              ./run

              The server was built as cmake . && make -j3.
              The feedback plugin is modified to send reports in 60+60 sec. The script starts the server, waits for 2 minutes, shuts down the server, checks the error log.
              Output is like this:

              Starting MariaDB server ...
              		 
               
              		 
              151216 19:56:53 [Note] /home/mariadb/git/10.0-mdev9088/sql/mysqld (mysqld 10.0.23-MariaDB) starting as process 13276 ...
              		 
              151216 19:56:53 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /home/mariadb/git/10.0-mdev9088/sql/share/english/
              		 
              151216 19:56:53 [Warning] Use --lc-messages-dir without language specific part instead.
              		 
               
              		 
              The feedback plugin was modified to fire in 1 min, and then again in 1 min, waiting...
              		 
              The error log should say that the report was sent twice
              		 
               
              		 
              Version: '10.0.23-MariaDB'  socket: '/home/mariadb/git/10.0-mdev9088/data/tmp/mysql.sock'  port: 9088  Source distribution
              		 
              151216 19:57:54 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent
              		 
              151216 19:57:56 [Note] feedback plugin: server replied 'ok'
              		 
              151216 19:58:57 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent
              		 
              151216 19:58:59 [Note] feedback plugin: server replied 'ok'
              		 
               
              		 
              Shutting down the server...
              		 
               
              		 
              ./run: line 18: 13276 Segmentation fault      (core dumped) /home/mariadb/git/10.0-mdev9088/sql/mysqld --no-defaults --basedir=/home/mariadb/git/10.0-mdev9088 --datadir=/home/mariadb/git/10.0-mdev9088/data --log-error=/home/mariadb/git/10.0-mdev9088/data/log.err --loose-lc-messages-dir=/home/mariadb/git/10.0-mdev9088/sql/share --loose-language=/home/mariadb/git/10.0-mdev9088/sql/share/english --port=9088 --socket=/home/mariadb/git/10.0-mdev9088/data/tmp/mysql.sock --tmpdir=/home/mariadb/git/10.0-mdev9088/data/tmp --loose-core --plugin-dir=/home/mariadb/git/10.0-mdev9088/plugin/feedback --plugin-load-add=feedback.so --feedback
              		 
               
              		 
              Checking the error log now
              		 
               
              		 
               
              		 
              Server version: 10.0.23-MariaDB
              		 
              key_buffer_size=134217728
              		 
              read_buffer_size=131072
              		 
              max_used_connections=1
              		 
              max_threads=153
              		 
              thread_count=0
              		 
              It is possible that mysqld could use up to 
              key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467003 K  bytes of memory
              		 
              Hope that's ok; if not, decrease some variables in the equation.
              		 
               
              		 
              Thread pointer: 0x0x7f05aecc4008
              		 
              Attempting backtrace. You can use the following information to find out
              		 
              where mysqld died. If you see no messages after this, something went
              		 
              terribly wrong...
              		 
              stack_bottom = 0x0 thread_stack 0x48000
              		 
              /home/mariadb/git/10.0-mdev9088/sql/mysqld(my_print_stacktrace+0x29)[0xac4959]
              		 
              /home/mariadb/git/10.0-mdev9088/sql/mysqld(handle_fatal_signal+0x398)[0x6edcb8]
              		 
              /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f05d969bcb0]
              		 
              sql/sql_class.cc:4556(thd_wait_begin)[0x58faa5]
              		 
              /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_io_wait+0x12f)[0xaf451f]
              		 
              /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_socket_io_wait+0x18)[0xaf45c8]
              		 
              /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_ssl_read+0x61)[0xaf50d1]
              		 
              /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(_ZN8feedback8Url_http4sendEPKcm+0x43c)[0x7f05d80748ac]
              		 
              /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(+0xa6f5)[0x7f05d80726f5]
              		 
              /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(background_thread+0x93)[0x7f05d8074323]
              		 
              /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f05d9693e9a]
              		 
              /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f05d8583cbd]
              		 
               
              		 
              Trying to get some variables.
              		 
              Some pointers may be invalid and cause the dump to abort.
              		 
              Query (0x9499f115528087f6): Can't read from address 0x9499f115528087f6
              		 
              Connection ID (thread ID): 0
              		 
              Status: NOT_KILLED
              		 
               
              		 
              Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=off,derived_merge=off,derived_with_keys=on,firstmatch=off,loosescan=on,materialization=off,in_to_exists=on,semijoin=off,partial_match_rowid_merge=on,partial_match_table_scan=off,subquery_cache=off,mrr=off,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=off,semijoin_with_cache=off,join_cache_incremental=off,join_cache_hashed=off,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=off,extended_keys=on,exists_to_in=off
              		 
               
              		 
              The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
              		 
              information that should help you find out what is causing the crash.
              		 
              Writing a core file

            elenst Elena Stepanova added a comment - - edited Still reproducible on 10.0 commit 90ea0145856338221803ebb9b446ed2a6e082412 (current pre-10.0.23 tree of December 16). I've set it up on perro. log in as usual cd mdev9088 ./run The server was built as cmake . && make -j3 . The feedback plugin is modified to send reports in 60+60 sec. The script starts the server, waits for 2 minutes, shuts down the server, checks the error log. Output is like this: Starting MariaDB server ...   151216 19:56:53 [Note] /home/mariadb/git/10.0-mdev9088/sql/mysqld (mysqld 10.0.23-MariaDB) starting as process 13276 ... 151216 19:56:53 [Warning] An old style --language or -lc-message-dir value with language specific part detected: /home/mariadb/git/10.0-mdev9088/sql/share/english/ 151216 19:56:53 [Warning] Use --lc-messages-dir without language specific part instead.   The feedback plugin was modified to fire in 1 min, and then again in 1 min, waiting... The error log should say that the report was sent twice   Version: '10.0.23-MariaDB' socket: '/home/mariadb/git/10.0-mdev9088/data/tmp/mysql.sock' port: 9088 Source distribution 151216 19:57:54 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent 151216 19:57:56 [Note] feedback plugin: server replied 'ok' 151216 19:58:57 [Note] feedback plugin: report to 'https://mariadb.org/feedback_plugin/post' was sent 151216 19:58:59 [Note] feedback plugin: server replied 'ok'   Shutting down the server...   ./run: line 18: 13276 Segmentation fault (core dumped) /home/mariadb/git/10.0-mdev9088/sql/mysqld --no-defaults --basedir=/home/mariadb/git/10.0-mdev9088 --datadir=/home/mariadb/git/10.0-mdev9088/data --log-error=/home/mariadb/git/10.0-mdev9088/data/log.err --loose-lc-messages-dir=/home/mariadb/git/10.0-mdev9088/sql/share --loose-language=/home/mariadb/git/10.0-mdev9088/sql/share/english --port=9088 --socket=/home/mariadb/git/10.0-mdev9088/data/tmp/mysql.sock --tmpdir=/home/mariadb/git/10.0-mdev9088/data/tmp --loose-core --plugin-dir=/home/mariadb/git/10.0-mdev9088/plugin/feedback --plugin-load-add=feedback.so --feedback   Checking the error log now     Server version: 10.0.23-MariaDB key_buffer_size=134217728 read_buffer_size=131072 max_used_connections=1 max_threads=153 thread_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467003 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x0x7f05aecc4008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48000 /home/mariadb/git/10.0-mdev9088/sql/mysqld(my_print_stacktrace+0x29)[0xac4959] /home/mariadb/git/10.0-mdev9088/sql/mysqld(handle_fatal_signal+0x398)[0x6edcb8] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f05d969bcb0] sql/sql_class.cc:4556(thd_wait_begin)[0x58faa5] /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_io_wait+0x12f)[0xaf451f] /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_socket_io_wait+0x18)[0xaf45c8] /home/mariadb/git/10.0-mdev9088/sql/mysqld(vio_ssl_read+0x61)[0xaf50d1] /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(_ZN8feedback8Url_http4sendEPKcm+0x43c)[0x7f05d80748ac] /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(+0xa6f5)[0x7f05d80726f5] /home/mariadb/git/10.0-mdev9088/plugin/feedback/feedback.so(background_thread+0x93)[0x7f05d8074323] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7f05d9693e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f05d8583cbd]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x9499f115528087f6): Can't read from address 0x9499f115528087f6 Connection ID (thread ID): 0 Status: NOT_KILLED   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=off,index_merge_intersection=off,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=off,derived_merge=off,derived_with_keys=on,firstmatch=off,loosescan=on,materialization=off,in_to_exists=on,semijoin=off,partial_match_rowid_merge=on,partial_match_table_scan=off,subquery_cache=off,mrr=off,mrr_cost_based=on,mrr_sort_keys=off,outer_join_with_cache=off,semijoin_with_cache=off,join_cache_incremental=off,join_cache_hashed=off,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=off,extended_keys=on,exists_to_in=off   The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. Writing a core file
            wlad Vladislav Vaintroub added a comment - - edited

            Another callstack at shutdown
            https://buildbot.askmonty.org/buildbot/builders/win32-debug2/builds/8757/steps/test/logs/stdio

            It crashes sql_print_information, and it looks like THD pointer would be uninitialized at that time sql_print_information.

            The address being accessed is 0xcdcdcdcd, which means I believe is uninitialized heap memory with C runtime debug library

            mysqld!print_buffer_to_file [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8074]
            mysqld!vprint_msg_to_log [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8125]
            mysqld!Log_to_file_event_handler::log_error [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 1004]
            mysqld!LOGGER::error_log_print [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 1112]
            mysqld!error_log_print [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 6058]
            mysqld!sql_print_information [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8168]
            mysqld!feedback::Url_http::send [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\url_http.cc @ 257]
            mysqld!feedback::send_report [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\sender_thread.cc @ 243]
            mysqld!background_thread [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\sender_thread.cc @ 298]
            mysqld!pthread_start [c:\bzr\bb-win32\win32-debug2\build\mysys\my_winthread.c @ 60]
            mysqld!_callthreadstartex [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 314]
            mysqld!_threadstartex [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 297]

            wlad Vladislav Vaintroub added a comment - - edited Another callstack at shutdown https://buildbot.askmonty.org/buildbot/builders/win32-debug2/builds/8757/steps/test/logs/stdio It crashes sql_print_information, and it looks like THD pointer would be uninitialized at that time sql_print_information. The address being accessed is 0xcdcdcdcd, which means I believe is uninitialized heap memory with C runtime debug library mysqld!print_buffer_to_file [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8074] mysqld!vprint_msg_to_log [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8125] mysqld!Log_to_file_event_handler::log_error [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 1004] mysqld!LOGGER::error_log_print [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 1112] mysqld!error_log_print [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 6058] mysqld!sql_print_information [c:\bzr\bb-win32\win32-debug2\build\sql\log.cc @ 8168] mysqld!feedback::Url_http::send [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\url_http.cc @ 257] mysqld!feedback::send_report [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\sender_thread.cc @ 243] mysqld!background_thread [c:\bzr\bb-win32\win32-debug2\build\plugin\feedback\sender_thread.cc @ 298] mysqld!pthread_start [c:\bzr\bb-win32\win32-debug2\build\mysys\my_winthread.c @ 60] mysqld!_callthreadstartex [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 314] mysqld!_threadstartex [f:\dd\vctools\crt_bld\self_x86\crt\src\threadex.c @ 297]

            People

              serg Sergei Golubchik
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.