Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-13779

InnoDB fails to shut down purge workers, causing hang

Details

    Description

      innodb_fts.innodb_fts_plugin 'innodb' w2 [ fail ] Found warnings/errors in server log file!
      http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest/builds/10121/steps/test_2/logs/mysqld.1.err.2

      CURRENT_TEST: innodb_fts.innodb_fts_plugin
      		 
      2017-09-08 14:13:13 0 [Note] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld (root[root] @ localhost []): Normal shutdown
      		 
       
      		 
      2017-09-08 14:13:13 0 [Note] InnoDB: FTS optimize thread exiting.
      		 
      2017-09-08 14:13:13 0 [Note] Event Scheduler: Purging the queue. 0 events
      		 
      2017-09-08 14:13:13 0 [Note] InnoDB: Starting shutdown...
      		 
      2017-09-08 14:13:13 0 [Note] InnoDB: Dumping buffer pool(s) to /dev/shm/var_auto_Jb6A/2/mysqld.1/data/ib_buffer_pool
      		 
      2017-09-08 14:13:13 0 [Note] InnoDB: Instance 0, restricted to 127 pages due to innodb_buf_pool_dump_pct=25
      		 
      2017-09-08 14:13:13 0 [Note] InnoDB: Buffer pool(s) dump completed at 170908 14:13:13
      		 
      2017-09-08 14:13:14 0 [Note] InnoDB: Shutdown completed; log sequence number 167888018
      		 
      2017-09-08 14:13:14 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      		 
      2017-09-08 14:13:14 0 [Note] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: Shutdown complete
      		 
       
      		 
      "2017-09-08 14:13:16 0 [Note] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld (mysqld 10.3.2-MariaDB-log) starting as process 5693 ...
      		 
      2017-09-08 14:13:16 0 [Warning] Could not increase number of max_open_files to more than 1024 (request: 4162)
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'partition' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Uses event mutexes
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Using Linux native AIO
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Number of pools: 1
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Using generic crc32 instructions
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Initializing buffer pool, total size = 8M, instances = 1, chunk size = 8M
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Waiting for purge to start
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: 5.7.19 started; log sequence number 167888018
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'SEQUENCE' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_SYS_DATAFILES' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_SYS_TABLESTATS' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_TABLESPACES_SCRUBBING' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_CMP' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_DELETED' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_CMP_RESET' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_LOCK_WAITS' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_TABLESPACES_ENCRYPTION' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_CMPMEM_RESET' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'FEEDBACK' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_INDEX_TABLE' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_CMP_PER_INDEX_RESET' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'user_variables' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_INDEX_CACHE' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_BEING_DELETED' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_DEFAULT_STOPWORD' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_FT_CONFIG' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_SYS_TABLESPACES' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_MUTEXES' is disabled.
      		 
      2017-09-08 14:13:16 0 [Note] Plugin 'INNODB_SYS_SEMAPHORE_WAITS' is disabled.
      		 
      2017-09-08 14:13:16 0 [Warning] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: unknown variable 'loose-feedback-debug-startup-interval=20'
      		 
      2017-09-08 14:13:16 0 [Warning] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: unknown variable 'loose-feedback-debug-first-interval=60'
      		 
      2017-09-08 14:13:16 0 [Warning] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: unknown variable 'loose-feedback-debug-interval=60'
      		 
      2017-09-08 14:13:16 0 [Warning] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: unknown option '--loose-pam-debug'
      		 
      2017-09-08 14:13:16 0 [Warning] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: unknown variable 'loose-debug-sync-timeout=300'
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Loading buffer pool(s) from /dev/shm/var_auto_Jb6A/2/mysqld.1/data/ib_buffer_pool
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: Buffer pool(s) load completed at 170908 14:13:16
      		 
      2017-09-08 14:13:16 0 [Note] Server socket created on IP: '127.0.0.1'.
      		 
      2017-09-08 14:13:16 0 [Note] Reading of all Master_info entries succeded
      		 
      2017-09-08 14:13:16 0 [Note] Added new Master_info '' to hash table
      		 
      2017-09-08 14:13:16 0 [Note] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld: ready for connections.
      		 
      Version: '10.3.2-MariaDB-log'  socket: '/mnt/buildbot/build/mariadb-10.3.2/mysql-test/var/tmp/2/mysqld.1.sock'  port: 16020  MariaDB Server
      		 
      2017-09-08 14:13:16 0 [Note] /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld (root[root] @ localhost []): Normal shutdown
      		 
       
      		 
      2017-09-08 14:13:16 0 [Note] Event Scheduler: Purging the queue. 0 events
      		 
      2017-09-08 14:13:16 0 [Note] InnoDB: FTS optimize thread exiting.
      		 
      170908 14:14:16 [ERROR] mysqld got signal 6 ;
      		 
      This could be because you hit a bug. It is also possible that this binary
      		 
      or one of the libraries it was linked against is corrupt, improperly built,
      		 
      or misconfigured. This error can also be caused by malfunctioning hardware.
      		 
       
      		 
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      		 
       
      		 
      We will try our best to scrape up some info that will hopefully help
      		 
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
      		 
       
      		 
      Server version: 10.3.2-MariaDB-log
      		 
      key_buffer_size=1048576
      		 
      read_buffer_size=131072
      		 
      max_used_connections=1
      		 
      max_threads=153
      		 
      thread_count=3
      		 
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 63038 K  bytes of memory
      		 
      Hope that's ok; if not, decrease some variables in the equation.
      		 
       
      		 
      Thread pointer: 0x0
      		 
      Attempting backtrace. You can use the following information to find out
      		 
      where mysqld died. If you see no messages after this, something went
      		 
      terribly wrong...
      		 
      stack_bottom = 0x0 thread_stack 0x49000
      		 
      /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld(my_print_stacktrace+0x2e)[0x55d16586fcee]
      		 
      /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld(handle_fatal_signal+0x307)[0x55d165308f67]
      		 
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f123a020390]
      		 
      /lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_wait+0xc0)[0x7f123a01c360]
      		 
      /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld(_Z11mysqld_mainiPPc+0x1232)[0x55d1650b0372]
      		 
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f12395db830]
      		 
      /mnt/buildbot/build/mariadb-10.3.2/sql/mysqld(_start+0x29)[0x55d1650a2b89]
      		 
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      		 
      information that should help you find out what is causing the crash.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15554 InnoDB page_cleaner shutdown sometimes hangs

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-11802 innodb.innodb_bug14676111 fails in buildbot due to InnoDB purge failing to start when there is work to do

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-14379 encryption.innodb_encrypt_log_corruption failed in buildbot, crash

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-15817 encryption tests fail futex(COND_thread_count/LOCK_thread_count)

          • Major - Major loss of function.
          • Open

          Activity

            Ascending order - Click to sort in descending order
            alice Alice Sherepa created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            Summary innodb_fts_plugin failed in buildbot (sporadic) innodb_fts_plugin failed in buildbot with server crash on shutdown (sporadic)
            ratzpo Rasmus Johansson (Inactive) made changes -
            Assignee Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            Summary innodb_fts_plugin failed in buildbot with server crash on shutdown (sporadic) innodb_fts_plugin failed in buildbot with server hang on shutdown (sporadic)
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Summary innodb_fts_plugin failed in buildbot with server hang on shutdown (sporadic) InnoDB fails to shut down purge workers, causing hang
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2018-05-16 13:44:03.0 2018-05-16 13:44:03.971
            marko Marko Mäkelä made changes -
            Component/s Storage Engine - InnoDB [ 10129 ]
            Component/s Tests [ 10800 ]
            Fix Version/s 10.2.15 [ 23006 ]
            Fix Version/s 10.3.7 [ 23005 ]
            Fix Version/s 10.3 [ 22126 ]
            Resolution Fixed [ 1 ]
            Status Open [ 1 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            Affects Version/s 10.3.0 [ 22127 ]
            Affects Version/s 10.2.3 [ 22115 ]
            Affects Version/s 10.3 [ 22126 ]
            marko Marko Mäkelä made changes -
            Labels hang shutdown
            marko Marko Mäkelä made changes -
            serg Sergei Golubchik made changes -
            Fix Version/s 10.2.16 [ 23110 ]
            Fix Version/s 10.2.15 [ 23006 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.3.8 [ 23113 ]
            Fix Version/s 10.3.7 [ 23005 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 82522 ] MariaDB v4 [ 152802 ]

            People

              marko Marko Mäkelä
              alice Alice Sherepa
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.