Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7634

mysqld hangs on shutdown waiting for galera service thread

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Cannot Reproduce
    • 10.0.16-galera
    • N/A
    • Galera

    Description

      1. Bootstrap a 3-node galera cluster.
      2. Stop bootstrap node (no clients connected, no transactions in progress)

      mysqld sometimes hangs indefinitely on shutdown

      150225 16:02:21 [Note] /usr/sbin/mysqld: Normal shutdown

      150225 16:02:21 [Note] WSREP: Stop replication
      150225 16:02:23 [Note] WSREP: rollbacker thread exiting
      150225 16:02:23 [Note] Event Scheduler: Purging the queue. 0 events
      150225 16:02:43 [Warning] /usr/sbin/mysqld: Forcing close of thread 26 user: ''

      Thread 26 (Thread 0x7fec93bb2700 (LWP 11840)):
      #0 0x00007feca06a15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
      #1 0x00007fec9bd5962b in wait (arg=0x7fec9cd54320) at galerautils/src/gu_lock.hpp:56
      #2 galera::ServiceThd::thd_func (arg=0x7fec9cd54320) at galera/src/galera_service_thd.cpp:30
      #3 0x00007feca069d9d1 in start_thread () from /lib64/libpthread.so.0
      #4 0x00007fec9edb78fd in clone () from /lib64/libc.so.6

      Attachments

        Activity

          rrva Ragnar Rova added a comment -

          Using libgalera 25.3.9

          rrva Ragnar Rova added a comment - Using libgalera 25.3.9

          Can you share the configuration (my.cnf) used for all 3 nodes?

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Can you share the configuration (my.cnf) used for all 3 nodes?
          rrva Ragnar Rova added a comment - - edited

          This is the template used

          [mysqld_safe]
          log-error = /var/log/mysqld.log

          [mysqld]
          server-id = %(server-id)s
          datadir = /var/lib/mysql
          socket = /var/lib/mysql/mysql.sock
          user = mysql
          symbolic-links = 0
          log-bin = mysqld-bin
          binlog-format = ROW
          default_storage_engine = InnoDB
          innodb_autoinc_lock_mode = 2
          sync_binlog = 2
          expire_logs_days = 4
          table_cache = 2048
          sort_buffer_size = 8M
          thread_cache_size = 32
          innodb_flush_log_at_trx_commit = 2
          skip-external-locking
          sql_mode = STRICT_ALL_TABLES
          innodb_buffer_pool_size = %(innodb_buffer_pool_size)s
          character-set-server = utf8
          collation-server = utf8_unicode_ci

          [mariadb]
          wsrep_provider = /usr/lib64/galera/libgalera_smm.so
          wsrep_cluster_address = gcomm://%(other_nodes)s
          wsrep_sst_method = rsync

          rrva Ragnar Rova added a comment - - edited This is the template used [mysqld_safe] log-error = /var/log/mysqld.log [mysqld] server-id = %(server-id)s datadir = /var/lib/mysql socket = /var/lib/mysql/mysql.sock user = mysql symbolic-links = 0 log-bin = mysqld-bin binlog-format = ROW default_storage_engine = InnoDB innodb_autoinc_lock_mode = 2 sync_binlog = 2 expire_logs_days = 4 table_cache = 2048 sort_buffer_size = 8M thread_cache_size = 32 innodb_flush_log_at_trx_commit = 2 skip-external-locking sql_mode = STRICT_ALL_TABLES innodb_buffer_pool_size = %(innodb_buffer_pool_size)s character-set-server = utf8 collation-server = utf8_unicode_ci [mariadb] wsrep_provider = /usr/lib64/galera/libgalera_smm.so wsrep_cluster_address = gcomm://%(other_nodes)s wsrep_sst_method = rsync
          rrva Ragnar Rova added a comment -

          template variables on db1 (symmetrical on db2,db3):

          other_nodes = db2,db3
          innodb_buffer_pool_size = MemTotal/10

          rrva Ragnar Rova added a comment - template variables on db1 (symmetrical on db2,db3): other_nodes = db2,db3 innodb_buffer_pool_size = MemTotal/10
          rrva Ragnar Rova added a comment - - edited

          Steps to reproduce

          1. Bootstrap db1 (service mysql bootstrap --init-file=/var/lib/mysql/reset_password.sql)
          2. Join db2 (service mysql start)
          3. Join db3 (service mysql start)
          4. Stop db1, bootstrap db1
          5. Stop db2, start db2 (fails to start, might be second bug?), start again with --wsrep-provider-options='pc.npvo=false'
          6. Stop db3, start db3 (succeeds)
          7. Stop db1

          -> hangs as described. Attaching full log from db1 of these events.

          rrva Ragnar Rova added a comment - - edited Steps to reproduce 1. Bootstrap db1 (service mysql bootstrap --init-file=/var/lib/mysql/reset_password.sql) 2. Join db2 (service mysql start) 3. Join db3 (service mysql start) 4. Stop db1, bootstrap db1 5. Stop db2, start db2 (fails to start, might be second bug?), start again with --wsrep-provider-options='pc.npvo=false' 6. Stop db3, start db3 (succeeds) 7. Stop db1 -> hangs as described. Attaching full log from db1 of these events.

          Can you try more recent release of MariaDB Galera cluster 10.0 ?

          jplindst Jan Lindström (Inactive) added a comment - Can you try more recent release of MariaDB Galera cluster 10.0 ?

          After several tries I could not repeat the issue so either it is already fixed or my environment does not match the environment where issue is repeatable.

          jplindst Jan Lindström (Inactive) added a comment - After several tries I could not repeat the issue so either it is already fixed or my environment does not match the environment where issue is repeatable.

          People

            jplindst Jan Lindström (Inactive)
            rrva Ragnar Rova
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.