[MDEV-7634] mysqld hangs on shutdown waiting for galera service thread Created: 2015-02-25  Updated: 2018-07-16  Resolved: 2018-07-16

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.0.16-galera
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Ragnar Rova Assignee: Jan Lindström (Inactive)
Resolution: Cannot Reproduce Votes: 0
Labels: docker, galera

Attachments: Text File gdb-backtrace.txt     Text File mysqld-stop-hangs-waiting-on-galera-thread.log    

 Description   

1. Bootstrap a 3-node galera cluster.
2. Stop bootstrap node (no clients connected, no transactions in progress)

mysqld sometimes hangs indefinitely on shutdown

150225 16:02:21 [Note] /usr/sbin/mysqld: Normal shutdown

150225 16:02:21 [Note] WSREP: Stop replication
150225 16:02:23 [Note] WSREP: rollbacker thread exiting
150225 16:02:23 [Note] Event Scheduler: Purging the queue. 0 events
150225 16:02:43 [Warning] /usr/sbin/mysqld: Forcing close of thread 26 user: ''

Thread 26 (Thread 0x7fec93bb2700 (LWP 11840)):
#0 0x00007feca06a15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007fec9bd5962b in wait (arg=0x7fec9cd54320) at galerautils/src/gu_lock.hpp:56
#2 galera::ServiceThd::thd_func (arg=0x7fec9cd54320) at galera/src/galera_service_thd.cpp:30
#3 0x00007feca069d9d1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007fec9edb78fd in clone () from /lib64/libc.so.6

 Comments   
Comment by Ragnar Rova [ 2015-02-25 ]

Using libgalera 25.3.9

Comment by Nirbhay Choubey (Inactive) [ 2015-02-25 ]

Can you share the configuration (my.cnf) used for all 3 nodes?

Comment by Ragnar Rova [ 2015-02-25 ]

This is the template used

[mysqld_safe]
log-error = /var/log/mysqld.log

[mysqld]
server-id = %(server-id)s
datadir = /var/lib/mysql
socket = /var/lib/mysql/mysql.sock
user = mysql
symbolic-links = 0
log-bin = mysqld-bin
binlog-format = ROW
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
sync_binlog = 2
expire_logs_days = 4
table_cache = 2048
sort_buffer_size = 8M
thread_cache_size = 32
innodb_flush_log_at_trx_commit = 2
skip-external-locking
sql_mode = STRICT_ALL_TABLES
innodb_buffer_pool_size = %(innodb_buffer_pool_size)s
character-set-server = utf8
collation-server = utf8_unicode_ci

[mariadb]
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address = gcomm://%(other_nodes)s
wsrep_sst_method = rsync

Comment by Ragnar Rova [ 2015-02-25 ]

template variables on db1 (symmetrical on db2,db3):

other_nodes = db2,db3
innodb_buffer_pool_size = MemTotal/10

Comment by Ragnar Rova [ 2015-02-25 ]

Steps to reproduce

1. Bootstrap db1 (service mysql bootstrap --init-file=/var/lib/mysql/reset_password.sql)
2. Join db2 (service mysql start)
3. Join db3 (service mysql start)
4. Stop db1, bootstrap db1
5. Stop db2, start db2 (fails to start, might be second bug?), start again with --wsrep-provider-options='pc.npvo=false'
6. Stop db3, start db3 (succeeds)
7. Stop db1

-> hangs as described. Attaching full log from db1 of these events.

Comment by Jan Lindström (Inactive) [ 2018-07-16 ]

Can you try more recent release of MariaDB Galera cluster 10.0 ?

Comment by Jan Lindström (Inactive) [ 2018-07-16 ]

After several tries I could not repeat the issue so either it is already fixed or my environment does not match the environment where issue is repeatable.

Generated at Thu Feb 08 07:21:06 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.