Maria won't shut down following upgrade to MariaDB Galera Cluster 5.5.35.
Logs with wsrep_debug = 1:
140216 0:25:02 [Note] /usr/sbin/mysqld: Normal shutdown
140216 0:25:02 [Note] WSREP: Stop replication
140216 0:25:02 [Note] WSREP: Provider disconnect
140216 0:25:02 [Note] WSREP: Closing send monitor...
140216 0:25:02 [Note] WSREP: Closed send monitor.
140216 0:25:02 [Note] WSREP: gcomm: terminating thread
140216 0:25:02 [Note] WSREP: gcomm: joining thread
140216 0:25:02 [Note] WSREP: gcomm: closing backend
140216 0:25:02 [Note] WSREP: view(view_id(NON_PRIM,0aaf4462-96a0-11e3-a1d0-a3367b5dbc93,178) memb {
ae5e2351-96a0-11e3-8c77-ce4c363e1cf4,0
} joined {
} left {
} partitioned {
0aaf4462-96a0-11e3-a1d0-a3367b5dbc93,0
458976e6-96a0-11e3-9a4d-d6f1862200ab,0
})
140216 0:25:02 [Note] WSREP: view((empty))
140216 0:25:02 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140216 0:25:02 [Note] WSREP: gcomm: closed
140216 0:25:02 [Note] WSREP: Flow-control interval: [512, 512]
140216 0:25:02 [Note] WSREP: Received NON-PRIMARY.
140216 0:25:02 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 45947383)
140216 0:25:02 [Note] WSREP: Received self-leave message.
140216 0:25:02 [Note] WSREP: Flow-control interval: [512, 512]
140216 0:25:02 [Note] WSREP: Received SELF-LEAVE. Closing connection.
140216 0:25:02 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 45947383)
140216 0:25:02 [Note] WSREP: New cluster view: global state: 64e060fb-10bd-11e3-0800-8ac8783f0ec6:45947383, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140216 0:25:02 [Note] WSREP: RECV thread exiting 0: Success
140216 0:25:02 [Note] WSREP: Setting wsrep_ready to 0
140216 0:25:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140216 0:25:02 [Note] WSREP: New cluster view: global state: 64e060fb-10bd-11e3-0800-8ac8783f0ec6:45947383, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
140216 0:25:02 [Note] WSREP: Setting wsrep_ready to 0
140216 0:25:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140216 0:25:02 [Note] WSREP: applier thread exiting (code:0)
140216 0:25:02 [Note] WSREP: closing applier 2
140216 0:25:02 [Note] WSREP: wsrep running threads now: 4
140216 0:25:02 [Note] WSREP: recv_thread() joined.
140216 0:25:02 [Note] WSREP: Closing replication queue.
140216 0:25:02 [Note] WSREP: Closing slave action queue.
140216 0:25:02 [Note] WSREP: applier thread exiting (code:6)
140216 0:25:02 [Note] WSREP: closing applier 22
140216 0:25:02 [Note] WSREP: wsrep running threads now: 3
140216 0:25:02 [Note] WSREP: applier thread exiting (code:6)
140216 0:25:02 [Note] WSREP: closing applier 20
140216 0:25:02 [Note] WSREP: wsrep running threads now: 2
140216 0:25:02 [Note] WSREP: applier thread exiting (code:6)
140216 0:25:02 [Note] WSREP: closing applier 21
140216 0:25:02 [Note] WSREP: wsrep running threads now: 1
140216 0:25:03 [Note] WSREP: Before Lock_thread_count
140216 0:25:04 [Note] WSREP: waiting for client connections to close: 22
140216 0:25:04 [Note] WSREP: closing wsrep thread 22
140216 0:25:04 [Note] WSREP: closing wsrep thread 21
140216 0:25:04 [Note] WSREP: closing wsrep thread 20
140216 0:25:04 [Note] WSREP: closing wsrep thread 2
140216 0:25:04 [Note] WSREP: closing wsrep thread 1
140216 0:25:04 [Note] WSREP: WSREP rollback thread wakes for signal
140216 0:25:04 [Note] WSREP: WSREP rollback thread has empty abort queue
140216 0:25:04 [Note] WSREP: rollbacker thread exiting
140216 0:25:04 [Note] WSREP: wsrep running threads now: 0
140216 0:25:04 [Note] Event Scheduler: Purging the queue. 0 events
140216 0:25:06 [Note] closing wsrep system thread
140216 0:25:06 [Note] closing wsrep system thread
140216 0:25:06 [Note] closing wsrep system thread
140216 0:25:06 [Note] closing wsrep system thread
140216 0:25:06 [Note] closing wsrep system thread
It seems to get stuck at this point and nothing else happens until I kill -9 the process. The cluster was upgraded with socket.checksum=1 (from 5.5.34) and I then disabled that option.
my.cnf:
[mysqld]
collation-server = utf8_unicode_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8
wsrep_cluster_address = 'gcomm://XXXX001,XXXX002,XXXX003'
wsrep_node_address = XXXX003
wsrep_provider = /usr/lib/galera/libgalera_smm.so
wsrep_sst_method = rsync
wsrep_sst_receive_address = XXXX003
wsrep_slave_threads = 4
wsrep_log_conflicts = 1
wsrep_retry_autocommit = 3
wsrep_provider_options = gcs.fc_limit=512; gcs.fc_master_slave=YES; gcs.fc_factor=1.0; gcache.size=5G;
datadir = /var/lib/mysql
default-storage-engine = InnoDB
user = mysql
max_allowed_packet = 16M
max_connect_errors = 1000000
transaction-isolation = REPEATABLE-READ
innodb_max_dirty_pages_pct = 30
innodb_file_per_table = 1
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 2
innodb_locks_unsafe_for_binlog = 1
innodb_autoinc_lock_mode = 2
innodb_print_all_deadlocks = 1
innodb_buffer_pool_instances = 4
innodb_buffer_pool_size = 11G
innodb_buffer_pool_populate = 1
innodb_file_format = Barracuda
innodb_thread_concurrency = 0
innodb_log_file_size = 64M
innodb_io_capacity = 300
innodb_read_io_threads = 32
innodb_write_io_threads = 32
innodb_flush_neighbor_pages = area
innodb_open_files = 600
thread_handling = pool-of-threads
thread_pool_size = 8
thread_pool_stall_limit = 500
thread_pool_max_threads = 500
thread_pool_idle_timeout = 60
extra_port = 63306
extra_max_connections = 5
tmpdir = /tmp
tmp_table_size = 32M
max_heap_table_size = 32M
query_cache_type = 0
query_cache_size = 0
max_connections = 300
thread_cache_size = 50
open_files_limit = 65535
table_definition_cache = 4096
table_open_cache = 16384
binlog_format = ROW
max_binlog_size = 100M
expire_logs_days = 1
log-bin = /var/log/mysql/mariadb-bin.log
slow_query_log = 1
long_query_time = 2
slow_query_log_file = /var/log/mysql/mariadb-slow-queries.log
general_log = 0
log-error = /var/log/mysql/mariadb-error.log
log_slow_verbosity = Query_plan
plugin-load = handlersocket.so
loose_handlersocket_port = 9998
loose_handlersocket_port_wr = 9999
loose_handlersocket_threads = 16
loose_handlersocket_threads_wr = 1
{"report":{"fcp":1157.5999999940395,"ttfb":190.19999998807907,"pageVisibility":"visible","entityId":30932,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"4c7206b4-d9d5-44d5-820e-b3c076c507d9","navigationType":0,"readyForUser":1241.5999999940395,"redirectCount":0,"resourceLoadedEnd":2103.800000011921,"resourceLoadedStart":195.19999998807907,"resourceTiming":[{"duration":258.2000000178814,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":195.19999998807907,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":195.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":453.40000000596046,"responseStart":0,"secureConnectionStart":0},{"duration":258.09999999403954,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":195.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":195.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":453.59999999403954,"responseStart":0,"secureConnectionStart":0},{"duration":514.5999999940395,"initiatorType":"script","name":"https://jira.mariadb.org/s/e9b27a47da5fb0f74a35acd57e9847fb-CDN/lu2bv2/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":195.59999999403954,"connectEnd":195.59999999403954,"connectStart":195.59999999403954,"domainLookupEnd":195.59999999403954,"domainLookupStart":195.59999999403954,"fetchStart":195.59999999403954,"redirectEnd":0,"redirectStart":0,"requestStart":195.59999999403954,"responseEnd":710.1999999880791,"responseStart":710.1999999880791,"secureConnectionStart":195.59999999403954},{"duration":603.7999999821186,"initiatorType":"script","name":"https://jira.mariadb.org/s/c32eb0da7ad9831253f8397e6cc26afd-CDN/lu2bv2/820016/12ta74/2380add21a9a1006587582385952de73/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":195.80000001192093,"connectEnd":195.80000001192093,"connectStart":195.80000001192093,"domainLookupEnd":195.80000001192093,"domainLookupStart":195.80000001192093,"fetchStart":195.80000001192093,"redirectEnd":0,"redirectStart":0,"requestStart":195.80000001192093,"responseEnd":799.5999999940395,"responseStart":799.5,"secureConnectionStart":195.80000001192093},{"duration":607.3000000119209,"initiatorType":"script","name":"https://jira.mariadb.org/s/bc0bcb146314416123c992714ee00ff7-CDN/lu2bv2/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":196,"connectEnd":196,"connectStart":196,"domainLookupEnd":196,"domainLookupStart":196,"fetchStart":196,"redirectEnd":0,"redirectStart":0,"requestStart":196,"responseEnd":803.3000000119209,"responseStart":803.3000000119209,"secureConnectionStart":196},{"duration":607.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":196.19999998807907,"connectEnd":196.19999998807907,"connectStart":196.19999998807907,"domainLookupEnd":196.19999998807907,"domainLookupStart":196.19999998807907,"fetchStart":196.19999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":196.19999998807907,"responseEnd":803.6999999880791,"responseStart":803.6999999880791,"secureConnectionStart":196.19999998807907},{"duration":607.5999999940395,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":196.40000000596046,"connectEnd":196.40000000596046,"connectStart":196.40000000596046,"domainLookupEnd":196.40000000596046,"domainLookupStart":196.40000000596046,"fetchStart":196.40000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":196.40000000596046,"responseEnd":804,"responseStart":804,"secureConnectionStart":196.40000000596046},{"duration":653.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bv2/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":196.59999999403954,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":196.59999999403954,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":850.0999999940395,"responseStart":0,"secureConnectionStart":0},{"duration":607.8000000119209,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":196.69999998807907,"connectEnd":196.69999998807907,"connectStart":196.69999998807907,"domainLookupEnd":196.69999998807907,"domainLookupStart":196.69999998807907,"fetchStart":196.69999998807907,"redirectEnd":0,"redirectStart":0,"requestStart":196.69999998807907,"responseEnd":804.5,"responseStart":804.5,"secureConnectionStart":196.69999998807907},{"duration":653.1999999880791,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":196.90000000596046,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":196.90000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":850.0999999940395,"responseStart":0,"secureConnectionStart":0},{"duration":607.9000000059605,"initiatorType":"script","name":"https://jira.mariadb.org/s/719848dd97ebe0663199f49a3936487a-CDN/lu2bv2/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":197.09999999403954,"connectEnd":197.09999999403954,"connectStart":197.09999999403954,"domainLookupEnd":197.09999999403954,"domainLookupStart":197.09999999403954,"fetchStart":197.09999999403954,"redirectEnd":0,"redirectStart":0,"requestStart":197.09999999403954,"responseEnd":805,"responseStart":805,"secureConnectionStart":197.09999999403954},{"duration":1123.2999999821186,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":197.90000000596046,"connectEnd":197.90000000596046,"connectStart":197.90000000596046,"domainLookupEnd":197.90000000596046,"domainLookupStart":197.90000000596046,"fetchStart":197.90000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":197.90000000596046,"responseEnd":1321.199999988079,"responseStart":1321.199999988079,"secureConnectionStart":197.90000000596046},{"duration":1905.9000000059605,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bv2/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":197.90000000596046,"connectEnd":197.90000000596046,"connectStart":197.90000000596046,"domainLookupEnd":197.90000000596046,"domainLookupStart":197.90000000596046,"fetchStart":197.90000000596046,"redirectEnd":0,"redirectStart":0,"requestStart":197.90000000596046,"responseEnd":2103.800000011921,"responseStart":2103.800000011921,"secureConnectionStart":197.90000000596046},{"duration":469.40000000596046,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":861.5999999940395,"connectEnd":861.5999999940395,"connectStart":861.5999999940395,"domainLookupEnd":861.5999999940395,"domainLookupStart":861.5999999940395,"fetchStart":861.5999999940395,"redirectEnd":0,"redirectStart":0,"requestStart":861.5999999940395,"responseEnd":1331,"responseStart":1331,"secureConnectionStart":861.5999999940395},{"duration":954.7999999821186,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1151.800000011921,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1151.800000011921,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":2106.5999999940395,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":31,"responseStart":190,"responseEnd":191,"domLoading":193,"domInteractive":2122,"domContentLoadedEventStart":2122,"domContentLoadedEventEnd":2161,"domComplete":3744,"loadEventStart":3744,"loadEventEnd":3746,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":2107.5},{"name":"bigPipe.sidebar-id.end","time":2108.4000000059605},{"name":"bigPipe.activity-panel-pipe-id.start","time":2108.5},{"name":"bigPipe.activity-panel-pipe-id.end","time":2110.800000011921},{"name":"activityTabFullyLoaded","time":2166.699999988079}],"measures":[],"correlationId":"7c839d5f8b36fc","effectiveType":"4g","downlink":10,"rtt":0,"serverDuration":105,"dbReadsTimeInMs":7,"dbConnsTimeInMs":15,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}