[MDEV-6481] Yum Upgrade on CentOS 6.5 causes instant crash of MariaDB/Galera Created: 2014-07-24  Updated: 2014-10-10  Resolved: 2014-10-10

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.0.12-galera
Fix Version/s: 10.0.14-galera

Type: Bug Priority: Minor
Reporter: Andy Repton Assignee: Nirbhay Choubey (Inactive)
Resolution: Fixed Votes: 0
Labels: galera
Environment:

CentOS 6.5 VM running on Linode environment


Attachments: Text File MariaDB-Crash-Orig.txt     Text File MariaDB-Crash.txt     File cluster.cnf     File mysql.log    

 Description   

Ran a yum upgrade, and following this yum hung. Opened a second shell, and found the attached error in the Mysql Log. Tried to restart with identical result: instant crash. Other nodes don't even see attempt to re-connect. Only way to fix it was to move my /var/lib/mysql directory and redo the SST.

Please let me know if I can provide any further info, I've tried to preserve as much as possible to help out with this.



 Comments   
Comment by Elena Stepanova [ 2014-07-24 ]

For a note: the log says "Server version: 10.0.11-MariaDB-wsrep", so apparently upgrade didn't work well enough.

Comment by Andy Repton [ 2014-07-24 ]

Oops, apologies. This was the second crash, after I attempted to roll back. Attached the original one.

Comment by Nirbhay Choubey (Inactive) [ 2014-07-25 ]

I couldn't reproduce it on CentoOS 6.5 32-bit, 10.0.11-galera -> 10.0.12-galera. Looking at both the attached files, it looks like crash occurred for both the versions during plugin initialization. Just to make myself clear : the node started fine after you removed the datadir for SST? Can you share the complete server error log and configurations?

Comment by Andy Repton [ 2014-07-26 ]

Sorry for the delay here. The setup is a three node master-master mariadb cluster, all running CentOS 6.5. The process to break was:

1. yum update on one of the nodes
2. MariaDB crashed during yum update, freezing yum.
3. Manually killed yum, tried to restart mariadb, instant crash
4. Ran yum remove 10.0.12-galera, downloaded and re-installed 10.0.11
5. Tried to restart mariadb, instant crash
6. Moved grastate file into home directory, tried to restart mariadb, instant crash
7. Rsync'd entire /var/lib/mysql into home directory, rm -rf /var/lib/mysql/* and then restarted mariaDB, SST completed and runs fine.

I still have all the files. Have uploaded the entire mysql.log and copies of my mysql config.

Comment by Nirbhay Choubey (Inactive) [ 2014-08-09 ]

Tried on CentOS-6.5_64 this time, no luck so far.

Comment by Nirbhay Choubey (Inactive) [ 2014-08-09 ]

Do you still have the copy of datadir?

Comment by Andy Repton [ 2014-09-29 ]

Hi there,

Yes I do, and my entire cluster now refuses to start. I can start them individually, but as soon as I try and enable clustering it crashes in fiery glory. Understandably, I'm hesitant to upload my data directory to the internet, but I'm willing to share it privately with you to help try and pin this down. Can you email me? mariaDB AT falconsupport.com.

Thanks!

Andy

Comment by Nirbhay Choubey (Inactive) [ 2014-10-08 ]

Can you upload them to ftp.askmonty.org?

https://mariadb.com/kb/en/mariadb/development/debugging-mariadb/reporting-bugs/

Comment by Andy Repton [ 2014-10-08 ]

I'd prefer to remove the sensitive information first... which rather removes the point of me uploading it I guess. What are you looking for in particular, the data files or the database contents?

Comment by Nirbhay Choubey (Inactive) [ 2014-10-08 ]

I see. What is the SE for mysql.plugin table? MyISAM or InnoDB?
SHOW CREATE TABLE mysql.plugin;

Comment by Nirbhay Choubey (Inactive) [ 2014-10-08 ]

http://lists.askmonty.org/pipermail/commits/2014-October/006749.html

Comment by Jan Lindström (Inactive) [ 2014-10-10 ]

Ok to push.

Comment by Nirbhay Choubey (Inactive) [ 2014-10-10 ]

Pushed to maria-10.0-galera.

http://bazaar.launchpad.net/~maria-captains/maria/maria-10.0-galera/revision/3899

Generated at Thu Feb 08 07:12:14 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.