Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6481

Yum Upgrade on CentOS 6.5 causes instant crash of MariaDB/Galera

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Fixed
    • 10.0.12-galera
    • 10.0.14-galera
    • Galera
    • CentOS 6.5 VM running on Linode environment

    Description

      Ran a yum upgrade, and following this yum hung. Opened a second shell, and found the attached error in the Mysql Log. Tried to restart with identical result: instant crash. Other nodes don't even see attempt to re-connect. Only way to fix it was to move my /var/lib/mysql directory and redo the SST.

      Please let me know if I can provide any further info, I've tried to preserve as much as possible to help out with this.

      Attachments

        1. MariaDB-Crash.txt
          5 kB
        2. MariaDB-Crash-Orig.txt
          4 kB
        3. mysql.log
          252 kB
        4. cluster.cnf
          0.9 kB

        Activity

          Seth Karlo Andy Repton created issue -

          For a note: the log says "Server version: 10.0.11-MariaDB-wsrep", so apparently upgrade didn't work well enough.

          elenst Elena Stepanova added a comment - For a note: the log says "Server version: 10.0.11-MariaDB-wsrep", so apparently upgrade didn't work well enough.
          elenst Elena Stepanova made changes -
          Field Original Value New Value
          Assignee Nirbhay Choubey [ nirbhay_c ]
          Seth Karlo Andy Repton added a comment -

          Oops, apologies. This was the second crash, after I attempted to roll back. Attached the original one.

          Seth Karlo Andy Repton added a comment - Oops, apologies. This was the second crash, after I attempted to roll back. Attached the original one.
          Seth Karlo Andy Repton made changes -
          Attachment MariaDB-Crash-Orig.txt [ 32603 ]

          I couldn't reproduce it on CentoOS 6.5 32-bit, 10.0.11-galera -> 10.0.12-galera. Looking at both the attached files, it looks like crash occurred for both the versions during plugin initialization. Just to make myself clear : the node started fine after you removed the datadir for SST? Can you share the complete server error log and configurations?

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - I couldn't reproduce it on CentoOS 6.5 32-bit, 10.0.11-galera -> 10.0.12-galera. Looking at both the attached files, it looks like crash occurred for both the versions during plugin initialization. Just to make myself clear : the node started fine after you removed the datadir for SST? Can you share the complete server error log and configurations?
          Seth Karlo Andy Repton added a comment -

          Sorry for the delay here. The setup is a three node master-master mariadb cluster, all running CentOS 6.5. The process to break was:

          1. yum update on one of the nodes
          2. MariaDB crashed during yum update, freezing yum.
          3. Manually killed yum, tried to restart mariadb, instant crash
          4. Ran yum remove 10.0.12-galera, downloaded and re-installed 10.0.11
          5. Tried to restart mariadb, instant crash
          6. Moved grastate file into home directory, tried to restart mariadb, instant crash
          7. Rsync'd entire /var/lib/mysql into home directory, rm -rf /var/lib/mysql/* and then restarted mariaDB, SST completed and runs fine.

          I still have all the files. Have uploaded the entire mysql.log and copies of my mysql config.

          Seth Karlo Andy Repton added a comment - Sorry for the delay here. The setup is a three node master-master mariadb cluster, all running CentOS 6.5. The process to break was: 1. yum update on one of the nodes 2. MariaDB crashed during yum update, freezing yum. 3. Manually killed yum, tried to restart mariadb, instant crash 4. Ran yum remove 10.0.12-galera, downloaded and re-installed 10.0.11 5. Tried to restart mariadb, instant crash 6. Moved grastate file into home directory, tried to restart mariadb, instant crash 7. Rsync'd entire /var/lib/mysql into home directory, rm -rf /var/lib/mysql/* and then restarted mariaDB, SST completed and runs fine. I still have all the files. Have uploaded the entire mysql.log and copies of my mysql config.
          Seth Karlo Andy Repton made changes -
          Attachment mysql.log [ 32800 ]
          Seth Karlo Andy Repton made changes -
          Attachment cluster.cnf [ 32801 ]
          nirbhay_c Nirbhay Choubey (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]

          Tried on CentOS-6.5_64 this time, no luck so far.

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Tried on CentOS-6.5_64 this time, no luck so far.

          Do you still have the copy of datadir?

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Do you still have the copy of datadir?
          Seth Karlo Andy Repton added a comment -

          Hi there,

          Yes I do, and my entire cluster now refuses to start. I can start them individually, but as soon as I try and enable clustering it crashes in fiery glory. Understandably, I'm hesitant to upload my data directory to the internet, but I'm willing to share it privately with you to help try and pin this down. Can you email me? mariaDB AT falconsupport.com.

          Thanks!

          Andy

          Seth Karlo Andy Repton added a comment - Hi there, Yes I do, and my entire cluster now refuses to start. I can start them individually, but as soon as I try and enable clustering it crashes in fiery glory. Understandably, I'm hesitant to upload my data directory to the internet, but I'm willing to share it privately with you to help try and pin this down. Can you email me? mariaDB AT falconsupport.com. Thanks! Andy
          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Can you upload them to ftp.askmonty.org? https://mariadb.com/kb/en/mariadb/development/debugging-mariadb/reporting-bugs/
          Seth Karlo Andy Repton added a comment -

          I'd prefer to remove the sensitive information first... which rather removes the point of me uploading it I guess. What are you looking for in particular, the data files or the database contents?

          Seth Karlo Andy Repton added a comment - I'd prefer to remove the sensitive information first... which rather removes the point of me uploading it I guess. What are you looking for in particular, the data files or the database contents?

          I see. What is the SE for mysql.plugin table? MyISAM or InnoDB?
          SHOW CREATE TABLE mysql.plugin;

          nirbhay_c Nirbhay Choubey (Inactive) added a comment - I see. What is the SE for mysql.plugin table? MyISAM or InnoDB? SHOW CREATE TABLE mysql.plugin;
          nirbhay_c Nirbhay Choubey (Inactive) added a comment - http://lists.askmonty.org/pipermail/commits/2014-October/006749.html
          nirbhay_c Nirbhay Choubey (Inactive) made changes -
          Assignee Nirbhay Choubey [ nirbhay_c ] Jan Lindström [ jplindst ]
          Status In Progress [ 3 ] In Review [ 10002 ]

          Ok to push.

          jplindst Jan Lindström (Inactive) added a comment - Ok to push.
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Jan Lindström [ jplindst ] Nirbhay Choubey [ nirbhay_c ]
          Status In Review [ 10002 ] Stalled [ 10000 ]
          nirbhay_c Nirbhay Choubey (Inactive) added a comment - Pushed to maria-10.0-galera. http://bazaar.launchpad.net/~maria-captains/maria/maria-10.0-galera/revision/3899
          nirbhay_c Nirbhay Choubey (Inactive) made changes -
          Component/s Galera [ 10124 ]
          Fix Version/s 10.0.14-galera [ 17000 ]
          Resolution Fixed [ 1 ]
          Status Stalled [ 10000 ] Closed [ 6 ]
          ratzpo Rasmus Johansson (Inactive) made changes -
          Workflow MariaDB v2 [ 50709 ] MariaDB v3 [ 64896 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 64896 ] MariaDB v4 [ 148059 ]

          People

            nirbhay_c Nirbhay Choubey (Inactive)
            Seth Karlo Andy Repton
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.