Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-17458

Unable to Start Galera Node

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Fixed
    • 10.3.9, 10.3.11, 10.3.14, 10.3.15
    • 10.3.16, 10.4.6
    • Galera
    • None
    • CentOS 7, running on XenServer hypervisor.

    Description

      Servers went down due to failed query. I bootstrapped cluster and was unable to start the remaining nodes with the following error:

      2018-10-12 20:12:15 0 [Warning] WSREP: SST position can't be set in past. Requested: 92172, Current: 3620686.
      2018-10-12 20:12:15 0 [Warning] WSREP: Can't continue.
      2018-10-12 20:12:15 0 [ERROR] Aborting

      It says the current commit is 3620686, but it was actually 9953033. Not sure why this matters since the cluster was restarted and that number restarted.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33540 mariabackup --prepare: [ERROR] InnoDB: Crash recovery is broken due to insufficient innodb_log_file_size

          • Major - Major loss of function.
          • Closed
          is caused by

          Task - A task that needs to be done. MDEV-15158 On commit, do not write to the TRX_SYS page

          • Major - Major loss of function.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-19193 The --wsrep-new-cluster option does not reset wsrep_cluster_state_uuid and wsrep_last_committed

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-18177 Galera test failure on galera_autoinc_sst_mariabackup

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 7 older comments
            GeoffMontee Geoff Montee (Inactive) added a comment -

            chanouha,

            I think this is a bug, --wsrep-new-cluster should start in a clean new cluster state. As i deleted all data in joiner nodes, sync should have been done successfully.

            I submitted a specific bug for that here: MDEV-19193

            GeoffMontee Geoff Montee (Inactive) added a comment - chanouha , I think this is a bug, --wsrep-new-cluster should start in a clean new cluster state. As i deleted all data in joiner nodes, sync should have been done successfully. I submitted a specific bug for that here: MDEV-19193
            jplindst Jan Lindström (Inactive) added a comment -

            https://github.com/MariaDB/server/pull/1305

            jplindst Jan Lindström (Inactive) added a comment - https://github.com/MariaDB/server/pull/1305
            marko Marko Mäkelä added a comment -

            Please request another review after addressing my comments.

            marko Marko Mäkelä added a comment - Please request another review after addressing my comments.
            marko Marko Mäkelä added a comment -

            The upgrade was broken due to MDEV-15158, which was introduced in MariaDB 10.3.5.

            marko Marko Mäkelä added a comment - The upgrade was broken due to MDEV-15158 , which was introduced in MariaDB 10.3.5.
            dalmeida Daniel Almeida (Inactive) added a comment -

            Thanks sir, appreciate the input. I think that's sounds about correct, related to mariabackup, as I was also able to duplicate this even doing a manual sst.
            I noticed the coords were wrong and usually off between 1 or 4 positions. Here's what I did to fix it:

            list of errors with wrong coords:

             grep "WSREP: SST position can't be set in past. Requested" /var/log/mysql/error.log
            		 
            2021-02-02 21:30:06 140122607290560 [Warning] WSREP: SST position can't be set in past. Requested: 688039012, Current:  688039013.
            		 
            2021-02-03 10:18:19 139797349947584 [Warning] WSREP: SST position can't be set in past. Requested: 688264699, Current:  688264701.
            		 
            2021-02-03 11:08:51 140313816185024 [Warning] WSREP: SST position can't be set in past. Requested: 688297756, Current:  688297759.
            		 
            2021-02-03 13:16:02 140193343183040 [Warning] WSREP: SST position can't be set in past. Requested: 688370258, Current:  688370259.
            		 
            2021-02-03 14:26:35 140649485912256 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current:  688408040.
            		 
            2021-02-03 14:59:56 140348690200768 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current:  688408040.

            Fix:

            1 remove all data from bad node
            2 do streaming backup from good node to bad node:

            mariabackup --backup --parallel=4 --tmpdir=/tmp --stream=mbstream --galera-info --datadir=/var/lib/mysql --user=$user --password=$pass | pigz -c --fast | nc -w 2 badNode 4444

            4 prepare, change perms on bad node
            5 get the info from here on bad node

            cat /var/lib/mysql/xtrabackup_galera_info
            		 
            bd34378f-525c-11e5-92d5-130282ff70e0:688408036

            6 edit on bad node /var/lib/mysql/grastate.dat and add the info from the above command

            # GALERA saved state
            		 
            version: 2.1
            		 
            uuid:    bd34378f-525c-11e5-92d5-130282ff70e0
            		 
            seqno:   688408036
            		 
            safe_to_bootstrap: 0

            7 change perms on that file so mysql owns it
            8 start mariadb again (this will fail again with the same error)

            2021-02-03 14:59:56 140348690200768 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current:  688408040.

            9 edit /var/lib/mysql/grastate.dat again, now replace the seqno with the seqno from the error message above
            from 688408036 to 688408040 and start mariadb again, i.e.

            # GALERA saved state
            		 
            version: 2.1
            		 
            uuid:    bd34378f-525c-11e5-92d5-130282ff70e0
            		 
            seqno:   688408040
            		 
            safe_to_bootstrap: 0

            dalmeida Daniel Almeida (Inactive) added a comment - Thanks sir, appreciate the input. I think that's sounds about correct, related to mariabackup, as I was also able to duplicate this even doing a manual sst. I noticed the coords were wrong and usually off between 1 or 4 positions. Here's what I did to fix it: list of errors with wrong coords: grep "WSREP: SST position can't be set in past. Requested" /var/log/mysql/error.log 2021-02-02 21:30:06 140122607290560 [Warning] WSREP: SST position can't be set in past. Requested: 688039012, Current: 688039013. 2021-02-03 10:18:19 139797349947584 [Warning] WSREP: SST position can't be set in past. Requested: 688264699, Current: 688264701. 2021-02-03 11:08:51 140313816185024 [Warning] WSREP: SST position can't be set in past. Requested: 688297756, Current: 688297759. 2021-02-03 13:16:02 140193343183040 [Warning] WSREP: SST position can't be set in past. Requested: 688370258, Current: 688370259. 2021-02-03 14:26:35 140649485912256 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current: 688408040. 2021-02-03 14:59:56 140348690200768 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current: 688408040. Fix: 1 remove all data from bad node 2 do streaming backup from good node to bad node: mariabackup --backup --parallel=4 --tmpdir=/tmp --stream=mbstream --galera-info --datadir=/var/lib/mysql --user=$user --password=$pass | pigz -c --fast | nc -w 2 badNode 4444 4 prepare, change perms on bad node 5 get the info from here on bad node cat /var/lib/mysql/xtrabackup_galera_info bd34378f-525c-11e5-92d5-130282ff70e0:688408036 6 edit on bad node /var/lib/mysql/grastate.dat and add the info from the above command # GALERA saved state version: 2.1 uuid: bd34378f-525c-11e5-92d5-130282ff70e0 seqno: 688408036 safe_to_bootstrap: 0 7 change perms on that file so mysql owns it 8 start mariadb again (this will fail again with the same error) 2021-02-03 14:59:56 140348690200768 [Warning] WSREP: SST position can't be set in past. Requested: 688408036, Current: 688408040. 9 edit /var/lib/mysql/grastate.dat again, now replace the seqno with the seqno from the error message above from 688408036 to 688408040 and start mariadb again, i.e. # GALERA saved state version: 2.1 uuid: bd34378f-525c-11e5-92d5-130282ff70e0 seqno: 688408040 safe_to_bootstrap: 0

            People

              sciascid Daniele Sciascia
              obissick Oren Bissick (Inactive)
              Votes:
              3 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.