Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-23647

Garbd can't initiate SST anymore in 10.5

Details

    Description

      The galera arbitrator binary garbd can be used to trigger any actual cluster node to act as a donor towards the arbitrator using the -sst and -donor options.

      Up to MariaDB 10.4 this works fine, but with 10.5 the SST request does not lead to the actual requested wsrep_sst_... script being executed anymore.

      The donor node will just log:

      Aug 21 15:10:27 node-1 mariadbd[2902]: 2020-08-21 15:10:27 0 [Note] WSREP: Member 0.0 (garbd) requested state transfer from 'node-1'. Selected 1.0 (node-1)(SYNCED) as donor.
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: 2020-08-21 15:10:27 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 3)
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: 2020-08-21 15:10:27 1 [Note] WSREP: ================================================
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: View:
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   id: a0e49d83-e3bd-11ea-905f-ff2c5b67f9c2:3
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   status: primary
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   protocol_version: 4
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   final: no
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   own_index: 1
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:   members(3):
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:         0: 71303bfe-e3c0-11ea-b367-d2112cd7bec7, garbd
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:         1: a0e2712c-e3bd-11ea-9f04-db21411dccbc, node-1
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]:         2: d2324935-e3bd-11ea-a3b6-da0d74e66e47, node-2
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: =================================================
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: 2020-08-21 15:10:27 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      		 
      Aug 21 15:10:27 node-1 mariadbd[2902]: 2020-08-21 15:10:27 0 [Note] WSREP: 0.0 (garbd): State transfer from 1.0 (node-1) complete.

      With 10.4 it shows the actual wsrep script being executed, but nothing like this is to be seen in the 10.5 donor log:

      Aug 21 15:35:07 node-1 mysqld[2778]: 2020-08-21 15:35:07 0 [Note] WSREP: Running: 'wsrep_sst_backup --role 'donor' --address '' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --gtid 'b21e7239-e3c1-11ea-a1cb-f36fe61c8c1c:0' --gtid-domain-id '0' --binlog 'node-1-bin' --mysqld-args --wsrep-new-cluster --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1'

      And on the garbd side: 

      2020-08-21 15:10:27.576  INFO: Shifting OPEN -> PRIMARY (TO: 3)
      		 
      2020-08-21 15:10:27.577  INFO: Sending state transfer request: 'backup', size: 6
      		 
      2020-08-21 15:10:27.578  INFO: Member 0.0 (garbd) requested state transfer from 'node-1'. Selected 1.0 (node-1)(SYNCED) as donor.
      		 
      2020-08-21 15:10:27.578  INFO: Shifting PRIMARY -> JOINER (TO: 3)
      		 
      2020-08-21 15:10:27.578  INFO: Closing send monitor...
      		 
      2020-08-21 15:10:27.578  INFO: Closed send monitor.
      		 
      2020-08-21 15:10:27.578  INFO: gcomm: terminating thread
      		 
      2020-08-21 15:10:27.578  INFO: gcomm: joining thread
      		 
      2020-08-21 15:10:27.579  INFO: gcomm: closing backend
      		 
      2020-08-21 15:10:27.580  INFO: 0.0 (garbd): State transfer from 1.0 (node-1) complete.
      		 
      2020-08-21 15:10:27.580  INFO: Shifting JOINER -> JOINED (TO: 3)

      Attachments

        Activity

          Ascending order - Click to sort in descending order
          hholzgra Hartmut Holzgraefe created issue -
          elenst Elena Stepanova made changes -
          Field Original Value New Value
          Fix Version/s 10.5 [ 23123 ]
          Assignee Jan Lindström [ jplindst ]
          julien.fritsch Julien Fritsch made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          jplindst Jan Lindström (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Jan Lindström [ jplindst ] Mario Karuza [ mkaruza ]
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Mario Karuza [ mkaruza ] Jan Lindström [ jplindst ]
          jplindst Jan Lindström (Inactive) made changes -
          Status In Progress [ 3 ] Stalled [ 10000 ]
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Jan Lindström [ jplindst ] Mario Karuza [ mkaruza ]
          mkaruza Mario Karuza (Inactive) added a comment -

          Hartmut, you mentioned that this was working on 10.4. Which MariaDB & Galera version were used when this was working ?

          mkaruza Mario Karuza (Inactive) added a comment - Hartmut, you mentioned that this was working on 10.4. Which MariaDB & Galera version were used when this was working ?
          julien.fritsch Julien Fritsch made changes -
          Labels need_feedback
          mkaruza Mario Karuza (Inactive) made changes -
          Status Stalled [ 10000 ] In Progress [ 3 ]
          mkaruza Mario Karuza (Inactive) made changes -
          Assignee Mario Karuza [ mkaruza ] Jan Lindström [ jplindst ]
          jplindst Jan Lindström (Inactive) made changes -
          Priority Critical [ 2 ] Major [ 3 ]
          jplindst Jan Lindström (Inactive) made changes -
          Labels need_feedback
          jplindst Jan Lindström (Inactive) made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          jplindst Jan Lindström (Inactive) made changes -
          issue.field.resolutiondate 2021-01-21 07:45:34.0 2021-01-21 07:45:34.423
          jplindst Jan Lindström (Inactive) made changes -
          Fix Version/s 10.4.18 [ 25110 ]
          Fix Version/s 10.5.9 [ 25109 ]
          Fix Version/s 10.5 [ 23123 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 113145 ] MariaDB v4 [ 158330 ]
          mariadb-jira-automation Jira Automation (IT) made changes -
          Zendesk Related Tickets 119292

          People

            jplindst Jan Lindström (Inactive)
            hholzgra Hartmut Holzgraefe
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.