Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21514

Galera test failure on galera.galera_wan_restart_sst

Details

    Description

      galera.galera_wan_restart_sst failed on CI.

      stdout.log:

      10.4.11-5-MariaDB-enterprise d4f88b8, rhel-6

      2019-12-27T06:13:29.1489186Z galera.galera_wan_restart_sst 'innodb'   w1 [ fail ]
      2019-12-27T06:13:29.1491212Z         Test ended at 2019-12-27 01:13:29
      2019-12-27T06:13:29.1492133Z 
      2019-12-27T06:13:29.1492691Z WSREP did not transition to state READY
      2019-12-27T06:13:29.1493291Z 
      2019-12-27T06:13:29.1611758Z  - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
      2019-12-27T06:13:29.1951938Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
      2019-12-27T06:13:29.1953228Z 
      2019-12-27T06:13:29.1954337Z 2019-12-27  1:09:56 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
      2019-12-27T06:13:29.1955461Z 2019-12-27  1:09:56 0 [Warning] Provider sst_sent() returned an error
      2019-12-27T06:13:29.1956809Z 2019-12-27  1:10:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
      2019-12-27T06:13:29.1957852Z 2019-12-27  1:10:01 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
      2019-12-27T06:13:29.1959150Z 2019-12-27  1:10:01 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
      2019-12-27T06:13:29.1960380Z 2019-12-27  1:10:01 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
      

      Attachments

        Activity

          stepan.patryshev Stepan Patryshev (Inactive) created issue -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Field Original Value New Value
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Description galera.galera_wan_restart_sst failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=3799&view=logs&j=87c5ea8e-aa4a-594f-39d2-16e12171cf31&t=2291973c-a970-5c45-9d10-ac8ae5f0a944&l=463].

          Azure: MariaDBEnterprise - MTR, run #20191227.3, TestTarball_2 rhel-6 > MTR - MTR Galera -big test:
          *[stdout.log|^20191227.3_ttb2_rhel6.txt]*:
          {code:title=10.4.11-5-MariaDB-enterprise d4f88b8}
          2019-12-27T06:13:29.1489186Z galera.galera_wan_restart_sst 'innodb' w1 [ fail ]
          2019-12-27T06:13:29.1491212Z Test ended at 2019-12-27 01:13:29
          2019-12-27T06:13:29.1492133Z
          2019-12-27T06:13:29.1492691Z WSREP did not transition to state READY
          2019-12-27T06:13:29.1493291Z
          2019-12-27T06:13:29.1611758Z - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
          2019-12-27T06:13:29.1951938Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
          2019-12-27T06:13:29.1953228Z
          2019-12-27T06:13:29.1954337Z 2019-12-27 1:09:56 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
          2019-12-27T06:13:29.1955461Z 2019-12-27 1:09:56 0 [Warning] Provider sst_sent() returned an error
          2019-12-27T06:13:29.1956809Z 2019-12-27 1:10:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
          2019-12-27T06:13:29.1957852Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
          2019-12-27T06:13:29.1959150Z 2019-12-27 1:10:01 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
          2019-12-27T06:13:29.1960380Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
          {code}
          galera.galera_wan_restart_sst failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=3799&view=logs&j=87c5ea8e-aa4a-594f-39d2-16e12171cf31&t=2291973c-a970-5c45-9d10-ac8ae5f0a944&l=463]:
          MariaDBEnterprise - MTR, run #20191227.3, TestTarball_2 rhel-6 > MTR - MTR Galera -big test:
          *[stdout.log|^20191227.3_ttb2_rhel6.txt]*:
          {code:title=10.4.11-5-MariaDB-enterprise d4f88b8}
          2019-12-27T06:13:29.1489186Z galera.galera_wan_restart_sst 'innodb' w1 [ fail ]
          2019-12-27T06:13:29.1491212Z Test ended at 2019-12-27 01:13:29
          2019-12-27T06:13:29.1492133Z
          2019-12-27T06:13:29.1492691Z WSREP did not transition to state READY
          2019-12-27T06:13:29.1493291Z
          2019-12-27T06:13:29.1611758Z - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
          2019-12-27T06:13:29.1951938Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
          2019-12-27T06:13:29.1953228Z
          2019-12-27T06:13:29.1954337Z 2019-12-27 1:09:56 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
          2019-12-27T06:13:29.1955461Z 2019-12-27 1:09:56 0 [Warning] Provider sst_sent() returned an error
          2019-12-27T06:13:29.1956809Z 2019-12-27 1:10:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
          2019-12-27T06:13:29.1957852Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
          2019-12-27T06:13:29.1959150Z 2019-12-27 1:10:01 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
          2019-12-27T06:13:29.1960380Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
          {code}
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Assignee Stepan Patryshev [ stepan.patryshev ] Jan Lindström [ jplindst ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Priority Major [ 3 ] Critical [ 2 ]

          Increased the priority since it fails on Azure.

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - Increased the priority since it fails on Azure.
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Jan Lindström [ jplindst ] Seppo Jaakola [ seppo ]
          seppo Seppo Jaakola made changes -
          Assignee Seppo Jaakola [ seppo ] Teemu Ollakka [ teemu.ollakka ]
          jplindst Jan Lindström (Inactive) made changes -
          Component/s Galera [ 10124 ]
          Component/s Tests [ 10800 ]
          Component/s Galera [ 14918 ]
          Component/s Tests [ 14999 ]
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.4 [ 23604 ]
          Key MENT-589 MDEV-21514
          Affects Version/s 10.4.11-5 [ 23927 ]
          Project MariaDB Enterprise [ 11500 ] MariaDB Server [ 10000 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Labels disabled
          jplindst Jan Lindström (Inactive) made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          jplindst Jan Lindström (Inactive) made changes -
          Fix Version/s 10.4.13 [ 24223 ]
          Fix Version/s 10.4 [ 22408 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Teemu Ollakka [ teemu.ollakka ] Jan Lindström [ jplindst ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Labels disabled
          julien.fritsch Julien Fritsch made changes -
          Labels Test_disabled
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Labels Test_disabled

          Reopened since it failed on CI again.

          stdio.log:

          10.4.12-6 86a634a0feaf7788c9bcf7cc763e500d2be97d75

          2020-03-11T06:03:53.7186772Z galera.galera_wan_restart_sst 'innodb'   w1 [ fail ]
          2020-03-11T06:03:53.7189830Z         Test ended at 2020-03-11 06:03:53
          2020-03-11T06:03:53.7190561Z 
          2020-03-11T06:03:53.7191172Z WSREP did not transition to state READY
          2020-03-11T06:03:53.7191915Z 
          2020-03-11T06:03:53.7297100Z  - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
          2020-03-11T06:03:53.7634124Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
          2020-03-11T06:03:53.7634915Z 
          2020-03-11T06:03:53.7636027Z 2020-03-11  6:00:25 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
          2020-03-11T06:03:53.7637500Z 2020-03-11  6:00:25 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
          2020-03-11T06:03:53.7638835Z 2020-03-11  6:00:25 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
          2020-03-11T06:03:53.7640796Z 2020-03-11  6:00:25 1 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(9258864b-635d-11ea-a5dc-9b4ff9f9667c:3) failed: 103 (Software caused connection abort)
          

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited Reopened since it failed on CI again. stdio.log : 10.4.12-6 86a634a0feaf7788c9bcf7cc763e500d2be97d75 2020-03-11T06:03:53.7186772Z galera.galera_wan_restart_sst 'innodb' w1 [ fail ] 2020-03-11T06:03:53.7189830Z Test ended at 2020-03-11 06:03:53 2020-03-11T06:03:53.7190561Z 2020-03-11T06:03:53.7191172Z WSREP did not transition to state READY 2020-03-11T06:03:53.7191915Z 2020-03-11T06:03:53.7297100Z - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/' 2020-03-11T06:03:53.7634124Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst 2020-03-11T06:03:53.7634915Z 2020-03-11T06:03:53.7636027Z 2020-03-11 6:00:25 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1 2020-03-11T06:03:53.7637500Z 2020-03-11 6:00:25 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) 2020-03-11T06:03:53.7638835Z 2020-03-11 6:00:25 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component. 2020-03-11T06:03:53.7640796Z 2020-03-11 6:00:25 1 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(9258864b-635d-11ea-a5dc-9b4ff9f9667c:3) failed: 103 (Software caused connection abort)
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Stalled [ 10000 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.4.13 [ 24223 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -

          It failed also on Azure, 10.3.

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited It failed also on Azure, 10.3 .
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Fix Version/s 10.3 [ 22126 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Affects Version/s 10.4.11 [ 24013 ]
          Affects Version/s 10.3.22 [ 24018 ]

          It failed also on Azure, 10.2

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited It failed also on Azure, 10.2
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Fix Version/s 10.2 [ 14601 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Affects Version/s 10.2.31 [ 24017 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Description galera.galera_wan_restart_sst failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=3799&view=logs&j=87c5ea8e-aa4a-594f-39d2-16e12171cf31&t=2291973c-a970-5c45-9d10-ac8ae5f0a944&l=463]:
          MariaDBEnterprise - MTR, run #20191227.3, TestTarball_2 rhel-6 > MTR - MTR Galera -big test:
          *[stdout.log|^20191227.3_ttb2_rhel6.txt]*:
          {code:title=10.4.11-5-MariaDB-enterprise d4f88b8}
          2019-12-27T06:13:29.1489186Z galera.galera_wan_restart_sst 'innodb' w1 [ fail ]
          2019-12-27T06:13:29.1491212Z Test ended at 2019-12-27 01:13:29
          2019-12-27T06:13:29.1492133Z
          2019-12-27T06:13:29.1492691Z WSREP did not transition to state READY
          2019-12-27T06:13:29.1493291Z
          2019-12-27T06:13:29.1611758Z - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
          2019-12-27T06:13:29.1951938Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
          2019-12-27T06:13:29.1953228Z
          2019-12-27T06:13:29.1954337Z 2019-12-27 1:09:56 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
          2019-12-27T06:13:29.1955461Z 2019-12-27 1:09:56 0 [Warning] Provider sst_sent() returned an error
          2019-12-27T06:13:29.1956809Z 2019-12-27 1:10:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
          2019-12-27T06:13:29.1957852Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
          2019-12-27T06:13:29.1959150Z 2019-12-27 1:10:01 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
          2019-12-27T06:13:29.1960380Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
          {code}
          galera.galera_wan_restart_sst failed on CI.

          *[stdout.log|^20191227.3_ttb2_rhel6.txt]*:
          {code:title=10.4.11-5-MariaDB-enterprise d4f88b8, rhel-6}
          2019-12-27T06:13:29.1489186Z galera.galera_wan_restart_sst 'innodb' w1 [ fail ]
          2019-12-27T06:13:29.1491212Z Test ended at 2019-12-27 01:13:29
          2019-12-27T06:13:29.1492133Z
          2019-12-27T06:13:29.1492691Z WSREP did not transition to state READY
          2019-12-27T06:13:29.1493291Z
          2019-12-27T06:13:29.1611758Z - saving '/var/tmp/mtr/1/log/galera.galera_wan_restart_sst-innodb/' to '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
          2019-12-27T06:13:29.1951938Z ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
          2019-12-27T06:13:29.1953228Z
          2019-12-27T06:13:29.1954337Z 2019-12-27 1:09:56 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
          2019-12-27T06:13:29.1955461Z 2019-12-27 1:09:56 0 [Warning] Provider sst_sent() returned an error
          2019-12-27T06:13:29.1956809Z 2019-12-27 1:10:01 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
          2019-12-27T06:13:29.1957852Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
          2019-12-27T06:13:29.1959150Z 2019-12-27 1:10:01 1 [Warning] WSREP: Sending JOIN failed: -103 (Software caused connection abort). Will retry in new primary component.
          2019-12-27T06:13:29.1960380Z 2019-12-27 1:10:01 1 [ERROR] WSREP: Failed to JOIN the cluster after SST
          {code}
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -

          It failed also on Jenkins, 10.5 ES, e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, ubuntu-1804.

          stepan.patryshev Stepan Patryshev (Inactive) added a comment - - edited It failed also on Jenkins, 10.5 ES, e7ca9ec6f6bd93883eeb35ebb2b09ffcbde35ad6, Build Debug, ubuntu-1804.
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Affects Version/s 10.5 [ 23123 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Fix Version/s 10.5 [ 23123 ]
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          Summary Galera test failure on galera.galera_wan_restart_sst on Azure Galera test failure on galera.galera_wan_restart_sst
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          stepan.patryshev Stepan Patryshev (Inactive) made changes -
          jplindst Jan Lindström (Inactive) made changes -
          Status Stalled [ 10000 ] In Progress [ 3 ]
          jplindst Jan Lindström (Inactive) added a comment - http://buildbot.askmonty.org/buildbot/grid?category=main&branch=bb-10.2-MDEV-21514 https://github.com/MariaDB/server/commit/250239a5b2d35b566befa0cb1a7c9608c10ecaa9
          jplindst Jan Lindström (Inactive) made changes -
          Assignee Jan Lindström [ jplindst ] Julius Goryavsky [ sysprg ]
          Status In Progress [ 3 ] In Review [ 10002 ]

          works fine, ok to push

          sysprg Julius Goryavsky added a comment - works fine, ok to push
          sysprg Julius Goryavsky made changes -
          Assignee Julius Goryavsky [ sysprg ] Jan Lindström [ jplindst ]
          Status In Review [ 10002 ] Stalled [ 10000 ]
          jplindst Jan Lindström (Inactive) made changes -
          Status Stalled [ 10000 ] In Progress [ 3 ]
          jplindst Jan Lindström (Inactive) made changes -
          issue.field.resolutiondate 2021-04-28 08:53:23.0 2021-04-28 08:53:23.778
          jplindst Jan Lindström (Inactive) made changes -
          Fix Version/s 10.2.38 [ 25207 ]
          Fix Version/s 10.3.29 [ 25206 ]
          Fix Version/s 10.4.19 [ 25205 ]
          Fix Version/s 10.5.10 [ 25204 ]
          Fix Version/s 10.2 [ 14601 ]
          Fix Version/s 10.3 [ 22126 ]
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.5 [ 23123 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]
          julien.fritsch Julien Fritsch made changes -
          julien.fritsch Julien Fritsch made changes -
          serg Sergei Golubchik made changes -
          Workflow MariaDB v3 [ 102761 ] MariaDB v4 [ 157161 ]
          tturenko Timofey Turenko made changes -

          People

            jplindst Jan Lindström (Inactive)
            stepan.patryshev Stepan Patryshev (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.