[MDEV-21806] galera.galera_partition MTR failed: failed to recover from DONOR state - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.4.12, 10.5.2, 10.2(EOL), 10.3(EOL)
Fix Version/s: 10.4.22, 10.5.13, 10.6.5
Component/s: Galera, Tests
Labels:
None
Environment:
TestTarball_2 rhel-8

Description

galera.galera_partition failed on CI: failed to recover from DONOR state.

stdio:

10.4.12-6 81415e046e0011dcdddb1fa70b1f72cf274083b7
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

200410_10.5e_galera_partition_logs.zip
242 kB
2020-04-10 17:11
200602_10.4e_mysqld.1.err
557 kB
2020-06-03 08:54
200602_galera_partition_10.4e.zip
135 kB
2020-06-03 08:54

Activity

Ascending order - Click to sort in descending order

Stepan Patryshev (Inactive) created issue - 2020-02-24 09:28

Stepan Patryshev (Inactive) made changes - 2020-02-24 09:28

Field	Original Value	New Value
Link		This issue blocks MDEV-17728 [ MDEV-17728 ]

Stepan Patryshev (Inactive) made changes - 2020-02-24 09:29

Priority

Major [ 3 ]

Critical [ 2 ]

Stepan Patryshev (Inactive) made changes - 2020-02-24 09:30

Assignee

Stepan Patryshev [ stepan.patryshev ]

Jan Lindström [ jplindst ]

Stepan Patryshev (Inactive) made changes - 2020-02-24 09:46

Description

galera.galera_partition failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=4735&view=logs&j=94f5bb79-c84f-5a90-82a1-c3eb3e61c2d8&t=7adbd942-163b-541b-1142-86f871f98ab5&l=580] failed to recover from DONOR state.

*[stdio.log|https://dev.azure.com/mariadbe/550599d3-6165-4abd-8c86-e3f7e53a1847/_apis/build/builds/4735/logs/338]*:
{code:title=10.4.12-6 commitABCD123}
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err
{code}

galera.galera_partition failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=4735&view=logs&j=94f5bb79-c84f-5a90-82a1-c3eb3e61c2d8&t=7adbd942-163b-541b-1142-86f871f98ab5&l=580] failed to recover from DONOR state.

*[stdio.log|https://dev.azure.com/mariadbe/550599d3-6165-4abd-8c86-e3f7e53a1847/_apis/build/builds/4735/logs/338]*:
{code:title=10.4.12-6 81415e046e0011dcdddb1fa70b1f72cf274083b7}
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err
{code}

Stepan Patryshev (Inactive) made changes - 2020-02-24 09:50

Link

This issue blocks MENT-565 [ MENT-565 ]

Stepan Patryshev (Inactive) made changes - 2020-04-06 11:20

Affects Version/s

10.3 [ 22126 ]

Stepan Patryshev (Inactive) made changes - 2020-04-06 11:21

Fix Version/s

10.3 [ 22126 ]

Stepan Patryshev (Inactive) added a comment - 2020-04-06 11:21 - edited

It failed also on bb, CS 10.3, kvm-deb-jessie-x86.

Stepan Patryshev (Inactive) added a comment - 2020-04-06 11:21 - edited It failed also on bb , CS 10.3, kvm-deb-jessie-x86.

Stepan Patryshev (Inactive) added a comment - 2020-04-07 12:08

It failed also on bb, CS 10.2, kvm-deb-stretch-amd64.

Stepan Patryshev (Inactive) added a comment - 2020-04-07 12:08 It failed also on bb , CS 10.2, kvm-deb-stretch-amd64.

Stepan Patryshev (Inactive) made changes - 2020-04-10 17:11

Attachment

200410_10.5e_galera_partition_logs.zip [ 51173 ]

Stepan Patryshev (Inactive) added a comment - 2020-04-10 17:11 - edited

It failed also on Jenkins, 10.5 ES, rhel-7, debug build: All logs.

Stepan Patryshev (Inactive) added a comment - 2020-04-10 17:11 - edited It failed also on Jenkins, 10.5 ES, rhel-7, debug build: All logs .

Stepan Patryshev (Inactive) made changes - 2020-04-10 17:12

Affects Version/s

10.5.2 [ 24030 ]

Stepan Patryshev (Inactive) made changes - 2020-04-10 17:12

Fix Version/s

10.5 [ 23123 ]

Stepan Patryshev (Inactive) added a comment - 2020-04-13 11:50 - edited

It failed also on CI, 10.2 ES, 2969d0702d56405d1aec8c16a272ac85fef7bd61, sles-15.

Stepan Patryshev (Inactive) added a comment - 2020-04-13 11:50 - edited It failed also on CI, 10.2 ES, 2969d0702d56405d1aec8c16a272ac85fef7bd61, sles-15.

Stepan Patryshev (Inactive) made changes - 2020-04-13 11:50

Affects Version/s

10.2 [ 14601 ]

Stepan Patryshev (Inactive) made changes - 2020-04-13 11:50

Fix Version/s

10.2 [ 14601 ]

Stepan Patryshev (Inactive) made changes - 2020-04-13 11:51

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Stepan Patryshev (Inactive) made changes - 2020-04-13 11:51

Link

This issue blocks MDEV-17728 [ MDEV-17728 ]

Stepan Patryshev (Inactive) made changes - 2020-04-15 09:33

Description

galera.galera_partition failed on [Azure|https://dev.azure.com/mariadbe/MariaDB%20Enterprise/_build/results?buildId=4735&view=logs&j=94f5bb79-c84f-5a90-82a1-c3eb3e61c2d8&t=7adbd942-163b-541b-1142-86f871f98ab5&l=580] failed to recover from DONOR state.

*[stdio.log|https://dev.azure.com/mariadbe/550599d3-6165-4abd-8c86-e3f7e53a1847/_apis/build/builds/4735/logs/338]*:
{code:title=10.4.12-6 81415e046e0011dcdddb1fa70b1f72cf274083b7}
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err
{code}

galera.galera_partition failed on CI: failed to recover from DONOR state.

*[stdio.log|https://dev.azure.com/mariadbe/550599d3-6165-4abd-8c86-e3f7e53a1847/_apis/build/builds/4735/logs/338]*:
{code:title=10.4.12-6 81415e046e0011dcdddb1fa70b1f72cf274083b7}
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err
{code}

Stepan Patryshev (Inactive) made changes - 2020-04-15 09:37

Description

galera.galera_partition failed on CI: failed to recover from DONOR state.

*stdio*:
{code:title=10.4.12-6 81415e046e0011dcdddb1fa70b1f72cf274083b7}
2020-02-23T03:52:25.8793361Z galera.galera_partition 'innodb' w2 [ fail ] Found warnings/errors in server log file!
2020-02-23T03:52:25.8795232Z Test ended at 2020-02-23 03:52:25
2020-02-23T03:52:25.8795970Z line
2020-02-23T03:52:25.8797106Z 2020-02-23 3:52:14 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.
2020-02-23T03:52:25.8798686Z 2020-02-23 3:52:14 0 [ERROR] WSREP: failed to recover from DONOR state: gcs_join(da568d9d-55ef-11ea-9450-1266ada12cdf:2) failed: 107 (Transport endpoint is not connected)
2020-02-23T03:52:25.8799999Z 2020-02-23 3:52:14 0 [Warning] Provider sst_sent() returned an error
2020-02-23T03:52:25.8800972Z ^ Found warnings in /var/tmp/mtr/2/log/mysqld.1.err
{code}

Stepan Patryshev (Inactive) made changes - 2020-06-03 08:54

Attachment		200602_galera_partition_10.4e.zip [ 52070 ]
Attachment		200602_10.4e_mysqld.1.err [ 52071 ]

Stepan Patryshev (Inactive) added a comment - 2020-06-03 08:54

It failed also on Jenkins, 10.4 ES.
mysqld.1.err:

10.4.13-7 ES, 528ff0121b1cb943a838b8e3f7fddf23fabb248e, RelWithDebInfo,rhel-7
2020-06-02 16:08:30 0 [Warning] WSREP: discarding established (time wait) 4bf3494a-94a1 (tcp://127.0.0.1:16033)
2020-06-02 16:08:30 0 [Warning] WSREP: Member 2.2 (jw-rhel-7-2h0dz0) requested state transfer from 'any', but it is impossible to select State Transfer donor: Resource temporarily unavailable
[...]
2020-06-02 16:08:37 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component.

Other logs.

Stepan Patryshev (Inactive) added a comment - 2020-06-03 08:54 It failed also on Jenkins, 10.4 ES. mysqld.1.err : 10.4.13-7 ES, 528ff0121b1cb943a838b8e3f7fddf23fabb248e, RelWithDebInfo,rhel-7 2020-06-02 16:08:30 0 [Warning] WSREP: discarding established (time wait) 4bf3494a-94a1 (tcp://127.0.0.1:16033) 2020-06-02 16:08:30 0 [Warning] WSREP: Member 2.2 (jw-rhel-7-2h0dz0) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable [...] 2020-06-02 16:08:37 0 [Warning] WSREP: Sending JOIN failed: -107 (Transport endpoint is not connected). Will retry in new primary component. Other logs .

Stepan Patryshev (Inactive) made changes - 2020-06-03 08:55

Assignee

Jan Lindström [ jplindst ]

Julius Goryavsky [ sysprg ]

Julius Goryavsky made changes - 2020-06-16 12:25

Status

Open [ 1 ]

In Progress [ 3 ]

Stepan Patryshev (Inactive) added a comment - 2021-02-12 14:51 - edited

It failed on BB, 10.2.
stdout:

10.2.37, c89f37983ec82e5c6140f098e5672fde7fbf1002, kvm-deb-xenial-amd64
galera.galera_partition 'innodb' w1 [ fail ]
Test ended at 2021-01-14 05:19:07

WSREP did not transition to state READY


Failed to start mysqld.3


- saving '/dev/shm/var/1/log/galera.galera_partition-innodb/' to '/dev/shm/var/log/galera.galera_partition-innodb/'

Retrying test galera.galera_partition, attempt(2/3)...

worker[1] > Restart - not started
worker[1] > Restart - not started
worker[1] > Restart - not started
worker[1] > Restart - not started

Stepan Patryshev (Inactive) added a comment - 2021-02-12 14:51 - edited It failed on BB, 10.2 . stdout : 10.2.37, c89f37983ec82e5c6140f098e5672fde7fbf1002, kvm-deb-xenial-amd64 galera.galera_partition 'innodb' w1 [ fail ] Test ended at 2021-01-14 05:19:07 WSREP did not transition to state READY Failed to start mysqld.3 - saving '/dev/shm/var/1/log/galera.galera_partition-innodb/' to '/dev/shm/var/log/galera.galera_partition-innodb/' Retrying test galera.galera_partition, attempt(2/3)... worker[1] > Restart - not started worker[1] > Restart - not started worker[1] > Restart - not started worker[1] > Restart - not started

Jan Lindström (Inactive) made changes - 2021-06-24 06:38

Assignee

Julius Goryavsky [ sysprg ]

Jan Lindström [ jplindst ]

Jan Lindström (Inactive) made changes - 2021-09-28 04:58

Fix Version/s		10.4.22 [ 26031 ]
Fix Version/s		10.5.13 [ 26026 ]
Fix Version/s		10.6.5 [ 26034 ]
Fix Version/s	10.2 [ 14601 ]
Fix Version/s	10.3 [ 22126 ]
Fix Version/s	10.4 [ 22408 ]
Fix Version/s	10.5 [ 23123 ]
Resolution		Fixed [ 1 ]
Status	In Progress [ 3 ]	Closed [ 6 ]

Julien Fritsch made changes - 2021-09-29 13:13

Link

This issue blocks ~~MDEV-19937~~ [ ~~MDEV-19937~~ ]

Julien Fritsch made changes - 2021-09-29 13:13

Link

This issue blocks MENT-565 [ MENT-565 ]

Sergei Golubchik made changes - 2021-12-06 21:51

Workflow

MariaDB v3 [ 104242 ]

MariaDB v4 [ 157369 ]

People

Assignee:: Jan Lindström (Inactive)

Reporter:: Stepan Patryshev (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 2020-02-24 09:28

Updated:: 2021-09-29 13:13

Resolved:: 2021-09-28 04:58

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Attachments

Activity

People

Dates

Git Integration