[MDEV-14394] Nodes can't join with socat Address already in use - Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Critical
Resolution: Fixed
Affects Version/s: 10.1.28
Fix Version/s: 10.1.30
Component/s: Galera SST
Labels:
None
Environment:
Ubuntu Xenial

Sprint:
10.1.30

Description

Hi,
Experiencing the same problem as ~~MDEV-10442~~, nodes can't join on Ubuntu Xenial.

mysqld stops with:

2017-11-14 16:11:29 140304279660288 [Note] WSREP: Member 0.0 (Galera_Node_02) requested state transfer from '*any*'. Selected 1.0 (Galera_Node_01)(SYNCED) as donor.

2017-11-14 16:11:29 140304279660288 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 786790589)

2017-11-14 16:11:29 140322124806912 [Note] WSREP: Requesting state transfer: success, donor: 1

2017-11-14 16:11:29 140322124806912 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(709f570f-c7d5-11e7-b709-e25129d5bf65:13)

WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20171114 16:11:29.639)

2017/11/14 16:11:29 socat[26364] E bind(6, {AF=2 0.0.0.0:4444}, 16): Address already in use

WSREP_SST: [ERROR] Error while getting data from donor node:  exit codes: 1 0 (20171114 16:11:29.645)

WSREP_SST: [ERROR] Cleanup after exit with status:32 (20171114 16:11:29.647)

2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.30.2.88' --datadir '/mnt/data/mysql/'   --parent '26044' --binlog 'mysqld-bin' : 32 (Broken pipe)

2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.

2017-11-14 16:11:29 140322125121792 [ERROR] WSREP: SST failed: 32 (Broken pipe)

2017-11-14 16:11:29 140322125121792 [ERROR] Aborting

and socat, xbstream, wsrep_sst_xtrabackup-v2 keep running, parenthood inherited by 'init'.
As soon as you kill all the sst processes systemd restarts mysqld and the problems is back, mysqld stops, sst processes keep running.

10.1.28 seems not to contain the fix from ~~MDEV-10442~~:

In sql/wsrep_utils.cc:

    err_ = posix_spawnattr_setflags (&attr, POSIX_SPAWN_SETSIGDEF  |

                                            POSIX_SPAWN_SETSIGMASK |

            /* start a new process group */ POSIX_SPAWN_SETPGROUP  |

                                            POSIX_SPAWN_USEVFORK);

    if (err_)

Attachments

Issue Links

relates to

MDEV-10442 "Address already in use" on restart

Closed

Activity

Ascending order - Click to sort in descending order

Claudio Nanni created issue - 2017-11-14 17:06

Claudio Nanni made changes - 2017-11-14 17:06

Field	Original Value	New Value
Link		This issue relates to ~~MDEV-10442~~ [ ~~MDEV-10442~~ ]

Claudio Nanni made changes - 2017-11-14 17:07

Description

Hi,
Experiencing the same problem as ~~MDEV-10442~~, nodes can't join on Ubuntu Xenial.

mysqld stops with:

{{
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Member 0.0 (Galera_Node_02) requested state transfer from '*any*'. Selected 1.0 (Galera_Node_01)(SYNCED) as donor.
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 786790589)
2017-11-14 16:11:29 140322124806912 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-11-14 16:11:29 140322124806912 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(709f570f-c7d5-11e7-b709-e25129d5bf65:13)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20171114 16:11:29.639)
2017/11/14 16:11:29 socat[26364] E bind(6, {AF=2 0.0.0.0:4444}, 16): Address already in use
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20171114 16:11:29.645)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20171114 16:11:29.647)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.30.2.88' --datadir '/mnt/data/mysql/' --parent '26044' --binlog 'mysqld-bin' : 32 (Broken pipe)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2017-11-14 16:11:29 140322125121792 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2017-11-14 16:11:29 140322125121792 [ERROR] Aborting

}}
and socat, xbstream, wsrep_sst_xtrabackup-v2 keep running, parenthood inherited by 'init'.
As soon as you kill all the sst processes systemd restarts mysqld and the problems is back, mysqld stops, sst processes keep running.

10.1.28 seems not to contain the fix from ~~MDEV-10442~~:

In sql/wsrep_utils.cc:
}

    err_ = posix_spawnattr_setflags (&attr, POSIX_SPAWN_SETSIGDEF |
                                            POSIX_SPAWN_SETSIGMASK |
            /* start a new process group */ POSIX_SPAWN_SETPGROUP |
                                            POSIX_SPAWN_USEVFORK);
    if (err_)
    {

Hi,
Experiencing the same problem as ~~MDEV-10442~~, nodes can't join on Ubuntu Xenial.

mysqld stops with:

{{2017-11-14 16:11:29 140304279660288 [Note] WSREP: Member 0.0 (Galera_Node_02) requested state transfer from '*any*'. Selected 1.0 (Galera_Node_01)(SYNCED) as donor.
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 786790589)
2017-11-14 16:11:29 140322124806912 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-11-14 16:11:29 140322124806912 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(709f570f-c7d5-11e7-b709-e25129d5bf65:13)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20171114 16:11:29.639)
2017/11/14 16:11:29 socat[26364] E bind(6, {AF=2 0.0.0.0:4444}, 16): Address already in use
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20171114 16:11:29.645)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20171114 16:11:29.647)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.30.2.88' --datadir '/mnt/data/mysql/' --parent '26044' --binlog 'mysqld-bin' : 32 (Broken pipe)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2017-11-14 16:11:29 140322125121792 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2017-11-14 16:11:29 140322125121792 [ERROR] Aborting

}}
and socat, xbstream, wsrep_sst_xtrabackup-v2 keep running, parenthood inherited by 'init'.
As soon as you kill all the sst processes systemd restarts mysqld and the problems is back, mysqld stops, sst processes keep running.

10.1.28 seems not to contain the fix from ~~MDEV-10442~~:

In sql/wsrep_utils.cc:
}

    err_ = posix_spawnattr_setflags (&attr, POSIX_SPAWN_SETSIGDEF |
                                            POSIX_SPAWN_SETSIGMASK |
            /* start a new process group */ POSIX_SPAWN_SETPGROUP |
                                            POSIX_SPAWN_USEVFORK);
    if (err_)
    {

Claudio Nanni made changes - 2017-11-14 17:08

Description

Hi,
Experiencing the same problem as ~~MDEV-10442~~, nodes can't join on Ubuntu Xenial.

mysqld stops with:

{{2017-11-14 16:11:29 140304279660288 [Note] WSREP: Member 0.0 (Galera_Node_02) requested state transfer from '*any*'. Selected 1.0 (Galera_Node_01)(SYNCED) as donor.
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 786790589)
2017-11-14 16:11:29 140322124806912 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-11-14 16:11:29 140322124806912 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(709f570f-c7d5-11e7-b709-e25129d5bf65:13)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20171114 16:11:29.639)
2017/11/14 16:11:29 socat[26364] E bind(6, {AF=2 0.0.0.0:4444}, 16): Address already in use
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20171114 16:11:29.645)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20171114 16:11:29.647)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.30.2.88' --datadir '/mnt/data/mysql/' --parent '26044' --binlog 'mysqld-bin' : 32 (Broken pipe)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2017-11-14 16:11:29 140322125121792 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2017-11-14 16:11:29 140322125121792 [ERROR] Aborting

}}
and socat, xbstream, wsrep_sst_xtrabackup-v2 keep running, parenthood inherited by 'init'.
As soon as you kill all the sst processes systemd restarts mysqld and the problems is back, mysqld stops, sst processes keep running.

10.1.28 seems not to contain the fix from ~~MDEV-10442~~:

In sql/wsrep_utils.cc:
}

    err_ = posix_spawnattr_setflags (&attr, POSIX_SPAWN_SETSIGDEF |
                                            POSIX_SPAWN_SETSIGMASK |
            /* start a new process group */ POSIX_SPAWN_SETPGROUP |
                                            POSIX_SPAWN_USEVFORK);
    if (err_)
    {

Hi,
Experiencing the same problem as ~~MDEV-10442~~, nodes can't join on Ubuntu Xenial.

mysqld stops with:

{noformat}
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Member 0.0 (Galera_Node_02) requested state transfer from '*any*'. Selected 1.0 (Galera_Node_01)(SYNCED) as donor.
2017-11-14 16:11:29 140304279660288 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 786790589)
2017-11-14 16:11:29 140322124806912 [Note] WSREP: Requesting state transfer: success, donor: 1
2017-11-14 16:11:29 140322124806912 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(709f570f-c7d5-11e7-b709-e25129d5bf65:13)
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20171114 16:11:29.639)
2017/11/14 16:11:29 socat[26364] E bind(6, {AF=2 0.0.0.0:4444}, 16): Address already in use
WSREP_SST: [ERROR] Error while getting data from donor node: exit codes: 1 0 (20171114 16:11:29.645)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20171114 16:11:29.647)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.30.2.88' --datadir '/mnt/data/mysql/' --parent '26044' --binlog 'mysqld-bin' : 32 (Broken pipe)
2017-11-14 16:11:29 140304250304256 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2017-11-14 16:11:29 140322125121792 [ERROR] WSREP: SST failed: 32 (Broken pipe)
2017-11-14 16:11:29 140322125121792 [ERROR] Aborting
{noformat}

and socat, xbstream, wsrep_sst_xtrabackup-v2 keep running, parenthood inherited by 'init'.
As soon as you kill all the sst processes systemd restarts mysqld and the problems is back, mysqld stops, sst processes keep running.

10.1.28 seems not to contain the fix from ~~MDEV-10442~~:

In sql/wsrep_utils.cc:

{code:java}
}

    err_ = posix_spawnattr_setflags (&attr, POSIX_SPAWN_SETSIGDEF |
                                            POSIX_SPAWN_SETSIGMASK |
            /* start a new process group */ POSIX_SPAWN_SETPGROUP |
                                            POSIX_SPAWN_USEVFORK);
    if (err_)
    {
{code}

Elena Stepanova made changes - 2017-11-14 17:50

Fix Version/s		10.1 [ 16100 ]
Assignee		Andrii Nikitin [ anikitin ]

Andrii Nikitin (Inactive) made changes - 2017-11-15 16:29

Assignee

Andrii Nikitin [ anikitin ]

Sergei Golubchik [ serg ]

Sergei Golubchik made changes - 2017-12-12 11:29

Sprint

10.1.30 [ 215 ]

Sergei Golubchik made changes - 2017-12-12 11:29

Rank

Ranked higher

Sergei Golubchik made changes - 2017-12-18 16:06

Fix Version/s		10.1.30 [ 22637 ]
Fix Version/s	10.1 [ 16100 ]
Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Closed [ 6 ]

Sergei Golubchik made changes - 2021-12-06 21:46

Workflow

MariaDB v3 [ 83795 ]

MariaDB v4 [ 153186 ]

Jira Automation (IT) made changes - 2024-07-04 08:10

Zendesk Related Tickets

112061

People

Assignee:: Sergei Golubchik

Reporter:: Claudio Nanni

Votes:: 1 Vote for this issue

Watchers:: 5 Start watching this issue

Dates

Created:: 2017-11-14 17:06

Updated:: 2024-07-08 00:37

Resolved:: 2017-12-18 16:06

Git Integration

Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.

MariaDB Server

Details

Description

Attachments

Issue Links

Activity

People

Dates

Git Integration