Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35023

WSREP: State transfer failed: -22 (Invalid argument)

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.9
    • None
    • Galera
    • Ubuntu 22.04.5

    Description

      Hi team,
      we have a 5 node Galera Cluster 26.4.19 with MariaDB Server 10.11.9.
      After a test update of MariaDB on one node (maria-ham-3) I switched back to the versions above. I cleared data directory and wanted to sync and join (what worked fine in the past) with the node:
      {{

      1. rm -rf /GALERANFS_HAM_DATA/data/mariadb/* /GALERANFS_HAM_DATA/InnoDB/ibdata1 /GALERANFS_HAM_LOG/ib_log/ib_logfile0
      2. systemctl start mariadb
        }}
        This leads to a transfer starting but the later failing:
        {{
        Sep 27 09:20:52 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:52 0 [Note] WSREP: REPORTING SST PROGRESS: ' { "from": 1, "to": 3, "total": 19922598377, "done": 19922598377, "indefinite": -1 }

        '
        Sep 27 09:20:58 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:20:58 0 [Note] WSREP: REPORTING SST PROGRESS: '

        { "from": 1, "to": 3, "total": 19935177778, "done": 19935177778, "indefinite": -1 }

        '
        Sep 27 09:21:04 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:04 0 [Note] WSREP: REPORTING SST PROGRESS: '

        { "from": 1, "to": 3, "total": 19947753124, "done": 19947753124, "indefinite": -1 }

        '
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Warning] WSREP: 0.0 (maria-muc-1): State transfer to 3.0 (maria-ham-3) failed: -22 (Invalid argument)
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [ERROR] WSREP: ./gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():1217: Will never receive state. Need to abort.
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: terminating thread
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: joining thread
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closing backend
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view(view_id(NON_PRIM,0abc8cb4-9ae0,609) memb

        { Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: a591b16e-9113,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }

        joined

        { Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }

        left

        { Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }

        partitioned

        { Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 0abc8cb4-9ae0,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 29958514-86e6,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 7150c82e-8f59,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: dfc13b8b-8838,0 Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: }

        )
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: PC protocol downgrade 1 -> 0
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: view((empty))
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: gcomm: closed
        Sep 27 09:21:08 maria-ham-3 mariadbd[2442465]: 2024-09-27 9:21:08 0 [Note] WSREP: /usr/sbin/mariadbd: Terminated.
        Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
        Sep 27 09:21:08 maria-ham-3 systemd[1]: mariadb.service: Failed with result 'signal'.
        Sep 27 09:21:08 maria-ham-3 systemd[1]: Failed to start MariaDB 10.11.9 database server.
        }}
        I configured the wsrep_sst_donor to one node to ease the debugging.
        The logs of both sides are attached.
        Filesystems are not full.
        I cannot find out what stopped the transfer. Can you give me any hint of what to do for further debugging?
        Thanks in advance
        Lars

      Attachments

        1. maria-donor.txt
          19 kB
          Lars Timmann
        2. maria-receiver.txt
          47 kB
          Lars Timmann

        Activity

          People

            Unassigned Unassigned
            lollypop Lars Timmann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.