Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-31888

galera.galera_wan, galera.galera_vote_rejoin_* fail

Details

    Description

      ES 23.07.0

      galera.galera_wan 'innodb'               [ fail ]  Found warnings/errors in server log file!
      		 
              Test ended at 2023-08-08 03:57:58
      		 
      line
      		 
      2023-08-08  3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
      		 
      2023-08-08  3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
      		 
      2023-08-08  3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
      		 
      ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
      		 
      ok


      galera.galera_vote_rejoin_ddl 'innodb'   [ fail ]  Found warnings/errors in server log file!
      		 
              Test ended at 2023-08-08 03:56:33
      		 
      line
      		 
      2023-08-08  3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
      		 
      2023-08-08  3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
      		 
      2023-08-08  3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
      		 
      ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
      		 
      ok

      Attachments

        1. logs (7).tar.gz
          9.87 MB
        2. messages.tar.gz
          10 kB
        3. mtr (7).log
          56 kB

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33778 galera.galera_wan_restart_ist test failure

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          is part of

          Task - A task that needs to be done. MDEV-33073 always green buildbot

          • Major - Major loss of function.
          • Stalled
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-33778 galera.galera_wan_restart_ist test failure

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            tturenko Timofey Turenko created issue -
            tturenko Timofey Turenko made changes -
            Field Original Value New Value
            Summary galera.galera_wan fails galera.galera_wan, galera.galera_vote_rejoin_* fail
            tturenko Timofey Turenko made changes -
            Description ```
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            ```             		```
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            ```

            ```
            galera.galera_vote_rejoin_ddl 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:56:33
            line
            2023-08-08 3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            ```
            tturenko Timofey Turenko made changes -
            Description ```
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            ```

            ```
            galera.galera_vote_rejoin_ddl 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:56:33
            line
            2023-08-08 3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            ```

            {noformat}
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}


            {noformat}
            galera.galera_vote_rejoin_ddl 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:56:33
            line
            2023-08-08 3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}

            tturenko Timofey Turenko made changes -
            Component/s Galera [ 10124 ]
            Component/s Tests [ 10800 ]
            Component/s Galera [ 14918 ]
            Component/s Tests [ 14999 ]
            Key MENT-1913 MDEV-31888
            Affects Version/s 11.1.1 [ 28704 ]
            Affects Version/s 23.07.0 [ 27403 ]
            Project MariaDB Enterprise [ 11500 ] MariaDB Server [ 10000 ]
            tturenko Timofey Turenko made changes -
            Description

            {noformat}
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}


            {noformat}
            galera.galera_vote_rejoin_ddl 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:56:33
            line
            2023-08-08 3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}

            		ES 23.07.0

            {noformat}
            galera.galera_wan 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:57:58
            line
            2023-08-08 3:57:45 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:57:49 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:57:49 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}


            {noformat}
            galera.galera_vote_rejoin_ddl 'innodb' [ fail ] Found warnings/errors in server log file!
                    Test ended at 2023-08-08 03:56:33
            line
            2023-08-08 3:55:51 2 [Warning] WSREP: server: mdbci-xpsburev-1691453554-build unallowed state transition: donor -> synced
            2023-08-08 3:55:55 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            2023-08-08 3:55:55 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            ^ Found warnings in /var/tmp/mtr/log/mysqld.1.err
            ok
            {noformat}

            tturenko Timofey Turenko added a comment -

            also reproducible with CS 11.1.1, test `galera.galera_wan_restart_sst `, Ubuntu Focal

            galera.galera_wan_restart_sst 'innodb'   [ fail ]
            		 
                    Test ended at 2023-08-09 19:37:48
            		 
             
            		 
            WSREP did not transition to state READY
            		 
             
            		 
             - skipping '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/'
            		 
            ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst
            		 
             
            		 
            2023-08-09 19:34:10 2 [Warning] WSREP: server: mdbci-yivvqij5-1691597248-build unallowed state transition: donor -> synced
            		 
            2023-08-09 19:34:12 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED
            		 
            2023-08-09 19:34:12 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component
            		 
            2023-08-09 19:34:16 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1
            		 
            2023-08-09 19:34:16 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error)
            		 
            2023-08-09 19:34:16 2 [ERROR] WSREP: /home/buildbot/buildbot/build/gcs/src/gcs.cpp:s_join():960: Sending JOIN failed: -103 (Software caused connection abort).
            		 
            2023-08-09 19:34:16 2 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(eb36e548-36da-11ee-9a24-12383ef8c7c5:3) failed: 103 (Software caused connection abort)
            		 
            2023-08-09 19:34:16 2 [Warning] WSREP: Cert position -1 less than last committed 3
            		 
            2023-08-09 19:34:16 2 [Warning] WSREP: Cert position -1 less than last committed 3

            tturenko Timofey Turenko added a comment - also reproducible with CS 11.1.1, test `galera.galera_wan_restart_sst `, Ubuntu Focal galera.galera_wan_restart_sst 'innodb' [ fail ] Test ended at 2023-08-09 19:37:48   WSREP did not transition to state READY   - skipping '/var/tmp/mtr/log/galera.galera_wan_restart_sst-innodb/' ***Warnings generated in error logs during shutdown after running tests: galera.galera_wan_restart_sst   2023-08-09 19:34:10 2 [Warning] WSREP: server: mdbci-yivvqij5-1691597248-build unallowed state transition: donor -> synced 2023-08-09 19:34:12 0 [ERROR] WSREP: sst sent called when not SST donor, state SYNCED 2023-08-09 19:34:12 0 [Warning] WSREP: wsrep::sst_sent() returned an error: Not connected to Primary Component 2023-08-09 19:34:16 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: -1 2023-08-09 19:34:16 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 71 (Protocol error) 2023-08-09 19:34:16 2 [ERROR] WSREP: /home/buildbot/buildbot/build/gcs/src/gcs.cpp:s_join():960: Sending JOIN failed: -103 (Software caused connection abort). 2023-08-09 19:34:16 2 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(eb36e548-36da-11ee-9a24-12383ef8c7c5:3) failed: 103 (Software caused connection abort) 2023-08-09 19:34:16 2 [Warning] WSREP: Cert position -1 less than last committed 3 2023-08-09 19:34:16 2 [Warning] WSREP: Cert position -1 less than last committed 3
            serg Sergei Golubchik made changes -
            Fix Version/s 11.1 [ 28549 ]
            serg Sergei Golubchik made changes -
            Fix Version/s 10.5 [ 23123 ]
            serg Sergei Golubchik made changes -
            Affects Version/s 1.5 [ 29825 ]
            serg Sergei Golubchik made changes -
            Affects Version/s 10.5 [ 23123 ]
            Affects Version/s 1.5 [ 29825 ]
            serg Sergei Golubchik added a comment -

            CS new buildbot: https://buildbot.mariadb.org/#/builders/595/builds/4000

            serg Sergei Golubchik added a comment - CS new buildbot: https://buildbot.mariadb.org/#/builders/595/builds/4000
            serg Sergei Golubchik made changes -
            danblack Daniel Black added a comment -

            10.5.26, 26.4.19, https://buildbot.mariadb.org/#/builders/619/builds/6383/steps/10/logs/stdio FreeBSD

            		 
             
            		 
            galera.galera_vote_rejoin_dml            w2 [ fail ]
            		 
                    Test ended at 2024-08-30 02:03:41
            		 
            WSREP did not transition to state READY
            		 
             - saving '/data/buildbot/workers/prod/amd64-freebsd-14/build/mysql-test/var/2/log/galera.galera_vote_rejoin_dml/' to '/data/buildbot/workers/prod/amd64-freebsd-14/build/mysql-test/var/log/galera.galera_vote_rejoin_dml/'
            		 
            Retrying test galera.galera_vote_rejoin_dml, attempt(2/3)...
            		 
            worker[02] > Restart  - not started
            		 
            worker[02] > Restart  - not started
            		 
            worker[02] > Restart  - not started
            		 
            worker[02] > Restart  - not started
            		 
            ***Warnings generated in error logs during shutdown after running tests: galera.galera_vote_rejoin_dml
            		 
            2024-08-30  1:59:41 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: 2
            		 
            2024-08-30  1:59:41 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 92 (Protocol error)
            		 
            2024-08-30  1:59:41 2 [ERROR] WSREP: /usr/ports/databases/galera26/work/galera-mariadb-26.4.19/gcs/src/gcs.cpp:s_join():965: Sending JOIN failed: -53 (Software caused connection abort).
            		 
            2024-08-30  1:59:41 2 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(b7b3b223-6662-11ef-9d86-5ea8bccc41a7:3) failed: 53 (Software caused connection abort)
            		 
            2024-08-30  1:59:41 2 [Warning] WSREP: Cert position 2 less than last committed 3
            		 
            2024-08-30  1:59:41 2 [Warning] WSREP: Cert position 2 less than last committed 3
            		 
            galera.galera_vote_rejoin_dml            w2 [ retry-pass ]  26944

            danblack Daniel Black added a comment - 10.5.26, 26.4.19, https://buildbot.mariadb.org/#/builders/619/builds/6383/steps/10/logs/stdio FreeBSD   galera.galera_vote_rejoin_dml w2 [ fail ] Test ended at 2024-08-30 02:03:41 WSREP did not transition to state READY - saving '/data/buildbot/workers/prod/amd64-freebsd-14/build/mysql-test/var/2/log/galera.galera_vote_rejoin_dml/' to '/data/buildbot/workers/prod/amd64-freebsd-14/build/mysql-test/var/log/galera.galera_vote_rejoin_dml/' Retrying test galera.galera_vote_rejoin_dml, attempt(2/3)... worker[02] > Restart - not started worker[02] > Restart - not started worker[02] > Restart - not started worker[02] > Restart - not started ***Warnings generated in error logs during shutdown after running tests: galera.galera_vote_rejoin_dml 2024-08-30 1:59:41 0 [ERROR] WSREP: IST didn't contain all write sets, expected last: 3 last received: 2 2024-08-30 1:59:41 2 [ERROR] WSREP: Receiving IST failed, node restart required: IST receiver reported failure: 92 (Protocol error) 2024-08-30 1:59:41 2 [ERROR] WSREP: /usr/ports/databases/galera26/work/galera-mariadb-26.4.19/gcs/src/gcs.cpp:s_join():965: Sending JOIN failed: -53 (Software caused connection abort). 2024-08-30 1:59:41 2 [Warning] WSREP: Failed to JOIN the cluster after SST gcs_join(b7b3b223-6662-11ef-9d86-5ea8bccc41a7:3) failed: 53 (Software caused connection abort) 2024-08-30 1:59:41 2 [Warning] WSREP: Cert position 2 less than last committed 3 2024-08-30 1:59:41 2 [Warning] WSREP: Cert position 2 less than last committed 3 galera.galera_vote_rejoin_dml w2 [ retry-pass ] 26944
            julien.fritsch Julien Fritsch made changes -
            Fix Version/s 11.1 [ 28549 ]
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            sysprg Julius Goryavsky made changes -
            Assignee Julius Goryavsky [ sysprg ] Jan Lindström [ JIRAUSER53125 ]
            janlindstrom Jan Lindström made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            janlindstrom Jan Lindström added a comment -

            https://github.com/MariaDB/server/pull/3550

            janlindstrom Jan Lindström added a comment - https://github.com/MariaDB/server/pull/3550
            janlindstrom Jan Lindström made changes -
            Assignee Jan Lindström [ JIRAUSER53125 ] Julius Goryavsky [ sysprg ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            janlindstrom Jan Lindström made changes -
            janlindstrom Jan Lindström made changes -
            sysprg Julius Goryavsky made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky made changes -
            Assignee Julius Goryavsky [ sysprg ] Jan Lindström [ JIRAUSER53125 ]
            janlindstrom Jan Lindström made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            janlindstrom Jan Lindström added a comment -

            sysprg I reverted part of configuration change that is not really needed to get these pass.

            janlindstrom Jan Lindström added a comment - sysprg I reverted part of configuration change that is not really needed to get these pass.
            janlindstrom Jan Lindström made changes -
            Assignee Jan Lindström [ JIRAUSER53125 ] Julius Goryavsky [ sysprg ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            sysprg Julius Goryavsky made changes -
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sysprg Julius Goryavsky made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sysprg Julius Goryavsky added a comment -

            Fix has been merged with the head revision: https://github.com/MariaDB/server/commit/7ffa7b6b0193738d011e514b66d875ce29bc11d2

            sysprg Julius Goryavsky added a comment - Fix has been merged with the head revision: https://github.com/MariaDB/server/commit/7ffa7b6b0193738d011e514b66d875ce29bc11d2
            sysprg Julius Goryavsky made changes -
            Fix Version/s 10.5.27 [ 29902 ]
            Fix Version/s 10.5 [ 23123 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            JIraAutomate JiraAutomate made changes -
            Fix Version/s 10.6.20 [ 29903 ]
            Fix Version/s 10.11.10 [ 29904 ]
            Fix Version/s 11.2.6 [ 29906 ]
            Fix Version/s 11.4.4 [ 29907 ]

            People

              sysprg Julius Goryavsky
              tturenko Timofey Turenko
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.