Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-18480

Galera: Rolling upgrade: Assertion `xid_seqno > wsrep_seqno' failed. Crash on node5 after upgrading node4 from 10.3 to 10.4

    Details

      Description

      Hi, I used following scenario:

      Tested on Ubuntu 18.04 desktop with -> 10.3.13-MariaDB-debug and 10.4.3-MariaDB-debug:

      • Start given number of nodes - with my test 5 nodes with 10.3 Maria + 3.x Galera.
      • Shutdown node5
      • Start node5 with 10.4 Maria + 4.x Galera.(upgrade)
      • Run mysqld_upgrade on node5
      • Shutdown node4
      • Start node4 with 10.4 Maria + 4.x Galera.(upgrade)
      • Lost node5 with following error:

      2019-02-05 16:13:56 3 [Note] WSREP:     id: 115052ad-293e-11e9-b321-f39d8b096a01 name: node1
      2019-02-05 16:13:56 3 [Note] WSREP:     id: 19b702ff-293e-11e9-8c1d-bfccc2b51c78 name: node2
      2019-02-05 16:13:56 3 [Note] WSREP:     id: 2216b05e-293e-11e9-a39e-2f095cd98421 name: node3
      2019-02-05 16:13:56 3 [Note] WSREP:     id: 51e56e60-293e-11e9-b14b-3ee4c00d6096 name: node5
      2019-02-05 16:13:56 3 [Note] WSREP: =================================================
      2019-02-05 16:13:56 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      mysqld: /home/shako/Galera_Tests/MariaDB/storage/innobase/trx/trx0rseg.cc:92: void trx_rseg_update_wsrep_checkpoint(trx_rsegf_t*, const XID*, mtr_t*): Assertion `xid_seqno > wsrep_seqno' failed.
      190205 16:13:56 [ERROR] mysqld got signal 6 ;
      

      /home/shako/Galera_Tests/dbs/maria_10.4/bin/mysqld(+0xed0d5b)[0x560ccc738d5b]
      trx/trx0rseg.cc:94(trx_rseg_update_wsrep_checkpoint(unsigned char*, xid_t const*, mtr_t*))[0x560ccc738eb6]
      trx/trx0rseg.cc:130(trx_rseg_update_wsrep_checkpoint(xid_t const*))[0x560ccc543ac8]
      handler/ha_innodb.cc:18664(innobase_wsrep_set_checkpoint(handlerton*, xid_t const*))[0x560ccc200442]
      sql/wsrep_xid.cc:121(set_SE_checkpoint(THD*, st_plugin_int**, void*))[0x560ccbfef53a]
      sql/sql_plugin.cc:2432(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int**, void*), int, unsigned int, void*))[0x560ccc20048b]
      sql/wsrep_xid.cc:128(wsrep_set_SE_checkpoint(xid_t&))[0x560ccc2004e5]
      sql/wsrep_xid.cc:135(wsrep_set_SE_checkpoint(wsrep::gtid const&))[0x560ccc50948e]
      sql/wsrep_server_service.cc:220(Wsrep_server_service::log_view(wsrep::high_priority_service*, wsrep::view const&))[0x560cccb541b1]
      src/server_state.cpp:858(wsrep::server_state::on_view(wsrep::view const&, wsrep::high_priority_service*))[0x560cccb6682d]
      src/replicator_smm.cpp:2411(galera::ReplicatorSMM::submit_view_info(void*, wsrep_view_info const*))[0x7f80a8ca189e]
      src/replicator_smm.cpp:2794(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7f80a8ca6afe]
      src/gcs_action_source.cpp:124(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f80a8c8e62a]
      src/gcs_action_source.cpp:182(galera::GcsActionSource::process(void*, bool&))[0x7f80a8c8e971]
      src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f80a8ca4eec]
      src/wsrep_provider.cpp:263(galera_recv)[0x7f80a8cc0ea8]
      /home/shako/Galera_Tests/dbs/maria_10.4/bin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x30)[0x560cccb67898]
      src/wsrep_provider_v26.cpp:646(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x560ccc21d14a]
      sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x560ccc20e8fe]
      nptl/pthread_create.c:463(start_thread)[0x7f80abf426db]
      x86_64/clone.S:97(clone)[0x7f80ab12888f]
      

        Attachments

        1. mynode4_10.3.cnf
          0.6 kB
        2. mynode4_10.4.cnf
          0.6 kB
        3. mynode5_10.3.cnf
          0.6 kB
        4. mynode5_10.4.cnf
          0.6 kB
        5. mysql-node5.err
          63 kB

          Issue Links

            Activity

              People

              • Assignee:
                teemu.ollakka Teemu Ollakka
                Reporter:
                shako Shahriyar Rzayev
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: