[MDEV-18480] Galera: Rolling upgrade: Assertion `xid_seqno > wsrep_seqno' failed. Crash on node5 after upgrading node4 from 10.3 to 10.4 Created: 2019-02-05  Updated: 2019-02-21  Resolved: 2019-02-21

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.4.2, 10.3.13
Fix Version/s: 10.4.3, 10.3.14

Type: Bug Priority: Major
Reporter: Shahriyar Rzayev (Inactive) Assignee: Teemu Ollakka
Resolution: Fixed Votes: 0
Labels: galera

Attachments: File mynode4_10.3.cnf     File mynode4_10.4.cnf     File mynode5_10.3.cnf     File mynode5_10.4.cnf     File mysql-node5.err    
Issue Links:
Relates
relates to MDEV-18580 Galera: Rolling upgrade: Upgraded nod... Closed

 Description   

Hi, I used following scenario:

Tested on Ubuntu 18.04 desktop with -> 10.3.13-MariaDB-debug and 10.4.3-MariaDB-debug:

  • Start given number of nodes - with my test 5 nodes with 10.3 Maria + 3.x Galera.
  • Shutdown node5
  • Start node5 with 10.4 Maria + 4.x Galera.(upgrade)
  • Run mysqld_upgrade on node5
  • Shutdown node4
  • Start node4 with 10.4 Maria + 4.x Galera.(upgrade)
  • Lost node5 with following error:

2019-02-05 16:13:56 3 [Note] WSREP:     id: 115052ad-293e-11e9-b321-f39d8b096a01 name: node1
2019-02-05 16:13:56 3 [Note] WSREP:     id: 19b702ff-293e-11e9-8c1d-bfccc2b51c78 name: node2
2019-02-05 16:13:56 3 [Note] WSREP:     id: 2216b05e-293e-11e9-a39e-2f095cd98421 name: node3
2019-02-05 16:13:56 3 [Note] WSREP:     id: 51e56e60-293e-11e9-b14b-3ee4c00d6096 name: node5
2019-02-05 16:13:56 3 [Note] WSREP: =================================================
2019-02-05 16:13:56 3 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
mysqld: /home/shako/Galera_Tests/MariaDB/storage/innobase/trx/trx0rseg.cc:92: void trx_rseg_update_wsrep_checkpoint(trx_rsegf_t*, const XID*, mtr_t*): Assertion `xid_seqno > wsrep_seqno' failed.
190205 16:13:56 [ERROR] mysqld got signal 6 ;

/home/shako/Galera_Tests/dbs/maria_10.4/bin/mysqld(+0xed0d5b)[0x560ccc738d5b]
trx/trx0rseg.cc:94(trx_rseg_update_wsrep_checkpoint(unsigned char*, xid_t const*, mtr_t*))[0x560ccc738eb6]
trx/trx0rseg.cc:130(trx_rseg_update_wsrep_checkpoint(xid_t const*))[0x560ccc543ac8]
handler/ha_innodb.cc:18664(innobase_wsrep_set_checkpoint(handlerton*, xid_t const*))[0x560ccc200442]
sql/wsrep_xid.cc:121(set_SE_checkpoint(THD*, st_plugin_int**, void*))[0x560ccbfef53a]
sql/sql_plugin.cc:2432(plugin_foreach_with_mask(THD*, char (*)(THD*, st_plugin_int**, void*), int, unsigned int, void*))[0x560ccc20048b]
sql/wsrep_xid.cc:128(wsrep_set_SE_checkpoint(xid_t&))[0x560ccc2004e5]
sql/wsrep_xid.cc:135(wsrep_set_SE_checkpoint(wsrep::gtid const&))[0x560ccc50948e]
sql/wsrep_server_service.cc:220(Wsrep_server_service::log_view(wsrep::high_priority_service*, wsrep::view const&))[0x560cccb541b1]
src/server_state.cpp:858(wsrep::server_state::on_view(wsrep::view const&, wsrep::high_priority_service*))[0x560cccb6682d]
src/replicator_smm.cpp:2411(galera::ReplicatorSMM::submit_view_info(void*, wsrep_view_info const*))[0x7f80a8ca189e]
src/replicator_smm.cpp:2794(galera::ReplicatorSMM::process_conf_change(void*, gcs_action const&))[0x7f80a8ca6afe]
src/gcs_action_source.cpp:124(galera::GcsActionSource::dispatch(void*, gcs_action const&, bool&))[0x7f80a8c8e62a]
src/gcs_action_source.cpp:182(galera::GcsActionSource::process(void*, bool&))[0x7f80a8c8e971]
src/replicator_smm.cpp:383(galera::ReplicatorSMM::async_recv(void*))[0x7f80a8ca4eec]
src/wsrep_provider.cpp:263(galera_recv)[0x7f80a8cc0ea8]
/home/shako/Galera_Tests/dbs/maria_10.4/bin/mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x30)[0x560cccb67898]
src/wsrep_provider_v26.cpp:646(wsrep::wsrep_provider_v26::run_applier(wsrep::high_priority_service*))[0x560ccc21d14a]
sql/wsrep_thd.cc:61(wsrep_replication_process(THD*, void*))[0x560ccc20e8fe]
nptl/pthread_create.c:463(start_thread)[0x7f80abf426db]
x86_64/clone.S:97(clone)[0x7f80ab12888f]



 Comments   
Comment by Shahriyar Rzayev (Inactive) [ 2019-02-21 ]

Could not reproduce anymore with latest rolling upgrade run.
Can be closed.

Generated at Thu Feb 08 08:44:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.