[MDEV-22745] node crash on upgrade from 10.3 to 10.4 writing on the 10.4 node Created: 2020-05-28  Updated: 2022-08-04  Resolved: 2020-08-20

Status: Closed
Project: MariaDB Server
Component/s: Galera
Affects Version/s: 10.3.23, 10.4.13
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Massimo Assignee: Alexey
Resolution: Cannot Reproduce Votes: 0
Labels: None
Environment:

CentOS Linux release 7.6.1810


Attachments: Text File error_log_scrash.txt    
Issue Links:
Relates
relates to MDEV-22723 Data loss when performing rolling upg... Closed
relates to MDEV-29246 WSREP_CLUSTER_SIZE at 0 after rolling... Closed

 Description   

We have a crash during upgrade:

galera cluster of 2 node 10.3.23. we upgrade to mariadb 10.4.13 one node, run mysql_upgrade, get it to rejoin the cluster.

At the moment all the data were in sync on the mdb2 node with 10.4 but the wsrep:

MariaDB mdb2 [(none)]> show global status like 'wsrep%';
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name                 | Value                                                                                                                                          |
+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| wsrep_local_state_uuid        | abc10424-9f6c-11ea-aefe-16619600de3a                                                                                                           |
| wsrep_protocol_version        | -1                                                                                                                                             |
| wsrep_last_committed          | 6                                                                                                                                              |
| wsrep_replicated              | 0                                                                                                                                              |
| wsrep_replicated_bytes        | 0                                                                                                                                              |
| wsrep_repl_keys               | 0                                                                                                                                              |
| wsrep_repl_keys_bytes         | 0                                                                                                                                              |
| wsrep_repl_data_bytes         | 0                                                                                                                                              |
| wsrep_repl_other_bytes        | 0                                                                                                                                              |
| wsrep_received                | 3                                                                                                                                              |
| wsrep_received_bytes          | 208                                                                                                                                            |
| wsrep_local_commits           | 0                                                                                                                                              |
| wsrep_local_cert_failures     | 0                                                                                                                                              |
| wsrep_local_replays           | 0                                                                                                                                              |
| wsrep_local_send_queue        | 0                                                                                                                                              |
| wsrep_local_send_queue_max    | 1                                                                                                                                              |
| wsrep_local_send_queue_min    | 0                                                                                                                                              |
| wsrep_local_send_queue_avg    | 0                                                                                                                                              |
| wsrep_local_recv_queue        | 0                                                                                                                                              |
| wsrep_local_recv_queue_max    | 1                                                                                                                                              |
| wsrep_local_recv_queue_min    | 0                                                                                                                                              |
| wsrep_local_recv_queue_avg    | 0                                                                                                                                              |
| wsrep_local_cached_downto     | 6                                                                                                                                              |
| wsrep_flow_control_paused_ns  | 0                                                                                                                                              |
| wsrep_flow_control_paused     | 0                                                                                                                                              |
| wsrep_flow_control_sent       | 0                                                                                                                                              |
| wsrep_flow_control_recv       | 0                                                                                                                                              |
| wsrep_cert_deps_distance      | 0                                                                                                                                              |
| wsrep_apply_oooe              | 0                                                                                                                                              |
| wsrep_apply_oool              | 0                                                                                                                                              |
| wsrep_apply_window            | 1                                                                                                                                              |
| wsrep_commit_oooe             | 0                                                                                                                                              |
| wsrep_commit_oool             | 0                                                                                                                                              |
| wsrep_commit_window           | 1                                                                                                                                              |
| wsrep_local_state             | 4                                                                                                                                              |
| wsrep_local_state_comment     | Synced                                                                                                                                         |
| wsrep_cert_index_size         | 0                                                                                                                                              |
| wsrep_causal_reads            | 0                                                                                                                                              |
| wsrep_cert_interval           | 0                                                                                                                                              |
| wsrep_open_transactions       | 0                                                                                                                                              |
| wsrep_open_connections        | 0                                                                                                                                              |
| wsrep_incoming_addresses      | AUTO,10.0.1.13:3306                                                                                                                            |
| wsrep_cluster_weight          | 2                                                                                                                                              |
| wsrep_desync_count            | 0                                                                                                                                              |
| wsrep_evs_delayed             |                                                                                                                                                |
| wsrep_evs_evict_list          |                                                                                                                                                |
| wsrep_evs_repl_latency        | 0.000443841/0.00142586/0.00253945/0.000707472/7                                                                                                |
| wsrep_evs_state               | OPERATIONAL                                                                                                                                    |
| wsrep_gcomm_uuid              | 4bf88128-9f74-11ea-8fa8-8686298518ac                                                                                                           |
| wsrep_applier_thread_count    | 32                                                                                                                                             |
| wsrep_cluster_capabilities    |                                                                                                                                                |
| wsrep_cluster_conf_id         | 18446744073709551615                                                                                                                           |
| wsrep_cluster_size            | 0                                                                                                                                              |
| wsrep_cluster_state_uuid      |                                                                                                                                                |
| wsrep_cluster_status          | Primary                                                                                                                                        |
| wsrep_connected               | ON                                                                                                                                             |

when we try to insert a record 1 line and we got the crash. see the error log attach and cluster die



 Comments   
Comment by Massimo [ 2020-05-28 ]

the issue is correlated to MDEV-22723 and the setting of the servers are the same

Comment by Stepan Patryshev (Inactive) [ 2020-08-20 ]

Closing since original MDEV-22723 is fixed and this crash I could not reproduced.

Generated at Thu Feb 08 09:17:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.