Details
-
Bug
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11.16
-
None
-
None
-
Ubuntu 22
Description
Last night be were in the process of upgrading the last of our 10.6.15 nodes in a three node cluster to 10.11.16.
However the moment I stop the mariadb service on node 2, node 1 crashed with:
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 11 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 11 [Note] WSREP: Lowest cert index boundary for CC from group: 16295626829 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 11 [Note] WSREP: Min available from gcache for CC from group: 16285754179 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Message with protocol version 2 != highest commonly supported: 6. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Discard message from member 1 because of not commonly supported version. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Message with protocol version 2 != highest commonly supported: 6. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Resend message because of not commonly supported version. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 238757 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 238757 [Warning] WSREP: Appending row key failed: insert into opmon_results.serviceResultSnmp set deviceServiceId=773029, IP='172.19.156.157', resultBoolean=1, resultState=1, da> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 1702017 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138653 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138653 [Warning] WSREP: Appending row key failed: insert into opmon_results.serviceResultSnmp set deviceServiceId=443702, IP='172.18.3.177', resultBoolean=1, resultState=1, dat> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 239490 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 239490 [Warning] WSREP: Appending row key failed: insert into opmon_results.serviceResultSnmp set deviceServiceId=446315, IP='172.18.4.15', resultState=3, datetime=FROM_UNIXTIME> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 1702017 [Warning] WSREP: Appending row key failed: insert into opmon_results.serviceResultSnmp set deviceServiceId=448487, IP='172.18.4.100', resultBoolean=1, resultState=1, dat> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [ERROR] WSREP: Appending table key failed: update alert set count=536394, lastUpdate=FROM_UNIXTIME(1778785394) where id=677451541 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Note] Failed Foreign key referenced table found: opmon.action |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2001597 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2001597 [Warning] WSREP: Appending row key failed: insert into opmon_results.serviceResultSnmp set deviceServiceId=382745, IP='172.16.2.39', resultBoolean=1, resultState=1, date> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [ERROR] WSREP: Appending table key failed: update alert set count=757942, lastUpdate=FROM_UNIXTIME(1778785394) where id=562546361 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Note] Failed Foreign key referenced table found: opmon.action |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [ERROR] WSREP: Appending table key failed: update alert set state=2, resolved=FROM_UNIXTIME(1778785394) where id=1459427161 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Note] Failed Foreign key referenced table found: opmon.action |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [Warning] WSREP: key version '6' does not match to trx version' 5' |
May 14 21:03:27 db1 mariadbd[1538183]: #011 at ./galera/src/trx_handle.hpp:append_key():894 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [ERROR] WSREP: Appending cascaded fk row key failed: insert into alert set alertTypeId=7, count=1, logTime=FROM_UNIXTIME(1778785395), lastUpdate=FROM_UNIXTIME(1778785395> |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 2138508 [ERROR] [FATAL] InnoDB: Unknown error Generic error |
May 14 21:03:27 db1 mariadbd[1538183]: 260514 21:03:27 [ERROR] /usr/sbin/mariadbd got signal 6 ; |
May 14 21:03:27 db1 mariadbd[1538183]: Sorry, we probably made a mistake, and this is a bug. |
May 14 21:03:27 db1 mariadbd[1538183]: Your assistance in bug reporting will enable us to fix this for the next release. |
May 14 21:03:27 db1 mariadbd[1538183]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs about how to report |
May 14 21:03:27 db1 mariadbd[1538183]: a bug on https://jira.mariadb.org/. |
May 14 21:03:27 db1 mariadbd[1538183]: Please include the information from the server start above, to the end of the |
May 14 21:03:27 db1 mariadbd[1538183]: information below. |
May 14 21:03:27 db1 mariadbd[1538183]: Server version: 10.11.16-MariaDB-ubu2204 source revision: 3218602d3100db9ce7a875511a591cddc173cc16 |
May 14 21:03:27 db1 mariadbd[1538183]: WSREP: Suppressing further logging |
May 14 21:03:27 db1 mariadbd[1538183]: WSREP: Shutting down network communications |
May 14 21:03:27 db1 mariadbd[1538183]: The information page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ |
May 14 21:03:27 db1 mariadbd[1538183]: contains instructions to obtain a better version of the backtrace below. |
May 14 21:03:27 db1 mariadbd[1538183]: Following these instructions will help MariaDB developers provide a fix quicker. |
May 14 21:03:27 db1 mariadbd[1538183]: Attempting backtrace. Include this in the bug report. |
May 14 21:03:27 db1 mariadbd[1538183]: (note: Retrieving this information may fail) |
May 14 21:03:27 db1 mariadbd[1538183]: Thread pointer: 0x7ebd64000c68 |
May 14 21:03:27 db1 mariadbd[1538183]: stack_bottom = 0x7f3854cd0000 thread_stack 0x49000 |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Shutdown replication |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Server status change synced -> disconnecting |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Closing send monitor... |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: Closed send monitor. |
May 14 21:03:27 db1 mariadbd[1538183]: 2026-05-14 21:03:27 0 [Note] WSREP: gcomm: terminating thread |
Shortly after that node 3 went into non-primary.
I shutdown all nodes and bootstrapped node 3 and then recovered node 1 and then proceeded to upgrade and bring node 2 online after which everything was stable.
It did however cause an outage of about 30 minute for us.
Why did the WSREP transaction key errors appear.
Should the two remaining nodes not both have started to communicate at the new protocol version once node 2 (10.6) left?
Is rolling upgrades then not reliable?