Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.11, 11.0.1, 11.0.2, 11.0.3, 11.0.4
-
bitnami/mariadb-galera:11.0.4-debian-11-r0 in k8s cluster
Description
After upgrading initially from 10.6->10.11->11.0.3 and now to 11.0.4 we see that 2 of the members start up without any issues but the 3rd member(db-0) fails to start up due to an old sequence no.(117376) that i believe is being passed on from the donor.
We are unable to find the old seqno anywhere except in the `ibdata1` file of the donor(by searching for the hex of it). But not sure how to get rid of this old seqno.
What we did try is starting db-0 with fresh volume by removing K8s PVC for it, so it does a SST but that fails as well.
Logs from member db-0:
[Note] WSREP: SST received
|
[Note] WSREP: Server status change joiner -> initializing
|
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
[Note] mysqld: Aria engine: starting recovery
|
recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 2 1(0.0 seconds);
|
[Note] mysqld: Aria engine: recovery done |
[Note] InnoDB: Compressed tables use zlib 1.2.11
|
[Note] InnoDB: Number of transaction pools: 1
|
[Note] InnoDB: Using crc32 + pclmulqdq instructions
|
[Note] InnoDB: Using Linux native AIO
|
[Note] InnoDB: Initializing buffer pool, total size = 2.000GiB, chunk size = 32.000MiB
|
[Note] InnoDB: Completed initialization of buffer pool
|
[Note] InnoDB: File system buffers for log disabled (block size=512 bytes) |
[Note] InnoDB: End of log at LSN=180964319
|
[Note] InnoDB: Resizing redo log from 12.016KiB to 96.000MiB; LSN=180964319
|
[Note] InnoDB: File system buffers for log disabled (block size=512 bytes) |
[Note] InnoDB: Reinitializing innodb_undo_tablespaces= 3 from 0
|
[Note] InnoDB: Data file .//undo001 did not exist: new to be created |
[Note] InnoDB: Setting file .//undo001 size to 10.000MiB |
[Note] InnoDB: Database physically writes the file full: wait... |
[Note] InnoDB: Data file .//undo002 did not exist: new to be created |
[Note] InnoDB: Setting file .//undo002 size to 10.000MiB |
[Note] InnoDB: Database physically writes the file full: wait... |
[Note] InnoDB: Data file .//undo003 did not exist: new to be created |
[Note] InnoDB: Setting file .//undo003 size to 10.000MiB |
[Note] InnoDB: Database physically writes the file full: wait... |
[Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. |
[Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... |
[Note] InnoDB: File './ibtmp1' size is now 12.000MiB. |
[Note] InnoDB: log sequence number 180964319; transaction id 73558 |
[Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool |
[Note] InnoDB: Cannot open '/bitnami/mariadb/data/ib_buffer_pool' for reading: No such file or directory |
[Note] Plugin 'FEEDBACK' is disabled. |
[Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
[Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
[Note] Recovering after a crash using mysql-bin
|
[Note] Starting table crash recovery...
|
[Note] Crash table recovery finished.
|
[Note] Server socket created on IP: '0.0.0.0'. |
[Warning] 'proxies_priv' entry '@% root@db-0' ignored in --skip-name-resolve mode. |
[Note] WSREP: wsrep_init_schema_and_SR (nil)
|
[Note] WSREP: Server initialized
|
[Note] WSREP: Server status change initializing -> initialized
|
[Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
|
[Note] WSREP: Recovered position from storage: 6aa53efc-db72-11ec-880f-a282ce494905:117376
|
[Note] WSREP: Starting applier thread 6
|
[Note] WSREP: Starting applier thread 7
|
[Note] WSREP: Starting applier thread 8
|
[Note] WSREP: Recovered view from SST:
|
id: 6aa53efc-db72-11ec-880f-a282ce494905:128934 |
status: primary
|
protocol_version: 4
|
capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
|
final: no
|
own_index: 0
|
members(3):
|
0: 46250027-95c8-11ee-9d94-06eb82413a07, db
|
1: 4ed771b5-941f-11ee-82bf-56ec952abcf6, db
|
2: bfa6e688-941f-11ee-97c5-d323f02068c1, db
|
 |
[ERROR] WSREP: sst_received failed: SST script passed bogus GTID: 6aa53efc-db72-11ec-880f-a282ce494905:117376. Preceding view GTID: 6aa53efc-db72-11ec-880f-a282ce494905:128934
|
[Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
|
[Note] WSREP: Joiner monitor thread ended with total time 14 sec |
[ERROR] WSREP: Application received wrong state:
|
Received: 00000000-0000-0000-0000-000000000000
|
Required: 6aa53efc-db72-11ec-880f-a282ce494905
|
[ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
|
[Note] WSREP: ReplicatorSMM::abort()
|
[Note] WSREP: Closing send monitor...
|
[Note] WSREP: Closed send monitor.
|
[Note] WSREP: gcomm: terminating thread
|
[Note] WSREP: gcomm: joining thread
|
[Note] WSREP: gcomm: closing backend
|
[Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections. |
Version: '11.0.3-MariaDB-log' socket: '/opt/bitnami/mariadb/tmp/mysql.sock' port: 3306 Source distribution |
[Note] WSREP: view(view_id(NON_PRIM,46250027-9d94,6636) memb {
|
46250027-9d94,0
|
} joined {
|
} left {
|
} partitioned {
|
4ed771b5-82bf,0
|
bfa6e688-97c5,0
|
})
|
[Note] WSREP: PC protocol downgrade 1 -> 0
|
[Note] WSREP: view((empty))
|
[Note] WSREP: gcomm: closed
|
[Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
|
[Note] WSREP: Flow-control interval: [128, 160]
|
[Note] WSREP: Received NON-PRIMARY.
|
[Note] WSREP: Shifting JOINER -> OPEN (TO: 128934)
|
[Note] WSREP: New SELF-LEAVE.
|
[Note] WSREP: Flow-control interval: [0, 0]
|
[Note] WSREP: Received SELF-LEAVE. Closing connection.
|
[Note] WSREP: Shifting OPEN -> CLOSED (TO: 128934)
|
[Note] WSREP: RECV thread exiting 0: Success
|
[Note] WSREP: recv_thread() joined.
|
[Note] WSREP: Closing replication queue.
|
[Note] WSREP: Closing slave action queue.
|
[Note] WSREP: /opt/bitnami/mariadb/sbin/mysqld: Terminated. |
Attachments
Issue Links
- is caused by
-
MDEV-19229 Allow innodb_undo_tablespaces to be changed after database creation
- Closed
-
MDEV-29986 Set innodb_undo_tablespaces=3 by default
- Closed