Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32974

Member fails to join due to old seqno in GTID

    XMLWordPrintable

Details

    Description

      After upgrading initially from 10.6->10.11->11.0.3 and now to 11.0.4 we see that 2 of the members start up without any issues but the 3rd member(db-0) fails to start up due to an old sequence no.(117376) that i believe is being passed on from the donor.

      We are unable to find the old seqno anywhere except in the `ibdata1` file of the donor(by searching for the hex of it). But not sure how to get rid of this old seqno.
      What we did try is starting db-0 with fresh volume by removing K8s PVC for it, so it does a SST but that fails as well.

      Logs from member db-0:

      [Note] WSREP: SST received
      [Note] WSREP: Server status change joiner -> initializing
      [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      [Note] mysqld: Aria engine: starting recovery
      recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 2 1(0.0 seconds); 
      [Note] mysqld: Aria engine: recovery done
      [Note] InnoDB: Compressed tables use zlib 1.2.11
      [Note] InnoDB: Number of transaction pools: 1
      [Note] InnoDB: Using crc32 + pclmulqdq instructions
      [Note] InnoDB: Using Linux native AIO
      [Note] InnoDB: Initializing buffer pool, total size = 2.000GiB, chunk size = 32.000MiB
      [Note] InnoDB: Completed initialization of buffer pool
      [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
      [Note] InnoDB: End of log at LSN=180964319
      [Note] InnoDB: Resizing redo log from 12.016KiB to 96.000MiB; LSN=180964319
      [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
      [Note] InnoDB: Reinitializing innodb_undo_tablespaces= 3 from 0
      [Note] InnoDB: Data file .//undo001 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo001 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: Data file .//undo002 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo002 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: Data file .//undo003 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo003 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
      [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      [Note] InnoDB: log sequence number 180964319; transaction id 73558
      [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
      [Note] InnoDB: Cannot open '/bitnami/mariadb/data/ib_buffer_pool' for reading: No such file or directory
      [Note] Plugin 'FEEDBACK' is disabled.
      [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      [Note] Recovering after a crash using mysql-bin
      [Note] Starting table crash recovery...
      [Note] Crash table recovery finished.
      [Note] Server socket created on IP: '0.0.0.0'.
      [Warning] 'proxies_priv' entry '@% root@db-0' ignored in --skip-name-resolve mode.
      [Note] WSREP: wsrep_init_schema_and_SR (nil)
      [Note] WSREP: Server initialized
      [Note] WSREP: Server status change initializing -> initialized
      [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      [Note] WSREP: Recovered position from storage: 6aa53efc-db72-11ec-880f-a282ce494905:117376
      [Note] WSREP: Starting applier thread 6
      [Note] WSREP: Starting applier thread 7
      [Note] WSREP: Starting applier thread 8
      [Note] WSREP: Recovered view from SST:
        id: 6aa53efc-db72-11ec-880f-a282ce494905:128934
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 0
        members(3):
      	0: 46250027-95c8-11ee-9d94-06eb82413a07, db
      	1: 4ed771b5-941f-11ee-82bf-56ec952abcf6, db
      	2: bfa6e688-941f-11ee-97c5-d323f02068c1, db
       
      [ERROR] WSREP: sst_received failed: SST script passed bogus GTID: 6aa53efc-db72-11ec-880f-a282ce494905:117376. Preceding view GTID: 6aa53efc-db72-11ec-880f-a282ce494905:128934
      [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
      [Note] WSREP: Joiner monitor thread ended with total time 14 sec
      [ERROR] WSREP: Application received wrong state: 
      	Received: 00000000-0000-0000-0000-000000000000
      	Required: 6aa53efc-db72-11ec-880f-a282ce494905
      [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
      [Note] WSREP: ReplicatorSMM::abort()
      [Note] WSREP: Closing send monitor...
      [Note] WSREP: Closed send monitor.
      [Note] WSREP: gcomm: terminating thread
      [Note] WSREP: gcomm: joining thread
      [Note] WSREP: gcomm: closing backend
      [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
      Version: '11.0.3-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
      [Note] WSREP: view(view_id(NON_PRIM,46250027-9d94,6636) memb {
      	46250027-9d94,0
      } joined {
      } left {
      } partitioned {
      	4ed771b5-82bf,0
      	bfa6e688-97c5,0
      })
      [Note] WSREP: PC protocol downgrade 1 -> 0
      [Note] WSREP: view((empty))
      [Note] WSREP: gcomm: closed
      [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      [Note] WSREP: Flow-control interval: [128, 160]
      [Note] WSREP: Received NON-PRIMARY.
      [Note] WSREP: Shifting JOINER -> OPEN (TO: 128934)
      [Note] WSREP: New SELF-LEAVE.
      [Note] WSREP: Flow-control interval: [0, 0]
      [Note] WSREP: Received SELF-LEAVE. Closing connection.
      [Note] WSREP: Shifting OPEN -> CLOSED (TO: 128934)
      [Note] WSREP: RECV thread exiting 0: Success
      [Note] WSREP: recv_thread() joined.
      [Note] WSREP: Closing replication queue.
      [Note] WSREP: Closing slave action queue.
      [Note] WSREP: /opt/bitnami/mariadb/sbin/mysqld: Terminated.
      

      Attachments

        1. datadir.tgz
          3.67 MB
        2. node1.cnf
          0.8 kB
        3. node2_after_prepare.tgz
          812 kB
        4. node2_before_prepare.tgz
          818 kB
        5. node2.cnf
          0.8 kB

        Issue Links

          Activity

            People

              janlindstrom Jan Lindström
              ihti Ihtisham ul Haq
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.