Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-32974

Member fails to join due to old seqno in GTID

Details

    Description

      After upgrading initially from 10.6->10.11->11.0.3 and now to 11.0.4 we see that 2 of the members start up without any issues but the 3rd member(db-0) fails to start up due to an old sequence no.(117376) that i believe is being passed on from the donor.

      We are unable to find the old seqno anywhere except in the `ibdata1` file of the donor(by searching for the hex of it). But not sure how to get rid of this old seqno.
      What we did try is starting db-0 with fresh volume by removing K8s PVC for it, so it does a SST but that fails as well.

      Logs from member db-0:

      [Note] WSREP: SST received
      [Note] WSREP: Server status change joiner -> initializing
      [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      [Note] mysqld: Aria engine: starting recovery
      recovered pages: 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% (0.0 seconds); tables to flush: 2 1(0.0 seconds); 
      [Note] mysqld: Aria engine: recovery done
      [Note] InnoDB: Compressed tables use zlib 1.2.11
      [Note] InnoDB: Number of transaction pools: 1
      [Note] InnoDB: Using crc32 + pclmulqdq instructions
      [Note] InnoDB: Using Linux native AIO
      [Note] InnoDB: Initializing buffer pool, total size = 2.000GiB, chunk size = 32.000MiB
      [Note] InnoDB: Completed initialization of buffer pool
      [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
      [Note] InnoDB: End of log at LSN=180964319
      [Note] InnoDB: Resizing redo log from 12.016KiB to 96.000MiB; LSN=180964319
      [Note] InnoDB: File system buffers for log disabled (block size=512 bytes)
      [Note] InnoDB: Reinitializing innodb_undo_tablespaces= 3 from 0
      [Note] InnoDB: Data file .//undo001 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo001 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: Data file .//undo002 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo002 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: Data file .//undo003 did not exist: new to be created
      [Note] InnoDB: Setting file .//undo003 size to 10.000MiB
      [Note] InnoDB: Database physically writes the file full: wait...
      [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
      [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      [Note] InnoDB: log sequence number 180964319; transaction id 73558
      [Note] InnoDB: Loading buffer pool(s) from /bitnami/mariadb/data/ib_buffer_pool
      [Note] InnoDB: Cannot open '/bitnami/mariadb/data/ib_buffer_pool' for reading: No such file or directory
      [Note] Plugin 'FEEDBACK' is disabled.
      [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      [Note] Recovering after a crash using mysql-bin
      [Note] Starting table crash recovery...
      [Note] Crash table recovery finished.
      [Note] Server socket created on IP: '0.0.0.0'.
      [Warning] 'proxies_priv' entry '@% root@db-0' ignored in --skip-name-resolve mode.
      [Note] WSREP: wsrep_init_schema_and_SR (nil)
      [Note] WSREP: Server initialized
      [Note] WSREP: Server status change initializing -> initialized
      [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      [Note] WSREP: Recovered position from storage: 6aa53efc-db72-11ec-880f-a282ce494905:117376
      [Note] WSREP: Starting applier thread 6
      [Note] WSREP: Starting applier thread 7
      [Note] WSREP: Starting applier thread 8
      [Note] WSREP: Recovered view from SST:
        id: 6aa53efc-db72-11ec-880f-a282ce494905:128934
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 0
        members(3):
      	0: 46250027-95c8-11ee-9d94-06eb82413a07, db
      	1: 4ed771b5-941f-11ee-82bf-56ec952abcf6, db
      	2: bfa6e688-941f-11ee-97c5-d323f02068c1, db
       
      [ERROR] WSREP: sst_received failed: SST script passed bogus GTID: 6aa53efc-db72-11ec-880f-a282ce494905:117376. Preceding view GTID: 6aa53efc-db72-11ec-880f-a282ce494905:128934
      [Note] WSREP: SST received: 00000000-0000-0000-0000-000000000000:-1
      [Note] WSREP: Joiner monitor thread ended with total time 14 sec
      [ERROR] WSREP: Application received wrong state: 
      	Received: 00000000-0000-0000-0000-000000000000
      	Required: 6aa53efc-db72-11ec-880f-a282ce494905
      [ERROR] WSREP: Application state transfer failed. This is unrecoverable condition, restart required.
      [Note] WSREP: ReplicatorSMM::abort()
      [Note] WSREP: Closing send monitor...
      [Note] WSREP: Closed send monitor.
      [Note] WSREP: gcomm: terminating thread
      [Note] WSREP: gcomm: joining thread
      [Note] WSREP: gcomm: closing backend
      [Note] /opt/bitnami/mariadb/sbin/mysqld: ready for connections.
      Version: '11.0.3-MariaDB-log'  socket: '/opt/bitnami/mariadb/tmp/mysql.sock'  port: 3306  Source distribution
      [Note] WSREP: view(view_id(NON_PRIM,46250027-9d94,6636) memb {
      	46250027-9d94,0
      } joined {
      } left {
      } partitioned {
      	4ed771b5-82bf,0
      	bfa6e688-97c5,0
      })
      [Note] WSREP: PC protocol downgrade 1 -> 0
      [Note] WSREP: view((empty))
      [Note] WSREP: gcomm: closed
      [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
      [Note] WSREP: Flow-control interval: [128, 160]
      [Note] WSREP: Received NON-PRIMARY.
      [Note] WSREP: Shifting JOINER -> OPEN (TO: 128934)
      [Note] WSREP: New SELF-LEAVE.
      [Note] WSREP: Flow-control interval: [0, 0]
      [Note] WSREP: Received SELF-LEAVE. Closing connection.
      [Note] WSREP: Shifting OPEN -> CLOSED (TO: 128934)
      [Note] WSREP: RECV thread exiting 0: Success
      [Note] WSREP: recv_thread() joined.
      [Note] WSREP: Closing replication queue.
      [Note] WSREP: Closing slave action queue.
      [Note] WSREP: /opt/bitnami/mariadb/sbin/mysqld: Terminated.
      

      Attachments

        1. datadir.tgz
          3.67 MB
          Jan Lindström
        2. node2_before_prepare.tgz
          818 kB
          Jan Lindström
        3. node2_after_prepare.tgz
          812 kB
          Jan Lindström
        4. node1.cnf
          0.8 kB
          Jan Lindström
        5. node2.cnf
          0.8 kB
          Jan Lindström

        Issue Links

          Activity

            Transition Time In Source Status Execution Times
            Jan Lindström made transition -
            Open Confirmed
            82d 23h 12m 1
            Jan Lindström made transition -
            Confirmed In Progress
            7d 18h 1m 1
            Jan Lindström made transition -
            Stalled In Review
            18d 2h 40m 3
            Marko Mäkelä made transition -
            In Review Stalled
            5d 6h 29m 5
            Jan Lindström made transition -
            Stalled In Progress
            5d 16h 6m 2
            Jan Lindström made transition -
            In Progress In Review
            5d 1m 3
            Marko Mäkelä made transition -
            In Review Closed
            2h 1

            People

              janlindstrom Jan Lindström
              ihti Ihtisham ul Haq
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.