Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35080

mariabackup: position saved in xtrabackup_binlog_info is incorrect for replication

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.6.19
    • None
    • Replication
    • None

    Description

      Hello,

      The bug that is supposed to be fixed in MDEV-23080 does not appear to be fixed.

      I try to set up an asynchronous replica with --no-lock enabled. The command I use to create the backup:

      mariabackup --innobackupex --no-lock --stream=xbstream --parallel=4 --port=... -host=... --socket=... --password=... --skip-rocksdb-backup /tmp

      The source system is a quiet reader node in a galera cluster with some traffic (not very busy, but a few dozen of transactions per second). I have experienced the same issue on writer node as well. I am not sure if it is related to Galera but I find it not likely. This cluster has data at rest encryption enabled, but I have seen the same issue on clusters without.

      Please tell me if this log output is sufficient. I can relatively easily reproduce the issue and/or add debug information or include more log files.

      The first and last bit of the backup output:

      root@mdb5:/data/data# head -n 30 setup_replication_backup_output.log
      		 
      241004 18:49:11 innobackupex: Starting the backup operation
      		 
       
      		 
      IMPORTANT: Please check that the backup run completes successfully.
      		 
                 At the end of a successful backup run innobackupex
      		 
                 prints "completed OK!".
      		 
       
      		 
      [00] 2024-10-04 18:49:11 Connecting to MariaDB server host: localhost, user: root, password: set, port: 3306, socket: /data/data/mysql.sock
      		 
      [00] 2024-10-04 18:49:11 Using server version 10.6.19-MariaDB-ubu2004-log
      		 
      --innobackupex based on MariaDB server 10.6.19-MariaDB debian-linux-gnu (x86_64)
      		 
      [00] 2024-10-04 18:49:11 uses posix_fadvise().
      		 
      [00] 2024-10-04 18:49:11 cd to /data/data/
      		 
      [00] 2024-10-04 18:49:11 Loading encryption plugin
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_endpoint_url='
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_key_spec=AES_128'
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_log_level=Off'
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_master_key_id=********************************'
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_region=eu-central-1'
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_request_timeout=0'
      		 
      [00] 2024-10-04 18:49:11         Encryption plugin parameter :  '--aws_key_management_rotate_key=0'
      		 
      [00] 2024-10-04 18:49:11 open files limit requested 100000, set to 100000
      		 
      [00] 2024-10-04 18:49:11 mariabackup: using the following InnoDB configuration:
      		 
      [00] 2024-10-04 18:49:11 innodb_data_home_dir = 
      [00] 2024-10-04 18:49:11 innodb_data_file_path = ibdata1:12M:autoextend
      		 
      [00] 2024-10-04 18:49:11 innodb_log_group_home_dir = /data/data
      		 
      [00] 2024-10-04 18:49:11 InnoDB: Using Linux native AIO
      		 
      2024-10-04 18:49:11 0 [Note] InnoDB: Number of pools: 1
      		 
      [00] 2024-10-04 18:49:11 mariabackup: Generating a list of tablespaces
      		 
      [00] 2024-10-04 18:49:14 mariabackup: Starting 4 threads for parallel data files transfer
      		 
      [01] 2024-10-04 18:49:14 Streaming ./first_customer_db/messagesent.ibd
      		 
      [02] 2024-10-04 18:49:14 Streaming ./first_customer_db/selection.ibd
      		 
      ............................................
      		 
      [01] 2024-10-04 19:19:39 Streaming ./performance_schema/db.opt to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [01] 2024-10-04 19:19:39 Streaming ./customer_db_name/db.opt to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [00] 2024-10-04 19:19:39 Finished backing up non-InnoDB tables and files
      		 
      [01] 2024-10-04 19:19:39 Streaming ./aws-kms-key.1.1 to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [01] 2024-10-04 19:19:39 Streaming ./aws-kms-key.2.1 to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [01] 2024-10-04 19:19:39 Streaming /data/data//aria_log_control to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [01] 2024-10-04 19:19:39 Streaming /data/data//aria_log.00000001 to <STDOUT>
      		 
      [01] 2024-10-04 19:19:39         ...done
      		 
      [00] 2024-10-04 19:19:39 Waiting for log copy thread to read lsn 0
      		 
      [00] 2024-10-04 19:19:39 Streaming xtrabackup_binlog_info
      		 
      [00] 2024-10-04 19:19:39         ...done
      		 
      [00] 2024-10-04 19:19:39 mariabackup: The latest check point (for incremental): '1181315416946'
      		 
      mariabackup: Stopping log copying thread.
      		 
      [00] 2024-10-04 19:19:40 Backup created in directory '/tmp/'
      		 
      [00] 2024-10-04 19:19:40 MySQL binlog position: filename 'mysql-bin.001719', position '923267527', GTID of the last change '0-1-647038260,3419099865-2545419552-6155'
      		 
      [00] 2024-10-04 19:19:40 Streaming backup-my.cnf
      		 
      [00] 2024-10-04 19:19:40         ...done
      		 
      [00] 2024-10-04 19:19:40 Streaming xtrabackup_info
      		 
      [00] 2024-10-04 19:19:40         ...done
      		 
      [00] 2024-10-04 19:19:40 Redo log (from LSN 1181129627031 to 1181493837646) was copied.

      The prepare output:

      root@mdb5:/data/data# cat setup_replication_prepare_output.log 
      mariabackup based on MariaDB server 10.6.19-MariaDB debian-linux-gnu (x86_64)
      		 
      [00] 2024-10-04 19:19:53 cd to /data/data/
      		 
      [00] 2024-10-04 19:19:53 open files limit requested 0, set to 1024
      		 
      [00] 2024-10-04 19:19:53 Loading encryption plugin from aws_key_management=aws_key_management
      		 
      [00] 2024-10-04 19:19:53 Loading encryption plugin
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--plugin_load=aws_key_management=aws_key_management'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_endpoint_url='
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_key_spec=AES_128'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_log_level=Off'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_master_key_id=********'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_region=eu-central-1'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_request_timeout=0'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--aws_key_management_rotate_key=0'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--innodb_encrypt_log=1'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--innodb_encrypt_tables=ON'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--prepare'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--use-memory=5952M'
      		 
      [00] 2024-10-04 19:19:53 	 Encryption plugin parameter :  '--target-dir=/data/data'
      		 
      [00] 2024-10-04 19:19:53 This target seems to be not prepared yet.
      		 
      [00] 2024-10-04 19:19:53 mariabackup: using the following InnoDB configuration for recovery:
      		 
      [00] 2024-10-04 19:19:53 innodb_data_home_dir = .
      		 
      [00] 2024-10-04 19:19:53 innodb_data_file_path = ibdata1:12M:autoextend
      		 
      [00] 2024-10-04 19:19:53 innodb_log_group_home_dir = .
      		 
      [00] 2024-10-04 19:19:53 InnoDB: Using Linux native AIO
      		 
      [00] 2024-10-04 19:19:53 Starting InnoDB instance for recovery.
      		 
      [00] 2024-10-04 19:19:53 mariabackup: Using 6241124352 bytes for buffer pool (set by --use-memory parameter)
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Using transactional memory
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Number of pools: 1
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Using Linux native AIO
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Initializing buffer pool, total size = 6241124352, chunk size = 6241124352
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Completed initialization of buffer pool
      		 
      2024-10-04 19:19:53 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1181129627031,1181306446426
      		 
      2024-10-04 19:19:55 0 [Note] InnoDB: To recover: 66550 pages
      		 
      2024-10-04 19:19:57 0 [Note] InnoDB: Last binlog file '/data/binlogs/mysql-bin.001719', position 923304079
      		 
      [00] 2024-10-04 19:19:57 Last binlog file /data/binlogs/mysql-bin.001719, position 923304079
      		 
      [00] 2024-10-04 19:19:57 mariabackup: Recovered WSREP position: 703c6d58-48d6-11ef-838c-723637965529:84305845 domain_id: 0

      The xtrabackup_binlog_info:

      root@mdb5:/data/data# cat xtrabackup_binlog_info 
      mysql-bin.001719	923267527	0-1-647038260,3419099865-2545419552-6155

      Thank you,

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MDEV-23080 mariabackup: position saved in xtrabackup_binlog_info is incorrect for replication

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              Unassigned Unassigned
              michaeldg Michaël de groot
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.