Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-22725

"Unable to find a record to delete-mark" errors lead to divergent slave when replicating from Enterprise to Community

    XMLWordPrintable

    Details

      Description

      User reports data inconsistency discovered on 10.4.12 servers installed with an Enterprise master replicating to both Enterprise and Community slaves. Inconsistency only happens on Community slaves.

      Master was populated form a logical dump, and slaves were replicated from a clean state using mixed-mode replication with FULL binlog_row_image. Enterprise and community slaves configured similarly, and these errors appear on the community slave, leading to the discrepancy:

      2020-04-15 23:21:21 1 [ERROR] InnoDB: Unable to find a record to delete-mark
      InnoDB: tuple DATA TUPLE: 2 fields;
       0: len 12; hex 2b3137373532393833363036; asc +17752983606;;
       1: len 8; hex 0000000026da4b0e; asc     & K ;;
       
      InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0
       0: len 12; hex 2b3137373532393833313432; asc +17752983142;;
       1: len 8; hex 00000000061b52ee; asc       R ;;
      2020-04-15 23:21:21 1 [ERROR] InnoDB: page [page id: space=23, page number=13542] (490 records, index id 60).
      2020-04-15 23:21:21 1 [ERROR] InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
      

      All errors report the same "InnoDB: record PHYSICAL RECORD: n_fields 2; compact format; info bits 0" with different values.

      Configuration differences between the Community server, getting the errors, on the left, and the Enterprise server, not seeing these errors, on the right:

      binlog_cache_size       32768                         | binlog_cache_size       262144
      binlog_stmt_cache_size  32768                         | binlog_stmt_cache_size  131072
      expire_logs_days        0                             | expire_logs_days        7
      general_log_file                  XXXXXdb-01a.log     | general_log_file                XXXXXdb-02a.log
      gtid_binlog_pos                                       | gtid_binlog_pos 0-1-9847781
      gtid_binlog_state                                     | gtid_binlog_state       0-1769-6341,0-1-9847781
      gtid_current_pos        0-1-9846831                   | gtid_current_pos        0-1-9847781
      gtid_slave_pos  0-1-9846831                           | gtid_slave_pos  0-1-9847781
      gtid_strict_mode        OFF                           | gtid_strict_mode        ON
      have_symlink    YES                                   | have_symlink    DISABLED
      host_cache_size 635                                   | host_cache_size 653
      hostname        XXXdb-01a   | hostname        XXXdb-02a
      innodb_buffer_pool_size 1073741824                    | innodb_buffer_pool_size 2147483648
      innodb_checksum_algorithm       crc32                 | innodb_checksum_algorithm       full_crc32
      innodb_checksums        ON                            <
      innodb_log_buffer_size  16777216                      | innodb_log_buffer_size  67108864
      innodb_max_dirty_pages_pct      75.000000             | innodb_max_dirty_pages_pct      50.000000
      innodb_read_io_threads  4                             | innodb_read_io_threads  8
      innodb_stats_sample_pages       8                     | innodb_stats_sample_pages       32
      innodb_stats_transient_sample_pages     8             | innodb_stats_transient_sample_pages     32
      innodb_write_io_threads 4                             | innodb_write_io_threads 8
      join_buffer_size        262144                        | join_buffer_size        16777216
      join_buffer_space_limit 2097152                       | join_buffer_space_limit 33554432
      join_cache_level        2                             | join_cache_level        6
      key_buffer_size 134217728                             | key_buffer_size 67108864
      local_infile    ON                                    | local_infile    OFF
      log_bin OFF                                           | log_bin ON
      log_bin_basename                                      | log_bin_basename        /mnt/data/mariadb/mariadb-bin
      log_bin_index                                         | log_bin_index   /mnt/data/mariadb/mariadb-bin.index
      log_bin_trust_function_creators OFF                   | log_bin_trust_function_creators ON
      log_error       /mnt/data/mariadb/XXXXdb-01a | log_error       /mnt/data/mariadb/XXXXXdb-02a
      log_slave_updates       OFF                           | log_slave_updates       ON
      max_connections 650                                   | max_connections 1000
      max_heap_table_size     16777216                      | max_heap_table_size     33554432
      myisam_recover_options  BACKUP,QUICK                  | myisam_recover_options  BACKUP,FORCE
      myisam_sort_buffer_size 134216704                     | myisam_sort_buffer_size 67108864
              rowid_filter=off,                             |         rowid_filter=on,
      pid_file        /mnt/data/mariadb/XXXXXdb-01a.      | pid_file        /mnt/data/mariadb/XXXXXdb-02a
      rpl_semi_sync_master_enabled    OFF                   | rpl_semi_sync_master_enabled    ON
      rpl_semi_sync_master_wait_point AFTER_COMMIT          | rpl_semi_sync_master_wait_point AFTER_SYNC
      rpl_semi_sync_slave_enabled     OFF                   | rpl_semi_sync_slave_enabled     ON
                                                            > s3_access_key
                                                            > s3_block_size   4194304
                                                            > s3_bucket       MariaDB
                                                            > s3_debug        OFF
                                                            > s3_host_name    s3.amazonaws.com
                                                            > s3_pagecache_age_threshold      300
                                                            > s3_pagecache_buffer_size        134217728
                                                            > s3_pagecache_division_limit     100
                                                            > s3_pagecache_file_hash_size     512
                                                            > s3_protocol_version     Auto
                                                            > s3_region
                                                            > s3_secret_key
      server_audit_excl_users root, openser, ap_backup, ap_ | server_audit_excl_users
      server_id       1813                                  | server_id       1769
      slow_query_log_file               XXXXXdb-01a-slow.lo | slow_query_log_file            XXXXXdb-02a-slow.log
      sql_mode        STRICT_TRANS_TABLES,ERROR_FOR_DIVISIO | sql_mode        STRICT_TRANS_TABLES
      sync_binlog     0                                     | sync_binlog     1
      tmp_memory_table_size   16777216                      | tmp_memory_table_size   33554432
      tmp_table_size  16777216                              | tmp_table_size  33554432
      version 10.4.12-MariaDB-log                           | version 10.4.12-6-MariaDB-enterprise-log
      version_comment MariaDB Server                        | version_comment MariaDB Enterprise Server
      version_source_revision ba6bfc402c352372cc1a9ec20b5dc | version_source_revision 86a634a0feaf7788c9bcf7cc763e5
      wsrep_node_name                XXXXXdb-01a            | wsrep_node_name                XXXXXdb-02a         
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              marko Marko Mäkelä
              Reporter:
              juan.vera Juan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated: