Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-28870

InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT during crash recovery

Details

    Description

      origin/10.6 5bb90cb2ac7b93f40c8ca4672ec7e2c18f0ea238 2022-06-16T10:01:29+03:00
      Scenario:
      1. Start the server and load some initial data.
      2. Several sessions run a DDL/DML mix
      3. During 2. is ongoing the server gets intentional killed (SIGSEGV) 
      4. The restart attempt fails with
      [rr 1379888 3340]2022-06-16 13:05:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=80040121,81223111
      [rr 1379888 5048]2022-06-16 13:05:16 0 [ERROR] InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT for tablespace 88
      [rr 1379888 5052]2022-06-16 13:05:16 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1447] with error Data structure corruption
       
      pluto:/data/results/1655382284/TBR-1547
      _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysqld-1
           # Fate of the server till the SIGKILL arrives.
      _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysql-2
          # The restart attempt
      pluto:/data/results/1655382284/TBR-1547/1/fbackup
          # File backup of the server datadirectory made after the SIGKILL
          # and before the restart attempt.
       
      RQG
      ===
      # git clone https://github.com/mleich1/rqg --branch experimental RQG
      #
      # GIT_SHOW: HEAD -> experimental, origin/experimental a32979c6d789ddf0982e6a56af687565be00590c 2022-06-14T17:21:56+02:00
      # rqg.pl  : Version 4.0.6 (2022-05)
      #
      # $RQG_HOME/rqg.pl \
      # --grammar=conf/mariadb/innodb_compression_encryption.yy \
      # --gendata=conf/mariadb/innodb_compression_encryption.zz \
      # --max_gd_duration=1800 \
      # --mysqld=--innodb-encrypt-log \
      # --mysqld=--innodb-encrypt-tables \
      # --reporters=CrashRecovery1 \
      # --redefine=conf/mariadb/redefine_innodb_undo.yy \
      # --mysqld=--innodb-immediate-scrub-data-uncompressed=1 \
      # --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
      # --mysqld=--loose-idle_write_transaction_timeout=0 \
      # --mysqld=--loose-idle_transaction_timeout=0 \
      # --mysqld=--loose-idle_readonly_transaction_timeout=0 \
      # --mysqld=--connect_timeout=60 \
      # --mysqld=--interactive_timeout=28800 \
      # --mysqld=--slave_net_timeout=60 \
      # --mysqld=--net_read_timeout=30 \
      # --mysqld=--net_write_timeout=60 \
      # --mysqld=--loose-table_lock_wait_timeout=50 \
      # --mysqld=--wait_timeout=28800 \
      # --mysqld=--lock-wait-timeout=86400 \
      # --mysqld=--innodb-lock-wait-timeout=50 \
      # --no-mask \
      # --queries=10000000 \
      # --seed=random \
      # --reporters=Backtrace \
      # --reporters=ErrorLog \
      # --reporters=Deadlock1 \
      # --validators=None \
      # --mysqld=--log_output=none \
      # --mysqld=--log_bin_trust_function_creators=1 \
      # --mysqld=--loose-debug_assert_on_not_freed_memory=0 \
      # --engine=InnoDB \
      # --restart_timeout=240 \
      # --mysqld=--plugin-load-add=file_key_management.so \
      # --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
      # --mysqld=--plugin-load-add=provider_lzo.so \
      # --mysqld=--plugin-load-add=provider_bzip2.so \
      # --mysqld=--plugin-load-add=provider_lzma.so \
      # --mysqld=--plugin-load-add=provider_snappy.so \
      # --mysqld=--plugin-load-add=provider_lz4.so \
      # --duration=300 \
      # --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
      # --mysqld=--innodb_file_per_table=1 \
      # --mysqld=--loose-innodb_read_only_compressed=OFF \
      # --mysqld=--loose-innodb-sync-debug \
      # --mysqld=--innodb_stats_persistent=off \
      # --mysqld=--innodb_adaptive_hash_index=on \
      # --mysqld=--log-bin \
      # --mysqld=--sync-binlog=1 \
      # --mysqld=--loose-innodb_evict_tables_on_commit_debug=on \
      # --mysqld=--loose-max-statement-time=30 \
      # --threads=33 \
      # --mysqld=--innodb-use-native-aio=0 \
      # --mysqld=--loose-gdb \
      # --mysqld=--loose-debug-gdb \
      # --rr=Extended \
      # --rr_options=--chaos --wait \
      # --mysqld=--loose_innodb_change_buffering=purges \
      # --mysqld=--innodb_rollback_on_timeout=OFF \
      # --vardir_type=slow \
      # --mysqld=--innodb_page_size=16K \
      # --mysqld=--innodb-buffer-pool-size=256M \
      # --no_mask \
      # <local settings>
      

      Attachments

        Issue Links

          Activity

            mleich Matthias Leich created issue -
            marko Marko Mäkelä made changes -
            Field Original Value New Value
            marko Marko Mäkelä made changes -
            Assignee Marko Mäkelä [ marko ] Thirunarayanan Balathandayuthapani [ thiru ]
            Description
            {noformat}
            origin/10.6 5bb90cb2ac7b93f40c8ca4672ec7e2c18f0ea238 2022-06-16T10:01:29+03:00
            Scenario:
            1. Start the server and load some initial data.
            2. Several sessions run a DDL/DML mix
            3. During 2. is ongoing the server gets intentional killed (SIGSEGV)
            4. The restart attempt fails with
            [rr 1379888 3340]2022-06-16 13:05:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=80040121,81223111
            [rr 1379888 5048]2022-06-16 13:05:16 0 [ERROR] InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT for tablespace 88
            [rr 1379888 5052]2022-06-16 13:05:16 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1447] with error Data structure corruption

            pluto:/data/results/1655382284/TBR-1547
            _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysqld-1
                 # Fate of the server till the SIGKILL arrives.
            _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysql-2
                # The restart attempt
            pluto:/data/results/1655382284/TBR-1547/1/fbackup
                # File backup of the server datadirectory made after the SIGKILL
                # and before the restart attempt.

            RQG
            ===
            # git clone https://github.com/mleich1/rqg --branch experimental RQG
            #
            # GIT_SHOW: HEAD -> experimental, origin/experimental a32979c6d789ddf0982e6a56af687565be00590c 2022-06-14T17:21:56+02:00
            # rqg.pl : Version 4.0.6 (2022-05)
            #
            # $RQG_HOME/rqg.pl \
            # --grammar=conf/mariadb/innodb_compression_encryption.yy \
            # --gendata=conf/mariadb/innodb_compression_encryption.zz \
            # --max_gd_duration=1800 \
            # --mysqld=--innodb-encrypt-log \
            # --mysqld=--innodb-encrypt-tables \
            # --reporters=CrashRecovery1 \
            # --redefine=conf/mariadb/redefine_innodb_undo.yy \
            # --mysqld=--innodb-immediate-scrub-data-uncompressed=1 \
            # --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
            # --mysqld=--loose-idle_write_transaction_timeout=0 \
            # --mysqld=--loose-idle_transaction_timeout=0 \
            # --mysqld=--loose-idle_readonly_transaction_timeout=0 \
            # --mysqld=--connect_timeout=60 \
            # --mysqld=--interactive_timeout=28800 \
            # --mysqld=--slave_net_timeout=60 \
            # --mysqld=--net_read_timeout=30 \
            # --mysqld=--net_write_timeout=60 \
            # --mysqld=--loose-table_lock_wait_timeout=50 \
            # --mysqld=--wait_timeout=28800 \
            # --mysqld=--lock-wait-timeout=86400 \
            # --mysqld=--innodb-lock-wait-timeout=50 \
            # --no-mask \
            # --queries=10000000 \
            # --seed=random \
            # --reporters=Backtrace \
            # --reporters=ErrorLog \
            # --reporters=Deadlock1 \
            # --validators=None \
            # --mysqld=--log_output=none \
            # --mysqld=--log_bin_trust_function_creators=1 \
            # --mysqld=--loose-debug_assert_on_not_freed_memory=0 \
            # --engine=InnoDB \
            # --restart_timeout=240 \
            # --mysqld=--plugin-load-add=file_key_management.so \
            # --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
            # --mysqld=--plugin-load-add=provider_lzo.so \
            # --mysqld=--plugin-load-add=provider_bzip2.so \
            # --mysqld=--plugin-load-add=provider_lzma.so \
            # --mysqld=--plugin-load-add=provider_snappy.so \
            # --mysqld=--plugin-load-add=provider_lz4.so \
            # --duration=300 \
            # --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
            # --mysqld=--innodb_file_per_table=1 \
            # --mysqld=--loose-innodb_read_only_compressed=OFF \
            # --mysqld=--loose-innodb-sync-debug \
            # --mysqld=--innodb_stats_persistent=off \
            # --mysqld=--innodb_adaptive_hash_index=on \
            # --mysqld=--log-bin \
            # --mysqld=--sync-binlog=1 \
            # --mysqld=--loose-innodb_evict_tables_on_commit_debug=on \
            # --mysqld=--loose-max-statement-time=30 \
            # --threads=33 \
            # --mysqld=--innodb-use-native-aio=0 \
            # --mysqld=--loose-gdb \
            # --mysqld=--loose-debug-gdb \
            # --rr=Extended \
            # --rr_options=--chaos --wait \
            # --mysqld=--loose_innodb_change_buffering=purges \
            # --mysqld=--innodb_rollback_on_timeout=OFF \
            # --vardir_type=slow \
            # --mysqld=--innodb_page_size=16K \
            # --mysqld=--innodb-buffer-pool-size=256M \
            # --no_mask \
            # <local settings>
            {noformat}
            {noformat}
            origin/10.6 5bb90cb2ac7b93f40c8ca4672ec7e2c18f0ea238 2022-06-16T10:01:29+03:00
            Scenario:
            1. Start the server and load some initial data.
            2. Several sessions run a DDL/DML mix
            3. During 2. is ongoing the server gets intentional killed (SIGSEGV)
            4. The restart attempt fails with
            [rr 1379888 3340]2022-06-16 13:05:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=80040121,81223111
            [rr 1379888 5048]2022-06-16 13:05:16 0 [ERROR] InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT for tablespace 88
            [rr 1379888 5052]2022-06-16 13:05:16 0 [ERROR] InnoDB: Plugin initialization aborted at srv0start.cc[1447] with error Data structure corruption

            pluto:/data/results/1655382284/TBR-1547
            _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysqld-1
                 # Fate of the server till the SIGKILL arrives.
            _RR_TRACE_DIR="./1/rr" rr replay --mark-stdio mysql-2
                # The restart attempt
            pluto:/data/results/1655382284/TBR-1547/1/fbackup
                # File backup of the server datadirectory made after the SIGKILL
                # and before the restart attempt.

            RQG
            ===
            # git clone https://github.com/mleich1/rqg --branch experimental RQG
            #
            # GIT_SHOW: HEAD -> experimental, origin/experimental a32979c6d789ddf0982e6a56af687565be00590c 2022-06-14T17:21:56+02:00
            # rqg.pl : Version 4.0.6 (2022-05)
            #
            # $RQG_HOME/rqg.pl \
            # --grammar=conf/mariadb/innodb_compression_encryption.yy \
            # --gendata=conf/mariadb/innodb_compression_encryption.zz \
            # --max_gd_duration=1800 \
            # --mysqld=--innodb-encrypt-log \
            # --mysqld=--innodb-encrypt-tables \
            # --reporters=CrashRecovery1 \
            # --redefine=conf/mariadb/redefine_innodb_undo.yy \
            # --mysqld=--innodb-immediate-scrub-data-uncompressed=1 \
            # --mysqld=--loose-innodb_lock_schedule_algorithm=fcfs \
            # --mysqld=--loose-idle_write_transaction_timeout=0 \
            # --mysqld=--loose-idle_transaction_timeout=0 \
            # --mysqld=--loose-idle_readonly_transaction_timeout=0 \
            # --mysqld=--connect_timeout=60 \
            # --mysqld=--interactive_timeout=28800 \
            # --mysqld=--slave_net_timeout=60 \
            # --mysqld=--net_read_timeout=30 \
            # --mysqld=--net_write_timeout=60 \
            # --mysqld=--loose-table_lock_wait_timeout=50 \
            # --mysqld=--wait_timeout=28800 \
            # --mysqld=--lock-wait-timeout=86400 \
            # --mysqld=--innodb-lock-wait-timeout=50 \
            # --no-mask \
            # --queries=10000000 \
            # --seed=random \
            # --reporters=Backtrace \
            # --reporters=ErrorLog \
            # --reporters=Deadlock1 \
            # --validators=None \
            # --mysqld=--log_output=none \
            # --mysqld=--log_bin_trust_function_creators=1 \
            # --mysqld=--loose-debug_assert_on_not_freed_memory=0 \
            # --engine=InnoDB \
            # --restart_timeout=240 \
            # --mysqld=--plugin-load-add=file_key_management.so \
            # --mysqld=--loose-file-key-management-filename=$RQG_HOME/conf/mariadb/encryption_keys.txt \
            # --mysqld=--plugin-load-add=provider_lzo.so \
            # --mysqld=--plugin-load-add=provider_bzip2.so \
            # --mysqld=--plugin-load-add=provider_lzma.so \
            # --mysqld=--plugin-load-add=provider_snappy.so \
            # --mysqld=--plugin-load-add=provider_lz4.so \
            # --duration=300 \
            # --mysqld=--loose-innodb_fatal_semaphore_wait_threshold=300 \
            # --mysqld=--innodb_file_per_table=1 \
            # --mysqld=--loose-innodb_read_only_compressed=OFF \
            # --mysqld=--loose-innodb-sync-debug \
            # --mysqld=--innodb_stats_persistent=off \
            # --mysqld=--innodb_adaptive_hash_index=on \
            # --mysqld=--log-bin \
            # --mysqld=--sync-binlog=1 \
            # --mysqld=--loose-innodb_evict_tables_on_commit_debug=on \
            # --mysqld=--loose-max-statement-time=30 \
            # --threads=33 \
            # --mysqld=--innodb-use-native-aio=0 \
            # --mysqld=--loose-gdb \
            # --mysqld=--loose-debug-gdb \
            # --rr=Extended \
            # --rr_options=--chaos --wait \
            # --mysqld=--loose_innodb_change_buffering=purges \
            # --mysqld=--innodb_rollback_on_timeout=OFF \
            # --vardir_type=slow \
            # --mysqld=--innodb_page_size=16K \
            # --mysqld=--innodb-buffer-pool-size=256M \
            # --no_mask \
            # <local settings>
            {noformat}
            thiru Thirunarayanan Balathandayuthapani made changes -
            Assignee Thirunarayanan Balathandayuthapani [ thiru ] Marko Mäkelä [ marko ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            Fix Version/s 10.9 [ 26905 ]
            Fix Version/s 10.10 [ 27530 ]
            Affects Version/s 10.5 [ 23123 ]
            Affects Version/s 10.6 [ 24028 ]
            Affects Version/s 10.7 [ 24805 ]
            Affects Version/s 10.8 [ 26121 ]
            Affects Version/s 10.9 [ 26905 ]
            Affects Version/s 10.10 [ 27530 ]
            Affects Version/s 10.6.9 [ 27507 ]
            Labels rr-profile corruption recovery rr-profile
            marko Marko Mäkelä made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            marko Marko Mäkelä made changes -
            issue.field.resolutiondate 2022-06-21 14:54:26.0 2022-06-21 14:54:26.808
            marko Marko Mäkelä made changes -
            Fix Version/s 10.6.9 [ 27507 ]
            Fix Version/s 10.7.5 [ 27505 ]
            Fix Version/s 10.8.4 [ 27503 ]
            Fix Version/s 10.9.2 [ 27115 ]
            Fix Version/s 10.10.1 [ 27913 ]
            Fix Version/s 10.5 [ 23123 ]
            Fix Version/s 10.6 [ 24028 ]
            Fix Version/s 10.7 [ 24805 ]
            Fix Version/s 10.8 [ 26121 ]
            Fix Version/s 10.9 [ 26905 ]
            Fix Version/s 10.10 [ 27530 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            marko Marko Mäkelä made changes -
            danblack Daniel Black made changes -
            marko Marko Mäkelä made changes -
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 203055
            Zendesk active tickets 203055
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk Related Tickets 203055 203055 203048
            Zendesk active tickets 203055 203055 203048
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk active tickets 203048 203055 203055
            mariadb-jira-automation Jira Automation (IT) made changes -
            Zendesk active tickets 203055

            People

              marko Marko Mäkelä
              mleich Matthias Leich
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.