Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20983

Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored

Details

    Description

      hi

      In a Mariadb replication setup (1 master, 1 slave)

      We got an
      InnoDB: Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored [3690179534:3690179534] key_version

      for no obvious reason, no HW failure or anything noticeable at OS level.

      Then a bit later : Got error 192 when reading table

      We had to switch role to let the application operate again. That went OK, although we lost the rows that could not be replicated for the corrupted table.

      After a while, the old master that was promoted to slaved crashed an could not be restarted, besides using recovery level 6.

      We ended resyncing with full backup but the issue is scary.

      Asking for analysis and feedback.

      Thank you
      Emmanuel

      Attachments

        1. before_innocrash_20191105.txt
          10 kB
          Emmanuel Torre
        2. innocrash_20191105.txt
          26 kB
          Emmanuel Torre

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. MDEV-12112 corruption in encrypted table may be overlooked

          • Major - Major loss of function.
          • Closed

          Activity

            Descending order - Click to sort in ascending order
            etorre Emmanuel Torre added a comment -

            thank you for the responses and explanations. We willl naturally upgrade.

            etorre Emmanuel Torre added a comment - thank you for the responses and explanations. We willl naturally upgrade.
            marko Marko Mäkelä added a comment -

            The first line of before_innocrash_20191105.txt is:

            2019-11-04 12:55:17 140692352591616 [ERROR] InnoDB:  Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored [3690179534:3690179534] key_version 1

            Note: it is showing the same checksum in all 3 places. This message is a telltale sign of severely broken logic. The code wrongly assumed that it would be impossible for the checksum of the data payload to be the same both before and after encryption. In fact, the probability of that is at least 1/4,294,967,296 (actually much more, due to checksum sloppiness issues that were addressed in MDEV-17957 and MDEV-17958).

            The broken logic was fixed in a follow-up fix to MDEV-12112, in MariaDB 10.1.38, 10.2.20, 10.3.12.

            marko Marko Mäkelä added a comment - The first line of before_innocrash_20191105.txt is: 2019-11-04 12:55:17 140692352591616 [ERROR] InnoDB: Page 38:4016 may be corrupted. Post encryption checksum 3690179534 stored [3690179534:3690179534] key_version 1 Note: it is showing the same checksum in all 3 places. This message is a telltale sign of severely broken logic. The code wrongly assumed that it would be impossible for the checksum of the data payload to be the same both before and after encryption. In fact, the probability of that is at least 1/4,294,967,296 (actually much more, due to checksum sloppiness issues that were addressed in MDEV-17957 and MDEV-17958 ). The broken logic was fixed in a follow-up fix to MDEV-12112 , in MariaDB 10.1.38, 10.2.20, 10.3.12.
            elenst Elena Stepanova added a comment - - edited

            10.2.8 is over 2 years old and 20 releases behind the current one. There have been numerous bugfixes since then, specifically around encryption. Please upgrade to a current version and report a bug against it if the problem persists.

            elenst Elena Stepanova added a comment - - edited 10.2.8 is over 2 years old and 20 releases behind the current one. There have been numerous bugfixes since then, specifically around encryption. Please upgrade to a current version and report a bug against it if the problem persists.

            People

              Unassigned Unassigned
              etorre Emmanuel Torre
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.