Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-15519

CHECKSUM TABLE gives different results across MariaDB 10.0 and 10.2

Details

    • Bug
    • Status: Confirmed (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.0(EOL), 10.2(EOL)
    • 10.4(EOL)
    • Server
    • None
    • GNU/Linux

    Description

      According to https://mariadb.com/kb/en/library/checksum-table/ only older MySQL servers would yield different results.

      As it turns out, supported, widely used and deployed MariaDB instances will also return different results for the exact same table structure and data?!

      Comparing 10.0 (Ubuntu LTS repo release) and 10.2 (MariaDB official stable repo release).

      CHECKSUM TABLE badcs;
      

      MariaDB 10.0: 1873571331
      MariaDB 10.2: 3472699532

      Even worse, MySQL is different on it's own:
      MySQL 5.6.34: 260370412

      But I'm interested here in MariaDB only.
      It seems that MyISAM tables will produce same results across each server version/flavour, so all OK as expected with checksumming process done on exact same table/data.

      Only InnoDB will (mis)behave like this, thus effectively breaking entire point of CHECKSUM-ming data across servers.

      Isn't this suppose to be standard's compliant command?

      In attachment there's my sample table with few rows. So it's easy to be reproduced.

      Attachments

        Issue Links

          Activity

            I can't reproduce it on a single machine either.
            It is possible that I was running 10.0 and 10.2 on different machines back then. I could have failed to mention it as I wouldn't expect it to be of any importance, but apparently it is.

            Below are the results I'm getting on Ubuntu Focal x86_64 vs Ubuntu Xenial x86_64. Both are running the same server from MariaDB 10.0.38 / 10.2.36 GLIBC_2.14 binary tarballs

            $ ls -l  *gz
            -rw-rw-r-- 1 buildbot buildbot 447462919 Jan 30  2019 mariadb-10.0.38-linux-glibc_214-x86_64.tar.gz
            -rw-rw-r-- 1 buildbot buildbot 814589442 Nov 10 20:21 mariadb-10.2.36-linux-glibc_214-x86_64.tar.gz
            

            Focal

            MariaDB [test]> CHECKSUM TABLE badcs100, badcs100_53temp;
            +----------------------+------------+
            | Table                | Checksum   |
            +----------------------+------------+
            | test.badcs100        | 3449432362 |
            | test.badcs100_53temp | 3449432362 |
            +----------------------+------------+
            2 rows in set (0.000 sec)
            

            Xenial

            MariaDB [test]> CHECKSUM TABLE badcs100, badcs100_53temp;
            +----------------------+------------+
            | Table                | Checksum   |
            +----------------------+------------+
            | test.badcs100        | 3088635921 |
            | test.badcs100_53temp | 3088635921 |
            +----------------------+------------+
            

            I don't know if it's expected.

            elenst Elena Stepanova added a comment - I can't reproduce it on a single machine either. It is possible that I was running 10.0 and 10.2 on different machines back then. I could have failed to mention it as I wouldn't expect it to be of any importance, but apparently it is. Below are the results I'm getting on Ubuntu Focal x86_64 vs Ubuntu Xenial x86_64. Both are running the same server from MariaDB 10.0.38 / 10.2.36 GLIBC_2.14 binary tarballs $ ls -l *gz -rw-rw-r-- 1 buildbot buildbot 447462919 Jan 30 2019 mariadb-10.0.38-linux-glibc_214-x86_64.tar.gz -rw-rw-r-- 1 buildbot buildbot 814589442 Nov 10 20:21 mariadb-10.2.36-linux-glibc_214-x86_64.tar.gz Focal MariaDB [test]> CHECKSUM TABLE badcs100, badcs100_53temp; + ----------------------+------------+ | Table | Checksum | + ----------------------+------------+ | test.badcs100 | 3449432362 | | test.badcs100_53temp | 3449432362 | + ----------------------+------------+ 2 rows in set (0.000 sec) Xenial MariaDB [test]> CHECKSUM TABLE badcs100, badcs100_53temp; + ----------------------+------------+ | Table | Checksum | + ----------------------+------------+ | test.badcs100 | 3088635921 | | test.badcs100_53temp | 3088635921 | + ----------------------+------------+ I don't know if it's expected.

            Did anyone test this with Valgrind or MemorySanitizer, while disabling the questionable function Field_varstring::mark_unused_memory_as_defined()? InnoDB will leave the unused part of VARCHAR buffers uninitialized when returning data to the SQL layer. That function (and equivalent Valgrind suppressions) are silencing warnings about uninitialized data being used.

            marko Marko Mäkelä added a comment - Did anyone test this with Valgrind or MemorySanitizer, while disabling the questionable function Field_varstring::mark_unused_memory_as_defined() ? InnoDB will leave the unused part of VARCHAR buffers uninitialized when returning data to the SQL layer. That function (and equivalent Valgrind suppressions) are silencing warnings about uninitialized data being used.

            I tested this with the following patch:

            diff --git a/sql/field.cc b/sql/field.cc
            index 08bde5f58e1..46e0a2e5954 100644
            --- a/sql/field.cc
            +++ b/sql/field.cc
            @@ -7865,8 +7865,10 @@ bool Field_varstring::send(Protocol *protocol)
             #ifdef HAVE_MEM_CHECK
             void Field_varstring::mark_unused_memory_as_defined()
             {
            +#if 0
               uint used_length __attribute__((unused)) = get_length();
               MEM_MAKE_DEFINED(get_data() + used_length, field_length - used_length);
            +#endif
             }
             #endif
             
            

            and the following test case on a recent 10.5. But, it did not generate any messages about using uninitialized memory. I invoked Valgrind manually while passing the --manual-gdb option to mtr, so no suppressions should have been present.

            --source include/have_innodb.inc
            CREATE TABLE badcs (
              bad_id int(10) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
              partner_id mediumint(8) NOT NULL,
              service_type varchar(50) NOT NULL,
              service_level tinyint(3) unsigned NOT NULL,
              service_start timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
              service_end timestamp NULL DEFAULT NULL,
              invoice tinyint(1) unsigned NOT NULL
            ) ENGINE=InnoDB DEFAULT CHARSET utf8mb3;
             
            INSERT INTO badcs VALUES
            (1,	123,	'CREATE TABLE badcs',	5,	'2018-03-02 10:48:23',	NULL,	1),
            (2,	123,	'zzzzzzzzzzzzz',	4,	'2018-03-02 10:48:46',	NULL,	1),
            (3,	255,	'trrrrrrrrrrrr',	4,	'2018-03-02 11:56:37',	NULL,	0);
            CHECKSUM TABLE badcs;
            DROP TABLE badcs;
            

            I got the same checksum reported when I invoked the test with

            ./mtr --mysqld=--innodb-default-row-format=redundant 
            

            marko Marko Mäkelä added a comment - I tested this with the following patch: diff --git a/sql/field.cc b/sql/field.cc index 08bde5f58e1..46e0a2e5954 100644 --- a/sql/field.cc +++ b/sql/field.cc @@ -7865,8 +7865,10 @@ bool Field_varstring::send(Protocol *protocol) #ifdef HAVE_MEM_CHECK void Field_varstring::mark_unused_memory_as_defined() { +#if 0 uint used_length __attribute__((unused)) = get_length(); MEM_MAKE_DEFINED(get_data() + used_length, field_length - used_length); +#endif } #endif and the following test case on a recent 10.5. But, it did not generate any messages about using uninitialized memory. I invoked Valgrind manually while passing the --manual-gdb option to mtr , so no suppressions should have been present. --source include/have_innodb.inc CREATE TABLE badcs ( bad_id int (10) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY , partner_id mediumint(8) NOT NULL , service_type varchar (50) NOT NULL , service_level tinyint(3) unsigned NOT NULL , service_start timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' , service_end timestamp NULL DEFAULT NULL , invoice tinyint(1) unsigned NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET utf8mb3;   INSERT INTO badcs VALUES (1, 123, 'CREATE TABLE badcs' , 5, '2018-03-02 10:48:23' , NULL , 1), (2, 123, 'zzzzzzzzzzzzz' , 4, '2018-03-02 10:48:46' , NULL , 1), (3, 255, 'trrrrrrrrrrrr' , 4, '2018-03-02 11:56:37' , NULL , 0); CHECKSUM TABLE badcs; DROP TABLE badcs; I got the same checksum reported when I invoked the test with ./mtr --mysqld=--innodb-default-row-format=redundant

            For the record, the checksum that I got reported was 3379190625, which is differs from any checksums that were reported here earlier.

            marko Marko Mäkelä added a comment - For the record, the checksum that I got reported was 3379190625, which is differs from any checksums that were reported here earlier.

            I also tested the original data and I got checksum 3379190625
            (This with 11.0)

            I don't think the VARCHAR or blobs are a problem.
            The code in handler.cc:calculate_checksum() ensures that for BLOB, VARCHAR, GEOMETRY and BIT,
            we only calculate the checksum for the active data.

            monty Michael Widenius added a comment - I also tested the original data and I got checksum 3379190625 (This with 11.0) I don't think the VARCHAR or blobs are a problem. The code in handler.cc:calculate_checksum() ensures that for BLOB, VARCHAR, GEOMETRY and BIT, we only calculate the checksum for the active data.

            People

              serg Sergei Golubchik
              stamster Nemanja Stambolic
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.