Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-27154

allkeys.txt based tests for Unicode-4.0.0 and 5.2.0

Details

    Description

      Let's add MTR tests which will load the default weight table allkeys.txt from Unicode-4.0.0 and Unicode-5.2.0 to check that the collations utf8mb4_unicode_ci and utf8mb4_unicode_520_ci work as expected.

      These new tests will cover all characters in the range U+0000..U+10FFFF and will make sure that nothing breaks after upcoming changes soon.

      The idea is to calculate weights for every Unicode character into two ways:

      1. Using WEIGHT_STRING() - this is the weight that the MariaDB collation returns for the character.
      2. Parsing implicit weights from the corresponding line in allkeys.txt (or by calculating its implicit weight) - this is the weight that the collation is supposed to return according to the Unicode standard.

      Both calculated values must produce equal results for every character.
      If for some character the weights calculated in two weights are different, it means the collation works incorrectly.

      The only exception character is "FDFA ARABIC LIGATURE SALLALLAHOU ALAYHE WASALLAM". It has 18 weights in allkeys.txt, while MariaDB has a limit of 8 weights per character.

      Attachments

        Issue Links

          Activity

            bar Alexander Barkov added a comment - - edited

            It's also repeatable with mtr --valgrind run with this smaller script:

            --source include/have_utf32.inc
            --source include/have_utf8mb4.inc
             
            SET NAMES latin1;
             
            CREATE TABLE t1 (
              code INT NOT NULL,
              str VARCHAR(1) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL
            ) ENGINE=MyISAM;
             
            DELIMITER $$;
            FOR i IN 0x0000..0x2FFF
            DO
              INSERT INTO t1 VALUES (i, CHAR(i USING utf32));
            END FOR;
            $$
            DELIMITER ;$$
            SELECT COUNT(*) FROM t1;
             
            SELECT HEX(code), HEX(str) FROM t1 ORDER BY HEX(str);
             
            DROP TABLE t1;
            

            bar Alexander Barkov added a comment - - edited It's also repeatable with mtr --valgrind run with this smaller script: --source include/have_utf32.inc --source include/have_utf8mb4.inc   SET NAMES latin1;   CREATE TABLE t1 ( code INT NOT NULL, str VARCHAR(1) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL ) ENGINE=MyISAM;   DELIMITER $$; FOR i IN 0x0000..0x2FFF DO INSERT INTO t1 VALUES (i, CHAR(i USING utf32)); END FOR; $$ DELIMITER ;$$ SELECT COUNT(*) FROM t1;   SELECT HEX(code), HEX(str) FROM t1 ORDER BY HEX(str);   DROP TABLE t1;

            People

              bar Alexander Barkov
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.