Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30577

Case folding for uca1400 collations is not up to date

Details

    Description

      UCA1400 collations (added by MDEV-27009) currently use Unicode-5.2.0 case folding tables.

      They should use Unicode-14.0.0 tables instead.

      The difference (see attached diff-520-1400.diff) between these two files:

      shows that a few hundred new case folding mapping pairs where added in these letter scripts:

      Cyrillic, Gergian, Cherokee, Glagolitic, Coptic, Latin, Osage, Vithkuqi, Old Hungarian, Warang Citi, Medefaidrin, Adlam.

      This SQL script demonstrates the out-dated case folding:

      CREATE OR REPLACE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8 COLLATE uca1400_ai_ci);
      		 
      # Insert letters appeared in Unicode-6.1 (released in January 2012)
      		 
      INSERT INTO t1 VALUES (_ucs2 0xA792) /* U+A792 LATIN CAPITAL LETTER C WITH BAR */;
      		 
      INSERT INTO t1 VALUES (_ucs2 0xA793) /* U+A793 LATIN SMALL LETTER C WITH BAR */;
      		 
      SELECT HEX(a), HEX(LOWER(a)), HEX(UPPER(a)), a, LOWER(a), UPPER(a) FROM t1;


      +--------+---------------+---------------+------+----------+----------+
      		 
      | HEX(a) | HEX(LOWER(a)) | HEX(UPPER(a)) | a    | LOWER(a) | UPPER(a) |
      		 
      +--------+---------------+---------------+------+----------+----------+
      		 
      | EA9E92 | EA9E92        | EA9E92        | Ꞓ    | Ꞓ        | Ꞓ        |
      		 
      | EA9E93 | EA9E93        | EA9E93        | ꞓ    | ꞓ        | ꞓ        |
      		 
      +--------+---------------+---------------+------+----------+----------+

      The above two characters (first appeared in Unicode-6.1) are expected to map to each other by functions UPPER and LOWER.

      Attachments

        Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. MDEV-19123 Change default charset from latin1 to utf8mb4

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. MDEV-25829 Change default Unicode collation to uca1400_ai_ci

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. MDEV-27490 Allow full utf8mb4 for identifiers

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Stalled
          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30692 conf_to_src is not up to date

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-30695 Refactor case folding data types in Asian collation

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30716 Wrong casefolding in xxx_unicode_520_ci for U+0700..U+07FF

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30746 Regression in ucs2_general_mysql500_ci

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-31068 Reuse duplicate case conversion code in ctype-utf8.c and ctype-ucs2.c

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-31069 Reuse duplicate char-to-weight conversion code in ctype-utf8.c and ctype-ucs2.c

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Task - A task that needs to be done. MDEV-31071 Refactor case folding data types in Unicode collations

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Task - A task that needs to be done. MDEV-27009 Add UCA-14.0.0 collations

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MDEV-30661 UPPER() returns an empty string for U+0251 in uca1400 collations for utf8

          • Major - Major loss of function.
          • Closed

          Activity

            There are no comments yet on this issue.

            People

              bar Alexander Barkov
              bar Alexander Barkov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.