Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-16857

system-invisible row_end is displayed in SHOW INDEX

Details

    Description

      There is no easy way to check the column information of the auto-generated row_start and row_end columns when a table is created WITH SYSTEM VERSIONING. Due to this, it is causing a bug with phpMyAdmin and other similar tools/libraries (including my own PHP Universal Database Library). These utilities usually do a SELECT * query on their (Browse) tab. In phpMyAdmin, most features, such as editing row values, is disabled because they assume they don't have the full PRIMARY KEY in the results set. We already know that row_end in the PRIMARY KEY is GENERATED, but there is no way to query this so the application software can be aware of this and handle it properly. MDEV-16804 is adding the GENERATED flag to these columns, however if the columns are automatically created rather than manually created, they don't even show up via DESC / SHOW COLUMNS, so there is no way to add the needed conditional code to handle these cases at the application layer.

      More details about how this is causing issues with phpMyAdmin can be found over in their GitHub: https://github.com/phpmyadmin/phpmyadmin/issues/14515

      The only other work around I can think of at the moment for the application layer would be to assume any column listed in the PRIMARY KEY that is NOT listed in SHOW COLUMNS is a system GENERATED column. But there may be other applications that want to know the data type information of the PRIMARY KEY, so this would still need addressed.

      As a note, according to documentation, user created INVISIBLE columns show up in SHOW COLUMNS, so system created INVISIBLE columns probably should too. https://mariadb.com/kb/en/library/invisible-columns/

      CREATE TABLE vertest (x INT PRIMARY KEY) WITH SYSTEM VERSIONING;
       
       
      SHOW COLUMNS FROM vertest;
      +-------+---------+------+-----+---------+-------+
      | Field | Type    | Null | Key | Default | Extra |
      +-------+---------+------+-----+---------+-------+
      | x     | int(11) | NO   | PRI | NULL    |       |
      +-------+---------+------+-----+---------+-------+
       
       
      SHOW FULL COLUMNS FROM vertest;
      +-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
      | Field | Type    | Collation | Null | Key | Default | Extra | Privileges                      | Comment |
      +-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
      | x     | int(11) | NULL      | NO   | PRI | NULL    |       | select,insert,update,references |         |
      +-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
       
       
      SHOW INDEX FROM vertest;
      +---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      | Table   | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
      +---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      | vertest |          0 | PRIMARY  |            1 | x           | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
      | vertest |          0 | PRIMARY  |            2 | row_end     | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
      +---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
      

      Attachments

        Issue Links

          Activity

            https://github.com/MariaDB/server/blob/c3289d27eef39a47fed2ce1ff239013ed6870f39/sql/table.h#L343

            enum field_visibility_t {
              VISIBLE= 0,
              INVISIBLE_USER,
              /* automatically added by the server. Can be queried explicitly
              in SELECT, otherwise invisible from anything" */
              INVISIBLE_SYSTEM,
              INVISIBLE_FULL
            };
            

            I'm guessing by this comment in the source that the row_start and row_end columns are defined as either INVISIBLE_SYSTEM or INVISIBLE_FULL - if that is the case, then shouldn't they also be hidden from SHOW INDEX? If so, the issues with applications like phpMyAdmin should go away, since the PRIMARY KEY would be "fully" displayed in a SELECT * statement.

            darkain Vincent Milum Jr added a comment - https://github.com/MariaDB/server/blob/c3289d27eef39a47fed2ce1ff239013ed6870f39/sql/table.h#L343 enum field_visibility_t { VISIBLE= 0, INVISIBLE_USER, /* automatically added by the server. Can be queried explicitly in SELECT, otherwise invisible from anything" */ INVISIBLE_SYSTEM, INVISIBLE_FULL }; I'm guessing by this comment in the source that the row_start and row_end columns are defined as either INVISIBLE_SYSTEM or INVISIBLE_FULL - if that is the case, then shouldn't they also be hidden from SHOW INDEX ? If so, the issues with applications like phpMyAdmin should go away, since the PRIMARY KEY would be "fully" displayed in a SELECT * statement.

            Yes, automatic versioning columns are SYSTEM_INVISIBLE, hence their visibility differs from regular user-invisible columns.
            However, I cannot argue with the point about SHOW INDEX, as I raised it myself before, see my comment to MDEV-14785 (last one as of now). I believe at that time it was decided to keep it the way it is now. Assigning to serg to consider if a change is due.

            elenst Elena Stepanova added a comment - Yes, automatic versioning columns are SYSTEM_INVISIBLE, hence their visibility differs from regular user-invisible columns. However, I cannot argue with the point about SHOW INDEX , as I raised it myself before, see my comment to MDEV-14785 (last one as of now). I believe at that time it was decided to keep it the way it is now. Assigning to serg to consider if a change is due.

            For the time being at least for phpMyAdmin, I have submitted a patch to them which checks the primary key columns against all "known" columns in the table. If a particular key column does not "appear" within the table's listed columns, ignore it. I've yet to see if I'll need to do the same within my PUDL library, depending on how applications start using system versioning (the first site to use it just started development today)

            darkain Vincent Milum Jr added a comment - For the time being at least for phpMyAdmin, I have submitted a patch to them which checks the primary key columns against all "known" columns in the table. If a particular key column does not "appear" within the table's listed columns, ignore it. I've yet to see if I'll need to do the same within my PUDL library, depending on how applications start using system versioning (the first site to use it just started development today)

            Yes, I think this is a bug and we should fix it.

            serg Sergei Golubchik added a comment - Yes, I think this is a bug and we should fix it.
            sevam Edmund Tam added a comment -

            This also affect ORM tools which syncs the application code with the database schema. For instance, the ORM tool will be confused as the primary key definition in the database doesn't match the one defined in application code after system versioning is enabled. For best compatibility with higher level tools (ORM and schema tools like phpMyAdmin) it'd be reasonable for the system generated column to (also) be invisible by default in the SHOW INDEXES result.

            sevam Edmund Tam added a comment - This also affect ORM tools which syncs the application code with the database schema. For instance, the ORM tool will be confused as the primary key definition in the database doesn't match the one defined in application code after system versioning is enabled. For best compatibility with higher level tools (ORM and schema tools like phpMyAdmin) it'd be reasonable for the system generated column to (also) be invisible by default in the SHOW INDEXES result.

            Add SHOW CREATE TABLE to the test as explanation why thay are not listed in IS. and OK to push.

            sanja Oleksandr Byelkin added a comment - Add SHOW CREATE TABLE to the test as explanation why thay are not listed in IS. and OK to push.

            People

              midenok Aleksey Midenkov
              darkain Vincent Milum Jr
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.