[MDEV-16857] system-invisible row_end is displayed in SHOW INDEX Created: 2018-07-30  Updated: 2021-07-06  Resolved: 2021-07-06

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Create Table, Information Schema, Versioned Tables
Affects Version/s: 10.3.8
Fix Version/s: 10.3.31, 10.4.21, 10.5.12, 10.6.4

Type: Bug Priority: Major
Reporter: Vincent Milum Jr Assignee: Aleksey Midenkov
Resolution: Fixed Votes: 0
Labels: None
Environment:

10.3.8-MariaDB-1:10.3.8+maria~stretch-log - mariadb.org binary distribution


Issue Links:
Relates
relates to MDEV-16804 SYSTEM VERSIONING columns not showing... Closed

 Description   

There is no easy way to check the column information of the auto-generated row_start and row_end columns when a table is created WITH SYSTEM VERSIONING. Due to this, it is causing a bug with phpMyAdmin and other similar tools/libraries (including my own PHP Universal Database Library). These utilities usually do a SELECT * query on their (Browse) tab. In phpMyAdmin, most features, such as editing row values, is disabled because they assume they don't have the full PRIMARY KEY in the results set. We already know that row_end in the PRIMARY KEY is GENERATED, but there is no way to query this so the application software can be aware of this and handle it properly. MDEV-16804 is adding the GENERATED flag to these columns, however if the columns are automatically created rather than manually created, they don't even show up via DESC / SHOW COLUMNS, so there is no way to add the needed conditional code to handle these cases at the application layer.

More details about how this is causing issues with phpMyAdmin can be found over in their GitHub: https://github.com/phpmyadmin/phpmyadmin/issues/14515

The only other work around I can think of at the moment for the application layer would be to assume any column listed in the PRIMARY KEY that is NOT listed in SHOW COLUMNS is a system GENERATED column. But there may be other applications that want to know the data type information of the PRIMARY KEY, so this would still need addressed.

As a note, according to documentation, user created INVISIBLE columns show up in SHOW COLUMNS, so system created INVISIBLE columns probably should too. https://mariadb.com/kb/en/library/invisible-columns/

CREATE TABLE vertest (x INT PRIMARY KEY) WITH SYSTEM VERSIONING;
 
 
SHOW COLUMNS FROM vertest;
+-------+---------+------+-----+---------+-------+
| Field | Type    | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| x     | int(11) | NO   | PRI | NULL    |       |
+-------+---------+------+-----+---------+-------+
 
 
SHOW FULL COLUMNS FROM vertest;
+-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
| Field | Type    | Collation | Null | Key | Default | Extra | Privileges                      | Comment |
+-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
| x     | int(11) | NULL      | NO   | PRI | NULL    |       | select,insert,update,references |         |
+-------+---------+-----------+------+-----+---------+-------+---------------------------------+---------+
 
 
SHOW INDEX FROM vertest;
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table   | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| vertest |          0 | PRIMARY  |            1 | x           | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
| vertest |          0 | PRIMARY  |            2 | row_end     | A         |           0 |     NULL | NULL   |      | BTREE      |         |               |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+



 Comments   
Comment by Vincent Milum Jr [ 2018-07-30 ]

https://github.com/MariaDB/server/blob/c3289d27eef39a47fed2ce1ff239013ed6870f39/sql/table.h#L343

enum field_visibility_t {
  VISIBLE= 0,
  INVISIBLE_USER,
  /* automatically added by the server. Can be queried explicitly
  in SELECT, otherwise invisible from anything" */
  INVISIBLE_SYSTEM,
  INVISIBLE_FULL
};

I'm guessing by this comment in the source that the row_start and row_end columns are defined as either INVISIBLE_SYSTEM or INVISIBLE_FULL - if that is the case, then shouldn't they also be hidden from SHOW INDEX? If so, the issues with applications like phpMyAdmin should go away, since the PRIMARY KEY would be "fully" displayed in a SELECT * statement.

Comment by Elena Stepanova [ 2018-08-03 ]

Yes, automatic versioning columns are SYSTEM_INVISIBLE, hence their visibility differs from regular user-invisible columns.
However, I cannot argue with the point about SHOW INDEX, as I raised it myself before, see my comment to MDEV-14785 (last one as of now). I believe at that time it was decided to keep it the way it is now. Assigning to serg to consider if a change is due.

Comment by Vincent Milum Jr [ 2018-08-04 ]

For the time being at least for phpMyAdmin, I have submitted a patch to them which checks the primary key columns against all "known" columns in the table. If a particular key column does not "appear" within the table's listed columns, ignore it. I've yet to see if I'll need to do the same within my PUDL library, depending on how applications start using system versioning (the first site to use it just started development today)

Comment by Sergei Golubchik [ 2018-08-06 ]

Yes, I think this is a bug and we should fix it.

Comment by Edmund Tam [ 2018-10-09 ]

This also affect ORM tools which syncs the application code with the database schema. For instance, the ORM tool will be confused as the primary key definition in the database doesn't match the one defined in application code after system versioning is enabled. For best compatibility with higher level tools (ORM and schema tools like phpMyAdmin) it'd be reasonable for the system generated column to (also) be invisible by default in the SHOW INDEXES result.

Comment by Oleksandr Byelkin [ 2021-04-29 ]

Add SHOW CREATE TABLE to the test as explanation why thay are not listed in IS. and OK to push.

Generated at Thu Feb 08 08:32:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.