[MDEV-12459] The information_schema tables for getting temporary tables info is missing, at least for innodb there is no INNODB_TEMP_TABLE_INFO Created: 2017-04-06 Updated: 2023-09-07 Resolved: 2023-08-12 |
|
| Status: | Closed |
| Project: | MariaDB Server |
| Component/s: | Data Definition - Temporary, Information Schema |
| Fix Version/s: | 11.2.1 |
| Type: | Task | Priority: | Critical |
| Reporter: | Francisco Dueñas | Assignee: | Anel Husakovic |
| Resolution: | Fixed | Votes: | 6 |
| Labels: | Preview_10.9, Preview_11.2, beginner-friendly, foundation | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Currently, before Mysql 5.7, there was no way to get information from a temporary table (session table) using the information_schema. Now with Mysql 5.7 there is a new table named 'INNODB_TEMP_TABLE_INFO'. According to documentation: https://mariadb.com/kb/en/mariadb/information-schema-temp_tables_info-table/ There was a table names 'TEMP_TABLES_INFO' which was introduced in 10.2.2 but removed in 10.2.4. I think maybe you're thinking to implement another solution which can cover all the needs. The example shows how to use 'INNODB_TEMP_TABLE_INFO' but it doesn't exist BTW that mysql's implementation is very limited, there no way you can query against temporary tables fields using a SELECT statement. Summary from comments This is what will be implemented: References: |
| Comments |
| Comment by Marko Mäkelä [ 2017-05-09 ] | |||||||||||||||||||||||||
|
I intentionally removed the MySQL implementation of the table in | |||||||||||||||||||||||||
| Comment by Francisco Dueñas [ 2017-05-09 ] | |||||||||||||||||||||||||
|
marko: I Agree with you , the current Implementation is very limited, for example there is no way you can get the metadata info of a temporary table, for example to get the field list, because sometimes we need to generate a temporary table based on a prepared statement, and use that table to do some advanced operations. But we need to parse the field list because, as being a prepared statement the is no way we can know in advance the fields that the table will have. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2018-02-14 ] | |||||||||||||||||||||||||
|
Standard INFORMATION_SCHEMA.TABLES should show temporary tables too. The fourth column, TABLE_TYPE should be "LOCAL TEMPORARY". Standard also has "GLOBAL TEMPORARY" tables, but we don't support them yet. | |||||||||||||||||||||||||
| Comment by Badrul Chowdhury [ 2018-04-03 ] | |||||||||||||||||||||||||
|
serg, I would like to work on this issue, is that alright? | |||||||||||||||||||||||||
| Comment by Julien Fritsch [ 2019-05-16 ] | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2019-05-16 ] | |||||||||||||||||||||||||
|
Hi julien.fritsch I have updated fix-version, currently there is a PR for `10.3`. | |||||||||||||||||||||||||
| Comment by Julien Fritsch [ 2019-10-14 ] | |||||||||||||||||||||||||
|
cvicentiu, when do you think you can review this issue, please? | |||||||||||||||||||||||||
| Comment by Francisco Dueñas [ 2020-01-21 ] | |||||||||||||||||||||||||
|
Also, aditionaly that they can shown in INFORMATION_SCHEMA.TABLES as 'LOCAL TEMPORARY'. MariaDB should be also able to show their fields in INFORMATION_SCHEMA.COLUMNS, and any other metadata associated to a normal table. But at least, implementing INFORMATION_SCHEMA.TABLES and INFORMATION_SCHEMA.COLUMNS will help a lot. One usage for this feature is to be able to generate a JSON object from a Temporary Table/query (i.e CREATE TEMPORARY TABLE from a SELECT), for that, we need to know the field type to generate the correct JSON representation (i.e. Date values, Blob value, etc). | |||||||||||||||||||||||||
| Comment by Aurélien LEQUOY [ 2020-02-24 ] | |||||||||||||||||||||||||
|
I looking for informations to analyse a trouble DDOS issue with replication. any update ? | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-09-21 ] | |||||||||||||||||||||||||
|
anel, I don't think there should be a special I_S table for that. Temporary tables should just show up normally in I_S.TABLES, I_S.COLUMNS, etc. The following should be kept in mind:
another, much simpler approach, would be to show only temp tables for the current connection. Then connection_id column isn't needed. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-09-21 ] | |||||||||||||||||||||||||
|
perhaps, let's split the task. this one will only show temporary tables for the current connection and MDEV-15623 will be for showing temporary tables for all connections? | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-09-21 ] | |||||||||||||||||||||||||
|
How will it work with "I_S.COLUMNS, etc."? | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2020-09-22 ] | |||||||||||||||||||||||||
|
Hi serg, elenst
| |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-09-22 ] | |||||||||||||||||||||||||
anel, | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-09-22 ] | |||||||||||||||||||||||||
|
elenst, good point. One possible solution: I_S tables show information on user visible tables, like, if you can do SELECT * FROM t1 then I_S tables will show information about t1. That is, if temporary table shadows the non-temporary table, then I_S tables only show the temporary table. | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-09-22 ] | |||||||||||||||||||||||||
|
serg, I'm not sure the users will appreciate it. I can think of two reasons right away, although there can be many more. 1) Currently if you happen to have a temporary table which is a namesake of a base table (as unwise as it may be), the only straightforward way to find out the structure of the base table is via the I_S tables. If you run SHOW CREATE TABLE, it will show you the temporary table, but you can get around it via the information_schema. With such a change, it will become much harder, if possible at all. 2) The inconsistency of I_S data can go very ugly very fast. Say, you have a relation (e.g. a foreign key) between two base tables, t2 references t1, and t1 has a temporary namesake. What should happen then when you're looking at a cross-reference I_S table, e.g. I_S.KEY_COLUMN_USAGE, for the table t2? You can't hide the entry for t2 because it is a normal unique base table, but it will show a reference to a column in the base table t1, and then if you try to find the column in other I_S tables, it might not be there because the temporary one doesn't have it (or even worse, it is there, but with a different column definition). | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-09-23 ] | |||||||||||||||||||||||||
|
1) One can argue that it's a bug. That SHOW CREATE TABLE and I_S.TABLES shows different information. 2) Right. I don't have a good solution for that. But the same logic as above applies. One can do SHOW CREATE TABLE for t2, see that it references t1, then one does SHOW CREATE TABLE t1 and the column is not there. Inconsistency is still there. And it not solved by the fact that this inconsistency is inconsistently present in only some commands, but not others. | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-09-23 ] | |||||||||||||||||||||||||
|
1) One can only argue that SHOW CREATE TABLE is wrong. The fact that I_S.TABLES returns information about all non-temporary tables is documented, both in MariaDB KB and MySQL manual. On the other hand, the fact that SHOW CREATE TABLE returns information about a temporary table, while well-known, is not documented, neither in MariaDB KB nor (oddly) in MySQL manual. 2) Once again, it looks like we are going to attempt to solve inconsistency at the expense of common sense and usability – that is, to make things worse than they were from most practical points of view; and we don't even really get rid of inconsistency this time, we just replace one with another. Now I_S tables are consistent among themselves, but inconsistent with SHOW CREATE TABLE. This change will make certain I_S tables be consistent with SHOW CREATE TABLE, but instead they'll become inconsistent among themselves and still some I_S tables will be inconsistent with SHOW CREATE TABLE. I don't think people query the information schema and/or run SHOW statements just to enjoy how perfectly they match, and neither do I believe that even users who now want to see temporary tables in the I_S will be happy when they realize that they have lost base tables as the result. Both I_S and SHOW are run to gather certain information. If one method provides this information, in practice nobody will use another one and thus won't suffer from the inconsistency among them (except for maybe tester-like minds). If one method doesn't provide the required information, then the other one would be tried, and users will surely appreciate if it helps. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-09-24 ] | |||||||||||||||||||||||||
|
it's not even a question of consistency between SHOW CREATE and I_S.TABLES, it's that one cannot get the information on selectable tables from I_S. One can query I_S.TABLES see some columns for test.t1, then query this table to see completely different table structure. I think it's better when I_S.TABLES shows the actual table that users see and can select from. | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2020-09-24 ] | |||||||||||||||||||||||||
|
It is better if users can see both. | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2020-09-28 ] | |||||||||||||||||||||||||
|
/me is still unclear design which should be used per redundancy which will occur. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-10-03 ] | |||||||||||||||||||||||||
|
For I_S.TABLES we can easily show both. They'll have different values in TABLE_TYPE, there can be no ambiguity. What to do with I_S.COLUMNS, I_S.STATISTICS, I_S.TABLE_CONSTRAINTS, I_S.KEY_COLUMN_USAGE, I_S.PARTITIONS, I_S.FILES (?), I_S.CHECK_CONSTRAINTS. They all use TABLE_SCHEMA+TABLE_NAME to identify a table. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-10-03 ] | |||||||||||||||||||||||||
|
One backward compatible but a really inconsistent approach would be for normal tables to shadow temporary tables in I_S (except in I_S.TABLES that can show both). This way everything will work intuitively correct unless one create a temporary table with the same name as a normal table. Long term we should probably outlaw this shadowing behavior altogether. | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2020-10-06 ] | |||||||||||||||||||||||||
|
So to proceed with and implement to I_S.TABLES only in this phase | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-11-06 ] | |||||||||||||||||||||||||
|
okay, let's start from I_S.TABLES. But it won't solve MDEV-23643. We'll need to do something about I_S.COLUMNS too. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2020-11-06 ] | |||||||||||||||||||||||||
|
Note, that as I_S.TABLES doesn't show session id, it should only show temporary tables from the current session. Thus MDEV-15623 also says unresolved | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2020-11-18 ] | |||||||||||||||||||||||||
|
So currently we have:
Based on above, I have created one patch with some open questions in commit message and github comments, hope serg or someone else? will have time to review and point me further, thanks. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2021-03-21 ] | |||||||||||||||||||||||||
|
One more option:
elenst, how about that? in normal use case (no shadowing) everything works normally. We sidestep and simply refuse to solve the case of shadowing. And there's a new table to see all temp tables in all sessions, but it generally cannot be joined with other I_S tables. | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2021-03-31 ] | |||||||||||||||||||||||||
|
serg, To me it sounds better (=~ more backward-compatible) than previous solutions, although please note that it won't be as straightforward as it sounds, right away there is a question of "isolation level", so to speak. Say, connection 1 creates a temporary table myPoorTable. It's somewhat rhetorical as the only practical answer that I can imagine is "new non-temporary table", and I don't think it's necessarily a critical problem, but it may cause confusion and should be at least documented. Also, I assume this solution still implies that the table type for temporary tables which are shown in the I_S (those which don't shadow anything) will be 'TEMPORARY', or there will be another column in I_S.TABLES indicating whether the table is temporary or not. It would make a lot of sense. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2021-03-31 ] | |||||||||||||||||||||||||
|
A "new non-temporary table", indeed. And yes, temporary tables in I_S.TABLES will be shown as "TEMPORARY" | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2021-04-21 ] | |||||||||||||||||||||||||
|
I want to point out that this problem is critical. There aare some business problems, namely converting an arbitrary "Select" statement to an HTML table, con only be solved by reading the metada of a temporary table. If we cannot, then we need to use real tables, and concurrency becomes close to impossible. | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2021-04-27 ] | |||||||||||||||||||||||||
|
It would be unnecessary to change anything if we could do this without returning an empty rowset, which is unnecessary for the logic. | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2021-04-29 ] | |||||||||||||||||||||||||
|
Specifically, how can I run this statement Empty set (0.00 sec) | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2021-05-01 ] | |||||||||||||||||||||||||
|
philip_38, within the approach I've suggested in this comment, you'll be able to read the structure of a temporary table by selecting FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA='your_schema' AND TABLE_NAME='you_table'. It'll work both for normal and temporary tables. | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2021-05-01 ] | |||||||||||||||||||||||||
|
That would be fantastic, but meanwhile, is there any obvious way that I have overlooked to avoid, cancel or redirect that extra empty result set? | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2021-08-07 ] | |||||||||||||||||||||||||
|
I have a question: Is there a way to retrieve metadata for a temporary table in version 5.6? | |||||||||||||||||||||||||
| Comment by Anel Husakovic [ 2021-09-06 ] | |||||||||||||||||||||||||
|
Based on suggested design here is the patch bb1737b04828686 which allows temporary tables to be visible in information_schema.tables per thread. Howerver, there is the failure in buildbot for big-tests default_row_format_create - explained more in | |||||||||||||||||||||||||
| Comment by Michael Widenius [ 2022-02-08 ] | |||||||||||||||||||||||||
|
There is a missmatch between the Title and description of the task and what The task of changing information_schema.tables not part of the Regarding 1) Please describe the reason why we want to change the current Yes, GLOBAL TEMPORARY and LOCAL TEMPORARY are standard, but as we have In SQL SERVER, it looks like LOCAL TEMPORARY is almost exactly In what way is our temporary tables so different from PostgreSQL and 2) and 3) are ok. 4) Don't do it. References: 3) Explain in the Jira what this is all about. | |||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-02-08 ] | |||||||||||||||||||||||||
|
Is it a good idea to have a shared data structure of local temporary tables? It would seem to have some potential to be a performance bottleneck. In InnoDB, I would love to remove the global hash table of temporary table names (MDEV-17805), but currently some handler::delete_table() calls are deleting temporary tables by name, not by an already opened handle. The names that are being used are not the user-specified connection-local table names, but some internally generated, globally unique names. Based on I think that we should thoroughly review what is really needed and wanted, before constraining ourselves by creating such a view. | |||||||||||||||||||||||||
| Comment by Michael Widenius [ 2022-02-15 ] | |||||||||||||||||||||||||
|
Marko, there is no plans to have a shared data structure for Temporary tables. The idea is to loop over THD's and use the existing list of created temporary tables for this | |||||||||||||||||||||||||
| Comment by Michael Widenius [ 2022-02-15 ] | |||||||||||||||||||||||||
|
Regarding why we should not use LOCAL TEMPORARY, according to Serg the main issues seams to be that LOCAL TEMPORARY should survive the connection and can be shared among user. To personally care more about interoperability for applications than following a standard that others may not follow either.
References: | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2022-03-02 ] | |||||||||||||||||||||||||
|
To add to the above, SQL Standard (2016) has no "TEMPORARY", there is no "CREATE TEMPORARY TABLE" at all. There is "CREATE GLOBAL TEMPORARY TABLE" and "CREATE LOCAL TEMPORARY TABLE", where the GLOBAL/LOCAL keyword is not optional. There's also "DECLARE LOCAL TEMPORARY TABLE" that we don't support. All standard global/local temporary tables do not disappear when a session ends. The difference between GLOBAL and LOCAL is minor and in my personal opinion not practically useful, I'd prefer the standard to have tables that are local to a session, but here we are. MariaDB and MySQL have "CREATE TEMPORARY TABLE", so it make sense for MariaDB to call those tables "TEMPORARY", like MySQL does — the table type is what's between "CREATE" and "TABLE". We can call them "PRIVATE TEMPORARY", kind of like Oracle, but Oracle doesn't use this wording in INFORMATION_SCHEMA.TABLES as far as I understand. So it'd be compatible with nothing. Or we can do like PostgreSQL does and incorrectly call those tables "LOCAL TEMPORARY", which will go both against the standard and against our own "CREATE TEMPORARY TABLE" syntax. Considering all the above, I believe we should use "TEMPORARY". This matches our existing syntax, is compatible with MySQL behavior, and does not violate standard definitions. If (when? one can hope) the standard introduces session-local temporary tables with a special name and syntax for them, then we can reconsider. | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2022-03-03 ] | |||||||||||||||||||||||||
|
Note: In MS SQL Server, there are also global temporary tables, prefixed with ##, and they survive a connection closure. They are created in TEMPDB, a special system database. | |||||||||||||||||||||||||
| Comment by Daniel Black [ 2022-03-03 ] | |||||||||||||||||||||||||
|
philip_38 its a new issue, please create a new task. How are they temporary if they survive a connection? (question for the new task) | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2022-03-03 ] | |||||||||||||||||||||||||
|
I was just adding to Michael Widenius' statement on how other databases work. Furthermore, there are a lot of business cases for a temporary table that all users and processes can write to and read from, is located in TEMPDB andis specially cached. It gets truncated on reboot. So the idea is not bad. | |||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2022-03-04 ] | |||||||||||||||||||||||||
|
Conceptually, how do global temporary tables differ from ENGINE=MEMORY tables? As far as I understand, those tables have persistent metadata, but the data will not survive a restart of the database server. | |||||||||||||||||||||||||
| Comment by Sergei Golubchik [ 2022-03-09 ] | |||||||||||||||||||||||||
|
For those tables, the data is connection-local, while the metadata is global. | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2022-03-09 ] | |||||||||||||||||||||||||
|
In SQL Server, global temporary tables are visible to all sessions (connections). So if you create a global temporary table in one session, you can start using it in other sessions. | |||||||||||||||||||||||||
| Comment by Elena Stepanova [ 2022-05-06 ] | |||||||||||||||||||||||||
|
Given that there are still open bugs related to the feature implementation and 10.9 is past the deadline, I recommend postponing the feature till 10.10. I don't think it's a critical loss for the potential users of 10.9 release line, since the implementation doesn't really address the demand which was mentioned in the original report and repeated several times in the comments: there is still no way to list temporary table columns by the means of SELECT, and/or retrieve a temporary table structure from INFORMATION_SCHEMA. The implementation only allows to see which temporary tables exist in the session, and get some basic information about them (e.g. the engine and such). | |||||||||||||||||||||||||
| Comment by Philip orleans [ 2022-08-11 ] | |||||||||||||||||||||||||
|
As Elena Stepanova said, wisely, an implementation that would not allow seeing the columns, is useless. I vote to delay it until it works. | |||||||||||||||||||||||||
| Comment by Juan Ferrer Toribio [ 2022-10-17 ] | |||||||||||||||||||||||||
|
I think different tables should be used in I_S for temporary tables, as far as I know, there is no way to know globally how many temporary tables there are, which session created them and how much space they use (on disk or memory for engine=memory). This makes it very difficult to debug memory usage in environments with heavy use of temporary tables with engine=memory. Also, actually a single connection can fill the entire memory of the server since temporary tables can be limited in space but not in number per session. Please correct me if I'm wrong... | |||||||||||||||||||||||||
| Comment by Vicențiu Ciorbaru [ 2023-04-18 ] | |||||||||||||||||||||||||
|
As agreed with Serg, moving this to 11.2 with the final specification: This task will cover only the following behaviour for I_S.tables:
| |||||||||||||||||||||||||
| Comment by Marko Mäkelä [ 2023-07-03 ] | |||||||||||||||||||||||||
|
The added test main.information_schema_temp_table is massively failing. I spotted a failure of a MemorySanitizer build:
As far as I understand, myrg_status() will not initialize the errkey field if flag=HA_STATUS_POS. I think that this must be fixed before the feature can be merged to any main branch. | |||||||||||||||||||||||||
| Comment by Alice Sherepa [ 2023-08-11 ] | |||||||||||||||||||||||||
|
Tested on preview-11.2-preview a238b9a87616794a24a9. OK to push into 11.2 |