[MDEV-13626] Merge InnoDB test cases from MySQL 5.7 Created: 2017-08-23  Updated: 2024-01-03  Resolved: 2023-11-08

Status: Closed
Project: MariaDB Server
Component/s: Tests
Fix Version/s: 10.4.33, 10.5.24, 10.6.17, 10.11.7, 11.0.5, 11.1.4, 11.2.3, 11.3.2

Type: Task Priority: Major
Reporter: Marko Mäkelä Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: compat57

Attachments: File force_restart.task    
Issue Links:
Blocks
is blocked by MDEV-13625 Merge InnoDB test cases from MySQL 5.6 Closed
PartOf
includes MDEV-4697 UPDATE_TIME field for XtraDB / InnoDB Closed
includes MDEV-12251 InnoDB: Failing assertion: table->dat... Confirmed
includes MDEV-19869 Port innodb_fts.fulltext2 from mysql ... Closed
includes MDEV-19886 InnoDB returns misleading ER_NO_SUCH_... Closed
includes MDEV-20000 Consider adopting innodb.innodb_pages... Open
Problem/Incident
causes MDEV-20138 innodb.trx_id_future fails in buildbot Closed
Relates
relates to MDEV-4784 merge test cases from 5.6 Stalled
relates to MDEV-9260 Improve progress report on on-line al... Open
relates to MDEV-13613 Dropping column from table that is pa... Closed
relates to MDEV-13818 CREATE INDEX leaks memory if running ... Closed
relates to MDEV-14218 rem0rec.cc:566: Assertion `!leaf' fai... Closed
relates to MDEV-14238 row0row.ic:52: ulint row_get_trx_id_o... Closed
relates to MDEV-14325 Mysqld crash, when innodb_open_files ... Closed
relates to MDEV-14335 foreign key silently becomes broken Closed
relates to MDEV-14418 Failing assertion: table->data_dir_pa... Confirmed
relates to MDEV-20094 MariaDB Server 10.2 grows .ibd files ... Stalled
relates to MDEV-32690 After CREATE INDEX, InnoDB reports th... Confirmed
relates to MDEV-32725 innodb.import_update_stats accesses u... Closed
relates to MDEV-10568 Implement support for ALTER TABLE ...... Open
relates to MDEV-13640 ALTER TABLE CHANGE and ADD INDEX on a... Closed
relates to MDEV-13716 point_big test fails in 10.2 Closed
relates to MDEV-14207 innodb_gis.rtree_compress, innodb_gis... Closed
relates to MDEV-14230 wrong relevance ranking for full-text... Open
relates to MDEV-14240 sync0rw.ic:285 Assertion `!rw_lock_ow... Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MDEV-14231 MATCH() AGAINST( IN BOOLEAN MODE), re... Technical task Open Thirunarayanan Balathandayuthapani  
MDEV-14232 innodb_fts.index_table test failed, r... Technical task Closed  
MDEV-14252 innodb_fts.limit_union failed, result... Technical task Open Thirunarayanan Balathandayuthapani  
MDEV-14258 fix innodb_fts.compatibility.test Technical task Closed Marko Mäkelä  
MDEV-14261 fix innobase_drop_fts_index_table.test Technical task Closed Marko Mäkelä  
MDEV-14348 deadlock with innodb_autoinc_lock_mo... Technical task Closed Marko Mäkelä  
MDEV-14403 Review and fix tests from innodb_fts,... Technical task Open Thirunarayanan Balathandayuthapani  
MDEV-14405 Review and fix tests from innodb suite Technical task Closed Marko Mäkelä  
MDEV-14406 Review and fix tests from innodb_gis ... Technical task Closed Marko Mäkelä  

 Description   

Until MDEV-11751 and MDEV-13481, the MariaDB process for merging changes from MySQL after 5.6 ignored additions to existing test cases or added test files.
The preferred way of merging changes should be like the MySQL 5.7.19 merge: a merge of a cherry-pick of each relevant commit, and add test cases; if substantial changes to tests or code are needed, do them in separate commits.

There are some behaviour changes in MariaDB that need to be taken into account. Most notably:

  • MariaDB 10.2 does not support CREATE TABLESPACE for InnoDB
  • MDEV-11426 Remove InnoDB INFORMATION_SCHEMA.FILES implementation
  • MDEV-11487 Revert InnoDB internal temporary tables from WL#7682
  • MDEV-5800 implemented indexed virtual columns in a slightly different way, and the InnoDB tests are in --suite=gcol
  • many startup&shutdown fixes (the innodb.log_* tests are already be present in 10.2 in modified form)
  • MDEV-13613 Dropping column from table that is part of a key fails in 10.2.8
  • MDEV-13564 made TRUNCATE TABLE and undo tablespace truncation compatible with backup
  • MDEV-14717 made renames crash-safe in InnoDB
  • MDEV-14585 Automatically remove #sql- tables in innodb dictionary during recovery

That said, we must ensure that all applicable test cases from the following MySQL 5.7 tests are present in MariaDB 10.0:

  • main.alter_table (WL#6555 "Online rename index", DISCARD/IMPORT, index comment, …)
  • main.alter_table-big
  • innodb.*
  • innodb_zip.*
  • innodb_fts.*
  • innodb_gis.*
  • innodb_stress.* (maybe not to be run on buildbot?)

All the affected tests should be documented in this ticket in detail (especially those that are already covered by other MariaDB tests).
The innodb_undo tests were rewritten in MDEV-17049.



 Comments   
Comment by Marko Mäkelä [ 2017-09-08 ]

I imported the test innodb.alter_crash.

Comment by Marko Mäkelä [ 2017-09-14 ]

I imported the changes to innodb.innodb-index and innodb.innodb-index-debug.

Comment by Jan Lindström (Inactive) [ 2017-10-25 ]

commit a6a4c25bf7d943ff5a429fbf32fe04fdaca25b16
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu Oct 12 17:21:04 2017 +0300

MariaDB adjustments to innodb_gis tests.

Some test are still disabled as they provide different results,
fail on different error or crash.

Fixes also MDEV-14060 InnoDB: Foreign key constraint to
geometry type does not work where test intentionally tried
incorrect cases.

commit 42e3e574260dc6ccfe161f8c0d8e8af41e599353
Author: Jan Lindström <jan.lindstrom@mariadb.com>
Date: Thu Oct 12 16:21:21 2017 +0300

MDEV-13626: Merge InnoDB test cases from MySQL 5.7

Add missing innodb_gis tests.

Comment by Alice Sherepa [ 2017-10-30 ]

Jan Lindström, please take a look on this test, what is the correct result her/
https://github.com/MariaDB/server/blob/mdev-13626/mysql-test/suite/innodb_gis/t/rtree_split.test

innodb_gis.rtree_split 'innodb'          [ fail ]
 
@@ -80,6 +80,5 @@
 set @g1 = ST_GeomFromText('Polygon((2 2,2 800,800 800,800 2,2 2))');
 select count(*) from t1 where MBRWithin(t1.c2, @g1);
 count(*)
-57344
+65536
 drop table t1;
mysqltest: Result length mismatch

Comment by Alice Sherepa [ 2017-10-30 ]

innodb_gis.gis_split_nan needed 4k size page on options, as --source include/have_innodb_4k.inc just checks if it is not 4k, then skip test. Now it fails with error 1416: Cannot get geometry object from data you send to the GEOMETRY field

https://github.com/MariaDB/server/blob/mdev-13626/mysql-test/suite/innodb_gis/t/gis_split_nan.test

innodb_gis.gis_split_nan 'innodb'        w1 [ fail ]
        Test ended at 2017-10-30 14:03:37
 
CURRENT_TEST: innodb_gis.gis_split_nan
mysqltest: At line 285: query 'INSERT INTO geom_index SELECT * FROM gis_geometrycollection' failed: 1416: Cannot get geometry object from data you send to the GEOMETRY field
 
The result from queries just before the failure was:
< snip >
                                                                      'LINESTRING(0 0,10 10),'
                                                                      'GEOMETRYCOLLECTION('
                                                                            'GEOMETRYCOLLECTION())),'
                                                                'GEOMETRYCOLLECTION(),'
                                                                'GEOMETRYCOLLECTION('
                                                                       'GEOMETRYCOLLECTION()),'
                                                                'POLYGON((0 0,0 10,10 10,10 0,0 0)),'
                                                                'MULTIPOINT(0 0,2 2,4 4,6 6,8 8,10 10),'
                                                                'MULTILINESTRING((0 0,10 10),(0 10,10 0)),'
                                                                'MULTIPOLYGON(((0 0,0 5,5 5,5 0,0 0)),((5 5,5 10,10 10,10 5,5 5))))'))));
# Creating a table with spatial index column
CREATE TABLE geom_index (fid INTEGER NOT NULL PRIMARY KEY, g GEOMETRY NOT NULL, SPATIAL INDEX(g));
# Inserting geometry values
INSERT INTO geom_index SELECT * FROM gis_point;
INSERT INTO geom_index SELECT * FROM gis_linestring;
INSERT INTO geom_index SELECT * FROM gis_polygon;
INSERT INTO geom_index SELECT * FROM gis_multi_point;
INSERT INTO geom_index SELECT * FROM gis_multi_linestring;
INSERT INTO geom_index SELECT * FROM gis_multi_polygon;
INSERT INTO geom_index SELECT * FROM gis_geometrycollection;

Comment by Alice Sherepa [ 2017-11-02 ]

please check if result is correct/ test is here https://github.com/MariaDB/server/blob/mdev-13626/mysql-test/suite/innodb_fts/t/sync.test

CURRENT_TEST: innodb_fts.sync
--- /home/alice/aliska/10.2/mysql-test/suite/innodb_fts/r/sync.result	2017-11-02 14:13:42.975845201 +0100
+++ /home/alice/aliska/10.2/mysql-test/suite/innodb_fts/r/sync.reject	2017-11-02 14:14:01.639849689 +0100
@@ -133,9 +133,9 @@
 SET GLOBAL innodb_ft_aux_table="test/t1";
 SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_INDEX_TABLE;
 WORD	FIRST_DOC_ID	LAST_DOC_ID	DOC_COUNT	DOC_ID	POSITION
-database	4	4	1	4	0
-good	5	5	1	5	0
-mysql	2	2	1	2	0
+database	2	2	1	2	0
+good	3	3	1	3	0
+mysql	1	1	1	1	0
 SELECT * FROM INFORMATION_SCHEMA.INNODB_FT_INDEX_CACHE;
 WORD	FIRST_DOC_ID	LAST_DOC_ID	DOC_COUNT	DOC_ID	POSITION
 SET GLOBAL innodb_ft_aux_table=default;
 
mysqltest: Result content mismatch

Comment by Alice Sherepa [ 2017-11-02 ]

also problem in MATCH AGAINST IN BOOLEAN MODE --https://github.com/MariaDB/server/blob/mdev-13626/mysql-test/suite/innodb_fts/t/opt.test

CURRENT_TEST: innodb_fts.opt
mysqltest: At line 977: query 'select * from t1 where MATCH a,b AGAINST('"Now sUPPort"' IN BOOLEAN MODE)' succeeded - should have failed with errno 1191...
 
The result from queries just before the failure was:
< snip >
DROP TABLE t1;
#
# Bug#20261601 ASSERTION FAILED: !FIRST_QEP_TAB->TABLE()->NO_KEYREAD
#
CREATE TABLE t1(a INT PRIMARY KEY);
INSERT INTO t1 VALUES(1),(2);
SELECT (SELECT MATCH(`a`)AGAINST('1') FROM t1) FROM t1;
ERROR HY000: Can't find FULLTEXT index matching the column list
SELECT 1, a IN (SELECT a FROM t1) FROM t1;
1	a IN (SELECT a FROM t1)
1	1
1	1
DROP TABLE t1;
#
# Bug#20442572 ASSERTION `!FIRST_QEP_TAB->TABLE()->NO_KEYREAD' FAILED.
# Bug#75688 Assertion `!first_qep_tab->table()->no_keyread' failed.
#
CREATE TABLE t1(a INT,b POINT NOT NULL,KEY(a));
HANDLER t1 OPEN;
select * from t1 where MATCH a,b AGAINST('"Now sUPPort"' IN BOOLEAN MODE);

Comment by Alice Sherepa [ 2017-11-15 ]

please see attached file force-restart.task, it lists tests, which had option --force-restart, that our mtr does not support.
I deleted it, but tests might fail because of it.

Comment by Marko Mäkelä [ 2018-10-25 ]

The file mysql-test/suite/innodb/t/cascade_lock_wait.test became useless in 5.7 after the needed DBUG_EXECUTE_IF was removed.
For any debug tests that we import from MySQL, we should check that the debug instrumentation is still available.

Comment by Marko Mäkelä [ 2019-06-20 ]

Resuming work on this. innodb.innodb_buffer_pool_dump_pct needed adjustment for MDEV-11454.

Comment by Marko Mäkelä [ 2019-07-02 ]

I will not import two tests for temporary tables:

  • innodb_zip.wl6915_1 does not seem to add much value to MariaDB. In MariaDB, the limits for rollback segments were preserved (MDEV-12289) for compatibility with older versions.
  • innodb_zip.wl6470_2 is only testing basic DML functionality with temporary tables. We can get better test coverage with RQG runs.

Both tests expect ROW_FORMAT=COMPRESSED to be available for temporary tables. This is intentionally not supported by MariaDB due to the increased memory and processing overhead.

Comment by Marko Mäkelä [ 2019-07-02 ]

The test innodb.blob-crash needs some code changes, to replace DBUG_SUICIDE with DEBUG_SYNC and external killing.

Comment by Marko Mäkelä [ 2023-11-03 ]

The test innodb.iodku was added in a fix of Bug #25966845 INSERT ON DUPLICATE KEY GENERATE A DEADLOCK. As I had noted in MDEV-20850, MariaDB fixed the problem differently:

the original problem was partly fixed in MDEV-17073 (and later fully fixed in MDEV-17614).

The MySQL fix ensures that statement-level replication for INSERT…ON DUPLICATE KEY UPDATE works without causing an increased amount of deadlocks. The MariaDB fix will simply mark such statements unsafe for statement-level replication, so that they will be deterministically replicated using row-level events. The MySQL fix avoids deadlocks by registering next-key (gap) locks on a partial rollback, that is, by escalating to coarser locking. That is why the "Test 6" and "Test 7" sections in the test result in a lock wait timeout in MySQL 5.7 but not MariaDB.

While I was discussing this with vlad.lesin, I pointed out that by design, InnoDB allows a transaction to modify a record that was committed by a later-started transaction. I verified that a lock wait timeout would be reported by MariaDB if both transactions were using the SERIALIZABLE isolation level.

Comment by Vladislav Lesin [ 2023-11-07 ]

I would also add to the comment that MySQL fix sets implicit lock on just-inserted clustered index record during IODKU execution. The lock is converted to explicit one during partial rollback. Then the lock is inherited to a gap lock when just inserted record is deleted on partial rollback. Such a way the gap should be locked in clustered index, and, as any IODKU statement inserts a record to clustered index before secondary indexes duplicates check, any other transaction which executes IODKU will be blocked on "INSERT" stage.

I think it explains the reason of implicit to explicit lock conversion in the fix.

Generated at Thu Feb 08 08:07:04 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.