[MDEV-12676] InnoDB FTS duplicate key error Created: 2017-05-04  Updated: 2020-08-25  Resolved: 2017-10-16

Status: Closed
Project: MariaDB Server
Component/s: Full-text Search, Storage Engine - InnoDB, Storage Engine - XtraDB
Affects Version/s: 10.0.29
Fix Version/s: 10.0.33, 10.1.29, 10.2.10, 10.3.3

Type: Bug Priority: Major
Reporter: Will Fong Assignee: Marko Mäkelä
Resolution: Fixed Votes: 2
Labels: upstream

Issue Links:
Relates
relates to MDEV-12636 SELECT UNION writes garbage tail of V... Closed
relates to MDEV-15237 "Can't write; duplicate key in table"... Closed
relates to MDEV-19073 FTS row mismatch after crash recovery Closed

 Description   

We have a customer continuously getting this in their error log:

InnoDB: Error (Duplicate key) writing word node to FTS auxiliary index table.

I haven't been able to reproduce this issue on my side. But looking through the source code, it doesn't seem like this type of error should be reported in the error log. Perhaps it could be better handled internally (silently ignored?).

This is a problem because it uselessly fills up the error log.

Thanks,
-will

2017-08-10 - A careful code review would be needed to answer the question if this has any user-visible impact, or why this occurs in the first place.



 Comments   
Comment by Marko Mäkelä [ 2017-05-09 ]

As demonstrated in MDEV-11649 and MDEV-12636, it is possible that FTS_DOC_ID sometimes is uninitialized garbage, causing this error.

Comment by Will Fong [ 2017-05-09 ]

marko Thanks for the update. Can I get a confirmation that this doesn't affect query results / accuracy? It is just "error log noise"? Thanks!

Comment by Marko Mäkelä [ 2017-08-10 ]

After analyzing both MDEV-11649 and MDEV-12636, I think that this bug is unrelated to them.
A careful code review would be needed to answer the question if this has any user-visible impact, or why this occurs in the first place.
A repeatable test case would greatly help.

Comment by Dimitris Chatzis [ 2017-11-18 ]

Not fixed on latest 10.1.29

I am getting :

InnoDB: Error (Duplicate key) writing word node to FTS auxiliary index table.

when i use:

journalctl -u mariadb --no-pager

Comment by Marko Mäkelä [ 2017-11-20 ]

pamamolf, the fix of this particular error definitely is included in MariaDB 10.1.29.

When you report an error, please include a minimal test case, preferrably based on SQL. If I understand correctly, the command "journalctl" is querying the logs via systemd. So, apparently you are seeing a message in the MariaDB error log. The message would seem to be generated by fts_sync_write_words():

		if (error != DB_SUCCESS && !print_error) {
			ut_print_timestamp(stderr);
			fprintf(stderr, "  InnoDB: Error (%s) writing "
				"word node to FTS auxiliary index "
				"table.\n", ut_strerr(error));
 
			print_error = TRUE;
		}

This looks like a different duplicate key error. The one that was fixed was a race condition when assigning values for the hidden FTS_DOC_ID column in the user table. That is, the duplicate key error was reported for the hidden FTS_DOC_ID_INDEX(FTS_DOC_ID) in the user table. This reported duplicate key error is something different, in an "auxiliary table" that stores a part of the inverted index from tokenized words to sets of FTS_DOC_ID.

The error would seem to be that instead of updating an existing word entry by adding more FTS_DOC_ID to the set, the logic is trying to insert a duplicate entry for the same tokenized word. Please file a separate issue for it, with a reduced SQL test case, so that we can repeat and fix it.

I have a wild guess (which may be wrong): somewhere the code is using wrong collation when comparing words, and then instead of updating an existing word, it is trying to insert a duplicate one. Example: both the words 'bar' and 'bär' exist in the indexed column(s), and the collation threats these two words as equivalent, but some code inside InnoDB FTS is wrongly treating them as distinct words.

Comment by Dimitris Chatzis [ 2017-11-20 ]

I am not expert at all and don't know how to post a test case Sorry.....

But it sounds that the issue probably is what you guess:

"somewhere the code is using wrong collation when comparing words, and then instead of updating an existing word, it is trying to insert a duplicate one. Example: both the words 'bar' and 'bär' exist in the indexed column(s), and the collation threats these two words as equivalent, but some code inside InnoDB FTS is wrongly treating them as distinct words."

Comment by Massimo [ 2018-06-01 ]

Not fixed on latest 10.1.30-MariaDB-1~xenial

after many errors report as " InnoDB: Error (Duplicate key) writing word node to FTS auxiliary index table." and not table mention, we fix the issue in rebuilds the tables with fulltext on innodb tables based on STATISTIC tables on information schema.

after that the error did not show up anymore.

Comment by Marko Mäkelä [ 2018-06-04 ]

For the record, I closed this bug on October 16, 2017 because I had applied a fix (with a test case) for the duplicate key error for the hidden unique FTS_DOC_ID column.

The remaining duplicate key error was reported as MDEV-15237. If you have any information that could help reproduce or fix the problem, please post it there, not here.

Generated at Thu Feb 08 07:59:34 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.