InnoDB ignores errors of a fulltext parser plugin, so a row can stay unindexed or partly indexed without a warning

fts_tokenize_by_parser() calls parser->parse() and ignores its return value. For an ordinary INSERT or UPDATE, InnoDB tokenizes the document when the transaction commits (fts_commit()), so when a parser plugin fails, the row is committed but its words are not, or only partly, in the FULLTEXT index, and the client gets no warning; later searches can miss the row. The same holds when an index is built over existing rows (row_merge_fts_doc_tokenize_by_parser(): "We assume parse returns successfully here.").

Found by reading the code: simple_parser always returns 0, and the ngram parser proposed in MDEV-10267 returns 0 on these paths too, because InnoDB's add_word callbacks there always return 0. A third-party parser can fail, for example at an internal limit, as MeCab does in MySQL.

MySQL fixed the commit path in Bug#39838534 (https://github.com/mysql/mysql-server/commit/9ca87cd15634cbbb5b710048d28af733a2f43e0b; see Marko's comment on MDEV-10268): when parse() fails, InnoDB pushes the warning "InnoDB: Full-text parser failed; text may not be indexed and FULLTEXT searches may fail. See the error log for details reported by the parser", and trx_commit_low() refreshes the statement's warning count, so that an autocommit INSERT reports it too. They kept the deferred tokenization.

The code is the same in 10.11 and main. Found while working on MDEV-10267.