# MDEV-10267 ngram full-text parser: design note for finishing the work

Status: proposal (Futureweb, 2026-10-23), attached to MDEV-10267. Basis: Sergei Golubchik's branch
`bb-12.3-MDEV-10267-ngram` (Rinat Ibragimov's 2020 plugin plus Serg's InnoDB follow-up), PR #1624, the
maria-developers reviews of 2020, and the MySQL 5.7–8.4 implementation.

## Goal

The goal is an `ngram` full-text parser plugin that:

- is merged into `main` (13.2), at the latest for 13.3 LTS, the target of the other features from the
  Foundation's MySQL-compatibility poll (ngram came third there, with 61 of 506 votes);
- behaves like MySQL's built-in `ngram` parser wherever this is cheap, so applications and dumps
  (`/*!50100 WITH PARSER ngram */`) move from MySQL unchanged;
- keeps Serg's direction: a plugin, no SQL-layer or plugin-API changes, and backportable later.

## Where the branch stands

- **Build:** the plugin does not compile on 12.x. It redefines `my_ci_ctype` and `my_ci_charlen` (now a macro and
  an inline function in `m_ctype.h`) and calls the removed `my_charlen()`. Buildbot fails on 13 of 16 builders.
- **InnoDB change (78c81737ac):** InnoDB no longer filters plugin tokens by length or stopword, like MyISAM and
  Aria. This is right, because bigrams would otherwise fall below `innodb_ft_min_token_size=3`. There is one
  regression: `fts_init_recover_all_docs` never sets `doc.stopwords`, so recovery re-tokenizes with the built-in
  parser without the stopword filter.
- **Plugin behaviour:** a pure character slicer. Spaces and punctuation become part of n-grams, which differs from
  MySQL. There is no MySQL short-wildcard rule and no stopword handling. `ngram_token_size` is GLOBAL, DYNAMIC and
  1..10.
- **Leftovers from review:**
  - the string is scanned twice;
  - `assert()` is used instead of `DBUG_ASSERT`;
  - a negative `ctype()` return (`MY_CS_TOOSMALLn`) is not handled;
  - no debian `.install` entry;
  - maturity is EXPERIMENTAL, so a stable server refuses it;
  - the Oracle copyright header is still in the file;
  - the MySQL tests from 5aa0752 have not been ported;
  - no benchmark (asked for by Serg in 2020).

## Proposed decisions (to confirm with Serg)

| # | Question | Proposal | Why |
|---|---|---|---|
| 1 | Tokenization | **As MySQL**: single-byte whitespace and punctuation separate words; n-grams are built within a word. Benchmark against the pure slicer on CJK and Latin text. | MDEV-28906 / compat80: the same index content and results as MySQL. Serg asked for a benchmark before choosing. |
| 2 | Token size | **Global `ngram_token_size`, 1..10, default 2** (MySQL-compatible). Keep it dynamic as in the branch, but document that existing indexes must be rebuilt after a change. A per-index `HA_IOPTION` stays a later step. | A parser cannot see index options without API changes. Marko accepted a global variable (Jira, 2020-07-31). Dynamic keeps the tests restart-free, as Marko asked. |
| 3 | Boolean mode / wildcards | Port MySQL's `ngram_term_convert`: a `*` term shorter than n stays a prefix search; otherwise the term becomes a phrase of n-grams. | Pure plugin code. It gives the same results as MySQL. |
| 4 | Stopwords | **First step: none inside the parser.** The difference from MySQL is documented: MySQL drops every n-gram that contains a stopword, so with the default list nothing containing "a" or "i" is indexed. | A parser has no access to InnoDB's per-table stopword lists without API changes. MySQL's rule mainly hurts Latin text. This can be revisited if Serg wants the MySQL rule. |
| 5 | Packaging | **Built in, but still a plugin** (as in MySQL): compiled into the server and loaded by default, can be disabled with `--ngram=OFF`. Maturity BETA in `main`. A dynamic module `fulltext_ngram` (`INSTALL SONAME`) is the fallback if Serg prefers it. | MySQL dumps and applications that use `WITH PARSER ngram` then load without a manual INSTALL. That is the migration case that brings most users to this issue. It is still "purely a plugin", with no server or API change. |
| 6 | InnoDB | Keep Serg's change. Fix the recovery path (`doc.stopwords`). Keep the `FTS_MAX_WORD_LEN` guard for plugin tokens. Add a release note: indexes that use a plugin parser should be rebuilt. | The regression and the guard matter for every parser plugin, not only ngram. |
| 7 | Out of scope | Hash partitioning of the aux tables for utf8mb4 CJK (on-disk format), `HA_IOPTION` per index, MeCab (MDEV-10268). | Each is a separate decision or ticket. |

## Work plan

1. **Rebase onto `main`** as two commits (InnoDB, plugin), on top of the MDEV-28730 refactor.
2. **InnoDB:** the recovery fix, the `FTS_MAX_WORD_LEN` guard, and re-recorded `innodb_fts` results.
3. **Plugin:**
   - use the current `m_ctype.h` API with a single scan, and handle negative `ctype()`;
   - MySQL tokenization (decision 1) and the wildcard rule (decision 3);
   - `DBUG_ASSERT`, a MariaDB header, maturity BETA.
4. **Tests:**
   - the existing ngram suite;
   - port MySQL's ngram tests (ngram, ngram_1, phrase, wildcard, limit, Bug #25873310 / #30933728 / #31228694 /
     #21638891 cases);
   - InnoDB, MyISAM and Aria;
   - big5, gbk, gb2312, sjis, ujis, euckr, gb18030, utf8mb4 (CJK and German compounds);
   - token size changes without restart;
   - recovery;
   - ASAN and UBSAN runs.
5. **Packaging and docs:** debian and rpm, a KB page draft, and a release note.
6. **Benchmark:** slicer vs MySQL tokenization; index size and query time on a CJK and a Latin corpus.
7. **PR against `main`:** contributed under the MCA, with buildbot green.

## Process (COMMUNITY_CONTRIBUTIONS.md)

- **Before starting:** the Jira is assigned, so reach out to Serg first (Jira and Zulip). Put the design into the
  Jira before the implementation.
- **Draft PR:** shows the work without entering any queue.
- **"Ready for review" PR:** joins the preliminary review queue run by the community team. The domain expert (Serg,
  for InnoDB also Marko) does the final review. As a new feature, it also gets a separate testing step by a test
  engineer.
- **Ready for review means:**
  - one commit per logical change (InnoDB, plugin);
  - the commit message follows CODING_STANDARDS and starts with "MDEV-10267";
  - attribution: Rinat as author of his commits, Serg as co-author of the InnoDB change;
  - the PR description repeats the commit message;
  - the target is `main`;
  - buildbot is green, or each failure is shown to be unrelated;
  - MCA accepted (CLA assistant) or 3-clause BSD.
- **Inactivity:** 21 days without activity in review moves the PR back to Draft, so answer review comments quickly.
- **Follow-ups:** what stays out of scope becomes its own MDEV, linked here (per-index token size, utf8mb4 CJK
  partitioning, MySQL stopword rule).