[MDEV-29342] Assertion failure in file que0que.cc line 728 Created: 2022-08-20  Updated: 2023-04-24  Resolved: 2022-09-06

Status: Closed
Project: MariaDB Server
Component/s: Data Manipulation - Insert, Data Manipulation - Update, Full-text Search, Storage Engine - InnoDB
Affects Version/s: 10.6.9, 10.6, 10.7, 10.8, 10.9, 10.10
Fix Version/s: 10.3.37, 10.4.27, 10.5.18, 10.6.10, 10.7.6, 10.8.5, 10.9.3, 10.10.2

Type: Bug Priority: Critical
Reporter: zEvilz Assignee: Thirunarayanan Balathandayuthapani
Resolution: Fixed Votes: 3
Labels: None
Environment:

Debian 10 amd64


Attachments: File archlinux-package-update-log-2022-08-22.log     File mariadb-10.8.4-crash-2022-08-22.log     File systemd-coredump-2022-08-22.log    
Issue Links:
Duplicate
duplicates MDEV-29335 Crash after Duplicate Key error Closed
is duplicated by MDEV-29053 safe_mutex: Trying to destroy a mutex... Closed
is duplicated by MDEV-29380 server crashes multiple times daily -... Closed
is duplicated by MDEV-29563 MariaDB 10.6.9 crashing every minute Closed
Problem/Incident
Relates
relates to MDEV-29511 "Page read from tablespace is corrupt... Closed

 Description   

Mariadb crash after each below query:

INSERT INTO `b_user_index` (`USER_ID`, `NAME`, `SECOND_NAME`, `LAST_NAME`, `WORK_POSITION`, `UF_DEPARTMENT_NAME`, `SEARCH_USER_CONTENT`, `SEARCH_ADMIN_CONTENT`, `SEARCH_DEPARTMENT_CONTENT`) VALUES (7000, '', '', '', '', '', '7000', '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989  не выбрано         0', '') ON DUPLICATE KEY UPDATE `NAME` = '', `SECOND_NAME` = '', `LAST_NAME` = '', `WORK_POSITION` = '', `UF_DEPARTMENT_NAME` = '', `SEARCH_USER_CONTENT` = IF(SHA1(SEARCH_USER_CONTENT) = 'b7525fa5653c5b45f75d490825626e0880b394f4', SEARCH_USER_CONTENT, '7000'), `SEARCH_ADMIN_CONTENT` = IF(SHA1(SEARCH_ADMIN_CONTENT) = 'ac932ff7b355d03b541911c13e64c2c600fd009c', SEARCH_ADMIN_CONTENT, '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989  не выбрано         0'), `SEARCH_DEPARTMENT_CONTENT` = IF(SHA1(SEARCH_DEPARTMENT_CONTENT) = 'da39a3ee5e6b4b0d3255bfef95601890afd80709', SEARCH_DEPARTMENT_CONTENT, '')

The problem started after updating to 10.6.9.

error log:

2022-08-20 14:02:28 62242 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `z2s_db`.`b_user_index`
2022-08-20 14:02:28 0x7f0fec0d6700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.9/storage/innobase/que/que0que.cc line 728
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220820 14:02:28 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
Server version: 10.6.9-MariaDB-1:10.6.9+maria~deb10-log
key_buffer_size=8589934592
read_buffer_size=131072
max_used_connections=50
max_threads=2502
thread_count=50
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 19022199 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
 
Thread pointer: 0x7f0960000c18
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f0fec0d5d98 thread_stack 0x49000
??:0(my_print_stacktrace)[0x559ba9edfb0e]
??:0(handle_fatal_signal)[0x559ba99c5555]
??:0(__restore_rt)[0x7f120640d730]
??:0(gsignal)[0x7f1205f647bb]
??:0(abort)[0x7f1205f4f535]
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559ba966b3d2]
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559ba96571ca]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e4ebb9]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e3c3ff]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e40ba1]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e426de]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e42945]
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e42ad5]
??:0(main)[0x559ba96cd5e8]
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559ba9d9565b]
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559ba9d9584f]
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x559ba9c82958]
??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x559ba99c93a3]
??:0(ha_commit_trans(THD*, bool))[0x559ba99ca3b5]
??:0(trans_commit_stmt(THD*))[0x559ba98b5483]
??:0(mysql_execute_command(THD*, bool))[0x559ba97a985f]
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x559ba979bf32]
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x559ba97a6b3a]
??:0(do_command(THD*, bool))[0x559ba97a81f7]
??:0(do_handle_one_connection(CONNECT*, bool))[0x559ba98a5d17]
??:0(handle_one_connection)[0x559ba98a605d]
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x559ba9be1b0e]
??:0(start_thread)[0x7f1206402fa3]
??:0(clone)[0x7f1206025eff]
 
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7f0960010b50): INSERT INTO `b_user_index` (`USER_ID`, `NAME`, `SECOND_NAME`, `LAST_NAME`, `WORK_POSITION`, `UF_DEPARTMENT_NAME`, `SEARCH_USER_CONTENT`, `SEARCH_ADMIN_CONTENT`, `SEARCH_DEPARTMENT_CONTENT`) VALUES (7000, '', '', '', '', '', '7000', '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989  не выбрано         0', '') ON DUPLICATE KEY UPDATE `NAME` = '', `SECOND_NAME` = '', `LAST_NAME` = '', `WORK_POSITION` = '', `UF_DEPARTMENT_NAME` = '', `SEARCH_USER_CONTENT` = IF(SHA1(SEARCH_USER_CONTENT) = 'b7525fa5653c5b45f75d490825626e0880b394f4', SEARCH_USER_CONTENT, '7000'), `SEARCH_ADMIN_CONTENT` = IF(SHA1(SEARCH_ADMIN_CONTENT) = 'ac932ff7b355d03b541911c13e64c2c600fd009c', SEARCH_ADMIN_CONTENT, '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989  не выбрано         0'), `SEARCH_DEPARTMENT_CONTENT` = IF(SHA1(SEARCH_DEPARTMENT_CONTENT) = 'da39a3ee5e6b4b0d3255bfef95601890afd80709', SEARCH_DEPARTMENT_CONTENT, '')
 
Connection ID (thread ID): 62242
Status: NOT_KILLED
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /var/lib/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             255271               255271               processes 
Max open files            1048576              1048576              files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       255271               255271               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
Kernel version: Linux version 4.19.0-20-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.235-1 (2022-03-17)
 
2022-08-20 14:02:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-08-20 14:02:35 0 [Note] InnoDB: Number of pools: 1
2022-08-20 14:02:35 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2022-08-20 14:02:35 0 [Note] InnoDB: Using Linux native AIO
2022-08-20 14:02:35 0 [Note] InnoDB: Initializing buffer pool, total size = 17179869184, chunk size = 134217728
2022-08-20 14:02:35 0 [Note] InnoDB: Completed initialization of buffer pool
2022-08-20 14:02:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=744061940175,744061940175
2022-08-20 14:02:42 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 104 row operations to undo
2022-08-20 14:02:42 0 [Note] InnoDB: Trx id counter is 2290494205
2022-08-20 14:02:42 0 [Note] InnoDB: Starting final batch to recover 63594 pages from redo log.
2022-08-20 14:02:44 0 [Note] InnoDB: 128 rollback segments are active.
2022-08-20 14:02:44 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2022-08-20 14:02:44 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2022-08-20 14:02:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-08-20 14:02:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-08-20 14:02:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-08-20 14:02:44 0 [Note] InnoDB: Rolled back recovered transaction 2290494166
2022-08-20 14:02:44 0 [Note] InnoDB: 10.6.9 started; log sequence number 744761503269; transaction id 2290494207
2022-08-20 14:02:44 0 [Note] InnoDB: Rolled back recovered transaction 2290494173
2022-08-20 14:02:44 0 [Note] InnoDB: Rollback of non-prepared transactions completed
2022-08-20 14:02:44 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-08-20 14:02:44 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-08-20 14:02:44 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
2022-08-20 14:02:44 0 [Note] Server socket created on IP: '127.0.0.1'.
2022-08-20 14:02:44 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.6.9-MariaDB-1:10.6.9+maria~deb10-log'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-08-20 14:02:46 0 [Note] InnoDB: Buffer pool(s) load completed at 220820 14:02:46



 Comments   
Comment by zEvilz [ 2022-08-21 ]

Similar queries break ib_logfile* files now. No problems after downgrading to version 10.6.8.

Comment by Marko Mäkelä [ 2022-08-22 ]

zEvilz, can you please provide the output of

SHOW CREATE TABLE b_user_index;

Also, please install the package mariadb-server-10.6-dbgsym and try to produce a full stack trace of the crash.

Comment by Thirunarayanan Balathandayuthapani [ 2022-08-22 ]

Previously, If there is a duplicate key happened due to FTS_DOC_ID(MDEV-15237) then only error message will happen in log file and it doesn't crash. But MDEV-25581 removes the background sync operation and allows user thread to do sync operation during transaction commit. It leads to crash. It is not a good idea to upgrade to 10.6.9 by those who're affected by MDEV-15237. Internally, we fail to repeat the MDEV-15237 problem, it would be great to get a test case from any affected user.

Comment by Arthur Borsboom [ 2022-08-22 ]

Yesterday, 21 August 2022, I have updated a Linux server including MariaDB which was updated from version 10.8.3 > 10.8.4
Since then, MariaDB crashes often with the same error message as the reporter.

[arthur@web1 ~]$ sudo coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Mon 2022-08-22 09:30:45 CEST 577 89 89 SIGABRT present /usr/bin/mariadbd 8.7M
Mon 2022-08-22 09:45:22 CEST 8143 89 89 SIGABRT present /usr/bin/mariadbd 4.7M
Mon 2022-08-22 09:50:42 CEST 8307 89 89 SIGABRT present /usr/bin/mariadbd 3.7M
Mon 2022-08-22 09:52:31 CEST 8430 89 89 SIGABRT present /usr/bin/mariadbd 2.2M
Mon 2022-08-22 09:53:45 CEST 8496 89 89 SIGABRT present /usr/bin/mariadbd 2.1M
Mon 2022-08-22 10:46:24 CEST 8563 89 89 SIGABRT present /usr/bin/mariadbd 4.7M
Mon 2022-08-22 11:02:53 CEST 563 89 89 SIGABRT present /usr/bin/mariadbd 3.1M

I have attached a snippet of the MariaDB log entries.
I have attached the last systemd-coredump as attachment.
I have attached the OS packages that have been updated, assuming that the cause might lie in the dependencies of MariaDB, such as libc or the Linux kernel.

If more information is needed, please let me know, what is needed and how I can retrieve this info.

Comment by Marko Mäkelä [ 2022-08-22 ]

ArthurBorsboom, can you please install the debug information for that server and then try to extract the stack trace from a core dump again? The .deb based package name should be mariadb-server-10.8-dbgsym. The .rpm based package name should start with MariaDB-server-debuginfo-10.8.4.

Comment by Arthur Borsboom [ 2022-08-22 ]

MariaDB [support]> SHOW CREATE TABLE ost__search;
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table       | Create Table                                                                                                                                                                                                                                                                                          |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ost__search | CREATE TABLE `ost__search` (
  `object_type` varchar(8) NOT NULL,
  `object_id` int(11) unsigned NOT NULL,
  `title` text DEFAULT NULL,
  `content` text DEFAULT NULL,
  PRIMARY KEY (`object_type`,`object_id`),
  FULLTEXT KEY `search` (`title`,`content`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.000 sec)
 
MariaDB [support]>

Comment by Arthur Borsboom [ 2022-08-22 ]

@Marko, I am not sure yet how to add debugging info.
The server is a production server, so I can't go too crazy replacing software.

However, if I can enable some flag and restart MariaDB or another service that would be fine.
MariaDB is running on Arch Linux.

Any suggestions?

Comment by Arthur Borsboom [ 2022-08-22 ]

[arthur@web1 ~]$ mariadbd --debug --help
2022-08-22 12:37:13 0 [Warning] 'debug' is disabled in this build

Comment by Marko Mäkelä [ 2022-08-22 ]

ArthurBorsboom, I am not familiar with the package structure on Arch Linux, so I do not know the name of the package that would contain the debug information that is normally omitted from the executables themselves. All I can say that in systemd-coredump-2022-08-22.log, the debug information is not available. Only some function names are available, but not function parameters or source file line numbers.

Debug instrumentation (cmake -DCMAKE_BUILD_TYPE=Debug) is something different. I would not advise anyone to run debug-instrumented executables in production.

Comment by Marko Mäkelä [ 2022-08-22 ]

ArthurBorsboom, please check https://archlinux.org/news/debug-packages-and-debuginfod/
For Debian, https://wiki.debian.org/AutomaticDebugPackages covers something similar.

Comment by Sergei Golubchik [ 2022-08-22 ]

ArthurBorsboom, according to https://wiki.archlinux.org/title/Debugging/Getting_traces you should use debuginfod, whatever it is, to download debug information for mariadbd. It seems to be supported, it appears in the https://debuginfod.archlinux.org/packages list

Comment by Arthur Borsboom [ 2022-08-22 ]

I have to dive into the debugging to retrieve this info.

With the command [sudo coredumpctl gdb 863] I have been able to retrieve an incomplete backtrace.

Core was generated by `/usr/bin/mariadbd'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fbd9fea14dc in ?? () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7fbd9f0b56c0 (LWP 884))]
(gdb) bt
#0 0x00007fbd9fea14dc in ?? () from /usr/lib/libc.so.6
#1 0x00007fbd9fe51998 in raise () from /usr/lib/libc.so.6
#2 0x00007fbd9fe3b5c7 in abort () from /usr/lib/libc.so.6
#3 0x00005563c7d29399 in ?? ()
#4 0x00005563c7d12a64 in ?? ()
#5 0x00005563c8518089 in ?? ()
#6 0x00005563c851e079 in ?? ()
#7 0x00005563c851f672 in ?? ()
#8 0x00005563c851f879 in ?? ()
#9 0x00005563c7d9825a in ?? ()
#10 0x00005563c84b4926 in ?? ()
#11 0x00005563c84b4ae2 in ?? ()
#12 0x00005563c83cd508 in ?? ()
#13 0x00005563c83cd68d in ?? ()
#14 0x00005563c83d84bb in ?? ()
#15 0x00005563c8162c5f in ?? ()
#16 0x00005563c8164006 in ha_commit_trans(THD*, bool) ()
#17 0x00005563c804866a in trans_commit_stmt(THD*) ()
#18 0x00005563c7ef934d in mysql_execute_command(THD*, bool) ()
#19 0x00005563c7f052e2 in mysql_parse(THD*, char*, unsigned int, Parser_state*) ()
#20 0x00005563c7ef543e in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) ()
#21 0x00005563c7ef75e8 in do_command(THD*, bool) ()
#22 0x00005563c8042724 in do_handle_one_connection(CONNECT*, bool) ()
#23 0x00005563c8042c0b in handle_one_connection ()
#24 0x00005563c8357051 in ?? ()
#25 0x00007fbd9fe9f78d in ?? () from /usr/lib/libc.so.6
#26 0x00007fbd9ff208e4 in clone () from /usr/lib/libc.so.6
(gdb)

I will try to fill in the '??' with more information, possibly with the information provided by Sergei.
Which debug information is missing? Is it mariadb, libc, both or others?

Comment by Marko Mäkelä [ 2022-08-22 ]

ArthurBorsboom, missing debuginfo for libc does not matter that much. I would assume that abort() was invoked somewhere inside fts_commit() of InnoDB. To see full stack traces of mariadbd, you will need the debug information for MariaDB Server.

Once we have a useful stack trace, we may need some more information from the crash. Some relevant information maybe unavailable due to MDEV-10814 and some special measures may be necessary due to it.

Comment by Arthur Borsboom [ 2022-08-22 ]

While trying to enable debugging, I can confirm (re)running the following query does trigger the core dump.

REPLACE INTO ost__search SET object_type='H', object_id=9387, content='Hi Chris, That\'s all good. See you this afternoon. BR, Arthur.', title='';

Comment by Marko Mäkelä [ 2022-08-22 ]

ArthurBorsboom, it is nice that you can reproduce this at will, hopefully also in a staging environment where you would copy the data from the production server.

For now, I am afraid that you’d better to revert the production server to the older version (10.8.3).

For the final debugging, I would suggest to attach GDB to the running server (sudo gdb -p $(pgrep mariadbd), continue), run that crashing statement, and once the SIGABRT is reported in GDB, execute backtrace. thiru would tell which other commands he would like to be executed, once he has a stack trace of the crash for reference.

Comment by Arthur Borsboom [ 2022-08-22 ]

Hi Marko,

After taking another backup for a debugging environment, I have rolled back the following packages.

downgraded mariadb-libs (10.8.4-1 -> 10.8.3-1)
downgraded mariadb-clients (10.8.4-1 -> 10.8.3-1)
downgraded mariadb (10.8.4-1 -> 10.8.3-1)

This has resolved the issue for me.
Once the debugging environment has been setup, I will try to compile a debug enabled version of MariaDB in order to create a better back trace.

Comment by Arthur Borsboom [ 2022-08-23 ]

I have created a copy of the VM and prepped it for debugging.
I have created a mariadb package with debugging enabled.
The crash is still present and this is the back trace.

#0  0x00007f23caea14dc in ?? () from /usr/lib/libc.so.6
#1  0x00007f23cae51998 in raise () from /usr/lib/libc.so.6
#2  0x00007f23cae3b5c7 in abort () from /usr/lib/libc.so.6
#3  0x00005581e9f382da in ut_dbg_assertion_failed (expr=expr@entry=0x5581ea9dbbd8 "trx->error_state == DB_SUCCESS", 
    file=file@entry=0x5581ea9dbb98 "/usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc", line=line@entry=728)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/ut/ut0dbg.cc:60
#4  0x00005581e9f25842 in que_run_threads (thr=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc:728
#5  que_run_threads (thr=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc:722
#6  0x00005581ea6f35bb in fts_eval_sql (trx=trx@entry=0x7f23b849e100, graph=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0sql.cc:163
#7  0x00005581ea6e2fe6 in fts_write_node (trx=0x7f23b849e100, graph=0x7f23abc14ed0, fts_table=0x7f23ca130ad0, word=<optimized out>, 
    node=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3763
#8  0x00005581ea6e6861 in fts_sync_write_words (index_cache=0x7f23abc14e80, trx=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3877
#9  fts_sync_index (index_cache=0x7f23abc14e80, sync=0x7f23ca9fdf90) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3959
#10 fts_sync (sync=0x7f23ca9fdf90) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4105
#11 0x00005581ea6e8182 in fts_sync_table (table=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4150
#12 fts_sync_table (table=0x7f23ca94f480) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4145
#13 fts_add_doc_by_id (ftt=<optimized out>, doc_id=18758) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3468
#14 0x00005581ea6e8295 in fts_add (ftt=<optimized out>, row=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2748
#15 0x00005581ea6e8461 in fts_modify (row=<optimized out>, ftt=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2862
#16 fts_commit_table (ftt=0x7f23abf69690) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2914
#17 fts_commit (trx=trx@entry=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2959
#18 0x00005581e9f9aecf in trx_t::commit_low (this=0x7f23b849cb00, mtr=0x7f23ca131650)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1414
#19 0x00005581ea659866 in trx_t::commit_persist (this=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1463
#20 0x00005581ea6599a2 in trx_t::commit (this=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1472
#21 trx_commit_for_mysql (trx=trx@entry=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1589
#22 0x00005581ea572b01 in innobase_commit_low (trx=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4348
#23 0x00005581ea572bd7 in innobase_commit_ordered_2 (trx=0x7f23b849cb00, thd=0x7f23abc25198)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4454
#24 0x00005581ea572d4b in innobase_commit (hton=<optimized out>, thd=0x7f23abc25198, commit_trx=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4555
#25 0x00005581ea2c82bf in commit_one_phase_2 (thd=0x7f23abc25198, all=<optimized out>, trans=0x7f23abc289a8, is_real_trans=true)
    at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:2067
#26 0x00005581ea2c8474 in ha_commit_one_phase (thd=<optimized out>, all=<optimized out>) at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:2020
#27 0x00005581ea2d46ae in ha_commit_trans (thd=thd@entry=0x7f23abc25198, all=all@entry=false) at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:1814
--Type <RET> for more, q to quit, c to continue without paging--
#28 0x00005581ea1be666 in trans_commit_stmt (thd=thd@entry=0x7f23abc25198) at /usr/src/debug/mariadb-10.8.4/sql/transaction.cc:472
#29 0x00005581ea09b6d5 in mysql_execute_command (thd=0x7f23abc25198, is_called_from_prepared_stmt=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:6049
#30 0x00005581ea09fea3 in mysql_parse (thd=0x7f23abc25198, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:8028
#31 0x00005581ea0a1ff6 in dispatch_command (command=COM_QUERY, thd=0x7f23abc25198, packet=<optimized out>, packet_length=<optimized out>, 
    blocking=<optimized out>) at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:1991
#32 0x00005581ea0a356b in do_command (thd=0x7f23abc25198, blocking=blocking@entry=true) at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:1407
#33 0x00005581ea1af199 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true)
    at /usr/src/debug/mariadb-10.8.4/sql/sql_connect.cc:1418
#34 0x00005581ea1af53e in handle_one_connection (arg=arg@entry=0x7f23ca858b98) at /usr/src/debug/mariadb-10.8.4/sql/sql_connect.cc:1312
#35 0x00005581ea4e445d in pfs_spawn_thread (arg=0x7f23ca94f418) at /usr/src/debug/mariadb-10.8.4/storage/perfschema/pfs.cc:2201
#36 0x00007f23cae9f78d in ?? () from /usr/lib/libc.so.6
#37 0x00007f23caf208e4 in clone () from /usr/lib/libc.so.6

I guess this is more helpful?

Comment by Thirunarayanan Balathandayuthapani [ 2022-08-23 ]

Hi ArthurBorsboom,
Thanks for sending the stack trace. Looks like InnoDB encountered duplicate entry in FTS auxiliary table. (word, first_doc_id).
first_doc_id could be 18578 based on fts_add_doc_by_id parameter. It would be great to find the word and first doc id

f 8
info locals
 
Above command should show something similar like the following:
 
(gdb) info locals
fts_node = ZZZ
word = YYY
fts_table = {type = FTS_INDEX_TABLE, table_id = 18, index_id = 30, suffix = 0x5555571684c3 "INDEX_3", table = 0x7fff901fd2c8, charset = 0xe969696900000008}
n_nodes = 0
n_words = 1
rbt_node = 0x7fff90074778
error = DB_SUCCESS
table = 0x7fff901fd2c8
 
After that, please type cast the fts_node and word like below:
 
p *(fts_node_t*) ZZZ
p *(fts_tokenizer_word_t*) YYY
 
f 30
p thd->query_string

Based on query_string output, please send the show create table output.

Do you run any DDL on the affected table ?

Comment by Arthur Borsboom [ 2022-08-23 ]

Hi Thirunarayanan,

I have (more or less blindly) executed the commands in gdb; the fts_node is unavailable (optimized out).

(gdb) f 8
#8  0x00005581ea6e6861 in fts_sync_write_words (index_cache=0x7f23abc14e80, trx=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3877
3877				error = fts_write_node(
(gdb) info locals
fts_node = <optimized out>
i = 0
selected = <optimized out>
word = 0x7f23abfef038
fts_table = {type = FTS_INDEX_TABLE, table_id = 65756, index_id = 33145, suffix = 0x5581ea9f2ed3 "INDEX_1", table = 0x7f23ca94f480, 
  charset = 0x5581ea661f24 <rbt_add_node(ib_rbt_t*, ib_rbt_bound_t*, void const*)+164>}
n_nodes = 16
n_words = 3666
rbt_node = 0x7f23abfef018
error = <optimized out>
table = 0x7f23ca94f480
err_exit = <optimized out>
fts_table = <optimized out>
n_nodes = <optimized out>
n_words = <optimized out>
rbt_node = <optimized out>
error = <optimized out>
table = <optimized out>
i = <optimized out>
selected = <optimized out>
word = <optimized out>
fts_node = <optimized out>
(gdb) p *(fts_tokenizer_word_t*) 0x7f23abfef038
$1 = {text = {f_str = 0x7f23abfc9910 "130", f_len = 3, f_n_char = 3}, nodes = 0x7f23abfc9828}
(gdb) f 30
#30 0x00005581ea09fea3 in mysql_parse (thd=0x7f23abc25198, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>)
    at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:8028
8028	          error= mysql_execute_command(thd);
(gdb) p thd->query_string
$2 = {string = {
    str = 0x7f23abc4c5b0 "REPLACE INTO ost__search SET object_type='H', object_id=9387, content='Hi Chris, That\\'s all good. See you this afternoon. BR, Arthur.', title=''", length = 145}, cs = 0x5581eb008040 <my_charset_utf8mb3_general_ci>}
(gdb)

create table output

+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table       | Create Table                                                                                                                                                                                                                                                                                          |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ost__search | CREATE TABLE `ost__search` (
  `object_type` varchar(8) NOT NULL,
  `object_id` int(11) unsigned NOT NULL,
  `title` text DEFAULT NULL,
  `content` text DEFAULT NULL,
  PRIMARY KEY (`object_type`,`object_id`),
  FULLTEXT KEY `search` (`title`,`content`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

TM> Do you run any DDL on the affected table ?

I have no idea. The database is part of an opensource project called osTicket, which is to me a black box.

Do you need more information?

Comment by Thirunarayanan Balathandayuthapani [ 2022-08-24 ]

Thanks for sending the results ArthurBorsboom. I would like to verify the first doc id and last doc id. The following could do the trick

 
f 7
info locals

Comment by zEvilz [ 2022-08-24 ]

@marko this is a production server. I can't test on it. Later I will reproduce the problem on a test server and make trace on it.

CREATE TABLE `b_user_index` (
  `USER_ID` int(11) NOT NULL,
  `SEARCH_USER_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `SEARCH_DEPARTMENT_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `SEARCH_ADMIN_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `LAST_NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `SECOND_NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `WORK_POSITION` varchar(255) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  `UF_DEPARTMENT_NAME` varchar(255) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
  PRIMARY KEY (`USER_ID`),
  FULLTEXT KEY `IXF_B_USER_INDEX_1` (`SEARCH_USER_CONTENT`),
  FULLTEXT KEY `IXF_B_USER_INDEX_2` (`SEARCH_DEPARTMENT_CONTENT`),
  FULLTEXT KEY `IXF_B_USER_INDEX_3` (`SEARCH_ADMIN_CONTENT`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci

Comment by Arthur Borsboom [ 2022-08-24 ]

@Thirunarayanan, f7 results in the first and last doc id, which seem to be similar.

(gdb) f 7
#7  0x00005581ea6e2fe6 in fts_write_node (trx=0x7f23b849e100, graph=0x7f23abc14ed0, fts_table=0x7f23ca130ad0, word=<optimized out>, 
    node=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3763
3763		error = fts_eval_sql(trx, *graph);
(gdb) info locals
info = 0x7f23abf25700
error = <optimized out>
doc_count = 16777216
start_time = 1661249843
last_doc_id = 16809122659206955008
first_doc_id = 16809122659206955008
table_name = "support/FTS_", '0' <repeats 11 times>, "100dc_", '0' <repeats 12 times>, "8179_INDEX_1\000\000\000\000\270ǫ#\177\000\000\002\000\000\000\000\000\000\000 *`\266#\177\000\000@\b\023\312#\177\000\000K][\352\201U\000\000P\017\023\312#\177\000\000\250\017\023\312#\177\000\000p\b\023\312>\000\000\000\002\000\000\000\000\000\000\000p\b\023\312#\177", '\000' <repeats 11 times>, "\200w\266#\177\000\000\247\063_\352\201U\000\000\001\000\000\000\000\000\000\000h\301\234\312#\177\000\000\000V\207\312#\177\000\000\300\340I\270#\177\000\000@\n\023\312#\177\000\000*"...
(gdb)

Comment by zEvilz [ 2022-08-24 ]

@marko the problem not reproduce on test server with same software configuration.

Comment by Arthur Borsboom [ 2022-08-25 ]

To summarize the bug report:

  • zEvilz and Arthur Borsboom both report a crash with the same error message "(Duplicate key) writing word node to FTS auxiliary".
  • zEvilz has upgraded from 10.6.8 > 10.6.9
  • Arthur Borsboom has upgraded from 10.8.3 > 10.8.4
  • Reverting the upgrade resolves the problem for both users.
  • zEvilz can reproduce the crash in the production environment, but not in the test environment
  • Arthur Borsboom can reproduce the crash both in the production and test environment, where debugging is enabled.

Can we provide more information to help solving the issue?

Comment by Harro Verton [ 2022-08-28 ]

Same problem here in our Galera cluster. Spend most of the afternoon, evening and night (now 01:38 here) trying to get things fixed. Ended up with multiple nodes with corrupt data, which I downgraded to 10.6.8, wiped and rebuild using SST from the only node that seemed to have ok data. The entire cluster is now back to 10.6.8, and so far, so good (famous last words...).

2022-08-27 19:41:58 7 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `IPBoard`.`topics`
2022-08-27 19:41:58 0x7f7ed00cf700  InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.9/storage/innobase/que/que0que.cc line 728
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mariadbd startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
InnoDB: about forcing recovery.
220827 19:41:58 [ERROR] mysqld got signal 6 ;

We run MariaDB on Almalinux 8.

Comment by Elena Stepanova [ 2022-08-28 ]

test case

--source include/have_innodb.inc
 
CREATE TABLE t (
  pk int,
  c01 varchar(255),
  c02 varchar(255),
  c03 varchar(255),
  c04 varchar(255),
  c05 varchar(255),
  c06 varchar(255),
  c07 varchar(255),
  c08 varchar(255),
  c09 varchar(255),
  c10 varchar(255),
  c11 varchar(255),
  c12 varchar(255),
  c13 varchar(255),
  c14 varchar(255),
  c15 varchar(255),
  c16 varchar(255),
  c17 varchar(255),
  c18 varchar(255),
  PRIMARY KEY (pk),
  FULLTEXT KEY idx01 (c01),
  FULLTEXT KEY idx02 (c02),
  FULLTEXT KEY idx03 (c03),
  FULLTEXT KEY idx04 (c04),
  FULLTEXT KEY idx05 (c05),
  FULLTEXT KEY idx06 (c06),
  FULLTEXT KEY idx07 (c07),
  FULLTEXT KEY idx08 (c08),
  FULLTEXT KEY idx09 (c09),
  FULLTEXT KEY idx10 (c10),
  FULLTEXT KEY idx11 (c11),
  FULLTEXT KEY idx12 (c12),
  FULLTEXT KEY idx13 (c13),
  FULLTEXT KEY idx14 (c14),
  FULLTEXT KEY idx15 (c15),
  FULLTEXT KEY idx16 (c16),
  FULLTEXT KEY idx17 (c17),
  FULLTEXT KEY idx18 (c18),
  FULLTEXT KEY idx19 (c05),
  FULLTEXT KEY idx20 (c04),
  FULLTEXT KEY idx21 (c01),
  FULLTEXT KEY idx22 (c09),
  FULLTEXT KEY idx23 (c15)
) ENGINE=InnoDB;
 
INSERT INTO t VALUES
(1,'drag','shortly','twelve','correlation','contribution','embrace','pilot','sprinkle','social','reaction','validity','view','tent','survey','fortune','certain','merchant','ladder'),
(2,'tale','warmth','skill','low','scan','whom','wave','commander','nerve','ah','apologize','prosecutor','continued','yourself','maintenance','cruise','yeah','gear'),
(3,'exhibition','upstairs','anyone','presidency','type','proper','nature','wealthy','jury','mostly','shorts','commit','concert','weekly','criteria','percentage','fold','complex'),
(4,'finding','demand','weekend','rabbit','dish','relief','counseling','so','the','chef','colony','instinct','widespread','additional','painter','outside','forbid','blanket'),
(7,'move','outside','realm','indication','bread','bitter','college','particular','mr','neat','belief','animal','conventional','honey','theoretical','slap','indigenous','desperate'),
(9,'respect','up','crime','hot','democracy','war','county','outfit','decrease','training','diplomat','terrific','score','pill','man','motor','united','joke'),
(10,'praise','considerably','characteristic','processor','breeze','stock','popular','flame','medal','behind','juice','rule','arm','opening','soul','verbal','gun','e-mail'),
(11,'strain','treaty','rival','european','stone','steep','lawsuit','climb','hostage','logic','condition','finger','clock','goal','investment','gravity','comment','count'),
(12,'them','plunge','fantastic','impressive','engagement','defensive','difficult','afraid','accident','genetic','frequency','accounting','environmental','reportedly','incident','assistant','engineer','foundation'),
(13,'survival','save','partially','track','lung','tall','story','center','plate','prayer','blank','screw','testimony','bean','developer','union','lock','purple'),
(16,'responsible','yell','break','squeeze','people','leather','muscle','think','consistently','concerned','mistake','persist','number','mobile','maintenance','visual','inquiry','interpretation'),
(17,'adventure','trip','chill','halfway','setting','battle','wise','factor','cop','recall','adopt','sir','municipal','fifteen','those','peace','particle','spot'),
(18,'former','execute','breathe','front','bus','violence','thrive','click','victim','chapter','pregnant','poke','integrity','matter','poll','province','except','reduce'),
(19,'magnitude','low','shame','announce','soccer','film','uncertainty','pen','pulse','event','cool','relax','commander','array','thrive','most','locate','mask'),
(20,'survey','effectiveness','scratch','document','without','law','rapid','around','length','boost','grass','yourself','poll','combined','surface','starting','reinforce','happy'),
(23,'recession','equation','lifestyle','watch','condition','wheelchair','variable','reliability','flexibility','acceptance','lend','beside','rock','diagnosis','govern','mountain','wonderful','conduct'),
(31,'prospect','station','chart','iraqi','imply','sand','evidence','protect','altogether','advertising','admit','comment','get','uh','therapist','harmony','edit','circuit'),
(33,'scream','input','flip','deficit','pepper','transformation','week','solely','needle','give','market','passing','bit','abstract','dark','politically','cook','branch'),
(34,'rage','eligible','cousin','forest','reason','color','stiff','encounter','angle','consequence','bold','negotiate','matter','emotion','proportion','emotion','draft','register'),
(35,'thanks','substantial','country','guilty','accurately','family','tradition','sun','independent','openly','so-called','cancer','tall','diagnosis','powerful','frustrate','you','solar'),
(36,'role','prepare','brick','replace','transmit','ah','silent','slot','sort','arrive','reputation','unhappy','adequate','feed','risky','cover','urge','tumor'),
(37,'advanced','yourself','sit','brilliant','calendar','acquisition','expected','whose','headline','eleven','exactly','evolve','sufficient','adoption','palestinian','adopt','turn','tumor'),
(40,'violation','plain','reliable','grow','thinking','lightning','stream','culture','invitation','illness','counterpart','overwhelming','nose','hurry','carve','option','great','instead'),
(43,'form','marker','earnings','custom','each','temple','tissue','lid','indicator','bulb','around','banking','bulk','construction','orange','programming','opposition','burning'),
(44,'negotiate','suffer','scandal','straighten','informal','ie','total','block','recipe','forever','practically','social','rear','thousand','short-term','drain','laughter','director'),
(46,'boring','tongue','legitimate','via','decision','deficit','others','ridiculous','cheat','angel','try','communication','diversity','bucket','trunk','impossible','cook','banker'),
(50,'tonight','ideological','scratch','plenty','communication','comparable','thrive','appear','difficult','chinese','likelihood','mention','message','seal','greek','gene','somewhere','shelter'),
(51,'high-tech','none','statistical','improvement','coup','enjoy','complain','suggestion','traditional','it','man','unity','convey','the','ice','corruption','palestinian','celebration'),
(56,'ass','receiver','husband','undertake','offense','billion','extensive','plus','confrontation','bury','back','settlement','currency','entire','buy','excellent','stomach','enact'),
(60,'silent','newspaper','elite','spoon','restriction','confront','fortune','time','improvement','forest','guest','prescription','tea','attract','adviser','tax','grace','hand'),
(61,'album','emphasis','originally','strengthen','local','gravity','equip','neck','string','cash','hand','mean','viewer','compose','virtue','too','scent','soap'),
(62,'quit','skin','elegant','engagement','handle','battle','label','collar','port','buck','smell','embarrassed','severe','diamond','translate','main','resist','slip'),
(63,'confront','scramble','succeed','convert','teammate','appreciation','fleet','agenda','universe','businessman','size','single','familiar','dense','sometime','dear','roman','physics'),
(65,'thin','starting','productivity','dna','level','mass','nominee','cling','cry','mountain','sure','earth','devote','substance','surround','interrupt','security','pole'),
(71,'lie','voluntary','cart','half','variety','whose','fact','aide','skirt','eating','intensity','acceptable','constitution','damn','plant','response','feminist','truck'),
(72,'provider','reader','information','continuous','health','steady','case','kit','habit','joy','account','register','basic','paint','sexual','convert','outstanding','therapy'),
(75,'reference','professor','importance','guard','taxpayer','resource','envelope','priest','painting','perfectly','nose','marketplace','somehow','alike','help','creation','toward','cloud'),
(77,'transmission','strictly','donate','sacrifice','rib','environment','early','then','shared','force','spell','room','staff','islam','tuck','cross','occasion','tune'),
(78,'spit','lie','attendance','battery','edge','hidden','broad','african-american','seed','house','random','outer','cite','liver','important','wealth','drift','mean'),
(79,'ring','plate','grow','page','forbid','athlete','extension','technological','chinese','incorporate','complaint','appear','apply','approach','throw','jaw','branch','down'),
(80,'inventory','somewhere','though','rabbit','smooth','will','happily','heritage','post','complexity','technological','logical','concentrate','critic','nurse','ballot','obstacle','others'),
(81,'lobby','intense','estimate','favorite','injury','tea','map','hard','ballot','contain','sing','copy','earth','worldwide','outcome','thanks','favorable','hay'),
(82,'meat','silent','purpose','democratic','reason','rely','ladder','legislation','lion','eating','shelter','charter','ecological','bible','evaluation','adopt','worth','human'),
(83,'repeat','heal','gathering','crazy','circle','stare','comply','persuade','create','color','substantially','exclusive','natural','anxiety','golf','commodity','wood','luck'),
(88,'wear','indigenous','disappointed','qualify','alley','accomplishment','hearing','melt','married','assemble','tight','scope','four','church','support','refuge','channel','employment'),
(89,'counter','spanish','range','incident','session','purse','egg','squeeze','frustrate','please','clue','math','awareness','effort','couch','sovereignty','hallway','obligation'),
(90,'accelerate','slam','export','parade','industrial','forest','whereas','injure','best','river','feeling','curtain','lobby','lung','jacket','breeze','frustrate','definitely'),
(93,'contest','nowhere','boring','sofa','guitar','click','manufacturer','resident','marble','education','therapy','grin','impress','undergo','ongoing','workout','modify','irish'),
(94,'civilian','fee','flag','infection','fatal','above','quality','announcement','bare','acknowledge','estate','refuse','frequently','size','holiday','scale','transmission','short-term'),
(99,'motor','weakness','organize','criteria','isolated','mathematics','shame','fun','concede','pattern','choose','gap','expect','suspect','shame','japanese','bill','rock'),
(100,'protein','trick','salad','privately','live','genuine','survivor','afternoon','component','comparison','church','sunny','satellite','dancing','southeast','smoke','journey','old-fashioned'),
(101,'disc','understand','throw','monitor','likelihood','top','psychology','instruct','subtle','ethics','hardware','retailer','immigration','paper','ask','drill','devastating','interested'),
(102,'hostile','committee','over','density','newly','franchise','married','sand','cover','banker','athletic','twentieth','face','wet','invention','waste','film','rebuild'),
(105,'politician','wolf','fever','interval','convert','limitation','jurisdiction','republican','whether','tree','construction','short-term','uncomfortable','dilemma','legally','significant','fever','science'),
(106,'picture','growth','league','hook','valuable','hypothesis','precisely','mortality','steel','emphasis','flesh','roughly','pressure','expert','distinctive','environmental','wash','hispanic'),
(107,'introduction','empty','collaboration','administrator','several','that','embrace','fist','panic','easy','blast','ease','prefer','magazine','garage','undergo','compensation','disk'),
(108,'achieve','yesterday','satisfaction','cast','relative','suicide','actual','member','internal','support','compelling','grave','various','cool','ocean','cash','performance','sector'),
(113,'portrait','sanction','anything','can','bombing','attend','conscience','dedicate','justice','pond','fighting','credit','murder','pollution','romance','frontier','eligible','textbook'),
(114,'study','worth','musical','trade','port','tourist','sing','sensitive','five','chapter','expedition','flame','spiritual','rib','please','slowly','ankle','boom'),
(115,'dry','unit','nearby','numerous','safety','initiative','headquarters','sibling','generation','eating','curriculum','activist','mistake','squeeze','land','anxiety','era','timber'),
(116,'prevention','such','integrate','drill','quote','commodity','implement','household','relief','softly','agenda','venture','headline','cheer','dawn','chance','crucial','contract'),
(117,'weave','flexibility','secular','confront','servant','rebuild','essential','quest','ceiling','equity','relative','squeeze','line','form','cross','economics','grandparent','change'),
(118,'push','scary','determine','implementation','similar','aids','hostage','oil','character','employer','addition','telescope','correctly','envelope','corporate','immigration','slavery','initial'),
(119,'furniture','ring','productivity','administrator','cold','uncomfortable','stretch','shallow','opposition','claim','strike','mechanical','ambitious','president','execution','team','modify','portion'),
(121,'spread','environment','rib','wilderness','arrival','efficiency','respond','husband','forgive','adolescent','camera','due','maintenance','toward','beach','detective','personnel','machine'),
(122,'poke','large','mostly','hardly','cartoon','attention','traditionally','reportedly','phone','application','contemporary','preach','suicide','mom','essential','concert','notice','slap'),
(123,'distance','lend','re','bathroom','bit','oak','skip','advertising','device','daughter','fantastic','scenario','fare','inner','backyard','objection','sell','frustration'),
(126,'restore','dessert','response','banana','seemingly','ambition','pen','alien','hard','arrangement','influence','aisle','limited','warmth','century','later','overwhelming','fun'),
(128,'plan','version','sometime','cut','ignore','slope','indicator','bed','bathroom','carpet','charter','championship','cat','negative','tie','peer','float','evaluation'),
(134,'hate','slight','cooperative','coffee','accountability','vocal','uncomfortable','hat','humor','racial','sentence','advice','serving','input','exercise','welfare','lens','dying'),
(136,'journey','patience','importance','mayor','wheat','beginning','leaf','cheap','blessing','error','pen','hockey','same','syndrome','scatter','mountain','globe','update'),
(137,'cargo','bullet','whoever','reduce','envelope','yell','failure','visit','creative','apologize','soften','plain','remind','highway','retail','weak','lead','male'),
(146,'corporate','wealth','faculty','ruin','complicated','surprisingly','cup','outsider','descend','hazard','attend','cattle','bunch','tie','premium','due','unity','retail'),
(149,'ensure','suck','spite','revolution','pipe','inquiry','frown','retain','dessert','drive','historic','bike','exclusive','promotion','attractive','pine','family','mortality'),
(151,'sweat','lady','essential','clothing','otherwise','extraordinary','yield','artist','reply','influential','piano','concern','wisdom','stir','acceptance','crowd','increased','shrimp'),
(152,'candy','divorce','compromise','color','twice','one','extend','league','telescope','abroad','available','realize','calculate','voting','disease','fork','concede','rubber'),
(155,'testify','shorts','automatically','leave','proud','scratch','short','symptom','involve','although','confirm','known','rate','remaining','sacred','fisherman','modest','satisfaction'),
(156,'conspiracy','whatever','genetic','rid','acid','swing','painting','gang','physics','a','import','tender','hell','teen','pleasure','above','painful','faith'),
(157,'i','recent','suicide','establish','name','literary','furniture','drug','twenty','debut','dawn','bury','ago','presumably','tool','any','two','grow'),
(159,'success','fat','persian','republic','tail','oven','exotic','glad','ambitious','abortion','impression','flower','salary','southwest','famous','humor','toxic','sign'),
(160,'sick','week','music','press','remove','count','wild','read','want','affect','choice','campaign','forward','substantially','jacket','instance','photographer','bit'),
(161,'feel','connect','scandal','deposit','traditional','campus','aide','video','virtue','bull','literature','partly','height','leadership','battle','action','celebration','promotion'),
(163,'similarly','wide','moreover','marketing','honest','combined','various','initial','bear','pose','trading','sink','unknown','bush','incorporate','salt','classroom','weapon'),
(166,'flavor','spread','gym','islamic','offering','pm','fish','dancer','particularly','nominee','simultaneously','ceo','tune','tear','accident','resort','endorse','peanut'),
(167,'their','supreme','childhood','technical','contact','skin','clothing','lie','critic','professor','discrimination','queen','queen','ban','beautiful','parish','oil','opening'),
(168,'owner','thumb','cry','venture','wander','group','desk','pace','array','functional','dedicate','acknowledge','enemy','devastating','regulate','inmate','closet','whisper'),
(171,'death','temple','ring','intend','cliff','unless','quality','fever','port','carrier','ribbon','shut','servant','welfare','commission','country','plain','result'),
(172,'gang','distinguish','norm','characterize','unity','internet','diamond','consensus','expose','climate','uncle','american','disappointed','meaning','infection','impress','interaction','colorful'),
(176,'companion','virtue','exchange','lucky','invest','sophisticated','desperately','thoroughly','essentially','hard','cottage','autonomy','therapy','sustain','cute','chin','regime','quantity'),
(177,'suit','organization','planner','waist','everyday','correct','drink','earn','quality','accurately','operator','alike','defendant','wall','as','tent','screening','mount'),
(180,'ensure','reliability','desperately','shoe','rough','global','reject','crack','comprise','adequate','appreciate','we','seminar','diversity','landscape','charm','laboratory','lay'),
(181,'hot','gravity','kill','people','opportunity','solution','architecture','funny','lifestyle','either','six','surprisingly','winter','custody','absence','lover','interfere','remaining'),
(183,'commitment','tradition','port','parade','chest','attack','have','present','string','style','factory','parental','outdoor','opt','easily','flesh','course','wrap'),
(186,'extended','roof','genetic','vegetable','display','democratic','following','logic','ideological','ms','two','sale','reportedly','button','horrible','emotional','heritage','absolute'),
(187,'possibly','month','situation','data','metaphor','fishing','equivalent','install','notion','drift','actor','short','investigation','cord','strategy','sprinkle','sad','decade'),
(188,'possibly','slight','nonetheless','following','underlying','political','product','assemble','pure','conduct','huh','refugee','catch','admit','outstanding','disappointment','practice','teaching'),
(190,'policeman','orange','liver','library','condemn','limit','walking','housing','ear','lower','between','rational','resemble','answer','show','license','e-mail','sensitivity'),
(193,'coach','necessary','shark','round','ratio','will','growing','click','heart','consideration','juror','await','weird','individual','living','beam','breakfast','explosion'),
(194,'illustrate','sky','concerning','intelligence','mad','contractor','regulator','simultaneously','corporate','accurate','earth','designer','store','instance','burn','driving','loyalty','length'),
(195,'half','loan','impressive','whoever','boot','so','moreover','stadium','genetic','finally','rough','designer','assert','often','russian','cope','producer','male'),
(197,'frequently','freedom','pilot','team','finger','latin','pitch','equally','family','experience','rhetoric','sign','reporting','surveillance','excited','cooperation','prosecutor','net'),
(198,'organize','utility','ever','catch','sensitive','offense','emphasis','slice','difficult','migration','sneak','creature','slide','goat','airport','league','convenience','overlook'),
(203,'reminder','stiff','quote','opportunity','diagnosis','provoke','attraction','bee','inside','huge','brand','write','propose','scientific','at','colony','life','straighten'),
(206,'alien','patience','flight','plain','crucial','pen','holy','era','roof','anxious','tale','gun','tree','kneel','sing','sentence','poetry','seriously'),
(208,'food','examination','dynamic','ideal','fifteen','exception','academic','custom','hostage','buy','repeatedly','concede','so','metaphor','bankruptcy','neither','participant','additional'),
(210,'admire','fork','hug','palace','elect','ethnic','super','circle','shuttle','battle','donor','enter','lately','efficient','pool','clothes','wherever','nearly'),
(211,'abortion','parent','message','please','latter','urban','receive','pine','coal','cotton','cabin','tie','hockey','wedding','associate','myth','crucial','crash'),
(212,'county','impulse','personality','cease','explain','proposal','accuse','popularity','experimental','actress','each','sleep','purchase','image','loyal','nurse','halfway','consequence'),
(214,'update','theology','skull','issue','meantime','pioneer','file','grandfather','willing','flight','attorney','ensure','stuff','training','simply','wind','charity','encounter'),
(215,'banking','up','personal','protein','absence','repeatedly','incident','knee','pool','ease','spoon','egg','too','ice','running','therapist','limited','anybody'),
(217,'minimum','dance','white','solution','false','loud','maintain','than','bounce','convey','attend','balloon','loan','anymore','far','wrist','rain','fellow'),
(218,'mineral','over','otherwise','prosecutor','training','championship','passing','advanced','adequate','dangerous','interpretation','divine','input','assure','juice','ban','description','coal'),
(220,'province','facilitate','future','characteristic','shoe','happiness','be','those','devastating','embarrassed','garage','rely','brief','inherit','predict','meantime','pull','hence'),
(223,'deadline','wander','fraud','pleased','seek','how','manner','bicycle','partial','he','timber','respond','scratch','presence','iraqi','destruction','hide','admission'),
(226,'fact','stroke','herself','supermarket','works','frustration','inside','discipline','equivalent','spine','clay','really','automatically','opt','wooden','nose','song','die'),
(229,'interpret','contain','beer','twelve','little','prevail','grandchild','horizon','bat','angle','european','magazine','natural','advantage','plus','ancestor','remark','foreigner'),
(230,'there','religious','output','amid','faith','contemporary','african-american','merchant','ankle','estimated','rural','condemn','insect','discuss','photo','target','training','uh'),
(231,'recipe','red','theological','plot','reflection','latter','musician','express','hug','headquarters','citizen','ring','inner','eligible','integration','born','artist','assure'),
(233,'depending','travel','dot','bid','regret','production','headline','straighten','psychological','southern','behind','separate','heavy','violent','fast','policy','float','totally'),
(235,'commitment','widely','ongoing','cell','component','neutral','harsh','college','four','immediately','prove','approach','accent','ballot','patent','this','inspector','strongly'),
(242,'history','explosion','managing','lady','bone','bit','restaurant','southeast','merely','accuse','dip','molecule','skirt','tropical','chapter','mill','entertainment','confidence'),
(244,'recover','significant','permission','besides','screw','conservative','lifestyle','satisfy','monkey','sigh','disorder','barn','fitness','prospect','conviction','counter','assessment','knee'),
(248,'imagine','butt','bush','skirt','headache','manufacturing','inmate','appreciation','architecture','athletic','booth','sheep','controversy','consent','core','decision','counterpart','sink'),
(250,'angry','pork','temporary','offensive','fluid','mostly','girlfriend','attitude','beauty','compel','norm','since','internet','father','rate','boy','court','catalog'),
(251,'terrible','standard','density','carrier','horizon','proof','implement','cliff','depression','platform','commission','boil','fighter','quarter','living','consultant','realm','promise'),
(252,'consciousness','mm-hmm','thereby','hug','architecture','well','own','stroke','exhaust','killer','haul','clay','prove','trip','left','policeman','absence','culture'),
(253,'victory','diary','brown','curriculum','study','long','publicly','slot','weakness','association','credit','communication','swim','proud','abroad','asset','reach','sympathy'),
(254,'by','exchange','anxiety','energy','marry','unit','elbow','atmosphere','slap','cue','leap','aide','gaze','championship','occur','knock','mysterious','puzzle'),
(257,'category','resist','lack','drum','uncertain','drift','law','travel','hypothesis','anticipate','mark','restaurant','hospital','earth','tale','mm-hmm','previous','turn'),
(258,'jew','towards','remaining','degree','scientific','intact','lemon','relative','cat','certainly','creativity','burn','improve','unless','impact','recording','specialty','experienced'),
(263,'iron','engage','achievement','sex','descend','cloud','generally','cue','hat','earn','i','apologize','faint','frown','common','behavioral','massive','bare'),
(266,'spine','senior','influential','variety','of','more','leg','distribution','seize','rape','telephone','adolescent','inmate','examination','mill','restaurant','double','warehouse'),
(267,'cute','internal','jazz','continued','asleep','we','fit','horizon','external','cow','hispanic','fisherman','whale','deep','debut','diversity','natural','project'),
(269,'seize','opportunity','regulator','mentor','united','river','hope','managing','operator','native','tunnel','advanced','faith','evening','claim','cut','thirty','angel'),
(271,'informal','time','serving','allow','goat','differ','apparent','virtue','barrel','bishop','improve','lean','super','unable','transaction','toy','theory','coastal'),
(273,'nature','flood','council','bolt','almost','venture','weekend','refrigerator','politician','coal','self-esteem','fun','blanket','path','implication','accept','commander','neighboring'),
(274,'grandfather','shelf','pray','departure','resist','interpret','canadian','piece','respectively','balance','spread','couple','camera','threaten','provoke','excessive','permit','fever'),
(279,'option','ticket','machine','nutrient','bubble','lie','embarrassed','ideal','weigh','alcohol','asian','notice','music','angle','recruit','cattle','exhibit','appoint'),
(280,'boundary','sexy','victim','corner','resolution','unique','heavily','occupation','greatly','encouraging','opinion','stomach','go','screen','flee','rid','guarantee','lightning'),
(282,'functional','relevant','timing','remaining','response','preach','always','circumstance','builder','cable','body','contact','attractive','device','replace','romantic','concert','target'),
(284,'high-tech','teach','logical','paint','help','trick','disappointed','realistic','water','complaint','committee','relieve','retirement','sympathy','corner','serious','joke','elegant'),
(285,'meat','swimming','regional','love','profound','essential','publicity','association','athlete','fade','bake','decorate','constitutional','composition','price','rain','metaphor','flexible'),
(290,'kid','specifically','conduct','killing','anniversary','wow','diabetes','nest','catch','jacket','airline','formal','top','pant','isolation','already','chocolate','industrial'),
(292,'equip','walk','encounter','comparable','shared','heat','far','waste','attention','pitch','panic','assist','layer','cop','epidemic','missile','removal','culture'),
(295,'scientist','tv','clue','use','it','highway','improvement','unknown','producer','arrangement','dish','handle','standing','learn','acceptable','shopping','risky','discourage'),
(302,'by','girl','characterize','fee','hint','fee','policy','equally','dare','better','drive','skill','billion','toll','expense','hopefully','incredible','disappear'),
(306,'lower','fail','settlement','allow','gear','oppose','carbon','dam','straw','both','realistic','encouraging','recommendation','fiscal','realize','beef','increased','arise'),
(310,'maintain','soil','soon','painful','supporter','correlation','musical','chin','current','die','call','stack','write','scientific','depict','route','exciting','mention'),
(311,'century','motion','legitimate','realize','super','interaction','computer','strong','call','correctly','sprinkle','piano','mortgage','send','dozen','forth','initiate','strictly'),
(312,'pole','drum','russian','sit','type','restriction','assignment','presence','aspect','attract','sack','obtain','disorder','hockey','allow','stiff','standing','campaign'),
(313,'practice','telephone','permanent','price','system','improved','stroke','traffic','math','copy','avoid','dense','pitch','contract','flee','project','influential','onto'),
(317,'taste','note','annual','announcement','forgive','hunt','explosion','impose','opposition','auction','if','sigh','expose','turkey','psychologist','seed','box','authority'),
(320,'activity','vaccine','sister','skip','everybody','revolutionary','shadow','assessment','fascinating','collector','attack','bicycle','hay','companion','section','distribute','trust','original'),
(323,'irish','plunge','she','scale','vaccine','thus','thus','speaker','universe','analyst','offense','concept','possibly','missing','vehicle','training','which','fight'),
(324,'bacteria','scheme','endure','answer','season','necessary','sing','racial','hour','model','arm','butt','secular','hour','young','sector','pit','forget'),
(331,'start','which','intelligent','knock','costume','fast','ambitious','withdraw','provide','spot','why','valuable','hunger','when','the','month','towards','totally'),
(334,'unknown','unit','honest','thinking','advocate','middle','address','symptom','every','another','latter','personal','hotel','freedom','steer','performer','introduce','pant'),
(335,'grocery','reliable','illness','college','grief','tonight','brain','six','routine','deficit','random','security','portfolio','romance','sometimes','closest','eye','various'),
(336,'praise','twenty','recruit','sure','actively','stress','read','sweater','one','gain','automatic','head','mrs','closer','spectacular','ready','tolerate','auction'),
(337,'peer','secretary','camp','aisle','consensus','silent','matter','energy','past','drama','completely','blink','quietly','soil','alter','way','drawing','hotel'),
(339,'perfect','sake','known','forbid','nowhere','real','reward','reserve','text','profound','activist','beyond','proposal','basement','contemporary','division','plain','bold'),
(342,'growing','foot','development','oh','relative','distant','cooperative','maximum','weakness','drawing','final','side','anniversary','dark','soar','screw','unknown','join'),
(343,'heal','training','load','choice','five','crucial','costume','envision','fiction','briefly','voter','ritual','patent','error','e-mail','square','hence','jewelry'),
(344,'citizenship','stop','to','carbon','relieve','threshold','authority','basketball','anonymous','response','become','gift','good','win','edge','needle','productivity','theology'),
(345,'upstairs','press','journalism','four','works','import','hallway','perform','effect','hot','confrontation','what','football','planet','warrior','severe','teaching','modify'),
(346,'concrete','youth','speak','direct','operator','suggestion','ranch','mate','suit','intense','punishment','if','interval','own','bell','satellite','team','truck'),
(354,'blend','side','fully','lower','colleague','effectively','properly','police','bath','emphasize','ms','driver','out','walk','ingredient','code','spend','protein'),
(355,'psychologist','permanent','increasing','spanish','follow','proof','driver','hip','husband','excellent','aside','director','negotiate','fuel','ought','goat','diabetes','scared'),
(359,'opposite','conspiracy','stick','multiple','drawing','care','nutrient','airplane','lately','absence','added','fiction','accurate','differ','organized','ceo','legend','rice'),
(360,'angel','puzzle','vacuum','skip','always','departure','balloon','beautiful','hundred','breast','feeling','likely','lifetime','crack','aids','chapter','perspective','civilian'),
(362,'obstacle','tolerance','scream','traveler','leave','unknown','performer','obvious','cover','leather','vote','prospect','trunk','core','preserve','building','horizon','isolation'),
(363,'armed','wheel','scientist','clothes','deposit','ash','wander','proposed','reinforce','professor','acceptable','separate','momentum','ego','emission','several','today','anxious'),
(364,'speak','forest','tolerance','membership','insist','sword','anyway','thin','risky','coat','at','basket','pepper','best','describe','pregnancy','faint','rating'),
(366,'onto','outcome','envelope','carrot','natural','grief','childhood','probably','read','exhaust','unlike','suddenly','ownership','man','array','legitimate','smile','fast'),
(369,'house','vessel','potato','franchise','ban','snap','admission','everything','streak','acid','their','citizen','solution','greek','assessment','minute','encourage','hotel'),
(370,'childhood','party','can','someone','somehow','investigate','assemble','blink','our','and','iraqi','drum','trunk','loose','harassment','drown','encounter','fool'),
(372,'offensive','routinely','oh','trace','dissolve','through','hip','alone','crime','resolve','spit','wheat','sometime','dried','horn','boost','motor','beginning'),
(373,'wheelchair','device','trash','assault','huh','priority','gas','appreciation','funding','border','molecule','creature','lately','satisfaction','man','lap','judgment','guard'),
(374,'exciting','exploit','yet','certain','i','smoke','least','reliability','inflation','closed','legally','limitation','thanksgiving','establishment','experimental','deer','each','lip'),
(376,'require','plot','point','library','reference','swallow','catalog','strong','glad','folk','size','ok','doubt','agree','update','yet','spouse','diet'),
(378,'radical','retirement','depression','land','educator','developmental','people','music','policy','electricity','miracle','commissioner','participant','layer','competitor','when','van','phone'),
(380,'leave','medicine','nominee','shine','personality','surface','ease','line','proper','empire','major','artistic','uncomfortable','branch','rather','carve','emphasize','sheer'),
(387,'port','terror','song','passenger','everyday','integrated','hesitate','reserve','scream','conservative','electronics','include','dominant','credibility','diverse','shuttle','works','retreat'),
(388,'rebuild','cup','opponent','closet','constitute','public','soviet','trunk','transmit','changing','sing','european','neutral','acceptable','greatly','idea','widely','flow'),
(389,'together','unique','angry','characterize','black','nod','win','texture','continuous','proper','add','due','bonus','withdrawal','final','sector','excited','loyal'),
(391,'sequence','popularity','insert','example','initiate','practitioner','responsible','shirt','productive','commit','choice','cow','mix','numerous','industry','compliance','profession','title'),
(394,'tree','nuclear','sheet','anyway','ball','accommodate','recall','badly','prohibit','exposure','infection','jar','rescue','horse','divine','fur','instantly','stomach'),
(400,'racism','rack','law','girl','analysis','decrease','cave','passing','speaker','interested','sign','bike','sexuality','bee','franchise','map','mentally','teacher'),
(402,'violence','transaction','borrow','sky','proper','lead','journalism','franchise','external','boss','chain','chin','overwhelming','popular','offender','movie','dump','lucky'),
(403,'kingdom','refrigerator','motivate','diet','blond','golden','today','expectation','specifically','crash','alter','wildlife','activity','crash','negotiate','disappear','symbolic','type'),
(407,'tolerate','marriage','elaborate','due','hay','lifetime','reduction','then','correspondent','incredibly','within','increasingly','perform','six','hearing','etc','investor','anonymous'),
(408,'eye','vulnerable','garage','inside','galaxy','parent','explore','unless','classroom','contribute','coverage','weed','outdoor','captain','nation','these','occupation','automatically'),
(409,'cruel','harvest','elderly','moreover','temperature','remind','nowhere','nightmare','reduction','certainly','ideological','predator','literary','concept','catalog','annual','preserve','gun'),
(411,'haul','fatal','conflict','folk','primary','retirement','undergo','publicity','objection','blame','hero','golden','stove','talented','rolling','ago','skull','enjoy'),
(412,'without','render','await','mr','yes','require','deadly','edit','religion','fix','realistic','earthquake','blade','victory','latter','import','photograph','inherit'),
(414,'tighten','now','actor','objection','contend','primarily','pan','courtroom','partly','apple','consensus','phenomenon','progressive','deliberately','mathematics','kill','spray','feather'),
(415,'organic','schedule','new','furthermore','past','analysis','practically','friendly','behavior','arrangement','count','firmly','pitch','palm','knife','predator','five','key'),
(416,'date','tropical','likewise','ride','dispute','importance','mushroom','curious','nest','declare','accident','consist','peace','prescription','tomato','persist','living','gently'),
(418,'producer','escape','psychology','likewise','see','auto','normally','script','soft','bedroom','next','depressed','black','entrance','faculty','search','multiple','leader'),
(420,'priest','guidance','medium','disappointment','unit','banker','civil','guest','bullet','fifty','tie','huh','essentially','rod','risky','order','intense','parking'),
(421,'explore','code','discussion','organized','rip','photograph','van','sphere','prepare','circuit','complaint','instructor','reveal','cart','reliability','junior','especially','uh'),
(422,'regularly','recipient','identify','manufacturing','blend','mill','courtroom','in','history','database','criticize','fiscal','rib','desk','contract','leg','meter','device'),
(424,'insight','discrimination','seminar','judgment','accommodate','curve','gold','spell','guilt','conflict','supermarket','carefully','convince','passage','mere','resist','municipal','halfway'),
(429,'formation','so-called','panic','breathe','piece','retreat','clinic','ear','map','disagree','face','rear','dramatic','married','indicator','male','lifestyle','hit'),
(430,'cooking','service','interview','angel','fruit','accessible','sustainable','arrangement','minimize','friendly','makeup','rat','chamber','planner','cloth','institutional','fate','hate'),
(432,'genetic','receiver','accurately','inevitable','conventional','ignore','pressure','brutal','conspiracy','lead','produce','order','and/or','justice','steep','would','technological','low'),
(436,'ear','strongly','letter','privately','bear','conflict','personality','courtroom','sole','denial','harsh','yesterday','twist','liver','constant','chest','required','than'),
(437,'harassment','trigger','format','quick','club','inspection','works','hit','cling','fishing','boast','satisfaction','coastal','tube','approximately','lion','found','tomorrow'),
(439,'running','tighten','campus','chill','revelation','articulate','hallway','indian','number','page','bias','input','crowded','peace','inherit','sufficient','crazy','progress'),
(440,'deer','transformation','interior','instructor','identity','on','jazz','grade','opposed','legislator','serving','our','alcohol','director','butt','fee','suitable','resolution'),
(442,'temple','telescope','household','both','corner','soap','unfortunately','temple','battery','alternative','voter','speak','dismiss','low','mystery','channel','cast','detect'),
(447,'absence','nation','human','knife','thirty','fiscal','clock','crowded','loss','frustration','plane','gallery','really','branch','commission','salt','helpful','rock'),
(449,'damn','vacuum','chair','surgeon','musician','lately','rely','emerging','partnership','stove','medium','key','adoption','hunter','awful','always','use','adult'),
(450,'crack','retire','regional','popular','gathering','skull','density','plea','one','name','costume','consist','planner','teen','carry','bottle','ticket','lesson'),
(454,'enough','shower','sad','costly','around','poke','sister','transmit','supporter','rural','introduce','grape','nightmare','us','affair','post','road','greatly'),
(457,'drink','diabetes','element','leather','teaspoon','confess','crucial','russian','commission','buddy','tent','depend','success','traditionally','alone','associate','equally','minimum'),
(458,'panel','tooth','surgery','various','program','lion','clearly','ridiculous','understand','impression','overwhelm','helmet','meantime','promising','skill','questionnaire','onion','puzzle'),
(459,'assemble','extend','responsibility','unfair','rice','missile','damn','childhood','ideological','booth','dam','shortly','await','gear','simultaneously','happiness','cat','feedback'),
(462,'extremely','necessary','grave','spark','morning','appointment','bush','regular','explicit','hopefully','equity','atmosphere','shoe','alone','publisher','duck','reliability','conservation'),
(464,'literally','trial','boast','function','shirt','expedition','club','little','competition','bomb','overwhelming','bullet','encouraging','theoretical','accelerate','famous','increase','rush'),
(466,'day','stair','limitation','with','port','observe','until','snap','corner','jaw','bedroom','documentary','nonprofit','afraid','slavery','year','rely','aisle'),
(470,'opera','substantial','wildlife','branch','at','headquarters','rate','shop','produce','heat','degree','exceed','fatal','charm','sensitive','astronomer','initiative','defensive'),
(471,'scare','minimum','gain','hand','belt','variety','pill','road','install','finding','shore','brave','dna','battle','law','jail','comparable','cabinet'),
(472,'eyebrow','previously','concert','telescope','desperate','pig','client','action','practitioner','incredibly','shooting','chemical','story','year','likely','disease','slam','hit'),
(474,'suspect','southwest','strike','consequently','peer','emotional','beside','alter','scope','density','guide','known','slight','creation','longtime','backyard','motive','hip'),
(476,'commodity','full-time','slight','candle','commander','freshman','fog','chance','structure','addition','province','translation','administrator','radar','judicial','very','boss','born'),
(477,'topic','neck','machine','deserve','whisper','recession','horse','asian','acquisition','changing','pass','council','mrs','eye','lap','inflation','understand','advance'),
(479,'angle','should','final','one-third','beginning','struggle','painful','exchange','clothing','leaf','product','depressed','launch','broad','define','show','pay','dig'),
(480,'allegedly','religious','legacy','awareness','pressure','dna','civilization','artistic','equality','dock','throat','rim','occasion','additional','treat','public','robot','associate'),
(481,'distribution','sustainable','today','demographic','hit','single','behavior','responsible','stage','outcome','rescue','firm','supposed','loop','poll','innocent','oxygen','confuse'),
(484,'establishment','moreover','lower','overcome','compliance','accelerate','fork','raise','convey','irony','explain','potentially','comprise','supporter','bread','the','grant','practice'),
(485,'civilization','milk','size','brake','prize','ministry','key','press','category','legitimate','automatic','remote','subject','conventional','labor','integration','expectation','vs'),
(487,'arrive','win','exhibit','wealth','yourself','sink','wheat','texture','marker','midst','gasoline','friendly','chicken','physician','born','headline','first','approve'),
(488,'season','voice','identity','democrat','gay','article','selected','legend','strain','fisherman','debate','host','start','understand','jump','demonstrate','access','edition'),
(489,'uh','jet','square','would','stare','control','professor','pink','guide','daily','educational','pipe','solar','ingredient','workout','duck','town','bicycle'),
(490,'consist','still','motion','replace','produce','listen','advice','uniform','seventh','cocaine','articulate','presence','supply','absolute','greatest','suggest','segment','okay'),
(495,'proclaim','statue','mountain','repair','aim','style','debate','project','stare','spare','sustainable','merit','arab','throughout','row','precious','twentieth','entitle'),
(499,'bell','resident','suffer','sock','persist','punishment','attention','backyard','historical','aide','hockey','cool','ecological','setting','short-term','multiple','copy','necessity');
 
ALTER TABLE t ADD FULLTEXT INDEX idx(c04);
 
UPDATE t SET c10 = 'foo';
UPDATE t SET c10 = 'bar';
 
--source include/restart_mysqld.inc
 
UPDATE t SET c09 = 'qux';
ALTER TABLE t FORCE;
 
# Cleanup
DROP TABLE t;

10.6 0fbcb0a2 non-debug

2022-08-28 18:29:49 3 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `test`.`t`
2022-08-28 18:29:49 0x7fa750ff3700  InnoDB: Assertion failure in file /data/src/10.6/storage/innobase/que/que0que.cc line 728
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
 
#5  0x00007fa757c6e537 in __GI_abort () at abort.c:79
#6  0x000055cd386e0b18 in ut_dbg_assertion_failed (expr=expr@entry=0x55cd39323fc0 "trx->error_state == DB_SUCCESS", file=file@entry=0x55cd39323f90 "/data/src/10.6/storage/innobase/que/que0que.cc", line=line@entry=728) at /data/src/10.6/storage/innobase/ut/ut0dbg.cc:60
#7  0x000055cd386cada0 in que_run_threads (thr=<optimized out>) at /data/src/10.6/storage/innobase/que/que0que.cc:728
#8  que_run_threads (thr=<optimized out>) at /data/src/10.6/storage/innobase/que/que0que.cc:722
#9  0x000055cd38f213e9 in fts_eval_sql (trx=trx@entry=0x7fa7522a7180, graph=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0sql.cc:163
#10 0x000055cd38f0e7df in fts_write_node (trx=0x7fa7522a7180, graph=0x7fa72406b488, fts_table=0x7fa750ff0330, word=<optimized out>, node=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3763
#11 0x000055cd38f13e3c in fts_sync_write_words (index_cache=0x7fa724079980, trx=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3877
#12 fts_sync_index (index_cache=0x7fa724079980, sync=0x7fa72406b330) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3959
#13 fts_sync (sync=0x7fa72406b330) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4105
#14 0x000055cd38f1587d in fts_sync_table (table=<optimized out>, table=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4150
#15 fts_sync_table (table=0x7fa72406a600) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4145
#16 fts_add_doc_by_id (ftt=<optimized out>, doc_id=779) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3468
#17 0x000055cd38f15aa5 in fts_add (ftt=<optimized out>, row=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2748
#18 0x000055cd38f15c76 in fts_modify (row=<optimized out>, ftt=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2862
#19 fts_commit_table (ftt=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2914
#20 fts_commit (trx=trx@entry=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2959
#21 0x000055cd3873dcbe in trx_t::commit_low (this=0x7fa7522a5b80, mtr=0x7fa750ff10a0) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1422
#22 0x000055cd38e6c6f5 in trx_t::commit_persist (this=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1471
#23 0x000055cd38e6c8e2 in trx_t::commit (this=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1480
#24 trx_commit_for_mysql (trx=trx@entry=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1597
#25 0x000055cd38d61bf0 in innobase_commit_low (trx=<optimized out>) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4416
#26 innobase_commit_ordered_2 (thd=0x7fa71c000c58, trx=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4522
#27 innobase_commit (hton=<optimized out>, thd=0x7fa71c000c58, commit_trx=<optimized out>) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4626
#28 0x000055cd38a58539 in commit_one_phase_2 (thd=thd@entry=0x7fa71c000c58, all=<optimized out>, trans=0x7fa71c004410, is_real_trans=<optimized out>) at /data/src/10.6/sql/handler.cc:2070
#29 0x000055cd38a58784 in ha_commit_one_phase (thd=thd@entry=0x7fa71c000c58, all=all@entry=false) at /data/src/10.6/sql/handler.cc:2023
#30 0x000055cd38a655de in ha_commit_trans (thd=thd@entry=0x7fa71c000c58, all=all@entry=false) at /data/src/10.6/sql/handler.cc:1817
#31 0x000055cd389494b3 in trans_commit_stmt (thd=thd@entry=0x7fa71c000c58) at /data/src/10.6/sql/transaction.cc:472
#32 0x000055cd388252d9 in mysql_execute_command (thd=0x7fa71c000c58, is_called_from_prepared_stmt=<optimized out>) at /data/src/10.6/sql/sql_parse.cc:6057
#33 0x000055cd3882a197 in mysql_parse (thd=0x7fa71c000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/10.6/sql/sql_parse.cc:8030
#34 0x000055cd3882ca5d in dispatch_command (command=COM_QUERY, thd=0x7fa71c000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /data/src/10.6/sql/sql_class.h:234
#35 0x000055cd3882e712 in do_command (thd=0x7fa71c000c58, blocking=blocking@entry=true) at /data/src/10.6/sql/sql_parse.cc:1409
#36 0x000055cd389395ff in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /data/src/10.6/sql/sql_connect.cc:1418
#37 0x000055cd3893994d in handle_one_connection (arg=arg@entry=0x55cd3adb3e88) at /data/src/10.6/sql/sql_connect.cc:1312
#38 0x000055cd38cbb6f2 in pfs_spawn_thread (arg=0x55cd3adc8c68) at /data/src/10.6/storage/perfschema/pfs.cc:2201
#39 0x00007fa75812fea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#40 0x00007fa757d46def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

10.6 0fbcb0a2b87 debug

2022-08-28 18:38:44 3 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `test`.`t`
mariadbd: /data/src/10.6-bug/storage/innobase/que/que0que.cc:223: que_thr_t* que_fork_start_command(que_fork_t*): Assertion `thr->state == QUE_THR_COMPLETED' failed.
220828 18:38:44 [ERROR] mysqld got signal 6 ;
 
#7  0x00007f20911af662 in __GI___assert_fail (assertion=0x55c3a31f83f0 "thr->state == QUE_THR_COMPLETED", file=0x55c3a31f8320 "/data/src/10.6-bug/storage/innobase/que/que0que.cc", line=223, function=0x55c3a31f8458 "que_thr_t* que_fork_start_command(que_fork_t*)") at assert.c:101
#8  0x000055c3a2a40f4a in que_fork_start_command (fork=0x7f1fd8a11b40) at /data/src/10.6-bug/storage/innobase/que/que0que.cc:223
#9  0x000055c3a2ca819e in fts_eval_sql (trx=0x7f2087400180, graph=0x7f1fd8a11b40) at /data/src/10.6-bug/storage/innobase/fts/fts0sql.cc:161
#10 0x000055c3a2c8fa45 in fts_write_node (trx=0x7f2087400180, graph=0x7f204c06bf70, fts_table=0x7f208c13db30, word=0x7f1fd89b9718, node=0x7f1fd89b7df0) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3763
#11 0x000055c3a2c8ff1c in fts_sync_write_words (trx=0x7f2087400180, index_cache=0x7f204c082ea8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3877
#12 0x000055c3a2c90501 in fts_sync_index (sync=0x7f204c06bec8, index_cache=0x7f204c082ea8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3959
#13 0x000055c3a2c90cdd in fts_sync (sync=0x7f204c06bec8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:4105
#14 0x000055c3a2c90f14 in fts_sync_table (table=0x7f204c052338) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:4150
#15 0x000055c3a2c8f091 in fts_add_doc_by_id (ftt=0x7f1fd8960e48, doc_id=779) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3468
#16 0x000055c3a2c8d4cb in fts_add (ftt=0x7f1fd8960e48, row=0x7f1fd8970658) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2748
#17 0x000055c3a2c8dc04 in fts_commit_table (ftt=0x7f1fd8960e48) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2910
#18 0x000055c3a2c8dcf7 in fts_commit (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2959
#19 0x000055c3a2b4d8be in trx_t::commit_low (this=0x7f20873feb80, mtr=0x7f208c13ebb0) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1422
#20 0x000055c3a2b4ea71 in trx_t::commit_persist (this=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1471
#21 0x000055c3a2b4eb62 in trx_t::commit (this=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1480
#22 0x000055c3a2b4efc9 in trx_commit_for_mysql (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1597
#23 0x000055c3a28e1bb1 in innobase_commit_low (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4416
#24 0x000055c3a28e1ee3 in innobase_commit_ordered_2 (trx=0x7f20873feb80, thd=0x7f1fd8000db8) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4522
#25 0x000055c3a28e2439 in innobase_commit (hton=0x55c3a5f0bd78, thd=0x7f1fd8000db8, commit_trx=false) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4626
#26 0x000055c3a24d5b32 in commit_one_phase_2 (thd=0x7f1fd8000db8, all=false, trans=0x7f1fd8004720, is_real_trans=true) at /data/src/10.6-bug/sql/handler.cc:2070
#27 0x000055c3a24d58ef in ha_commit_one_phase (thd=0x7f1fd8000db8, all=false) at /data/src/10.6-bug/sql/handler.cc:2023
#28 0x000055c3a24d4a37 in ha_commit_trans (thd=0x7f1fd8000db8, all=false) at /data/src/10.6-bug/sql/handler.cc:1817
#29 0x000055c3a230ef00 in trans_commit_stmt (thd=0x7f1fd8000db8) at /data/src/10.6-bug/sql/transaction.cc:472
#30 0x000055c3a2147a83 in mysql_execute_command (thd=0x7f1fd8000db8, is_called_from_prepared_stmt=false) at /data/src/10.6-bug/sql/sql_parse.cc:6057
#31 0x000055c3a214d450 in mysql_parse (thd=0x7f1fd8000db8, rawbuf=0x7f1fd8017d40 "UPDATE t SET c09 = 'qux'", length=24, parser_state=0x7f208c1403c0) at /data/src/10.6-bug/sql/sql_parse.cc:8030
#32 0x000055c3a2139ab3 in dispatch_command (command=COM_QUERY, thd=0x7f1fd8000db8, packet=0x7f1fd800b879 "UPDATE t SET c09 = 'qux'", packet_length=24, blocking=true) at /data/src/10.6-bug/sql/sql_parse.cc:1896
#33 0x000055c3a21384ae in do_command (thd=0x7f1fd8000db8, blocking=true) at /data/src/10.6-bug/sql/sql_parse.cc:1409
#34 0x000055c3a22f2a7c in do_handle_one_connection (connect=0x55c3a64021d8, put_in_cache=true) at /data/src/10.6-bug/sql/sql_connect.cc:1418
#35 0x000055c3a22f271b in handle_one_connection (arg=0x55c3a64021d8) at /data/src/10.6-bug/sql/sql_connect.cc:1312
#36 0x000055c3a280f942 in pfs_spawn_thread (arg=0x55c3a60588c8) at /data/src/10.6-bug/storage/perfschema/pfs.cc:2201
#37 0x00007f209167bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#38 0x00007f2091278def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment by Ilkka Myller [ 2022-08-29 ]

I can confirm this bug with MariaDB 10.6.9 from Debian/Ubuntu packages:

  • 10.6.9+maria~deb11
  • 10.6.9+maria~ubu2004
  • 10.6.9+maria~ubu2204

It is not dependent on used table schema or amount of data in table, but occurs randomly when inserting rows to FTS enabled tables.

As a workaround we downgraded to 10.6.8 release. The same database directory worked after downgrade without further repairs.

Comment by Ilkka Myller [ 2022-08-29 ]

Daemon log for one of the affected 10.6.9 servers (some information has been redacted for security):

Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 2022-08-29 14:20:00 236 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `redacted`.`forsecurity`
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 2022-08-29 14:20:00 0x7fd0041f6640  InnoDB: Assertion failure in file ./storage/innobase/que/que0que.cc line 728
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: We intentionally generate a memory trap.
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: If you get repeated assertion failures or crashes, even
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: immediately after the mariadbd startup, there may be
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: corruption in the InnoDB tablespace. Please refer to
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: about forcing recovery.
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 220829 14:20:00 [ERROR] mysqld got signal 6 ;
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: This could be because you hit a bug. It is also possible that this binary
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: or one of the libraries it was linked against is corrupt, improperly built,
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: or misconfigured. This error can also be caused by malfunctioning hardware.
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: We will try our best to scrape up some info that will hopefully help
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: diagnose the problem, but since we have already crashed,
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: something is definitely wrong and this may fail.
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Server version: 10.6.9-MariaDB-1:10.6.9+maria~ubu2204
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: key_buffer_size=134217728
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: read_buffer_size=131072
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: max_used_connections=7
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: max_threads=153
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: thread_count=7
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: It is possible that mysqld could use up to
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K  bytes of memory
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Hope that's ok; if not, decrease some variables in the equation.
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Thread pointer: 0x7fcc58000c68
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Attempting backtrace. You can use the following information to find out
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: where mysqld died. If you see no messages after this, something went
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: terribly wrong...
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: stack_bottom = 0x7fd0041f5c78 thread_stack 0x49000
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(my_print_stacktrace)[0x5587a78a2332]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(handle_fatal_signal)[0x5587a7360ae8]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(__sigaction)[0x7fd00c027520]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(pthread_kill)[0x7fd00c07ba7c]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(raise)[0x7fd00c027476]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(abort)[0x7fd00c00d7f3]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x5587a6fdfb98]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x5587a6fc9a98]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(main)[0x5587a703f8e6]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5587a77606e9]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5587a77608e2]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5587a7655c30]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x5587a73643a9]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(ha_commit_trans(THD*, bool))[0x5587a73716a6]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(trans_commit_stmt(THD*))[0x5587a7251227]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(mysql_execute_command(THD*, bool))[0x5587a7129fcc]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5587a712ef57]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x5587a71316a5]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(do_command(THD*, bool))[0x5587a7133382]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(do_handle_one_connection(CONNECT*, bool))[0x5587a7240f6f]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(handle_one_connection)[0x5587a72412bd]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5587a75acdb6]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(pthread_condattr_setpshared)[0x7fd00c079b43]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(__xmknodat)[0x7fd00c10ba00]
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Trying to get some variables.
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Some pointers may be invalid and cause the dump to abort.
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Query (0x7fcc58012e40): insert into forsecurity <REDACTED FOR SECURITY>
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Connection ID (thread ID): 236
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Status: NOT_KILLED

Comment by Ilkka Myller [ 2022-08-30 ]

Any chance that 10.6 LTS minor version including this patch could be released at an expedited pace?

Since both FTS and MariaDB LTS is very popular for example in hosting industry (mainly WordPress, which commonly triggers this bug), this is hitting heavily on many hosting 10.6 LTS based services.

Also, this bug is affecting Docker containers using :10.6 release tag, which is expected to be relatively stable within minor release versions.

Comment by Marko Mäkelä [ 2022-08-30 ]

The fix looks OK to me. It consists of two parts:

  • fixing the so far only known underlying cause (MDEV-15237), in fts_cmp_set_sync_doc_id().
  • preventing crashes in case the duplicate key would occur
Comment by Arthur Borsboom [ 2022-09-05 ]

Any ETA on fixed versions?

Comment by Matthias Leich [ 2022-09-06 ]

The tree origin/bb-10.6-MDEV-29342 e8bc9d499ce4b6f6ec63f1a962f4af4cb9ef537f 2022-08-29T23:36:23+05:30
behaved well in RQG testing. Please be aware that up till now none of the RQG tests was ever capable to replay the problem.

Comment by Marko Mäkelä [ 2022-09-06 ]

The fix (to all or part of MDEV-15237) was pushed to 10.3. Some additional fixes were pushed to 10.6. The first version that could crash due to this bug was 10.6.9. We do not know yet if MDEV-15237 was fixed by this.

Comment by Marko Mäkelä [ 2022-09-06 ]

ArthurBorsboom, by default MariaDB Server releases occur 4 times per year. A fix for this crash is included in the Enterprise Server 10.6.9-5 release.

I have advocated for unscheduled releases, because a combination of this bug and MDEV-29374 can result in a permanently corrupted database, but it does not currently seem that one will be made.

Comment by Ilkka Myller [ 2022-09-06 ]

Thank you for your efforts to get an unscheduled release out.

I'd still suggest at least rolling back official MariaDB Docker images released with LTS tag mariadb:10.6 to version 10.6.8.

Those are spreading this db corruption issue like a wildfire without people making choice or even being aware what patch-version of 10.6 they run.

Comment by Arthur Borsboom [ 2022-09-07 ]

Arch Linux uses 10.9.2 at 7 september 2022.

I am able to reproduce the issue in a test environment, so I am able to test and confirm the fix.

Which version in the 10.9 release is be expected to have the fix?

Comment by Marko Mäkelä [ 2022-09-07 ]

ArthurBorsboom, the fix of the duplicate key error has been merged up to 10.9, but as part of doing that, I forgot to apply some additional changes to 10.6 (see my previous comment). Those additional changes (which should prevent a crash if something else than what we fixed is causing the duplicate key error) have not been merged to 10.9 yet.

Please try the latest 10.9 branch. If it crashes, then MDEV-15237 was not fully fixed.

Comment by Sergei Golubchik [ 2022-09-08 ]

Summary: if InnoDB table had a fulltext index and only one new row was inserted between the last sync (performed asynchronously by a dedicated thread) and server shutdown, the fulltext index wasn't properly updated and became out of sync with the data. Before 10.6.9 it was a silent error, in 10.6.9 an assertion crashed the server when detecting the inconsistency.

Comment by Arthur Borsboom [ 2022-09-08 ]

Is validation/fixing of the database / indices necessary?

If so, will the fixed version do this automatically or is manual intervention needed?

If manual work is needed, what are the steps to find and fix the inconsistencies?

Comment by Marko Mäkelä [ 2022-09-08 ]

ArthurBorsboom, this type of duplicate key error will lead to all FULLTEXT INDEX of the affected table to ‘freeze’, that is, no further updates to the indexes will be made, or the fulltext indexes will not reflect the latest changes to the table. That can be fixed by rebuilding the table (OPTIMIZE TABLE).

Comment by Arthur Borsboom [ 2022-09-27 ]

I am hereby confirming that upgrading mariadb from 10.9.2 > 10.9.3 solves the problem for me.

Generated at Thu Feb 08 10:07:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.