Details
-
Bug
-
Status: Closed (View Workflow)
-
Critical
-
Resolution: Fixed
-
10.6.9, 10.6, 10.7(EOL), 10.8(EOL), 10.9(EOL), 10.10(EOL)
-
None
-
Debian 10 amd64
Description
Mariadb crash after each below query:
INSERT INTO `b_user_index` (`USER_ID`, `NAME`, `SECOND_NAME`, `LAST_NAME`, `WORK_POSITION`, `UF_DEPARTMENT_NAME`, `SEARCH_USER_CONTENT`, `SEARCH_ADMIN_CONTENT`, `SEARCH_DEPARTMENT_CONTENT`) VALUES (7000, '', '', '', '', '', '7000', '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989 не выбрано 0', '') ON DUPLICATE KEY UPDATE `NAME` = '', `SECOND_NAME` = '', `LAST_NAME` = '', `WORK_POSITION` = '', `UF_DEPARTMENT_NAME` = '', `SEARCH_USER_CONTENT` = IF(SHA1(SEARCH_USER_CONTENT) = 'b7525fa5653c5b45f75d490825626e0880b394f4', SEARCH_USER_CONTENT, '7000'), `SEARCH_ADMIN_CONTENT` = IF(SHA1(SEARCH_ADMIN_CONTENT) = 'ac932ff7b355d03b541911c13e64c2c600fd009c', SEARCH_ADMIN_CONTENT, '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989 не выбрано 0'), `SEARCH_DEPARTMENT_CONTENT` = IF(SHA1(SEARCH_DEPARTMENT_CONTENT) = 'da39a3ee5e6b4b0d3255bfef95601890afd80709', SEARCH_DEPARTMENT_CONTENT, '')
|
The problem started after updating to 10.6.9.
error log:
2022-08-20 14:02:28 62242 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `z2s_db`.`b_user_index`
|
2022-08-20 14:02:28 0x7f0fec0d6700 InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.6.9/storage/innobase/que/que0que.cc line 728
|
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
220820 14:02:28 [ERROR] mysqld got signal 6 ;
|
This could be because you hit a bug. It is also possible that this binary
|
or one of the libraries it was linked against is corrupt, improperly built,
|
or misconfigured. This error can also be caused by malfunctioning hardware.
|
|
|
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
|
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
|
Server version: 10.6.9-MariaDB-1:10.6.9+maria~deb10-log
|
key_buffer_size=8589934592
|
read_buffer_size=131072
|
max_used_connections=50
|
max_threads=2502
|
thread_count=50
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 19022199 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation.
|
|
|
Thread pointer: 0x7f0960000c18
|
Attempting backtrace. You can use the following information to find out
|
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7f0fec0d5d98 thread_stack 0x49000
|
??:0(my_print_stacktrace)[0x559ba9edfb0e]
|
??:0(handle_fatal_signal)[0x559ba99c5555]
|
??:0(__restore_rt)[0x7f120640d730]
|
??:0(gsignal)[0x7f1205f647bb]
|
??:0(abort)[0x7f1205f4f535]
|
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559ba966b3d2]
|
??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x559ba96571ca]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e4ebb9]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e3c3ff]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e40ba1]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e426de]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e42945]
|
??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&))[0x559ba9e42ad5]
|
??:0(main)[0x559ba96cd5e8]
|
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559ba9d9565b]
|
??:0(std::_Rb_tree<unsigned int, unsigned int, std::_Identity<unsigned int>, std::less<unsigned int>, std::allocator<unsigned int> >::_M_erase(std::_Rb_tree_node<unsigned int>*))[0x559ba9d9584f]
|
??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x559ba9c82958]
|
??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x559ba99c93a3]
|
??:0(ha_commit_trans(THD*, bool))[0x559ba99ca3b5]
|
??:0(trans_commit_stmt(THD*))[0x559ba98b5483]
|
??:0(mysql_execute_command(THD*, bool))[0x559ba97a985f]
|
??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x559ba979bf32]
|
??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x559ba97a6b3a]
|
??:0(do_command(THD*, bool))[0x559ba97a81f7]
|
??:0(do_handle_one_connection(CONNECT*, bool))[0x559ba98a5d17]
|
??:0(handle_one_connection)[0x559ba98a605d]
|
??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x559ba9be1b0e]
|
??:0(start_thread)[0x7f1206402fa3]
|
??:0(clone)[0x7f1206025eff]
|
|
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7f0960010b50): INSERT INTO `b_user_index` (`USER_ID`, `NAME`, `SECOND_NAME`, `LAST_NAME`, `WORK_POSITION`, `UF_DEPARTMENT_NAME`, `SEARCH_USER_CONTENT`, `SEARCH_ADMIN_CONTENT`, `SEARCH_DEPARTMENT_CONTENT`) VALUES (7000, '', '', '', '', '', '7000', '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989 не выбрано 0', '') ON DUPLICATE KEY UPDATE `NAME` = '', `SECOND_NAME` = '', `LAST_NAME` = '', `WORK_POSITION` = '', `UF_DEPARTMENT_NAME` = '', `SEARCH_USER_CONTENT` = IF(SHA1(SEARCH_USER_CONTENT) = 'b7525fa5653c5b45f75d490825626e0880b394f4', SEARCH_USER_CONTENT, '7000'), `SEARCH_ADMIN_CONTENT` = IF(SHA1(SEARCH_ADMIN_CONTENT) = 'ac932ff7b355d03b541911c13e64c2c600fd009c', SEARCH_ADMIN_CONTENT, '7000 +7 (962) 981-7989 +7(962)981-7989 +79629817989 89629817989 79629817989 9629817989 629817989 29817989 9817989 817989 17989 7989 989 не выбрано 0'), `SEARCH_DEPARTMENT_CONTENT` = IF(SHA1(SEARCH_DEPARTMENT_CONTENT) = 'da39a3ee5e6b4b0d3255bfef95601890afd80709', SEARCH_DEPARTMENT_CONTENT, '')
|
|
|
Connection ID (thread ID): 62242
|
Status: NOT_KILLED
|
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
|
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
|
information that should help you find out what is causing the crash.
|
Writing a core file...
|
Working directory at /var/lib/mysql
|
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds
|
Max file size unlimited unlimited bytes
|
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes
|
Max resident set unlimited unlimited bytes
|
Max processes 255271 255271 processes
|
Max open files 1048576 1048576 files
|
Max locked memory 65536 65536 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks
|
Max pending signals 255271 255271 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0
|
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
|
Kernel version: Linux version 4.19.0-20-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.235-1 (2022-03-17)
|
|
|
2022-08-20 14:02:35 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2022-08-20 14:02:35 0 [Note] InnoDB: Number of pools: 1
|
2022-08-20 14:02:35 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2022-08-20 14:02:35 0 [Note] InnoDB: Using Linux native AIO
|
2022-08-20 14:02:35 0 [Note] InnoDB: Initializing buffer pool, total size = 17179869184, chunk size = 134217728
|
2022-08-20 14:02:35 0 [Note] InnoDB: Completed initialization of buffer pool
|
2022-08-20 14:02:35 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=744061940175,744061940175
|
2022-08-20 14:02:42 0 [Note] InnoDB: 2 transaction(s) which must be rolled back or cleaned up in total 104 row operations to undo
|
2022-08-20 14:02:42 0 [Note] InnoDB: Trx id counter is 2290494205
|
2022-08-20 14:02:42 0 [Note] InnoDB: Starting final batch to recover 63594 pages from redo log.
|
2022-08-20 14:02:44 0 [Note] InnoDB: 128 rollback segments are active.
|
2022-08-20 14:02:44 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
|
2022-08-20 14:02:44 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
|
2022-08-20 14:02:44 0 [Note] InnoDB: Creating shared tablespace for temporary tables
|
2022-08-20 14:02:44 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
|
2022-08-20 14:02:44 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
|
2022-08-20 14:02:44 0 [Note] InnoDB: Rolled back recovered transaction 2290494166
|
2022-08-20 14:02:44 0 [Note] InnoDB: 10.6.9 started; log sequence number 744761503269; transaction id 2290494207
|
2022-08-20 14:02:44 0 [Note] InnoDB: Rolled back recovered transaction 2290494173
|
2022-08-20 14:02:44 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2022-08-20 14:02:44 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
|
2022-08-20 14:02:44 0 [Note] Plugin 'FEEDBACK' is disabled.
|
2022-08-20 14:02:44 0 [Warning] You need to use --log-bin to make --expire-logs-days or --binlog-expire-logs-seconds work.
|
2022-08-20 14:02:44 0 [Note] Server socket created on IP: '127.0.0.1'.
|
2022-08-20 14:02:44 0 [Note] /usr/sbin/mariadbd: ready for connections.
|
Version: '10.6.9-MariaDB-1:10.6.9+maria~deb10-log' socket: '/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
|
2022-08-20 14:02:46 0 [Note] InnoDB: Buffer pool(s) load completed at 220820 14:02:46
|
Attachments
Issue Links
- duplicates
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
Activity
zEvilz, can you please provide the output of
SHOW CREATE TABLE b_user_index; |
Also, please install the package mariadb-server-10.6-dbgsym and try to produce a full stack trace of the crash.
Previously, If there is a duplicate key happened due to FTS_DOC_ID(MDEV-15237) then only error message will happen in log file and it doesn't crash. But MDEV-25581 removes the background sync operation and allows user thread to do sync operation during transaction commit. It leads to crash. It is not a good idea to upgrade to 10.6.9 by those who're affected by MDEV-15237. Internally, we fail to repeat the MDEV-15237 problem, it would be great to get a test case from any affected user.
Yesterday, 21 August 2022, I have updated a Linux server including MariaDB which was updated from version 10.8.3 > 10.8.4
Since then, MariaDB crashes often with the same error message as the reporter.
[arthur@web1 ~]$ sudo coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE
Mon 2022-08-22 09:30:45 CEST 577 89 89 SIGABRT present /usr/bin/mariadbd 8.7M
Mon 2022-08-22 09:45:22 CEST 8143 89 89 SIGABRT present /usr/bin/mariadbd 4.7M
Mon 2022-08-22 09:50:42 CEST 8307 89 89 SIGABRT present /usr/bin/mariadbd 3.7M
Mon 2022-08-22 09:52:31 CEST 8430 89 89 SIGABRT present /usr/bin/mariadbd 2.2M
Mon 2022-08-22 09:53:45 CEST 8496 89 89 SIGABRT present /usr/bin/mariadbd 2.1M
Mon 2022-08-22 10:46:24 CEST 8563 89 89 SIGABRT present /usr/bin/mariadbd 4.7M
Mon 2022-08-22 11:02:53 CEST 563 89 89 SIGABRT present /usr/bin/mariadbd 3.1M
I have attached a snippet of the MariaDB log entries.
I have attached the last systemd-coredump as attachment.
I have attached the OS packages that have been updated, assuming that the cause might lie in the dependencies of MariaDB, such as libc or the Linux kernel.
If more information is needed, please let me know, what is needed and how I can retrieve this info.
ArthurBorsboom, can you please install the debug information for that server and then try to extract the stack trace from a core dump again? The .deb based package name should be mariadb-server-10.8-dbgsym. The .rpm based package name should start with MariaDB-server-debuginfo-10.8.4.
MariaDB [support]> SHOW CREATE TABLE ost__search;
|
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Table | Create Table |
|
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| ost__search | CREATE TABLE `ost__search` (
|
`object_type` varchar(8) NOT NULL,
|
`object_id` int(11) unsigned NOT NULL,
|
`title` text DEFAULT NULL,
|
`content` text DEFAULT NULL,
|
PRIMARY KEY (`object_type`,`object_id`),
|
FULLTEXT KEY `search` (`title`,`content`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 |
|
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
1 row in set (0.000 sec)
|
|
|
MariaDB [support]>
|
@Marko, I am not sure yet how to add debugging info.
The server is a production server, so I can't go too crazy replacing software.
However, if I can enable some flag and restart MariaDB or another service that would be fine.
MariaDB is running on Arch Linux.
Any suggestions?
[arthur@web1 ~]$ mariadbd --debug --help
2022-08-22 12:37:13 0 [Warning] 'debug' is disabled in this build
ArthurBorsboom, I am not familiar with the package structure on Arch Linux, so I do not know the name of the package that would contain the debug information that is normally omitted from the executables themselves. All I can say that in systemd-coredump-2022-08-22.log
, the debug information is not available. Only some function names are available, but not function parameters or source file line numbers.
Debug instrumentation (cmake -DCMAKE_BUILD_TYPE=Debug) is something different. I would not advise anyone to run debug-instrumented executables in production.
ArthurBorsboom, please check https://archlinux.org/news/debug-packages-and-debuginfod/
For Debian, https://wiki.debian.org/AutomaticDebugPackages covers something similar.
ArthurBorsboom, according to https://wiki.archlinux.org/title/Debugging/Getting_traces you should use debuginfod, whatever it is, to download debug information for mariadbd. It seems to be supported, it appears in the https://debuginfod.archlinux.org/packages list
I have to dive into the debugging to retrieve this info.
With the command [sudo coredumpctl gdb 863] I have been able to retrieve an incomplete backtrace.
Core was generated by `/usr/bin/mariadbd'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fbd9fea14dc in ?? () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7fbd9f0b56c0 (LWP 884))]
(gdb) bt
#0 0x00007fbd9fea14dc in ?? () from /usr/lib/libc.so.6
#1 0x00007fbd9fe51998 in raise () from /usr/lib/libc.so.6
#2 0x00007fbd9fe3b5c7 in abort () from /usr/lib/libc.so.6
#3 0x00005563c7d29399 in ?? ()
#4 0x00005563c7d12a64 in ?? ()
#5 0x00005563c8518089 in ?? ()
#6 0x00005563c851e079 in ?? ()
#7 0x00005563c851f672 in ?? ()
#8 0x00005563c851f879 in ?? ()
#9 0x00005563c7d9825a in ?? ()
#10 0x00005563c84b4926 in ?? ()
#11 0x00005563c84b4ae2 in ?? ()
#12 0x00005563c83cd508 in ?? ()
#13 0x00005563c83cd68d in ?? ()
#14 0x00005563c83d84bb in ?? ()
#15 0x00005563c8162c5f in ?? ()
#16 0x00005563c8164006 in ha_commit_trans(THD*, bool) ()
#17 0x00005563c804866a in trans_commit_stmt(THD*) ()
#18 0x00005563c7ef934d in mysql_execute_command(THD*, bool) ()
#19 0x00005563c7f052e2 in mysql_parse(THD*, char*, unsigned int, Parser_state*) ()
#20 0x00005563c7ef543e in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool) ()
#21 0x00005563c7ef75e8 in do_command(THD*, bool) ()
#22 0x00005563c8042724 in do_handle_one_connection(CONNECT*, bool) ()
#23 0x00005563c8042c0b in handle_one_connection ()
#24 0x00005563c8357051 in ?? ()
#25 0x00007fbd9fe9f78d in ?? () from /usr/lib/libc.so.6
#26 0x00007fbd9ff208e4 in clone () from /usr/lib/libc.so.6
(gdb)
I will try to fill in the '??' with more information, possibly with the information provided by Sergei.
Which debug information is missing? Is it mariadb, libc, both or others?
ArthurBorsboom, missing debuginfo for libc does not matter that much. I would assume that abort() was invoked somewhere inside fts_commit() of InnoDB. To see full stack traces of mariadbd, you will need the debug information for MariaDB Server.
Once we have a useful stack trace, we may need some more information from the crash. Some relevant information maybe unavailable due to MDEV-10814 and some special measures may be necessary due to it.
While trying to enable debugging, I can confirm (re)running the following query does trigger the core dump.
REPLACE INTO ost__search SET object_type='H', object_id=9387, content='Hi Chris, That\'s all good. See you this afternoon. BR, Arthur.', title='';
ArthurBorsboom, it is nice that you can reproduce this at will, hopefully also in a staging environment where you would copy the data from the production server.
For now, I am afraid that you’d better to revert the production server to the older version (10.8.3).
For the final debugging, I would suggest to attach GDB to the running server (sudo gdb -p $(pgrep mariadbd), continue), run that crashing statement, and once the SIGABRT is reported in GDB, execute backtrace. thiru would tell which other commands he would like to be executed, once he has a stack trace of the crash for reference.
Hi Marko,
After taking another backup for a debugging environment, I have rolled back the following packages.
downgraded mariadb-libs (10.8.4-1 -> 10.8.3-1)
downgraded mariadb-clients (10.8.4-1 -> 10.8.3-1)
downgraded mariadb (10.8.4-1 -> 10.8.3-1)
This has resolved the issue for me.
Once the debugging environment has been setup, I will try to compile a debug enabled version of MariaDB in order to create a better back trace.
I have created a copy of the VM and prepped it for debugging.
I have created a mariadb package with debugging enabled.
The crash is still present and this is the back trace.
#0 0x00007f23caea14dc in ?? () from /usr/lib/libc.so.6
|
#1 0x00007f23cae51998 in raise () from /usr/lib/libc.so.6
|
#2 0x00007f23cae3b5c7 in abort () from /usr/lib/libc.so.6
|
#3 0x00005581e9f382da in ut_dbg_assertion_failed (expr=expr@entry=0x5581ea9dbbd8 "trx->error_state == DB_SUCCESS",
|
file=file@entry=0x5581ea9dbb98 "/usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc", line=line@entry=728)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/ut/ut0dbg.cc:60
|
#4 0x00005581e9f25842 in que_run_threads (thr=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc:728
|
#5 que_run_threads (thr=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/que/que0que.cc:722
|
#6 0x00005581ea6f35bb in fts_eval_sql (trx=trx@entry=0x7f23b849e100, graph=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0sql.cc:163
|
#7 0x00005581ea6e2fe6 in fts_write_node (trx=0x7f23b849e100, graph=0x7f23abc14ed0, fts_table=0x7f23ca130ad0, word=<optimized out>,
|
node=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3763
|
#8 0x00005581ea6e6861 in fts_sync_write_words (index_cache=0x7f23abc14e80, trx=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3877
|
#9 fts_sync_index (index_cache=0x7f23abc14e80, sync=0x7f23ca9fdf90) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3959
|
#10 fts_sync (sync=0x7f23ca9fdf90) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4105
|
#11 0x00005581ea6e8182 in fts_sync_table (table=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4150
|
#12 fts_sync_table (table=0x7f23ca94f480) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:4145
|
#13 fts_add_doc_by_id (ftt=<optimized out>, doc_id=18758) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3468
|
#14 0x00005581ea6e8295 in fts_add (ftt=<optimized out>, row=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2748
|
#15 0x00005581ea6e8461 in fts_modify (row=<optimized out>, ftt=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2862
|
#16 fts_commit_table (ftt=0x7f23abf69690) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2914
|
#17 fts_commit (trx=trx@entry=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:2959
|
#18 0x00005581e9f9aecf in trx_t::commit_low (this=0x7f23b849cb00, mtr=0x7f23ca131650)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1414
|
#19 0x00005581ea659866 in trx_t::commit_persist (this=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1463
|
#20 0x00005581ea6599a2 in trx_t::commit (this=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1472
|
#21 trx_commit_for_mysql (trx=trx@entry=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/trx/trx0trx.cc:1589
|
#22 0x00005581ea572b01 in innobase_commit_low (trx=0x7f23b849cb00) at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4348
|
#23 0x00005581ea572bd7 in innobase_commit_ordered_2 (trx=0x7f23b849cb00, thd=0x7f23abc25198)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4454
|
#24 0x00005581ea572d4b in innobase_commit (hton=<optimized out>, thd=0x7f23abc25198, commit_trx=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/handler/ha_innodb.cc:4555
|
#25 0x00005581ea2c82bf in commit_one_phase_2 (thd=0x7f23abc25198, all=<optimized out>, trans=0x7f23abc289a8, is_real_trans=true)
|
at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:2067
|
#26 0x00005581ea2c8474 in ha_commit_one_phase (thd=<optimized out>, all=<optimized out>) at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:2020
|
#27 0x00005581ea2d46ae in ha_commit_trans (thd=thd@entry=0x7f23abc25198, all=all@entry=false) at /usr/src/debug/mariadb-10.8.4/sql/handler.cc:1814
|
--Type <RET> for more, q to quit, c to continue without paging--
|
#28 0x00005581ea1be666 in trans_commit_stmt (thd=thd@entry=0x7f23abc25198) at /usr/src/debug/mariadb-10.8.4/sql/transaction.cc:472
|
#29 0x00005581ea09b6d5 in mysql_execute_command (thd=0x7f23abc25198, is_called_from_prepared_stmt=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:6049
|
#30 0x00005581ea09fea3 in mysql_parse (thd=0x7f23abc25198, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:8028
|
#31 0x00005581ea0a1ff6 in dispatch_command (command=COM_QUERY, thd=0x7f23abc25198, packet=<optimized out>, packet_length=<optimized out>,
|
blocking=<optimized out>) at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:1991
|
#32 0x00005581ea0a356b in do_command (thd=0x7f23abc25198, blocking=blocking@entry=true) at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:1407
|
#33 0x00005581ea1af199 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true)
|
at /usr/src/debug/mariadb-10.8.4/sql/sql_connect.cc:1418
|
#34 0x00005581ea1af53e in handle_one_connection (arg=arg@entry=0x7f23ca858b98) at /usr/src/debug/mariadb-10.8.4/sql/sql_connect.cc:1312
|
#35 0x00005581ea4e445d in pfs_spawn_thread (arg=0x7f23ca94f418) at /usr/src/debug/mariadb-10.8.4/storage/perfschema/pfs.cc:2201
|
#36 0x00007f23cae9f78d in ?? () from /usr/lib/libc.so.6
|
#37 0x00007f23caf208e4 in clone () from /usr/lib/libc.so.6
|
I guess this is more helpful?
Hi ArthurBorsboom,
Thanks for sending the stack trace. Looks like InnoDB encountered duplicate entry in FTS auxiliary table. (word, first_doc_id).
first_doc_id could be 18578 based on fts_add_doc_by_id parameter. It would be great to find the word and first doc id
f 8
|
info locals
|
|
|
Above command should show something similar like the following:
|
|
|
(gdb) info locals
|
fts_node = ZZZ
|
word = YYY
|
fts_table = {type = FTS_INDEX_TABLE, table_id = 18, index_id = 30, suffix = 0x5555571684c3 "INDEX_3", table = 0x7fff901fd2c8, charset = 0xe969696900000008}
|
n_nodes = 0
|
n_words = 1
|
rbt_node = 0x7fff90074778
|
error = DB_SUCCESS
|
table = 0x7fff901fd2c8
|
|
|
After that, please type cast the fts_node and word like below:
|
|
|
p *(fts_node_t*) ZZZ
|
p *(fts_tokenizer_word_t*) YYY
|
|
|
f 30
|
p thd->query_string
|
|
Based on query_string output, please send the show create table output.
Do you run any DDL on the affected table ?
Hi Thirunarayanan,
I have (more or less blindly) executed the commands in gdb; the fts_node is unavailable (optimized out).
(gdb) f 8
|
#8 0x00005581ea6e6861 in fts_sync_write_words (index_cache=0x7f23abc14e80, trx=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3877
|
3877 error = fts_write_node(
|
(gdb) info locals
|
fts_node = <optimized out>
|
i = 0
|
selected = <optimized out>
|
word = 0x7f23abfef038
|
fts_table = {type = FTS_INDEX_TABLE, table_id = 65756, index_id = 33145, suffix = 0x5581ea9f2ed3 "INDEX_1", table = 0x7f23ca94f480,
|
charset = 0x5581ea661f24 <rbt_add_node(ib_rbt_t*, ib_rbt_bound_t*, void const*)+164>}
|
n_nodes = 16
|
n_words = 3666
|
rbt_node = 0x7f23abfef018
|
error = <optimized out>
|
table = 0x7f23ca94f480
|
err_exit = <optimized out>
|
fts_table = <optimized out>
|
n_nodes = <optimized out>
|
n_words = <optimized out>
|
rbt_node = <optimized out>
|
error = <optimized out>
|
table = <optimized out>
|
i = <optimized out>
|
selected = <optimized out>
|
word = <optimized out>
|
fts_node = <optimized out>
|
(gdb) p *(fts_tokenizer_word_t*) 0x7f23abfef038
|
$1 = {text = {f_str = 0x7f23abfc9910 "130", f_len = 3, f_n_char = 3}, nodes = 0x7f23abfc9828}
|
(gdb) f 30
|
#30 0x00005581ea09fea3 in mysql_parse (thd=0x7f23abc25198, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>)
|
at /usr/src/debug/mariadb-10.8.4/sql/sql_parse.cc:8028
|
8028 error= mysql_execute_command(thd);
|
(gdb) p thd->query_string
|
$2 = {string = {
|
str = 0x7f23abc4c5b0 "REPLACE INTO ost__search SET object_type='H', object_id=9387, content='Hi Chris, That\\'s all good. See you this afternoon. BR, Arthur.', title=''", length = 145}, cs = 0x5581eb008040 <my_charset_utf8mb3_general_ci>}
|
(gdb)
|
create table output
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| Table | Create Table |
|
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
| ost__search | CREATE TABLE `ost__search` (
|
`object_type` varchar(8) NOT NULL,
|
`object_id` int(11) unsigned NOT NULL,
|
`title` text DEFAULT NULL,
|
`content` text DEFAULT NULL,
|
PRIMARY KEY (`object_type`,`object_id`),
|
FULLTEXT KEY `search` (`title`,`content`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 |
|
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|
TM> Do you run any DDL on the affected table ?
I have no idea. The database is part of an opensource project called osTicket, which is to me a black box.
Do you need more information?
Thanks for sending the results ArthurBorsboom. I would like to verify the first doc id and last doc id. The following could do the trick
|
|
f 7
|
info locals
|
@marko this is a production server. I can't test on it. Later I will reproduce the problem on a test server and make trace on it.
CREATE TABLE `b_user_index` (
|
`USER_ID` int(11) NOT NULL,
|
`SEARCH_USER_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`SEARCH_DEPARTMENT_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`SEARCH_ADMIN_CONTENT` text COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`LAST_NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`SECOND_NAME` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`WORK_POSITION` varchar(255) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
`UF_DEPARTMENT_NAME` varchar(255) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
|
PRIMARY KEY (`USER_ID`),
|
FULLTEXT KEY `IXF_B_USER_INDEX_1` (`SEARCH_USER_CONTENT`),
|
FULLTEXT KEY `IXF_B_USER_INDEX_2` (`SEARCH_DEPARTMENT_CONTENT`),
|
FULLTEXT KEY `IXF_B_USER_INDEX_3` (`SEARCH_ADMIN_CONTENT`)
|
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci
|
@Thirunarayanan, f7 results in the first and last doc id, which seem to be similar.
(gdb) f 7
|
#7 0x00005581ea6e2fe6 in fts_write_node (trx=0x7f23b849e100, graph=0x7f23abc14ed0, fts_table=0x7f23ca130ad0, word=<optimized out>,
|
node=<optimized out>) at /usr/src/debug/mariadb-10.8.4/storage/innobase/fts/fts0fts.cc:3763
|
3763 error = fts_eval_sql(trx, *graph);
|
(gdb) info locals
|
info = 0x7f23abf25700
|
error = <optimized out>
|
doc_count = 16777216
|
start_time = 1661249843
|
last_doc_id = 16809122659206955008
|
first_doc_id = 16809122659206955008
|
table_name = "support/FTS_", '0' <repeats 11 times>, "100dc_", '0' <repeats 12 times>, "8179_INDEX_1\000\000\000\000\270ǫ#\177\000\000\002\000\000\000\000\000\000\000 *`\266#\177\000\000@\b\023\312#\177\000\000K][\352\201U\000\000P\017\023\312#\177\000\000\250\017\023\312#\177\000\000p\b\023\312>\000\000\000\002\000\000\000\000\000\000\000p\b\023\312#\177", '\000' <repeats 11 times>, "\200w\266#\177\000\000\247\063_\352\201U\000\000\001\000\000\000\000\000\000\000h\301\234\312#\177\000\000\000V\207\312#\177\000\000\300\340I\270#\177\000\000@\n\023\312#\177\000\000*"...
|
(gdb)
|
@marko the problem not reproduce on test server with same software configuration.
To summarize the bug report:
- zEvilz and Arthur Borsboom both report a crash with the same error message "(Duplicate key) writing word node to FTS auxiliary".
- zEvilz has upgraded from 10.6.8 > 10.6.9
- Arthur Borsboom has upgraded from 10.8.3 > 10.8.4
- Reverting the upgrade resolves the problem for both users.
- zEvilz can reproduce the crash in the production environment, but not in the test environment
- Arthur Borsboom can reproduce the crash both in the production and test environment, where debugging is enabled.
Can we provide more information to help solving the issue?
Same problem here in our Galera cluster. Spend most of the afternoon, evening and night (now 01:38 here) trying to get things fixed. Ended up with multiple nodes with corrupt data, which I downgraded to 10.6.8, wiped and rebuild using SST from the only node that seemed to have ok data. The entire cluster is now back to 10.6.8, and so far, so good (famous last words...).
2022-08-27 19:41:58 7 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `IPBoard`.`topics`
|
2022-08-27 19:41:58 0x7f7ed00cf700 InnoDB: Assertion failure in file /home/buildbot/buildbot/padding_for_CPACK_RPM_BUILD_SOURCE_DIRS_PREFIX/mariadb-10.6.9/storage/innobase/que/que0que.cc line 728
|
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
|
InnoDB: We intentionally generate a memory trap.
|
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
InnoDB: If you get repeated assertion failures or crashes, even
|
InnoDB: immediately after the mariadbd startup, there may be
|
InnoDB: corruption in the InnoDB tablespace. Please refer to
|
InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
InnoDB: about forcing recovery.
|
220827 19:41:58 [ERROR] mysqld got signal 6 ;
|
We run MariaDB on Almalinux 8.
|
test case |
--source include/have_innodb.inc
|
|
|
CREATE TABLE t ( |
pk int, |
c01 varchar(255), |
c02 varchar(255), |
c03 varchar(255), |
c04 varchar(255), |
c05 varchar(255), |
c06 varchar(255), |
c07 varchar(255), |
c08 varchar(255), |
c09 varchar(255), |
c10 varchar(255), |
c11 varchar(255), |
c12 varchar(255), |
c13 varchar(255), |
c14 varchar(255), |
c15 varchar(255), |
c16 varchar(255), |
c17 varchar(255), |
c18 varchar(255), |
PRIMARY KEY (pk), |
FULLTEXT KEY idx01 (c01), |
FULLTEXT KEY idx02 (c02), |
FULLTEXT KEY idx03 (c03), |
FULLTEXT KEY idx04 (c04), |
FULLTEXT KEY idx05 (c05), |
FULLTEXT KEY idx06 (c06), |
FULLTEXT KEY idx07 (c07), |
FULLTEXT KEY idx08 (c08), |
FULLTEXT KEY idx09 (c09), |
FULLTEXT KEY idx10 (c10), |
FULLTEXT KEY idx11 (c11), |
FULLTEXT KEY idx12 (c12), |
FULLTEXT KEY idx13 (c13), |
FULLTEXT KEY idx14 (c14), |
FULLTEXT KEY idx15 (c15), |
FULLTEXT KEY idx16 (c16), |
FULLTEXT KEY idx17 (c17), |
FULLTEXT KEY idx18 (c18), |
FULLTEXT KEY idx19 (c05), |
FULLTEXT KEY idx20 (c04), |
FULLTEXT KEY idx21 (c01), |
FULLTEXT KEY idx22 (c09), |
FULLTEXT KEY idx23 (c15) |
) ENGINE=InnoDB;
|
|
|
INSERT INTO t VALUES |
(1,'drag','shortly','twelve','correlation','contribution','embrace','pilot','sprinkle','social','reaction','validity','view','tent','survey','fortune','certain','merchant','ladder'), |
(2,'tale','warmth','skill','low','scan','whom','wave','commander','nerve','ah','apologize','prosecutor','continued','yourself','maintenance','cruise','yeah','gear'), |
(3,'exhibition','upstairs','anyone','presidency','type','proper','nature','wealthy','jury','mostly','shorts','commit','concert','weekly','criteria','percentage','fold','complex'), |
(4,'finding','demand','weekend','rabbit','dish','relief','counseling','so','the','chef','colony','instinct','widespread','additional','painter','outside','forbid','blanket'), |
(7,'move','outside','realm','indication','bread','bitter','college','particular','mr','neat','belief','animal','conventional','honey','theoretical','slap','indigenous','desperate'), |
(9,'respect','up','crime','hot','democracy','war','county','outfit','decrease','training','diplomat','terrific','score','pill','man','motor','united','joke'), |
(10,'praise','considerably','characteristic','processor','breeze','stock','popular','flame','medal','behind','juice','rule','arm','opening','soul','verbal','gun','e-mail'), |
(11,'strain','treaty','rival','european','stone','steep','lawsuit','climb','hostage','logic','condition','finger','clock','goal','investment','gravity','comment','count'), |
(12,'them','plunge','fantastic','impressive','engagement','defensive','difficult','afraid','accident','genetic','frequency','accounting','environmental','reportedly','incident','assistant','engineer','foundation'), |
(13,'survival','save','partially','track','lung','tall','story','center','plate','prayer','blank','screw','testimony','bean','developer','union','lock','purple'), |
(16,'responsible','yell','break','squeeze','people','leather','muscle','think','consistently','concerned','mistake','persist','number','mobile','maintenance','visual','inquiry','interpretation'), |
(17,'adventure','trip','chill','halfway','setting','battle','wise','factor','cop','recall','adopt','sir','municipal','fifteen','those','peace','particle','spot'), |
(18,'former','execute','breathe','front','bus','violence','thrive','click','victim','chapter','pregnant','poke','integrity','matter','poll','province','except','reduce'), |
(19,'magnitude','low','shame','announce','soccer','film','uncertainty','pen','pulse','event','cool','relax','commander','array','thrive','most','locate','mask'), |
(20,'survey','effectiveness','scratch','document','without','law','rapid','around','length','boost','grass','yourself','poll','combined','surface','starting','reinforce','happy'), |
(23,'recession','equation','lifestyle','watch','condition','wheelchair','variable','reliability','flexibility','acceptance','lend','beside','rock','diagnosis','govern','mountain','wonderful','conduct'), |
(31,'prospect','station','chart','iraqi','imply','sand','evidence','protect','altogether','advertising','admit','comment','get','uh','therapist','harmony','edit','circuit'), |
(33,'scream','input','flip','deficit','pepper','transformation','week','solely','needle','give','market','passing','bit','abstract','dark','politically','cook','branch'), |
(34,'rage','eligible','cousin','forest','reason','color','stiff','encounter','angle','consequence','bold','negotiate','matter','emotion','proportion','emotion','draft','register'), |
(35,'thanks','substantial','country','guilty','accurately','family','tradition','sun','independent','openly','so-called','cancer','tall','diagnosis','powerful','frustrate','you','solar'), |
(36,'role','prepare','brick','replace','transmit','ah','silent','slot','sort','arrive','reputation','unhappy','adequate','feed','risky','cover','urge','tumor'), |
(37,'advanced','yourself','sit','brilliant','calendar','acquisition','expected','whose','headline','eleven','exactly','evolve','sufficient','adoption','palestinian','adopt','turn','tumor'), |
(40,'violation','plain','reliable','grow','thinking','lightning','stream','culture','invitation','illness','counterpart','overwhelming','nose','hurry','carve','option','great','instead'), |
(43,'form','marker','earnings','custom','each','temple','tissue','lid','indicator','bulb','around','banking','bulk','construction','orange','programming','opposition','burning'), |
(44,'negotiate','suffer','scandal','straighten','informal','ie','total','block','recipe','forever','practically','social','rear','thousand','short-term','drain','laughter','director'), |
(46,'boring','tongue','legitimate','via','decision','deficit','others','ridiculous','cheat','angel','try','communication','diversity','bucket','trunk','impossible','cook','banker'), |
(50,'tonight','ideological','scratch','plenty','communication','comparable','thrive','appear','difficult','chinese','likelihood','mention','message','seal','greek','gene','somewhere','shelter'), |
(51,'high-tech','none','statistical','improvement','coup','enjoy','complain','suggestion','traditional','it','man','unity','convey','the','ice','corruption','palestinian','celebration'), |
(56,'ass','receiver','husband','undertake','offense','billion','extensive','plus','confrontation','bury','back','settlement','currency','entire','buy','excellent','stomach','enact'), |
(60,'silent','newspaper','elite','spoon','restriction','confront','fortune','time','improvement','forest','guest','prescription','tea','attract','adviser','tax','grace','hand'), |
(61,'album','emphasis','originally','strengthen','local','gravity','equip','neck','string','cash','hand','mean','viewer','compose','virtue','too','scent','soap'), |
(62,'quit','skin','elegant','engagement','handle','battle','label','collar','port','buck','smell','embarrassed','severe','diamond','translate','main','resist','slip'), |
(63,'confront','scramble','succeed','convert','teammate','appreciation','fleet','agenda','universe','businessman','size','single','familiar','dense','sometime','dear','roman','physics'), |
(65,'thin','starting','productivity','dna','level','mass','nominee','cling','cry','mountain','sure','earth','devote','substance','surround','interrupt','security','pole'), |
(71,'lie','voluntary','cart','half','variety','whose','fact','aide','skirt','eating','intensity','acceptable','constitution','damn','plant','response','feminist','truck'), |
(72,'provider','reader','information','continuous','health','steady','case','kit','habit','joy','account','register','basic','paint','sexual','convert','outstanding','therapy'), |
(75,'reference','professor','importance','guard','taxpayer','resource','envelope','priest','painting','perfectly','nose','marketplace','somehow','alike','help','creation','toward','cloud'), |
(77,'transmission','strictly','donate','sacrifice','rib','environment','early','then','shared','force','spell','room','staff','islam','tuck','cross','occasion','tune'), |
(78,'spit','lie','attendance','battery','edge','hidden','broad','african-american','seed','house','random','outer','cite','liver','important','wealth','drift','mean'), |
(79,'ring','plate','grow','page','forbid','athlete','extension','technological','chinese','incorporate','complaint','appear','apply','approach','throw','jaw','branch','down'), |
(80,'inventory','somewhere','though','rabbit','smooth','will','happily','heritage','post','complexity','technological','logical','concentrate','critic','nurse','ballot','obstacle','others'), |
(81,'lobby','intense','estimate','favorite','injury','tea','map','hard','ballot','contain','sing','copy','earth','worldwide','outcome','thanks','favorable','hay'), |
(82,'meat','silent','purpose','democratic','reason','rely','ladder','legislation','lion','eating','shelter','charter','ecological','bible','evaluation','adopt','worth','human'), |
(83,'repeat','heal','gathering','crazy','circle','stare','comply','persuade','create','color','substantially','exclusive','natural','anxiety','golf','commodity','wood','luck'), |
(88,'wear','indigenous','disappointed','qualify','alley','accomplishment','hearing','melt','married','assemble','tight','scope','four','church','support','refuge','channel','employment'), |
(89,'counter','spanish','range','incident','session','purse','egg','squeeze','frustrate','please','clue','math','awareness','effort','couch','sovereignty','hallway','obligation'), |
(90,'accelerate','slam','export','parade','industrial','forest','whereas','injure','best','river','feeling','curtain','lobby','lung','jacket','breeze','frustrate','definitely'), |
(93,'contest','nowhere','boring','sofa','guitar','click','manufacturer','resident','marble','education','therapy','grin','impress','undergo','ongoing','workout','modify','irish'), |
(94,'civilian','fee','flag','infection','fatal','above','quality','announcement','bare','acknowledge','estate','refuse','frequently','size','holiday','scale','transmission','short-term'), |
(99,'motor','weakness','organize','criteria','isolated','mathematics','shame','fun','concede','pattern','choose','gap','expect','suspect','shame','japanese','bill','rock'), |
(100,'protein','trick','salad','privately','live','genuine','survivor','afternoon','component','comparison','church','sunny','satellite','dancing','southeast','smoke','journey','old-fashioned'), |
(101,'disc','understand','throw','monitor','likelihood','top','psychology','instruct','subtle','ethics','hardware','retailer','immigration','paper','ask','drill','devastating','interested'), |
(102,'hostile','committee','over','density','newly','franchise','married','sand','cover','banker','athletic','twentieth','face','wet','invention','waste','film','rebuild'), |
(105,'politician','wolf','fever','interval','convert','limitation','jurisdiction','republican','whether','tree','construction','short-term','uncomfortable','dilemma','legally','significant','fever','science'), |
(106,'picture','growth','league','hook','valuable','hypothesis','precisely','mortality','steel','emphasis','flesh','roughly','pressure','expert','distinctive','environmental','wash','hispanic'), |
(107,'introduction','empty','collaboration','administrator','several','that','embrace','fist','panic','easy','blast','ease','prefer','magazine','garage','undergo','compensation','disk'), |
(108,'achieve','yesterday','satisfaction','cast','relative','suicide','actual','member','internal','support','compelling','grave','various','cool','ocean','cash','performance','sector'), |
(113,'portrait','sanction','anything','can','bombing','attend','conscience','dedicate','justice','pond','fighting','credit','murder','pollution','romance','frontier','eligible','textbook'), |
(114,'study','worth','musical','trade','port','tourist','sing','sensitive','five','chapter','expedition','flame','spiritual','rib','please','slowly','ankle','boom'), |
(115,'dry','unit','nearby','numerous','safety','initiative','headquarters','sibling','generation','eating','curriculum','activist','mistake','squeeze','land','anxiety','era','timber'), |
(116,'prevention','such','integrate','drill','quote','commodity','implement','household','relief','softly','agenda','venture','headline','cheer','dawn','chance','crucial','contract'), |
(117,'weave','flexibility','secular','confront','servant','rebuild','essential','quest','ceiling','equity','relative','squeeze','line','form','cross','economics','grandparent','change'), |
(118,'push','scary','determine','implementation','similar','aids','hostage','oil','character','employer','addition','telescope','correctly','envelope','corporate','immigration','slavery','initial'), |
(119,'furniture','ring','productivity','administrator','cold','uncomfortable','stretch','shallow','opposition','claim','strike','mechanical','ambitious','president','execution','team','modify','portion'), |
(121,'spread','environment','rib','wilderness','arrival','efficiency','respond','husband','forgive','adolescent','camera','due','maintenance','toward','beach','detective','personnel','machine'), |
(122,'poke','large','mostly','hardly','cartoon','attention','traditionally','reportedly','phone','application','contemporary','preach','suicide','mom','essential','concert','notice','slap'), |
(123,'distance','lend','re','bathroom','bit','oak','skip','advertising','device','daughter','fantastic','scenario','fare','inner','backyard','objection','sell','frustration'), |
(126,'restore','dessert','response','banana','seemingly','ambition','pen','alien','hard','arrangement','influence','aisle','limited','warmth','century','later','overwhelming','fun'), |
(128,'plan','version','sometime','cut','ignore','slope','indicator','bed','bathroom','carpet','charter','championship','cat','negative','tie','peer','float','evaluation'), |
(134,'hate','slight','cooperative','coffee','accountability','vocal','uncomfortable','hat','humor','racial','sentence','advice','serving','input','exercise','welfare','lens','dying'), |
(136,'journey','patience','importance','mayor','wheat','beginning','leaf','cheap','blessing','error','pen','hockey','same','syndrome','scatter','mountain','globe','update'), |
(137,'cargo','bullet','whoever','reduce','envelope','yell','failure','visit','creative','apologize','soften','plain','remind','highway','retail','weak','lead','male'), |
(146,'corporate','wealth','faculty','ruin','complicated','surprisingly','cup','outsider','descend','hazard','attend','cattle','bunch','tie','premium','due','unity','retail'), |
(149,'ensure','suck','spite','revolution','pipe','inquiry','frown','retain','dessert','drive','historic','bike','exclusive','promotion','attractive','pine','family','mortality'), |
(151,'sweat','lady','essential','clothing','otherwise','extraordinary','yield','artist','reply','influential','piano','concern','wisdom','stir','acceptance','crowd','increased','shrimp'), |
(152,'candy','divorce','compromise','color','twice','one','extend','league','telescope','abroad','available','realize','calculate','voting','disease','fork','concede','rubber'), |
(155,'testify','shorts','automatically','leave','proud','scratch','short','symptom','involve','although','confirm','known','rate','remaining','sacred','fisherman','modest','satisfaction'), |
(156,'conspiracy','whatever','genetic','rid','acid','swing','painting','gang','physics','a','import','tender','hell','teen','pleasure','above','painful','faith'), |
(157,'i','recent','suicide','establish','name','literary','furniture','drug','twenty','debut','dawn','bury','ago','presumably','tool','any','two','grow'), |
(159,'success','fat','persian','republic','tail','oven','exotic','glad','ambitious','abortion','impression','flower','salary','southwest','famous','humor','toxic','sign'), |
(160,'sick','week','music','press','remove','count','wild','read','want','affect','choice','campaign','forward','substantially','jacket','instance','photographer','bit'), |
(161,'feel','connect','scandal','deposit','traditional','campus','aide','video','virtue','bull','literature','partly','height','leadership','battle','action','celebration','promotion'), |
(163,'similarly','wide','moreover','marketing','honest','combined','various','initial','bear','pose','trading','sink','unknown','bush','incorporate','salt','classroom','weapon'), |
(166,'flavor','spread','gym','islamic','offering','pm','fish','dancer','particularly','nominee','simultaneously','ceo','tune','tear','accident','resort','endorse','peanut'), |
(167,'their','supreme','childhood','technical','contact','skin','clothing','lie','critic','professor','discrimination','queen','queen','ban','beautiful','parish','oil','opening'), |
(168,'owner','thumb','cry','venture','wander','group','desk','pace','array','functional','dedicate','acknowledge','enemy','devastating','regulate','inmate','closet','whisper'), |
(171,'death','temple','ring','intend','cliff','unless','quality','fever','port','carrier','ribbon','shut','servant','welfare','commission','country','plain','result'), |
(172,'gang','distinguish','norm','characterize','unity','internet','diamond','consensus','expose','climate','uncle','american','disappointed','meaning','infection','impress','interaction','colorful'), |
(176,'companion','virtue','exchange','lucky','invest','sophisticated','desperately','thoroughly','essentially','hard','cottage','autonomy','therapy','sustain','cute','chin','regime','quantity'), |
(177,'suit','organization','planner','waist','everyday','correct','drink','earn','quality','accurately','operator','alike','defendant','wall','as','tent','screening','mount'), |
(180,'ensure','reliability','desperately','shoe','rough','global','reject','crack','comprise','adequate','appreciate','we','seminar','diversity','landscape','charm','laboratory','lay'), |
(181,'hot','gravity','kill','people','opportunity','solution','architecture','funny','lifestyle','either','six','surprisingly','winter','custody','absence','lover','interfere','remaining'), |
(183,'commitment','tradition','port','parade','chest','attack','have','present','string','style','factory','parental','outdoor','opt','easily','flesh','course','wrap'), |
(186,'extended','roof','genetic','vegetable','display','democratic','following','logic','ideological','ms','two','sale','reportedly','button','horrible','emotional','heritage','absolute'), |
(187,'possibly','month','situation','data','metaphor','fishing','equivalent','install','notion','drift','actor','short','investigation','cord','strategy','sprinkle','sad','decade'), |
(188,'possibly','slight','nonetheless','following','underlying','political','product','assemble','pure','conduct','huh','refugee','catch','admit','outstanding','disappointment','practice','teaching'), |
(190,'policeman','orange','liver','library','condemn','limit','walking','housing','ear','lower','between','rational','resemble','answer','show','license','e-mail','sensitivity'), |
(193,'coach','necessary','shark','round','ratio','will','growing','click','heart','consideration','juror','await','weird','individual','living','beam','breakfast','explosion'), |
(194,'illustrate','sky','concerning','intelligence','mad','contractor','regulator','simultaneously','corporate','accurate','earth','designer','store','instance','burn','driving','loyalty','length'), |
(195,'half','loan','impressive','whoever','boot','so','moreover','stadium','genetic','finally','rough','designer','assert','often','russian','cope','producer','male'), |
(197,'frequently','freedom','pilot','team','finger','latin','pitch','equally','family','experience','rhetoric','sign','reporting','surveillance','excited','cooperation','prosecutor','net'), |
(198,'organize','utility','ever','catch','sensitive','offense','emphasis','slice','difficult','migration','sneak','creature','slide','goat','airport','league','convenience','overlook'), |
(203,'reminder','stiff','quote','opportunity','diagnosis','provoke','attraction','bee','inside','huge','brand','write','propose','scientific','at','colony','life','straighten'), |
(206,'alien','patience','flight','plain','crucial','pen','holy','era','roof','anxious','tale','gun','tree','kneel','sing','sentence','poetry','seriously'), |
(208,'food','examination','dynamic','ideal','fifteen','exception','academic','custom','hostage','buy','repeatedly','concede','so','metaphor','bankruptcy','neither','participant','additional'), |
(210,'admire','fork','hug','palace','elect','ethnic','super','circle','shuttle','battle','donor','enter','lately','efficient','pool','clothes','wherever','nearly'), |
(211,'abortion','parent','message','please','latter','urban','receive','pine','coal','cotton','cabin','tie','hockey','wedding','associate','myth','crucial','crash'), |
(212,'county','impulse','personality','cease','explain','proposal','accuse','popularity','experimental','actress','each','sleep','purchase','image','loyal','nurse','halfway','consequence'), |
(214,'update','theology','skull','issue','meantime','pioneer','file','grandfather','willing','flight','attorney','ensure','stuff','training','simply','wind','charity','encounter'), |
(215,'banking','up','personal','protein','absence','repeatedly','incident','knee','pool','ease','spoon','egg','too','ice','running','therapist','limited','anybody'), |
(217,'minimum','dance','white','solution','false','loud','maintain','than','bounce','convey','attend','balloon','loan','anymore','far','wrist','rain','fellow'), |
(218,'mineral','over','otherwise','prosecutor','training','championship','passing','advanced','adequate','dangerous','interpretation','divine','input','assure','juice','ban','description','coal'), |
(220,'province','facilitate','future','characteristic','shoe','happiness','be','those','devastating','embarrassed','garage','rely','brief','inherit','predict','meantime','pull','hence'), |
(223,'deadline','wander','fraud','pleased','seek','how','manner','bicycle','partial','he','timber','respond','scratch','presence','iraqi','destruction','hide','admission'), |
(226,'fact','stroke','herself','supermarket','works','frustration','inside','discipline','equivalent','spine','clay','really','automatically','opt','wooden','nose','song','die'), |
(229,'interpret','contain','beer','twelve','little','prevail','grandchild','horizon','bat','angle','european','magazine','natural','advantage','plus','ancestor','remark','foreigner'), |
(230,'there','religious','output','amid','faith','contemporary','african-american','merchant','ankle','estimated','rural','condemn','insect','discuss','photo','target','training','uh'), |
(231,'recipe','red','theological','plot','reflection','latter','musician','express','hug','headquarters','citizen','ring','inner','eligible','integration','born','artist','assure'), |
(233,'depending','travel','dot','bid','regret','production','headline','straighten','psychological','southern','behind','separate','heavy','violent','fast','policy','float','totally'), |
(235,'commitment','widely','ongoing','cell','component','neutral','harsh','college','four','immediately','prove','approach','accent','ballot','patent','this','inspector','strongly'), |
(242,'history','explosion','managing','lady','bone','bit','restaurant','southeast','merely','accuse','dip','molecule','skirt','tropical','chapter','mill','entertainment','confidence'), |
(244,'recover','significant','permission','besides','screw','conservative','lifestyle','satisfy','monkey','sigh','disorder','barn','fitness','prospect','conviction','counter','assessment','knee'), |
(248,'imagine','butt','bush','skirt','headache','manufacturing','inmate','appreciation','architecture','athletic','booth','sheep','controversy','consent','core','decision','counterpart','sink'), |
(250,'angry','pork','temporary','offensive','fluid','mostly','girlfriend','attitude','beauty','compel','norm','since','internet','father','rate','boy','court','catalog'), |
(251,'terrible','standard','density','carrier','horizon','proof','implement','cliff','depression','platform','commission','boil','fighter','quarter','living','consultant','realm','promise'), |
(252,'consciousness','mm-hmm','thereby','hug','architecture','well','own','stroke','exhaust','killer','haul','clay','prove','trip','left','policeman','absence','culture'), |
(253,'victory','diary','brown','curriculum','study','long','publicly','slot','weakness','association','credit','communication','swim','proud','abroad','asset','reach','sympathy'), |
(254,'by','exchange','anxiety','energy','marry','unit','elbow','atmosphere','slap','cue','leap','aide','gaze','championship','occur','knock','mysterious','puzzle'), |
(257,'category','resist','lack','drum','uncertain','drift','law','travel','hypothesis','anticipate','mark','restaurant','hospital','earth','tale','mm-hmm','previous','turn'), |
(258,'jew','towards','remaining','degree','scientific','intact','lemon','relative','cat','certainly','creativity','burn','improve','unless','impact','recording','specialty','experienced'), |
(263,'iron','engage','achievement','sex','descend','cloud','generally','cue','hat','earn','i','apologize','faint','frown','common','behavioral','massive','bare'), |
(266,'spine','senior','influential','variety','of','more','leg','distribution','seize','rape','telephone','adolescent','inmate','examination','mill','restaurant','double','warehouse'), |
(267,'cute','internal','jazz','continued','asleep','we','fit','horizon','external','cow','hispanic','fisherman','whale','deep','debut','diversity','natural','project'), |
(269,'seize','opportunity','regulator','mentor','united','river','hope','managing','operator','native','tunnel','advanced','faith','evening','claim','cut','thirty','angel'), |
(271,'informal','time','serving','allow','goat','differ','apparent','virtue','barrel','bishop','improve','lean','super','unable','transaction','toy','theory','coastal'), |
(273,'nature','flood','council','bolt','almost','venture','weekend','refrigerator','politician','coal','self-esteem','fun','blanket','path','implication','accept','commander','neighboring'), |
(274,'grandfather','shelf','pray','departure','resist','interpret','canadian','piece','respectively','balance','spread','couple','camera','threaten','provoke','excessive','permit','fever'), |
(279,'option','ticket','machine','nutrient','bubble','lie','embarrassed','ideal','weigh','alcohol','asian','notice','music','angle','recruit','cattle','exhibit','appoint'), |
(280,'boundary','sexy','victim','corner','resolution','unique','heavily','occupation','greatly','encouraging','opinion','stomach','go','screen','flee','rid','guarantee','lightning'), |
(282,'functional','relevant','timing','remaining','response','preach','always','circumstance','builder','cable','body','contact','attractive','device','replace','romantic','concert','target'), |
(284,'high-tech','teach','logical','paint','help','trick','disappointed','realistic','water','complaint','committee','relieve','retirement','sympathy','corner','serious','joke','elegant'), |
(285,'meat','swimming','regional','love','profound','essential','publicity','association','athlete','fade','bake','decorate','constitutional','composition','price','rain','metaphor','flexible'), |
(290,'kid','specifically','conduct','killing','anniversary','wow','diabetes','nest','catch','jacket','airline','formal','top','pant','isolation','already','chocolate','industrial'), |
(292,'equip','walk','encounter','comparable','shared','heat','far','waste','attention','pitch','panic','assist','layer','cop','epidemic','missile','removal','culture'), |
(295,'scientist','tv','clue','use','it','highway','improvement','unknown','producer','arrangement','dish','handle','standing','learn','acceptable','shopping','risky','discourage'), |
(302,'by','girl','characterize','fee','hint','fee','policy','equally','dare','better','drive','skill','billion','toll','expense','hopefully','incredible','disappear'), |
(306,'lower','fail','settlement','allow','gear','oppose','carbon','dam','straw','both','realistic','encouraging','recommendation','fiscal','realize','beef','increased','arise'), |
(310,'maintain','soil','soon','painful','supporter','correlation','musical','chin','current','die','call','stack','write','scientific','depict','route','exciting','mention'), |
(311,'century','motion','legitimate','realize','super','interaction','computer','strong','call','correctly','sprinkle','piano','mortgage','send','dozen','forth','initiate','strictly'), |
(312,'pole','drum','russian','sit','type','restriction','assignment','presence','aspect','attract','sack','obtain','disorder','hockey','allow','stiff','standing','campaign'), |
(313,'practice','telephone','permanent','price','system','improved','stroke','traffic','math','copy','avoid','dense','pitch','contract','flee','project','influential','onto'), |
(317,'taste','note','annual','announcement','forgive','hunt','explosion','impose','opposition','auction','if','sigh','expose','turkey','psychologist','seed','box','authority'), |
(320,'activity','vaccine','sister','skip','everybody','revolutionary','shadow','assessment','fascinating','collector','attack','bicycle','hay','companion','section','distribute','trust','original'), |
(323,'irish','plunge','she','scale','vaccine','thus','thus','speaker','universe','analyst','offense','concept','possibly','missing','vehicle','training','which','fight'), |
(324,'bacteria','scheme','endure','answer','season','necessary','sing','racial','hour','model','arm','butt','secular','hour','young','sector','pit','forget'), |
(331,'start','which','intelligent','knock','costume','fast','ambitious','withdraw','provide','spot','why','valuable','hunger','when','the','month','towards','totally'), |
(334,'unknown','unit','honest','thinking','advocate','middle','address','symptom','every','another','latter','personal','hotel','freedom','steer','performer','introduce','pant'), |
(335,'grocery','reliable','illness','college','grief','tonight','brain','six','routine','deficit','random','security','portfolio','romance','sometimes','closest','eye','various'), |
(336,'praise','twenty','recruit','sure','actively','stress','read','sweater','one','gain','automatic','head','mrs','closer','spectacular','ready','tolerate','auction'), |
(337,'peer','secretary','camp','aisle','consensus','silent','matter','energy','past','drama','completely','blink','quietly','soil','alter','way','drawing','hotel'), |
(339,'perfect','sake','known','forbid','nowhere','real','reward','reserve','text','profound','activist','beyond','proposal','basement','contemporary','division','plain','bold'), |
(342,'growing','foot','development','oh','relative','distant','cooperative','maximum','weakness','drawing','final','side','anniversary','dark','soar','screw','unknown','join'), |
(343,'heal','training','load','choice','five','crucial','costume','envision','fiction','briefly','voter','ritual','patent','error','e-mail','square','hence','jewelry'), |
(344,'citizenship','stop','to','carbon','relieve','threshold','authority','basketball','anonymous','response','become','gift','good','win','edge','needle','productivity','theology'), |
(345,'upstairs','press','journalism','four','works','import','hallway','perform','effect','hot','confrontation','what','football','planet','warrior','severe','teaching','modify'), |
(346,'concrete','youth','speak','direct','operator','suggestion','ranch','mate','suit','intense','punishment','if','interval','own','bell','satellite','team','truck'), |
(354,'blend','side','fully','lower','colleague','effectively','properly','police','bath','emphasize','ms','driver','out','walk','ingredient','code','spend','protein'), |
(355,'psychologist','permanent','increasing','spanish','follow','proof','driver','hip','husband','excellent','aside','director','negotiate','fuel','ought','goat','diabetes','scared'), |
(359,'opposite','conspiracy','stick','multiple','drawing','care','nutrient','airplane','lately','absence','added','fiction','accurate','differ','organized','ceo','legend','rice'), |
(360,'angel','puzzle','vacuum','skip','always','departure','balloon','beautiful','hundred','breast','feeling','likely','lifetime','crack','aids','chapter','perspective','civilian'), |
(362,'obstacle','tolerance','scream','traveler','leave','unknown','performer','obvious','cover','leather','vote','prospect','trunk','core','preserve','building','horizon','isolation'), |
(363,'armed','wheel','scientist','clothes','deposit','ash','wander','proposed','reinforce','professor','acceptable','separate','momentum','ego','emission','several','today','anxious'), |
(364,'speak','forest','tolerance','membership','insist','sword','anyway','thin','risky','coat','at','basket','pepper','best','describe','pregnancy','faint','rating'), |
(366,'onto','outcome','envelope','carrot','natural','grief','childhood','probably','read','exhaust','unlike','suddenly','ownership','man','array','legitimate','smile','fast'), |
(369,'house','vessel','potato','franchise','ban','snap','admission','everything','streak','acid','their','citizen','solution','greek','assessment','minute','encourage','hotel'), |
(370,'childhood','party','can','someone','somehow','investigate','assemble','blink','our','and','iraqi','drum','trunk','loose','harassment','drown','encounter','fool'), |
(372,'offensive','routinely','oh','trace','dissolve','through','hip','alone','crime','resolve','spit','wheat','sometime','dried','horn','boost','motor','beginning'), |
(373,'wheelchair','device','trash','assault','huh','priority','gas','appreciation','funding','border','molecule','creature','lately','satisfaction','man','lap','judgment','guard'), |
(374,'exciting','exploit','yet','certain','i','smoke','least','reliability','inflation','closed','legally','limitation','thanksgiving','establishment','experimental','deer','each','lip'), |
(376,'require','plot','point','library','reference','swallow','catalog','strong','glad','folk','size','ok','doubt','agree','update','yet','spouse','diet'), |
(378,'radical','retirement','depression','land','educator','developmental','people','music','policy','electricity','miracle','commissioner','participant','layer','competitor','when','van','phone'), |
(380,'leave','medicine','nominee','shine','personality','surface','ease','line','proper','empire','major','artistic','uncomfortable','branch','rather','carve','emphasize','sheer'), |
(387,'port','terror','song','passenger','everyday','integrated','hesitate','reserve','scream','conservative','electronics','include','dominant','credibility','diverse','shuttle','works','retreat'), |
(388,'rebuild','cup','opponent','closet','constitute','public','soviet','trunk','transmit','changing','sing','european','neutral','acceptable','greatly','idea','widely','flow'), |
(389,'together','unique','angry','characterize','black','nod','win','texture','continuous','proper','add','due','bonus','withdrawal','final','sector','excited','loyal'), |
(391,'sequence','popularity','insert','example','initiate','practitioner','responsible','shirt','productive','commit','choice','cow','mix','numerous','industry','compliance','profession','title'), |
(394,'tree','nuclear','sheet','anyway','ball','accommodate','recall','badly','prohibit','exposure','infection','jar','rescue','horse','divine','fur','instantly','stomach'), |
(400,'racism','rack','law','girl','analysis','decrease','cave','passing','speaker','interested','sign','bike','sexuality','bee','franchise','map','mentally','teacher'), |
(402,'violence','transaction','borrow','sky','proper','lead','journalism','franchise','external','boss','chain','chin','overwhelming','popular','offender','movie','dump','lucky'), |
(403,'kingdom','refrigerator','motivate','diet','blond','golden','today','expectation','specifically','crash','alter','wildlife','activity','crash','negotiate','disappear','symbolic','type'), |
(407,'tolerate','marriage','elaborate','due','hay','lifetime','reduction','then','correspondent','incredibly','within','increasingly','perform','six','hearing','etc','investor','anonymous'), |
(408,'eye','vulnerable','garage','inside','galaxy','parent','explore','unless','classroom','contribute','coverage','weed','outdoor','captain','nation','these','occupation','automatically'), |
(409,'cruel','harvest','elderly','moreover','temperature','remind','nowhere','nightmare','reduction','certainly','ideological','predator','literary','concept','catalog','annual','preserve','gun'), |
(411,'haul','fatal','conflict','folk','primary','retirement','undergo','publicity','objection','blame','hero','golden','stove','talented','rolling','ago','skull','enjoy'), |
(412,'without','render','await','mr','yes','require','deadly','edit','religion','fix','realistic','earthquake','blade','victory','latter','import','photograph','inherit'), |
(414,'tighten','now','actor','objection','contend','primarily','pan','courtroom','partly','apple','consensus','phenomenon','progressive','deliberately','mathematics','kill','spray','feather'), |
(415,'organic','schedule','new','furthermore','past','analysis','practically','friendly','behavior','arrangement','count','firmly','pitch','palm','knife','predator','five','key'), |
(416,'date','tropical','likewise','ride','dispute','importance','mushroom','curious','nest','declare','accident','consist','peace','prescription','tomato','persist','living','gently'), |
(418,'producer','escape','psychology','likewise','see','auto','normally','script','soft','bedroom','next','depressed','black','entrance','faculty','search','multiple','leader'), |
(420,'priest','guidance','medium','disappointment','unit','banker','civil','guest','bullet','fifty','tie','huh','essentially','rod','risky','order','intense','parking'), |
(421,'explore','code','discussion','organized','rip','photograph','van','sphere','prepare','circuit','complaint','instructor','reveal','cart','reliability','junior','especially','uh'), |
(422,'regularly','recipient','identify','manufacturing','blend','mill','courtroom','in','history','database','criticize','fiscal','rib','desk','contract','leg','meter','device'), |
(424,'insight','discrimination','seminar','judgment','accommodate','curve','gold','spell','guilt','conflict','supermarket','carefully','convince','passage','mere','resist','municipal','halfway'), |
(429,'formation','so-called','panic','breathe','piece','retreat','clinic','ear','map','disagree','face','rear','dramatic','married','indicator','male','lifestyle','hit'), |
(430,'cooking','service','interview','angel','fruit','accessible','sustainable','arrangement','minimize','friendly','makeup','rat','chamber','planner','cloth','institutional','fate','hate'), |
(432,'genetic','receiver','accurately','inevitable','conventional','ignore','pressure','brutal','conspiracy','lead','produce','order','and/or','justice','steep','would','technological','low'), |
(436,'ear','strongly','letter','privately','bear','conflict','personality','courtroom','sole','denial','harsh','yesterday','twist','liver','constant','chest','required','than'), |
(437,'harassment','trigger','format','quick','club','inspection','works','hit','cling','fishing','boast','satisfaction','coastal','tube','approximately','lion','found','tomorrow'), |
(439,'running','tighten','campus','chill','revelation','articulate','hallway','indian','number','page','bias','input','crowded','peace','inherit','sufficient','crazy','progress'), |
(440,'deer','transformation','interior','instructor','identity','on','jazz','grade','opposed','legislator','serving','our','alcohol','director','butt','fee','suitable','resolution'), |
(442,'temple','telescope','household','both','corner','soap','unfortunately','temple','battery','alternative','voter','speak','dismiss','low','mystery','channel','cast','detect'), |
(447,'absence','nation','human','knife','thirty','fiscal','clock','crowded','loss','frustration','plane','gallery','really','branch','commission','salt','helpful','rock'), |
(449,'damn','vacuum','chair','surgeon','musician','lately','rely','emerging','partnership','stove','medium','key','adoption','hunter','awful','always','use','adult'), |
(450,'crack','retire','regional','popular','gathering','skull','density','plea','one','name','costume','consist','planner','teen','carry','bottle','ticket','lesson'), |
(454,'enough','shower','sad','costly','around','poke','sister','transmit','supporter','rural','introduce','grape','nightmare','us','affair','post','road','greatly'), |
(457,'drink','diabetes','element','leather','teaspoon','confess','crucial','russian','commission','buddy','tent','depend','success','traditionally','alone','associate','equally','minimum'), |
(458,'panel','tooth','surgery','various','program','lion','clearly','ridiculous','understand','impression','overwhelm','helmet','meantime','promising','skill','questionnaire','onion','puzzle'), |
(459,'assemble','extend','responsibility','unfair','rice','missile','damn','childhood','ideological','booth','dam','shortly','await','gear','simultaneously','happiness','cat','feedback'), |
(462,'extremely','necessary','grave','spark','morning','appointment','bush','regular','explicit','hopefully','equity','atmosphere','shoe','alone','publisher','duck','reliability','conservation'), |
(464,'literally','trial','boast','function','shirt','expedition','club','little','competition','bomb','overwhelming','bullet','encouraging','theoretical','accelerate','famous','increase','rush'), |
(466,'day','stair','limitation','with','port','observe','until','snap','corner','jaw','bedroom','documentary','nonprofit','afraid','slavery','year','rely','aisle'), |
(470,'opera','substantial','wildlife','branch','at','headquarters','rate','shop','produce','heat','degree','exceed','fatal','charm','sensitive','astronomer','initiative','defensive'), |
(471,'scare','minimum','gain','hand','belt','variety','pill','road','install','finding','shore','brave','dna','battle','law','jail','comparable','cabinet'), |
(472,'eyebrow','previously','concert','telescope','desperate','pig','client','action','practitioner','incredibly','shooting','chemical','story','year','likely','disease','slam','hit'), |
(474,'suspect','southwest','strike','consequently','peer','emotional','beside','alter','scope','density','guide','known','slight','creation','longtime','backyard','motive','hip'), |
(476,'commodity','full-time','slight','candle','commander','freshman','fog','chance','structure','addition','province','translation','administrator','radar','judicial','very','boss','born'), |
(477,'topic','neck','machine','deserve','whisper','recession','horse','asian','acquisition','changing','pass','council','mrs','eye','lap','inflation','understand','advance'), |
(479,'angle','should','final','one-third','beginning','struggle','painful','exchange','clothing','leaf','product','depressed','launch','broad','define','show','pay','dig'), |
(480,'allegedly','religious','legacy','awareness','pressure','dna','civilization','artistic','equality','dock','throat','rim','occasion','additional','treat','public','robot','associate'), |
(481,'distribution','sustainable','today','demographic','hit','single','behavior','responsible','stage','outcome','rescue','firm','supposed','loop','poll','innocent','oxygen','confuse'), |
(484,'establishment','moreover','lower','overcome','compliance','accelerate','fork','raise','convey','irony','explain','potentially','comprise','supporter','bread','the','grant','practice'), |
(485,'civilization','milk','size','brake','prize','ministry','key','press','category','legitimate','automatic','remote','subject','conventional','labor','integration','expectation','vs'), |
(487,'arrive','win','exhibit','wealth','yourself','sink','wheat','texture','marker','midst','gasoline','friendly','chicken','physician','born','headline','first','approve'), |
(488,'season','voice','identity','democrat','gay','article','selected','legend','strain','fisherman','debate','host','start','understand','jump','demonstrate','access','edition'), |
(489,'uh','jet','square','would','stare','control','professor','pink','guide','daily','educational','pipe','solar','ingredient','workout','duck','town','bicycle'), |
(490,'consist','still','motion','replace','produce','listen','advice','uniform','seventh','cocaine','articulate','presence','supply','absolute','greatest','suggest','segment','okay'), |
(495,'proclaim','statue','mountain','repair','aim','style','debate','project','stare','spare','sustainable','merit','arab','throughout','row','precious','twentieth','entitle'), |
(499,'bell','resident','suffer','sock','persist','punishment','attention','backyard','historical','aide','hockey','cool','ecological','setting','short-term','multiple','copy','necessity'); |
|
|
ALTER TABLE t ADD FULLTEXT INDEX idx(c04); |
|
|
UPDATE t SET c10 = 'foo'; |
UPDATE t SET c10 = 'bar'; |
|
|
--source include/restart_mysqld.inc
|
|
|
UPDATE t SET c09 = 'qux'; |
ALTER TABLE t FORCE; |
|
|
# Cleanup
|
DROP TABLE t; |
|
10.6 0fbcb0a2 non-debug |
2022-08-28 18:29:49 3 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `test`.`t`
|
2022-08-28 18:29:49 0x7fa750ff3700 InnoDB: Assertion failure in file /data/src/10.6/storage/innobase/que/que0que.cc line 728
|
InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
|
|
|
#5 0x00007fa757c6e537 in __GI_abort () at abort.c:79
|
#6 0x000055cd386e0b18 in ut_dbg_assertion_failed (expr=expr@entry=0x55cd39323fc0 "trx->error_state == DB_SUCCESS", file=file@entry=0x55cd39323f90 "/data/src/10.6/storage/innobase/que/que0que.cc", line=line@entry=728) at /data/src/10.6/storage/innobase/ut/ut0dbg.cc:60
|
#7 0x000055cd386cada0 in que_run_threads (thr=<optimized out>) at /data/src/10.6/storage/innobase/que/que0que.cc:728
|
#8 que_run_threads (thr=<optimized out>) at /data/src/10.6/storage/innobase/que/que0que.cc:722
|
#9 0x000055cd38f213e9 in fts_eval_sql (trx=trx@entry=0x7fa7522a7180, graph=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0sql.cc:163
|
#10 0x000055cd38f0e7df in fts_write_node (trx=0x7fa7522a7180, graph=0x7fa72406b488, fts_table=0x7fa750ff0330, word=<optimized out>, node=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3763
|
#11 0x000055cd38f13e3c in fts_sync_write_words (index_cache=0x7fa724079980, trx=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3877
|
#12 fts_sync_index (index_cache=0x7fa724079980, sync=0x7fa72406b330) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3959
|
#13 fts_sync (sync=0x7fa72406b330) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4105
|
#14 0x000055cd38f1587d in fts_sync_table (table=<optimized out>, table=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4150
|
#15 fts_sync_table (table=0x7fa72406a600) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:4145
|
#16 fts_add_doc_by_id (ftt=<optimized out>, doc_id=779) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:3468
|
#17 0x000055cd38f15aa5 in fts_add (ftt=<optimized out>, row=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2748
|
#18 0x000055cd38f15c76 in fts_modify (row=<optimized out>, ftt=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2862
|
#19 fts_commit_table (ftt=<optimized out>) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2914
|
#20 fts_commit (trx=trx@entry=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/fts/fts0fts.cc:2959
|
#21 0x000055cd3873dcbe in trx_t::commit_low (this=0x7fa7522a5b80, mtr=0x7fa750ff10a0) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1422
|
#22 0x000055cd38e6c6f5 in trx_t::commit_persist (this=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1471
|
#23 0x000055cd38e6c8e2 in trx_t::commit (this=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1480
|
#24 trx_commit_for_mysql (trx=trx@entry=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/trx/trx0trx.cc:1597
|
#25 0x000055cd38d61bf0 in innobase_commit_low (trx=<optimized out>) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4416
|
#26 innobase_commit_ordered_2 (thd=0x7fa71c000c58, trx=0x7fa7522a5b80) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4522
|
#27 innobase_commit (hton=<optimized out>, thd=0x7fa71c000c58, commit_trx=<optimized out>) at /data/src/10.6/storage/innobase/handler/ha_innodb.cc:4626
|
#28 0x000055cd38a58539 in commit_one_phase_2 (thd=thd@entry=0x7fa71c000c58, all=<optimized out>, trans=0x7fa71c004410, is_real_trans=<optimized out>) at /data/src/10.6/sql/handler.cc:2070
|
#29 0x000055cd38a58784 in ha_commit_one_phase (thd=thd@entry=0x7fa71c000c58, all=all@entry=false) at /data/src/10.6/sql/handler.cc:2023
|
#30 0x000055cd38a655de in ha_commit_trans (thd=thd@entry=0x7fa71c000c58, all=all@entry=false) at /data/src/10.6/sql/handler.cc:1817
|
#31 0x000055cd389494b3 in trans_commit_stmt (thd=thd@entry=0x7fa71c000c58) at /data/src/10.6/sql/transaction.cc:472
|
#32 0x000055cd388252d9 in mysql_execute_command (thd=0x7fa71c000c58, is_called_from_prepared_stmt=<optimized out>) at /data/src/10.6/sql/sql_parse.cc:6057
|
#33 0x000055cd3882a197 in mysql_parse (thd=0x7fa71c000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/10.6/sql/sql_parse.cc:8030
|
#34 0x000055cd3882ca5d in dispatch_command (command=COM_QUERY, thd=0x7fa71c000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /data/src/10.6/sql/sql_class.h:234
|
#35 0x000055cd3882e712 in do_command (thd=0x7fa71c000c58, blocking=blocking@entry=true) at /data/src/10.6/sql/sql_parse.cc:1409
|
#36 0x000055cd389395ff in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /data/src/10.6/sql/sql_connect.cc:1418
|
#37 0x000055cd3893994d in handle_one_connection (arg=arg@entry=0x55cd3adb3e88) at /data/src/10.6/sql/sql_connect.cc:1312
|
#38 0x000055cd38cbb6f2 in pfs_spawn_thread (arg=0x55cd3adc8c68) at /data/src/10.6/storage/perfschema/pfs.cc:2201
|
#39 0x00007fa75812fea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
#40 0x00007fa757d46def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
|
10.6 0fbcb0a2b87 debug |
2022-08-28 18:38:44 3 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `test`.`t`
|
mariadbd: /data/src/10.6-bug/storage/innobase/que/que0que.cc:223: que_thr_t* que_fork_start_command(que_fork_t*): Assertion `thr->state == QUE_THR_COMPLETED' failed.
|
220828 18:38:44 [ERROR] mysqld got signal 6 ;
|
|
|
#7 0x00007f20911af662 in __GI___assert_fail (assertion=0x55c3a31f83f0 "thr->state == QUE_THR_COMPLETED", file=0x55c3a31f8320 "/data/src/10.6-bug/storage/innobase/que/que0que.cc", line=223, function=0x55c3a31f8458 "que_thr_t* que_fork_start_command(que_fork_t*)") at assert.c:101
|
#8 0x000055c3a2a40f4a in que_fork_start_command (fork=0x7f1fd8a11b40) at /data/src/10.6-bug/storage/innobase/que/que0que.cc:223
|
#9 0x000055c3a2ca819e in fts_eval_sql (trx=0x7f2087400180, graph=0x7f1fd8a11b40) at /data/src/10.6-bug/storage/innobase/fts/fts0sql.cc:161
|
#10 0x000055c3a2c8fa45 in fts_write_node (trx=0x7f2087400180, graph=0x7f204c06bf70, fts_table=0x7f208c13db30, word=0x7f1fd89b9718, node=0x7f1fd89b7df0) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3763
|
#11 0x000055c3a2c8ff1c in fts_sync_write_words (trx=0x7f2087400180, index_cache=0x7f204c082ea8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3877
|
#12 0x000055c3a2c90501 in fts_sync_index (sync=0x7f204c06bec8, index_cache=0x7f204c082ea8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3959
|
#13 0x000055c3a2c90cdd in fts_sync (sync=0x7f204c06bec8) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:4105
|
#14 0x000055c3a2c90f14 in fts_sync_table (table=0x7f204c052338) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:4150
|
#15 0x000055c3a2c8f091 in fts_add_doc_by_id (ftt=0x7f1fd8960e48, doc_id=779) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:3468
|
#16 0x000055c3a2c8d4cb in fts_add (ftt=0x7f1fd8960e48, row=0x7f1fd8970658) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2748
|
#17 0x000055c3a2c8dc04 in fts_commit_table (ftt=0x7f1fd8960e48) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2910
|
#18 0x000055c3a2c8dcf7 in fts_commit (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/fts/fts0fts.cc:2959
|
#19 0x000055c3a2b4d8be in trx_t::commit_low (this=0x7f20873feb80, mtr=0x7f208c13ebb0) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1422
|
#20 0x000055c3a2b4ea71 in trx_t::commit_persist (this=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1471
|
#21 0x000055c3a2b4eb62 in trx_t::commit (this=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1480
|
#22 0x000055c3a2b4efc9 in trx_commit_for_mysql (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/trx/trx0trx.cc:1597
|
#23 0x000055c3a28e1bb1 in innobase_commit_low (trx=0x7f20873feb80) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4416
|
#24 0x000055c3a28e1ee3 in innobase_commit_ordered_2 (trx=0x7f20873feb80, thd=0x7f1fd8000db8) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4522
|
#25 0x000055c3a28e2439 in innobase_commit (hton=0x55c3a5f0bd78, thd=0x7f1fd8000db8, commit_trx=false) at /data/src/10.6-bug/storage/innobase/handler/ha_innodb.cc:4626
|
#26 0x000055c3a24d5b32 in commit_one_phase_2 (thd=0x7f1fd8000db8, all=false, trans=0x7f1fd8004720, is_real_trans=true) at /data/src/10.6-bug/sql/handler.cc:2070
|
#27 0x000055c3a24d58ef in ha_commit_one_phase (thd=0x7f1fd8000db8, all=false) at /data/src/10.6-bug/sql/handler.cc:2023
|
#28 0x000055c3a24d4a37 in ha_commit_trans (thd=0x7f1fd8000db8, all=false) at /data/src/10.6-bug/sql/handler.cc:1817
|
#29 0x000055c3a230ef00 in trans_commit_stmt (thd=0x7f1fd8000db8) at /data/src/10.6-bug/sql/transaction.cc:472
|
#30 0x000055c3a2147a83 in mysql_execute_command (thd=0x7f1fd8000db8, is_called_from_prepared_stmt=false) at /data/src/10.6-bug/sql/sql_parse.cc:6057
|
#31 0x000055c3a214d450 in mysql_parse (thd=0x7f1fd8000db8, rawbuf=0x7f1fd8017d40 "UPDATE t SET c09 = 'qux'", length=24, parser_state=0x7f208c1403c0) at /data/src/10.6-bug/sql/sql_parse.cc:8030
|
#32 0x000055c3a2139ab3 in dispatch_command (command=COM_QUERY, thd=0x7f1fd8000db8, packet=0x7f1fd800b879 "UPDATE t SET c09 = 'qux'", packet_length=24, blocking=true) at /data/src/10.6-bug/sql/sql_parse.cc:1896
|
#33 0x000055c3a21384ae in do_command (thd=0x7f1fd8000db8, blocking=true) at /data/src/10.6-bug/sql/sql_parse.cc:1409
|
#34 0x000055c3a22f2a7c in do_handle_one_connection (connect=0x55c3a64021d8, put_in_cache=true) at /data/src/10.6-bug/sql/sql_connect.cc:1418
|
#35 0x000055c3a22f271b in handle_one_connection (arg=0x55c3a64021d8) at /data/src/10.6-bug/sql/sql_connect.cc:1312
|
#36 0x000055c3a280f942 in pfs_spawn_thread (arg=0x55c3a60588c8) at /data/src/10.6-bug/storage/perfschema/pfs.cc:2201
|
#37 0x00007f209167bea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
|
#38 0x00007f2091278def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
|
I can confirm this bug with MariaDB 10.6.9 from Debian/Ubuntu packages:
- 10.6.9+maria~deb11
- 10.6.9+maria~ubu2004
- 10.6.9+maria~ubu2204
It is not dependent on used table schema or amount of data in table, but occurs randomly when inserting rows to FTS enabled tables.
As a workaround we downgraded to 10.6.8 release. The same database directory worked after downgrade without further repairs.
Daemon log for one of the affected 10.6.9 servers (some information has been redacted for security):
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 2022-08-29 14:20:00 236 [ERROR] InnoDB: (Duplicate key) writing word node to FTS auxiliary index table `redacted`.`forsecurity`
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 2022-08-29 14:20:00 0x7fd0041f6640 InnoDB: Assertion failure in file ./storage/innobase/que/que0que.cc line 728
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: Failing assertion: trx->error_state == DB_SUCCESS
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: We intentionally generate a memory trap.
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: If you get repeated assertion failures or crashes, even
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: immediately after the mariadbd startup, there may be
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: corruption in the InnoDB tablespace. Please refer to
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: InnoDB: about forcing recovery.
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: 220829 14:20:00 [ERROR] mysqld got signal 6 ;
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: This could be because you hit a bug. It is also possible that this binary
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: or one of the libraries it was linked against is corrupt, improperly built,
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: We will try our best to scrape up some info that will hopefully help
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: diagnose the problem, but since we have already crashed,
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: something is definitely wrong and this may fail.
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Server version: 10.6.9-MariaDB-1:10.6.9+maria~ubu2204
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: key_buffer_size=134217728
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: read_buffer_size=131072
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: max_used_connections=7
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: max_threads=153
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: thread_count=7
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: It is possible that mysqld could use up to
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467959 K bytes of memory
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Hope that's ok; if not, decrease some variables in the equation.
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Thread pointer: 0x7fcc58000c68
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: Attempting backtrace. You can use the following information to find out
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: where mysqld died. If you see no messages after this, something went
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: terribly wrong...
|
Aug 29 14:20:00 mariadb.redacted.for.security mariadbd[238011]: stack_bottom = 0x7fd0041f5c78 thread_stack 0x49000
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(my_print_stacktrace)[0x5587a78a2332]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(handle_fatal_signal)[0x5587a7360ae8]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(__sigaction)[0x7fd00c027520]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(pthread_kill)[0x7fd00c07ba7c]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(raise)[0x7fd00c027476]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(abort)[0x7fd00c00d7f3]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x5587a6fdfb98]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(Wsrep_server_service::log_dummy_write_set(wsrep::client_state&, wsrep::ws_meta const&))[0x5587a6fc9a98]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> >>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(main)[0x5587a703f8e6]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5587a77606e9]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(void std::this_thread::sleep_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&))[0x5587a77608e2]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(wsrep_notify_status(wsrep::server_state::state, wsrep::view const*))[0x5587a7655c30]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(ha_check_and_coalesce_trx_read_only(THD*, Ha_trx_info*, bool))[0x5587a73643a9]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(ha_commit_trans(THD*, bool))[0x5587a73716a6]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(trans_commit_stmt(THD*))[0x5587a7251227]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(mysql_execute_command(THD*, bool))[0x5587a7129fcc]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5587a712ef57]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool))[0x5587a71316a5]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(do_command(THD*, bool))[0x5587a7133382]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(do_handle_one_connection(CONNECT*, bool))[0x5587a7240f6f]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(handle_one_connection)[0x5587a72412bd]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(MyCTX_nopad::finish(unsigned char*, unsigned int*))[0x5587a75acdb6]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(pthread_condattr_setpshared)[0x7fd00c079b43]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: ??:0(__xmknodat)[0x7fd00c10ba00]
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Trying to get some variables.
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Some pointers may be invalid and cause the dump to abort.
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Query (0x7fcc58012e40): insert into forsecurity <REDACTED FOR SECURITY>
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Connection ID (thread ID): 236
|
Aug 29 14:20:01 mariadb.redacted.for.security mariadbd[238011]: Status: NOT_KILLED
|
Any chance that 10.6 LTS minor version including this patch could be released at an expedited pace?
Since both FTS and MariaDB LTS is very popular for example in hosting industry (mainly WordPress, which commonly triggers this bug), this is hitting heavily on many hosting 10.6 LTS based services.
Also, this bug is affecting Docker containers using :10.6 release tag, which is expected to be relatively stable within minor release versions.
The fix looks OK to me. It consists of two parts:
- fixing the so far only known underlying cause (
MDEV-15237), in fts_cmp_set_sync_doc_id(). - preventing crashes in case the duplicate key would occur
The tree origin/bb-10.6-MDEV-29342 e8bc9d499ce4b6f6ec63f1a962f4af4cb9ef537f 2022-08-29T23:36:23+05:30
behaved well in RQG testing. Please be aware that up till now none of the RQG tests was ever capable to replay the problem.
The fix (to all or part of MDEV-15237) was pushed to 10.3. Some additional fixes were pushed to 10.6. The first version that could crash due to this bug was 10.6.9. We do not know yet if MDEV-15237 was fixed by this.
ArthurBorsboom, by default MariaDB Server releases occur 4 times per year. A fix for this crash is included in the Enterprise Server 10.6.9-5 release.
I have advocated for unscheduled releases, because a combination of this bug and MDEV-29374 can result in a permanently corrupted database, but it does not currently seem that one will be made.
Thank you for your efforts to get an unscheduled release out.
I'd still suggest at least rolling back official MariaDB Docker images released with LTS tag mariadb:10.6 to version 10.6.8.
Those are spreading this db corruption issue like a wildfire without people making choice or even being aware what patch-version of 10.6 they run.
Arch Linux uses 10.9.2 at 7 september 2022.
I am able to reproduce the issue in a test environment, so I am able to test and confirm the fix.
Which version in the 10.9 release is be expected to have the fix?
ArthurBorsboom, the fix of the duplicate key error has been merged up to 10.9, but as part of doing that, I forgot to apply some additional changes to 10.6 (see my previous comment). Those additional changes (which should prevent a crash if something else than what we fixed is causing the duplicate key error) have not been merged to 10.9 yet.
Please try the latest 10.9 branch. If it crashes, then MDEV-15237 was not fully fixed.
Summary: if InnoDB table had a fulltext index and only one new row was inserted between the last sync (performed asynchronously by a dedicated thread) and server shutdown, the fulltext index wasn't properly updated and became out of sync with the data. Before 10.6.9 it was a silent error, in 10.6.9 an assertion crashed the server when detecting the inconsistency.
Is validation/fixing of the database / indices necessary?
If so, will the fixed version do this automatically or is manual intervention needed?
If manual work is needed, what are the steps to find and fix the inconsistencies?
ArthurBorsboom, this type of duplicate key error will lead to all FULLTEXT INDEX of the affected table to ‘freeze’, that is, no further updates to the indexes will be made, or the fulltext indexes will not reflect the latest changes to the table. That can be fixed by rebuilding the table (OPTIMIZE TABLE).
I am hereby confirming that upgrading mariadb from 10.9.2 > 10.9.3 solves the problem for me.
Similar queries break ib_logfile* files now. No problems after downgrading to version 10.6.8.