[MDEV-20559] Crash and autorestarting MariaDB 10.4.7 server Created: 2019-09-11  Updated: 2022-02-03

Status: Open
Project: MariaDB Server
Component/s: Full-text Search, Storage Engine - InnoDB
Affects Version/s: 10.4.7
Fix Version/s: 10.4

Type: Bug Priority: Major
Reporter: Yury Assignee: Vladislav Lesin
Resolution: Unresolved Votes: 0
Labels: None
Environment:

Debian 9

Attachments: HTML File mysql    

 Description   

MariaDB 10.4.7 крашится, после чего перезапускается.

На 10.4.6 beta было так же. После нескольких таких падений это привело к поломке таблиц и восстановлению через innodb_force_recovery=5 с полным восстановлением всех таблиц из дампа. После этого обновились до MariaDB 10.4.7 в надежде что в релизной версии такого не будет.

Sep 11 12:42:51 host_master mysqld[2609]: 2019-09-11 12:42:51 0x7f0e64663700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.7/storage/innobase/row/row0mysql.cc line 188
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: Failing assertion: len < 256
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: We intentionally generate a memory trap.
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: If you get repeated assertion failures or crashes, even
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: immediately after the mysqld startup, there may be
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: corruption in the InnoDB tablespace. Please refer to
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
Sep 11 12:42:51 host_master mysqld[2609]: InnoDB: about forcing recovery.
 
Sep 11 12:42:51 host_master mysqld[2609]: %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%190911 12:42:51 [ERROR] mysqld got signal 6 ;
 
Sep 11 12:42:51 host_master mysqld[2609]: This could be because you hit a bug. It is also possible that this binary
 
Sep 11 12:42:51 host_master mysqld[2609]: or one of the libraries it was linked against is corrupt, improperly built,
 
Sep 11 12:42:51 host_master mysqld[2609]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Sep 11 12:42:51 host_master mysqld[2609]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Sep 11 12:42:51 host_master mysqld[2609]: We will try our best to scrape up some info that will hopefully help
 
Sep 11 12:42:51 host_master mysqld[2609]: diagnose the problem, but since we have already crashed,
 
Sep 11 12:42:51 host_master mysqld[2609]: something is definitely wrong and this may fail.
 
Sep 11 12:42:51 host_master mysqld[2609]: Server version: 10.4.7-MariaDB-1:10.4.7+maria~stretch-log
 
Sep 11 12:42:51 host_master mysqld[2609]: key_buffer_size=134217728
 
Sep 11 12:42:51 host_master mysqld[2609]: read_buffer_size=2097152
 
Sep 11 12:42:51 host_master mysqld[2609]: max_used_connections=248
 
Sep 11 12:42:51 host_master mysqld[2609]: max_threads=502
 
Sep 11 12:42:51 host_master mysqld[2609]: thread_count=239
 
Sep 11 12:42:51 host_master mysqld[2609]: It is possible that mysqld could use up to
 
Sep 11 12:42:51 host_master mysqld[2609]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3227677 K  bytes of memory
 
Sep 11 12:42:51 host_master mysqld[2609]: Hope that's ok; if not, decrease some variables in the equation.
 
Sep 11 12:42:51 host_master mysqld[2609]: Thread pointer: 0x7f0e60040f88
 
Sep 11 12:42:51 host_master mysqld[2609]: Attempting backtrace. You can use the following information to find out
 
Sep 11 12:42:51 host_master mysqld[2609]: where mysqld died. If you see no messages after this, something went
 
Sep 11 12:42:51 host_master mysqld[2609]: terribly wrong...
 
Sep 11 12:42:51 host_master mysqld[2609]: stack_bottom = 0x7f0e64662cf8 thread_stack 0x49000
 
Sep 11 12:42:52 host_master mysqld[2609]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55a4e15d3b8e]
 
Sep 11 12:42:52 host_master mysqld[2609]: /usr/sbin/mysqld(handle_fatal_signal+0x3af)[0x55a4e106379f]
 
Sep 11 12:42:55 host_master mysqld[2609]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0)[0x7f281e0be0e0]
 
Sep 11 12:42:57 host_master mysqld[2609]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7f281c75dfff]
 
Sep 11 12:42:57 host_master mysqld[2609]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f281c75f42a]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0x562cc8)[0x55a4e0d66cc8]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0xacce68)[0x55a4e12d0e68]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0xaebf98)[0x55a4e12eff98]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0x561ccd)[0x55a4e0d65ccd]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0x561f81)[0x55a4e0d65f81]
 
Sep 11 12:43:00 host_master mysqld[2609]: /usr/sbin/mysqld(+0xaed3cf)[0x55a4e12f13cf]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(+0xa091e2)[0x55a4e120d1e2]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(+0x6901aa)[0x55a4e0e941aa]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x120)[0x55a4e0e89cc0]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xbcb)[0x55a4e0eab42b]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55a4e0eab693]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x55a4e0ea9a66]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x55a4e0eaa3b6]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(+0x55351a)[0x55a4e0d5751a]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1fa0)[0x55a4e0e4f140]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x55a4e0e55efa]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x134b)[0x55a4e0e5840b]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x55a4e0e59cec]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x281)[0x55a4e0f36431]
 
Sep 11 12:43:01 host_master mysqld[2609]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55a4e0f3655d]
 
Sep 11 12:43:02 host_master mysqld[2609]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f281e0b44a4]
 
Sep 11 12:43:05 host_master mysqld[2609]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f281c813d0f]
 
Sep 11 12:43:05 host_master mysqld[2609]: Trying to get some variables.
 
Sep 11 12:43:05 host_master mysqld[2609]: Some pointers may be invalid and cause the dump to abort.
 
Sep 11 12:43:05 host_master mysqld[2609]: Query (0x7f0e60050e30): SELECT c.id,                         SUM(COALESCE(rcd.members_growth,0)) as members_growth,                         SUM(rcd.views_count) as views_count,                         SUM(rcd.views_growth) as views_growth,
 
c.ci_index AS ic_count,                         rank as rank,                         ROUND(IF ((c.views_per_post / c.participants_count) * 100 < 1000, IF (c.participants_count > 30, (c.views_per_post / c.participants_count) * 100, NULL), NULL), 1) as perc
 
FROM channels c                     LEFT JOIN rating_channels_daily rcd ON rcd.channel_id = c.id AND rcd.date >= '2019-09-05'                 WHERE  c.category_id = '8' AND MATCH(username
 
, title, hash, about) AGAINST ('*peri* peri*' IN BOOLEAN MODE) AND c.language IN ('english') AND c.country IN ('ru')                 GROUP BY c.id
 
ORDER BY participants_count asc                 LIMIT 100                 OFFSET 0
 
Sep 11 12:43:05 host_master mysqld[2609]: Connection ID (thread ID): 3961639
 
Sep 11 12:43:05 host_master mysqld[2609]: Status: NOT_KILLED
 
Sep 11 12:43:05 host_master mysqld[2609]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firs
 
tmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,j
 
oin_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Sep 11 12:43:05 host_master mysqld[2609]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Sep 11 12:43:05 host_master mysqld[2609]: information that should help you find out what is causing the crash.
 
Sep 11 12:43:05 host_master mysqld[2609]: Writing a core file...
 
Sep 11 12:43:05 host_master mysqld[2609]: Working directory at /var/lib/mysql
 
Sep 11 12:43:05 host_master mysqld[2609]: Resource Limits:
 
Sep 11 12:43:05 host_master mysqld[2609]: Limit                     Soft Limit           Hard Limit           Units
 
Sep 11 12:43:05 host_master mysqld[2609]: Max cpu time              unlimited            unlimited            seconds
 
Sep 11 12:43:05 host_master mysqld[2609]: Max file size             unlimited            unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max data size             unlimited            unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max stack size            8388608              unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max core file size        0                    unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max resident set          unlimited            unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max processes             514387               514387               processes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max open files            16364                16364                files
 
Sep 11 12:43:05 host_master mysqld[2609]: Max locked memory         65536                65536                bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max address space         unlimited            unlimited            bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max file locks            unlimited            unlimited            locks
 
Sep 11 12:43:05 host_master mysqld[2609]: Max pending signals       514387               514387               signals
 
Sep 11 12:43:05 host_master mysqld[2609]: Max msgqueue size         819200               819200               bytes
 
Sep 11 12:43:05 host_master mysqld[2609]: Max nice priority         0                    0
 
Sep 11 12:43:05 host_master mysqld[2609]: Max realtime priority     0                    0
 
Sep 11 12:43:05 host_master mysqld[2609]: Max realtime timeout      unlimited            unlimited            us
 
Sep 11 12:43:05 host_master mysqld[2609]: Core pattern: core
 
Sep 11 12:43:10 host_master systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
 
Sep 11 12:43:10 host_master systemd[1]: mariadb.service: Unit entered failed state.
 
Sep 11 12:43:10 host_master systemd[1]: mariadb.service: Failed with result 'signal'.
 
Sep 11 12:43:15 host_master systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
 
Sep 11 12:43:15 host_master systemd[1]: Stopped MariaDB 10.4.7 database server.
 
Sep 11 12:43:15 host_master systemd[1]: Starting MariaDB 10.4.7 database server...



 Comments   
Comment by Eugene Kosov (Inactive) [ 2019-09-11 ]

Здравствуйте. Пытаюсь понять, что у вас произошло. Для этого смотрю на stacktrace из лога. Там не везде есть названия функций. Я вижу в каком месте программа умерла, но не вижу как туда пришла. У вас, если креши были неоднократно, все stacktrace похожие? Хотя бы, всегда ли MariaDB умирает в точке len < 256?

Чтобы примерно прнять ваш stacktrace я могу посмотреть ассемблерный код. Но для этого мне нужен ровно ваш бинарник. Можете его куда-нибудь выложить? Или, если вы взяли его в каком-нибудь репозитории можете дать ссылку, по которой я смогу его скачать? Но учтите, что нужна строго ваша версия, а то адреса будут вообще в другие места показывать.

С предыдущего запуска MariaDB вы делали ALTER TABLE с табличками channels или rating_channels_daily? Вообще, часто делаете ALTER TABLE? Бывает ли так, что в InnoDB табличку только добавляете новые поля или только удалаете? Мне интересно, срабатывает ли у вас INSTANT ALTER TABLE.

Comment by Yury [ 2019-09-12 ]

Евгений, добрый день.

На mariadb 10.4.7 пока что было только одно падение.

Посмотрел сейчас логи, которые остались от предыдущей версии mariadb 10.4.6 beta
Они идентичные, падения в той же строке row0mysql.cc line 188

--
 
Sep  5 19:55:41 host_master mysqld[24939]: 2019-09-05 19:55:41 0x7fa290219700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.6/storage/innobase/row/row0mysql.cc line 188
 
Sep  5 19:55:41 host_master mysqld[24939]: InnoDB: Failing assertion: len < 256
 
Sep  5 19:55:41 host_master mysqld[24939]: InnoDB: We intentionally generate a memory trap.
 
--
 
Sep  6 00:14:18 host_master mysqld[26401]: 2019-09-06 00:14:18 0x7f8118b29700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.6/storage/innobase/row/row0mysql.cc line 188
 
Sep  6 00:14:18 host_master mysqld[26401]: InnoDB: Failing assertion: len < 256
 
Sep  6 00:14:20 host_master mysqld[26401]: InnoDB: We intentionally generate a memory trap.
 
--
 
Sep  6 00:14:31 host_master mysqld[26401]: 2019-09-06 00:14:31 0x7f815769f700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.6/storage/innobase/row/row0mysql.cc line 188
 
Sep  6 00:14:31 host_master mysqld[26401]: InnoDB: Failing assertion: len < 256
 
Sep  6 00:14:31 host_master mysqld[26401]: InnoDB: We intentionally generate a memory trap.

Падения происходят на одном и том же SELECT-запросе. Из экзотики - там FTS индексы на табличку channels следующей структуры:

  FULLTEXT KEY `channel_fulltext_idx` (`username`,`title`,`about`),
 
  FULLTEXT KEY `channel_username_title_ft_idx` (`username`,`title`),
 
  FULLTEXT KEY `channel_fulltext_idx3` (`username`,`title`,`hash`,`about`),
 
  FULLTEXT KEY `channel_fulltext_idx4` (`username`,`title`,`hash`)

В запросах, на которых происходили падения на 10.4.7 присутствуют конструкции вида:

AND MATCH(username, title, hash, about) AGAINST ('*        *         *' IN BOOLEAN MODE)
 
AND MATCH(username, title, hash, about) AGAINST ('*        *         *' IN BOOLEAN MODE)
 
AND MATCH(username, title, hash, about) AGAINST ('*          * Nauka*' IN BOOLEAN MODE)

В запросе на 10.4.7:

AND MATCH(username, title, hash, about) AGAINST ('*peri* peri*' IN BOOLEAN MODE)

2. MariaDB 10.4.7 устанавливалась отсюда https://prnt.sc/p52g6h

apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xF1656F24C74CD1D8
add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mirrors.ukfast.co.uk/sites/mariadb/repo/10.4/debian stretch main'

Бинарник: mysql

3. ALTER TABLE для этих таблиц не выполнялся очень давно, а те что были пару месяцев назад связаны с добавлением новым колонок и индексов.

Comment by Yury [ 2019-09-12 ]

Еще один краш произошел только что.
Что может помочь Марии не падать?

Пересоздание индекса или изменения каких-то параметров конфигурации может исправить ситуацию?
Не совсем понимаю, что именно означает len < 256 и как это можно подлечить не дожидаясь фикса с вашей стороны ?

Спасибо

Comment by Eugene Kosov (Inactive) [ 2019-09-12 ]

Добрый день. Я еще не восстановил stacktrace и пока не знаю что у вас происходит. Быстро это сделать не смогу. Соответственно, не знаю как это можно временно подлечить.

Если у вас все падает на одном и том же запросе, можете попробовать сделать минимальный тест для воспроизведения бага и поделиться им? Поудалять из табличек записи и столбцы, чтобы запрос все еще крешил демон? Это если у вас, конечно, данные не секретные. Или если получится все секретное оттуда убрать. В идеале было бы здорово чтобы получился один файлик с запросами на создание таблиц, на заполнение их и потом один смертельный селект. Назовите его, например, MDEV-20559.tgz и положите в ftp://ftp.mariadb.com/uploads

Я не смог с наскоку расшифровать ваш stacktrace. Это займет время. Потом я буду пытаться понять как это происходит, буду пытаться сделать тест, чтобы все падало там же где и у вас. Это все долго, сложно и, на самом деле, не всегда получается. А если вы поможете мне воспроизвести баг, это сильно ускорит фикс.

Вот в этой функции демон умирает. ut_ad() это проверка типа assert(), только она и в релизной версии присутствует. https://github.com/MariaDB/server/blob/9a78a283f4ee7e8ccc4afb1d8a24c662fa4c634b/storage/innobase/row/row0mysql.cc#L188

Comment by Eugene Kosov (Inactive) [ 2019-09-12 ]

Ну и, собственно, если у вас проблемный запрос именно один и тот же с точностью до байта, то можете в своем приложении ловить его и показывать пользователю ошибку. Это лучше, чем ронять сервер и показывать ошибку всем.

Comment by Vladimir Polischuk [ 2019-12-02 ]

Столкнулись с похожей ошибкой

Dec  2 09:43:40 prod-db-01 mysqld[1712]: 2019-12-02 09:43:40 0x7fc88cd73700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.10/storage/innobase/row/row0mysql.cc line 188
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: Failing assertion: len < 256
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: We intentionally generate a memory trap.
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: If you get repeated assertion failures or crashes, even
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: immediately after the mysqld startup, there may be
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: corruption in the InnoDB tablespace. Please refer to
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: InnoDB: about forcing recovery.
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: 191202  9:43:40 [ERROR] mysqld got signal 6 ;
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: This could be because you hit a bug. It is also possible that this binary
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: or one of the libraries it was linked against is corrupt, improperly built,
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: or misconfigured. This error can also be caused by malfunctioning hardware.
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: We will try our best to scrape up some info that will hopefully help
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: diagnose the problem, but since we have already crashed,
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: something is definitely wrong and this may fail.
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: Server version: 10.4.10-MariaDB-1:10.4.10+maria~bionic-log
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: key_buffer_size=805306368
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: read_buffer_size=262144
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: max_used_connections=401
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: max_threads=2502
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: thread_count=407
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: It is possible that mysqld could use up to
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 329430655 K  bytes of memory
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: Hope that's ok; if not, decrease some variables in the equation.
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: Thread pointer: 0x7fc62801a5e8
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: Attempting backtrace. You can use the following information to find out
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: where mysqld died. If you see no messages after this, something went
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: terribly wrong...
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: stack_bottom = 0x7fc88cd72da8 thread_stack 0x80000
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55d7cb19c13e]
 
Dec  2 09:43:40 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55d7cac105d5]
 
Dec  2 09:43:41 prod-db-01 mysqld[1712]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fcfc5ecd890]
 
Dec  2 09:43:42 prod-db-01 mysqld[1712]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fcfc4801e97]
 
Dec  2 09:43:42 prod-db-01 mysqld[1712]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fcfc4803801]
 
 
 
 
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0x56ab90)[0x55d7ca918b90]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0xad1078)[0x55d7cae7f078]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0xaf1a78)[0x55d7cae9fa78]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0x56a351)[0x55d7ca918351]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0x56a5fd)[0x55d7ca9185fd]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0xaf44d2)[0x55d7caea24d2]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0xa0db3d)[0x55d7cadbbb3d]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0x694aa2)[0x55d7caa42aa2]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x163)[0x55d7caa389c3]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa8b)[0x55d7caa59cbb]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x55d7caa5a063]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x131)[0x55d7caa58421]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x141)[0x55d7caa58dd1]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0x647a31)[0x55d7ca9f5a31]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x12ab)[0x55d7ca9fdd5b]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x22a)[0x55d7caa0558a]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x13b5)[0x55d7caa07b35]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z10do_commandP3THD+0x148)[0x55d7caa09458]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x25e)[0x55d7caae4fce]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(handle_one_connection+0x3d)[0x55d7caae508d]
 
Dec  2 09:43:43 prod-db-01 mysqld[1712]: /usr/sbin/mysqld(+0xd9919a)[0x55d7cb14719a]
 
Dec  2 09:43:44 prod-db-01 mysqld[1712]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fcfc5ec26db]
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fcfc48e488f]
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Trying to get some variables.
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Some pointers may be invalid and cause the dump to abort.
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Query (0x7fc628028ce0): SELECT COUNT(*) FROM `sales_order_grid` AS `main_table`  LEFT JOIN `mavencommerce_ordermanager_status` AS `mos` ON main_table.entity_id = mos.order_id WHERE (MATCH(`main_table`.increment_id,`main_table`.billing_name,`main_table`.shipping_name,`main_table`.shipping_address,`main_table`.billing_address,`main_table`.customer_name,`main_table`.customer_email) AGAINST('0171482')) AND (`main_table`.`status` = 'holded') AND (`main_table`.`status` = 'holded') AND (`main_table`.`status` = 'holded')
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Connection ID (thread ID): 1305
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Status: NOT_KILLED
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: information that should help you find out what is causing the crash.
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Writing a core file...
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Working directory at /var/lib/mysql
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Resource Limits:
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Limit                     Soft Limit           Hard Limit           Units
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max cpu time              unlimited            unlimited            seconds
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max file size             unlimited            unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max data size             unlimited            unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max stack size            8388608              unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max core file size        0                    unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max resident set          unlimited            unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max processes             257576               257576               processes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max open files            64000                64000                files
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max locked memory         16777216             16777216             bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max address space         unlimited            unlimited            bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max file locks            unlimited            unlimited            locks
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max pending signals       257576               257576               signals
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max msgqueue size         819200               819200               bytes
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max nice priority         0                    0
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max realtime priority     0                    0
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Max realtime timeout      unlimited            unlimited            us
 
Dec  2 09:43:45 prod-db-01 mysqld[1712]: Core pattern: |/usr/share/apport/apport %p %s %c %d %P
 
Dec  2 09:43:47 prod-db-01 systemd[1]: mariadb.service: Main process exited, code=dumped, status=6/ABRT

версия марии:

mysqld  Ver 10.4.10-MariaDB-1:10.4.10+maria~bionic-log for debian-linux-gnu on x86_64 (mariadb.org binary distribution)

Собственно всё падает наследующем запросе 

SELECT COUNT(*) FROM `sales_order_grid` AS `main_table`  LEFT JOIN `mavencommerce_ordermanager_status` AS `mos` ON main_table.entity_id = mos.order_id WHERE (MATCH(`main_table`.increment_id,`main_table`.billing_name,`main_table`.shipping_name,`main_table`.shipping_address,`main_table`.billing_address,`main_table`.customer_name,`main_table`.customer_email) AGAINST('0171482')) AND (`main_table`.`status` = 'holded') AND (`main_table`.`status` = 'holded') AND (`main_table`.`status` = 'holded')

Перепроверил, дейсвтительно при выполнение этого запроса падает.
Подозреваю что проблема вылазит изза AGAINST который выполняется на инно дб таблице

Собственно ушёл искать в куче легаси эту штуку.
Если чем то могу помочь ещё.

Структура таблицы 

       Table: sales_order_grid
 
Create Table: CREATE TABLE `sales_order_grid` (
 
  `entity_id` int(10) unsigned NOT NULL COMMENT 'Entity Id',
 
  `status` varchar(32) DEFAULT NULL COMMENT 'Status',
 
  `store_id` smallint(5) unsigned DEFAULT NULL COMMENT 'Store Id',
 
  `store_name` varchar(255) DEFAULT NULL COMMENT 'Store Name',
 
  `customer_id` int(10) unsigned DEFAULT NULL COMMENT 'Customer Id',
 
  `base_grand_total` decimal(12,4) DEFAULT NULL COMMENT 'Base Grand Total',
 
  `base_total_paid` decimal(12,4) DEFAULT NULL COMMENT 'Base Total Paid',
 
  `grand_total` decimal(12,4) DEFAULT NULL COMMENT 'Grand Total',
 
  `total_paid` decimal(12,4) DEFAULT NULL COMMENT 'Total Paid',
 
  `increment_id` varchar(50) DEFAULT NULL COMMENT 'Increment Id',
 
  `base_currency_code` varchar(3) DEFAULT NULL COMMENT 'Base Currency Code',
 
  `order_currency_code` varchar(255) DEFAULT NULL COMMENT 'Order Currency Code',
 
  `shipping_name` varchar(255) DEFAULT NULL COMMENT 'Shipping Name',
 
  `billing_name` varchar(255) DEFAULT NULL COMMENT 'Billing Name',
 
  `created_at` timestamp NULL DEFAULT NULL COMMENT 'Created At',
 
  `updated_at` timestamp NULL DEFAULT NULL COMMENT 'Updated At',
 
  `billing_address` varchar(255) DEFAULT NULL COMMENT 'Billing Address',
 
  `shipping_address` varchar(255) DEFAULT NULL COMMENT 'Shipping Address',
 
  `shipping_information` varchar(255) DEFAULT NULL COMMENT 'Shipping Method Name',
 
  `customer_email` varchar(255) DEFAULT NULL COMMENT 'Customer Email',
 
  `customer_group` varchar(255) DEFAULT NULL COMMENT 'Customer Group',
 
  `subtotal` decimal(12,4) DEFAULT NULL COMMENT 'Subtotal',
 
  `shipping_and_handling` decimal(12,4) DEFAULT NULL COMMENT 'Shipping and handling amount',
 
  `customer_name` varchar(255) DEFAULT NULL COMMENT 'Customer Name',
 
  `payment_method` varchar(255) DEFAULT NULL COMMENT 'Payment Method',
 
  `total_refunded` decimal(12,4) DEFAULT NULL COMMENT 'Total Refunded',
 
  `Attr113` varchar(255) DEFAULT NULL COMMENT 'Delivery Date Info',
 
  PRIMARY KEY (`entity_id`),
 
  UNIQUE KEY `SALES_ORDER_GRID_INCREMENT_ID_STORE_ID` (`increment_id`,`store_id`),
 
  KEY `SALES_ORDER_GRID_STATUS` (`status`),
 
  KEY `SALES_ORDER_GRID_STORE_ID` (`store_id`),
 
  KEY `SALES_ORDER_GRID_BASE_GRAND_TOTAL` (`base_grand_total`),
 
  KEY `SALES_ORDER_GRID_BASE_TOTAL_PAID` (`base_total_paid`),
 
  KEY `SALES_ORDER_GRID_GRAND_TOTAL` (`grand_total`),
 
  KEY `SALES_ORDER_GRID_TOTAL_PAID` (`total_paid`),
 
  KEY `SALES_ORDER_GRID_SHIPPING_NAME` (`shipping_name`),
 
  KEY `SALES_ORDER_GRID_BILLING_NAME` (`billing_name`),
 
  KEY `SALES_ORDER_GRID_CREATED_AT` (`created_at`),
 
  KEY `SALES_ORDER_GRID_CUSTOMER_ID` (`customer_id`),
 
  KEY `SALES_ORDER_GRID_UPDATED_AT` (`updated_at`),
 
  FULLTEXT KEY `FTI_65B9E9925EC58F0C7C2E2F6379C233E7` (`increment_id`,`billing_name`,`shipping_name`,`shipping_address`,`billing_address`,`customer_name`,`customer_email`)
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Sales Flat Order Grid'

Generated at Thu Feb 08 09:00:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.