Details
-
Bug
-
Status: Needs Feedback (View Workflow)
-
Major
-
Resolution: Unresolved
-
10.11.10
-
None
Description
After upgrading from 10.5 to 10.11 following the documentation, we have several issues with our MariaDB.
And yesterday we get a "segfault" on the primary node:
241216 21:58:48 [ERROR] mysqld got signal 11 ;
|
Sorry, we probably made a mistake, and this is a bug.
|
|
Your assistance in bug reporting will enable us to fix this for the next release. |
To report this bug, see https://mariadb.com/kb/en/reporting-bugs |
|
We will try our best to scrape up some info that will hopefully help
|
diagnose the problem, but since we have already crashed,
|
something is definitely wrong and this may fail.
|
|
Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646 |
key_buffer_size=0
|
read_buffer_size=4194304
|
max_used_connections=834
|
max_threads=1502
|
thread_count=735
|
It is possible that mysqld could use up to
|
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9268439 K bytes of memory
|
Hope that's ok; if not, decrease some variables in the equation. |
|
Thread pointer: 0x7eee1820ad68
|
Attempting backtrace. You can use the following information to find out |
where mysqld died. If you see no messages after this, something went
|
terribly wrong...
|
stack_bottom = 0x7ef25cc4a000 thread_stack 0x49000
|
2024-12-16 21:58:59 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown |
2024-12-16 21:58:59 201911 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 201911, query: select `sc_collection`.`gen_review_comment_count` as `commentCount`, `sc_collection`.`gen_review_like_count` as `likeCount`, `sc_collection`.`gen_review_like_positive_count` as `likePositiveCount`, `sc_collection`.`id`, `sc_collection`.`product_id` as `productId`, `sc_collection`.`rating`, `sc_collection`.`review_comment_privacy` as `commentPrivacy`, `sc_collection`.`review_content_autosave` as `bodyAutoSaved`, `sc_collection`.`review_content_autosave_source` as `bodySourceAutoSaved`, `sc_collection`.`review_content` as `bodyText`, `sc_collection`.`review_content_source` as `body`, `sc_collection`.`review_date_creation` as `dateCreation`, `sc_collection`.`review_date_last_update` as `dateLastUpdate`, `sc_collection`.`review_hits` as `hits`, `sc_collection`.`review_state` as `isPublished`, `sc_collection`.`review_title_autosave` as `titleAutoSaved`, `sc_collection`.`review_title` as `title`, `sc_collection`.`user_id` as `user`, `product`.` |
2024-12-16 21:58:59 198757 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 198757, query: select `c`.`id` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.`id` = `c`.`user_id` where `c`.`product_id` = 20643012 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c.id DESC limit 10 |
2024-12-16 21:58:59 86910 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 86910, query: select `user_id` from `sc_collection` where `rating` = 7 and `product_id` = 418945 and `rating` is not null order by `date_done` desc limit 30 |
2024-12-16 21:58:59 137086 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 137086, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8 |
2024-12-16 21:58:59 87648 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 87648, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8 |
Printing to addr2line failed
|
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x563027cfaffe] |
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5630277c0a75] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f25f032b140] |
|
|
/usr/sbin/mariadbd(_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x5630274ee158] |
/usr/sbin/mariadbd(_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x5630274ef1a3] |
/usr/sbin/mariadbd(_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x5630274f0fd6] |
/usr/sbin/mariadbd(+0x83b5d1)[0x56302754b5d1] |
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x47ef)[0x56302755b51f] |
/usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x56302755cca7] |
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x56302755f55d] |
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x143)[0x5630275613d3] |
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x56302768910d] |
/usr/sbin/mariadbd(handle_one_connection+0x5d)[0x56302768946d] |
/usr/sbin/mariadbd(+0xce9de2)[0x5630279f9de2] |
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f25f031fea7] |
2024-12-16 21:59:36 0 [Note] InnoDB: Memory pressure event freed 178062 pages
|
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f25eff1cdef] |
|
Trying to get some variables.
|
Some pointers may be invalid and cause the dump to abort.
|
Query (0x7eee182f5210): select `sc_list`.`id`, CASE sc_list.comment_privacy WHEN 0 |
THEN 'EVERYONE' WHEN 1 |
THEN 'SCOUTS' WHEN 2 |
THEN 'NOBODY' |
ELSE NULL
|
END AS commentPrivacy, `sc_list`.`date_creation` as `dateCreation`, `sc_list`.`date_last_action` as `dateLastEdit`, `sc_list`.`date_last_update` as `dateLastUpdate`, `sc_list`.`description`, `sc_list`.`gen_comment_count` as `commentCount`, `sc_list`.`gen_count` as `productCount`, `sc_list`.`gen_like_positive_count` as `likePositiveCount`, `sc_list`.`gen_type_id_related`, `sc_list`.`hits`, `sc_list`.`is_ordered`, `sc_list`.`is_public`, `sc_list`.`json`, `sc_list`.`label`, `sc_list`.`parent_id` as `parent`, CASE WHEN sc_list.gen_like_count >= sc_list.gen_like_positive_count
|
THEN sc_list.gen_like_count - sc_list.gen_like_positive_count
|
ELSE 0
|
END AS likeNegativeCount, `sc_list`.`slug`, `sc_list`.`subtype_id`, `sc_list`.`subtype_id_related`, `sc_list`.`user_id_creation` as `user_id`, CONCAT("https://media.senscritique.com/media/", m.filename, "/500/cover.", m.extension) AS first_product_backdrop from `sc_list` left join `sc_list_product` as `lp` on `lp`.`list_id` = `sc_list`.`id` and `lp`.`position` = 1 left join `sc_param_index` as `pi` on `pi`.`product_id_key` = `lp`.`product_id` and `pi`.`param_id` in (326, 368, 424, 425) and `pi`.`state` > -1 left join `sc_media` as `m` on `m`.`id` = `pi`.`media_id_value` where `sc_list`.`id` = 3356937 and `sc_list`.`state` = 1 and `sc_list`.`gen_state_user` > -1 limit 1 |
|
Connection ID (thread ID): 199450
|
Status: KILL_SERVER
|
|
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
|
|
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains |
information that should help you find out what is causing the crash. |
Writing a core file... |
Working directory at /data/mysql |
Resource Limits:
|
Limit Soft Limit Hard Limit Units
|
Max cpu time unlimited unlimited seconds |
Max file size unlimited unlimited bytes |
Max data size unlimited unlimited bytes
|
Max stack size 8388608 unlimited bytes
|
Max core file size 0 unlimited bytes |
Max resident set unlimited unlimited bytes |
Max processes 1031124 1031124 processes
|
Max open files 32768 32768 files |
Max locked memory 524288 524288 bytes
|
Max address space unlimited unlimited bytes
|
Max file locks unlimited unlimited locks |
Max pending signals 1031124 1031124 signals
|
Max msgqueue size 819200 819200 bytes
|
Max nice priority 0 0 |
Max realtime priority 0 0
|
Max realtime timeout unlimited unlimited us
|
Core pattern: core
|
|
Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30) |
|
2024-12-16 22:00:05 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid Y8nq1+0PFBI3t+kgYmZD2KMfQ4Q= as process 42254 |
2024-12-16 22:00:05 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead. |
2024-12-16 22:00:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
|
2024-12-16 22:00:05 0 [Note] InnoDB: Using transactional memory
|
2024-12-16 22:00:05 0 [Note] InnoDB: Number of transaction pools: 1
|
2024-12-16 22:00:05 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
|
2024-12-16 22:00:05 0 [Note] InnoDB: Using liburing
|
2024-12-16 22:00:05 0 [Note] InnoDB: Initializing buffer pool, total size = 200.000GiB, chunk size = 3.125GiB
|
2024-12-16 22:00:06 0 [Note] InnoDB: Initialized memory pressure event listener
|
2024-12-16 22:00:06 0 [Note] InnoDB: Completed initialization of buffer pool
|
2024-12-16 22:00:07 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes)
|
2024-12-16 22:00:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=214350961313938
|
2024-12-16 22:00:11 0 [Note] InnoDB: End of log at LSN=214351431883947
|
2024-12-16 22:00:11 0 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 4 row operations to undo |
2024-12-16 22:00:11 0 [Note] InnoDB: Trx id counter is 60405952135 |
2024-12-16 22:00:11 0 [Note] InnoDB: To recover: 367581 pages
|
2024-12-16 22:00:22 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.018825', position 399196458 |
2024-12-16 22:00:22 0 [Note] InnoDB: 128 rollback segments are active.
|
2024-12-16 22:00:22 0 [Note] InnoDB: Starting in background the rollback of recovered transactions |
2024-12-16 22:00:22 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" |
2024-12-16 22:00:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... |
2024-12-16 22:00:22 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. |
2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952132
|
2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952133
|
2024-12-16 22:00:22 0 [Note] InnoDB: log sequence number 214351431883947; transaction id 60405952138 |
2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952049
|
2024-12-16 22:00:22 0 [Note] InnoDB: Rollback of non-prepared transactions completed
|
2024-12-16 22:00:22 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool |
2024-12-16 22:00:22 0 [Note] Plugin 'FEEDBACK' is disabled. |
2024-12-16 22:00:22 0 [Warning] Plugin 'SPHINX' is of maturity level gamma while the server is stable |
2024-12-16 22:00:22 0 [ERROR] mariadbd: Can't open shared library '/usr/lib/mysql/plugin/tokudb_backup.so' (errno: 0, cannot open shared object file: No such file or directory) |
2024-12-16 22:00:22 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
2024-12-16 22:00:22 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
2024-12-16 22:00:22 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
2024-12-16 22:00:22 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files. |
2024-12-16 22:00:22 0 [Note] Recovering after a crash using /data/binlog/mysql-bin |
2024-12-16 22:00:22 0 [Note] Starting table crash recovery...
|
2024-12-16 22:00:22 0 [Note] Crash table recovery finished.
|
2024-12-16 22:00:23 0 [Note] Server socket created on IP: '0.0.0.0'. |
2024-12-16 22:00:23 0 [Warning] 'user' entry 'root@mysql-master' ignored in --skip-name-resolve mode. |
2024-12-16 22:00:23 0 [Warning] 'user' entry '@mysql-master' ignored in --skip-name-resolve mode. |
2024-12-16 22:00:23 0 [Warning] 'proxies_priv' entry '@% root@mysql-master' ignored in --skip-name-resolve mode. |
2024-12-16 22:00:23 0 [Note] /usr/sbin/mariadbd: ready for connections. |
Version: '10.11.10-MariaDB-deb11-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution |
2024-12-16 22:00:23 40 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES) |
2024-12-16 22:00:23 43 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES) |
You can find the our configuration in the attachments.
I share you also some screenshot of our monitoring tool (datadog) to show the difference between "before" the upgrade and "after" the upgrade (the upgrade have been performed december the 4th).
We don't have perform any major change on our application, and after the upgrade we get slow queries, huge lock struct, and so on, like there was a change on how mariadb response.
Regards,
Attachments
Issue Links
- relates to
-
MDEV-23256 SIGSEGV's in Query_cache::unlink_table, my_hash_delete, Query_cache::double_linked_list_exclude, Assertion `table_block_data->m_cached_query_count >= 0', InnoDB assertion: table->foreign_set.empty(), ASAN: heap-buffer-overflow in Query_cache_block::init
- Confirmed