Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35671

MariaDB got signal 11 after upgrading

Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.10
    • None

    Description

      After upgrading from 10.5 to 10.11 following the documentation, we have several issues with our MariaDB.

      And yesterday we get a "segfault" on the primary node:

      241216 21:58:48 [ERROR] mysqld got signal 11 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
      key_buffer_size=0
      read_buffer_size=4194304
      max_used_connections=834
      max_threads=1502
      thread_count=735
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9268439 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7eee1820ad68
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7ef25cc4a000 thread_stack 0x49000
      2024-12-16 21:58:59 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      2024-12-16 21:58:59 201911 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 201911, query: select `sc_collection`.`gen_review_comment_count` as `commentCount`, `sc_collection`.`gen_review_like_count` as `likeCount`, `sc_collection`.`gen_review_like_positive_count` as `likePositiveCount`, `sc_collection`.`id`, `sc_collection`.`product_id` as `productId`, `sc_collection`.`rating`, `sc_collection`.`review_comment_privacy` as `commentPrivacy`, `sc_collection`.`review_content_autosave` as `bodyAutoSaved`, `sc_collection`.`review_content_autosave_source` as `bodySourceAutoSaved`, `sc_collection`.`review_content` as `bodyText`, `sc_collection`.`review_content_source` as `body`, `sc_collection`.`review_date_creation` as `dateCreation`, `sc_collection`.`review_date_last_update` as `dateLastUpdate`, `sc_collection`.`review_hits` as `hits`, `sc_collection`.`review_state` as `isPublished`, `sc_collection`.`review_title_autosave` as `titleAutoSaved`, `sc_collection`.`review_title` as `title`, `sc_collection`.`user_id` as `user`, `product`.`
      2024-12-16 21:58:59 198757 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 198757, query: select `c`.`id` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.`id` = `c`.`user_id` where `c`.`product_id` = 20643012 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c.id DESC limit 10
      2024-12-16 21:58:59 86910 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 86910, query: select `user_id` from `sc_collection` where `rating` = 7 and `product_id` = 418945 and `rating` is not null order by `date_done` desc limit 30
      2024-12-16 21:58:59 137086 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 137086, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8
      2024-12-16 21:58:59 87648 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 87648, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8
      Printing to addr2line failed
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x563027cfaffe]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5630277c0a75]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f25f032b140]
       
       
      /usr/sbin/mariadbd(_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x5630274ee158]
      /usr/sbin/mariadbd(_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x5630274ef1a3]
      /usr/sbin/mariadbd(_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x5630274f0fd6]
      /usr/sbin/mariadbd(+0x83b5d1)[0x56302754b5d1]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x47ef)[0x56302755b51f]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x56302755cca7]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x56302755f55d]
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x143)[0x5630275613d3]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x56302768910d]
      /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x56302768946d]
      /usr/sbin/mariadbd(+0xce9de2)[0x5630279f9de2]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f25f031fea7]
      2024-12-16 21:59:36 0 [Note] InnoDB: Memory pressure event freed 178062 pages
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f25eff1cdef]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7eee182f5210): select `sc_list`.`id`, CASE sc_list.comment_privacy WHEN 0
                THEN 'EVERYONE' WHEN 1
                THEN 'SCOUTS' WHEN 2
                THEN 'NOBODY'
                ELSE NULL
              END AS commentPrivacy, `sc_list`.`date_creation` as `dateCreation`, `sc_list`.`date_last_action` as `dateLastEdit`, `sc_list`.`date_last_update` as `dateLastUpdate`, `sc_list`.`description`, `sc_list`.`gen_comment_count` as `commentCount`, `sc_list`.`gen_count` as `productCount`, `sc_list`.`gen_like_positive_count` as `likePositiveCount`, `sc_list`.`gen_type_id_related`, `sc_list`.`hits`, `sc_list`.`is_ordered`, `sc_list`.`is_public`, `sc_list`.`json`, `sc_list`.`label`, `sc_list`.`parent_id` as `parent`, CASE WHEN sc_list.gen_like_count >= sc_list.gen_like_positive_count
                THEN sc_list.gen_like_count - sc_list.gen_like_positive_count
                ELSE 0
              END AS likeNegativeCount, `sc_list`.`slug`, `sc_list`.`subtype_id`, `sc_list`.`subtype_id_related`, `sc_list`.`user_id_creation` as `user_id`, CONCAT("https://media.senscritique.com/media/", m.filename, "/500/cover.", m.extension) AS first_product_backdrop from `sc_list` left join `sc_list_product` as `lp` on `lp`.`list_id` = `sc_list`.`id` and `lp`.`position` = 1 left join `sc_param_index` as `pi` on `pi`.`product_id_key` = `lp`.`product_id` and `pi`.`param_id` in (326, 368, 424, 425) and `pi`.`state` > -1 left join `sc_media` as `m` on `m`.`id` = `pi`.`media_id_value` where `sc_list`.`id` = 3356937 and `sc_list`.`state` = 1 and `sc_list`.`gen_state_user` > -1 limit 1
       
      Connection ID (thread ID): 199450
      Status: KILL_SERVER
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /data/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             1031124              1031124              processes 
      Max open files            32768                32768                files     
      Max locked memory         524288               524288               bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       1031124              1031124              signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
       
      2024-12-16 22:00:05 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid Y8nq1+0PFBI3t+kgYmZD2KMfQ4Q= as process 42254
      2024-12-16 22:00:05 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
      2024-12-16 22:00:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2024-12-16 22:00:05 0 [Note] InnoDB: Using transactional memory
      2024-12-16 22:00:05 0 [Note] InnoDB: Number of transaction pools: 1
      2024-12-16 22:00:05 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2024-12-16 22:00:05 0 [Note] InnoDB: Using liburing
      2024-12-16 22:00:05 0 [Note] InnoDB: Initializing buffer pool, total size = 200.000GiB, chunk size = 3.125GiB
      2024-12-16 22:00:06 0 [Note] InnoDB: Initialized memory pressure event listener
      2024-12-16 22:00:06 0 [Note] InnoDB: Completed initialization of buffer pool
      2024-12-16 22:00:07 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes)
      2024-12-16 22:00:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=214350961313938
      2024-12-16 22:00:11 0 [Note] InnoDB: End of log at LSN=214351431883947
      2024-12-16 22:00:11 0 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 4 row operations to undo
      2024-12-16 22:00:11 0 [Note] InnoDB: Trx id counter is 60405952135
      2024-12-16 22:00:11 0 [Note] InnoDB: To recover: 367581 pages
      2024-12-16 22:00:22 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.018825', position 399196458
      2024-12-16 22:00:22 0 [Note] InnoDB: 128 rollback segments are active.
      2024-12-16 22:00:22 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      2024-12-16 22:00:22 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
      2024-12-16 22:00:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      2024-12-16 22:00:22 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952132
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952133
      2024-12-16 22:00:22 0 [Note] InnoDB: log sequence number 214351431883947; transaction id 60405952138
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952049
      2024-12-16 22:00:22 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      2024-12-16 22:00:22 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool
      2024-12-16 22:00:22 0 [Note] Plugin 'FEEDBACK' is disabled.
      2024-12-16 22:00:22 0 [Warning] Plugin 'SPHINX' is of maturity level gamma while the server is stable
      2024-12-16 22:00:22 0 [ERROR] mariadbd: Can't open shared library '/usr/lib/mysql/plugin/tokudb_backup.so' (errno: 0, cannot open shared object file: No such file or directory)
      2024-12-16 22:00:22 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Note] Recovering after a crash using /data/binlog/mysql-bin
      2024-12-16 22:00:22 0 [Note] Starting table crash recovery...
      2024-12-16 22:00:22 0 [Note] Crash table recovery finished.
      2024-12-16 22:00:23 0 [Note] Server socket created on IP: '0.0.0.0'.
      2024-12-16 22:00:23 0 [Warning] 'user' entry 'root@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Warning] 'user' entry '@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Warning] 'proxies_priv' entry '@% root@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Note] /usr/sbin/mariadbd: ready for connections.
      Version: '10.11.10-MariaDB-deb11-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
      2024-12-16 22:00:23 40 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)
      2024-12-16 22:00:23 43 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)
      

      You can find the our configuration in the attachments.

      I share you also some screenshot of our monitoring tool (datadog) to show the difference between "before" the upgrade and "after" the upgrade (the upgrade have been performed december the 4th).

      We don't have perform any major change on our application, and after the upgrade we get slow queries, huge lock struct, and so on, like there was a change on how mariadb response.

      Regards,

      Attachments

        1. metrics_mysql.png
          metrics_mysql.png
          274 kB
        2. my-slave.cnf
          6 kB
        3. sql-master.cnf
          6 kB

        Issue Links

          Activity

            Hello Team,

            I have the same issue on my replica this morning. Here the stack trace:

            2024-12-18  9:10:00 0 [Note] InnoDB: Memory pressure event freed 14831856 pages
            241218  9:10:00 [ERROR] mysqld got signal 11 ;
            Sorry, we probably made a mistake, and this is a bug.
             
            Your assistance in bug reporting will enable us to fix this for the next release.
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed, 
            something is definitely wrong and this may fail.
             
            Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
            key_buffer_size=0
            read_buffer_size=4194304
            max_used_connections=608
            max_threads=1502
            thread_count=616
            It is possible that mysqld could use up to 
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7730391 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7fad24064998
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7fad2e3e0000 thread_stack 0x49000
            2024-12-18  9:10:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
            /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x55a2db6bbffe]
            /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x55a2db181a75]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7fe2f5b34140]
            /usr/sbin/mariadbd(_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x55a2daeaf158]
            /usr/sbin/mariadbd(_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x55a2daeb01a3]
            /usr/sbin/mariadbd(_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x55a2daeb1fd6]
            /usr/sbin/mariadbd(+0x83b5d1)[0x55a2daf0c5d1]
            /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x47ef)[0x55a2daf1c51f]
            /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x55a2daf1dca7]
            /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x55a2daf2055d]
            /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x143)[0x55a2daf223d3]
            /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x55a2db04a10d]
            /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x55a2db04a46d]
            /usr/sbin/mariadbd(+0xce9de2)[0x55a2db3bade2]
            2024-12-18  9:10:35 0 [Note] InnoDB: Memory pressure event freed 15893 pages
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7fe2f5b28ea7]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fe2f5715a2f]
             
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7fad24073530): select count(distinct `p`.`id`) as `count` from `sc_param_index` as `ParametersIndices` left join `sc_gen_product_fr_FR` as `p` on `ParametersIndices`.`product_id_key` = `p`.`id` left join `sc_param` as `Parameters` on `ParametersIndices`.`param_id` = `Parameters`.`id` where `ParametersIndices`.`contact_id_value` = 284031 and `ParametersIndices`.`state` >= 0 and `p`.`id` is not null and `p`.`subtype_id` not in (5, 32) and `p`.`subtype_id` = 36
             
            Connection ID (thread ID): 486967
            Status: KILL_SERVER
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
             
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /data/mysql
            Resource Limits:
            Limit                     Soft Limit           Hard Limit           Units     
            Max cpu time              unlimited            unlimited            seconds   
            Max file size             unlimited            unlimited            bytes     
            Max data size             unlimited            unlimited            bytes     
            Max stack size            8388608              unlimited            bytes     
            Max core file size        0                    unlimited            bytes     
            Max resident set          unlimited            unlimited            bytes     
            Max processes             1031142              1031142              processes 
            Max open files            32768                32768                files     
            Max locked memory         524288               524288               bytes     
            Max address space         unlimited            unlimited            bytes     
            Max file locks            unlimited            unlimited            locks     
            Max pending signals       1031142              1031142              signals   
            Max msgqueue size         819200               819200               bytes     
            Max nice priority         0                    0                    
            Max realtime priority     0                    0                    
            Max realtime timeout      unlimited            unlimited            us        
            Core pattern: core
             
            Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
             
            2024-12-18  9:11:07 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid wSrlp7YFoHJH/pOvshJeZmFnvrc= as process 2870069
            2024-12-18  9:11:07 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
            2024-12-18  9:11:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            2024-12-18  9:11:07 0 [Note] InnoDB: Using transactional memory
            2024-12-18  9:11:07 0 [Note] InnoDB: Number of transaction pools: 1
            2024-12-18  9:11:07 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
            2024-12-18  9:11:07 0 [Note] InnoDB: Using liburing
            2024-12-18  9:11:07 0 [Note] InnoDB: Initializing buffer pool, total size = 200.000GiB, chunk size = 3.125GiB
            2024-12-18  9:11:08 0 [Note] InnoDB: Initialized memory pressure event listener
            2024-12-18  9:11:08 0 [Note] InnoDB: Completed initialization of buffer pool
            2024-12-18  9:11:09 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes)
            2024-12-18  9:11:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=176057303566279
            2024-12-18  9:11:12 0 [Note] InnoDB: End of log at LSN=176057738960032
            2024-12-18  9:11:13 0 [Note] InnoDB: To recover: 244840 pages
            2024-12-18  9:11:24 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.004696', position 378101349
            2024-12-18  9:11:24 0 [Note] InnoDB: 128 rollback segments are active.
            2024-12-18  9:11:24 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
            2024-12-18  9:11:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
            2024-12-18  9:11:24 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
            2024-12-18  9:11:24 0 [Note] InnoDB: log sequence number 176057738960032; transaction id 58559332764
            2024-12-18  9:11:24 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool
            2024-12-18  9:11:24 0 [Note] Plugin 'FEEDBACK' is disabled.
            2024-12-18  9:11:24 0 [Warning] Plugin 'SPHINX' is of maturity level gamma while the server is stable
            2024-12-18  9:11:24 0 [ERROR] mariadbd: Can't open shared library '/usr/lib/mysql/plugin/tokudb_backup.so' (errno: 0, cannot open shared object file: No such file or directory)
            2024-12-18  9:11:24 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2024-12-18  9:11:24 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2024-12-18  9:11:24 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2024-12-18  9:11:24 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2024-12-18  9:11:24 0 [Note] Recovering after a crash using /data/binlog/mysql-bin
            2024-12-18  9:11:24 0 [Note] Starting table crash recovery...
            2024-12-18  9:11:24 0 [Note] Crash table recovery finished.
            2024-12-18  9:11:24 0 [Note] Server socket created on IP: '0.0.0.0'.
            2024-12-18  9:11:24 0 [Warning] 'user' entry 'root@mysql-master' ignored in --skip-name-resolve mode.
            2024-12-18  9:11:24 0 [Warning] 'user' entry '@mysql-master' ignored in --skip-name-resolve mode.
            2024-12-18  9:11:24 0 [Warning] 'proxies_priv' entry '@% root@mysql-master' ignored in --skip-name-resolve mode.
            2024-12-18  9:11:24 5 [Note] Slave I/O thread: Start asynchronous replication to master 'repl@100.121.142.60:3306' in log 'mysql-bin.018866' at position 473995758
            2024-12-18  9:11:24 0 [Note] /usr/sbin/mariadbd: ready for connections.
            Version: '10.11.10-MariaDB-deb11-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
            2024-12-18  9:11:24 6 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.018866' at position 432400629, relay log './slave2-relay-bin.028514' position: 332293053
            

            Kriko Pierre Krikorian added a comment - Hello Team, I have the same issue on my replica this morning. Here the stack trace: 2024-12-18 9:10:00 0 [Note] InnoDB: Memory pressure event freed 14831856 pages 241218 9:10:00 [ERROR] mysqld got signal 11 ; Sorry, we probably made a mistake, and this is a bug.   Your assistance in bug reporting will enable us to fix this for the next release. To report this bug, see https: //mariadb .com /kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646 key_buffer_size=0 read_buffer_size=4194304 max_used_connections=608 max_threads=1502 thread_count=616 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7730391 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7fad24064998 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fad2e3e0000 thread_stack 0x49000 2024-12-18 9:10:07 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown /usr/sbin/mariadbd (my_print_stacktrace+0x2e)[0x55a2db6bbffe] /usr/sbin/mariadbd (handle_fatal_signal+0x485)[0x55a2db181a75] /lib/x86_64-linux-gnu/libpthread .so.0(+0x13140)[0x7fe2f5b34140] /usr/sbin/mariadbd (_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x55a2daeaf158] /usr/sbin/mariadbd (_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x55a2daeb01a3] /usr/sbin/mariadbd (_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x55a2daeb1fd6] /usr/sbin/mariadbd (+0x83b5d1)[0x55a2daf0c5d1] /usr/sbin/mariadbd (_Z21mysql_execute_commandP3THDb+0x47ef)[0x55a2daf1c51f] /usr/sbin/mariadbd (_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x55a2daf1dca7] /usr/sbin/mariadbd (_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x55a2daf2055d] /usr/sbin/mariadbd (_Z10do_commandP3THDb+0x143)[0x55a2daf223d3] /usr/sbin/mariadbd (_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x55a2db04a10d] /usr/sbin/mariadbd (handle_one_connection+0x5d)[0x55a2db04a46d] /usr/sbin/mariadbd (+0xce9de2)[0x55a2db3bade2] 2024-12-18 9:10:35 0 [Note] InnoDB: Memory pressure event freed 15893 pages /lib/x86_64-linux-gnu/libpthread .so.0(+0x7ea7)[0x7fe2f5b28ea7] /lib/x86_64-linux-gnu/libc .so.6(clone+0x3f)[0x7fe2f5715a2f]   Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fad24073530): select count(distinct `p`.` id `) as `count` from `sc_param_index` as `ParametersIndices` left join `sc_gen_product_fr_FR` as `p` on `ParametersIndices`.`product_id_key` = `p`.` id ` left join `sc_param` as `Parameters` on `ParametersIndices`.`param_id` = `Parameters`.` id ` where `ParametersIndices`.`contact_id_value` = 284031 and `ParametersIndices`.`state` >= 0 and `p`.` id ` is not null and `p`.`subtype_id` not in (5, 32) and `p`.`subtype_id` = 36   Connection ID (thread ID): 486967 Status: KILL_SERVER   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off   The manual page at https: //mariadb .com /kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains information that should help you find out what is causing the crash. Writing a core file ... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1031142 1031142 processes Max open files 32768 32768 files Max locked memory 524288 524288 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1031142 1031142 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core   Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)   2024-12-18 9:11:07 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid wSrlp7YFoHJH /pOvshJeZmFnvrc = as process 2870069 2024-12-18 9:11:07 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead. 2024-12-18 9:11:07 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2024-12-18 9:11:07 0 [Note] InnoDB: Using transactional memory 2024-12-18 9:11:07 0 [Note] InnoDB: Number of transaction pools: 1 2024-12-18 9:11:07 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2024-12-18 9:11:07 0 [Note] InnoDB: Using liburing 2024-12-18 9:11:07 0 [Note] InnoDB: Initializing buffer pool, total size = 200.000GiB, chunk size = 3.125GiB 2024-12-18 9:11:08 0 [Note] InnoDB: Initialized memory pressure event listener 2024-12-18 9:11:08 0 [Note] InnoDB: Completed initialization of buffer pool 2024-12-18 9:11:09 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes) 2024-12-18 9:11:09 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=176057303566279 2024-12-18 9:11:12 0 [Note] InnoDB: End of log at LSN=176057738960032 2024-12-18 9:11:13 0 [Note] InnoDB: To recover: 244840 pages 2024-12-18 9:11:24 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.004696' , position 378101349 2024-12-18 9:11:24 0 [Note] InnoDB: 128 rollback segments are active. 2024-12-18 9:11:24 0 [Note] InnoDB: Removed temporary tablespace data file : "./ibtmp1" 2024-12-18 9:11:24 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... 2024-12-18 9:11:24 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. 2024-12-18 9:11:24 0 [Note] InnoDB: log sequence number 176057738960032; transaction id 58559332764 2024-12-18 9:11:24 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool 2024-12-18 9:11:24 0 [Note] Plugin 'FEEDBACK' is disabled. 2024-12-18 9:11:24 0 [Warning] Plugin 'SPHINX' is of maturity level gamma while the server is stable 2024-12-18 9:11:24 0 [ERROR] mariadbd: Can 't open shared library ' /usr/lib/mysql/plugin/tokudb_backup .so' (errno: 0, cannot open shared object file : No such file or directory) 2024-12-18 9:11:24 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2024-12-18 9:11:24 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2024-12-18 9:11:24 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2024-12-18 9:11:24 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2024-12-18 9:11:24 0 [Note] Recovering after a crash using /data/binlog/mysql-bin 2024-12-18 9:11:24 0 [Note] Starting table crash recovery... 2024-12-18 9:11:24 0 [Note] Crash table recovery finished. 2024-12-18 9:11:24 0 [Note] Server socket created on IP: '0.0.0.0' . 2024-12-18 9:11:24 0 [Warning] 'user' entry 'root@mysql-master' ignored in --skip-name-resolve mode. 2024-12-18 9:11:24 0 [Warning] 'user' entry '@mysql-master' ignored in --skip-name-resolve mode. 2024-12-18 9:11:24 0 [Warning] 'proxies_priv' entry '@% root@mysql-master' ignored in --skip-name-resolve mode. 2024-12-18 9:11:24 5 [Note] Slave I /O thread: Start asynchronous replication to master 'repl@100.121.142.60:3306' in log 'mysql-bin.018866' at position 473995758 2024-12-18 9:11:24 0 [Note] /usr/sbin/mariadbd : ready for connections. Version: '10.11.10-MariaDB-deb11-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2024-12-18 9:11:24 6 [Note] Slave SQL thread initialized, starting replication in log 'mysql-bin.018866' at position 432400629, relay log './slave2-relay-bin.028514' position: 332293053

            Is there any way we can repeat this?
            Or, at least, can you try to get a core dump from the crash?

            serg Sergei Golubchik added a comment - Is there any way we can repeat this? Or, at least, can you try to get a core dump from the crash?

            Hello @Sergei.

            We got some bad issues with our mariadb database.

            We're currently restoring our servers with the same dataset in the same version.

            Once the restore is over, we will try again the mariabackup and tell you if it's still not working.

            Kriko Pierre Krikorian added a comment - Hello @Sergei. We got some bad issues with our mariadb database. We're currently restoring our servers with the same dataset in the same version. Once the restore is over, we will try again the mariabackup and tell you if it's still not working.

            Hello Sergei,

            We just restart our production in 10.11 and still the same config file, and we got this error:

            2025-01-07 10:41:21 0 [Note] InnoDB: Memory pressure event freed 4109074 pages
            250107 10:41:23 [ERROR] mysqld got signal 11 ;
            Sorry, we probably made a mistake, and this is a bug.
             
            Your assistance in bug reporting will enable us to fix this for the next release.
            To report this bug, see https://mariadb.com/kb/en/reporting-bugs
             
            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.
             
            Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
            key_buffer_size=0
            read_buffer_size=2097152
            max_used_connections=1203
            max_threads=1502
            thread_count=1203
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6192343 K  bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.
             
            Thread pointer: 0x7f90cc113e88
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            stack_bottom = 0x7f90162ff000 thread_stack 0x49000
            /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x5634fc2ffffe]
            /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5634fbdc5a75]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x13140)[0x7fabb7db0140]
            2025-01-07 10:41:35 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
            2025-01-07 10:41:35 4232 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 4232, query: select `c`.`id` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.`id` = `c`.`user_id` where `c`.`product_id` = 404334 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c.id DESC limit 5 offset 1
            2025-01-07 10:41:35 3985 [Warning] Sort aborted, host: 100.121.142.31, user: prod, thread: 3985, query: select `sc_list`.`id`, `r`.`rank` from `sc_list` inner join `sc_list_product_rank` as `r` on `r`.`list_id` = `sc_list`.`id` where `r`.`product_id` = 133190 and `r`.`is_archive` = 0 and `sc_list`.`state` = 1 and `sc_list`.`is_public` = 1 and `sc_list`.`parent_id` is null and `sc_list`.`id` not in (1212377) order by r.rank ASC limit 11
            2025-01-07 10:41:35 2144 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 2144, query: select `c`.`id` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.`id` = `c`.`user_id` where `c`.`product_id` = 12707585 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c.id DESC limit 10
            /usr/sbin/mariadbd(_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x5634fbaf3158]
            /usr/sbin/mariadbd(_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x5634fbaf41a3]
            /usr/sbin/mariadbd(_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x5634fbaf5fd6]
            /usr/sbin/mariadbd(+0x83b5d1)[0x5634fbb505d1]
            /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x47ef)[0x5634fbb6051f]
            /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x5634fbb61ca7]
            /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x5634fbb6455d]
            /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x143)[0x5634fbb663d3]
            /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x5634fbc8e10d]
            /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x5634fbc8e46d]
            /usr/sbin/mariadbd(+0xce9de2)[0x5634fbffede2]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7fabb7da4ea7]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fabb7991a2f]
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7f90cc11dc90): select `sc_list`.`id`, CASE sc_list.comment_privacy WHEN 0
                      THEN 'EVERYONE' WHEN 1
                      THEN 'SCOUTS' WHEN 2
                      THEN 'NOBODY'
                      ELSE NULL
                    END AS commentPrivacy, `sc_list`.`date_creation` as `dateCreation`, `sc_list`.`date_last_action` as `dateLastEdit`, `sc_list`.`date_last_update` as `dateLastUpdate`, `sc_list`.`description`, `sc_list`.`gen_comment_count` as `commentCount`, `sc_list`.`gen_count` as `productCount`, `sc_list`.`gen_like_positive_count` as `likePositiveCount`, `sc_list`.`gen_type_id_related`, `sc_list`.`hits`, `sc_list`.`is_ordered`, `sc_list`.`is_public`, `sc_list`.`json`, `sc_list`.`label`, `sc_list`.`parent_id` as `parent`, CASE WHEN sc_list.gen_like_count >= sc_list.gen_like_positive_count
                      THEN sc_list.gen_like_count - sc_list.gen_like_positive_count
                      ELSE 0
                    END AS likeNegativeCount, `sc_list`.`slug`, `sc_list`.`subtype_id`, `sc_list`.`subtype_id_related`, `sc_list`.`user_id_creation` as `user_id`, CONCAT("https://media.senscritique.com/media/", m.filename, "/500/cover.", m.extension) AS first_product_backdrop from `sc_list` left join `sc_list_product` as `lp` on `lp`.`list_id` = `sc_list`.`id` and `lp`.`position` = 1 left join `sc_param_index` as `pi` on `pi`.`product_id_key` = `lp`.`product_id` and `pi`.`param_id` in (326, 368, 424, 425) and `pi`.`state` > -1 left join `sc_media` as `m` on `m`.`id` = `pi`.`media_id_value` where `sc_list`.`id` = 2913285 and `sc_list`.`state` = 1 and `sc_list`.`gen_state_user` > -1 limit 1
             
            Connection ID (thread ID): 4682
            Status: KILL_SERVER
             
            Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
             
            The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
            information that should help you find out what is causing the crash.
            Writing a core file...
            Working directory at /data/mysql
            Resource Limits:
            Limit                     Soft Limit           Hard Limit           Units
            Max cpu time              unlimited            unlimited            seconds
            Max file size             unlimited            unlimited            bytes
            Max data size             unlimited            unlimited            bytes
            Max stack size            8388608              unlimited            bytes
            Max core file size        0                    unlimited            bytes
            Max resident set          unlimited            unlimited            bytes
            Max processes             1031142              1031142              processes
            Max open files            32768                32768                files
            Max locked memory         524288               524288               bytes
            Max address space         unlimited            unlimited            bytes
            Max file locks            unlimited            unlimited            locks
            Max pending signals       1031142              1031142              signals
            Max msgqueue size         819200               819200               bytes
            Max nice priority         0                    0
            Max realtime priority     0                    0
            Max nice priority         0                    0
            Max realtime priority     0                    0
            Max realtime timeout      unlimited            unlimited            us
            Core pattern: core
             
            Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
             
            2025-01-07 10:41:59 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid wSrlp7YFoHJH/pOvshJeZmFnvrc= as process 3728649
            2025-01-07 10:41:59 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
            2025-01-07 10:41:59 0 [Note] mariadbd: Aria engine: starting recovery
            tables to flush: 2 1 0
             (0.0 seconds);
            2025-01-07 10:41:59 0 [Note] mariadbd: Aria engine: recovery done
            2025-01-07 10:41:59 0 [Note] Using unique option prefix 'innodb_flush_neighbor' is error-prone and can break in the future. Please use the full name 'innodb-flush-neighbors' instead.
            2025-01-07 10:41:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
            2025-01-07 10:41:59 0 [Note] InnoDB: Using transactional memory
            2025-01-07 10:41:59 0 [Note] InnoDB: Number of transaction pools: 1
            2025-01-07 10:41:59 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
            2025-01-07 10:41:59 0 [Note] InnoDB: Using liburing
            2025-01-07 10:41:59 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000GiB, chunk size = 1.562GiB
            2025-01-07 10:42:00 0 [Note] InnoDB: Initialized memory pressure event listener
            2025-01-07 10:42:00 0 [Note] InnoDB: Completed initialization of buffer pool
            2025-01-07 10:42:00 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes)
            2025-01-07 10:42:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=6106753213007
            2025-01-07 10:42:00 0 [Note] InnoDB: End of log at LSN=6106786058899
            2025-01-07 10:42:00 0 [Note] InnoDB: 93 transaction(s) which must be rolled back or cleaned up in total 93 row operations to undo
            2025-01-07 10:42:00 0 [Note] InnoDB: Trx id counter is 1891459
            2025-01-07 10:42:00 0 [Note] InnoDB: To recover: 16058 pages
            2025-01-07 10:42:01 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.000001', position 21457665
            2025-01-07 10:42:01 0 [Note] InnoDB: 128 rollback segments are active.
            2025-01-07 10:42:01 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
            2025-01-07 10:42:01 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
            2025-01-07 10:42:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
            2025-01-07 10:42:01 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891362
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891378
            2025-01-07 10:42:01 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2025-01-07 10:42:01 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2025-01-07 10:42:01 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2025-01-07 10:42:01 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2025-01-07 10:42:01 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
            2025-01-07 10:42:01 0 [Note] Recovering after a crash using /data/binlog/mysql-bin
            2025-01-07 10:42:01 0 [Note] Starting table crash recovery...
            2025-01-07 10:42:01 0 [Note] Crash table recovery finished.
            2025-01-07 10:42:01 0 [Note] Server socket created on IP: '0.0.0.0'.
            2025-01-07 10:42:01 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem.
            2025-01-07 10:42:01 0 [Note] /usr/sbin/mariadbd: ready for connections.
            Version: '10.11.10-MariaDB-deb11-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891410
            2025-01-07 10:42:01 5 [Warning] IP address '100.121.142.35' could not be resolved: Name or service not known
            2025-01-07 10:42:01 6 [Warning] IP address '100.121.142.34' could not be resolved: Name or service not known
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891450
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891386
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891418
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891402
            2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891338
            
            

            Kriko Pierre Krikorian added a comment - Hello Sergei, We just restart our production in 10.11 and still the same config file, and we got this error: 2025-01-07 10:41:21 0 [Note] InnoDB: Memory pressure event freed 4109074 pages 250107 10:41:23 [ERROR] mysqld got signal 11 ; Sorry, we probably made a mistake, and this is a bug.   Your assistance in bug reporting will enable us to fix this for the next release. To report this bug, see https: //mariadb .com /kb/en/reporting-bugs   We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.   Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646 key_buffer_size=0 read_buffer_size=2097152 max_used_connections=1203 max_threads=1502 thread_count=1203 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6192343 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.   Thread pointer: 0x7f90cc113e88 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f90162ff000 thread_stack 0x49000 /usr/sbin/mariadbd (my_print_stacktrace+0x2e)[0x5634fc2ffffe] /usr/sbin/mariadbd (handle_fatal_signal+0x485)[0x5634fbdc5a75] /lib/x86_64-linux-gnu/libpthread .so.0(+0x13140)[0x7fabb7db0140] 2025-01-07 10:41:35 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown 2025-01-07 10:41:35 4232 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 4232, query: select `c`.` id ` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.` id ` = `c`.`user_id` where `c`.`product_id` = 404334 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c. id DESC limit 5 offset 1 2025-01-07 10:41:35 3985 [Warning] Sort aborted, host: 100.121.142.31, user: prod, thread: 3985, query: select `sc_list`.` id `, `r`.`rank` from `sc_list` inner join `sc_list_product_rank` as `r` on `r`.`list_id` = `sc_list`.` id ` where `r`.`product_id` = 133190 and `r`.`is_archive` = 0 and `sc_list`.`state` = 1 and `sc_list`.`is_public` = 1 and `sc_list`.`parent_id` is null and `sc_list`.` id ` not in (1212377) order by r.rank ASC limit 11 2025-01-07 10:41:35 2144 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 2144, query: select `c`.` id ` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.` id ` = `c`.`user_id` where `c`.`product_id` = 12707585 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c. id DESC limit 10 /usr/sbin/mariadbd (_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x5634fbaf3158] /usr/sbin/mariadbd (_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x5634fbaf41a3] /usr/sbin/mariadbd (_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x5634fbaf5fd6] /usr/sbin/mariadbd (+0x83b5d1)[0x5634fbb505d1] /usr/sbin/mariadbd (_Z21mysql_execute_commandP3THDb+0x47ef)[0x5634fbb6051f] /usr/sbin/mariadbd (_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x5634fbb61ca7] /usr/sbin/mariadbd (_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x5634fbb6455d] /usr/sbin/mariadbd (_Z10do_commandP3THDb+0x143)[0x5634fbb663d3] /usr/sbin/mariadbd (_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x5634fbc8e10d] /usr/sbin/mariadbd (handle_one_connection+0x5d)[0x5634fbc8e46d] /usr/sbin/mariadbd (+0xce9de2)[0x5634fbffede2] /lib/x86_64-linux-gnu/libpthread .so.0(+0x7ea7)[0x7fabb7da4ea7] /lib/x86_64-linux-gnu/libc .so.6(clone+0x3f)[0x7fabb7991a2f] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f90cc11dc90): select `sc_list`.` id `, CASE sc_list.comment_privacy WHEN 0 THEN 'EVERYONE' WHEN 1 THEN 'SCOUTS' WHEN 2 THEN 'NOBODY' ELSE NULL END AS commentPrivacy, `sc_list`.`date_creation` as `dateCreation`, `sc_list`.`date_last_action` as `dateLastEdit`, `sc_list`.`date_last_update` as `dateLastUpdate`, `sc_list`.`description`, `sc_list`.`gen_comment_count` as `commentCount`, `sc_list`.`gen_count` as `productCount`, `sc_list`.`gen_like_positive_count` as `likePositiveCount`, `sc_list`.`gen_type_id_related`, `sc_list`.`hits`, `sc_list`.`is_ordered`, `sc_list`.`is_public`, `sc_list`.`json`, `sc_list`.`label`, `sc_list`.`parent_id` as `parent`, CASE WHEN sc_list.gen_like_count >= sc_list.gen_like_positive_count THEN sc_list.gen_like_count - sc_list.gen_like_positive_count ELSE 0 END AS likeNegativeCount, `sc_list`.`slug`, `sc_list`.`subtype_id`, `sc_list`.`subtype_id_related`, `sc_list`.`user_id_creation` as `user_id`, CONCAT( "https://media.senscritique.com/media/" , m.filename, "/500/cover." , m.extension) AS first_product_backdrop from `sc_list` left join `sc_list_product` as `lp` on `lp`.`list_id` = `sc_list`.` id ` and `lp`.`position` = 1 left join `sc_param_index` as `pi` on `pi`.`product_id_key` = `lp`.`product_id` and `pi`.`param_id` in (326, 368, 424, 425) and `pi`.`state` > -1 left join `sc_media` as `m` on `m`.` id ` = `pi`.`media_id_value` where `sc_list`.` id ` = 2913285 and `sc_list`.`state` = 1 and `sc_list`.`gen_state_user` > -1 limit 1   Connection ID (thread ID): 4682 Status: KILL_SERVER   Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off   The manual page at https: //mariadb .com /kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains information that should help you find out what is causing the crash. Writing a core file ... Working directory at /data/mysql Resource Limits: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 1031142 1031142 processes Max open files 32768 32768 files Max locked memory 524288 524288 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1031142 1031142 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Core pattern: core   Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)   2025-01-07 10:41:59 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid wSrlp7YFoHJH /pOvshJeZmFnvrc = as process 3728649 2025-01-07 10:41:59 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead. 2025-01-07 10:41:59 0 [Note] mariadbd: Aria engine: starting recovery tables to flush: 2 1 0 (0.0 seconds); 2025-01-07 10:41:59 0 [Note] mariadbd: Aria engine: recovery done 2025-01-07 10:41:59 0 [Note] Using unique option prefix 'innodb_flush_neighbor' is error-prone and can break in the future. Please use the full name 'innodb-flush-neighbors' instead. 2025-01-07 10:41:59 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2025-01-07 10:41:59 0 [Note] InnoDB: Using transactional memory 2025-01-07 10:41:59 0 [Note] InnoDB: Number of transaction pools: 1 2025-01-07 10:41:59 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions 2025-01-07 10:41:59 0 [Note] InnoDB: Using liburing 2025-01-07 10:41:59 0 [Note] InnoDB: Initializing buffer pool, total size = 100.000GiB, chunk size = 1.562GiB 2025-01-07 10:42:00 0 [Note] InnoDB: Initialized memory pressure event listener 2025-01-07 10:42:00 0 [Note] InnoDB: Completed initialization of buffer pool 2025-01-07 10:42:00 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes) 2025-01-07 10:42:00 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=6106753213007 2025-01-07 10:42:00 0 [Note] InnoDB: End of log at LSN=6106786058899 2025-01-07 10:42:00 0 [Note] InnoDB: 93 transaction(s) which must be rolled back or cleaned up in total 93 row operations to undo 2025-01-07 10:42:00 0 [Note] InnoDB: Trx id counter is 1891459 2025-01-07 10:42:00 0 [Note] InnoDB: To recover: 16058 pages 2025-01-07 10:42:01 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.000001' , position 21457665 2025-01-07 10:42:01 0 [Note] InnoDB: 128 rollback segments are active. 2025-01-07 10:42:01 0 [Note] InnoDB: Starting in background the rollback of recovered transactions 2025-01-07 10:42:01 0 [Note] InnoDB: Removed temporary tablespace data file : "./ibtmp1" 2025-01-07 10:42:01 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... 2025-01-07 10:42:01 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891362 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891378 2025-01-07 10:42:01 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2025-01-07 10:42:01 0 [Warning] 'innodb-buffer-pool-instances' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2025-01-07 10:42:01 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2025-01-07 10:42:01 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2025-01-07 10:42:01 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files. 2025-01-07 10:42:01 0 [Note] Recovering after a crash using /data/binlog/mysql-bin 2025-01-07 10:42:01 0 [Note] Starting table crash recovery... 2025-01-07 10:42:01 0 [Note] Crash table recovery finished. 2025-01-07 10:42:01 0 [Note] Server socket created on IP: '0.0.0.0' . 2025-01-07 10:42:01 0 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MariaDB server acts as a replica and has its hostname changed. Please use '--log-basename=#' or '--relay-log=mysqld-relay-bin' to avoid this problem. 2025-01-07 10:42:01 0 [Note] /usr/sbin/mariadbd : ready for connections. Version: '10.11.10-MariaDB-deb11-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891410 2025-01-07 10:42:01 5 [Warning] IP address '100.121.142.35' could not be resolved: Name or service not known 2025-01-07 10:42:01 6 [Warning] IP address '100.121.142.34' could not be resolved: Name or service not known 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891450 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891386 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891418 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891402 2025-01-07 10:42:01 0 [Note] InnoDB: Rolled back recovered transaction 1891338

            You have "mariadbd (initiated by: unknown): Normal shutdown" in the middle of a stack trace. Looks like you tried to shutdown a heavily loaded server. Sounds plausible? And that hit a race condition somewhere in the query cache. This is something we can try to repeat.

            serg Sergei Golubchik added a comment - You have "mariadbd (initiated by: unknown): Normal shutdown" in the middle of a stack trace. Looks like you tried to shutdown a heavily loaded server. Sounds plausible? And that hit a race condition somewhere in the query cache. This is something we can try to repeat.

            Hi Sergei,

            I'm not able to reproduce the issue because we have downgrade our server from 10.11.10 to 10.11.2 because of that error: https://jira.mariadb.org/projects/MDEV/issues/MDEV-35413?filter=allissues
            And for the moment (it's been 4 days): the backups are working and we don't get any error.

            Kriko Pierre Krikorian added a comment - Hi Sergei, I'm not able to reproduce the issue because we have downgrade our server from 10.11.10 to 10.11.2 because of that error: https://jira.mariadb.org/projects/MDEV/issues/MDEV-35413?filter=allissues And for the moment (it's been 4 days): the backups are working and we don't get any error.

            let's assume, the bug is not present in 10.11.2 and is present in 10.11.10. There was only one commit in sql_cache.cc in that range, https://github.com/MariaDB/server/commit/4980fcb99094

            sanja, do you think this change could've caused a crash with the stack trace as above when the server is being shutdown under heavy load?

            serg Sergei Golubchik added a comment - let's assume, the bug is not present in 10.11.2 and is present in 10.11.10. There was only one commit in sql_cache.cc in that range, https://github.com/MariaDB/server/commit/4980fcb99094 sanja , do you think this change could've caused a crash with the stack trace as above when the server is being shutdown under heavy load?

            People

              ramesh Ramesh Sivaraman
              Kriko Pierre Krikorian
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.