Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35671

MariaDB got signal 11 after upgrading

    XMLWordPrintable

Details

    • Bug
    • Status: Needs Feedback (View Workflow)
    • Major
    • Resolution: Unresolved
    • 10.11.10
    • None

    Description

      After upgrading from 10.5 to 10.11 following the documentation, we have several issues with our MariaDB.

      And yesterday we get a "segfault" on the primary node:

      241216 21:58:48 [ERROR] mysqld got signal 11 ;
      Sorry, we probably made a mistake, and this is a bug.
       
      Your assistance in bug reporting will enable us to fix this for the next release.
      To report this bug, see https://mariadb.com/kb/en/reporting-bugs
       
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
       
      Server version: 10.11.10-MariaDB-deb11-log source revision: 3d0fb150289716ca75cd64d62823cf715ee47646
      key_buffer_size=0
      read_buffer_size=4194304
      max_used_connections=834
      max_threads=1502
      thread_count=735
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 9268439 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
       
      Thread pointer: 0x7eee1820ad68
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7ef25cc4a000 thread_stack 0x49000
      2024-12-16 21:58:59 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
      2024-12-16 21:58:59 201911 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 201911, query: select `sc_collection`.`gen_review_comment_count` as `commentCount`, `sc_collection`.`gen_review_like_count` as `likeCount`, `sc_collection`.`gen_review_like_positive_count` as `likePositiveCount`, `sc_collection`.`id`, `sc_collection`.`product_id` as `productId`, `sc_collection`.`rating`, `sc_collection`.`review_comment_privacy` as `commentPrivacy`, `sc_collection`.`review_content_autosave` as `bodyAutoSaved`, `sc_collection`.`review_content_autosave_source` as `bodySourceAutoSaved`, `sc_collection`.`review_content` as `bodyText`, `sc_collection`.`review_content_source` as `body`, `sc_collection`.`review_date_creation` as `dateCreation`, `sc_collection`.`review_date_last_update` as `dateLastUpdate`, `sc_collection`.`review_hits` as `hits`, `sc_collection`.`review_state` as `isPublished`, `sc_collection`.`review_title_autosave` as `titleAutoSaved`, `sc_collection`.`review_title` as `title`, `sc_collection`.`user_id` as `user`, `product`.`
      2024-12-16 21:58:59 198757 [Warning] Sort aborted, host: 100.121.142.32, user: prod, thread: 198757, query: select `c`.`id` from `sc_collection` as `c` left join `sc_user` as `u` on `u`.`state` = 1 and `u`.`id` = `c`.`user_id` where `c`.`product_id` = 20643012 and `c`.`gen_state_user` = 1 and `c`.`is_review` = 1 and `c`.`review_state` = 1 and `c`.`review_state_validation` > -2 order by gen_review_like_positive_count DESC, c.id DESC limit 10
      2024-12-16 21:58:59 86910 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 86910, query: select `user_id` from `sc_collection` where `rating` = 7 and `product_id` = 418945 and `rating` is not null order by `date_done` desc limit 30
      2024-12-16 21:58:59 137086 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 137086, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8
      2024-12-16 21:58:59 87648 [Warning] Sort aborted, host: 100.121.142.11, user: prod, thread: 87648, query: select `id` from `sc_gen_product_fr_FR` where `date_release` between NOW() + INTERVAL 3 DAY and NOW() + INTERVAL 6 MONTH and `subtype_id` = 1 order by `collection_count` DESC limit 8
      Printing to addr2line failed
      /usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x563027cfaffe]
      /usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x5630277c0a75]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f25f032b140]
       
       
      /usr/sbin/mariadbd(_ZN11Query_cache12unlink_tableEP23Query_cache_block_table+0x8)[0x5630274ee158]
      /usr/sbin/mariadbd(_ZN11Query_cache19register_all_tablesEP3THDP17Query_cache_blockP10TABLE_LISTj+0xb3)[0x5630274ef1a3]
      /usr/sbin/mariadbd(_ZN11Query_cache11store_queryEP3THDP10TABLE_LIST+0x376)[0x5630274f0fd6]
      /usr/sbin/mariadbd(+0x83b5d1)[0x56302754b5d1]
      /usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x47ef)[0x56302755b51f]
      /usr/sbin/mariadbd(_Z11mysql_parseP3THDPcjP12Parser_state+0x1e7)[0x56302755cca7]
      /usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0x15bd)[0x56302755f55d]
      /usr/sbin/mariadbd(_Z10do_commandP3THDb+0x143)[0x5630275613d3]
      /usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x37d)[0x56302768910d]
      /usr/sbin/mariadbd(handle_one_connection+0x5d)[0x56302768946d]
      /usr/sbin/mariadbd(+0xce9de2)[0x5630279f9de2]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f25f031fea7]
      2024-12-16 21:59:36 0 [Note] InnoDB: Memory pressure event freed 178062 pages
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f25eff1cdef]
       
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0x7eee182f5210): select `sc_list`.`id`, CASE sc_list.comment_privacy WHEN 0
                THEN 'EVERYONE' WHEN 1
                THEN 'SCOUTS' WHEN 2
                THEN 'NOBODY'
                ELSE NULL
              END AS commentPrivacy, `sc_list`.`date_creation` as `dateCreation`, `sc_list`.`date_last_action` as `dateLastEdit`, `sc_list`.`date_last_update` as `dateLastUpdate`, `sc_list`.`description`, `sc_list`.`gen_comment_count` as `commentCount`, `sc_list`.`gen_count` as `productCount`, `sc_list`.`gen_like_positive_count` as `likePositiveCount`, `sc_list`.`gen_type_id_related`, `sc_list`.`hits`, `sc_list`.`is_ordered`, `sc_list`.`is_public`, `sc_list`.`json`, `sc_list`.`label`, `sc_list`.`parent_id` as `parent`, CASE WHEN sc_list.gen_like_count >= sc_list.gen_like_positive_count
                THEN sc_list.gen_like_count - sc_list.gen_like_positive_count
                ELSE 0
              END AS likeNegativeCount, `sc_list`.`slug`, `sc_list`.`subtype_id`, `sc_list`.`subtype_id_related`, `sc_list`.`user_id_creation` as `user_id`, CONCAT("https://media.senscritique.com/media/", m.filename, "/500/cover.", m.extension) AS first_product_backdrop from `sc_list` left join `sc_list_product` as `lp` on `lp`.`list_id` = `sc_list`.`id` and `lp`.`position` = 1 left join `sc_param_index` as `pi` on `pi`.`product_id_key` = `lp`.`product_id` and `pi`.`param_id` in (326, 368, 424, 425) and `pi`.`state` > -1 left join `sc_media` as `m` on `m`.`id` = `pi`.`media_id_value` where `sc_list`.`id` = 3356937 and `sc_list`.`state` = 1 and `sc_list`.`gen_state_user` > -1 limit 1
       
      Connection ID (thread ID): 199450
      Status: KILL_SERVER
       
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off,hash_join_cardinality=off,cset_narrowing=off
       
      The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mariadbd/ contains
      information that should help you find out what is causing the crash.
      Writing a core file...
      Working directory at /data/mysql
      Resource Limits:
      Limit                     Soft Limit           Hard Limit           Units     
      Max cpu time              unlimited            unlimited            seconds   
      Max file size             unlimited            unlimited            bytes     
      Max data size             unlimited            unlimited            bytes     
      Max stack size            8388608              unlimited            bytes     
      Max core file size        0                    unlimited            bytes     
      Max resident set          unlimited            unlimited            bytes     
      Max processes             1031124              1031124              processes 
      Max open files            32768                32768                files     
      Max locked memory         524288               524288               bytes     
      Max address space         unlimited            unlimited            bytes     
      Max file locks            unlimited            unlimited            locks     
      Max pending signals       1031124              1031124              signals   
      Max msgqueue size         819200               819200               bytes     
      Max nice priority         0                    0                    
      Max realtime priority     0                    0                    
      Max realtime timeout      unlimited            unlimited            us        
      Core pattern: core
       
      Kernel version: Linux version 5.10.0-9-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)
       
      2024-12-16 22:00:05 0 [Note] Starting MariaDB 10.11.10-MariaDB-deb11-log source revision 3d0fb150289716ca75cd64d62823cf715ee47646 server_uid Y8nq1+0PFBI3t+kgYmZD2KMfQ4Q= as process 42254
      2024-12-16 22:00:05 0 [Note] Using unique option prefix 'myisam-recover' is error-prone and can break in the future. Please use the full name 'myisam-recover-options' instead.
      2024-12-16 22:00:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
      2024-12-16 22:00:05 0 [Note] InnoDB: Using transactional memory
      2024-12-16 22:00:05 0 [Note] InnoDB: Number of transaction pools: 1
      2024-12-16 22:00:05 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
      2024-12-16 22:00:05 0 [Note] InnoDB: Using liburing
      2024-12-16 22:00:05 0 [Note] InnoDB: Initializing buffer pool, total size = 200.000GiB, chunk size = 3.125GiB
      2024-12-16 22:00:06 0 [Note] InnoDB: Initialized memory pressure event listener
      2024-12-16 22:00:06 0 [Note] InnoDB: Completed initialization of buffer pool
      2024-12-16 22:00:07 0 [Note] InnoDB: Buffered log writes (block size=4096 bytes)
      2024-12-16 22:00:07 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=214350961313938
      2024-12-16 22:00:11 0 [Note] InnoDB: End of log at LSN=214351431883947
      2024-12-16 22:00:11 0 [Note] InnoDB: 3 transaction(s) which must be rolled back or cleaned up in total 4 row operations to undo
      2024-12-16 22:00:11 0 [Note] InnoDB: Trx id counter is 60405952135
      2024-12-16 22:00:11 0 [Note] InnoDB: To recover: 367581 pages
      2024-12-16 22:00:22 0 [Note] InnoDB: Last binlog file '/data/binlog/mysql-bin.018825', position 399196458
      2024-12-16 22:00:22 0 [Note] InnoDB: 128 rollback segments are active.
      2024-12-16 22:00:22 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
      2024-12-16 22:00:22 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
      2024-12-16 22:00:22 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
      2024-12-16 22:00:22 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952132
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952133
      2024-12-16 22:00:22 0 [Note] InnoDB: log sequence number 214351431883947; transaction id 60405952138
      2024-12-16 22:00:22 0 [Note] InnoDB: Rolled back recovered transaction 60405952049
      2024-12-16 22:00:22 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      2024-12-16 22:00:22 0 [Note] InnoDB: Loading buffer pool(s) from /data/mysql/ib_buffer_pool
      2024-12-16 22:00:22 0 [Note] Plugin 'FEEDBACK' is disabled.
      2024-12-16 22:00:22 0 [Warning] Plugin 'SPHINX' is of maturity level gamma while the server is stable
      2024-12-16 22:00:22 0 [ERROR] mariadbd: Can't open shared library '/usr/lib/mysql/plugin/tokudb_backup.so' (errno: 0, cannot open shared object file: No such file or directory)
      2024-12-16 22:00:22 0 [Warning] 'innodb-file-format' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-log-files-in-group' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-thread-concurrency' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Warning] 'innodb-stats-sample-pages' was removed. It does nothing now and exists only for compatibility with old my.cnf files.
      2024-12-16 22:00:22 0 [Note] Recovering after a crash using /data/binlog/mysql-bin
      2024-12-16 22:00:22 0 [Note] Starting table crash recovery...
      2024-12-16 22:00:22 0 [Note] Crash table recovery finished.
      2024-12-16 22:00:23 0 [Note] Server socket created on IP: '0.0.0.0'.
      2024-12-16 22:00:23 0 [Warning] 'user' entry 'root@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Warning] 'user' entry '@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Warning] 'proxies_priv' entry '@% root@mysql-master' ignored in --skip-name-resolve mode.
      2024-12-16 22:00:23 0 [Note] /usr/sbin/mariadbd: ready for connections.
      Version: '10.11.10-MariaDB-deb11-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
      2024-12-16 22:00:23 40 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)
      2024-12-16 22:00:23 43 [Warning] Access denied for user 'debian-sys-maint'@'localhost' (using password: YES)
      

      You can find the our configuration in the attachments.

      I share you also some screenshot of our monitoring tool (datadog) to show the difference between "before" the upgrade and "after" the upgrade (the upgrade have been performed december the 4th).

      We don't have perform any major change on our application, and after the upgrade we get slow queries, huge lock struct, and so on, like there was a change on how mariadb response.

      Regards,

      Attachments

        1. metrics_mysql.png
          metrics_mysql.png
          274 kB
        2. my-slave.cnf
          6 kB
        3. sql-master.cnf
          6 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Kriko Pierre Krikorian
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.