Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-30947

Galera server crashes during IST 1:10.6.12+maria~ubu2004

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.3.36, 10.4.26
    • N/A
    • Galera

    Description

      The bug is exactly as https://jira.mariadb.org/browse/MDEV-29375 but reproduced during powering off/powering on node with mariadb.
      One of nodes failed to do IST. Attaching configs and logs from all nodes in archive. Could you please help to resolve it, thank you.

      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - 2023-03-28  6:29:56 2 [ERROR] WSREP: Corrupt buffer header: addr: 0x7f617bfff518, seqno: 3543537071533338624, size: 8084
      63664, ctx: 0x558bd59ab298, flags: 12337. store: 45, type: 48
      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - 230328  6:29:56 [ERROR] mysqld got signal 6 ;
      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - This could be because you hit a bug. It is also possible that this binary
      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - or one of the libraries it was linked against is corrupt, improperly built,
      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - or misconfigured. This error can also be caused by malfunctioning hardware.
      2023-03-28 06:29:56,837 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - We will try our best to scrape up some info that will hopefully help
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - diagnose the problem, but since we have already crashed, 
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - something is definitely wrong and this may fail.
      2023-03-28 06:29:56,838 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - Server version: 10.6.12-MariaDB-1:10.6.12+maria~ubu2004 source revision: 4c79e15cc3716f69c044d4287ad2160da8101cdc
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - key_buffer_size=0
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - read_buffer_size=131072
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - max_used_connections=0
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - max_threads=8194
      2023-03-28 06:29:56,839 - OpenStack-Helm Mariadb - INFO - thread_count=3
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO - It is possible that mysqld could use up to 
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO - key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 18043994 K  bytes of memory
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO - Hope that's ok; if not, decrease some variables in the equation.
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO -
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO - Thread pointer: 0x7f6164000c58
      2023-03-28 06:29:56,840 - OpenStack-Helm Mariadb - INFO - Attempting backtrace. You can use the following information to find out
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - where mysqld died. If you see no messages after this, something went
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - terribly wrong...
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - stack_bottom = 0x7f61c139ad88 thread_stack 0x49000
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - Printing to addr2line failed
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - mysqld(my_print_stacktrace+0x32)[0x558bd39d90c2]
      2023-03-28 06:29:56,841 - OpenStack-Helm Mariadb - INFO - mysqld(handle_fatal_signal+0x485)[0x558bd34a0c55]
      2023-03-28 06:29:56,842 - OpenStack-Helm Mariadb - INFO - /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f61c3f70420]
      2023-03-28 06:29:56,842 - OpenStack-Helm Mariadb - INFO - /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f61c3a7400b]
      2023-03-28 06:29:56,843 - OpenStack-Helm Mariadb - INFO - /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f61c3a53859]
      2023-03-28 06:29:56,844 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x3ebc7)[0x7f61c3572bc7]
      2023-03-28 06:29:56,844 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x1cc3fe)[0x7f61c37003fe]
      2023-03-28 06:29:56,844 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x1b12da)[0x7f61c36e52da]
      2023-03-28 06:29:56,844 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x91ea0)[0x7f61c35c5ea0]
      2023-03-28 06:29:56,844 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x7d9f7)[0x7f61c35b19f7]
      2023-03-28 06:29:56,846 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x7e42f)[0x7f61c35b242f]
      2023-03-28 06:29:56,846 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x7ea7d)[0x7f61c35b2a7d]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0xb019b)[0x7f61c35e419b]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0xb0682)[0x7f61c35e4682]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x7cef0)[0x7f61c35b0ef0]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - /usr/lib/galera/libgalera_smm.so(+0x504a1)[0x7f61c35844a1]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - mysqld(_ZN5wsrep18wsrep_provider_v2611run_applierEPNS_21high_priority_serviceE+0x12)[0x558bd3a763b2]
      2023-03-28 06:29:56,847 - OpenStack-Helm Mariadb - INFO - mysqld(+0xcbe651)[0x558bd3771651]
      2023-03-28 06:29:56,849 - OpenStack-Helm Mariadb - INFO - mysqld(_Z15start_wsrep_THDPv+0x26b)[0x558bd37603bb]
      2023-03-28 06:29:56,850 - OpenStack-Helm Mariadb - INFO - mysqld(+0xc36766)[0x558bd36e9766]
      2023-03-28 06:29:56,851 - OpenStack-Helm Mariadb - INFO - /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609)[0x7f61c3f64609]
      2023-03-28 06:29:56,852 - OpenStack-Helm Mariadb - INFO - /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f61c3b50133]
      2023-03-28 06:29:56,852 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,852 - OpenStack-Helm Mariadb - INFO - Trying to get some variables.
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - Some pointers may be invalid and cause the dump to abort.
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - Query (0x0): (null)
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - Connection ID (thread ID): 2
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - Status: NOT_KILLED
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,853 - OpenStack-Helm Mariadb - INFO - Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
      2023-03-28 06:29:56,856 - OpenStack-Helm Mariadb - INFO - 
      2023-03-28 06:29:56,856 - OpenStack-Helm Mariadb - INFO -
      

      mysql@mariadb-server-0:/$ dpkg -l |grep -e mariadb -e  galera
      ii  galera-4                   26.4.14-ubu2004                   amd64        Replication framework for transactional applications
      ii  libdbd-mariadb-perl        1.11-3ubuntu2                     amd64        Perl5 database interface to the MariaDB/MySQL databases
      ii  libmariadb3:amd64          1:10.6.12+maria~ubu2004           amd64        MariaDB database client library
      ii  mariadb-backup             1:10.6.12+maria~ubu2004           amd64        Backup tool for MariaDB server
      ii  mariadb-client-10.6        1:10.6.12+maria~ubu2004           amd64        MariaDB database client binaries
      ii  mariadb-client-core-10.6   1:10.6.12+maria~ubu2004           amd64        MariaDB database core client binaries
      ii  mariadb-common             1:10.6.12+maria~ubu2004           all          MariaDB common configuration files
      ii  mariadb-server             1:10.6.12+maria~ubu2004           all          MariaDB database server (metapackage depending on the latest version)
      ii  mariadb-server-10.6        1:10.6.12+maria~ubu2004           amd64        MariaDB database server binaries
      ii  mariadb-server-core-10.6   1:10.6.12+maria~ubu2004           amd64
      

      Attachments

        Issue Links

          Activity

            People

              Yurchenko Alexey
              jumpojoy Vasyl Saienko
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.