Details

    • Bug
    • Status: Open (View Workflow)
    • Major
    • Resolution: Unresolved
    • 11.4.0
    • None
    • None
    • None
    • Debian GNU/Linux 12 (bookworm)

    Description

      We are running a 3 node cluster, with wsrep_slave_threads=4 set.

      After an index change in the database we see this error: "WSREP: MDL BF-BF conflict". The Cluster is falling apart. The Cluster Status is this afterwards:

      node-1: wesrep_cluster_size = 1
      node-2: wesrep_cluster_size = 2
      node-3: wesrep_cluster_size = 2

      But only node-1 is the real one, the other two has no connection to node-1 anymore.

      "cinder-cinder-db-2","2025-02-25T08:22:11.158576901Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: declaring 3f24b45d-a4d0 at ssl://XXX.XXX.XXX.XXX:4567 stable"
      "cinder-cinder-db-2","2025-02-25T08:22:11.15859764Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: forgetting 55a20c81-a890 (ssl://10XXX.XXX.XXX.XXX:4567)"
      "cinder-cinder-db-2","2025-02-25T08:22:11.159083462Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: Node 3f24b45d-a4d0 state prim"
      "cinder-cinder-db-2","2025-02-25T08:22:11.16014208Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: view(view_id(PRIM,3f24b45d-a4d0,1444) memb {"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160162177Z stderr F 	3f24b45d-a4d0,0"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160168279Z stderr F 	6a7bfa18-9efe,0"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160173138Z stderr F } joined {"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160177175Z stderr F } left {"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160181564Z stderr F } partitioned {"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160185351Z stderr F 	55a20c81-a890,0"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160189088Z stderr F })"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160193035Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: save pc into disk"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160654241Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: forgetting 55a20c81-a890 (ssl://XXX.XXX.XXX.XXX:4567)"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160676904Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2"
      "cinder-cinder-db-2","2025-02-25T08:22:11.160685109Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID."
      "cinder-cinder-db-2","2025-02-25T08:22:11.161071815Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 9c476a1c-f351-11ef-ad85-dabb0dc0cc85"
      "cinder-cinder-db-2","2025-02-25T08:22:11.16144705Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9c476a1c-f351-11ef-ad85-dabb0dc0cc85 from 0 (db-0.XXX)"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161460215Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: STATE EXCHANGE: got state msg: 9c476a1c-f351-11ef-ad85-dabb0dc0cc85 from 1 (db-2.XXX)"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161464653Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: Quorum results:"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161470234Z stderr F 	version    = 6,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161476065Z stderr F 	component  = PRIMARY,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161480553Z stderr F 	conf_id    = 1437,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161485803Z stderr F 	members    = 2/2 (joined/total),"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161489359Z stderr F 	act_id     = 181296661,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161493137Z stderr F 	last_appl. = 181296600,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161497034Z stderr F 	protocols  = 5/11/4 (gcs/repl/appl),"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161500571Z stderr F 	vote policy= 0,"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161504157Z stderr F 	group UUID = 4cff59e7-272e-11ef-acfd-2fa7271b71c2"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161517422Z stderr F 2025-02-25  8:22:11 0 [Note] WSREP: Flow-control interval: [181, 226]"
      "cinder-cinder-db-2","2025-02-25T08:22:11.161547849Z stderr F 2025-02-25  8:22:11 6 [Note] WSREP: ####### processing CC 181296662, local, ordered"
      "cinder-cinder-db-2","2025-02-25T08:22:11.172985115Z stderr F 2025-02-25  8:22:11 9 [Note] WSREP: MDL BF-BF conflict"
      "cinder-cinder-db-2","2025-02-25T08:22:11.173001996Z stderr F schema:  cinder"
      "cinder-cinder-db-2","2025-02-25T08:22:11.173017395Z stderr F request: (9 	seqno 181296659 	wsrep (toi, exec, committed) cmd 0 2 	CREATE INDEX volumes_deleted_project_id_idx ON volumes (deleted, project_id))"
      "cinder-cinder-db-2","2025-02-25T08:22:11.173028336Z stderr F granted: (7 	seqno 181296660 	wsrep (high priority, exec, committing) cmd 0 161 	(null))"
      "cinder-cinder-db-2","2025-02-25T08:22:11.173067659Z stderr F 2025-02-25  8:22:11 9 [ERROR] Aborting"
      "cinder-cinder-db-2","2025-02-25T08:47:00.034222022Z stderr F 2025-02-25  8:47:00 70223 [Warning] WSREP: Node desync failed: File descriptor in bad state"
      "cinder-cinder-db-2","2025-02-25T08:47:00.034266646Z stderr F 	 at /bitnami/blacksmith-sandox/libgalera-26.4.21/galera/src/replicator_smm.cpp:desync():3164"
      "cinder-cinder-db-2","2025-02-25T08:47:00.034275372Z stderr F ERROR 1396 (HY000) at line 1: Operation 'desync' failed for SET GLOBAL wsrep_desync = ON"
      "cinder-cinder-db-2","2025-02-25T08:47:00.034282986Z stderr F 2025-02-25  8:47:00 70223 [Warning] WSREP: SET desync failed 1 for schema: (null), query: SET GLOBAL wsrep_desync = ON"
      

      we could resolve it, by setting wsrep_slave_threads=1.

      Attachments

        Issue Links

          Activity

            There are no comments yet on this issue.

            People

              janlindstrom Jan Lindström
              f.hiekel Friedrich Hiekel
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.