Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-20104

MariaDB crashes when in galera-cluster behind ProxySQL 2.0 .

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Incomplete
    • 10.4.4, 10.4.5, 10.4.6
    • N/A
    • Galera, Server
    • Ubuntu 18.04.2 (LTS)

    Description

      Hello,

      I have a galera cluster setup with 3 nodes. This cluster is being used behind a ProxySQL v2.0.

      When I execute a select query to view a parameter and exit the client interface, it causes the galera node currently ONLINE in the writer hostgroup to crash. For example, when I execute this query and exits, it crashes:

      SELECT @@hostname;
      +------------+
      | @@hostname |
      +------------+
      | mariadb3   |
      +------------+
      

      The crash generates this error log:

      2019-07-19 12:56:20 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.100.4:4567 
      2019-07-19 12:56:21 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') reconnecting to b3b145cc (tcp://192.168.100.4:4567), attempt 0
      2019-07-19 12:56:25 0 [Note] WSREP: evs::proto(f400fc1e, OPERATIONAL, view_id(REG,b3b145cc,78)) suspecting node: b3b145cc
      2019-07-19 12:56:25 0 [Note] WSREP: evs::proto(f400fc1e, OPERATIONAL, view_id(REG,b3b145cc,78)) suspected node without join message, declaring inactive
      2019-07-19 12:56:26 0 [Note] WSREP: declaring c9ed7fd8 at tcp://192.168.100.3:4567 stable
      2019-07-19 12:56:26 0 [Note] WSREP: Node c9ed7fd8 state prim
      2019-07-19 12:56:26 0 [Note] WSREP: view(view_id(PRIM,c9ed7fd8,79) memb {
      	c9ed7fd8,0
      	f400fc1e,0
      } joined {
      } left {
      } partitioned {
      	b3b145cc,0
      })
      2019-07-19 12:56:26 0 [Note] WSREP: save pc into disk
      2019-07-19 12:56:26 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
      2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      2019-07-19 12:56:26 0 [Note] WSREP: forgetting b3b145cc (tcp://192.168.100.4:4567)
      2019-07-19 12:56:26 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c
      2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c from 0 (mariadb2)
      2019-07-19 12:56:26 0 [Note] WSREP: STATE EXCHANGE: got state msg: 176378f2-aa03-11e9-a484-8a52dea1ff8c from 1 (mariadb1)
      2019-07-19 12:56:26 0 [Note] WSREP: Quorum results:
      	version    = 5,
      	component  = PRIMARY,
      	conf_id    = 42,
      	members    = 2/2 (joined/total),
      	act_id     = 256,
      	last_appl. = 0,
      	protocols  = 1/10/4 (gcs/repl/appl),
      	vote policy= 0,
      	group UUID = a0e4ee53-a2ed-11e9-8785-77f49c647069
      2019-07-19 12:56:26 0 [Note] WSREP: Writing down CC checksum: ef242699 07eacac7 fb0c42ea 3001f3d1 at offset 184
      2019-07-19 12:56:26 0 [Note] WSREP: Flow-control interval: [23, 23]
      2019-07-19 12:56:26 0 [Note] WSREP: Trying to continue unpaused monitor
      2019-07-19 12:56:26 1 [Note] WSREP: ####### processing CC 257, local, ordered
      2019-07-19 12:56:26 1 [Note] WSREP: ####### drain monitors upto 256
      2019-07-19 12:56:26 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
      2019-07-19 12:56:26 1 [Note] WSREP: ####### My UUID: f400fc1e-a6cf-11e9-ab61-eac7c8a52069
      2019-07-19 12:56:26 1 [Note] WSREP: ####### ST not required
      2019-07-19 12:56:26 1 [Note] WSREP: Skipping cert index reset
      2019-07-19 12:56:26 1 [Note] WSREP: ####### Adjusting cert position: 256 -> 257
      2019-07-19 12:56:26 0 [Note] WSREP: Service thread queue flushed.
      2019-07-19 12:56:26 1 [Note] WSREP: ####### Setting monitor position to 257
      2019-07-19 12:56:26 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 257
      2019-07-19 12:56:26 1 [Note] WSREP: Min available from gcache for CC from group: 125
      2019-07-19 12:56:26 1 [Note] WSREP: ================================================
      View:
        id: a0e4ee53-a2ed-11e9-8785-77f49c647069:257
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 1
        members(2):
      	0: c9ed7fd8-a2f6-11e9-b3a5-3e00a5379103, mariadb2
      	1: f400fc1e-a6cf-11e9-ab61-eac7c8a52069, mariadb1
      =================================================
      2019-07-19 12:56:26 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-07-19 12:56:29 0 [Note] WSREP:  cleaning up b3b145cc (tcp://192.168.100.4:4567)
      2019-07-19 12:56:30 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') connection established to b3b145cc tcp://192.168.100.4:4567
      2019-07-19 12:56:30 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
      2019-07-19 12:56:30 0 [Note] WSREP: declaring b3b145cc at tcp://192.168.100.4:4567 stable
      2019-07-19 12:56:30 0 [Note] WSREP: declaring c9ed7fd8 at tcp://192.168.100.3:4567 stable
      2019-07-19 12:56:30 0 [Note] WSREP: Node c9ed7fd8 state prim
      2019-07-19 12:56:30 0 [Note] WSREP: view(view_id(PRIM,b3b145cc,80) memb {
      	b3b145cc,0
      	c9ed7fd8,0
      	f400fc1e,0
      } joined {
      } left {
      } partitioned {
      })
      2019-07-19 12:56:30 0 [Note] WSREP: save pc into disk
      2019-07-19 12:56:30 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
      2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
      2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: sent state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af
      2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 0 (mariadb3)
      2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 1 (mariadb2)
      2019-07-19 12:56:30 0 [Note] WSREP: STATE EXCHANGE: got state msg: 19e956d9-aa03-11e9-b008-a6b67e5882af from 2 (mariadb1)
      2019-07-19 12:56:30 0 [Note] WSREP: Quorum results:
      	version    = 5,
      	component  = PRIMARY,
      	conf_id    = 43,
      	members    = 2/3 (joined/total),
      	act_id     = 257,
      	last_appl. = 0,
      	protocols  = 1/10/4 (gcs/repl/appl),
      	vote policy= 0,
      	group UUID = a0e4ee53-a2ed-11e9-8785-77f49c647069
      2019-07-19 12:56:30 0 [Note] WSREP: Writing down CC checksum: 555e7bf5 849283ab a4fcdd10 db627a65 at offset 240
      2019-07-19 12:56:30 0 [Note] WSREP: Flow-control interval: [28, 28]
      2019-07-19 12:56:30 0 [Note] WSREP: Trying to continue unpaused monitor
      2019-07-19 12:56:30 1 [Note] WSREP: ####### processing CC 258, local, ordered
      2019-07-19 12:56:30 1 [Note] WSREP: ####### drain monitors upto 257
      2019-07-19 12:56:30 1 [Note] WSREP: REPL Protocols: 10 (5, 3)
      2019-07-19 12:56:30 1 [Note] WSREP: ####### My UUID: f400fc1e-a6cf-11e9-ab61-eac7c8a52069
      2019-07-19 12:56:30 1 [Note] WSREP: ####### ST not required
      2019-07-19 12:56:30 1 [Note] WSREP: Skipping cert index reset
      2019-07-19 12:56:30 1 [Note] WSREP: ####### Adjusting cert position: 257 -> 258
      2019-07-19 12:56:30 0 [Note] WSREP: Service thread queue flushed.
      2019-07-19 12:56:30 1 [Note] WSREP: ####### Setting monitor position to 258
      2019-07-19 12:56:30 1 [Note] WSREP: Lowest cert indnex boundary for CC from group: 258
      2019-07-19 12:56:30 1 [Note] WSREP: Min available from gcache for CC from group: 125
      2019-07-19 12:56:30 1 [Note] WSREP: ================================================
      View:
        id: a0e4ee53-a2ed-11e9-8785-77f49c647069:258
        status: primary
        protocol_version: 4
        capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
        final: no
        own_index: 2
        members(3):
      	0: b3b145cc-a3f3-11e9-8d5d-b6f514e33bcd, mariadb3
      	1: c9ed7fd8-a2f6-11e9-b3a5-3e00a5379103, mariadb2
      	2: f400fc1e-a6cf-11e9-ab61-eac7c8a52069, mariadb1
      =================================================
      2019-07-19 12:56:30 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2019-07-19 12:56:31 0 [Note] WSREP: Member 0.0 (mariadb3) requested state transfer from '*any*'. Selected 1.0 (mariadb2)(SYNCED) as donor.
      2019-07-19 12:56:31 0 [Note] WSREP: 1.0 (mariadb2): State transfer to 0.0 (mariadb3) complete.
      2019-07-19 12:56:31 0 [Note] WSREP: Member 1.0 (mariadb2) synced with group.
      2019-07-19 12:56:33 0 [Note] WSREP: (f400fc1e, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2019-07-19 12:56:34 0 [Note] WSREP: 0.0 (mariadb3): State transfer from 1.0 (mariadb2) complete.
      2019-07-19 12:56:34 0 [Note] WSREP: Member 0.0 (mariadb3) synced with group.
      

      I have already created an issue at proxysql [1] and galera [2] concerning this matter.

      Regards,
      Shravan

      [1] - https://github.com/sysown/proxysql/issues/2132
      [2] - https://github.com/codership/galera/issues/542

      Attachments

        Activity

          People

            janlindstrom Jan Lindström
            dshravan Shravan Dwarka
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.