Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-4711

Maxscale crashes on network errors

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Incomplete
    • 22.08.1, 23.02.3
    • N/A
    • Core
    • RHEL v8.2
      VMware vCenter v7
      MariaDB v10.7 (3 node Galera cluster)
      Maxcsale v23.02.3 (one instance of each per MariaDB VM for redundancy)

    Description

      Hi There,

      We're experiencing regular load balancer crashes that seem to occur while vCenter commences vMotion activities on the same physical hosts where the MariaDB/Maxscale load balancer VMs reside. We have pinned the MariaDB/Maxscale load balancer VMs, as well as the application VMs that connect to the cluster, to the physical host they reside on, but to no avail. We have a three node cluster, and at least one of them experiences this event daily. Memory consumption varies between 50-75% (256Gb total), CPU 30-40% (16 vcores). Network connections immediately preceed the crashes. Upstream Java applications interface the cluster, DB connections vary between 20 to 80. Output below logs the core dump produced, please advise where the dumps are located as they don't appear in the system default location. Would gdb need to be configured to enable this?;

      2023-04-05 23:06:12 error : (871) Network error in connection to server 'viexh-session-usage-mdb-01', session in state 'Stopping session' (DCB::State::POLLING): 104, Connection reset by peer
      2023-04-05 23:06:12 error : (874) Network error in connection to server 'viexh-session-usage-mdb-01', session in state 'Stopping session' (DCB::State::POLLING): 104, Connection reset by peer
      2023-04-05 23:06:12 error : (879) Network error in connection to server 'viexh-session-usage-mdb-01', session in state 'Stopping session' (DCB::State::POLLING): 104, Connection reset by peer
      2023-04-05 23:06:12 error : (838) Network error in connection to server 'viexh-session-usage-mdb-01', session in state 'Stopping session' (DCB::State::POLLING): 104, Connection reset by peer (subsequent similar messages suppressed for 10000 milliseconds)
      2023-04-05 23:08:32 notice : Server changed state: viexh-session-usage-mdb-03[10.195.241.81:3306]: lost_slave. [Slave, Synced, Running] -> [Running]
      alert : MaxScale 22.08.1 received fatal signal 6. Commit ID: 2a533b7bce81e767ef5b263b0b32ebb509dbfe4c System name: Linux Release string: Red Hat Enterprise Linux release 8.2 (Ootpa)

      2023-04-05 23:08:52 alert : MaxScale 22.08.1 received fatal signal 6. Commit ID: 2a533b7bce81e767ef5b263b0b32ebb509dbfe4c System name: Linux Release string: Red Hat Enterprise Linux release 8.2 (Ootpa)
      2023-04-05 23:08:52 alert : Statement currently being classified: none/unknown
      2023-04-05 23:08:52 notice : For a more detailed stacktrace, install GDB and add 'debug=gdb-stacktrace' under the [maxscale] section.
      /lib64/libc.so.6(epoll_wait+0x57): ??:?
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x120): maxutils/maxbase/src/worker.cc:1099
      /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x4f): maxutils/maxbase/src/worker.cc:822
      /usr/bin/maxscale(main+0x214c): server/core/gateway.cc:2235
      /lib64/libc.so.6(__libc_start_main+0xf3): ??:?
      /usr/bin/maxscale(_start+0x2e): ??:?
      alert : Writing core dump.

      Thanks.

      Attachments

        1. MXS-4711_maxscale-logs-04.txt
          21 kB
        2. MXS-4711_maxscale-logs-03.txt
          40 kB
        3. MXS-4711_maxscale-logs-02.txt
          6 kB
        4. MXS-4711_maxscale-logs-01.txt
          241 kB
        5. MXS-4711_maxscale-graphs-01.PNG
          MXS-4711_maxscale-graphs-01.PNG
          21 kB
        6. MXS-4711_keepalived-config-01.txt
          0.6 kB
        7. maxscale-server-3.txt
          3 kB
        8. maxscale-server-2.txt
          3 kB
        9. maxscale-server-1.txt
          5 kB
        10. mariadb-3.txt
          3 kB
        11. mariadb-2.txt
          3 kB
        12. mariadb-1.txt
          3 kB
        13. galera-3.txt
          0.7 kB
        14. galera-2.txt
          0.7 kB
        15. galera-1.txt
          0.7 kB

        Activity

          People

            markus makela markus makela
            Presnickety Presnickety
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.