Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2753

MXS crash on cdc stream request

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 2.4.2, 2.4.3
    • Fix Version/s: N/A
    • Component/s: avrorouter, binlogrouter, cdc
    • Labels:
      None
    • Environment:
      CentOS7, 8 core Xeon, 8GB RAM.
      Maxscale 2.4.3 (updated today). MXS CDC Connector 2.4.3.

      Description

      Currently seeing Maxscale hang, fail and then restart via systemd watchdog whenever a CDC request is made. This is both via a local cdc.py request and from an external server running mxs_adapter.

      The service had been running for around 6 hours without failure, handling circa 1k new rows into Columnstore every 10 seconds successfully.

      Nothing changed in the config of mxs_adapter or maxscale between running state and failed state.

      When failure occured, an update from 2.4.2 to 2.4.3 was performed today.

      Additionally, following advice in https://jira.mariadb.org/browse/MXS-964 the router_options entry was added to the avro-router.

      On failure Maxscale service outputs this...

      Nov 06 16:35:05 maxscale1 systemd[1]: maxscale.service watchdog timeout (limit 1min)!
      Nov 06 16:35:05 maxscale1 maxscale[63821]: Fatal: MaxScale 2.4.3 received fatal signal 6. Commit ID: b33ef98f6c26b71e3cc9ea44b398776d51b35664 System name: Linux Release string: NAME="CentOS Linux"
      Nov 06 16:35:05 maxscale1 maxscale[63821]: 
                                                          /lib64/libc.so.6(epoll_wait+0x33): :?
                                                          /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0xd0): maxutils/maxbase/src/worker.cc:795
                                                          /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:559
                                                          /usr/bin/maxscale(main+0x2a76): server/core/gateway.cc:2265
                                                          /lib64/libc.so.6(__libc_start_main+0xf5): ??:?
                                                          /usr/bin/maxscale(): ??:?
      Nov 06 16:35:05 maxscale1 systemd[1]: maxscale.service: main process exited, code=killed, status=6/ABRT
      Nov 06 16:35:05 maxscale1 systemd[1]: Unit maxscale.service entered failed state.
      Nov 06 16:35:05 maxscale1 systemd[1]: maxscale.service failed.
      

      And here is the relevant section from Maxscale config file:

      [replication-listener]
      type=listener
      service=replication
      protocol=MariaDBClient
      port=3311
       
      [replication]
      type=service
      router=binlogrouter
      master_id=50
      server_id=50
      binlogdir=/home/binlogs
      filestem=mysql-bin
      user=username
      password=password
       
      [avro-router]
      type=service
      router=avrorouter
      source=replication
      router_options=disable_sescmd_history=true
      match=/databasename\.tablename/
      binlogdir=/home/binlogs
      avrodir=/home/binlogs
      filestem=mysql-bin
      start_index=17
       
      [avro-listener]
      type=listener
      service=avro-router
      protocol=CDC
      port=4001
      

        Attachments

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            DBA666 DBA666
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration