Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-559

Crash due to debug assertion in readwritesplit

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.0
    • Component/s: readwritesplit
    • Labels:
      None

      Description

      Originally reported by Florian Engelmann:

      It looks like the "jan21" version is still crashing.

      The crash was caused by using IP 10.0.248.202 which is listening to the RW split router of 5 MariaDB 5.5.47 nodes to dump a DB to the cluster:

      mysql -h10.0.248.202  -uadmin -pxxxxxxxxxxxxxxxxxxxxxxxx ops_meta_db < ops_meta_db.sql

      2016-01-25 16:21:34.150   error  : Backend hangup error handling.
      2016-01-25 16:21:34.150   error  : Backend hangup error handling.
      2016-01-25 19:04:29.502   error  : Backend hangup error handling.
      2016-01-25 19:04:29.503   error  : Backend hangup error handling.
      2016-01-25 19:34:53.271   error  : Backend hangup error handling.
      2016-01-25 19:34:53.271   error  : Backend hangup error handling.
      2016-01-25 19:34:53.273   error  : Backend hangup error handling.
      2016-01-25 19:34:53.273   error  : Backend hangup error handling.
      2016-01-25 20:06:34.445   error  : Backend hangup error handling.
      2016-01-25 20:06:34.449   error  : Backend hangup error handling.
      2016-01-25 20:06:34.452   error  : Backend hangup error handling.
      2016-01-25 20:06:45.636   error  : Backend hangup error handling.
      2016-01-25 20:07:26.176   error  : Backend hangup error handling.
      2016-01-25 20:11:53.358   error  : debug assert /home/vagrant/workspace/server/modules/routing/readwritesplit/readwritesplit.c:1689 Dcb under- or overflow 
      2016-01-25 20:11:53.358   error  : Fatal: MaxScale 1.3.0 received fatal signal 6. Attempting backtrace.
      2016-01-25 20:11:53.358   error  : Commit ID: 99f39cb213511b95552c49052e77a12e46090129 System name: Linux Release string: Ubuntu 14.04.3 LTS Embedded library version: 5.5.42-MariaDB
      2016-01-25 20:11:53.359   error  :   /usr/bin/maxscale() [0x54cb9e] 
      2016-01-25 20:11:53.359   error  :   /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7fd6263c7340] 
      2016-01-25 20:11:53.359   error  :   /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7fd624ffecc9] 
      2016-01-25 20:11:53.359   error  :   /lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7fd6250020d8] 
      2016-01-25 20:11:53.359   error  :   /lib/x86_64-linux-gnu/libc.so.6(+0x2fb86) [0x7fd624ff7b86] 
      2016-01-25 20:11:53.359   error  :   /lib/x86_64-linux-gnu/libc.so.6(+0x2fc32) [0x7fd624ff7c32] 
      2016-01-25 20:11:53.359   error  :   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(+0x4c67) [0x7fd61d420c67] 
      2016-01-25 20:11:53.359   error  :   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(+0x62a0) [0x7fd61d4222a0] 
      2016-01-25 20:11:53.359   error  :   /usr/lib/x86_64-linux-gnu/maxscale/libreadwritesplit.so(+0x5a80) [0x7fd61d421a80] 
      2016-01-25 20:11:53.360   error  :   /usr/lib/x86_64-linux-gnu/maxscale/libMySQLClient.so(+0x751c) [0x7fd608e2251c] 
      2016-01-25 20:11:53.360   error  :   /usr/lib/x86_64-linux-gnu/maxscale/libMySQLClient.so(+0x51fa) [0x7fd608e201fa] 
      2016-01-25 20:11:53.360   error  :   /usr/bin/maxscale() [0x565b59] 
      2016-01-25 20:11:53.360   error  :   /usr/bin/maxscale(poll_waitevents+0x67e) [0x565137] 
      2016-01-25 20:11:53.360   error  :   /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fd6263bf182] 
      2016-01-25 20:11:53.360   error  :   /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fd6250c247d] 

      /var/log/syslog from one of the Galera DB nodes:

      Jan 25 16:21:34 wsccms-db05f-prod mysqld: 160125 16:21:34 [Warning] Aborted connection 694195 to db: 'ops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
      Jan 25 19:04:29 wsccms-db05f-prod mysqld: 160125 19:04:29 [Warning] Aborted connection 704612 to db: 'ops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
      Jan 25 19:34:53 wsccms-db05f-prod mysqld: 160125 19:34:53 [Warning] Aborted connection 706553 to db: 'ops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
      Jan 25 20:06:34 wsccms-db05f-prod mysqld: 160125 20:06:34 [Warning] Aborted connection 708905 to db: 'ops_meta_db' user: 'admin' host: '10.0.249.211' (Unknown error)
      Jan 25 20:11:53 wsccms-db05f-prod mysqld: 160125 20:11:53 [Warning] Aborted connection 331417 to db: 'unconnected' user: 'maxscalemon' host: '10.0.249.211' (Unknown error)

      [MaxScale]
      threads=4
      syslog=1
      maxlog=1
      ms_timestamp=1
      log_to_shm=0
      log_warning=1
      log_notice=1
      log_info=0
      log_debug=0
       
      [CLI_listener]
      type=listener
      service=CLI
      protocol=maxscaled
      address=localhost
      port=6603
       
      [CLI]
      type=service
      router=cli
       
      [debug]
      type=service
      router=debugcli
      router_options=user
       
      [debug_listener]
      type=listener
      service=debug
      protocol=telnetd
      port=4442
       
      [galera_5_monitor]
      type=monitor
      module=galeramon
      servers=dbf1,dbf2,dbf3,dbf4,dbf5
      user=maxscalemon
      passwd=euxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      monitor_interval=2000
      disable_master_failback=1
      available_when_donor=1
      disable_master_role_setting=0
       
      [galera_10_monitor]
      type=monitor
      module=galeramon
      servers=dbg1,dbg2,dbg3,dbg4,dbg5
      user=maxscalemon
      passwd=exxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      monitor_interval=2000
      disable_master_failback=1
      available_when_donor=1
      disable_master_role_setting=0
       
       
      [galera_rr_service]
      type=service
      router=readconnroute
      #connection_timeout=300
      router_options=synced
      servers=dbf1,dbf2,dbf3,dbf4,dbf5
      user=maxscaleroute
      passwd=Koxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      enable_root_user=0
      version_string=5.5.47-MariaDB-RR
      #optimize_wildcard=true
      log_auth_warnings=true
       
      [galera_rr10_service]
      type=service
      router=readconnroute
      #connection_timeout=300
      router_options=synced
      servers=dbg1,dbg2,dbg3,dbg4,dbg5
      user=maxscaleroute
      passwd=Koxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      enable_root_user=0
      version_string=10.1-MariaDB-RR
      #optimize_wildcard=true
      log_auth_warnings=true
       
       
      [galera_rw_service]
      type=service
      router=readwritesplit
      #connection_timeout=300
      router_options=slave_selection_criteria=LEAST_CURRENT_OPERATIONS,disable_sescmd_history=true
      max_slave_connections=100%
      servers=dbf1,dbf2,dbf3,dbf4,dbf5
      user=maxscaleroute
      passwd=Kohxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      enable_root_user=0
      version_string=5.5.47-MariaDB-RWsplit
      #optimize_wildcard=true
      log_auth_warnings=true
       
      [galera_rw10_service]
      type=service
      router=readwritesplit
      #connection_timeout=300
      router_options=slave_selection_criteria=LEAST_CURRENT_OPERATIONS,disable_sescmd_history=true
      max_slave_connections=100%
      servers=dbg1,dbg2,dbg3,dbg4,dbg5
      user=maxscaleroute
      passwd=Koxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      enable_root_user=0
      version_string=10.1-MariaDB-RWsplit
      #optimize_wildcard=true
      log_auth_warnings=true
       
       
      [galera_rr_listener]
      type=listener
      service=galera_rr_service
      protocol=MySQLClient
      address=10.0.248.201
      port=3306
       
      [galera_rw_listener]
      type=listener
      service=galera_rw_service
      protocol=MySQLClient
      address=10.0.248.202
      port=3306
       
      [galera_rr10_listener]
      type=listener
      service=galera_rr10_service
      protocol=MySQLClient
      address=10.0.248.205
      port=3306
       
      [galera_rw10_listener]
      type=listener
      service=galera_rw10_service
      protocol=MySQLClient
      address=10.0.248.206
      port=3306
       
      [dbf1]
      type=server
      address=10.0.249.11
      port=3306
      protocol=MySQLBackend
       
      [dbf2]
      type=server
      address=10.0.249.12
      port=3306
      protocol=MySQLBackend
       
      [dbf3]
      type=server
      address=10.0.249.13
      port=3306
      protocol=MySQLBackend
       
      [dbf4]
      type=server
      address=10.0.249.14
      port=3306
      protocol=MySQLBackend
       
      [dbf5]
      type=server
      address=10.0.249.15
      port=3306
      protocol=MySQLBackend
       
      [dbg1]
      type=server
      address=10.0.249.16
      port=3306
      protocol=MySQLBackend
       
      [dbg2]
      type=server
      address=10.0.249.17
      port=3306
      protocol=MySQLBackend
       
      [dbg3]
      type=server
      address=10.0.249.18
      port=3306
      protocol=MySQLBackend
       
      [dbg4]
      type=server
      address=10.0.249.19
      port=3306
      protocol=MySQLBackend
       
      [dbg5]
      type=server
      address=10.0.249.20
      port=3306
      protocol=MySQLBackend

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markus makela markus makela
                Reporter:
                markus makela markus makela
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: