Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-3230

Primary/Replica MaxScale crashes during Bulk Insert

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.5.3
    • Fix Version/s: N/A
    • Component/s: N/A
    • Labels:
    • Environment:
      Test environment

      Description

      Following MaxScale images crash during bulk ingestions:

      • mariadb/maxscale:2.5.3-1
      • mariadb/maxscale:2.5.3-2
      • mariadb/maxscale:2.5.4-1

      Following MaxScale images don't crash during bulk ingestions:

      • mariadb/maxscale:2.4.10-1

      How to reproduce:
      Execute following script locally against a SkySQL 10.5 primary/replica topology.

      #!/bin/bash
       
      MARIADB_HOST='test-jens-01.mdb0001554.test.skysql.net'
      MARIADB_PORT='5001'
      MARIADB_USER='skysql_admin'
      MARIADB_PASSWORD='please replace me'
       
      # define MariaDB client
      MARIADB_CLIENT="mariadb -h ${MARIADB_HOST} -P ${MARIADB_PORT} -u ${MARIADB_USER} -p${MARIADB_PASSWORD} --ssl"
       
      # temporary data store
      mkdir -p /tmp/bookstore
      cd /tmp/bookstore
       
      # download online data
      curl https://www.dropbox.com/s/ikmy15xp5nk684v/books5001xpand.tar?dl=0 -J -L -k | tar -x 
       
      # expand csv files
      gzip -d *.gz
       
      # Execute the inject
      echo "DROP DATABASE IF EXISTS test; CREATE DATABASE test;" | ${MARIADB_CLIENT}
      echo "set global local_infile=ON ;" | ${MARIADB_CLIENT}
      sed -e "s/%DB%/test/g" 01_load_tx_init.sql | ${MARIADB_CLIENT}
      sed -e "s/%DB%/test/g" -e "s/%CSV%/$(pwd | sed -e 's/\//\\\//g')\\//g" 02_load_tx_ldi.sql | ${MARIADB_CLIENT}
       
      # Cleanup
      rm -rf /tmp/bookstore
      

      Client side:

      jens@ubuntu18:~$ ./DBAAS-5327-reproduce.sh
        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                       Dload  Upload   Total   Spent    Left  Speed
        0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
        0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      100  1034    0  1034    0     0    702      0 --:--:--  0:00:01 --:--:--  1576
      100  291M  100  291M    0     0  13.2M      0  0:00:21  0:00:21 --:--:-- 15.2M
      ERROR 2013 (HY000) at line 34: Lost connection to MySQL server during query
      jens@ubuntu18:~$ mariadb -V
      mariadb  Ver 15.1 Distrib 10.5.5-3-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
      

      Tests were reproducible with the 10.4 MariaDB client as well.

      MaxScale side:

      test-jump:~ $ kubectl logs -f test-jens-01-mdb-mxs-55cfb7cccc-lnwmc maxscale
      2020/10/09 13:36:19 start maxscale called
      info   : MaxScale will be run in the terminal process.
      2020-10-09 13:36:19   notice : syslog logging is enabled.
      2020-10-09 13:36:19   notice : maxlog logging is enabled.
      2020-10-09 13:36:19   notice : The collection of SQLite memory allocation statistics turned off.
      2020-10-09 13:36:19   notice : Threading mode of SQLite set to Multi-thread.
      2020-10-09 13:36:19   notice : Worker message queue size: 1MiB
      2020-10-09 13:36:19   warning: users_refresh_time: Specifying durations without a suffix denoting the unit has been deprecated: 0. Use the suffixes 'h' (hour), 'm' (minute) 's' (second) or 'ms' (milliseconds).
      2020-10-09 13:36:19   notice : Using up to 2.2GiB of memory for query classifier cache
      2020-10-09 13:36:19   notice : Running OS: Linux@4.19.112+, #1 SMP Thu May 21 12:32:38 PDT 2020, x86_64 with 4 processor cores.
      2020-10-09 13:36:19   notice : Total usable main memory: 14.68GiB.
      2020-10-09 13:36:19   notice : MariaDB MaxScale 2.5.3 started (Commit: de3770579523e8115da79b1696e600cce1087664)
      2020-10-09 13:36:19   notice : MaxScale is running in process 231
      2020-10-09 13:36:19   notice : Configuration file: /etc/maxscale-cfg/maxscale.cnf
      2020-10-09 13:36:19   notice : Log directory: /var/log/maxscale
      2020-10-09 13:36:19   notice : Data directory: /var/lib/maxscale
      2020-10-09 13:36:19   notice : Module directory: /usr/lib64/maxscale
      2020-10-09 13:36:19   notice : Service cache: /var/cache/maxscale
       
      Configuration file : /etc/maxscale-cfg/maxscale.cnf
      Log directory      : /var/log/maxscale
      Data directory     : /var/lib/maxscale
      Module directory   : /usr/lib64/maxscale
      Service cache      : /var/cache/maxscale
       
      2020-10-09 13:36:19   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
      2020-10-09 13:36:19   notice : Query classification results are cached and reused. Memory used per thread: 563.88MiB
      2020-10-09 13:36:19   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
      2020-10-09 13:36:19   notice : MaxScale started with 4 worker threads, each with a stack size of 8388608 bytes.
      2020-10-09 13:36:19   notice : Loading /etc/maxscale-cfg/maxscale.cnf.
      2020-10-09 13:36:19   notice : /etc/maxscale-cfg/maxscale.cnf.d does not exist, not reading.
      2020-10-09 13:36:19   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
      2020-10-09 13:36:19   notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
      2020-10-09 13:36:19   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
      2020-10-09 13:36:19   notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so
      2020-10-09 13:36:19   notice : (Read-Write-Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib64/maxscale/libmariadbauth.so
      2020-10-09 13:36:19   warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
      2020-10-09 13:36:19   notice : Started REST API on [0.0.0.0]:8989
      2020-10-09 13:36:19   warning: [mariadbmon] Tried to find a master but no valid master server found.
      2020-10-09 13:36:19   notice : Starting a total of 2 services...
      2020-10-09 13:36:19   notice : (Read-Write-Listener) Listening for connections at [0.0.0.0]:4006
      2020-10-09 13:36:19   warning: Service 'Read-Write-Service' has a listener but no servers
      2020-10-09 13:36:19   notice : Service 'Read-Write-Service' started (1/2)
      2020-10-09 13:36:19   notice : (Read-Only-Listener) Listening for connections at [0.0.0.0]:4008
      2020-10-09 13:36:19   warning: Service 'Read-Only-Service' has a listener but no servers
      2020-10-09 13:36:19   notice : Service 'Read-Only-Service' started (2/2)
      2020-10-09 13:36:20   warning: Authentication failed for 'skysql_admin', using password. Request: GET /v1/maxscale
      2020-10-09 13:36:23   warning: Authentication failed for 'skysql_admin', using password. Request: GET /v1/maxscale
      2020-10-09 13:36:24   notice : Create network user 'skysql_admin'
      OK
      2020-10-09 13:36:25   notice : Deleted network user 'admin'
      OK
      2020-10-09 13:36:26   notice : Value of passive changed to false
      2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'MariaDB-Monitor'
      2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'Read-Only-Service'
      2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'Read-Write-Service'
      2020-10-09 13:36:26   notice : Server '10.0.0.227' charset: utf8mb4
      2020-10-09 13:36:26   notice : Server '10.0.0.227' version: 10.5.5-3-MariaDB-enterprise-log
      2020-10-09 13:36:26   warning: [mariadbmon] Tried to find a master but no valid master server found.
      2020-10-09 13:36:26   warning: [mariadbmon] '10.0.0.227' is not a valid master candidate because it's read_only.
      2020-10-09 13:36:26   notice : Server changed state: 10.0.0.227[10.0.0.227:3306]: server_up. [Down] -> [Slave of External Server, Running]
      2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'MariaDB-Monitor'
      2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'Read-Only-Service'
      2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'Read-Write-Service'
      2020-10-09 13:36:26   notice : Server '10.0.1.3' charset: utf8mb4
      2020-10-09 13:36:26   notice : Server '10.0.1.3' version: 10.5.5-3-MariaDB-enterprise-log
      2020-10-09 13:36:26   warning: [mariadbmon] Tried to find a master but no valid master server found.
      2020-10-09 13:36:26   warning: [mariadbmon] '10.0.0.227' is not a valid master candidate because it's read_only.
      '10.0.1.3' is not a valid master candidate because it's read_only.
      2020-10-09 13:36:26   notice : Server changed state: 10.0.1.3[10.0.1.3:3306]: server_up. [Down] -> [Slave of External Server, Running]
      2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'MariaDB-Monitor'
      2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'Read-Only-Service'
      2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'Read-Write-Service'
      2020-10-09 13:36:26   notice : Server '10.0.0.195' charset: utf8mb4
      2020-10-09 13:36:26   notice : Server '10.0.0.195' version: 10.5.5-3-MariaDB-enterprise-log
      2020-10-09 13:36:26   notice : [mariadbmon] Selecting new master server.
      2020-10-09 13:36:26   notice : [mariadbmon] Setting '10.0.0.195' as master.
      2020-10-09 13:36:26   notice : Server changed state: 10.0.0.227[10.0.0.227:3306]: new_slave. [Slave of External Server, Running] -> [Slave, Running]
      2020-10-09 13:36:26   notice : Server changed state: 10.0.1.3[10.0.1.3:3306]: new_slave. [Slave of External Server, Running] -> [Slave, Running]
      2020-10-09 13:36:26   notice : Server changed state: 10.0.0.195[10.0.0.195:3306]: master_up. [Down] -> [Master, Running]
      2020-10-09 13:36:26   notice : Read 21 user@host entries from '10.0.0.195' for service 'Read-Write-Service'.
      2020-10-09 13:36:26   notice : Read 21 user@host entries from '10.0.0.195' for service 'Read-Only-Service'.
      alert  : MaxScale 2.5.3 received fatal signal 11. Commit ID: de3770579523e8115da79b1696e600cce1087664 System name: Linux Release string: CentOS Linux release 8.2.2004 (Core)
       
       
      2020-10-09 14:19:38   error  : (102) [qc_sqlite] The provided buffer does not contain a COM_QUERY, but a COM_UNKNOWN(61).
      2020-10-09 14:19:38   error  : (102) [qc_sqlite] The query could not be parsed. Response not valid.
      2020-10-09 14:19:38   alert  : (102) MaxScale 2.5.3 received fatal signal 11. Commit ID: de3770579523e8115da79b1696e600cce1087664 System name: Linux Release string: CentOS Linux release 8.2.2004 (Core)
      2020-10-09 14:19:38   alert  : (102) Statement currently being classified: none/unknown
      alert  : Writing core dump.
      2020-10-09 14:19:39   alert  : (102)
        /usr/lib64/maxscale/libqc_sqlite.so(+0xc653): query_classifier/qc_sqlite/qc_sqlite.cc:248
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_Z21qc_get_database_namesB5cxx11P5GWBUF+0x281): server/core/query_classifier.cc:219
        /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection19handle_use_databaseEP5GWBUF+0x19): /usr/include/c++/8/bits/stl_vector.h:895
        /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection24process_special_commandsEP5GWBUFh+0x1a3): server/modules/protocol/MariaDB/mariadb_client.cc:1174
        /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection15route_statementEON8maxscale6BufferE+0x65): server/modules/protocol/MariaDB/mariadb_client.cc:1230
        /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection19process_normal_readEv+0x352): server/modules/protocol/MariaDB/mariadb_client.cc:1335
        /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection17ready_for_readingEP3DCB+0x30): server/modules/protocol/MariaDB/mariadb_client.cc:1426
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1299
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x49): server/core/dcb.cc:1357
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
        /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
        /lib64/libstdc++.so.6(+0xc2b73): ??:?
        /lib64/libpthread.so.0(+0x82de): pthread_create.c:?
        /lib64/libc.so.6(clone+0x43): :?
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              markus makela markus makela
              Reporter:
              jens.rowekamp Jens Röwekamp
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: