[MXS-3230] Primary/Replica MaxScale crashes during Bulk Insert Created: 2020-10-09  Updated: 2020-10-12  Resolved: 2020-10-12

Status: Closed
Project: MariaDB MaxScale
Component/s: N/A
Affects Version/s: 2.5.3
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Jens Röwekamp (Inactive) Assignee: markus makela
Resolution: Duplicate Votes: 0
Labels: Maxscale
Environment:

Test environment


Issue Links:
Duplicate
duplicates MXS-3218 Crash with LOAD DATA LOCAL INFILE Closed

 Description   

Following MaxScale images crash during bulk ingestions:

  • mariadb/maxscale:2.5.3-1
  • mariadb/maxscale:2.5.3-2
  • mariadb/maxscale:2.5.4-1

Following MaxScale images don't crash during bulk ingestions:

  • mariadb/maxscale:2.4.10-1

How to reproduce:
Execute following script locally against a SkySQL 10.5 primary/replica topology.

#!/bin/bash
 
MARIADB_HOST='test-jens-01.mdb0001554.test.skysql.net'
MARIADB_PORT='5001'
MARIADB_USER='skysql_admin'
MARIADB_PASSWORD='please replace me'
 
# define MariaDB client
MARIADB_CLIENT="mariadb -h ${MARIADB_HOST} -P ${MARIADB_PORT} -u ${MARIADB_USER} -p${MARIADB_PASSWORD} --ssl"
 
# temporary data store
mkdir -p /tmp/bookstore
cd /tmp/bookstore
 
# download online data
curl https://www.dropbox.com/s/ikmy15xp5nk684v/books5001xpand.tar?dl=0 -J -L -k | tar -x 
 
# expand csv files
gzip -d *.gz
 
# Execute the inject
echo "DROP DATABASE IF EXISTS test; CREATE DATABASE test;" | ${MARIADB_CLIENT}
echo "set global local_infile=ON ;" | ${MARIADB_CLIENT}
sed -e "s/%DB%/test/g" 01_load_tx_init.sql | ${MARIADB_CLIENT}
sed -e "s/%DB%/test/g" -e "s/%CSV%/$(pwd | sed -e 's/\//\\\//g')\\//g" 02_load_tx_ldi.sql | ${MARIADB_CLIENT}
 
# Cleanup
rm -rf /tmp/bookstore

Client side:

jens@ubuntu18:~$ ./DBAAS-5327-reproduce.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  1034    0  1034    0     0    702      0 --:--:--  0:00:01 --:--:--  1576
100  291M  100  291M    0     0  13.2M      0  0:00:21  0:00:21 --:--:-- 15.2M
ERROR 2013 (HY000) at line 34: Lost connection to MySQL server during query
jens@ubuntu18:~$ mariadb -V
mariadb  Ver 15.1 Distrib 10.5.5-3-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Tests were reproducible with the 10.4 MariaDB client as well.

MaxScale side:

test-jump:~ $ kubectl logs -f test-jens-01-mdb-mxs-55cfb7cccc-lnwmc maxscale
2020/10/09 13:36:19 start maxscale called
info   : MaxScale will be run in the terminal process.
2020-10-09 13:36:19   notice : syslog logging is enabled.
2020-10-09 13:36:19   notice : maxlog logging is enabled.
2020-10-09 13:36:19   notice : The collection of SQLite memory allocation statistics turned off.
2020-10-09 13:36:19   notice : Threading mode of SQLite set to Multi-thread.
2020-10-09 13:36:19   notice : Worker message queue size: 1MiB
2020-10-09 13:36:19   warning: users_refresh_time: Specifying durations without a suffix denoting the unit has been deprecated: 0. Use the suffixes 'h' (hour), 'm' (minute) 's' (second) or 'ms' (milliseconds).
2020-10-09 13:36:19   notice : Using up to 2.2GiB of memory for query classifier cache
2020-10-09 13:36:19   notice : Running OS: Linux@4.19.112+, #1 SMP Thu May 21 12:32:38 PDT 2020, x86_64 with 4 processor cores.
2020-10-09 13:36:19   notice : Total usable main memory: 14.68GiB.
2020-10-09 13:36:19   notice : MariaDB MaxScale 2.5.3 started (Commit: de3770579523e8115da79b1696e600cce1087664)
2020-10-09 13:36:19   notice : MaxScale is running in process 231
2020-10-09 13:36:19   notice : Configuration file: /etc/maxscale-cfg/maxscale.cnf
2020-10-09 13:36:19   notice : Log directory: /var/log/maxscale
2020-10-09 13:36:19   notice : Data directory: /var/lib/maxscale
2020-10-09 13:36:19   notice : Module directory: /usr/lib64/maxscale
2020-10-09 13:36:19   notice : Service cache: /var/cache/maxscale
 
Configuration file : /etc/maxscale-cfg/maxscale.cnf
Log directory      : /var/log/maxscale
Data directory     : /var/lib/maxscale
Module directory   : /usr/lib64/maxscale
Service cache      : /var/cache/maxscale
 
2020-10-09 13:36:19   notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
2020-10-09 13:36:19   notice : Query classification results are cached and reused. Memory used per thread: 563.88MiB
2020-10-09 13:36:19   notice : Password encryption key file '/var/lib/maxscale/.secrets' not found, using configured passwords as plaintext.
2020-10-09 13:36:19   notice : MaxScale started with 4 worker threads, each with a stack size of 8388608 bytes.
2020-10-09 13:36:19   notice : Loading /etc/maxscale-cfg/maxscale.cnf.
2020-10-09 13:36:19   notice : /etc/maxscale-cfg/maxscale.cnf.d does not exist, not reading.
2020-10-09 13:36:19   notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
2020-10-09 13:36:19   notice : Loaded module readconnroute: V2.0.0 from /usr/lib64/maxscale/libreadconnroute.so
2020-10-09 13:36:19   notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
2020-10-09 13:36:19   notice : Loaded module mariadbmon: V1.5.0 from /usr/lib64/maxscale/libmariadbmon.so
2020-10-09 13:36:19   notice : (Read-Write-Listener) Loaded module MariaDBAuth: V2.1.0 from /usr/lib64/maxscale/libmariadbauth.so
2020-10-09 13:36:19   warning: The MaxScale GUI is enabled but encryption for the REST API is not enabled, the GUI will not be enabled. Configure `admin_ssl_key` and `admin_ssl_cert` to enable HTTPS or add `admin_secure_gui=false` to allow use of the GUI without encryption.
2020-10-09 13:36:19   notice : Started REST API on [0.0.0.0]:8989
2020-10-09 13:36:19   warning: [mariadbmon] Tried to find a master but no valid master server found.
2020-10-09 13:36:19   notice : Starting a total of 2 services...
2020-10-09 13:36:19   notice : (Read-Write-Listener) Listening for connections at [0.0.0.0]:4006
2020-10-09 13:36:19   warning: Service 'Read-Write-Service' has a listener but no servers
2020-10-09 13:36:19   notice : Service 'Read-Write-Service' started (1/2)
2020-10-09 13:36:19   notice : (Read-Only-Listener) Listening for connections at [0.0.0.0]:4008
2020-10-09 13:36:19   warning: Service 'Read-Only-Service' has a listener but no servers
2020-10-09 13:36:19   notice : Service 'Read-Only-Service' started (2/2)
2020-10-09 13:36:20   warning: Authentication failed for 'skysql_admin', using password. Request: GET /v1/maxscale
2020-10-09 13:36:23   warning: Authentication failed for 'skysql_admin', using password. Request: GET /v1/maxscale
2020-10-09 13:36:24   notice : Create network user 'skysql_admin'
OK
2020-10-09 13:36:25   notice : Deleted network user 'admin'
OK
2020-10-09 13:36:26   notice : Value of passive changed to false
2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'MariaDB-Monitor'
2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'Read-Only-Service'
2020-10-09 13:36:26   notice : Added '10.0.0.227' to 'Read-Write-Service'
2020-10-09 13:36:26   notice : Server '10.0.0.227' charset: utf8mb4
2020-10-09 13:36:26   notice : Server '10.0.0.227' version: 10.5.5-3-MariaDB-enterprise-log
2020-10-09 13:36:26   warning: [mariadbmon] Tried to find a master but no valid master server found.
2020-10-09 13:36:26   warning: [mariadbmon] '10.0.0.227' is not a valid master candidate because it's read_only.
2020-10-09 13:36:26   notice : Server changed state: 10.0.0.227[10.0.0.227:3306]: server_up. [Down] -> [Slave of External Server, Running]
2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'MariaDB-Monitor'
2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'Read-Only-Service'
2020-10-09 13:36:26   notice : Added '10.0.1.3' to 'Read-Write-Service'
2020-10-09 13:36:26   notice : Server '10.0.1.3' charset: utf8mb4
2020-10-09 13:36:26   notice : Server '10.0.1.3' version: 10.5.5-3-MariaDB-enterprise-log
2020-10-09 13:36:26   warning: [mariadbmon] Tried to find a master but no valid master server found.
2020-10-09 13:36:26   warning: [mariadbmon] '10.0.0.227' is not a valid master candidate because it's read_only.
'10.0.1.3' is not a valid master candidate because it's read_only.
2020-10-09 13:36:26   notice : Server changed state: 10.0.1.3[10.0.1.3:3306]: server_up. [Down] -> [Slave of External Server, Running]
2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'MariaDB-Monitor'
2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'Read-Only-Service'
2020-10-09 13:36:26   notice : Added '10.0.0.195' to 'Read-Write-Service'
2020-10-09 13:36:26   notice : Server '10.0.0.195' charset: utf8mb4
2020-10-09 13:36:26   notice : Server '10.0.0.195' version: 10.5.5-3-MariaDB-enterprise-log
2020-10-09 13:36:26   notice : [mariadbmon] Selecting new master server.
2020-10-09 13:36:26   notice : [mariadbmon] Setting '10.0.0.195' as master.
2020-10-09 13:36:26   notice : Server changed state: 10.0.0.227[10.0.0.227:3306]: new_slave. [Slave of External Server, Running] -> [Slave, Running]
2020-10-09 13:36:26   notice : Server changed state: 10.0.1.3[10.0.1.3:3306]: new_slave. [Slave of External Server, Running] -> [Slave, Running]
2020-10-09 13:36:26   notice : Server changed state: 10.0.0.195[10.0.0.195:3306]: master_up. [Down] -> [Master, Running]
2020-10-09 13:36:26   notice : Read 21 user@host entries from '10.0.0.195' for service 'Read-Write-Service'.
2020-10-09 13:36:26   notice : Read 21 user@host entries from '10.0.0.195' for service 'Read-Only-Service'.
alert  : MaxScale 2.5.3 received fatal signal 11. Commit ID: de3770579523e8115da79b1696e600cce1087664 System name: Linux Release string: CentOS Linux release 8.2.2004 (Core)
 
 
2020-10-09 14:19:38   error  : (102) [qc_sqlite] The provided buffer does not contain a COM_QUERY, but a COM_UNKNOWN(61).
2020-10-09 14:19:38   error  : (102) [qc_sqlite] The query could not be parsed. Response not valid.
2020-10-09 14:19:38   alert  : (102) MaxScale 2.5.3 received fatal signal 11. Commit ID: de3770579523e8115da79b1696e600cce1087664 System name: Linux Release string: CentOS Linux release 8.2.2004 (Core)
2020-10-09 14:19:38   alert  : (102) Statement currently being classified: none/unknown
alert  : Writing core dump.
2020-10-09 14:19:39   alert  : (102)
  /usr/lib64/maxscale/libqc_sqlite.so(+0xc653): query_classifier/qc_sqlite/qc_sqlite.cc:248
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_Z21qc_get_database_namesB5cxx11P5GWBUF+0x281): server/core/query_classifier.cc:219
  /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection19handle_use_databaseEP5GWBUF+0x19): /usr/include/c++/8/bits/stl_vector.h:895
  /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection24process_special_commandsEP5GWBUFh+0x1a3): server/modules/protocol/MariaDB/mariadb_client.cc:1174
  /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection15route_statementEON8maxscale6BufferE+0x65): server/modules/protocol/MariaDB/mariadb_client.cc:1230
  /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection19process_normal_readEv+0x352): server/modules/protocol/MariaDB/mariadb_client.cc:1335
  /usr/lib64/maxscale/libmariadbclient.so(_ZN23MariaDBClientConnection17ready_for_readingEP3DCB+0x30): server/modules/protocol/MariaDB/mariadb_client.cc:1426
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB14process_eventsEj+0xcc): server/core/dcb.cc:1299
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN3DCB13event_handlerEPS_j+0x49): server/core/dcb.cc:1357
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1be): maxutils/maxbase/src/worker.cc:879
  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x53): maxutils/maxbase/src/worker.cc:574
  /lib64/libstdc++.so.6(+0xc2b73): ??:?
  /lib64/libpthread.so.0(+0x82de): pthread_create.c:?
  /lib64/libc.so.6(clone+0x43): :?



 Comments   
Comment by markus makela [ 2020-10-11 ]

Looks like a duplicate of MXS-3218.

Generated at Thu Feb 08 04:19:54 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.