[MXS-183] MaxScale crash after 'reload config' Created: 2015-06-08  Updated: 2015-12-23  Resolved: 2015-12-23

Status: Closed
Project: MariaDB MaxScale
Component/s: maxadmin
Affects Version/s: 1.1.1
Fix Version/s: 1.3.0

Type: Bug Priority: Critical
Reporter: aftab khan Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

test


 Description   

Maxscale crashes when a new database connection is made after 'reload config' command.

Steps to reproduce this issue:

1. Update Maxscale.cnf file
e.g.

++ max_slave_connections=70%
 
-- max_slave_connections=50%

2. Reload configuration using maxadmin

/usr/local/mariadb-maxscale/bin/maxadmin -pmariadb "reload config"
 
Reloading configuration from file.

3. Connect to mysql 

mysql -h10.10.16.11 -udba -p -P4306
 
Enter password:
 
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 0 "Internal error/check (Not system error)"

this would cause maxscale to crash:

2015-06-08 11:26:46   Fatal: MaxScale received fatal signal 11. Attempting backtrace.
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/bin/maxscale() [0x558658]
 
2015-06-08 11:26:46     /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f9f48137cb0]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/modules/libreadwritesplit.so(+0x9b9a) [0x7f9f2d15bb9a]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/modules/libreadwritesplit.so(+0x277a) [0x7f9f2d15477a]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/bin/maxscale(session_alloc+0x1ed) [0x562f89]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/modules/libMySQLClient.so(+0x36cf) [0x7f9f2c33a6cf]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/bin/maxscale() [0x56b3c4]
 
2015-06-08 11:26:46     /usr/local/mariadb-maxscale/bin/maxscale(poll_waitevents+0x61b) [0x56ac8d]
 
2015-06-08 11:26:46     /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f9f4812fe9a]
 
2015-06-08 11:26:46     /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9f46e408bd]



 Comments   
Comment by Dipti Joshi (Inactive) [ 2015-06-08 ]

Which MaxScale version are you using ?

Comment by aftab khan [ 2015-06-08 ] 

$ apt-cache show maxscale
 
Package: maxscale
 
Version: 1.1.1
 
Architecture: amd64
 
Maintainer: MariaDB Corporation Ab
 
Installed-Size: 76132
 
Depends: libaio1 (>= 0.3.93), libc6 (>= 2.14), libcurl3 (>= 7.16.2-1), libgcc1 (>= 1:4.1.1), libssl1.0.0 (>= 1.0.0), libstdc++6 (>= 4.1.1)
 
Filename: dists/precise/main/binary-amd64//maxscale-1.1.1.deb
 
Size: 28256084
 
MD5sum: ec52bf0ac7bf1333b0bc3d6249b3b046
 
SHA1: 370e21bffcde4da3d2a52dbd2b1d9e4586a62480
 
SHA256: 812c76876068bde34a8e43ccc0379581e2adfd5e44c8ed1c2ed48b647d687d6a
 
Section: devel
 
Priority: optional
 
Description: MaxScale

On 8 Jun 2015 17:19, "Dipti Joshi (JIRA)" <jira@mariadb.atlassian.net>

Comment by Dipti Joshi (Inactive) [ 2015-06-11 ]

markus makela Please test this for 1.2.0 - if this is fixed as part of 1.2.0, we can close this.

Comment by markus makela [ 2015-06-11 ]

Could not reproduce by changing the max_slave_connections value and issuing a config reload. If possible, please provide the MaxScale.cnf file so it can be tested.

Since the 1.2 release will contain the update for dynamic configuration, this bug will be obsolete.

Comment by aftab khan [ 2015-06-11 ]

Maxscale.cnf

[maxscale]
 
threads=1
 
 
 
#[Galera Service]
 
#type=service
 
#router=readconnroute
 
#router_options=synced
 
#servers=dbserv1, dbserv2, dbserv3
 
#user=maxscale
 
#passwd=96F99AA1315BDC3604B006F427DD9484
 
 
 
 
 
[RWSplit Service]
 
type=service
 
router=readwritesplit
 
max_slave_connections=50%
 
servers=dbserv1, dbserv2, dbserv3
 
user=maxscale
 
passwd=96F99AA1315BDC3604B006F427DD9484
 
filters=qla|fetch|from
 
 
 
[RWSplit Listener]
 
type=listener
 
service=RWSplit Service
 
protocol=MySQLClient
 
port=4306
 
socket=/tmp/DB.Cluster
 
 
 
[dbserv1]
 
type=server
 
address=10.10.16.216
 
port=3306
 
protocol=MySQLBackend
 
[dbserv2]
 
type=server
 
address=10.10.16.217
 
port=3306
 
protocol=MySQLBackend
 
[dbserv3]
 
type=server
 
address=10.10.16.218
 
port=3306
 
protocol=MySQLBackend
 
 
 
[Galera Monitor]
 
type=monitor
 
module=galeramon
 
servers=dbserv1, dbserv2, dbserv3
 
user=maxscale
 
passwd=96F99AA1315BDC3604B006F427DD9484
 
 
 
[CLI]
 
type=service
 
router=cli
 
 
 
[CLI Listener]
 
type=listener
 
service=CLI
 
protocol=maxscaled
 
address=localhost
 
port=6603
 
 
 
[counter]
 
type=filter
 
module=testfilter
 
 

Comment by Florian Engelmann [ 2015-12-23 ]

still happens to me with Maxscale 1.2.1 ubuntu 14.04.3 LTS

apt-cache show maxscale
 

Package: maxscale
 
Version: 1.2.1
 
Architecture: amd64
 
Maintainer: MariaDB Corporation Ab
 
Installed-Size: 78563
 
Depends: libaio1 (>= 0.3.93), libc6 (>= 2.14), libcurl3 (>= 7.16.2), libgcc1 (>= 1:4.1.1), libssl1.0.0 (>= 1.0.1), libstdc++6 (>= 4.1.1)
 
Filename: dists/trusty/main/binary-amd64//maxscale-1.2.1-1.ubuntu_trusty.x86_64.deb
 
Size: 29452694
 
MD5sum: 1ba7e151ddf3f2c6900137b87d1c1bca
 
SHA1: bb1b4916dcb05639665053204ed8c4b62c5885e6
 
SHA256: 40534e0f42ef291a7286cb6d4596bf0b96a8892ab75e037780b026325148e15a
 
Section: devel
 
Priority: optional
 
Description: MaxScale
 
Description-md5: 3cb6f3b4fb6ec25683a17d09f584adcd
 

maxadmin -pmariadb reload config

/var/log/maxscale/debug1.log

 
2015-12-23 10:32:01.314   140311883937536 [poll_waitevents] epoll_wait found 1 fds
 
2015-12-23 10:32:01.314   140311883937536 [poll_waitevents] event 1 dcb 0x2848040 role DCB_ROLE_SERVICE_LISTENER
 
2015-12-23 10:32:01.314   140311883937536 [poll_waitevents] Accept in fd 12
 
2015-12-23 10:32:01.314   140311883937536 [poll_add_dcb] Added dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING to poll set.
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] epoll_wait found 1 fds
 
2015-12-23 10:32:01.314   140311883937536 [dcb_write] Wrote 4 Bytes to dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] epoll_wait found 1 fds
 
2015-12-23 10:32:01.314   140311883937536 [poll_waitevents] event 4 dcb 0x7f9cd40009c0 role DCB_ROLE_REQUEST_HANDLER
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] event 5 dcb 0x7f9cd40009c0 role DCB_ROLE_REQUEST_HANDLER
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] Read in dcb 0x7f9cd40009c0 fd 21
 
2015-12-23 10:32:01.314   140312169125632 [dcb_read] Read 5 bytes from dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21.
 
2015-12-23 10:32:01.314   140312169125632 [dcb_write] Wrote 8 Bytes to dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] epoll_wait found 1 fds
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] event 5 dcb 0x7f9cd40009c0 role DCB_ROLE_REQUEST_HANDLER
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] Read in dcb 0x7f9cd40009c0 fd 21
 
2015-12-23 10:32:01.314   140312169125632 [dcb_read] Read 7 bytes from dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21.
 
2015-12-23 10:32:01.314   140312169125632 [dcb_write] Wrote 6 Bytes to dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] epoll_wait found 1 fds
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] event 5 dcb 0x7f9cd40009c0 role DCB_ROLE_REQUEST_HANDLER
 
2015-12-23 10:32:01.314   140312169125632 [poll_waitevents] Read in dcb 0x7f9cd40009c0 fd 21
 
2015-12-23 10:32:01.314   140312169125632 [dcb_read] Read 13 bytes from dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21.
 
2015-12-23 10:32:01.314   140312169125632 [dcb_write] Wrote 35 Bytes to dcb 0x7f9cd40009c0 in state DCB_STATE_POLLING fd 21
 
2015-12-23 10:32:01.314   ---	Logging to file is enabled	--

/var/log/maxscale/error1.log

 
2015-12-23 10:32:01.314   Fatal: MaxScale 1.2.1 received fatal signal 11. Attempting backtrace.
 
2015-12-23 10:32:01.314   Commit ID: 7bfda81b098bfd0db3c306495725d1910294d2e8 System name: Linux Release string: Ubuntu 14.04.3 LTS Embedded library version: 5.5.42-MariaDB
 
2015-12-23 10:32:01.315     /usr/bin/maxscale() [0x549855] 
2015-12-23 10:32:01.315     /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f9d1cf19340] 
2015-12-23 10:32:01.315     /usr/bin/maxscale() [0x5651b5] 
2015-12-23 10:32:01.315     /usr/bin/maxscale(config_reload+0x84) [0x561eab] 
2015-12-23 10:32:01.315     /usr/lib/x86_64-linux-gnu/maxscale/libcli.so(+0x7071) [0x7f9d01221071] 
2015-12-23 10:32:01.316     /usr/lib/x86_64-linux-gnu/maxscale/libcli.so(execute_cmd+0x8ae) [0x7f9d012208cd] 
2015-12-23 10:32:01.316     /usr/lib/x86_64-linux-gnu/maxscale/libcli.so(+0x5c54) [0x7f9d0121fc54] 
2015-12-23 10:32:01.316     /usr/lib/x86_64-linux-gnu/maxscale/libmaxscaled.so(+0x36ae) [0x7f9d006076ae] 
2015-12-23 10:32:01.316     /usr/bin/maxscale() [0x55fcca] 
2015-12-23 10:32:01.316     /usr/bin/maxscale(poll_waitevents+0x6d4) [0x55f57b] 
2015-12-23 10:32:01.316     /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f9d1cf11182] 
2015-12-23 10:32:01.316     /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f9d1bc1447d]

/etc/maxscale.cnf

 
[MaxScale]
 
threads=8
 
ms_timestamp=1
 
log_messages=1
 
log_trace=1
 
log_debug=1
 
 
 
[CLI_listener]
 
type=listener
 
service=CLI
 
protocol=maxscaled
 
address=localhost
 
port=6603
 
 
 
[CLI]
 
type=service
 
router=cli
 
 
 
[debug]
 
type=service
 
router=debugcli
 
router_options=user
 
 
 
[debug_listener]
 
type=listener
 
service=debug
 
protocol=telnetd
 
port=4442
 
 
 
[galera_monitor]
 
type=monitor
 
module=galeramon
 
servers=db1,db2,db3,db4,db5
 
user=maxscaleuser
 
passwd=53B89EBxxxxxx57xxxxxxxxxxxxxxxxxxE4E59
 
monitor_interval=2000
 
disable_master_failback=1
 
available_when_donor=1
 
disable_master_role_setting=0
 
 
 
[galera_rr_service]
 
type=service
 
router=readconnroute
 
#connection_timeout=300
 
router_options=synced
 
servers=db1,db2,db3,db4,db5
 
user=maxscaleuser
 
passwd=xxxxxxE91557E2CBxxxxxxxxxxxxD2A86D679B58Exxxx
 
enable_root_user=0
 
 
 
[galera_rw_service]
 
type=service
 
router=readwritesplit
 
#connection_timeout=300
 
router_options="slave_selection_criteria=LEAST_CURRENT_OPERATIONS"
 
max_slave_connections="100%"
 
servers=db1,db2,db3,db4,db5
 
user=maxscaleuser
 
passwd=xxxxE91557xxxxxxx8E4xxxxxxxxxxxxxxxxxxxxxxxxx
 
enable_root_user=0
 
 
 
[galera_rr_listener]
 
type=listener
 
service=galera_rr_service
 
protocol=MySQLClient
 
address=10.0.248.201
 
port=3306
 
 
 
[galera_rw_listener]
 
type=listener
 
service=galera_rw_service
 
protocol=MySQLClient
 
address=10.0.248.202
 
port=3306
 
 
 
[db1]
 
type=server
 
address=10.0.249.11
 
port=3306
 
protocol=MySQLBackend
 
 
 
[db2]
 
type=server
 
address=10.0.249.12
 
port=3306
 
protocol=MySQLBackend
 
 
 
[db3]
 
type=server
 
address=10.0.249.13
 
port=3306
 
protocol=MySQLBackend
 
 
 
[db4]
 
type=server
 
address=10.0.249.14
 
port=3306
 
protocol=MySQLBackend
 
 
 
[db5]
 
type=server
 
address=10.0.249.15
 
port=3306
 
protocol=MySQLBackend

Comment by markus makela [ 2015-12-23 ]

Managed to reproduce this by quoting the values on 1.3.0. Fix is simple since wrong value was used.

Comment by markus makela [ 2015-12-23 ]

Fixed in commit 2b6b8f87dbbcf11f606c6d31f24fc338ef5961fa

Comment by markus makela [ 2015-12-23 ]

Also, the values shouldn't be quoted for any of the parameters.

Comment by Florian Engelmann [ 2015-12-23 ]

Thank you so much! Removing all quotes from my configuration fixed it.

Generated at Thu Feb 08 03:57:24 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.