[MXS-799] fatal signal 11 when socket could not be opened Created: 2016-07-08  Updated: 2016-07-26  Resolved: 2016-07-26

Status: Closed
Project: MariaDB MaxScale
Component/s: Core
Affects Version/s: 1.4.3
Fix Version/s: 2.1.0

Type: Bug Priority: Minor
Reporter: Christian Schnidrig Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7

Issue Links:
Relates
relates to MXS-711 All service ports use the same user data Closed
Sprint: 2016-14

 Description   

Jul 8 11:05:54 ubuntu maxscale[1]: Fatal: MaxScale 1.4.3 received fatal signal 11. Attempting backtrace.
Jul 8 11:05:54 ubuntu maxscale[1]: Commit ID: c2706bab69d1f3b5f1f3ec70f78b1a4803788fec System name: Linux Release string: CentOS Linux release 7.2.1511 (Core) Embedded library version: (null)
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/bin/maxscale() [0x403b80]
Jul 8 11:05:54 ubuntu maxscale[1]: /lib64/libpthread.so.0(+0xf100) [0x7febfae9b100]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(atomic_add+0x12) [0x7febfb7a32be]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(mysql_users_fetch+0x33) [0x7febfb7acb23]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libMySQLClient.so(gw_find_mysql_user_password_sha1+0x19c) [0x7febec053ea6]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libMySQLClient.so(gw_check_mysql_scramble_data+0x134) [0x7febec05572f]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libMySQLClient.so(+0x8957) [0x7febec055957]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libMySQLClient.so(mysql_auth_authenticate+0x139) [0x7febec0550b1]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libMySQLClient.so(+0x3238) [0x7febec050238]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x43f90) [0x7febfb7c2f90]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(poll_waitevents+0x5de) [0x7febfb7c286e]
Jul 8 11:05:54 ubuntu maxscale[1]: /usr/bin/maxscale(worker_thread_main+0x2a) [0x404b7f]
Jul 8 11:05:54 ubuntu maxscale[1]: /lib64/libpthread.so.0(+0x7dc5) [0x7febfae93dc5]
Jul 8 11:05:54 ubuntu maxscale[1]: /lib64/libc.so.6(clone+0x6d) [0x7febf9975ced]
Jul 8 11:05:54 ubuntu kernel: [53550.709817] maxscale[30889]: segfault at 1c ip 00007febfb7a32be sp 00007febd7d4a850 error 6 in libmaxscale-common.so.1.0.0[7febfb77f000+e4000]

===========

When there is a socket configured, which cannot not be opened:

maxscale[1]: Unable to start to listen port 0 for MySQLClient Splitter Service.
maxscale[1]: Loaded module maxscaled: V1.0.0 from /usr/lib64/maxscale/libmaxscaled.so

Then opon trying to connect to maxscale with an mysql client will trigger the crash above.

=========
Log of startup:

Jul 8 11:04:43 ubuntu maxscale[1]: Configuration file: /etc/maxscale.cnf
Jul 8 11:04:43 ubuntu maxscale[1]: Log directory: /var/log/maxscale
Jul 8 11:04:43 ubuntu maxscale[1]: Data directory: /var/lib/maxscale
Jul 8 11:04:43 ubuntu maxscale[1]: Module directory: /usr/lib64/maxscale
Jul 8 11:04:43 ubuntu maxscale[1]: Service cache: /var/cache/maxscale
Jul 8 11:04:43 ubuntu maxscale[1]: Initialise CLI router module V1.0.0.
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
Jul 8 11:04:43 ubuntu maxscale[1]: Initializing statemend-based read/write split router module.
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module readwritesplit: V1.0.2 from /usr/lib64/maxscale/libreadwritesplit.so
Jul 8 11:04:43 ubuntu maxscale[1]: Initialise the MySQL Galera Monitor module V2.0.0.
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module galeramon: V2.0.0 from /usr/lib64/maxscale/libgaleramon.so
Jul 8 11:04:43 ubuntu maxscale[1]: Encrypted password file /var/lib/maxscale/.secrets can't be accessed (No such file or directory). Password encryption is not used.
Jul 8 11:04:43 ubuntu maxscale[1]: No query classifier specified, using default 'qc_mysqlembedded'.
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module qc_mysqlembedded: V1.0.0 from /usr/lib64/maxscale/libqc_mysqlembedded.so
Jul 8 11:04:43 ubuntu maxscale[1]: qc_mysqlembedded loaded.
Jul 8 11:04:43 ubuntu maxscale[1]: Server changed state: 10.0.23.31[10.0.23.31:3306]: new_master
Jul 8 11:04:43 ubuntu maxscale[1]: Server changed state: 10.0.23.32[10.0.23.32:3306]: new_slave
Jul 8 11:04:43 ubuntu maxscale[1]: Query classifier initialized.
Jul 8 11:04:43 ubuntu maxscale[1]: MariaDB Corporation MaxScale 1.4.3 (C) MariaDB Corporation Ab 2013-2015
Jul 8 11:04:43 ubuntu maxscale[1]: MaxScale is running in process 1
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded 2 MySQL Users for service [Splitter Service].
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module MySQLClient: V1.0.0 from /usr/lib64/maxscale/libMySQLClient.so
Jul 8 11:04:43 ubuntu maxscale[1]: Listening MySQL connections at 0.0.0.0:3306
Jul 8 11:04:43 ubuntu maxscale[1]: Failed to bind to UNIX Domain socket '/var/run/mysqld/mysqld.sock': 13, Permission denied
Jul 8 11:04:43 ubuntu maxscale[1]: Unable to start to listen port 0 for MySQLClient Splitter Service.
Jul 8 11:04:43 ubuntu maxscale[1]: Loaded module maxscaled: V1.0.0 from /usr/lib64/maxscale/libmaxscaled.so
Jul 8 11:04:43 ubuntu maxscale[1]: Listening maxscale connections at localhost:6603
Jul 8 11:04:43 ubuntu maxscale[1]: MaxScale started with 4 server threads.
Jul 8 11:04:43 ubuntu maxscale[1]: Started MaxScale log flusher.
Jul 8 11:05:01 ubuntu CRON[30916]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
==================

Config file:

[maxscale]
threads=4

[Splitter Service]
type=service
router=readwritesplit
localhost_match_wildcard_host=1
servers=10.0.23.31,10.0.23.32,
user=maxscale
passwd=xxx

[Splitter Listener]
type=listener
service=Splitter Service
protocol=MySQLClient
port=3306
socket=/var/run/mysqld/mysqld.sock

[10.0.23.31]
type=server
address=10.0.23.31
port=3306
protocol=MySQLBackend

[10.0.23.32]
type=server
address=10.0.23.32
port=3306
protocol=MySQLBackend

[Galera Monitor]
type=monitor
module=galeramon
monitor_interval=1000
disable_master_failback=1
servers=10.0.23.31,10.0.23.32,
user=maxscale
passwd=xxx

[CLI]
type=service
router=cli

[CLI Listener]
type=listener
service=CLI
protocol=maxscaled
address=localhost
port=6603

 Comments   
Comment by Timofey Turenko [ 2016-07-14 ]

(gdb) bt
#0 0x00007f254a999cd6 in atomic_add (variable=0x1c, value=1) at /home/vagrant/workspace/server/core/atomic.c:46
#1 0x00007f254a9a4d60 in mysql_users_fetch (users=0x0, key=0x7f2541361900) at /home/vagrant/workspace/server/core/dbusers.c:1759
#2 0x00007f2542f8fdb8 in gw_find_mysql_user_password_sha1 (username=0x7f2530011cd8 "skysql", gateway_password=0x7f2541361a10 "", dcb=0x7f254ae7aa88)
at /home/vagrant/workspace/server/modules/authenticator/mysql_auth.c:442
#3 0x00007f2542f90217 in gw_check_mysql_scramble_data (dcb=0x7f254ae7aa88,
token=0x7f2530000fd0 "\214\312\033\371p\006\256\204I\354\005qI\214yJ߷\243n", token_len=20,
mxs_scramble=0x7f2534012390 "!e9i#)]PAO<U7A\036&
B8W", scramble_len=20, username=0x7f2530011cd8 "skysql", stage1_hash=0x7f2530011cc4 "")
at /home/vagrant/workspace/server/modules/authenticator/mysql_auth.c:604
#4 0x00007f2542f90470 in combined_auth_check (dcb=0x7f254ae7aa88, auth_token=0x7f2530000fd0 "\214\312\033\371p\006\256\204I\354\005qI\214yJ߷\243n",
auth_token_len=20, protocol=0x7f2534012350, username=0x7f2530011cd8 "skysql", stage1_hash=0x7f2530011cc4 "", database=0x7f2530011d59 "test")
at /home/vagrant/workspace/server/modules/authenticator/mysql_auth.c:770
#5 0x00007f2542f8f53d in mysql_auth_authenticate (dcb=0x7f254ae7aa88) at /home/vagrant/workspace/server/modules/authenticator/mysql_auth.c:154
#6 0x00007f25431a3add in gw_read_do_authentication (dcb=0x7f254ae7aa88, read_buffer=0x7f2530011bf0, nbytes_read=91)
at /home/vagrant/workspace/server/modules/protocol/mysql_client.c:585
#7 0x00007f25431a3a1f in gw_read_client_event (dcb=0x7f254ae7aa88) at /home/vagrant/workspace/server/modules/protocol/mysql_client.c:522
#8 0x00007f254a9c7f3c in process_pollq (thread_id=2) at /home/vagrant/workspace/server/core/poll.c:1003
#9 0x00007f254a9c7457 in poll_waitevents (arg=0x2) at /home/vagrant/workspace/server/core/poll.c:724
#10 0x00000000004050c0 in worker_thread_main (arg=0x2) at /home/vagrant/workspace/server/core/gateway.c:929
#11 0x00007f254a08bdf5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f2548b701ad in clone () from /lib64/libc.so.6

[vagrant@maxscale ~]$ maxscale --version-full

MariaDB Corporation MaxScale beta-2.0.0 Thu Jul 14 10:27:40 2016
------------------------------------------------------
MaxScale beta-2.0.0 - 90b23774d85ca6e86dd1181d0b50f0d3783ebc93
[vagrant@maxscale ~]$

Comment by markus makela [ 2016-07-26 ]

Each listener now individually stores the user data and the related caches.

Generated at Thu Feb 08 04:02:01 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.