[#MXS-1536] Fatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace. Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux"

[MXS-1536] Fatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace. Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux" Created: 2017-11-21 Updated: 2018-10-11 Resolved: 2017-11-30
Status:	Closed
Project:	MariaDB MaxScale
Component/s:	avrorouter, binlogrouter
Affects Version/s:	None
Fix Version/s:	2.1.12

Type:

Bug

Priority:

Blocker

Reporter:

james mclaurin (Inactive)

Assignee:

markus makela

Resolution:

Fixed

Votes:

Labels:

None

Environment:

CentOS Linux

Attachments:

binlog.000001

Issue Links:

Relates
relates to	~~MXS-2095~~	Fatal: MaxScale 2.2.15 received fatal...	Closed

Description

[root@nf-mdb-maxscale01 dschatt]# maxscale
[root@nf-mdb-maxscale01 dschatt]# tFatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace.
Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux"

maxscale[0x405cea]
/lib64/libpthread.so.0(+0xf5e0)[0x7f33d557b5e0]
/lib64/libc.so.6(+0x134fd8)[0x7f33d3e03fd8]
/usr/lib64/maxscale/libavrorouter.so(count_columns+0x21)[0x7f33d0165611]
/usr/lib64/maxscale/libavrorouter.so(table_create_alloc+0x1aa)[0x7f33d016588a]
/usr/lib64/maxscale/libavrorouter.so(handle_query_event+0x212)[0x7f33d0169db2]
/usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0x76d)[0x7f33d016a6ed]
/usr/lib64/maxscale/libavrorouter.so(converter_func+0x98)[0x7f33d01615e8]
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x3a174)[0x7f33d5c9d174]
/lib64/libpthread.so.0(+0x7e25)[0x7f33d5573e25]
/lib64/libc.so.6(clone+0x6d)[0x7f33d3dc734d]

Writing core dump

Fatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace.
Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux"

maxscale[0x405cea]
/lib64/libpthread.so.0(+0xf5e0)[0x7fba791a95e0]
/lib64/libc.so.6(+0x134fd8)[0x7fba77a31fd8]
/usr/lib64/maxscale/libavrorouter.so(count_columns+0x21)[0x7fba73d93611]
/usr/lib64/maxscale/libavrorouter.so(table_create_alloc+0x1aa)[0x7fba73d9388a]
/usr/lib64/maxscale/libavrorouter.so(handle_query_event+0x212)[0x7fba73d97db2]
/usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0x76d)[0x7fba73d986ed]
/usr/lib64/maxscale/libavrorouter.so(converter_func+0x98)[0x7fba73d8f5e8]
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x3a174)[0x7fba798cb174]
/lib64/libpthread.so.0(+0x7e25)[0x7fba791a1e25]
/lib64/libc.so.6(clone+0x6d)[0x7fba779f534d]
2017-11-20 20:45:20 info : [avrorouter] Create table: CREATE TABLE `flyway`.`schema_history` ( `installed_rank` INT NOT NULL, `version` VARCHAR(50), `description` VARCHAR(200) NOT NULL, `type` VARCHAR(20) NOT NULL, `script` VARCHAR(1000) NOT NULL, `checksum` INT, `installed_by` VARCHAR(100) NOT NULL, `installed_on` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, `execution_time` INT NOT NULL, `success` BOOL NOT NULL,
2017-11-20 20:45:20 alert : Fatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace.
2017-11-20 20:45:20 alert : Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux"
2017-11-20 20:45:20 alert : maxscale() [0x405cea]
2017-11-20 20:45:20 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7fba791a95e0]
2017-11-20 20:45:20 alert : /lib64/libc.so.6(+0x134fd8) [0x7fba77a31fd8]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libavrorouter.so(count_columns+0x21) [0x7fba73d93611]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libavrorouter.so(table_create_alloc+0x1aa) [0x7fba73d9388a]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libavrorouter.so(handle_query_event+0x212) [0x7fba73d97db2]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0x76d) [0x7fba73d986ed]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libavrorouter.so(converter_func+0x98) [0x7fba73d8f5e8]
2017-11-20 20:45:20 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x3a174) [0x7fba798cb174]
2017-11-20 20:45:20 alert : /lib64/libpthread.so.0(+0x7e25) [0x7fba791a1e25]
2017-11-20 20:45:20 alert : /lib64/libc.so.6(clone+0x6d) [0x7fba779f534d]
2017-11-20 20:45:20 info : Starting log flushing to disk.

Writing core dump

Comments

Comment by markus makela [ 2017-11-29 ]

Managed to find a few issues in the ALTER TABLE handling and the table map event processing which together caused the crash.

Comment by james mclaurin (Inactive) [ 2017-12-02 ]

I had customer test this update and he reports the following results:

It gets slightly further, but still crashes. New maxscale.log attached:

MariaDB Corporation MaxScale 2.1.11 Fri Dec 1 20:18:16 2017
------------------------------------------------------

MariaDB MaxScale /var/log/maxscale/maxscale.log Fri Dec 1 20:18:16 2017
----------------------------------------------------------------------------
2017-12-01 20:18:16 notice : Working directory: /var/log/maxscale
2017-12-01 20:18:16 notice : MariaDB MaxScale 2.1.11 started
2017-12-01 20:18:16 notice : MaxScale is running in process 9033
2017-12-01 20:18:16 notice : Commit: source-build
2017-12-01 20:18:16 notice : Configuration file: /etc/maxscale.cnf
2017-12-01 20:18:16 notice : Log directory: /var/log/maxscale
2017-12-01 20:18:16 notice : Data directory: /var/lib/maxscale
2017-12-01 20:18:16 notice : Module directory: /usr/lib64/maxscale
2017-12-01 20:18:16 notice : Service cache: /var/cache/maxscale
2017-12-01 20:18:16 notice : Loading /etc/maxscale.cnf.
2017-12-01 20:18:16 notice : The logging of informational messages has been enabled.
2017-12-01 20:18:16 notice : /etc/maxscale.cnf.d does not exist, not reading.
2017-12-01 20:18:16 notice : Loaded module avrorouter: V1.0.0 from /usr/lib64/maxscale/libavrorouter.so
2017-12-01 20:18:16 notice : [binlogrouter] Initialise binlog router module.
2017-12-01 20:18:16 notice : Loaded module binlogrouter: V2.1.0 from /usr/lib64/maxscale/libbinlogrouter.so
2017-12-01 20:18:16 notice : [cli] Initialise CLI router module
2017-12-01 20:18:16 notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
2017-12-01 20:18:16 notice : [readwritesplit] Initializing statement-based read/write split router module.
2017-12-01 20:18:16 notice : Loaded module readwritesplit: V1.1.0 from /usr/lib64/maxscale/libreadwritesplit.so
2017-12-01 20:18:16 notice : [galeramon] Initialise the MySQL Galera Monitor module.
2017-12-01 20:18:16 notice : Loaded module galeramon: V2.0.0 from /usr/lib64/maxscale/libgaleramon.so
2017-12-01 20:18:16 notice : Loaded module MySQLBackend: V2.0.0 from /usr/lib64/maxscale/libMySQLBackend.so
2017-12-01 20:18:16 notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libMySQLBackendAuth.so
2017-12-01 20:18:16 notice : Loaded module CDC: V1.0.0 from /usr/lib64/maxscale/libCDC.so
2017-12-01 20:18:16 notice : Loaded module CDCPlainAuth: V1.1.0 from /usr/lib64/maxscale/libCDCPlainAuth.so
2017-12-01 20:18:16 notice : Loaded module MySQLClient: V1.1.0 from /usr/lib64/maxscale/libMySQLClient.so
2017-12-01 20:18:16 notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libMySQLAuth.so
2017-12-01 20:18:16 info : [maxscaled] Initialise MaxScaled Protocol module.
2017-12-01 20:18:16 notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
2017-12-01 20:18:16 notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libMaxAdminAuth.so
2017-12-01 20:18:16 notice : No query classifier specified, using default 'qc_sqlite'.
2017-12-01 20:18:16 notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
2017-12-01 20:18:16 info : qc_sqlite loaded.
2017-12-01 20:18:16 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 139758150863488.
2017-12-01 20:18:16 notice : Encrypted password file /var/lib/maxscale/.secrets can't be accessed (No such file or directory). Password encryption is not used.
2017-12-01 20:18:17 info : Notification service feedback is not enabled.
2017-12-01 20:18:17 notice : Server changed state: mysql01[10.64.24.42:3306]: new_master. [Running] -> [Master, Synced, Running]
2017-12-01 20:18:17 notice : Server changed state: mysql02[10.64.24.66:3306]: new_slave. [Running] -> [Slave, Synced, Running]
2017-12-01 20:18:17 notice : Server changed state: mysql03[10.64.24.67:3306]: new_slave. [Running] -> [Slave, Synced, Running]
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('', 'localhost', NULL, 0, NULL)
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('', 'nf-mdb-mysql01.arch.fonality.com', NULL, 0, NULL)
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('cluster', '%', NULL, 1, '4C92A9EA77687A04EBF3DF0746CC783364BF09AC')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('hdb', '10.%', NULL, 1, '188FB57F186061FEEC658BD24A8C4E0A7DAB3125')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('maxuser', '%', NULL, 0, '5EDBD32E469DAE0CE10E6999C3899DEFCB9F12E0')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_asterisk', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_central', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_chat', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_opensips', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_asterisk', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_central', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_chat', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_opensips', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('opensips', '%', 'nf_opensips', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('hdb', '%', NULL, 1, '188FB57F186061FEEC658BD24A8C4E0A7DAB3125')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('asterisk', '%', 'nf_asterisk', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('info', '%', 'flyway', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('exporter', 'localhost', NULL, 1, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('exporter', '%', NULL, 1, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('maxuser', '%', 'mysql', 0, '5EDBD32E469DAE0CE10E6999C3899DEFCB9F12E0')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', NULL, 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', NULL, 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('opensips', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('asterisk', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('info', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 notice : [MySQLAuth] [Read-Write Service] Loaded 25 MySQL users for listener Read-Write Listener.
2017-12-01 20:18:17 notice : Listening for connections at [10.64.24.88]:3306 with protocol MySQL
2017-12-01 20:18:17 info : Started session [1] for Read-Write Service service
2017-12-01 20:18:17 notice : Listening for connections at [/tmp/maxadmin.sock]:0 with protocol MaxScale Admin
2017-12-01 20:18:17 info : Started session [2] for MaxAdmin Service service
2017-12-01 20:18:17 info : [binlogrouter] Replication: /var/binlogs/master.ini parse result is 0
2017-12-01 20:18:17 notice : [binlogrouter] Replication: Service has transaction safety option set to ON
2017-12-01 20:18:17 notice : [binlogrouter] Validating binlog file 'binlog.000001' ...
2017-12-01 20:18:17 notice : [binlogrouter] 1510358530 @ 256, GTID List Event, (Sat Nov 11 00:02:10 2017), First EventTime
2017-12-01 20:18:17 notice : [binlogrouter] 1510684002 @ 6911186, Query Event, (Tue Nov 14 18:26:42 2017), Last EventTime
2017-12-01 20:18:17 notice : [binlogrouter] Transaction Summary for binlog 'binlog.000001'
Description Total Average Max
No. of Transactions 1555
No. of Events 26721 17.2 1723
No. of Bytes 3.8M 2.5k 315.6k
2017-12-01 20:18:17 info : [binlogrouter] Current binlog file is binlog.000001, safe pos 6911335, current pos is 6911335
2017-12-01 20:18:17 info : Started session [3] for Replication service
2017-12-01 20:18:17 notice : [binlogrouter] Replication: attempting to connect to master server [nf-mdb-mysql01.arch.fonality.com]:3306, binlog binlog.000001, pos 6911335
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('', 'localhost', NULL, 0, NULL)
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('', 'nf-mdb-mysql01.arch.fonality.com', NULL, 0, NULL)
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('cluster', '%', NULL, 1, '4C92A9EA77687A04EBF3DF0746CC783364BF09AC')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('hdb', '10.%', NULL, 1, '188FB57F186061FEEC658BD24A8C4E0A7DAB3125')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('maxuser', '%', NULL, 0, '5EDBD32E469DAE0CE10E6999C3899DEFCB9F12E0')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_asterisk', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_central', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_chat', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', 'nf_opensips', 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_asterisk', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_central', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_chat', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', 'nf_opensips', 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('opensips', '%', 'nf_opensips', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('hdb', '%', NULL, 1, '188FB57F186061FEEC658BD24A8C4E0A7DAB3125')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('asterisk', '%', 'nf_asterisk', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('info', '%', 'flyway', 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('exporter', 'localhost', NULL, 1, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('exporter', '%', NULL, 1, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('maxuser', '%', 'mysql', 0, '5EDBD32E469DAE0CE10E6999C3899DEFCB9F12E0')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfonread', '%', NULL, 0, '9C974CE5D3CCA42F5D9D38537C40FE805C386641')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('nfon', '%', NULL, 0, 'AA5A1CD2A0EBCF5E3CE5212A745D254428998E16')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('opensips', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('asterisk', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 info : [MySQLAuth] Added user: INSERT OR REPLACE INTO mysqlauth_users VALUES ('info', '%', NULL, 0, '34FD07B148CCA9FCB2BE6C2EA20689053E748B13')
2017-12-01 20:18:17 notice : [MySQLAuth] [Replication] Loaded 25 MySQL users for listener Replication Listener.
2017-12-01 20:18:17 notice : Listening for connections at [::]:5308 with protocol MySQL
2017-12-01 20:18:17 info : Started session [4] for Replication service
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] Using configuration options from service 'Replication'.
2017-12-01 20:18:17 info : [avrorouter] Reading MySQL binlog files from /var/binlogs
2017-12-01 20:18:17 warning: [avrorouter] Router options for Avrorouter are deprecated. Please convert them to parameters.
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] Reading MySQL binlog files from /var/binlogs
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] Avro files stored at: /var/lib/maxscale/avro/
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] First binlog is: binlog.000001
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] Using existing GTID index: //var/lib/maxscale/avro//avro.index
2017-12-01 20:18:17 notice : [avrorouter] [CDCService] Loading stored conversion state: /var/lib/maxscale/avro//avro-conversion.ini
2017-12-01 20:18:17 notice : [avrorouter] Loaded stored binary log conversion state: File: [binlog.000001] Position: [5921] GTID: [0-1-36:0]
2017-12-01 20:18:17 info : [avrorouter] current MySQL binlog file is binlog.000001, pos is 5921
2017-12-01 20:18:17 notice : Listening for connections at [::]:4001 with protocol CDC
2017-12-01 20:18:17 info : Started session [5] for CDCService service
2017-12-01 20:18:17 notice : Started MaxScale log flusher.
2017-12-01 20:18:17 notice : MaxScale started with 4 server threads.
2017-12-01 20:18:17 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 139757984225024.
2017-12-01 20:18:17 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 139757975832320.
2017-12-01 20:18:17 info : [qc_sqlite] In-memory sqlite database successfully opened for thread 139757900265216.
2017-12-01 20:18:17 notice : (3) [binlogrouter] Replication: checking Semi-Sync replication capability for master server [nf-mdb-mysql01.arch.fonality.com]:3306
2017-12-01 20:18:17 notice : (3) [binlogrouter] Replication: master server [nf-mdb-mysql01.arch.fonality.com]:3306 doesn't have semi_sync capability
2017-12-01 20:18:17 notice : (3) [binlogrouter] Replication: Request binlog records from binlog.000001 at position 6911335 from master server [nf-mdb-mysql01.arch.fonality.com]:3306
2017-12-01 20:18:17 notice : (3) [binlogrouter] Replication: identity seen by the master: Server_id: 1001, Slave_UUID: c48f8294-d6d4-11e7-8edf-6667ee7a2a3e, Host: maxscale-blr-1
2017-12-01 20:18:17 notice : (3) [binlogrouter] Replication: identity seen by the slaves: server_id: 1, hostname: nf-mdb-mysql01.arch.fonality.com, MySQL version: 10.2.9-MariaDB-log
2017-12-01 20:18:18 info : [avrorouter] Create table: CREATE TABLE `flyway`.`schema_history` ( `installed_rank` INT NOT NULL, `version` VARCHAR(50), `description` VARCHAR(200) NOT NULL, `type` VARCHAR(20) NOT NULL, `script` VARCHAR(1000) NOT NULL, `checksum` INT, `installed_by` VARCHAR(100) NOT NULL, `installed_on` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, `execution_time` INT NOT NULL, `success` BOOL NOT NULL, CONSTRAINT `schema_history_pk`PRIMARY KEY (`installed_rank`) ) ENGINE=InnoDB?l?7
2017-12-01 20:18:18 info : [avrorouter] Annotate_rows_event: INSERT INTO `flyway`.`schema_history` (`installed_rank`,`version`,`description`,`type`,`script`,`checksum`,`installed_by`,`execution_time`,`success`) VALUES (1, NULL, '<< Flyway Schema Creation >>', 'SCHEMA', '`flyway`,`nf_asterisk`,`nf_central`,`nf_chat`,`nf_opensips`', NULL, SUBSTRING_INDEX(USER(),'@',1), 0, 1)??*J
2017-12-01 20:18:18 info : [avrorouter] Table Map for 'flyway.schema_history' at 7877
2017-12-01 20:18:18 info : [avrorouter] Row Event for 'flyway.schema_history' at 7957
2017-12-01 20:18:18 info : [avrorouter] Row 1
2017-12-01 20:18:18 info : [avrorouter] [1] NULL
2017-12-01 20:18:18 info : [avrorouter] [2] VARCHAR: field: 200 bytes, data: 28 bytes
2017-12-01 20:18:18 info : [avrorouter] [3] VARCHAR: field: 20 bytes, data: 6 bytes
2017-12-01 20:18:18 info : [avrorouter] [4] VARCHAR: field: 1000 bytes, data: 59 bytes
2017-12-01 20:18:18 info : [avrorouter] [5] NULL
2017-12-01 20:18:18 info : [avrorouter] [6] VARCHAR: field: 100 bytes, data: 4 bytes
2017-12-01 20:18:18 info : [avrorouter] [7] TEMPORAL: 2017-11-11 00:18:47
2017-12-01 20:18:18 info : [avrorouter] Row 2
2017-12-01 20:18:18 info : [avrorouter] [1] NULL
2017-12-01 20:18:18 info : [avrorouter] [2] NULL
2017-12-01 20:18:18 info : [avrorouter] [3] NULL
2017-12-01 20:18:18 info : [avrorouter] [4] VARCHAR: field: 1000 bytes, data: 24942 bytes
2017-12-01 20:18:18 error : [avrorouter] debug assert at /home/vagrant/MaxScale/server/modules/routing/avrorouter/avro_rbr.c:630 failed: ptr < end
2017-12-01 20:18:18 info : Starting log flushing to disk.
maxscale: /home/vagrant/MaxScale/server/modules/routing/avrorouter/avro_rbr.c:630: process_row_event_data: Assertion `ptr < end' failed.
Fatal: MaxScale 2.1.11 received fatal signal 6. Attempting backtrace.
Commit ID: source-build System name: Linux Release string: NAME="CentOS Linux"

maxscale[0x4043a0]
/lib64/libpthread.so.0(+0xf5e0)[0x7f1bfa2ae5e0]
/lib64/libc.so.6(gsignal+0x37)[0x7f1bf8a371f7]
/lib64/libc.so.6(abort+0x148)[0x7f1bf8a388e8]
/lib64/libc.so.6(+0x2e266)[0x7f1bf8a30266]
/lib64/libc.so.6(+0x2e312)[0x7f1bf8a30312]
/usr/lib64/maxscale/libavrorouter.so(process_row_event_data+0x11a7)[0x7f1bf4e96eb1]
/usr/lib64/maxscale/libavrorouter.so(handle_row_event+0x56a)[0x7f1bf4e95514]
/usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0xa50)[0x7f1bf4e9963e]
/usr/lib64/maxscale/libavrorouter.so(converter_func+0xbd)[0x7f1bf4e8c231]
/usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x466a4)[0x7f1bfa9dc6a4]
/lib64/libpthread.so.0(+0x7e25)[0x7f1bfa2a6e25]
/lib64/libc.so.6(clone+0x6d)[0x7f1bf8afa34d]
2017-12-01 20:18:18 alert : Fatal: MaxScale 2.1.11 received fatal signal 6. Attempting backtrace.
2017-12-01 20:18:18 alert : Commit ID: source-build System name: Linux Release string: NAME="CentOS Linux"
2017-12-01 20:18:18 alert : maxscale() [0x4043a0]
2017-12-01 20:18:18 alert : /lib64/libpthread.so.0(+0xf5e0) [0x7f1bfa2ae5e0]
2017-12-01 20:18:18 alert : /lib64/libc.so.6(gsignal+0x37) [0x7f1bf8a371f7]
2017-12-01 20:18:18 alert : /lib64/libc.so.6(abort+0x148) [0x7f1bf8a388e8]
2017-12-01 20:18:18 alert : /lib64/libc.so.6(+0x2e266) [0x7f1bf8a30266]
2017-12-01 20:18:18 alert : /lib64/libc.so.6(+0x2e312) [0x7f1bf8a30312]
2017-12-01 20:18:18 alert : /usr/lib64/maxscale/libavrorouter.so(process_row_event_data+0x11a7) [0x7f1bf4e96eb1]
2017-12-01 20:18:18 alert : /usr/lib64/maxscale/libavrorouter.so(handle_row_event+0x56a) [0x7f1bf4e95514]
2017-12-01 20:18:18 alert : /usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0xa50) [0x7f1bf4e9963e]
2017-12-01 20:18:18 alert : /usr/lib64/maxscale/libavrorouter.so(converter_func+0xbd) [0x7f1bf4e8c231]
2017-12-01 20:18:18 alert : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x466a4) [0x7f1bfa9dc6a4]
2017-12-01 20:18:18 alert : /lib64/libpthread.so.0(+0x7e25) [0x7f1bfa2a6e25]
2017-12-01 20:18:18 alert : /lib64/libc.so.6(clone+0x6d) [0x7f1bf8afa34d]
2017-12-01 20:18:18 info : Starting log flushing to disk.

Writing core dump

Comment by james mclaurin (Inactive) [ 2017-12-02 ]

customer reports the following after fix:

It gets slightly further, but still crashes. New maxscale.log attached:

MariaDB Corporation MaxScale 2.1.11 Fri Dec 1 20:18:16 2017
------------------------------------------------------

Writing core dump

Generated at Thu Feb 08 04:07:30 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.

[MXS-1536] Fatal: MaxScale 2.1.10 received fatal signal 11. Attempting backtrace. Commit ID: 96c3f0dda3b5a9640c4995f46ac8efec77686269 System name: Linux Release string: NAME="CentOS Linux" Created: 2017-11-21 Updated: 2018-10-11 Resolved: 2017-11-30