[MXS-2188] MaxScale crashing when replicating Created: 2018-11-26  Updated: 2018-12-03  Resolved: 2018-12-03

Status: Closed
Project: MariaDB MaxScale
Component/s: avrorouter
Affects Version/s: 2.3.0
Fix Version/s: 2.3.2

Type: Bug Priority: Major
Reporter: Dipti Joshi (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

CentOS 7


Attachments: File maxscale.cnf    

 Description   

MaxScale is configured with binlog router and avro router

2018-11-26 14:32:22  notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
2018-11-26 14:32:22  notice : replication-router: attempting to connect to master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306, binlog='mariadb-bin.000003', pos=1561866
2018-11-26 14:32:22  notice : (4) replication-router: Request binlog records from mariadb-bin.000003 at position 1561866 from master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306
2018-11-26 14:32:22  notice : (4) replication-router: identity seen by the master: Server_id: 5, Slave_UUID: 163fcb48-f188-11e8-8d7e-0ad760efc9d2, Host: not set
2018-11-26 14:32:22  notice : (4) replication-router: identity seen by the slaves: server_id: 1, hostname: ip-172-31-33-161.us-east-2.compute.internal, MySQL version: 10.3.10-MariaDB-log
2018-11-26 14:32:23  alert : Fatal: MaxScale 2.3.0 received fatal signal 11. Attempting backtrace.
2018-11-26 14:32:23  alert : Commit ID: e3ef99f1f75880683e718d0d77ff45df76288fb2 System name: Linux Release string: Red Hat Enterprise Linux Server release 7.5 (Maipo)
2018-11-26 14:32:23  alert :  /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x40cadb]: /home/ec2-user/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
2018-11-26 14:32:23  alert :  /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x4e) [0x40ce3e]: /usr/include/c++/4.8.2/functional:2029
2018-11-26 14:32:23  alert :  /usr/bin/maxscale() [0x4094b9]: ??:0
2018-11-26 14:32:23  alert :  /lib64/libpthread.so.0(+0xf680) [0x7f8707f01680]: sigaction.c:?
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x4fd3e) [0x7f8702896d3e]: st.c:?
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x4f161) [0x7f8702896161]: :?
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(avro_schema_record_field_get_index+0x2b) [0x7f87028928ee]: ??:?
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x394c1) [0x7f87028804c1]: generic.c:?
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN13AvroConverter6columnEiSs+0x12) [0x7f8702867c82]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_converter.cc:440
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_Z22process_row_event_dataSt10shared_ptrI13TableMapEventES_I16TableCreateEventERSt8auto_ptrI15RowEventHandlerEPhS8_S8_+0x608) [0x7f8702861498]: /usr/include/c++/4.8.2/bits/basic_string.h:293
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN3Rpl16handle_row_eventEP10rep_headerPh+0x72b) [0x7f870286276b]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_rbr.cc:646
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN3Rpl12handle_eventE10rep_headerPh+0x8f) [0x7f870286432f]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_rbr.cc:1000
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(avro_read_all_events+0x14c) [0x7f870286622c]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_file.cc:612
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavrorouter.so(_Z14converter_funcN7maxbase6Worker4Call8action_tEP4Avro+0x78) [0x7f8702ac1f58]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_main.cc:279
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker4tickEv+0xe6) [0x7f870844f436]: /home/ec2-user/MaxScale/maxutils/maxbase/include/maxbase/worker.hh:780
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase11WorkerTimer6handleEPNS_6WorkerEj+0x36) [0x7f870844dbd6]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:253
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1b5) [0x7f870844e5d5]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:841
2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x51) [0x7f870844e7d1]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:542
2018-11-26 14:32:23  alert :  /usr/bin/maxscale(main+0x2019) [0x4086e9]: /home/ec2-user/MaxScale/server/core/gateway.cc:2248
2018-11-26 14:32:23  alert :  /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8705dfe3d5]: ??:?
2018-11-26 14:32:23  alert :  /usr/bin/maxscale() [0x408f12]: ??:0
MariaDB MaxScale /var/log/maxscale/maxscale.log Mon Nov 26 14:32:24 2018
----------------------------------------------------------------------------
2018-11-26 14:32:24  notice : syslog logging is enabled.
2018-11-26 14:32:24  notice : maxlog logging is enabled.
2018-11-26 14:32:24  notice : Working directory: /var/log/maxscale
2018-11-26 14:32:24  notice : The collection of SQLite memory allocation statistics turned off.
2018-11-26 14:32:24  notice : Threading mode of SQLite set to Multi-thread.
2018-11-26 14:32:24  notice : MariaDB MaxScale 2.3.0 started (Commit: e3ef99f1f75880683e718d0d77ff45df76288fb2)
2018-11-26 14:32:24  notice : MaxScale is running in process 19829
2018-11-26 14:32:24  notice : Configuration file: /etc/maxscale.cnf
2018-11-26 14:32:24  notice : Log directory: /var/log/maxscale
2018-11-26 14:32:24  notice : Data directory: /var/lib/maxscale
2018-11-26 14:32:24  notice : Module directory: /usr/lib64/maxscale
2018-11-26 14:32:24  notice : Service cache: /var/cache/maxscale
2018-11-26 14:32:24  notice : No query classifier specified, using default 'qc_sqlite'.
2018-11-26 14:32:24  notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
2018-11-26 14:32:24  notice : Query classification results are cached and reused, cache max size: 415293440
2018-11-26 14:32:24  notice : Loading /etc/maxscale.cnf.
2018-11-26 14:32:24  notice : /etc/maxscale.cnf.d does not exist, not reading.
2018-11-26 14:32:24  notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
2018-11-26 14:32:24  notice : Initialise CLI router module
2018-11-26 14:32:24  notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
2018-11-26 14:32:24  notice : Loaded module cdc: V1.0.0 from /usr/lib64/maxscale/libcdc.so
2018-11-26 14:32:24  notice : Loaded module avrorouter: V1.0.0 from /usr/lib64/maxscale/libavrorouter.so
2018-11-26 14:32:24  warning: Protocol module 'mysqlclient' has been deprecated, use 'mariadbclient' instead.
2018-11-26 14:32:24  notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
2018-11-26 14:32:24  notice : Initialise binlog router module.
2018-11-26 14:32:24  notice : Loaded module binlogrouter: V2.1.0 from /usr/lib64/maxscale/libbinlogrouter.so
2018-11-26 14:32:24  notice : replication-router: storing binlog files in 'flat' mode
2018-11-26 14:32:24  notice : replication-router: Service has MariaDB GTID otion set to ON
2018-11-26 14:32:24  notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so
2018-11-26 14:32:24  notice : Validating last binlog file 'mariadb-bin.000003' ...
2018-11-26 14:32:24  notice : 1543202433 @ 256, GTID List Event, (Mon Nov 26 03:20:33 2018), First EventTime
2018-11-26 14:32:24  notice : 1543238469 @ 1561835, Transaction ID Event (2 Phase Commit), (Mon Nov 26 13:21:09 2018), Last EventTime
2018-11-26 14:32:24  notice : Transaction Summary for binlog 'mariadb-bin.000003'
      Description          Total     Average       Max
      No. of Transactions       184
      No. of Events          7921       43.0       100
      No. of Bytes          1.5M       8.3k      19.8k
2018-11-26 14:32:24  notice : Reading MySQL binlog files from /var/lib/maxscale
2018-11-26 14:32:24  notice : Avro files stored at: /var/lib/maxscale
2018-11-26 14:32:24  notice : First binlog is: mariadb-bin.000001
2018-11-26 14:32:24  notice : [avro-router] Loading stored conversion state: /var/lib/maxscale/avro-conversion.ini
2018-11-26 14:32:24  notice : Loaded stored binary log conversion state: File: [mariadb-bin.000003] Position: [1083294] GTID: [0-1-618:29]
2018-11-26 14:32:24  notice : Loaded module CDCPlainAuth: V1.1.0 from /usr/lib64/maxscale/libcdcplainauth.so
2018-11-26 14:32:24  notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so
2018-11-26 14:32:24  notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so
2018-11-26 14:32:24  notice : Housekeeper thread started.
2018-11-26 14:32:24  notice : Starting a total of 3 services...
2018-11-26 14:32:24  notice : Encrypted password file /var/lib/maxscale/.secrets can't be accessed (No such file or directory). Password encryption is not used.
2018-11-26 14:32:24  warning: The user for service 'replication-router' might be missing the SELECT grant on `mysql.roles_mapping` or `mysql.user`. Use of default roles is disabled until the missing privileges are added. Error was: SELECT command denied to user 'repl'@'ip-172-31-38-194.us-east-2.compute.internal' for table 'roles_mapping'
2018-11-26 14:32:24  notice : [replication-router] Loaded 9 MySQL users for listener replication-listener.
2018-11-26 14:32:24  notice : Listening for connections at [::]:6603 with protocol MySQL
2018-11-26 14:32:24  notice : Service 'replication-router' started (1/3)
2018-11-26 14:32:24  notice : Listening for connections at [::]:4001 with protocol CDC
2018-11-26 14:32:24  notice : Service 'avro-router' started (2/3)
2018-11-26 14:32:24  notice : Listening for connections at [/var/run/maxscale/maxadmin.sock]:0 with protocol MaxScale Admin
2018-11-26 14:32:24  notice : Service 'MaxAdmin-Service' started (3/3)
2018-11-26 14:32:24  notice : Started REST API on [127.0.0.1]:8989
2018-11-26 14:32:24  notice : MaxScale started with 1 worker threads, each with a stack size of 8388608 bytes.
2018-11-26 14:32:24  warning: Protocol module 'mysqlbackend' has been deprecated, use 'mariadbbackend' instead.
2018-11-26 14:32:24  notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
2018-11-26 14:32:24  notice : replication-router: attempting to connect to master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306, binlog='mariadb-bin.000003', pos=1561866
2018-11-26 14:32:24  notice : (4) replication-router: Request binlog records from mariadb-bin.000003 at position 1561866 from master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306
2018-11-26 14:32:24  notice : (4) replication-router: identity seen by the master: Server_id: 5, Slave_UUID: 174ac6fa-f188-11e8-825d-0ad760efc9d2, Host: not set
2018-11-26 14:32:24  notice : (4) replication-router: identity seen by the slaves: server_id: 1, hostname: ip-172-31-33-161.us-east-2.compute.internal, MySQL version: 10.3.10-MariaDB-log


Generated at Thu Feb 08 04:12:19 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.