Uploaded image for project: 'MariaDB MaxScale'
  1. MariaDB MaxScale
  2. MXS-2188

MaxScale crashing when replicating

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.2
    • Component/s: avrorouter
    • Labels:
      None
    • Environment:
      CentOS 7

      Description

      MaxScale is configured with binlog router and avro router

      2018-11-26 14:32:22  notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
      2018-11-26 14:32:22  notice : replication-router: attempting to connect to master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306, binlog='mariadb-bin.000003', pos=1561866
      2018-11-26 14:32:22  notice : (4) replication-router: Request binlog records from mariadb-bin.000003 at position 1561866 from master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306
      2018-11-26 14:32:22  notice : (4) replication-router: identity seen by the master: Server_id: 5, Slave_UUID: 163fcb48-f188-11e8-8d7e-0ad760efc9d2, Host: not set
      2018-11-26 14:32:22  notice : (4) replication-router: identity seen by the slaves: server_id: 1, hostname: ip-172-31-33-161.us-east-2.compute.internal, MySQL version: 10.3.10-MariaDB-log
      2018-11-26 14:32:23  alert : Fatal: MaxScale 2.3.0 received fatal signal 11. Attempting backtrace.
      2018-11-26 14:32:23  alert : Commit ID: e3ef99f1f75880683e718d0d77ff45df76288fb2 System name: Linux Release string: Red Hat Enterprise Linux Server release 7.5 (Maipo)
      2018-11-26 14:32:23  alert :  /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceESt8functionIFvPKcS2_EE+0x2b) [0x40cadb]: /home/ec2-user/MaxScale/maxutils/maxbase/src/stacktrace.cc:130
      2018-11-26 14:32:23  alert :  /usr/bin/maxscale(_ZN7maxbase15dump_stacktraceEPFvPKcS1_E+0x4e) [0x40ce3e]: /usr/include/c++/4.8.2/functional:2029
      2018-11-26 14:32:23  alert :  /usr/bin/maxscale() [0x4094b9]: ??:0
      2018-11-26 14:32:23  alert :  /lib64/libpthread.so.0(+0xf680) [0x7f8707f01680]: sigaction.c:?
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x4fd3e) [0x7f8702896d3e]: st.c:?
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x4f161) [0x7f8702896161]: :?
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(avro_schema_record_field_get_index+0x2b) [0x7f87028928ee]: ??:?
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(+0x394c1) [0x7f87028804c1]: generic.c:?
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN13AvroConverter6columnEiSs+0x12) [0x7f8702867c82]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_converter.cc:440
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_Z22process_row_event_dataSt10shared_ptrI13TableMapEventES_I16TableCreateEventERSt8auto_ptrI15RowEventHandlerEPhS8_S8_+0x608) [0x7f8702861498]: /usr/include/c++/4.8.2/bits/basic_string.h:293
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN3Rpl16handle_row_eventEP10rep_headerPh+0x72b) [0x7f870286276b]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_rbr.cc:646
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(_ZN3Rpl12handle_eventE10rep_headerPh+0x8f) [0x7f870286432f]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_rbr.cc:1000
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavro-common.so.1.0.0(avro_read_all_events+0x14c) [0x7f870286622c]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_file.cc:612
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libavrorouter.so(_Z14converter_funcN7maxbase6Worker4Call8action_tEP4Avro+0x78) [0x7f8702ac1f58]: /home/ec2-user/MaxScale/server/modules/routing/avrorouter/avro_main.cc:279
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker4tickEv+0xe6) [0x7f870844f436]: /home/ec2-user/MaxScale/maxutils/maxbase/include/maxbase/worker.hh:780
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase11WorkerTimer6handleEPNS_6WorkerEj+0x36) [0x7f870844dbd6]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:253
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker15poll_waiteventsEv+0x1b5) [0x7f870844e5d5]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:841
      2018-11-26 14:32:23  alert :  /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(_ZN7maxbase6Worker3runEPNS_9SemaphoreE+0x51) [0x7f870844e7d1]: /home/ec2-user/MaxScale/maxutils/maxbase/src/worker.cc:542
      2018-11-26 14:32:23  alert :  /usr/bin/maxscale(main+0x2019) [0x4086e9]: /home/ec2-user/MaxScale/server/core/gateway.cc:2248
      2018-11-26 14:32:23  alert :  /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f8705dfe3d5]: ??:?
      2018-11-26 14:32:23  alert :  /usr/bin/maxscale() [0x408f12]: ??:0
      MariaDB MaxScale /var/log/maxscale/maxscale.log Mon Nov 26 14:32:24 2018
      ----------------------------------------------------------------------------
      2018-11-26 14:32:24  notice : syslog logging is enabled.
      2018-11-26 14:32:24  notice : maxlog logging is enabled.
      2018-11-26 14:32:24  notice : Working directory: /var/log/maxscale
      2018-11-26 14:32:24  notice : The collection of SQLite memory allocation statistics turned off.
      2018-11-26 14:32:24  notice : Threading mode of SQLite set to Multi-thread.
      2018-11-26 14:32:24  notice : MariaDB MaxScale 2.3.0 started (Commit: e3ef99f1f75880683e718d0d77ff45df76288fb2)
      2018-11-26 14:32:24  notice : MaxScale is running in process 19829
      2018-11-26 14:32:24  notice : Configuration file: /etc/maxscale.cnf
      2018-11-26 14:32:24  notice : Log directory: /var/log/maxscale
      2018-11-26 14:32:24  notice : Data directory: /var/lib/maxscale
      2018-11-26 14:32:24  notice : Module directory: /usr/lib64/maxscale
      2018-11-26 14:32:24  notice : Service cache: /var/cache/maxscale
      2018-11-26 14:32:24  notice : No query classifier specified, using default 'qc_sqlite'.
      2018-11-26 14:32:24  notice : Loaded module qc_sqlite: V1.0.0 from /usr/lib64/maxscale/libqc_sqlite.so
      2018-11-26 14:32:24  notice : Query classification results are cached and reused, cache max size: 415293440
      2018-11-26 14:32:24  notice : Loading /etc/maxscale.cnf.
      2018-11-26 14:32:24  notice : /etc/maxscale.cnf.d does not exist, not reading.
      2018-11-26 14:32:24  notice : Loaded module maxscaled: V2.0.0 from /usr/lib64/maxscale/libmaxscaled.so
      2018-11-26 14:32:24  notice : Initialise CLI router module
      2018-11-26 14:32:24  notice : Loaded module cli: V1.0.0 from /usr/lib64/maxscale/libcli.so
      2018-11-26 14:32:24  notice : Loaded module cdc: V1.0.0 from /usr/lib64/maxscale/libcdc.so
      2018-11-26 14:32:24  notice : Loaded module avrorouter: V1.0.0 from /usr/lib64/maxscale/libavrorouter.so
      2018-11-26 14:32:24  warning: Protocol module 'mysqlclient' has been deprecated, use 'mariadbclient' instead.
      2018-11-26 14:32:24  notice : Loaded module mariadbclient: V1.1.0 from /usr/lib64/maxscale/libmariadbclient.so
      2018-11-26 14:32:24  notice : Initialise binlog router module.
      2018-11-26 14:32:24  notice : Loaded module binlogrouter: V2.1.0 from /usr/lib64/maxscale/libbinlogrouter.so
      2018-11-26 14:32:24  notice : replication-router: storing binlog files in 'flat' mode
      2018-11-26 14:32:24  notice : replication-router: Service has MariaDB GTID otion set to ON
      2018-11-26 14:32:24  notice : Loaded module MySQLBackendAuth: V1.0.0 from /usr/lib64/maxscale/libmysqlbackendauth.so
      2018-11-26 14:32:24  notice : Validating last binlog file 'mariadb-bin.000003' ...
      2018-11-26 14:32:24  notice : 1543202433 @ 256, GTID List Event, (Mon Nov 26 03:20:33 2018), First EventTime
      2018-11-26 14:32:24  notice : 1543238469 @ 1561835, Transaction ID Event (2 Phase Commit), (Mon Nov 26 13:21:09 2018), Last EventTime
      2018-11-26 14:32:24  notice : Transaction Summary for binlog 'mariadb-bin.000003'
            Description          Total     Average       Max
            No. of Transactions       184
            No. of Events          7921       43.0       100
            No. of Bytes          1.5M       8.3k      19.8k
      2018-11-26 14:32:24  notice : Reading MySQL binlog files from /var/lib/maxscale
      2018-11-26 14:32:24  notice : Avro files stored at: /var/lib/maxscale
      2018-11-26 14:32:24  notice : First binlog is: mariadb-bin.000001
      2018-11-26 14:32:24  notice : [avro-router] Loading stored conversion state: /var/lib/maxscale/avro-conversion.ini
      2018-11-26 14:32:24  notice : Loaded stored binary log conversion state: File: [mariadb-bin.000003] Position: [1083294] GTID: [0-1-618:29]
      2018-11-26 14:32:24  notice : Loaded module CDCPlainAuth: V1.1.0 from /usr/lib64/maxscale/libcdcplainauth.so
      2018-11-26 14:32:24  notice : Loaded module MySQLAuth: V1.1.0 from /usr/lib64/maxscale/libmysqlauth.so
      2018-11-26 14:32:24  notice : Loaded module MaxAdminAuth: V2.1.0 from /usr/lib64/maxscale/libmaxadminauth.so
      2018-11-26 14:32:24  notice : Housekeeper thread started.
      2018-11-26 14:32:24  notice : Starting a total of 3 services...
      2018-11-26 14:32:24  notice : Encrypted password file /var/lib/maxscale/.secrets can't be accessed (No such file or directory). Password encryption is not used.
      2018-11-26 14:32:24  warning: The user for service 'replication-router' might be missing the SELECT grant on `mysql.roles_mapping` or `mysql.user`. Use of default roles is disabled until the missing privileges are added. Error was: SELECT command denied to user 'repl'@'ip-172-31-38-194.us-east-2.compute.internal' for table 'roles_mapping'
      2018-11-26 14:32:24  notice : [replication-router] Loaded 9 MySQL users for listener replication-listener.
      2018-11-26 14:32:24  notice : Listening for connections at [::]:6603 with protocol MySQL
      2018-11-26 14:32:24  notice : Service 'replication-router' started (1/3)
      2018-11-26 14:32:24  notice : Listening for connections at [::]:4001 with protocol CDC
      2018-11-26 14:32:24  notice : Service 'avro-router' started (2/3)
      2018-11-26 14:32:24  notice : Listening for connections at [/var/run/maxscale/maxadmin.sock]:0 with protocol MaxScale Admin
      2018-11-26 14:32:24  notice : Service 'MaxAdmin-Service' started (3/3)
      2018-11-26 14:32:24  notice : Started REST API on [127.0.0.1]:8989
      2018-11-26 14:32:24  notice : MaxScale started with 1 worker threads, each with a stack size of 8388608 bytes.
      2018-11-26 14:32:24  warning: Protocol module 'mysqlbackend' has been deprecated, use 'mariadbbackend' instead.
      2018-11-26 14:32:24  notice : Loaded module mariadbbackend: V2.0.0 from /usr/lib64/maxscale/libmariadbbackend.so
      2018-11-26 14:32:24  notice : replication-router: attempting to connect to master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306, binlog='mariadb-bin.000003', pos=1561866
      2018-11-26 14:32:24  notice : (4) replication-router: Request binlog records from mariadb-bin.000003 at position 1561866 from master server [ec2-18-188-20-38.us-east-2.compute.amazonaws.com]:3306
      2018-11-26 14:32:24  notice : (4) replication-router: identity seen by the master: Server_id: 5, Slave_UUID: 174ac6fa-f188-11e8-825d-0ad760efc9d2, Host: not set
      2018-11-26 14:32:24  notice : (4) replication-router: identity seen by the slaves: server_id: 1, hostname: ip-172-31-33-161.us-east-2.compute.internal, MySQL version: 10.3.10-MariaDB-log
      

        Attachments

          Activity

            People

            Assignee:
            markus makela markus makela
            Reporter:
            dshjoshi Dipti Joshi (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Git Integration