[MXS-1191] Alter statement to a table with no create statement. Created: 2017-03-20  Updated: 2017-05-18  Resolved: 2017-05-18

Status: Closed
Project: MariaDB MaxScale
Component/s: avrorouter
Affects Version/s: 2.0.5
Fix Version/s: 2.0.6

Type: Bug Priority: Major
Reporter: Josh Becker Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MXS-1216 Fatal error while converting data Closed
Sprint: 2017-32, 2017-33

 Description   

I was running MaxScale perfectly fine for about 3 days and then this error appeared. I used the provided go tool to create all avro schemas before maxscale was ever started.

2017-03-18 00:38:08 error : Alter statement to a table with no create statement.
2017-03-18 00:38:08 error : Fatal: MaxScale 2.0.5 received fatal signal 11. Attempting backtrace.
2017-03-18 00:38:08 error : Commit ID: 60c69d0f7f461a1773a4e848e87fa3fcc7b0e421 System name: Linux Release string: Ubuntu 14.04.5 LTS
2017-03-18 00:38:08 error : /usr/bin/maxscale() [0x403c80]
2017-03-18 00:38:08 error : /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330) [0x7f4072ba5330]
2017-03-18 00:38:08 error : /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a) [0x7f407247c9da]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(+0x20381) [0x7f406c9ff381]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(+0x2094c) [0x7f406c9ff94c]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(+0x20a29) [0x7f406c9ffa29]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(+0x20739) [0x7f406c9ff739]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(+0x209f9) [0x7f406c9ff9f9]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(json_vpack_ex+0x104) [0x7f406ca00fe5]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(json_pack_ex+0xaa) [0x7f406ca010f9]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(json_new_schema_from_table+0x46b) [0x7f406c9f3784]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(handle_table_map_event+0x152) [0x7f406c9f6014]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(avro_read_all_events+0x880) [0x7f406c9f99a3]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libavrorouter.so(converter_func+0xbd) [0x7f406c9f0360]
2017-03-18 00:38:08 error : /usr/lib/x86_64-linux-gnu/maxscale/libmaxscale-common.so.1.0.0(+0x4237e) [0x7f407305437e]
2017-03-18 00:38:08 error : /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184) [0x7f4072b9d184]
2017-03-18 00:38:08 error : /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f40724ee37d]
2017-03-18 00:38:08 info : Starting log flushing to disk.

Let me know what else would be helpful to provide to debug.

 Comments   
Comment by markus makela [ 2017-04-04 ]

This is possibly caused by an ALTER TABLE DROP COLUMN statement being executed with user generated schemas. It will cause the avrorouter to mix up the column counts and it will use unallocated memory.

Comment by Josh Becker [ 2017-04-04 ]

By user generated, do you mean by using the go tool to generate the avsc files?

I've also noticed that new tables that have been created aren't showing a corresponding avsc file in the avro directory, this should occur as the create table statement comes through, right?

Comment by markus makela [ 2017-04-04 ]

Yes, either the go tool or by manually defining the JSON schema.

The fact that there are missing avsc files does point to the fact that there might be something wrong in the schema versioning. Each CREATE TABLE or ALTER TABLE should create a new .avsc file, and if this is not happening, something is broken.

I'll look into the schema generation with ALTER TABLE and CREATE TABLE but the immediate problem of ALTER TABLE causing a crash has been fixed. Could you open a new issue for the .avsc file problem?

Comment by Josh Becker [ 2017-04-04 ]

Perfect will do. Which branch is the fix on?

Comment by markus makela [ 2017-04-04 ]

It's on the 2.0 branch. I've also pushed a Python 3 version of the cdc_schema Go tool for those cases where building the tool is not an option.

Comment by markus makela [ 2017-04-20 ]

I've build packages from commit a418387d0a8fa2372f78eb2fe351122c6b3ab024: http://max-tst-01.mariadb.com/ci-repository/2.0-apr20/mariadb-maxscale/

These packages contain the fixes for the handling of ALTER TABLE statements that can cause crashes.

Comment by Josh Becker [ 2017-04-24 ]

Just installed that package and receive the following error on startup (I suspect it wasn't compiled with the avrorouter?):

2017-04-24 13:23:19 error : Unable to find library for module: avrorouter. Module dir: /usr/lib/x86_64-linux-gnu/maxscale
2017-04-24 13:23:19 error : Unable to load Router module "avrorouter".
Ensure that libavrorouter.so exists in one of the following directories :

  • /usr/lib/x86_64-linux-gnu/maxscale
    2017-04-24 13:23:19 error : Service creation failed.
    2017-04-24 13:23:19 notice : Initialise binlog router module V2.0.0.
    2017-04-24 13:23:19 notice : Loaded module binlogrouter: V2.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libbinlogrouter.so
    2017-04-24 13:23:19 notice : Initialise CLI router module V1.0.0.
    2017-04-24 13:23:19 notice : Loaded module cli: V1.0.0 from /usr/lib/x86_64-linux-gnu/maxscale/libcli.so
    2017-04-24 13:23:19 error : 1 errors were encountered while processing the configuration file '/etc/maxscale.cnf'.
    2017-04-24 13:23:19 error : Failed to open, read or process the MaxScale configuration file /etc/maxscale.cnf. Exiting.
    2017-04-24 13:23:19 info : Starting log flushing to disk.
    2017-04-24 13:23:19 MaxScale is shut down.
Comment by markus makela [ 2017-05-08 ]

Yes, apologies for this mistake, we had some problems which we solved since then. I've build a new set of packages where the avrorouter is now properly built: http://max-tst-01.mariadb.com/ci-repository/2.0-may8/mariadb-maxscale/

Comment by markus makela [ 2017-05-18 ]

I've created packages from commit 5a0d2c54bd564688af44695067953ac16a09ee85 which fixes a few bugs. The packages can be found here: http://max-tst-01.mariadb.com/ci-repository/2.0-may15/mariadb-maxscale/

Please test these packages and see if they fix your problems.

Comment by markus makela [ 2017-05-18 ]

Closing as fixed. Please try the packages and if the problem persists, reopen this issue.

Generated at Thu Feb 08 04:04:53 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.