[MXS-1575] maxscale crashed with Avro conversion service Created: 2017-12-10  Updated: 2018-02-02  Resolved: 2018-02-02

Status: Closed
Project: MariaDB MaxScale
Component/s: avrorouter, binlogrouter
Affects Version/s: 2.1.11, 2.1.12
Fix Version/s: 2.1.14

Type: Bug Priority: Major
Reporter: Jiafu Wang Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

maxscale:maxscale-2.1.11-1.x86_64
[root@10-10-103-62 log]# lsb_release -a
LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description: CentOS Linux release 7.2.1511 (Core)
Release: 7.2.1511
Codename: Core
[root@10-10-103-62 log]# uname -a
Linux 10-10-103-62 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


Attachments: Zip Archive crash.zip     Zip Archive maxscale.zip     Zip Archive maxscalelog1.zip    
Sprint: 2017-49, MXS-SPRINT-50, MXS-SPRINT-51

 Description   

I have confiured the maxscale 2.1.11 with binlogrouter and avrorouter.
The step is:
1)start maxscale
2)configure the binlog master
3)start slave
then maxscale crashed with the core file。
The attachment are maxscale configure file、logfile、corefile。
Please checked them as quickly as possible;thx very much。



 Comments   
Comment by markus makela [ 2017-12-11 ]

Please try if the pre-release candidate packages for 2.1.12 fix the problem: http://max-tst-01.mariadb.com/ci-repository/maxscale-2.1.12-release/mariadb-maxscale/

Comment by Jiafu Wang [ 2017-12-11 ]

@markus makela thanks for reply,maxscale-2.1.12 was also crashed.
I have tried the maxscale version including:
maxscale-2.0.6-1.rhel.7.x86_64.rpm
maxscale-2.1.11-1.rhel.7.x86_64.rpm
maxscale-2.1.12-x86_64.rpm
maxscale-2.2.0-1.centos.7.x86_64.rpm

Comment by markus makela [ 2017-12-11 ]

OK, thanks for verifying this.

Also, the archive file is empty. If possible, please upload the MaxScale log file first so that we can see where it crashed. Only upload the binary log files and the core dump if it contains no confidential data.

Comment by Jiafu Wang [ 2017-12-11 ]

sorry,it's my fault.I have uploaded the relative files.
Because the upload filesize limit,I haved split the maxscale log to two parts.

Comment by Jiafu Wang [ 2017-12-13 ]

Hi markus,I haved upload the files.What about the progress of this issue?

Comment by markus makela [ 2018-01-02 ]

If possible, please try out this debug build of MaxScale. It contains a very minor fix (that could prevent the crash but still cause problems) and some added debug assertions to catch the real reason of the crash: http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan2/

Comment by Jiafu Wang [ 2018-01-05 ]

The MaxScale 2.1.13 from http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan2/ was crashed again.
The detail logfile is crash.zip which in the upload files.

Comment by markus makela [ 2018-01-05 ]

I think the crash could be caused by the comments in the CREATE TABLE statements. I think that the way the avrorouter parses the statements is not compatible with the character set that is used for the comments.

Edit: This was a false alarm, it was a simple mistake in the CREATE TABLE processing that failed to identify the explicit database name used in the CREATE statement.

Comment by markus makela [ 2018-01-06 ]

We have added a small fix that should fix the problems with the CREATE TABLE processing as well as built packages for testing the fix. You can find the packages here: http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan6/mariadb-maxscale/

If you can reproduce the crash, we'd appreciate it if you can try and see what sort of output you get with these packages.

Comment by Jiafu Wang [ 2018-01-09 ]

The recent core file was uploaded to the ftp://ftp.mariadb.com/uploads/.The filename is crash20170109.zip.
Installed image is from http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan6/mariadb-maxscale/.Thanks.

Comment by markus makela [ 2018-01-09 ]

Would it be possible for you to upload the binary log file that was being converted when the crash happened? This would be the fastest way to resolve this issue. You can upload the binary logs to the same place where you uploaded the core file.

Comment by Jiafu Wang [ 2018-01-10 ]

I have uploaded the binlog file.The filename is binlog.zip.
avro-conversion.ini stop at

[avro-conversion]
position=293872191
gtid=0-168403306-99552:219
file=mysql-bin.000007

Comment by markus makela [ 2018-01-10 ]

Thank you, I'll start testing with the binlog you provided and report back once I have some results.

Comment by markus makela [ 2018-01-11 ]

Managed to reproduce the debug assertion with the uploaded binlog. Now it's just a matter of finding and fixing the bug.

Comment by markus makela [ 2018-01-11 ]

The avrorouter appeared to be having problems with large DECIMAL fields. This has now been fixed along with a few minor bugs that were found.

wjf870128 I have built new packages for testing, you can find them here: http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan11/mariadb-maxscale/

With the new package, I was able to fully convert the binary log file.

Comment by markus makela [ 2018-01-16 ]

wjf870128 Have you tried the new custom version?

Comment by Jiafu Wang [ 2018-01-18 ]

Thanks,it was looked better.
The lastest crashed file were uploaded to ftp server.The files were named of "crash_binlog20180118.zip" and "crash_log20180118.zip".
avro-conversion.ini stop at

[avro-conversion]
position=403677237
gtid=0-168403306-1559086:1
file=mysql-bin.000010

Comment by markus makela [ 2018-01-18 ]

Thanks for the reply, we'll look into the binary log to see what is going on. I suspect that some of the data types used in your setup are not very common and they are the ones causing problems. I'll let you know once we've built another package for you to test. A big thank you for doing all this testing!

Comment by markus makela [ 2018-01-20 ]

This time it seems to be a problem with

ALTER TABLE { ADD | MODIFY | CHANGE } COLUMN ... AFTER ...

causing problems. This appears to be a simple oversight in the ALTER TABLE processing that the avrorouter does.

Comment by markus makela [ 2018-01-29 ]

The ALTER TABLE handling revealed a few bugs in other parts of the code. With these packages, I was able to fully convert both binlog files. wjf870128 If possible, please try them out.

Comment by Jiafu Wang [ 2018-02-02 ]

Hi makela,I have tested the packages from http://max-tst-01.mariadb.com/ci-repository/2.1-markusjm-jan29/mariadb-maxscale/centos/7/x86_64/
Up to now,my instance of maxscale is running stabilitily and the binlogs are parsed rightly.
Thanks for your help!

Comment by markus makela [ 2018-02-02 ]

OK, that is very good news. I'll close this issue and the fix should get into the next 2.1 release.

Generated at Thu Feb 08 04:07:48 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.