[MXS-1106] Failed to initialize avrofile, fatal signal 11 Created: 2017-01-25  Updated: 2017-02-01  Resolved: 2017-02-01

Status: Closed
Project: MariaDB MaxScale
Component/s: avrorouter
Affects Version/s: 2.0.3
Fix Version/s: 2.0.4

Type: Bug Priority: Major
Reporter: Frédérick Pop Assignee: markus makela
Resolution: Fixed Votes: 0
Labels: None
Environment:

Centos 6.7


Attachments: Text File maxscale203_startup.log    
Sprint: 2017-27

 Description   

Hi,

I'm having an issue while starting maxscale when I set up avrorouter service :

error : Failed to initialize avrofile.
error : Fatal: MaxScale 2.0.3 received fatal signal 11. Attempting backtrace.
error : Commit ID: 9e27e6c002b297d6ddbdbdb34c3de849069a9d7c System name: Linux Release string: CentOS release 6.7 (Final)
error : /usr/bin/maxscale() [0x403b9c]
error : /lib64/libpthread.so.0(+0xf7e0) [0x7fbd6574a7e0]
error : /usr/lib64/maxscale/libavrorouter.so(maxavro_file_open+0x25f) [0x7fbd60304395]
error : /usr/lib64/maxscale/libavrorouter.so(avro_index_file+0x29) [0x7fbd602ce3c0]
error : /usr/lib64/maxscale/libavrorouter.so(avro_update_index+0x97) [0x7fbd602ce8e3]
error : /usr/lib64/maxscale/libavrorouter.so(avro_flush_all_tables+0x90) [0x7fbd602cdbbb]
error : /usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0xb06) [0x7fbd602cd63b]
error : /usr/lib64/maxscale/libavrorouter.so(converter_func+0x69) [0x7fbd602c4c00]
error : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x3b8a2) [0x7fbd660558a2]
error : /lib64/libpthread.so.0(+0x7aa1) [0x7fbd65742aa1]
error : /lib64/libc.so.6(clone+0x6d) [0x7fbd642c8aad]

Context :

1) I setup a binlogrouter service and replicate binlogs from my master mariadb node, binlog started at index 32
2) I used cdc_schema.go to generate avro schema files (some create table statements where missing)
3) I setup avrorouter service and try to start but it fail

Here is my configuration :

  1. The Replication Proxy service

[replication-service]
type=service
router=binlogrouter
router_options=server_id=4000,master_id=3000,binlogdir=/var/lib/maxscale/binlog/,mariadb10-compatibility=1,
user=user
passwd=pass

  1. The Avro conversion service

[avro-service]
type=service
router=avrorouter
source=replication-service
router_options=binlogdir=/var/lib/maxscale/binlog/,avrodir=/var/lib/maxscale/avro/,filestem=mariadb-bin,start_index=32,



 Comments   
Comment by markus makela [ 2017-01-26 ]

Can you upload the .avsc files that were generated? One possible cause for this might be a malformed schema definition.

Comment by Frédérick Pop [ 2017-01-26 ]

Is there a way to anonymize tables definitions into avsc files ? I'm doing this setup with a production like database and I work in a financial company so unfortunately I won't have authorization to upload those informations.

Comment by markus makela [ 2017-01-26 ]

Ah, I think validating the JSON in the .avsc files would be enough. The actual values aren't relevant as long as they conform to the Avro specification. You can use something like jq to validate it.

You can also enable the info log level to get some extra information about the binlog conversion process. If you spot a certain spot in the binlogs that causes this, it would be good to take a look at the binlogs at that point with mysqlbinlog. Since it crashes on startup, I'd assume it's the first binlog that is causing the problems.

You can also try and see what happens if you create a similar table, insert some data and start MaxScale without the generated .avsc files. This way MaxScale itself will generate the .avsc files from the CREATE TABLE statements in the binlogs and we can see if the problem is with the generated .avsc files or something else.

We've also made a few minor fixes for the next 2.0 release that could fix this bug. If you have the possibility of building MaxScale from the source, I'd recommend you try out the 2.0 branch on GitHub and see if this still happens.

Comment by Frédérick Pop [ 2017-01-26 ]

JSON structure seems good for every .avsc files, I tried to open the first binlog but I get this message :

" mysqlbinlog: unknown variable 'default-character-set=utf8' "

It seems to be caused by the 'default-character-set=utf8' statement in my.cnf configuration, which is deprecated.
Is that blocking the binlog conversion process ?

Comment by markus makela [ 2017-01-26 ]

I don't think that should prevent the binlogs from being processed.

Does the error log have any errors apart from the one that's already shown?

Comment by Frédérick Pop [ 2017-01-26 ]

maxscale203_startup.log Here is the full startup log.

Comment by Frédérick Pop [ 2017-01-26 ]

I opened the first binlog with these options " mysqlbinlog --no-defaults --base64-output=DECODE-ROWS mariadb-bin.000032 " and what we can see at pos 526 is :

# at 407
# at 460
#170109 14:23:24 server id 38  end_log_pos 460  Table_map: `bbdb`.`_partition` mapped to number 100
#170109 14:23:24 server id 38  end_log_pos 526  Update_rows: table id 100 flags: STMT_END_F
# at 526
#170109 14:23:24 server id 38  end_log_pos 553  Xid = 63945799
COMMIT/*!*/;
# at 553
#170109 14:23:24 server id 38  end_log_pos 591  GTID 0-38-45226
/*!100001 SET @@session.gtid_seq_no=45226*//*!*/;
BEGIN
/*!*/;
# at 591

Comment by Frédérick Pop [ 2017-01-30 ]

I tried to build from source in order to test fixes from 2.0.4 but it fail during 'make test' command on the last test (binlogrouter).

Comment by markus makela [ 2017-01-30 ]

We've pushed a fix to the test failure in the 2.0 branch on GitHub. The test should pass now as it was a simple configuration error in the test declaration.

Comment by Frédérick Pop [ 2017-02-01 ]

I tried with version 2.0.4 and still can't start Maxscale, I have the same log :

2017-02-01 11:53:01 error : Failed to initialize avrofile.
2017-02-01 11:53:01 error : Fatal: MaxScale 2.0.4 received fatal signal 11. Attempting backtrace.
2017-02-01 11:53:01 error : Commit ID: a8780b892498f0a4df03ab8c09625603874582d5 System name: Linux Release string: CentOS release 6.7 (Final)
2017-02-01 11:53:01 error : /usr/bin/maxscale() [0x403b9c]
2017-02-01 11:53:01 error : /lib64/libpthread.so.0(+0xf7e0) [0x7feb226de7e0]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(maxavro_file_open+0x25f) [0x7feb1d298395]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(avro_index_file+0x29) [0x7feb1d2623c0]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(avro_update_index+0x97) [0x7feb1d2628e3]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(avro_flush_all_tables+0x90) [0x7feb1d261bbb]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(avro_read_all_events+0xb06) [0x7feb1d26163b]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libavrorouter.so(converter_func+0x69) [0x7feb1d258c00]
2017-02-01 11:53:01 error : /usr/lib64/maxscale/libmaxscale-common.so.1.0.0(+0x40ad2) [0x7feb22fefad2]
2017-02-01 11:53:01 error : /lib64/libpthread.so.0(+0x7aa1) [0x7feb226d6aa1]
2017-02-01 11:53:01 error : /lib64/libc.so.6(clone+0x6d) [0x7feb2125caad]
2017-02-01 11:53:01 info : Starting log flushing to disk.

Comment by Frédérick Pop [ 2017-02-01 ]

I tried with an other binlog start index, I always have this error.

Generated at Thu Feb 08 04:04:17 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.