[MXS-1316] error using Kafka with binlog router Created: 2017-07-11  Updated: 2020-08-25  Resolved: 2017-07-18

Status: Closed
Project: MariaDB MaxScale
Component/s: binlogrouter
Affects Version/s: 2.1.4
Fix Version/s: 2.1.5

Type: Bug Priority: Major
Reporter: Kyle Joiner (Inactive) Assignee: markus makela
Resolution: Fixed Votes: 2
Labels: None

Attachments: File binlog.000001     File binlog.000002     File maxscale.cnf     Text File maxscale.log     File test.maxwell.000001.avro     File test.maxwell.000001.avsc    

 Description   

Broken pipe error when connecting kafka to binlog router:

The sequence of the error is as follows:

1. Run the command above (cdc.py -umaxscale -pmaxscale -h127.0.0.1 -P4001 test.maxwell | cdc_kafka_producer.py --kafka-broker=10.2.9.147:9092 --kafka-topic=maxscale)
2. Get message 'value must be bytes'
3. The program will continue to run without problems (timed it to 20 mins once)
4. As soon as the insert is executed on MariaDB (insert into `test`.`maxwell` values ('89','daemon test') the following error appears: [Errno 32] Broken pipe

 Comments   
Comment by Chris Calender (Inactive) [ 2017-07-13 ]

FYI, the customer installed MaxScale 2.0.3, re-ran the test, and the cdc_kafka_producer.py passes data to Kafka without problems. So it seems to be a regression bug somewhere between 2.0.3 and 2.1.4.

Comment by markus makela [ 2017-07-13 ]

This is caused by the fact that the values read from the standard input are interpreted as Unicode strings when the Kafka library requires them to be in byte format. As to why it works on some systems but not on others, I'm not completely sure. Adding the following to the start of the script seems to remedy this issue:

# Add the following on line 33
 
sys.stdin = sys.stdin.detach()

Generated at Thu Feb 08 04:05:49 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.