[MXS-2] MaxScale filter to capture incoming operations for consumption in external sources Created: 2015-03-02  Updated: 2016-10-10  Resolved: 2016-10-10

Status: Closed
Project: MariaDB MaxScale
Component/s: Plugin
Affects Version/s: None
Fix Version/s: N/A

Type: Task Priority: Major
Reporter: Sergei Golubchik Assignee: Unassigned
Resolution: Won't Do Votes: 4
Labels: gsoc15, gsoc16


 Description   

Design a filter that will capture incoming inserts, updates and deletes, for specified tables (as regex) in a separate log file that is consumable as JSON or CSV form. So that external ETL processes can process it for data uploading into DWH or big data platform. Optionally a plugin that takes this log into a Kafka broker that can put this data on Hadoop node can be developed as next step.

 Comments   
Comment by Anshu Ranjan [ 2015-03-13 ]

Hi,

I am a PhD student at University of Florida. I have plenty of programming experiences including a 7 month co-op at Amazon, where I had to work a lot on JSON on CSV files. I have built plenty of project in databases. One of them required an implementation of a new database system. My undergrad final year project also involved databases for which I got an A+ grade an IEEE paper. I am working in working on this issue as this matches my background. Please let me know of future steps to come up with a proper proposal for the issue.

Thank you.

Comment by Ioannis Feidaros [ 2015-03-14 ]

Hi,

Our company have already built such filters and they are working pretty stable.
They designed as pipeline filters, recording vital information (including statement type and table info) for all queries executed via MaxScale and store it to external storage. Currently we support 2 external storage appliances:

  • Redis Server
  • ZeromMQ Consumer

Let me know if you need further help on this.

Comment by Timofey Turenko [ 2015-05-13 ]

What is the current status? Should we close this task?

Comment by Sriram Patil [ 2015-06-17 ]

I have been working on this as part of Google Summer of Code - 2015. I usually post a blog on every Monday giving the details of my work. One can check the details at http://srirampatil.tumblr.com/.

Comment by Johan Wikman [ 2016-10-10 ]

Avro router provides this functionality.

Generated at Thu Feb 08 03:56:02 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.