Details
-
Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Do
-
None
Description
Design a filter that will capture incoming inserts, updates and deletes, for specified tables (as regex) in a separate log file that is consumable as JSON or CSV form. So that external ETL processes can process it for data uploading into DWH or big data platform. Optionally a plugin that takes this log into a Kafka broker that can put this data on Hadoop node can be developed as next step.
Attachments
Activity
Hi,
Our company have already built such filters and they are working pretty stable.
They designed as pipeline filters, recording vital information (including statement type and table info) for all queries executed via MaxScale and store it to external storage. Currently we support 2 external storage appliances:
- Redis Server
- ZeromMQ Consumer
Let me know if you need further help on this.
I have been working on this as part of Google Summer of Code - 2015. I usually post a blog on every Monday giving the details of my work. One can check the details at http://srirampatil.tumblr.com/.
Hi,
I am a PhD student at University of Florida. I have plenty of programming experiences including a 7 month co-op at Amazon, where I had to work a lot on JSON on CSV files. I have built plenty of project in databases. One of them required an implementation of a new database system. My undergrad final year project also involved databases for which I got an A+ grade an IEEE paper. I am working in working on this issue as this matches my background. Please let me know of future steps to come up with a proper proposal for the issue.
Thank you.