Details
-
Task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Do
-
None
Description
Design a filter that will capture incoming inserts, updates and deletes, for specified tables (as regex) in a separate log file that is consumable as JSON or CSV form. So that external ETL processes can process it for data uploading into DWH or big data platform. Optionally a plugin that takes this log into a Kafka broker that can put this data on Hadoop node can be developed as next step.
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Workflow | jira [ 59915 ] | Test MaxScale Workflow [ 60023 ] |
| Comment |
[ Hi, I am Kapil Chhajer student from IIIT Hyderabad pursuing M.tech. CSE. I am comfortable with c, c++ and java. From Database perspective , I developed Simple Database engine in java as academic project. It seems "UDFs returning arrays" as interesting one but I am not getting completely. Please explain it or give me some related material link so I can understand. ] |
Hi,
Our company have already built such filters and they are working pretty stable.
They designed as pipeline filters, recording vital information (including statement type and table info) for all queries executed via MaxScale and store it to external storage. Currently we support 2 external storage appliances:
- Redis Server
- ZeromMQ Consumer
Let me know if you need further help on this.
| Status | Open [ 1 ] | Confirmed [ 10101 ] |
| Status | Confirmed [ 10101 ] | In Progress [ 3 ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| Status | Resolved [ 5 ] | Verified [ 10102 ] |
| Status | Verified [ 10102 ] | Closed [ 6 ] |
| Status | Closed [ 6 ] | Reopened [ 4 ] |
| Status | Reopened [ 4 ] | Confirmed [ 10101 ] |
| Resolution | Fixed [ 1 ] | Not a Bug [ 6 ] |
| Status | Confirmed [ 10101 ] | Resolved [ 5 ] |
| Workflow | Test MaxScale Workflow [ 60023 ] | MariaDB v3 [ 67479 ] |
| Status | Resolved [ 5 ] | In Review [ 10002 ] |
| Assignee | markus makela [ <markus.makela ] |
| Assignee | markus makela [ <markus.makela ] | Sriram Patil [ srirampatil ] |
| Component/s | Plugin [ 11601 ] | |
| Fix Version/s | 1.4.0 [ 19407 ] | |
| Status | In Review [ 10002 ] | Closed [ 6 ] |
| Resolution | Not a Bug [ 6 ] | |
| Status | Closed [ 6 ] | Stalled [ 10000 ] |
| Status | Stalled [ 10000 ] | Confirmed [ 10101 ] |
I have been working on this as part of Google Summer of Code - 2015. I usually post a blog on every Monday giving the details of my work. One can check the details at http://srirampatil.tumblr.com/.
| Fix Version/s | 1.4.0 [ 19407 ] |
| Labels | gsoc15 | gsoc16 |
| Labels | gsoc16 | gsoc15 gsoc16 |
| Assignee | Sriram Patil [ srirampatil ] |
| Fix Version/s | N/A [ 22001 ] | |
| Resolution | Won't Do [ 10201 ] | |
| Status | Confirmed [ 10101 ] | Closed [ 6 ] |
| Workflow | MariaDB v3 [ 67479 ] | MariaDB v4 [ 129084 ] |
Hi,
I am a PhD student at University of Florida. I have plenty of programming experiences including a 7 month co-op at Amazon, where I had to work a lot on JSON on CSV files. I have built plenty of project in databases. One of them required an implementation of a new database system. My undergrad final year project also involved databases for which I got an A+ grade an IEEE paper. I am working in working on this issue as this matches my background. Please let me know of future steps to come up with a proper proposal for the issue.
Thank you.