Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
-
None
Description
LOAD DATA currently supports reading from files and from the client (LOAD DATA LOCAL).
There's a certain interest in the ability to load data from more sources, in particular from AWS S3, but also from http[s]. This could also enable us to handle compressed files that is using any of the compression format the server supports.
We'll solve it by abstracting file reading code into the plugin. Initially there will be two plugins, file and "local".
The syntax
LOAD { DATA | XML } [ LOCAL ] INFILE ... |
will be generalized to
LOAD { DATA | XML } [ plugin ] INFILE ... |
We might need some kind of plugin-specific syntax extension for LOAD, so that AWS plugin would be able to specify the credentials. Or may be not, if everything can be part of the "filename", like in http://user:password@host.name/path/to/file
Optionally it should work for SELECT ... INTO OUTFILE too. In this case:
- a plugin descriptor will have two callbacks. For reading and writing (receiving and sending).
- either one can be NULL, in this case a plugin will only work one way (we likely won't want to support sending for http and client transfer plugins)
Attachments
Issue Links
- blocks
-
MCOL-5013 Support Load data from AWS S3 : UDF : columnstore_info.load_from_s3
-
- Closed
-
- is duplicated by
-
MDEV-23814 LOAD DATA INFILE 's3://mybucket/myobject.json'
-
- Closed
-
- relates to
-
MDEV-33188 Enhance mariadb-dump and mariadb-import capabilities similar to MyDumper
-
- Stalled
-
-
MXS-4618 Load data from S3
-
- Closed
-
-
MDEV-32061 INTO OUTFILE should work with named pipes
-
- Open
-