[CONC-478] Feature Request: Batch mode and query pipelining Created: 2020-06-19  Updated: 2020-07-31

Status: Open
Project: MariaDB Connector/C
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major
Reporter: James Dominic Guana Assignee: Unassigned
Resolution: Unresolved Votes: 1
Labels: Connectors, performance


 Description   

Requesting new feature similar to postgresql batch mode and query pipelining

https://2ndquadrant.github.io/postgres/libpq-batch-mode.html

Most of the high performing app, utilize those features in postgresql

https://www.techempower.com/benchmarks/#section=data-r19&hw=ph&test=fortune



 Comments   
Comment by Vladislav Vaintroub [ 2020-06-19 ]

I moved this to appropriate project. MariaDB Server, or the protocol, do support query pipelining, and so does MySQL Server (AWS Aurora is known to break pipelining, but it is their problem). MariaDB JDBC driver is using pipelining in JDBC batch processing, only C client does not support it. I guess only splitting of mysql_query() into mysql_send_query() and mysql_read_result() is needed, to support it.

Comment by James Dominic Guana [ 2020-06-19 ]

That is good to hear!

How about the batching mode? I think the server should also support the batching mode.

From my understanding, batching mode is implemented collectively, like even if there are multiple clients submitting request, the database server still process it collectively as a batch .

Comment by Vladislav Vaintroub [ 2020-06-20 ]

Can you give a specific example of server batching ? Can you point to Postgres documentation of that?

Comment by James Dominic Guana [ 2020-06-20 ]

The link above mentions about the v3 extended query protocol implemented in PostgreSQL as a requirement.

So my guesstimate is that MariaDB should also have a similar implementations in place to take advantage of batching, unless of course the dev might have a better high performance approach in place to implement batching, that will also be welcome!

Comment by Vladislav Vaintroub [ 2020-06-20 ]

there is nothing I can read from this query protocol that tell about batching. Taking advantage of batching is simple, you write query multiple times, then read result multiple times. It is called pipelining. mariadb does not have any similarity to Postgres protocol, but almost any protocol can support pipelining , it is implementation detail of the client. our JDBC does have it, to support JDBC batch. HTTP can pipeline, and it is not similar to postgres protocol.

Comment by Vladislav Vaintroub [ 2020-06-20 ]

As for non-pipelined batching, MySQL protocol has also had semicolon batching for ages

"select 1; insert...;select .." gives you 3 semicolon-separated queries in one. Some people consider the feature a security weakness because it allows to exploit badly written application, as exemplified in https://xkcd.com/327/

Comment by Vladislav Vaintroub [ 2020-06-24 ]

JamesG, also. the link you sent, vaguely describes server side prepared statements. I guess this was 2005-ish, when it was implemented in MySQL protocol, on the client and server side ( I guess, because it was before my time)

Comment by James Dominic Guana [ 2020-06-25 ]

Thank you for enlightening. Then this would make this solely a C connector issue. I would be happy to see that soon.

Comment by Matthieu Garrigues [ 2020-07-31 ]

I'm working on adding sql pipelining support on the lithium framework, so I'll be happy to test mariadb pipelining as soon as it is available. I'm pretty sure this will boost the performances of the mariadb drivers and lower the gap between postgresql and mysql in this benchmark. This is definitely something that can be done without changing the protocol but may need changes also in the server if it's blocking the processing of request N+1 until the full result of request N has been read. You can also check the postgresql implementation this if you need inspiration:
https://2ndquadrant.github.io/postgres/libpq-batch-mode.html

Generated at Thu Feb 08 03:05:37 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.