[MDEV-22041] Allow to use binary row format even for COM_QUERY Created: 2020-03-25  Updated: 2024-01-15

Status: Open
Project: MariaDB Server
Component/s: Binary Protocol
Fix Version/s: None

Type: New Feature Priority: Major
Reporter: Diego Dupin Assignee: Ralf Gebhardt
Resolution: Unresolved Votes: 1
Labels: None


 Description   

There are two different row format for resultset : TEXT and BINARY format.

When retrieving numeric values in text format there is a lot of overhead:

  • Server converts numeric values into string
  • String values (which might be larger than the native binary format) need to be transferred via wire
  • Client needs to store the string and convert it back to a numeric value

The difference mainly concern :

  • null that are encoded using 1 bit, compare to 1 byte
  • numeric: text versus native. example "2147483647" will take 11 bytes (1 for length) vs 4 bytes for binary
  • date/time/timestamp: text versus "semi-native". example: "2001-01-01 00:00:00" takes 20 bytes (1 for length) vs 8 bytes

-When tested, representative workload TPC-C data is 40208 bytes for text vs 31325 for binary row (without header).-
That much exchange on network improve drastically performance.
Parsing native data will improve client execution too, for example in Python benchmarks fetching numeric values is up to 60% faster.
A minor annoyance is that using BINARY format is less readable for debugging

Proposal: Add an additionally capability MARIADB_BINARY_RESULT. If supported, the client will send this capability flag during handshake. The server afterwards will send result sets for COM_QUERY in binary format.



 Comments   
Comment by Vladislav Vaintroub [ 2020-03-26 ]

numeric: text vs native will "0" will take 2 bytes vs 8 byte for binary.

Comment by Vladislav Vaintroub [ 2020-03-26 ]

if there are no NULLs, null bitmap wastes column_count/8 + 1 bytes space.
with a typical result set row, where lets say 10 columns are returned, and none of them are NULL,
it wastes 2 bytes.

parsing integers is not that bad, and most languages that do not support memcpy, making an int out of 4 bytes buffer is

parsing int, binary protocol style
b[0] | b[1]<<8 | b[2] << 16|b[3] << 24

parsing int, text style
int s = 0;
for(int i=0; I < n; i++)
s = 10*s + b[i] -'0';

That's not too bad, compared to alleged horrors of parsing I've heard about

for fixed size int, e.g in dates you can use better routines : parsing 4 digit int

(a[0]-'0')*10000 + (a[1]-'0')*1000 + (a[2]-'0')*100+a[3]-'0' = 1000*a[0] + 100*a[1]+10*a[2] + a[3] - 1111*'0'
=1000*a[0] + 100*a[1]+10*a[2] + a[3]  - 53328

there is some insignificant overhead compared to shifts and OR, but it is miniscule.

As for size of data, how do rows look like, in TPCC queries? What's DDL and result set. It would be nice if you could break it down , for the result sets.

Comment by Diego Dupin [ 2020-04-17 ]

I've run TPCC new time, because after reflexion, that much difference surprise me.
running in detail and checking all bytes, i obtain very similar results (1% difference) using binary compare to text.

Row data bytes represent 26% of read exchanges and size is very similar using binary protocol compare to text (1% difference)

A surprise is that metadata represent 65% ! of read bytes.

To give an idea of what this 1% represent : ok packet represent 6% of read exchanges. (4.5% of all exchanges are from text info ok OkPacket = "Rows matched: 1 Changed: 1 Warnings: 0"

Comment by Vladislav Vaintroub [ 2020-04-17 ]

the OK packet text maybe should be optionally suppressed, this is really just for the command line client.

Comment by Georg Richter [ 2020-04-19 ]

Python benchmark:

CREATE TABLE `num_test` (
  `col1` smallint(6) DEFAULT NULL,
  `col2` int(11) DEFAULT NULL,
  `col3` smallint(6) DEFAULT NULL,
  `col4` bigint(20) DEFAULT NULL,
  `col5` float DEFAULT NULL,
  `col6` decimal(10,5) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1

   cursor = conn.cursor()
    range_it = range(loops)
    t0 = pyperf.perf_counter()
    for value in range_it:
        cursor.execute('select col1,col2,col3,col4,col5 from num_test')
        row= cursor.fetchone()
        while row is not None:
            row= cursor.fetchone()
    del cursor

num_fetchloop: Mean +- std dev: 839 us +- 34 us
num_fetchloop_bin: Mean +- std dev: 530 us +- 25 us

Comment by Sergei Golubchik [ 2020-08-16 ]

wouldn't EXECUTE IMMEDIATE be an exact replacement of COM_QUERY with binary protocol?

protocol command EXECUTE IMMEDIATE, not COM_QUERY that sends "EXECUTE IMMEDIATE" string, of course

Comment by Georg Richter [ 2020-08-16 ]

Serg,

unless EXECUTE IMMEDIATE doesn't have the same limitations as prepare (MEV-16708).
But I agree COM_EXECUTE_IMMEDIATE and COM_STMT_EXECUTE_IMMEDIATE (instead of the stmt_execute_direct hack) would be an idea.

Generated at Thu Feb 08 09:11:44 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.