The current way of handling uuid's in MariaDB is not very user friendly. If you want to do it right you have to replace all 4 "-" chars with "" and store it in a binary(16) type and so on. That is a lot of work and sadly people start to use just varchar() for it because it's easier. But that is a huge performence problem.
MDEV-35427Assertion `is_null() >= item->null_value' failed in Timestamp_or_zero_datetime_native_null::Timestamp_or_zero_datetime_native_null on EXECUTE
A datatype will maybe less useful for current applications, because of BC schemas, but I would recommend it for new schema.
For current schemas some (robust) conversion between textual and binary representation of UUIDs would be very helpful, although this functions can be shimed on legacy systems.
Oliver Hoff (Inactive)
added a comment - - edited A datatype will maybe less useful for current applications, because of BC schemas, but I would recommend it for new schema.
For current schemas some (robust) conversion between textual and binary representation of UUIDs would be very helpful, although this functions can be shimed on legacy systems.
You can convert from text uuid to binary with something like unhex(replace(uuid, "-", "")).
But yes, a dedicated type would be much more convenient, I agree.
Sergei Golubchik
added a comment - You can convert from text uuid to binary with something like unhex(replace(uuid, "-", "")).
But yes, a dedicated type would be much more convenient, I agree.
Recommend that this is implemented as a native data type, not plugin or other method, given the fact that people will be using this as a primary key.
Jacob Rhoden (Inactive)
added a comment - Recommend that this is implemented as a native data type, not plugin or other method, given the fact that people will be using this as a primary key.
Also, the benefits of the dedicated type would be (quoting MDEV-5593):
more comfortable because they do not have to do hex / unhex and remove dash
faster index than varchar variant
automatic generation of UUID if inserted record with NULL PK
Sergei Golubchik
added a comment - Also, the benefits of the dedicated type would be (quoting MDEV-5593 ):
more comfortable because they do not have to do hex / unhex and remove dash
faster index than varchar variant
automatic generation of UUID if inserted record with NULL PK
Just in case performance could be improved in the initial implementation I thought I'd link this article: https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/ (author: Karthik Appigatla)
It suggests reordering the bits when storing UUIDs (version 1) has a big performance benefit when the UUID is used as a PK for InnoDB. I assume XtraDB would be the same. Ideally this bit shuffling would be handled so clients can naively insert/select the conventionally arranged version 1 UUIDs, but get the performance benefits of the reordering.
You can count me as someone who would benefit greatly from a performant UUID data type. Thanks!
Michael Amado
added a comment - Just in case performance could be improved in the initial implementation I thought I'd link this article:
https://www.percona.com/blog/2014/12/19/store-uuid-optimized-way/ (author: Karthik Appigatla)
It suggests reordering the bits when storing UUIDs (version 1) has a big performance benefit when the UUID is used as a PK for InnoDB. I assume XtraDB would be the same. Ideally this bit shuffling would be handled so clients can naively insert/select the conventionally arranged version 1 UUIDs, but get the performance benefits of the reordering.
You can count me as someone who would benefit greatly from a performant UUID data type. Thanks!
Recommend: When storing the bits into BINARY(16), rearrange them. Rationale...
Type-1 UUIDs (which MySQL has always used) include a timestamp. But the bits of the time are scrambled. This means that indexing on a UUID leads to random accesses. When an index (or table) is big, 'consecutive' rows tend to be scattered.
By simply shuffling the bits, you can make a Type-1 UUID act like an AUTO_INCREMENT or TIMESTAMP – you can get "locality of reference" for accessing chronologically 'close' records.
More discussion here: http://mysql.rjweb.org/doc.php/uuid , and the 8.0.0 Changelog: UUID_TO_BIN() and BIN_TO_UUID() convert between UUID values in string and binary formats (represented as hexadecimal characters and VARBINARY(16), respectively). This permits conversion of string UUID values to binary values that take less storage space. UUID values converted to binary can be represented in a way that permits improved indexing efficiency.
Rick James
added a comment - Recommend: When storing the bits into BINARY(16), rearrange them. Rationale...
Type-1 UUIDs (which MySQL has always used) include a timestamp. But the bits of the time are scrambled. This means that indexing on a UUID leads to random accesses. When an index (or table) is big, 'consecutive' rows tend to be scattered.
By simply shuffling the bits, you can make a Type-1 UUID act like an AUTO_INCREMENT or TIMESTAMP – you can get "locality of reference" for accessing chronologically 'close' records.
More discussion here: http://mysql.rjweb.org/doc.php/uuid , and the 8.0.0 Changelog: UUID_TO_BIN() and BIN_TO_UUID() convert between UUID values in string and binary formats (represented as hexadecimal characters and VARBINARY(16), respectively). This permits conversion of string UUID values to binary values that take less storage space. UUID values converted to binary can be represented in a way that permits improved indexing efficiency.
Alexander Barkov
added a comment - SQL Server introduces a separate data type for this:
https://docs.microsoft.com/en-us/sql/t-sql/functions/newsequentialid-transact-sql?view=sql-server-ver15
Rick, would you suggest that it always reorders? That incurs a small performance penalty which is unnecessary for v4 UUIDs.
Perhaps a UUID(0) and UUID(1) argument to replicate the swap_flag on UUID_TO_BIN() could be used to toggle that behaviour?
Alexander, NEWSEQUENTIALID is a function returning a uuid-like 128bit value to store in a uniqueidentifier column. It isn't compatible with v1/v2 UUIDs. Are you suggesting two separate data types?
A flag affecting the storage would seem preferable to me, similar to the fsp argument on the DATETIME type.
Steven Ayre
added a comment - - edited Rick, would you suggest that it always reorders? That incurs a small performance penalty which is unnecessary for v4 UUIDs.
Perhaps a UUID(0) and UUID(1) argument to replicate the swap_flag on UUID_TO_BIN() could be used to toggle that behaviour?
Alexander, NEWSEQUENTIALID is a function returning a uuid-like 128bit value to store in a uniqueidentifier column. It isn't compatible with v1/v2 UUIDs. Are you suggesting two separate data types?
A flag affecting the storage would seem preferable to me, similar to the fsp argument on the DATETIME type.
I lean toward always reordering. (But I don't have a strong opinion.) Some debating points (on both sides):
Nothing else will look at the layout on disk, so always swapping works.
MySQL 8 has a flag – I see this as clutter with no benefit.
Would mysqldump (etc) always convert to the standard 36-char HEX layout?
Performance penalty is insignificant. (I repeatedly remind users that all the overhead of fetching a row is much higher than simple function calls.)
Compatibility with MySQL 8 (with its optional flag) may be more important than other arguments. (MariaDB has been drifting away from its "drop-in compatibility" claim.)
Swapping does not hurt type-4 (etc) uuids.
Rick James
added a comment - I lean toward always reordering. (But I don't have a strong opinion.) Some debating points (on both sides):
Nothing else will look at the layout on disk, so always swapping works.
MySQL 8 has a flag – I see this as clutter with no benefit.
Would mysqldump (etc) always convert to the standard 36-char HEX layout?
Performance penalty is insignificant. (I repeatedly remind users that all the overhead of fetching a row is much higher than simple function calls.)
Compatibility with MySQL 8 (with its optional flag) may be more important than other arguments. (MariaDB has been drifting away from its "drop-in compatibility" claim.)
Swapping does not hurt type-4 (etc) uuids.
> MySQL 8 has a flag – I see this as clutter with no benefit.
I've implemented the MySQL 8 functions as a UDF (https://github.com/SteveAyre/uuid2bin) to use until this type is added, and the reordering has a 5% performance hit. So there is a benefit. What impact it will have on the data type will depend on the implementation and yes fetching the row is more significant, but it won't be free. If you're working on millions of rows it may be important.
Steven Ayre
added a comment - - edited > MySQL 8 has a flag – I see this as clutter with no benefit.
I've implemented the MySQL 8 functions as a UDF ( https://github.com/SteveAyre/uuid2bin ) to use until this type is added, and the reordering has a 5% performance hit. So there is a benefit. What impact it will have on the data type will depend on the implementation and yes fetching the row is more significant, but it won't be free. If you're working on millions of rows it may be important.
In some situations, there is a significant performance advantage for the reshuffle.
Given:
A huge table (bigger than the buffer_pool)
PRIMARY KEY(uuid) – Type 1 with the bits rearranged as described
The working set is "recent" data
Then the activity on the table will use much less I/O, hence be much faster than if the bits were not rearranged.
(A similar argument can be made for a secondary index starting with UUID.)
Rick James
added a comment - In some situations, there is a significant performance advantage for the reshuffle.
Given:
A huge table (bigger than the buffer_pool)
PRIMARY KEY(uuid) – Type 1 with the bits rearranged as described
The working set is "recent" data
Then the activity on the table will use much less I/O, hence be much faster than if the bits were not rearranged.
(A similar argument can be made for a secondary index starting with UUID.)
Should consider an upgrade path from existing databases using byte reordering (MySQL 8 or MariaDB with either stored functions or UDF)...
Importing a dump containing BINARY(16) into a schema using the UUID type
ALTER TABLE MODIFY on a column from the BINARY(16) to UUID type
In these cases the UUID type will need to know whether the timestamp bytes are reordered or not.
Personally I like the idea of UUID(0) and UUID(1) a lot as it gives a way to specify this, or as an alternative two different UUID types.
Perhaps the default behaviour should be to reorder if the DBA hasn't specified one.
Steven Ayre
added a comment - - edited Should consider an upgrade path from existing databases using byte reordering (MySQL 8 or MariaDB with either stored functions or UDF)...
Importing a dump containing BINARY(16) into a schema using the UUID type
ALTER TABLE MODIFY on a column from the BINARY(16) to UUID type
In these cases the UUID type will need to know whether the timestamp bytes are reordered or not.
Personally I like the idea of UUID(0) and UUID(1) a lot as it gives a way to specify this, or as an alternative two different UUID types.
Perhaps the default behaviour should be to reorder if the DBA hasn't specified one.
Another thought... Instead of shuffling the bits when the UUID is stored, make use of the "datatype UUID" to shuffle the bits as it is being used in BTree accesses.
That is, the storage has the straightforward mapping between bits and hex with dashes; no rearranging.
Just as FLOAT and DECIMAL and SIGNED/UNSIGNED must interpret the bits differently when comparing, UUID would shuffle the bits when comparing.
Rick James
added a comment - Another thought... Instead of shuffling the bits when the UUID is stored, make use of the "datatype UUID" to shuffle the bits as it is being used in BTree accesses.
That is, the storage has the straightforward mapping between bits and hex with dashes; no rearranging.
Just as FLOAT and DECIMAL and SIGNED/UNSIGNED must interpret the bits differently when comparing, UUID would shuffle the bits when comparing.
Daniel Black
added a comment - To all those that are watching, a preview release is available for testing and feedback:
https://mariadb.org/10-7-preview-feature-uuid-data-type/
Hi guys, so excited to see the UUID data type as a feature! One question - does the actually implemented feature use the bits-shuffle solution to optimize performance? Or was this only an idea that didn't get implemented?
Slawomir Dymitrow
added a comment - Hi guys, so excited to see the UUID data type as a feature! One question - does the actually implemented feature use the bits-shuffle solution to optimize performance? Or was this only an idea that didn't get implemented?
Alexander Barkov
added a comment - eXsio ,
Also please see these comments:
https://github.com/MariaDB/server/blob/10.7/plugin/type_uuid/sql_type_uuid.h#L29
People
Alice Sherepa
Leo Unglaub
Votes:
36Vote for this issue
Watchers:
37Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":849.7000000476837,"ttfb":319.89999997615814,"pageVisibility":"visible","entityId":25411,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"5cfa782b-b271-48a6-af79-347fc850ae2d","navigationType":0,"readyForUser":939.8999999761581,"redirectCount":0,"resourceLoadedEnd":803.8999999761581,"resourceLoadedStart":326.89999997615814,"resourceTiming":[{"duration":11.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":326.89999997615814,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":326.89999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":338.39999997615814,"responseStart":0,"secureConnectionStart":0},{"duration":11.600000023841858,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":327.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":327.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":338.8000000715256,"responseStart":0,"secureConnectionStart":0},{"duration":66.10000002384186,"initiatorType":"script","name":"https://jira.mariadb.org/s/fbf975c0cce4b1abf04784eeae9ba1f4-CDN/lu2bu7/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":327.39999997615814,"connectEnd":327.39999997615814,"connectStart":327.39999997615814,"domainLookupEnd":327.39999997615814,"domainLookupStart":327.39999997615814,"fetchStart":327.39999997615814,"redirectEnd":0,"redirectStart":0,"requestStart":327.39999997615814,"responseEnd":393.5,"responseStart":393.5,"secureConnectionStart":327.39999997615814},{"duration":160.89999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/099b33461394b8015fc36c0a4b96e19f-CDN/lu2bu7/820016/12ta74/8679b4946efa1a0bb029a3a22206fb5d/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true","startTime":327.5,"connectEnd":327.5,"connectStart":327.5,"domainLookupEnd":327.5,"domainLookupStart":327.5,"fetchStart":327.5,"redirectEnd":0,"redirectStart":0,"requestStart":327.5,"responseEnd":488.39999997615814,"responseStart":488.39999997615814,"secureConnectionStart":327.5},{"duration":164.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/94c15bff32baef80f4096a08aceae8bc-CDN/lu2bu7/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":327.8000000715256,"connectEnd":327.8000000715256,"connectStart":327.8000000715256,"domainLookupEnd":327.8000000715256,"domainLookupStart":327.8000000715256,"fetchStart":327.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":327.8000000715256,"responseEnd":492.3000000715256,"responseStart":492.3000000715256,"secureConnectionStart":327.8000000715256},{"duration":164.80000007152557,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":328,"connectEnd":328,"connectStart":328,"domainLookupEnd":328,"domainLookupStart":328,"fetchStart":328,"redirectEnd":0,"redirectStart":0,"requestStart":328,"responseEnd":492.8000000715256,"responseStart":492.8000000715256,"secureConnectionStart":328},{"duration":164.89999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":328.2000000476837,"connectEnd":328.2000000476837,"connectStart":328.2000000476837,"domainLookupEnd":328.2000000476837,"domainLookupStart":328.2000000476837,"fetchStart":328.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":328.2000000476837,"responseEnd":493.10000002384186,"responseStart":493.10000002384186,"secureConnectionStart":328.2000000476837},{"duration":207.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2bu7/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":328.3000000715256,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":328.3000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":535.8999999761581,"responseStart":0,"secureConnectionStart":0},{"duration":165.20000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":328.5,"connectEnd":328.5,"connectStart":328.5,"domainLookupEnd":328.5,"domainLookupStart":328.5,"fetchStart":328.5,"redirectEnd":0,"redirectStart":0,"requestStart":328.5,"responseEnd":493.7000000476837,"responseStart":493.7000000476837,"secureConnectionStart":328.5},{"duration":207.39999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":328.60000002384186,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":328.60000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":536,"responseStart":0,"secureConnectionStart":0},{"duration":165.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/3339d87fa2538a859872f2df449bf8d0-CDN/lu2bu7/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":328.8000000715256,"connectEnd":328.8000000715256,"connectStart":328.8000000715256,"domainLookupEnd":328.8000000715256,"domainLookupStart":328.8000000715256,"fetchStart":328.8000000715256,"redirectEnd":0,"redirectStart":0,"requestStart":328.8000000715256,"responseEnd":494.3000000715256,"responseStart":494.3000000715256,"secureConnectionStart":328.8000000715256},{"duration":252.29999995231628,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":331.10000002384186,"connectEnd":331.10000002384186,"connectStart":331.10000002384186,"domainLookupEnd":331.10000002384186,"domainLookupStart":331.10000002384186,"fetchStart":331.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":331.10000002384186,"responseEnd":583.3999999761581,"responseStart":583.3999999761581,"secureConnectionStart":331.10000002384186},{"duration":288.39999997615814,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2bu7/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":331.10000002384186,"connectEnd":331.10000002384186,"connectStart":331.10000002384186,"domainLookupEnd":331.10000002384186,"domainLookupStart":331.10000002384186,"fetchStart":331.10000002384186,"redirectEnd":0,"redirectStart":0,"requestStart":331.10000002384186,"responseEnd":619.5,"responseStart":619.5,"secureConnectionStart":331.10000002384186},{"duration":42.60000002384186,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":547.2000000476837,"connectEnd":547.2000000476837,"connectStart":547.2000000476837,"domainLookupEnd":547.2000000476837,"domainLookupStart":547.2000000476837,"fetchStart":547.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":547.2000000476837,"responseEnd":589.8000000715256,"responseStart":589.8000000715256,"secureConnectionStart":547.2000000476837},{"duration":210.39999997615814,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2bu7/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true","startTime":593.5,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":593.5,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":803.8999999761581,"responseStart":0,"secureConnectionStart":0}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":91,"responseStart":320,"responseEnd":323,"domLoading":323,"domInteractive":1064,"domContentLoadedEventStart":1064,"domContentLoadedEventEnd":1133,"domComplete":1986,"loadEventStart":1986,"loadEventEnd":1986,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1021.5},{"name":"bigPipe.sidebar-id.end","time":1022.3000000715256},{"name":"bigPipe.activity-panel-pipe-id.start","time":1022.3999999761581},{"name":"bigPipe.activity-panel-pipe-id.end","time":1027},{"name":"activityTabFullyLoaded","time":1158.2000000476837}],"measures":[],"correlationId":"c0df9215157cfb","effectiveType":"4g","downlink":9.5,"rtt":0,"serverDuration":168,"dbReadsTimeInMs":37,"dbConnsTimeInMs":50,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
A datatype will maybe less useful for current applications, because of BC schemas, but I would recommend it for new schema.
For current schemas some (robust) conversion between textual and binary representation of UUIDs would be very helpful, although this functions can be shimed on legacy systems.