SELECT INTO OUTFILE handles very badly a mixuture of multibyte character data and binary data. See MDEV-34883 as and example.
With utf8mb4 being the default character set since MDEV-19123 (11.6.0) the problem becomes even more critical.
LOAD DATA INFILE uses @@character_set_database, which is now typically utf8mb4, as the character set of the loaded file by default.
In order for text and binary data to co-exists in a dump file, let's add a way to dump binary data using HEX representation.
Let's also fix LOAD DATA INFILE to detect and decode a hex representation.
Let's add a new option as follows:
SELECT ... INTO OUTFILE 'file_name'
[CHARACTERSET charset_name]
[binary_encoding]
[export_options]
binary_encoding: BINARY ENCODED USING HEX
If this option is specified then columns of the data types BINARY, VARBINARY, BLOB, Geometry (and its variants) will be dumped using HEX encoding.
Columns with hex encoded will have the "\X" prefix followed by HEX data.
Let's also change LOAD DATA INFILE to automatically decode HEX representation if a field data in the file starts with "\X".
This does not conflict with previous versions because SELECT INTO OUTFILE cannot produce a sequence of "\X" in the very beginning of a field data.
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE" to dump load anything including binary data? — that works already now just fine. Use SELECT ... INTO OUTFILE without charset specifier and LOAD DATA INFILE CHARSET BINARY
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE" to dump load anything including binary data by default without any additional syntax? — this problem isn't solved, because the suggested solution requires the user to specify BINARY ENCODED USING HEX.
So, what is it?
Sergei Golubchik
added a comment - What problem we're solving here?
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE " to dump load anything including binary data? — that works already now just fine. Use SELECT ... INTO OUTFILE without charset specifier and LOAD DATA INFILE CHARSET BINARY
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE " to dump load anything including binary data by default without any additional syntax? — this problem isn't solved, because the suggested solution requires the user to specify BINARY ENCODED USING HEX .
So, what is it?
People
Unassigned
Alexander Barkov
Votes:
0Vote for this issue
Watchers:
2Start watching this issue
Dates
Created:
Updated:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":836.1000003814697,"ttfb":188.2000002861023,"pageVisibility":"visible","entityId":130808,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":1,"journeyId":"ba5eabf0-be87-4192-8f10-74b5ad080140","navigationType":0,"readyForUser":921.1000003814697,"redirectCount":0,"resourceLoadedEnd":782.8000001907349,"resourceLoadedStart":193.80000019073486,"resourceTiming":[{"duration":47.59999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":193.80000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":193.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":241.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":51.39999961853027,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":194.10000038146973,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":194.10000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":245.5,"responseStart":0,"secureConnectionStart":0},{"duration":108.90000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":194.2000002861023,"connectEnd":194.2000002861023,"connectStart":194.2000002861023,"domainLookupEnd":194.2000002861023,"domainLookupStart":194.2000002861023,"fetchStart":194.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":194.2000002861023,"responseEnd":303.1000003814697,"responseStart":303.1000003814697,"secureConnectionStart":194.2000002861023},{"duration":191.40000009536743,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":194.5,"connectEnd":194.5,"connectStart":194.5,"domainLookupEnd":194.5,"domainLookupStart":194.5,"fetchStart":194.5,"redirectEnd":0,"redirectStart":0,"requestStart":194.5,"responseEnd":385.90000009536743,"responseStart":385.90000009536743,"secureConnectionStart":194.5},{"duration":195.2999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":194.60000038146973,"connectEnd":194.60000038146973,"connectStart":194.60000038146973,"domainLookupEnd":194.60000038146973,"domainLookupStart":194.60000038146973,"fetchStart":194.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":194.60000038146973,"responseEnd":389.90000009536743,"responseStart":389.90000009536743,"secureConnectionStart":194.60000038146973},{"duration":195.59999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":194.90000009536743,"connectEnd":194.90000009536743,"connectStart":194.90000009536743,"domainLookupEnd":194.90000009536743,"domainLookupStart":194.90000009536743,"fetchStart":194.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":194.90000009536743,"responseEnd":390.5,"responseStart":390.5,"secureConnectionStart":194.90000009536743},{"duration":195.80000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":195,"connectEnd":195,"connectStart":195,"domainLookupEnd":195,"domainLookupStart":195,"fetchStart":195,"redirectEnd":0,"redirectStart":0,"requestStart":195,"responseEnd":390.80000019073486,"responseStart":390.80000019073486,"secureConnectionStart":195},{"duration":252.69999980926514,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":195.30000019073486,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":195.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":448,"responseStart":0,"secureConnectionStart":0},{"duration":196,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":195.40000009536743,"connectEnd":195.40000009536743,"connectStart":195.40000009536743,"domainLookupEnd":195.40000009536743,"domainLookupStart":195.40000009536743,"fetchStart":195.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":195.40000009536743,"responseEnd":391.40000009536743,"responseStart":391.40000009536743,"secureConnectionStart":195.40000009536743},{"duration":252.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":195.60000038146973,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":195.60000038146973,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":448.1000003814697,"responseStart":0,"secureConnectionStart":0},{"duration":196.30000019073486,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":195.80000019073486,"connectEnd":195.80000019073486,"connectStart":195.80000019073486,"domainLookupEnd":195.80000019073486,"domainLookupStart":195.80000019073486,"fetchStart":195.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":195.80000019073486,"responseEnd":392.1000003814697,"responseStart":392.1000003814697,"secureConnectionStart":195.80000019073486},{"duration":565,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":196.80000019073486,"connectEnd":196.80000019073486,"connectStart":196.80000019073486,"domainLookupEnd":196.80000019073486,"domainLookupStart":196.80000019073486,"fetchStart":196.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":196.80000019073486,"responseEnd":761.8000001907349,"responseStart":761.8000001907349,"secureConnectionStart":196.80000019073486},{"duration":565.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":196.90000009536743,"connectEnd":196.90000009536743,"connectStart":196.90000009536743,"domainLookupEnd":196.90000009536743,"domainLookupStart":196.90000009536743,"fetchStart":196.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":196.90000009536743,"responseEnd":762.4000000953674,"responseStart":762.4000000953674,"secureConnectionStart":196.90000009536743},{"duration":250.39999961853027,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":459.6000003814697,"connectEnd":459.6000003814697,"connectStart":459.6000003814697,"domainLookupEnd":459.6000003814697,"domainLookupStart":459.6000003814697,"fetchStart":459.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":459.6000003814697,"responseEnd":710,"responseStart":710,"secureConnectionStart":459.6000003814697},{"duration":57.5,"initiatorType":"link","name":"https://jira.mariadb.org/s/d5715adaadd168a9002b108b2b039b50-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/css/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":725.3000001907349,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":725.3000001907349,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":782.8000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":294.69999980926514,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":726.2000002861023,"connectEnd":726.2000002861023,"connectStart":726.2000002861023,"domainLookupEnd":726.2000002861023,"domainLookupStart":726.2000002861023,"fetchStart":726.2000002861023,"redirectEnd":0,"redirectStart":0,"requestStart":726.2000002861023,"responseEnd":1020.9000000953674,"responseStart":1020.9000000953674,"secureConnectionStart":726.2000002861023},{"duration":299.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/097ae97cb8fbec7d6ea4bbb1f26955b9-CDN/lu2cib/820016/12ta74/be4b45e9cec53099498fa61c8b7acba4/_/download/contextbatch/js/jira.project.sidebar,-_super,-project.issue.navigator,-jira.general,-jira.browse.project,-jira.view.issue,-jira.global,-atl.general,-com.atlassian.jira.projects.sidebar.init/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":726.6000003814697,"connectEnd":726.6000003814697,"connectStart":726.6000003814697,"domainLookupEnd":726.6000003814697,"domainLookupStart":726.6000003814697,"fetchStart":726.6000003814697,"redirectEnd":0,"redirectStart":0,"requestStart":726.6000003814697,"responseEnd":1025.7000002861023,"responseStart":1025.7000002861023,"secureConnectionStart":726.6000003814697}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":28,"responseStart":188,"responseEnd":190,"domLoading":191,"domInteractive":976,"domContentLoadedEventStart":976,"domContentLoadedEventEnd":1023,"domComplete":1621,"loadEventStart":1621,"loadEventEnd":1621,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":959},{"name":"bigPipe.sidebar-id.end","time":959.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.start","time":960},{"name":"bigPipe.activity-panel-pipe-id.end","time":961.1000003814697},{"name":"activityTabFullyLoaded","time":1041.8000001907349}],"measures":[],"correlationId":"5c291067dd6ff9","effectiveType":"4g","downlink":9.2,"rtt":0,"serverDuration":97,"dbReadsTimeInMs":11,"dbConnsTimeInMs":19,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
What problem we're solving here?
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE" to dump load anything including binary data? — that works already now just fine. Use SELECT ... INTO OUTFILE without charset specifier and LOAD DATA INFILE CHARSET BINARY
"allow SELECT ... INTO OUTFILE + LOAD DATA INFILE" to dump load anything including binary data by default without any additional syntax? — this problem isn't solved, because the suggested solution requires the user to specify BINARY ENCODED USING HEX.
So, what is it?