The day after upgrading to 10.0.13, we found our daily ETL processes hung (we use MariaDB for a data warehouse). We have CONNECT tables that point to various databases: MySQL, DB2, and Oracle. Not all remote databases are available 24/7--we stop them during off hours to reduce our costs. Some of the remote tables are quite large and exceed 10 million rows.
We frequently query the INFORMATION_SCHEMA database during our ETL cycle. When querying information_schema.tables in particular, instead of returning instantly as it did on earlier versions of MariaDB, it would appear to hang for 30-45 minutes. I determined that if you query particular columns such as "create_time", the CONNECT engine appears to be reaching out to the remote databases to retrieve information about the underlying tables. If the remote database was unavailable, it would hang and eventually time out, and if it was available, it appeared to be getting a rowcount for the remote table. This causes the query to "hang" if a large number of CONNECT tables refer to databases that are temporarily unavailable or if the tables are very large. This was definitely true of remote MySQL tables--we downgraded back to 10.0.12 before I was able to finish testing with the ODBC connections to DB2 and Oracle.
To be clear, this behavior is occurring when querying INFORMATION_SCHEMA. I'm not actually trying to pull data from the CONNECT tables via a select.
After downgrading to 10.0.12, the behavior returned to normal, and querying INFORMATION_SCHEMA only returned whatever information had been stored locally about the remote tables.
Is there a way to disable this remote querying? Since remote connections/queries can be expensive, I would prefer to only access the underlying tables when querying the CONNECT tables directly, and not when performing simple queries of the INFORMATION_SCHEMA. I'm not sure if this new behavior was intentional, but it is undesirable for us and will limit our ability to use the CONNECT engine going forward when we eventually upgrade MariaDB. For now, we are staying on 10.0.12.
{"report":{"fcp":1097.3000001907349,"ttfb":317.7000000476837,"pageVisibility":"visible","entityId":43519,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":64,"apdex":0.5,"journeyId":"f7e804ad-6611-4c09-adfe-4d82acf4e16c","navigationType":0,"readyForUser":1159.4000000953674,"redirectCount":0,"resourceLoadedEnd":1023.6000001430511,"resourceLoadedStart":323,"resourceTiming":[{"duration":7.400000095367432,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":323,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":323,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":330.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":7.400000095367432,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":323.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":323.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":330.60000014305115,"responseStart":0,"secureConnectionStart":0},{"duration":305,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":323.30000019073486,"connectEnd":323.30000019073486,"connectStart":323.30000019073486,"domainLookupEnd":323.30000019073486,"domainLookupStart":323.30000019073486,"fetchStart":323.30000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":323.30000019073486,"responseEnd":628.3000001907349,"responseStart":628.3000001907349,"secureConnectionStart":323.30000019073486},{"duration":157.70000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":323.60000014305115,"connectEnd":323.60000014305115,"connectStart":323.60000014305115,"domainLookupEnd":323.60000014305115,"domainLookupStart":323.60000014305115,"fetchStart":323.60000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":323.60000014305115,"responseEnd":481.30000019073486,"responseStart":481.30000019073486,"secureConnectionStart":323.60000014305115},{"duration":161.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":323.80000019073486,"connectEnd":323.80000019073486,"connectStart":323.80000019073486,"domainLookupEnd":323.80000019073486,"domainLookupStart":323.80000019073486,"fetchStart":323.80000019073486,"redirectEnd":0,"redirectStart":0,"requestStart":323.80000019073486,"responseEnd":484.90000009536743,"responseStart":484.90000009536743,"secureConnectionStart":323.80000019073486},{"duration":160.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":323.90000009536743,"connectEnd":323.90000009536743,"connectStart":323.90000009536743,"domainLookupEnd":323.90000009536743,"domainLookupStart":323.90000009536743,"fetchStart":323.90000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":323.90000009536743,"responseEnd":484.40000009536743,"responseStart":484.40000009536743,"secureConnectionStart":323.90000009536743},{"duration":161.20000004768372,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":324.10000014305115,"connectEnd":324.10000014305115,"connectStart":324.10000014305115,"domainLookupEnd":324.10000014305115,"domainLookupStart":324.10000014305115,"fetchStart":324.10000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":324.10000014305115,"responseEnd":485.30000019073486,"responseStart":485.30000019073486,"secureConnectionStart":324.10000014305115},{"duration":162.20000004768372,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":324.2000000476837,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":324.2000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":486.40000009536743,"responseStart":0,"secureConnectionStart":0},{"duration":306.7999999523163,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":324.40000009536743,"connectEnd":324.40000009536743,"connectStart":324.40000009536743,"domainLookupEnd":324.40000009536743,"domainLookupStart":324.40000009536743,"fetchStart":324.40000009536743,"redirectEnd":0,"redirectStart":0,"requestStart":324.40000009536743,"responseEnd":631.2000000476837,"responseStart":631.2000000476837,"secureConnectionStart":324.40000009536743},{"duration":306.7000000476837,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":324.60000014305115,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":324.60000014305115,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":631.3000001907349,"responseStart":0,"secureConnectionStart":0},{"duration":307,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":324.7000000476837,"connectEnd":324.7000000476837,"connectStart":324.7000000476837,"domainLookupEnd":324.7000000476837,"domainLookupStart":324.7000000476837,"fetchStart":324.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":324.7000000476837,"responseEnd":631.7000000476837,"responseStart":631.7000000476837,"secureConnectionStart":324.7000000476837},{"duration":479.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":333.7000000476837,"connectEnd":333.7000000476837,"connectStart":333.7000000476837,"domainLookupEnd":333.7000000476837,"domainLookupStart":333.7000000476837,"fetchStart":333.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":333.7000000476837,"responseEnd":813.2000000476837,"responseStart":813.2000000476837,"secureConnectionStart":333.7000000476837},{"duration":683.9000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":339.7000000476837,"connectEnd":339.7000000476837,"connectStart":339.7000000476837,"domainLookupEnd":339.7000000476837,"domainLookupStart":339.7000000476837,"fetchStart":339.7000000476837,"redirectEnd":0,"redirectStart":0,"requestStart":339.7000000476837,"responseEnd":1023.6000001430511,"responseStart":1023.6000001430511,"secureConnectionStart":339.7000000476837},{"duration":181.59999990463257,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":839.4000000953674,"connectEnd":839.4000000953674,"connectStart":839.4000000953674,"domainLookupEnd":839.4000000953674,"domainLookupStart":839.4000000953674,"fetchStart":839.4000000953674,"redirectEnd":0,"redirectStart":0,"requestStart":839.4000000953674,"responseEnd":1021,"responseStart":1021,"secureConnectionStart":839.4000000953674},{"duration":112.90000009536743,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1075,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1075,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1187.9000000953674,"responseStart":0,"secureConnectionStart":0},{"duration":151.09999990463257,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1097.1000001430511,"connectEnd":1097.1000001430511,"connectStart":1097.1000001430511,"domainLookupEnd":1097.1000001430511,"domainLookupStart":1097.1000001430511,"fetchStart":1097.1000001430511,"redirectEnd":0,"redirectStart":0,"requestStart":1097.1000001430511,"responseEnd":1248.2000000476837,"responseStart":1248.2000000476837,"secureConnectionStart":1097.1000001430511}],"fetchStart":0,"domainLookupStart":89,"domainLookupEnd":137,"connectStart":137,"connectEnd":162,"secureConnectionStart":145,"requestStart":162,"responseStart":318,"responseEnd":340,"domLoading":322,"domInteractive":1222,"domContentLoadedEventStart":1222,"domContentLoadedEventEnd":1263,"domComplete":1424,"loadEventStart":1424,"loadEventEnd":1424,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":1191},{"name":"bigPipe.sidebar-id.end","time":1191.8000001907349},{"name":"bigPipe.activity-panel-pipe-id.start","time":1192},{"name":"bigPipe.activity-panel-pipe-id.end","time":1194.1000001430511},{"name":"activityTabFullyLoaded","time":1282.5}],"measures":[],"correlationId":"75fb0ab277eb68","effectiveType":"4g","downlink":9.1,"rtt":0,"serverDuration":93,"dbReadsTimeInMs":9,"dbConnsTimeInMs":17,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Indeed, I recently modified the way some information about tables was returned in order to have more precise values. In particular, the number of records info may be long to retrieve on some large tables. However, this happens only when a fast way to get it is not possible. Normally, retrieving this info is fast for all fixed record size table types (FIX, BIN, DBF,and VEC)
It is also fast for variable record length tables (DOS, CSV, FMT) when they are indexed because the table size can be retrieved from an index. The only long case are variable record length tables when they are not indexed.
For remote tables, to get the number of records info, CONNECT sends
select count(*) from remtab
to the remote server and some ODBC data source have not a fast way to return that (before, I was just returning 10)
Now, to handle your problems, I must have some examples of INFORMATION_SCHEMA queries that take a long time to be sure to find when and where there is a performance issue to fix. Then I could for instance add a CONNECT global variable allowing to tell whether or not one needs to have exact values when querying for table info or if just an estimate is enough.