With CREATE FUNCTION one can create functions in SQL, but this syntax doesn't allow one to create an aggregate function (like SUM, AVG, etc). This task is to add support for aggregate stored functions.
Syntax
is not decided yet. SQL Standard 2003 doesn't support aggregate stored functions (may be the newer standard does). Oracle, PostgreSQL, HSQL (may be more) all implement aggregate stored functions using their own incompatible syntax extensions. SQL Server and DB2 do not support creating new aggregate stored functions in SQL.
The syntax should at least allow for
the code to return the function value
the code to be called per row in a group, it accumulates data, does not return a value
storage that is preserved between calls within a group
and possibly
code to invoke between groups to reset the storage
code to remove data from the group (useful for window functions)
Attachments
Issue Links
relates to
MDEV-14520Custom aggregate functions work incorrectly with WITH ROLLUP clause
Closed
MDEV-24580Aggregate stored function return always NULL in case of empty input set
Good idea to build CREATE TYPE inside CREATE [AGGREGATE] FUNCTION.
We should probably try to define method bodies inline (without having separate CREATE METHOD statements).
This would solve the problem with concurrent connections.
Alexander Barkov
added a comment - - edited Good idea to build CREATE TYPE inside CREATE [AGGREGATE] FUNCTION.
We should probably try to define method bodies inline (without having separate CREATE METHOD statements).
This would solve the problem with concurrent connections.
The current test case seems to corrupt the procs table:
delimiter |;
create aggregate function agg_sum(x INT) returnsdouble
begin
declare z doubledefault 0;
declarecontinue handler fornot found return z;
loop
fetchgroupnext row;
set z = z + x;
end loop;
end|
delimiter ;|
createtable t1 (id int, salary int);
INSERTINTO t1 VALUES (1, 100), (2, 40), (3, 6);
show createfunction agg_sum;
alterfunction agg_sum aggregate none;
show createfunction agg_sum;
Final query fails with:
# This fails with 'show create function agg_sum' failed: 1457: Failed to load routine test.agg_sum. The table mysql.proc is missing, corrupt, or contains bad data (internal code -6)
Vicențiu Ciorbaru
added a comment - The current test case seems to corrupt the procs table:
delimiter |;
create aggregate function agg_sum(x INT ) returns double
begin
declare z double default 0;
declare continue handler for not found return z;
loop
fetch group next row;
set z = z + x;
end loop;
end |
delimiter ;|
create table t1 (id int , salary int );
INSERT INTO t1 VALUES (1, 100), (2, 40), (3, 6);
show create function agg_sum;
alter function agg_sum aggregate none;
show create function agg_sum;
Final query fails with:
# This fails with 'show create function agg_sum' failed: 1457: Failed to load routine test.agg_sum. The table mysql.proc is missing, corrupt, or contains bad data (internal code -6)
A final rebase was required, resolved the conflicts and have sent it to cvicentiu for review.
Have pushed it to bb-10.3-varun for final testing
Varun Gupta (Inactive)
added a comment - A final rebase was required, resolved the conflicts and have sent it to cvicentiu for review.
Have pushed it to bb-10.3-varun for final testing
I haven't got anything of significance in a round of tests on bb-10.3-varun 7d8c5ee4039.
Testing of correctness of results is limited due to numerous existing problems in this area, unrelated to the new functionality. I haven't got any functionality-specific crashes.
After fixing MDEV-14520 and addressing all reviews, please go ahead and push to 10.3. If reviews require significant changes in the code, please notify so that I re-run the tests.
Please make sure the functionality is documented in the KB, if not done yet.
Among other things, I think it should be specifically mentioned in the documentation that unlike built-in functions, the custom ones don't provide automatic type recognition and act according to the types specified in the function itself. For a simple example, consider the following
createorreplacetable t1 (i int, c char(8), ci char(8));
createorreplace aggregate function agg_max1(x INT) returnsINT
begin
declare m INTdefaultNULL;
declarecontinue handler fornot found return m;
loop
fetchgroupnext row;
if (x isnotnull) and (m isnullor m < x) then
set m= x;
endif;
end loop;
end $
createorreplace aggregate function agg_max2(x BLOB) returns BLOB
begin
declare m BLOB defaultNULL;
declarecontinue handler fornot found return m;
loop
fetchgroupnext row;
if (x isnotnull) and (m isnullor m < x) then
set m= x;
endif;
end loop;
end $
delimiter ;
Here, column `i` is integer, column `ci` is text which contains integer values, and column `c` is text.
The in-built function MAX will treat each of them accordingly – the result will be based on integer comparison for `i`, on text comparison for `ci` and `c`.
MariaDB [test]> selectmax(i), max(ci), max(c) from t1;
+--------+---------+--------+
| max(i) | max(ci) | max(c) |
+--------+---------+--------+
| 10 | 2 | foo |
+--------+---------+--------+
1 row inset (0.00 sec)
Definitions of functions agg_max1 and agg_max2 are identical, except that agg_max1 works with integers and agg_max2 with blobs.
On the same data, agg_max2 will do text comparison for all columns, regardless their initial type
MariaDB [test]> select agg_max2(i), agg_max2(ci), agg_max2(c) from t1;
+-------------+--------------+-------------+
| agg_max2(i) | agg_max2(ci) | agg_max2(c) |
+-------------+--------------+-------------+
| 2 | 2 | foo |
+-------------+--------------+-------------+
1 row inset (0.00 sec)
Function agg_max1 will do integer comparison for anything that looks like integer – in this case, for `i` and for `ci`:
MariaDB [test]> select agg_max1(i), agg_max1(ci) from t1;
+-------------+--------------+
| agg_max1(i) | agg_max1(ci) |
+-------------+--------------+
| 10 | 10 |
+-------------+--------------+
1 row inset (0.00 sec)
And for something that doesn't look like integer, it doesn't work at all:
MariaDB [test]> set sql_mode= 'STRICT_ALL_TABLES';
Given that functions must define explicit types, I don't see a bug here, but it might be something users don't expect, so better to document it.
Elena Stepanova
added a comment - I haven't got anything of significance in a round of tests on bb-10.3-varun 7d8c5ee4039.
Testing of correctness of results is limited due to numerous existing problems in this area, unrelated to the new functionality. I haven't got any functionality-specific crashes.
After fixing MDEV-14520 and addressing all reviews, please go ahead and push to 10.3. If reviews require significant changes in the code, please notify so that I re-run the tests.
Please make sure the functionality is documented in the KB, if not done yet.
Among other things, I think it should be specifically mentioned in the documentation that unlike built-in functions, the custom ones don't provide automatic type recognition and act according to the types specified in the function itself. For a simple example, consider the following
create or replace table t1 (i int , c char (8), ci char (8));
insert into t1 values (10, 'foo' , '10' ),(2, 'bar' , '2' );
delimiter $
create or replace aggregate function agg_max1(x INT ) returns INT
begin
declare m INT default NULL ;
declare continue handler for not found return m;
loop
fetch group next row;
if (x is not null ) and (m is null or m < x) then
set m= x;
end if ;
end loop;
end $
create or replace aggregate function agg_max2(x BLOB) returns BLOB
begin
declare m BLOB default NULL ;
declare continue handler for not found return m;
loop
fetch group next row;
if (x is not null ) and (m is null or m < x) then
set m= x;
end if ;
end loop;
end $
delimiter ;
Here, column `i` is integer, column `ci` is text which contains integer values, and column `c` is text.
The in-built function MAX will treat each of them accordingly – the result will be based on integer comparison for `i` , on text comparison for `ci` and `c` .
MariaDB [test]> select max (i), max (ci), max (c) from t1;
+ --------+---------+--------+
| max (i) | max (ci) | max (c) |
+ --------+---------+--------+
| 10 | 2 | foo |
+ --------+---------+--------+
1 row in set (0.00 sec)
Definitions of functions agg_max1 and agg_max2 are identical, except that agg_max1 works with integers and agg_max2 with blobs.
On the same data, agg_max2 will do text comparison for all columns, regardless their initial type
MariaDB [test]> select agg_max2(i), agg_max2(ci), agg_max2(c) from t1;
+ -------------+--------------+-------------+
| agg_max2(i) | agg_max2(ci) | agg_max2(c) |
+ -------------+--------------+-------------+
| 2 | 2 | foo |
+ -------------+--------------+-------------+
1 row in set (0.00 sec)
Function agg_max1 will do integer comparison for anything that looks like integer – in this case, for `i` and for `ci` :
MariaDB [test]> select agg_max1(i), agg_max1(ci) from t1;
+ -------------+--------------+
| agg_max1(i) | agg_max1(ci) |
+ -------------+--------------+
| 10 | 10 |
+ -------------+--------------+
1 row in set (0.00 sec)
And for something that doesn't look like integer, it doesn't work at all:
MariaDB [test]> set sql_mode= 'STRICT_ALL_TABLES' ;
Query OK, 0 rows affected (0.02 sec)
MariaDB [test]> select agg_max1(c) from t1;
ERROR 1366 (22007): Incorrect integer value: 'foo' for column 'x' at row 1
MariaDB [test]> set sql_mode= '' ;
Query OK, 0 rows affected (0.00 sec)
MariaDB [test]> select agg_max1(c) from t1;
+ -------------+
| agg_max1(c) |
+ -------------+
| 0 |
+ -------------+
1 row in set , 3 warnings (0.00 sec)
Given that functions must define explicit types, I don't see a bug here, but it might be something users don't expect, so better to document it.
People
Varun Gupta (Inactive)
Sergei Golubchik
Votes:
3Vote for this issue
Watchers:
14Start watching this issue
Dates
Created:
Updated:
Resolved:
Git Integration
Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.
{"report":{"fcp":1959.5999999046326,"ttfb":671.3999996185303,"pageVisibility":"visible","entityId":50498,"key":"jira.project.issue.view-issue","isInitial":true,"threshold":1000,"elementTimings":{},"userDeviceMemory":8,"userDeviceProcessors":32,"apdex":0.5,"journeyId":"955abd5f-2930-4272-99c2-03e6c69deac3","navigationType":0,"readyForUser":2086.5999999046326,"redirectCount":0,"resourceLoadedEnd":2922.5999999046326,"resourceLoadedStart":680.8999996185303,"resourceTiming":[{"duration":447.1000003814697,"initiatorType":"link","name":"https://jira.mariadb.org/s/2c21342762a6a02add1c328bed317ffd-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/css/_super/batch.css","startTime":680.8999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":680.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1128,"responseStart":0,"secureConnectionStart":0},{"duration":447.09999990463257,"initiatorType":"link","name":"https://jira.mariadb.org/s/7ebd35e77e471bc30ff0eba799ebc151-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/css/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.css?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":681.1999998092651,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":681.1999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1128.2999997138977,"responseStart":0,"secureConnectionStart":0},{"duration":463.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/0917945aaa57108d00c5076fea35e069-CDN/lu2cib/820016/12ta74/0a8bac35585be7fc6c9cc5a0464cd4cf/_/download/contextbatch/js/_super/batch.js?locale=en","startTime":681.3999996185303,"connectEnd":681.3999996185303,"connectStart":681.3999996185303,"domainLookupEnd":681.3999996185303,"domainLookupStart":681.3999996185303,"fetchStart":681.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":681.3999996185303,"responseEnd":1144.8999996185303,"responseStart":1144.8999996185303,"secureConnectionStart":681.3999996185303},{"duration":598.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/2d8175ec2fa4c816e8023260bd8c1786-CDN/lu2cib/820016/12ta74/494e4c556ecbb29f90a3d3b4f09cb99c/_/download/contextbatch/js/jira.browse.project,project.issue.navigator,jira.view.issue,jira.general,jira.global,atl.general,-_super/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&locale=en&slack-enabled=true&whisper-enabled=true","startTime":682.6999998092651,"connectEnd":682.6999998092651,"connectStart":682.6999998092651,"domainLookupEnd":682.6999998092651,"domainLookupStart":682.6999998092651,"fetchStart":682.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":682.6999998092651,"responseEnd":1280.7999997138977,"responseStart":1280.7999997138977,"secureConnectionStart":682.6999998092651},{"duration":604.7999997138977,"initiatorType":"script","name":"https://jira.mariadb.org/s/a9324d6758d385eb45c462685ad88f1d-CDN/lu2cib/820016/12ta74/c92c0caa9a024ae85b0ebdbed7fb4bd7/_/download/contextbatch/js/atl.global,-_super/batch.js?locale=en","startTime":683.0999999046326,"connectEnd":683.0999999046326,"connectStart":683.0999999046326,"domainLookupEnd":683.0999999046326,"domainLookupStart":683.0999999046326,"fetchStart":683.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":683.0999999046326,"responseEnd":1287.8999996185303,"responseStart":1287.8999996185303,"secureConnectionStart":683.0999999046326},{"duration":608,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-en/jira.webresources:calendar-en.js","startTime":683.3999996185303,"connectEnd":683.3999996185303,"connectStart":683.3999996185303,"domainLookupEnd":683.3999996185303,"domainLookupStart":683.3999996185303,"fetchStart":683.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":683.3999996185303,"responseEnd":1291.3999996185303,"responseStart":1291.3999996185303,"secureConnectionStart":683.3999996185303},{"duration":608.5,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:calendar-localisation-moment/jira.webresources:calendar-localisation-moment.js","startTime":683.6999998092651,"connectEnd":683.6999998092651,"connectStart":683.6999998092651,"domainLookupEnd":683.6999998092651,"domainLookupStart":683.6999998092651,"fetchStart":683.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":683.6999998092651,"responseEnd":1292.1999998092651,"responseStart":1292.1999998092651,"secureConnectionStart":683.6999998092651},{"duration":669.1000003814697,"initiatorType":"link","name":"https://jira.mariadb.org/s/b04b06a02d1959df322d9cded3aeecc1-CDN/lu2cib/820016/12ta74/a2ff6aa845ffc9a1d22fe23d9ee791fc/_/download/contextbatch/css/jira.global.look-and-feel,-_super/batch.css","startTime":683.8999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":683.8999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1353,"responseStart":0,"secureConnectionStart":0},{"duration":609,"initiatorType":"script","name":"https://jira.mariadb.org/rest/api/1.0/shortcuts/820016/47140b6e0a9bc2e4913da06536125810/shortcuts.js?context=issuenavigation&context=issueaction","startTime":684.0999999046326,"connectEnd":684.0999999046326,"connectStart":684.0999999046326,"domainLookupEnd":684.0999999046326,"domainLookupStart":684.0999999046326,"fetchStart":684.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":684.0999999046326,"responseEnd":1293.0999999046326,"responseStart":1293.0999999046326,"secureConnectionStart":684.0999999046326},{"duration":668.8000001907349,"initiatorType":"link","name":"https://jira.mariadb.org/s/3ac36323ba5e4eb0af2aa7ac7211b4bb-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/css/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.css?jira.create.linked.issue=true","startTime":684.3999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":684.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":1353.1999998092651,"responseStart":0,"secureConnectionStart":0},{"duration":609.4000000953674,"initiatorType":"script","name":"https://jira.mariadb.org/s/5d5e8fe91fbc506585e83ea3b62ccc4b-CDN/lu2cib/820016/12ta74/d176f0986478cc64f24226b3d20c140d/_/download/contextbatch/js/com.atlassian.jira.projects.sidebar.init,-_super,-project.issue.navigator,-jira.view.issue/batch.js?jira.create.linked.issue=true&locale=en","startTime":684.5999999046326,"connectEnd":684.5999999046326,"connectStart":684.5999999046326,"domainLookupEnd":684.5999999046326,"domainLookupStart":684.5999999046326,"fetchStart":684.5999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":684.5999999046326,"responseEnd":1294,"responseStart":1294,"secureConnectionStart":684.5999999046326},{"duration":1247.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-js/jira.webresources:bigpipe-js.js","startTime":688.0999999046326,"connectEnd":688.0999999046326,"connectStart":688.0999999046326,"domainLookupEnd":688.0999999046326,"domainLookupStart":688.0999999046326,"fetchStart":688.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":688.0999999046326,"responseEnd":1935.1999998092651,"responseStart":1935.1999998092651,"secureConnectionStart":688.0999999046326},{"duration":2188.5999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/1.0/_/download/batch/jira.webresources:bigpipe-init/jira.webresources:bigpipe-init.js","startTime":693.6999998092651,"connectEnd":693.6999998092651,"connectStart":693.6999998092651,"domainLookupEnd":693.6999998092651,"domainLookupStart":693.6999998092651,"fetchStart":693.6999998092651,"redirectEnd":0,"redirectStart":0,"requestStart":693.6999998092651,"responseEnd":2882.2999997138977,"responseStart":2882.2999997138977,"secureConnectionStart":693.6999998092651},{"duration":546.0999999046326,"initiatorType":"xmlhttprequest","name":"https://jira.mariadb.org/rest/webResources/1.0/resources","startTime":1397.0999999046326,"connectEnd":1397.0999999046326,"connectStart":1397.0999999046326,"domainLookupEnd":1397.0999999046326,"domainLookupStart":1397.0999999046326,"fetchStart":1397.0999999046326,"redirectEnd":0,"redirectStart":0,"requestStart":1397.0999999046326,"responseEnd":1943.1999998092651,"responseStart":1943.1999998092651,"secureConnectionStart":1397.0999999046326},{"duration":1109.1000003814697,"initiatorType":"script","name":"https://www.google-analytics.com/analytics.js","startTime":1950.3999996185303,"connectEnd":0,"connectStart":0,"domainLookupEnd":0,"domainLookupStart":0,"fetchStart":1950.3999996185303,"redirectEnd":0,"redirectStart":0,"requestStart":0,"responseEnd":3059.5,"responseStart":0,"secureConnectionStart":0},{"duration":945.0999999046326,"initiatorType":"script","name":"https://jira.mariadb.org/s/d41d8cd98f00b204e9800998ecf8427e-CDN/lu2cib/820016/12ta74/e65b778d185daf5aee24936755b43da6/_/download/contextbatch/js/browser-metrics-plugin.contrib,-_super,-project.issue.navigator,-jira.view.issue,-atl.general/batch.js?agile_global_admin_condition=true&jag=true&jira.create.linked.issue=true&slack-enabled=true&whisper-enabled=true","startTime":1977.5,"connectEnd":1977.5,"connectStart":1977.5,"domainLookupEnd":1977.5,"domainLookupStart":1977.5,"fetchStart":1977.5,"redirectEnd":0,"redirectStart":0,"requestStart":1977.5,"responseEnd":2922.5999999046326,"responseStart":2922.5999999046326,"secureConnectionStart":1977.5}],"fetchStart":0,"domainLookupStart":0,"domainLookupEnd":0,"connectStart":0,"connectEnd":0,"requestStart":488,"responseStart":672,"responseEnd":693,"domLoading":675,"domInteractive":3068,"domContentLoadedEventStart":3068,"domContentLoadedEventEnd":3127,"domComplete":4301,"loadEventStart":4301,"loadEventEnd":4301,"userAgent":"Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)","marks":[{"name":"bigPipe.sidebar-id.start","time":2900},{"name":"bigPipe.sidebar-id.end","time":2901.199999809265},{"name":"bigPipe.activity-panel-pipe-id.start","time":2901.3999996185303},{"name":"bigPipe.activity-panel-pipe-id.end","time":2917.8999996185303},{"name":"activityTabFullyLoaded","time":3146}],"measures":[],"correlationId":"bf5df42f17a38f","effectiveType":"4g","downlink":9.3,"rtt":0,"serverDuration":128,"dbReadsTimeInMs":19,"dbConnsTimeInMs":30,"applicationHash":"9d11dbea5f4be3d4cc21f03a88dd11d8c8687422","experiments":[]}}
Good idea to build CREATE TYPE inside CREATE [AGGREGATE] FUNCTION.
We should probably try to define method bodies inline (without having separate CREATE METHOD statements).
This would solve the problem with concurrent connections.