[MDEV-17785] Window functions not working in ONLY_FULL_GROUP_BY mode II Created: 2018-11-20  Updated: 2022-04-04  Resolved: 2022-02-07

Status: Closed
Project: MariaDB Server
Component/s: Optimizer - Window functions
Affects Version/s: 10.3.11, 10.2, 10.3, 10.4, 10.5
Fix Version/s: 10.2.43, 10.3.34, 10.4.24, 10.5.15, 10.6.7, 10.7.3

Type: Bug Priority: Critical
Reporter: Jonas Staudenmeir Assignee: Sergei Petrunia
Resolution: Fixed Votes: 14
Labels: None
Environment:

Ubuntu 18.04.1


Issue Links:
Duplicate
is duplicated by MDEV-24409 NTILE example throws: ERROR 1140 (420... Closed

 Description   

This is a follow-up to MDEV-17525.

The patch fixed the AVG() function from the example query, but other window functions still don't work in ONLY_FULL_GROUP_BY mode.

Using a different example from the documentation:

CREATE TABLE student(course VARCHAR(10), mark int, name varchar(10));
 
INSERT INTO student VALUES 
  ('Maths', 60, 'Thulile'),
  ('Maths', 60, 'Pritha'),
  ('Maths', 70, 'Voitto'),
  ('Maths', 55, 'Chun'),
  ('Biology', 60, 'Bilal'),
   ('Biology', 70, 'Roger');
 
SET sql_mode = 'ONLY_FULL_GROUP_BY';
 
SELECT 
  RANK() OVER (PARTITION BY course ORDER BY mark DESC) AS rank, 
  DENSE_RANK() OVER (PARTITION BY course ORDER BY mark DESC) AS dense_rank, 
  ROW_NUMBER() OVER (PARTITION BY course ORDER BY mark DESC) AS row_num, 
  course, mark, name 
FROM student ORDER BY course, mark DESC;

The SELECT query fails with an error:

Mixing of GROUP columns (MIN(),MAX(),COUNT(),...) with no GROUP columns is illegal if there is no GROUP BY clause



 Comments   
Comment by Varun Gupta (Inactive) [ 2020-07-09 ]

This is another bug where we treat the window function as an Item_sum. Window function is treated as an aggregate function and this leads to the problem here.
The query does not have any aggregate function and we still hit the problem

Comment by Varun Gupta (Inactive) [ 2020-07-09 ]

Here is the code where we see the error being thrown

/*
    Check if there are references to un-aggregated columns when computing 
    aggregate functions with implicit grouping (there is no GROUP BY).
  */
  if (thd->variables.sql_mode & MODE_ONLY_FULL_GROUP_BY && !group_list &&
      !(select_lex->master_unit()->item &&
        select_lex->master_unit()->item->is_in_predicate() &&
        ((Item_in_subselect*)select_lex->master_unit()->item)->
        test_set_strategy(SUBS_MAXMIN_INJECTED)) &&
      select_lex->non_agg_field_used() &&
      select_lex->agg_func_used())
  {
    my_message(ER_MIX_OF_GROUP_FUNC_AND_FIELDS,
               ER_THD(thd, ER_MIX_OF_GROUP_FUNC_AND_FIELDS), MYF(0));
    DBUG_RETURN(-1);
  }

In the debugger I see

(lldb) p select_lex->agg_func_used()
(bool) $4 = true
(lldb) p select_lex->non_agg_field_used()
(bool) $5 = true

This means that the query has an aggregate function and atleast one field with non-aggregate function.
This does not happen in our case, we have window functions in the query, no aggregate functions.
This means that the window function is being treated as an aggregate function here.

The function select_lex->agg_func_used should return FALSE in this case

Comment by Varun Gupta (Inactive) [ 2020-07-09 ]

CREATE TABLE t1(a VARCHAR(10), b int);
INSERT INTO t1 VALUES
('Maths', 60),('Maths', 60), ('Maths', 70),('Maths', 55), ('Biology', 60), ('Biology', 70);
SET sql_mode = 'ONLY_FULL_GROUP_BY';

MariaDB [test]> SELECT a, b, SUM(b) OVER (PARTITION BY a ORDER BY b) AS rank FROM t1 ORDER BY a, b DESC;
+---------+------+------+
| a       | b    | rank |
+---------+------+------+
| Biology |   70 |  130 |
| Biology |   60 |   60 |
| Maths   |   70 |  245 |
| Maths   |   60 |  175 |
| Maths   |   60 |  175 |
| Maths   |   55 |   55 |
+---------+------+------+
6 rows in set (0.003 sec)

With SUM function used as a window function, there is no problem here.

MariaDB [test]> SELECT a, b, RANK() OVER (PARTITION BY a ORDER BY b) AS rank FROM t1 ORDER BY a, b DESC;
ERROR 1140 (42000): Mixing of GROUP columns (MIN(),MAX(),COUNT(),...) with no GROUP columns is illegal if there is no GROUP BY clause

With RANK function we hit the error.

Comment by Varun Gupta (Inactive) [ 2020-07-09 ]

Debugging in the function Item_sum::check_sum_func

For the query:

SELECT a, b, SUM(b) OVER (PARTITION BY a ORDER BY b) AS rank FROM t1 ORDER BY a, b DESC;

(lldb) p window_func_sum_expr_flag
(bool) $0 = true

and we return from this function.

For the second query

SELECT a, b, RANK() OVER (PARTITION BY a ORDER BY b) AS rank FROM t1 ORDER BY a, b DESC;

(lldb) p window_func_sum_expr_flag
(bool) $1 = false
(lldb) p this
(Item_sum_rank *) $2 = 0x000062b000000648

Debugging till the end of the function

(lldb) p aggr_sel->agg_func_used()
(bool) $3 = false
(lldb) n
Process 66555 stopped
* thread #2, stop reason = step over
    frame #0: 0x00000001004ff508 mysqld`Item_sum::check_sum_func(this=0x000062b000000648, thd=0x000062a00005a270, ref=0x000062b000000c38) at item_sum.cc:318:7
   315 	    }
   316 	  }
   317 	  aggr_sel->set_agg_func_used(true);
-> 318 	  if (sum_func() == SP_AGGREGATE_FUNC)
   319 	    aggr_sel->set_custom_agg_func_used(true);
   320 	  update_used_tables();
   321 	  thd->lex->in_sum_func= in_sum_func;
Target 0: (mysqld) stopped.
(lldb) p aggr_sel->agg_func_used()
(bool) $4 = true

We set that this is marked as an aggregate function. This is wrong, we need to make sure that this is an Item_sum instance that is used as a window function.

Comment by Varun Gupta (Inactive) [ 2020-07-09 ]

Patch
http://lists.askmonty.org/pipermail/commits/2020-July/014283.html

Comment by Nuno [ 2020-12-27 ]

I've just come across this now.

I was wondering why was MariaDB complaining about GROUP BY clauses/columns, when I was trying to use ROW_NUMBER(), without any GROUP BY stuff.
Then I found this Issue Ticket here.

My @@sql_mode has "ONLY_FULL_GROUP_BY", as should be, and I can confirm that ROW_NUMBER() works if I disable it.

SET sql_mode = (SELECT REPLACE(REPLACE(@@sql_mode, ",ONLY_FULL_GROUP_BY", ""), "ONLY_FULL_GROUP_BY,", ""));
SELECT
ROW_NUMBER() OVER (PARTITION BY `col1` ORDER BY `col2` DESC) AS `rownum`,
...

Of course, this is not ideal...

Comment by Nuno [ 2021-02-23 ]

igor Any idea when the fix can be reviewed and merged?
Thanks!

Comment by Nuno [ 2021-10-28 ]

psergei igor - any news about this one, please?

Comment by Dan Livingston [ 2021-11-17 ]

This is something that is critical for us as well as it is causing problems - any updates?

Comment by Paulus Limma [ 2022-01-20 ]

Is this going to be fixed anytime soon? IMO this is a really critical bug which is blocking me to use basic features of the SQL.

Comment by Sergei Petrunia [ 2022-02-07 ]

The second patch is ok. I've edited the commit comment to have more information.

Comment by Sergei Petrunia [ 2022-02-07 ]

bb-10.2-mdev17785

Generated at Thu Feb 08 08:39:05 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.