[MDEV-5719] Wrong result with GROUP BY and LEFT OUTER JOIN Created: 2014-02-24  Updated: 2015-01-27  Resolved: 2015-01-27

Status: Closed
Project: MariaDB Server
Component/s: Optimizer
Affects Version/s: 10.0.8
Fix Version/s: 10.0.16

Type: Bug Priority: Major
Reporter: Pavel Ivanov Assignee: Sergei Petrunia
Resolution: Fixed Votes: 12
Labels: None

Issue Links:
Duplicate
is duplicated by MDEV-7193 Incorrect Query Result (MySQL Bug 688... Closed
is duplicated by MDEV-7240 Cannot use "group by" with "left join" Closed
Relates
relates to MDEV-6417 Wrong results when using join and group Closed
relates to MDEV-6786 Wrong join query result with outer_jo... Closed

 Description   

Here's the mysql session log:

Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.0.8-MariaDB-log Source distribution
 
Copyright (c) 2000, 2013, Oracle, Monty Program Ab and others.
 
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
 
MariaDB [(none)]> create database test;
Query OK, 1 row affected (0.00 sec)
 
MariaDB [(none)]> use test;
Database changed
MariaDB [test]> CREATE TABLE t (id bigint(20), name varchar(30),
parent bigint(20), primary key (id)) ENGINE=InnoDB;
Query OK, 0 rows affected (0.27 sec)
 
MariaDB [test]> INSERT INTO t values (1, 'a', NULL), (4, 'b', 1), (5,
'c', 4), (10, 'd', 1);
Query OK, 4 rows affected (0.04 sec)
Records: 4  Duplicates: 0  Warnings: 0

MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT
OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a') GROUP BY
t1.id;
+----+------+------+------+
| id | name | id   | name |
+----+------+------+------+
|  1 | a    | NULL | NULL |
|  4 | b    | NULL | NULL |
|  5 | c    | NULL | NULL |
| 10 | d    | NULL | NULL |
+----+------+------+------+
4 rows in set (0.00 sec)

MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT
OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a');
+----+------+------+------+
| id | name | id   | name |
+----+------+------+------+
|  1 | a    | NULL | NULL |
|  4 | b    | NULL | NULL |
|  5 | c    |    4 | b    |
| 10 | d    | NULL | NULL |
+----+------+------+------+
4 rows in set (0.00 sec)

MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT
OUTER JOIN t t2 ON (t1.parent = t2.id and t2.name <> 'a') GROUP BY
t1.id, t1.name;
+----+------+------+------+
| id | name | id   | name |
+----+------+------+------+
|  1 | a    | NULL | NULL |
|  4 | b    | NULL | NULL |
|  5 | c    |    4 | b    |
| 10 | d    | NULL | NULL |
+----+------+------+------+
4 rows in set (0.00 sec)

MariaDB [test]> SELECT t1.id, t1.name, t2.id, t2.name FROM t t1 LEFT
OUTER JOIN t t2 ON (t1.parent = t2.id) GROUP BY t1.id;
+----+------+------+------+
| id | name | id   | name |
+----+------+------+------+
|  1 | a    | NULL | NULL |
|  4 | b    |    1 | a    |
|  5 | c    |    4 | b    |
| 10 | d    |    1 | a    |
+----+------+------+------+
4 rows in set (0.00 sec)

As you see the first query returns NULLs instead of real values which
can be seen in second query without GROUP BY. And if I add another
column to GROUP BY or remove one condition from JOIN I can see real
values again. I've also tested with the table without PRIMARY KEY and
it also returned real values in the first query.



 Comments   
Comment by Sergei Petrunia [ 2014-02-25 ]

Investigation details have been posted here:
https://lists.launchpad.net/maria-developers/msg06830.html

Comment by Sergei Petrunia [ 2014-02-25 ]

if I roll back the change we've got with merge from mysql-5.6, then the testcase provided in this bug passes, and the 'main' testsuite passes, too.

=== modified file 'sql/field.h'
--- sql/field.h 2013-11-20 11:05:39 +0000
+++ sql/field.h 2014-02-25 16:16:22 +0000
@@ -599,25 +599,8 @@ class Field
   */
   virtual void sql_type(String &str) const =0;
   virtual uint size_of() const =0;             // For new field
-  inline bool is_null(my_ptrdiff_t row_offset= 0) const
-  {
-    /*
-      The table may have been marked as containing only NULL values
-      for all fields if it is a NULL-complemented row of an OUTER JOIN                                             
-      or if the query is an implicitly grouped query (has aggregate                                                
-      functions but no GROUP BY clause) with no qualifying rows. If                                                
-      this is the case (in which TABLE::null_row is true), the field                                               
-      is considered to be NULL.                                                                                    
-      Note that if a table->null_row is set then also all null_bits are                                            
-      set for the row.                                                                                             
-
-      Otherwise, if the field is NULLable, it has a valid null_ptr
-      pointer, and its NULLity is recorded in the "null_bit" bit of
-      null_ptr[row_offset].
-    */
-    return (table->null_row ? TRUE :
-            null_ptr ? test(null_ptr[row_offset] & null_bit) : 0);
-  }
+  inline bool is_null(my_ptrdiff_t row_offset= 0)
+  { return null_ptr ? (null_ptr[row_offset] & null_bit ? 1 : 0) : table->null_row; }
   inline bool is_real_null(my_ptrdiff_t row_offset= 0) const
     { return null_ptr ? (null_ptr[row_offset] & null_bit ? 1 : 0) : 0; }
   inline bool is_null_in_record(const uchar *record) const

Comment by Sergei Petrunia [ 2014-02-25 ]

Related patches in MySQL:

revid: sergey.glukhov@oracle.com-20130619102408-0a0bf6sl5d7tsysa
Bug#16620047 INCORRECT QUERY RESULT (AFTER SERVER UPGRADE)
Regression is introduced in Bug11760517. According to the
comment for the Bug11760517 fix, there is an additional cleanup
in field.h(Field::is_null() function):
...

^ this one doesn't have a testcase

jorgen.loland@oracle.com-20111220120431-pxmaax8e3effzhj0
BUG#13430588: WRONG RESULT FROM IMPLICITLY GROUPED QUERY 
              WITH CONST TABLE AND NO MATCHING ROWS
BUG#13422961: WRONG RESULTS FROM SELECT WITH AGGREGATES
              AND IMPLICIT GROUPING + MYISAM OR MEM
BUG#11760517: 52935: MIN/MAX FAILS TO EVALUATE HAVING 
              CONDITION, RETURNS INCORRECT NULL RESULT
 
When implicitly grouped queries do not have any rows matching
the join predicates, a row based on NULL values is returned
(if not filtered out by the HAVING clause). This means that
(a) non-aggregated fields should have the value NULL and (b)
aggregated fields should be calculated based on NULL.
 
For (a), this is achieved by marking all tables as containing
only NULL values by calling mark_as_null_row(). For (b),
this is achieved by calling Item::no_rows_in_result().
 
There were multiple bugs in this area:
 1) In locations where mark_as_null_row() was called, const
    tables were ignored. This has been fixed by storing the
    info mark_as_null_row() modifies for const tables, call 
    mark_as_null_row() for all tables including const tables,
    send the result and finally restore const table info.
    Note that save/restore is only required for subqueries 
    since the value will not be reused otherwise. This was 
    the case for end_send_group() and end_write_group()
 2) Item::no_rows_in_result() was not called everywhere where
    aggregate items needed to be calculated on NULL values. This
    was the case for do_select().
 3) mark_as_null_row() was not called everywhere non-aggregate
    items needed to be set to NULL. This was the case for
    do_select() and end_write_group().
 4) the having clause was evaluated after (a) but before (b).
    This was the case in return_zero_rows()
 
Additional cleanup: 
 * mark_as_null_row() no longer sets the NULL bits, only
   table->null_row and table->status
 * Field::is_null() now checks table->null_row before checking
   the NULL bits.

^ this one has three testcases.

Comment by Sergei Petrunia [ 2014-02-25 ]

I'm trying testcases from MySQL bug on mariadb-10.0 + the above patch.

Bug#11760517: passes, the result is same as in mysql-5.6

Bug#13422961:

MySQL has:

SELECT col_int_nokey, MAX( pk ) 
FROM ot 
WHERE (8, 1) IN ( SELECT pk, COUNT( col_int_nokey ) FROM it );
col_int_nokey   MAX( pk )
NULL    NULL

MariaDB 10.0 + patch has:

SELECT col_int_nokey, MAX( pk ) 
FROM ot 
WHERE (8, 1) IN ( SELECT pk, COUNT( col_int_nokey ) FROM it );
col_int_nokey  MAX( pk )
8      NULL

As for 13430588, MariaDB and MySQL produce the same result, but is it really
the correct one:

#
# Bug#13430588: WRONG RESULT FROM IMPLICITLY GROUPED QUERY WITH
#               CONST TABLE AND NO MATCHING ROWS
#
CREATE TABLE t1 (i INT) ENGINE=MyISAM;
INSERT INTO t1 VALUES (1);
CREATE TABLE t2 (j INT) ENGINE=MyISAM;
INSERT INTO t2 VALUES (1),(2);
 
SELECT i, j, COUNT(i) FROM t1 JOIN t2 WHERE j=3;
i       j       COUNT(i)
1       NULL    0

The join didn't produce a single row. Is it correct that output has i=1 ?

Comment by Sergei Petrunia [ 2014-03-04 ]

In MySQL 5.6, they've changed it to produce a non-NULL value intentionally:
http://bazaar.launchpad.net/~mysql/mysql-server/5.6/revision/tor.didriksen@oracle.com-20130628082431-1ebjbeqcu0fet5ik

Comment by Elena Stepanova [ 2014-05-17 ]

Another test case from https://mariadb.com/kb/en/select-with-group-by-behavior-in-55-vs-100/ :

--source include/have_innodb.inc
 
create table table_one (oidGroup int, oid int primary key) engine=InnoDB;
insert into table_one values (1,1),(1,2),(1,3),(1,4);
 
create table table_two (oid int primary key) engine=InnoDB;
insert into table_two values (3);
 
# Returns a value
select a.oidGroup, a.oid, b.oid from table_one a left join table_two b on a.oid=b.oid where a.oidGroup=1;
 
# Returns only NULLs
select a.oidGroup, a.oid, b.oid from table_one a left join table_two b on a.oid=b.oid where a.oidGroup=1 group by a.oid;

Comment by alex stacey [ 2014-05-29 ]

Hello

Can anyone tell me under which conditions this bug happens? From the first test-case, I thought it can happen on any 2 tables joined with a LEFT JOIN with >1 join conditions and GROUP BY, but it seems from the last test case above that it doesn't need >1 join conditions.

I have found that there are lots of ways the query can be slightly changed which will un-trigger the bug and return correct values but I still don't understand the conditions that the bug happen under.

We are running this version of MariaDB in a production environment and it would be great to know what to look for to see if this is affecting us.

Thanks

Comment by Aaron [ 2014-11-11 ]

Hi,

Can anyone provide information about which release the bug fix for this will be in? Or at least a status update?

Cheers

Comment by Sergei Petrunia [ 2014-11-26 ]

Hi Aaron,

Thanks for your patience in waiting for this reply. Unfortunately, no work has been done so far. This bug is scheduled for the next 10.0 release (10.0.16) but I cannot promise at the moment.

Comment by Cameron Dale [ 2014-12-10 ]

It seems this was fixed in version 5.6.13 of MySQL. Doesn't that mean the work needed to include it in Maria DB is minimal?

http://bugs.mysql.com/bug.php?id=68897

Comment by Nirbhay Choubey (Inactive) [ 2015-01-23 ]

http://lists.askmonty.org/pipermail/commits/2015-January/007324.html

Comment by Nirbhay Choubey (Inactive) [ 2015-01-23 ]

Updated the test case to create INNODB tables (instead of default MyISAM) :
http://lists.askmonty.org/pipermail/commits/2015-January/007327.html

Comment by Sergei Petrunia [ 2015-01-25 ]

Explanations about why the fix above actually fixes the problem:
https://lists.launchpad.net/maria-developers/msg08094.html

Comment by Sergei Petrunia [ 2015-01-25 ]

Pushed the fix into 10.0 tree. Will close this MDEV when we know which version it is released in.

Comment by Nirbhay Choubey (Inactive) [ 2015-01-27 ]

Fix released in 10.0.16.

Generated at Thu Feb 08 07:06:32 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.