[MCOL-3828] UNION: complains about table not in query Created: 2020-02-21  Updated: 2020-11-12  Resolved: 2020-03-30

Status: Closed
Project: MariaDB ColumnStore
Component/s: ExeMgr
Affects Version/s: 1.4.3
Fix Version/s: 1.4.3-5, 1.4.4, 1.5.1

Type: Bug Priority: Critical
Reporter: David Hall (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
Relates
relates to MCOL-3747 Regression in 1.4 working_ssb_compare... Closed
Sprint: 2020-4, 2020-5
Epic/Theme: 1.4_regressions

 Description   

Consider the following sequence from working_tpch/union/union.sql

CREATE TABLE t1 (
cid smallint(5) ,
cv varchar(250)
) engine=columnstore;

INSERT INTO t1 VALUES (8,'dummy');
CREATE TABLE t2 (
cid bigint(20),
cap varchar(255)
) engine=columnstore;
CREATE TABLE t3 (
gid bigint(20) ,
gn varchar(255),
must tinyint(4)
) engine=columnstore;
INSERT INTO t3 VALUES (1,'V1',NULL);
CREATE TABLE t4 (
uid bigint(20),
gid bigint(20),
rid bigint(20),
cid bigint(20)
) engine=columnstore;

INSERT INTO t4 VALUES (1,1,NULL,NULL);
CREATE TABLE t5 (
rid bigint(20),
rl varchar(255)
) engine=columnstore;
CREATE TABLE t6 (
uid bigint(20),
un varchar(250),
uc smallint(5)
) engine=columnstore;
INSERT INTO t6 VALUES (1,'test',8);

SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test";
SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t3.must IS NOT NULL AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test";
(SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t3.must IS NOT NULL AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test") UNION (SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test");
drop table if exists t1;
drop table if exists t2;
drop table if exists t3;
drop table if exists t4;
drop table if exists t5;
drop table if exists t6;

create table t1 (a int) engine=columnstore;
insert into t1 values (1),(2),(3);
create table t2 (a int) engine=columnstore;
insert into t2 values (3),(4),(5);

SELECT * FROM t1 UNION SELECT * FROM t2 ORDER BY a desc;
ERROR 1815 (HY000): Internal error: IDB-2006: 'tpch1.t6' does not exist in Columnstore.

This only happens if you run the full sequence. Otherwise, there's no complaint about a table not existing – one that isn't in the query at all.



 Comments   
Comment by David Hall (Inactive) [ 2020-02-25 ]

This appears to be a similar problem that sometimes occurs in working_tpch1_comparelogonly.view/view.sql
CREATE TABLE t1 (lid int, name char(10)) engine=columnstore;
INSERT INTO t1 (lid, name) VALUES (1, 'YES'), (2, 'NO');

CREATE TABLE t2 ( id int, gid int, lid int, dt date) engine=columnstore;
INSERT INTO t2 (id, gid, lid, dt) VALUES
(1, 1, 1, '2007-01-01'),(2, 1, 2, '2007-01-02'),
(3, 2, 2, '2007-02-01'),(4, 2, 1, '2007-02-02');

SELECT DISTINCT t2.gid AS lgid,(SELECT t1.name FROM t1, t2 WHERE t1.lid = t2.lid AND t2.gid = t1.lid and t1.lid =1 ) as clid FROM t2 order by 1, 2;

CREATE VIEW v1 AS SELECT DISTINCT t2.gid AS lgid,(SELECT t1.name FROM t1, t2 WHERE t1.lid = t2.lid AND t2.gid = t1.lid and t1.lid =1 ) as clid FROM t2;
SELECT * FROM v1 order by 1, 2;

DROP VIEW v1;
DROP table t1;
drop table t2;

CREATE TABLE t1 (a INT) engine=columnstore;
INSERT INTO t1 VALUES (1),(2),(3);
CREATE VIEW v1 AS SELECT a FROM t1 ORDER BY a;

SELECT * FROM t1 UNION SELECT * FROM v1;
ERROR 1815 (HY000) at line 896: Internal error: IDB-2006: 'tpch1.t2' does not exist in Columnstore.

Comment by David Hall (Inactive) [ 2020-03-05 ]

Got a different error this time:
create table t1 (a int) engine=columnstore;
insert into t1 values (1),(2),(3);
create table t2 (a int) engine=columnstore;
insert into t2 values (3),(4),(5);
SELECT * FROM t1 UNION SELECT * FROM t2 ORDER BY a desc;

ERROR 1815 (HY000) at line 115: Internal error: value is not numerical.

Comment by David Hall (Inactive) [ 2020-03-11 ]

Found a similar problem in working_tpch1_compareLogOnly/view/mts_view.sql lines 896 and 897

Comment by Roman [ 2020-03-20 ]

So, there were two bugs that caused the issues:
"Internal error: value is not numerical" and "'tpch1.t6' does not exist in Columnstore" were caused by our conditions pushdown mechanism that doesn't clean after query execution with table API. I fixed the issue but the whole mechanism needs to be rewritten given this is an easy thing to do.

tables are not joined was caused by join order and conditions replacement introduced by MDB's optimizer that was called by optimize_unflattened_subqueries_mcs() function under the hood. I replaced the f() to address equi-JOIN conditions injection for IN + subquery predicates.

Comment by Roman [ 2020-03-20 ]

Plz review.

Comment by Daniel Lee (Inactive) [ 2020-03-30 ]

Builds verified: 1.4.4-1 source, 1.5.0-1 BB

1.4.4-1

/root/ColumnStore/buildColumnstoreFromGithubSource/server
commit ec0071afa50b78930860a14802b8cfc9791f7d13
Author: Sergei Petrunia <psergey@askmonty.org>
Date: Thu Mar 26 01:26:39 2020 +0300

MDEV-21887: federatedx crashes on SELECT ... INTO query in select_handler code

/root/ColumnStore/buildColumnstoreFromGithubSource/server/engine
commit 6e953d01d268161ac50dabb6aa9a122ed1d43e0f
Merge: 4468c93 ba06727
Author: Gagan Goel <gagan.nith@gmail.com>
Date: Thu Mar 26 11:41:35 2020 -0400

Merge pull request #1117 from drrtuy/MCOL-3828_1_4

MCOL-3828 This commit replaces the method that calls JOIN::optimise()

1.5.0-1

engine commit:
a35b208

Verified tests cases in both bug description and in comments.

Comment by Daniel Lee (Inactive) [ 2020-04-01 ]

Build verified: 1.4.3-5 hot fix, Azure 20200401.6

MariaDB [mytest]> CREATE TABLE t1 (
-> cid smallint(5) ,
-> cv varchar(250)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.161 sec)

MariaDB [mytest]> INSERT INTO t1 VALUES (8,'dummy');
Query OK, 1 row affected (0.222 sec)

MariaDB [mytest]> CREATE TABLE t2 (
-> cid bigint(20),
-> cap varchar(255)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.164 sec)

MariaDB [mytest]> CREATE TABLE t3 (
-> gid bigint(20) ,
-> gn varchar(255),
-> must tinyint(4)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.160 sec)

MariaDB [mytest]> INSERT INTO t3 VALUES (1,'V1',NULL);
Query OK, 1 row affected (0.343 sec)

MariaDB [mytest]> CREATE TABLE t4 (
-> uid bigint(20),
-> gid bigint(20),
-> rid bigint(20),
-> cid bigint(20)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.184 sec)

MariaDB [mytest]> INSERT INTO t4 VALUES (1,1,NULL,NULL);
t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test");
drop table if exists t1;
drop table if exists t2;
drop table if exists t3;
drop table if exists t4;
drop table if exists t5;
drop table if exists t6;
Query OK, 1 row affected (0.369 sec)

MariaDB [mytest]> CREATE TABLE t5 (
-> rid bigint(20),
-> rl varchar(255)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.170 sec)

MariaDB [mytest]> CREATE TABLE t6 (
-> uid bigint(20),
-> un varchar(250),
-> uc smallint(5)
-> ) engine=columnstore;
Query OK, 0 rows affected (0.177 sec)

MariaDB [mytest]> INSERT INTO t6 VALUES (1,'test',8);
Query OK, 1 row affected (0.295 sec)

MariaDB [mytest]> SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test";
----------------------

uid rl g1 cid gg

----------------------

1 NULL V1 NULL 1

----------------------
1 row in set (0.236 sec)

MariaDB [mytest]> SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t3.must IS NOT NULL AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test";
Empty set (0.064 sec)

MariaDB [mytest]> (SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t3.must IS NOT NULL AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test") UNION (SELECT t4.uid, t5.rl, t3.gn as g1, t4.cid, t4.gid as gg FROM t3, t6, t1, t4 left join t5 on t5.rid = t4.rid left join t2 on t2.cid = t4.cid WHERE t3.gid=t4.gid AND t6.uid = t4.uid AND t6.uc = t1.cid AND t1.cv = "dummy" AND t6.un = "test");
----------------------

uid rl g1 cid gg

----------------------

1 NULL V1 NULL 1

----------------------
1 row in set (0.138 sec)

MariaDB [mytest]> drop table if exists t1;
Query OK, 0 rows affected (0.293 sec)

MariaDB [mytest]> drop table if exists t2;
Query OK, 0 rows affected (0.199 sec)

MariaDB [mytest]> drop table if exists t3;
Query OK, 0 rows affected (0.193 sec)

MariaDB [mytest]> drop table if exists t4;
Query OK, 0 rows affected (0.209 sec)

MariaDB [mytest]> drop table if exists t5;
Query OK, 0 rows affected (0.395 sec)

MariaDB [mytest]> drop table if exists t6;
Query OK, 0 rows affected (0.211 sec)

MariaDB [mytest]> create table t1 (a int) engine=columnstore;
Query OK, 0 rows affected (0.156 sec)

MariaDB [mytest]> insert into t1 values (1),(2),(3);
Query OK, 3 rows affected (0.327 sec)
Records: 3 Duplicates: 0 Warnings: 0

MariaDB [mytest]> create table t2 (a int) engine=columnstore;
Query OK, 0 rows affected (0.142 sec)

MariaDB [mytest]> insert into t2 values (3),(4),(5);
Query OK, 3 rows affected (0.327 sec)
Records: 3 Duplicates: 0 Warnings: 0

MariaDB [mytest]> SELECT * FROM t1 UNION SELECT * FROM t2 ORDER BY a desc;
------

a

------

5
4
3
2
1

------
5 rows in set (0.177 sec)

Generated at Thu Feb 08 02:45:45 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.