[MCOL-964] tpcds query78 alternately fails and works with incorrect results Created: 2017-10-09  Updated: 2018-02-02  Resolved: 2018-02-02

Status: Closed
Project: MariaDB ColumnStore
Component/s: MariaDB Server
Affects Version/s: 1.1.0
Fix Version/s: 1.1.3

Type: Bug Priority: Major
Reporter: David Thompson (Inactive) Assignee: Daniel Lee (Inactive)
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
includes MCOL-963 self join cte queries from tpcds alte... Closed
Relates
relates to MCOL-963 self join cte queries from tpcds alte... Closed
Sprint: 2017-21, 2017-25, 2018-01, 2018-02, 2018-03

 Description   

The following query utilizing CTE's will alternately fail and then work and so on:

#QUERY_78
with ws as
  (select d_year AS ws_sold_year, ws_item_sk,
    ws_bill_customer_sk ws_customer_sk,
    sum(ws_quantity) ws_qty,
    sum(ws_wholesale_cost) ws_wc,
    sum(ws_sales_price) ws_sp
   from web_sales
   left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk
   join date_dim on ws_sold_date_sk = d_date_sk
   where wr_order_number is null
   group by d_year, ws_item_sk, ws_bill_customer_sk
   ),
cs as
  (select d_year AS cs_sold_year, cs_item_sk,
    cs_bill_customer_sk cs_customer_sk,
    sum(cs_quantity) cs_qty,
    sum(cs_wholesale_cost) cs_wc,
    sum(cs_sales_price) cs_sp
   from catalog_sales
   left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk
   join date_dim on cs_sold_date_sk = d_date_sk
   where cr_order_number is null
   group by d_year, cs_item_sk, cs_bill_customer_sk
   ),
ss as
  (select d_year AS ss_sold_year, ss_item_sk,
    ss_customer_sk,
    sum(ss_quantity) ss_qty,
    sum(ss_wholesale_cost) ss_wc,
    sum(ss_sales_price) ss_sp
   from store_sales
   left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk
   join date_dim on ss_sold_date_sk = d_date_sk
   where sr_ticket_number is null
   group by d_year, ss_item_sk, ss_customer_sk
   )
 select 
ss_sold_year, ss_item_sk, ss_customer_sk,
round(ss_qty/(coalesce(ws_qty,0)+coalesce(cs_qty,0)),2) ratio,
ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price,
coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty,
coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost,
coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price
from ss
left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk)
left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=ss_item_sk and cs_customer_sk=ss_customer_sk)
where (coalesce(ws_qty,0)>0 or coalesce(cs_qty, 0)>0) and ss_sold_year=2000
order by 
  ss_sold_year, ss_item_sk, ss_customer_sk,
  ss_qty desc, ss_wc desc, ss_sp desc,
  other_chan_qty,
  other_chan_wholesale_cost,
  other_chan_sales_price,
  round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
limit 100;

with the following error:
ERROR 1054 (42S22) at line 2 in file: 'query78.sql': Unknown column 'tpcds_1.ss.ss_sold_year' in 'field list'



 Comments   
Comment by David Thompson (Inactive) [ 2017-10-30 ]

This jira will cover a complete fix for this and should also work for the additional queries in MCOL-963

Comment by David Hall (Inactive) [ 2017-10-30 ]

There's a problem with math in the order by clause. removing
round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2)
stops the problem.

Tests will be done to see if math on aggregates is the problem, or if any math causes issues.

Comment by David Hall (Inactive) [ 2018-01-15 ]

The query in question now results in the following error:
ERROR 1815 (HY000): Internal error: Query cannot be processed using operational mode 1 (vtable mode)
MariaDB [tpcds_1]>

Running in mode 0 results in the correct result, but execution time is unacceptable (hours).
MCOL-964 is left open to fix the vtable mode 1 problem, but is not needed for any specific release.

Comment by David Hall (Inactive) [ 2018-01-17 ]

A simpler query that demonstrates the problem:

with ss as
(select ss_item_sk, ss_customer_sk,
sum(ss_quantity) ss_qty
from store_sales
group by ss_item_sk, ss_customer_sk
)

select
ss_item_sk, ss_customer_sk
from ss
order by
ss_item_sk, ss_customer_sk,
round(ss_qty,2)
limit 10;

This even simpler query asserts mysqld in debug (see MDEV-14981)

with ss as
(select ss_item_sk,
sum(ss_quantity) ss_qty
from store_sales
group by ss_item_sk
)

select
ss_item_sk
from ss
order by
ss_item_sk,
round(ss_qty,2)
limit 10;

Comment by David Hall (Inactive) [ 2018-01-17 ]

The logic say that if an order by column isn't already in the select – round(ss_qty,2) is not already selected – then add it to the select as a hidden column and re-run the query. Something breaks when it's a cte.

Comment by David Hall (Inactive) [ 2018-01-30 ]

The bug is caused by a field in the order by that isn't in the select list. Columnstore attempts to re-write and re-run the query. During the rewrite, the CTE is converted to a derived table, which works fine. However, it attempts to prepend the names of the select fields with the database name. This is invalid for derived tables which have no database. the fields couldn't be found by the fully qualified name and it errored.

By changing the place in server (Columnstore only code) where the name is written to not prepend with the database name, it works like a normal derived table.

I have a small amount of trepidation that there is some use case for the database prepend other than for views, but I can't think of one. It passes the standard regression test.

Comment by Daniel Lee (Inactive) [ 2018-02-02 ]

Reproduced the issue in 1.1.2-1
Build verified: Github source 1.1.3-1

root@stretch:~# cat mariadb-columnstore-1.1.3-1-stretch.x86_64.bin.tar.txt
/root/columnstore/mariadb-columnstore-server
commit e5499e513d88a3dfefbe9a356e20a1bceb1bde38
Merge: 99cdb0a4b5 4840a435aa
Author: david hill <david.hill@mariadb.com>
Date: Wed Jan 31 16:53:52 2018 -0600

Merge pull request #92 from mariadb-corporation/MCOL-1152

MCOL-1152: Change the debian package names.

/root/columnstore/mariadb-columnstore-server/mariadb-columnstore-engine
commit 446cf74a68eaaa8697a448d8de0327e0459d4341
Merge: c792188a 6f4cfcdc
Author: David.Hall <david.hall@mariadb.com>
Date: Fri Feb 2 13:09:51 2018 -0600

Merge pull request #392 from mariadb-corporation/MCOL-1070

MCOL-1070 Fix exists in view subquery

Both the original query 78, and Mr. Hall's simpler query returned the following error.

ERROR 1815 (HY000): Internal error: Query cannot be processed using operational mode 1 (vtable mode)

The original error was reported in 1.1.0 and the above was put in in 1.1.2-1

Both queries returned result in 1.1.3-1

Generated at Thu Feb 08 02:25:08 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.