Uploaded image for project: 'MariaDB ColumnStore'
  1. MariaDB ColumnStore
  2. MCOL-683

ADDDATE & DATE_FORMAT combo breaks datetime

Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Fixed
    • 1.0.8
    • 1.0.9, 1.1.0
    • None
    • None
    • centos7
    • 2017-9

    Description

      Tableau generated queries which manipulate dates are broken in Columnstore but worked in InfiniDB. In the query below you can see that the first column "breaks" in that the returned value is not a date.

      Query example:

      SELECT 
          ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) AS `tableau_date`,
          cast(ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) as datetime) AS `casted_date`,
          datecol as raw_date
      FROM
          columnstore_table
      WHERE
          datecol >= CURRENT_DATE()
      

      Attachments

        Activity

          Cause is MariaDB tells the engine DATE_ADD/ADDDATE is 3 byte UTF8 and we are handling the length as if it is 1 byte. So we truncate at 1/3rd of the length.

          LinuxJedi Andrew Hutchings (Inactive) added a comment - Cause is MariaDB tells the engine DATE_ADD/ADDDATE is 3 byte UTF8 and we are handling the length as if it is 1 byte. So we truncate at 1/3rd of the length.

          Pull requests open for develop and develop-1.0 as well as regression suite.

          For testing:

          CREATE TABLE `columnstore_table` (
            `a` int(11) DEFAULT NULL,
            `datecol` date DEFAULT NULL
          ) ENGINE=Columnstore
           
          INSERT INTO columnstore_table values (1, '2017-04-25');
          

          Then the queries in the example, on failure the first column will have a date truncated at 6 chars. Also regression suite changes cover this.

          LinuxJedi Andrew Hutchings (Inactive) added a comment - - edited Pull requests open for develop and develop-1.0 as well as regression suite. For testing: CREATE TABLE `columnstore_table` ( `a` int (11) DEFAULT NULL , `datecol` date DEFAULT NULL ) ENGINE=Columnstore   INSERT INTO columnstore_table values (1, '2017-04-25' ); Then the queries in the example, on failure the first column will have a date truncated at 6 chars. Also regression suite changes cover this.

          [root@localhost mariadb-columnstore-server]# git show
          commit 5a090c64bced6532578dd8910f15530fd37fce2c
          Merge: 9efb0a7 b062156
          Author: david hill <david.hill@mariadb.com>
          Date: Thu Apr 27 16:13:15 2017 -0500
          Merge pull request #45 from mariadb-corporation/FixPackageName
          Change the package name to match engines format
          [root@localhost mariadb-columnstore-server]# cd mariadb-columnstore-engine/
          [root@localhost mariadb-columnstore-engine]# git show
          commit cb788c0be215ad30e77e35298142fef43c61e70e
          Merge: 775c605 de06c48
          Author: dhall-InfiniDB <david.hall@mariadb.com>
          Date: Wed May 3 09:43:29 2017 -0500
          Merge pull request #163 from mariadb-corporation/MCOL-697
          MCOL-697 Remove 64KB VARCHAR response limit

          Reproduced the issue in 1.0.8-1
          MariaDB [tpch1]> SELECT ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) AS `tableau_date`, cast(ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) as datetime) AS `casted_date`, datecol as raw_date FROM columnstore_table WHERE datecol <= CURRENT_DATE();
          -------------------------------------------

          tableau_date casted_date raw_date

          -------------------------------------------

          2017-0 2017-04-25 00:00:00 2017-04-25

          -------------------------------------------
          1 row in set (0.01 sec)

          The same query in 1.1.0-1 returned:
          --------------------------------------------------

          tableau_date casted_date raw_date

          --------------------------------------------------

          2017-04-25 00:00:00 2017-04-25 00:00:00 2017-04-25

          --------------------------------------------------
          1 row in set (0.01 sec)

          dleeyh Daniel Lee (Inactive) added a comment - [root@localhost mariadb-columnstore-server] # git show commit 5a090c64bced6532578dd8910f15530fd37fce2c Merge: 9efb0a7 b062156 Author: david hill <david.hill@mariadb.com> Date: Thu Apr 27 16:13:15 2017 -0500 Merge pull request #45 from mariadb-corporation/FixPackageName Change the package name to match engines format [root@localhost mariadb-columnstore-server] # cd mariadb-columnstore-engine/ [root@localhost mariadb-columnstore-engine] # git show commit cb788c0be215ad30e77e35298142fef43c61e70e Merge: 775c605 de06c48 Author: dhall-InfiniDB <david.hall@mariadb.com> Date: Wed May 3 09:43:29 2017 -0500 Merge pull request #163 from mariadb-corporation/ MCOL-697 MCOL-697 Remove 64KB VARCHAR response limit Reproduced the issue in 1.0.8-1 MariaDB [tpch1] > SELECT ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) AS `tableau_date`, cast(ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) as datetime) AS `casted_date`, datecol as raw_date FROM columnstore_table WHERE datecol <= CURRENT_DATE(); ------------- ------------------- ----------- tableau_date casted_date raw_date ------------- ------------------- ----------- 2017-0 2017-04-25 00:00:00 2017-04-25 ------------- ------------------- ----------- 1 row in set (0.01 sec) The same query in 1.1.0-1 returned: -------------------- ------------------- ----------- tableau_date casted_date raw_date -------------------- ------------------- ----------- 2017-04-25 00:00:00 2017-04-25 00:00:00 2017-04-25 -------------------- ------------------- ----------- 1 row in set (0.01 sec)

          Build verified: Github source

          [root@localhost mariadb-columnstore-server]# git show
          commit 6e3afaaca8abf9903662878b457bdd63af6adbea
          Merge: 8ef673b 719b99d
          Author: dhall-InfiniDB <david.hall@mariadb.com>
          Date: Thu May 4 10:31:50 2017 -0500

          Merge pull request #49 from mariadb-corporation/MCOL-679-1.0

          MCOL-679 Fix the vtable prep stmt parser

          [root@localhost mariadb-columnstore-server]# cd mariadb-columnstore-engine/
          [root@localhost mariadb-columnstore-engine]# git show
          commit 85c7be75625b7b3f515b83956ee2b0acb6c550a3
          Merge: b6efffb f5881ce
          Author: dhall-InfiniDB <david.hall@mariadb.com>
          Date: Wed Apr 26 09:42:51 2017 -0500

          Merge pull request #159 from mariadb-corporation/MCOL-683-1.0

          MCOL-683 Fix nested ADDDATE and DATE_FORMAT

          Also verified the fix in 1.0.9-1
          [root@localhost ~]# mcsmysql mytest
          Welcome to the MariaDB monitor. Commands end with ; or \g.
          Your MariaDB connection id is 6
          Server version: 10.1.22-MariaDB Columnstore 1.0.9-1

          Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

          Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

          MariaDB [mytest]> CREATE TABLE `columnstore_table` (
          -> `a` int(11) DEFAULT NULL,
          -> `datecol` date DEFAULT NULL
          -> ) ENGINE=Columnstore;
          Query OK, 0 rows affected (0.11 sec)

          MariaDB [mytest]> INSERT INTO columnstore_table values (1, '2017-04-25');
          Query OK, 1 row affected (0.19 sec)

          MariaDB [mytest]> SELECT ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) AS `tableau_date`, cast(ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) as datetime) AS `casted_date`, datecol as raw_date FROM columnstore_table WHERE datecol <= CURRENT_DATE();
          --------------------------------------------------

          tableau_date casted_date raw_date

          --------------------------------------------------

          2017-04-25 00:00:00 2017-04-25 00:00:00 2017-04-25

          --------------------------------------------------
          1 row in set (0.05 sec)

          dleeyh Daniel Lee (Inactive) added a comment - Build verified: Github source [root@localhost mariadb-columnstore-server] # git show commit 6e3afaaca8abf9903662878b457bdd63af6adbea Merge: 8ef673b 719b99d Author: dhall-InfiniDB <david.hall@mariadb.com> Date: Thu May 4 10:31:50 2017 -0500 Merge pull request #49 from mariadb-corporation/ MCOL-679 -1.0 MCOL-679 Fix the vtable prep stmt parser [root@localhost mariadb-columnstore-server] # cd mariadb-columnstore-engine/ [root@localhost mariadb-columnstore-engine] # git show commit 85c7be75625b7b3f515b83956ee2b0acb6c550a3 Merge: b6efffb f5881ce Author: dhall-InfiniDB <david.hall@mariadb.com> Date: Wed Apr 26 09:42:51 2017 -0500 Merge pull request #159 from mariadb-corporation/ MCOL-683 -1.0 MCOL-683 Fix nested ADDDATE and DATE_FORMAT Also verified the fix in 1.0.9-1 [root@localhost ~] # mcsmysql mytest Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 6 Server version: 10.1.22-MariaDB Columnstore 1.0.9-1 Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [mytest] > CREATE TABLE `columnstore_table` ( -> `a` int(11) DEFAULT NULL, -> `datecol` date DEFAULT NULL -> ) ENGINE=Columnstore; Query OK, 0 rows affected (0.11 sec) MariaDB [mytest] > INSERT INTO columnstore_table values (1, '2017-04-25'); Query OK, 1 row affected (0.19 sec) MariaDB [mytest] > SELECT ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) AS `tableau_date`, cast(ADDDATE(DATE_FORMAT(datecol, '%Y-%m-%d %H:00:00'), INTERVAL 0 SECOND) as datetime) AS `casted_date`, datecol as raw_date FROM columnstore_table WHERE datecol <= CURRENT_DATE(); -------------------- ------------------- ----------- tableau_date casted_date raw_date -------------------- ------------------- ----------- 2017-04-25 00:00:00 2017-04-25 00:00:00 2017-04-25 -------------------- ------------------- ----------- 1 row in set (0.05 sec)
          gcleaves Geoff Cleaves added a comment -

          Just noticed a problem which may be related:

          select least(max(datefield),cast('2000-01-01' as datetime)) from cstable
          results in '0000-00-00 00:00:00'

          when the expected value is the maximum date in the column.

          gcleaves Geoff Cleaves added a comment - Just noticed a problem which may be related: select least(max(datefield),cast('2000-01-01' as datetime)) from cstable results in '0000-00-00 00:00:00' when the expected value is the maximum date in the column.

          Hi Geoff,

          That will be a completely separate problem. MCOL-683 is after the functions have processed and the result is going back to the MariaDB server. This new case is something happening earlier.

          Can you please file a new bug report for this with some test data?

          LinuxJedi Andrew Hutchings (Inactive) added a comment - Hi Geoff, That will be a completely separate problem. MCOL-683 is after the functions have processed and the result is going back to the MariaDB server. This new case is something happening earlier. Can you please file a new bug report for this with some test data?

          People

            dleeyh Daniel Lee (Inactive)
            gcleaves Geoff Cleaves
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.