Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-29307

Wrong result when joining two derived tables over the same view

Details

    Description

      I found an interesting bug introduced between 10.3.9 which was working fine and 10.3.34 which is already broken. I tested it up to 10.8.4 and it is still broken.

      I have a table of water consumption measurements (attached) in regular times for hot and cold water. And I need to convert it to a daily water consumption table. I'm using this.

      	WITH
      		twater_daily_raw AS (
      			SELECT DATE_FORMAT(`timestamp`, '%Y-%m-%d') 'date',
      			EXTRACT(YEAR FROM `timestamp`) 'year',
      			EXTRACT(MONTH FROM `timestamp`) 'month',
      			EXTRACT(DAY FROM `timestamp`) 'day',
      			`flat`, `type`,
      			(SELECT `value` FROM water sw WHERE sw.`timestamp` = MAX(w.`timestamp`) AND sw.`flat` = w.`flat` AND sw.`type` = w.`type`) 'total',
      			MAX(timestamp) 'last'
      			FROM `test2` w
      			GROUP BY `date`, `flat`, `type`
      			ORDER by `date`, `flat`, `type`
      		),
       
      		twater_daily_usage AS (
      			SELECT *,
      			(`total` - IFNULL(LAG(`total`, 1) OVER (partition by `type`, `flat` ORDER by `type`, `flat`, `date`),0)) 'usage'
      			FROM `twater_daily_raw`
                  ORDER by `date`, `flat`, `type`
      		),
       
      		water_daily_combined AS (
      			SELECT wc.`date`, wh.`date`'date2', wc.year, wc.month, wc.day, wc.flat,
      					wc.`usage` 'cold', wh.`usage` 'hot',
      					wh.total 'hot_total', wc.total 'cold_total'
      			FROM twater_daily_usage wc, twater_daily_usage wh
      			WHERE wc.`date` = wh.`date` AND wc.`flat` = wh.`flat` AND wh.`type` = 'WATER_HOT' AND wc.`type` = 'WATER_COLD'
       
      		)
       
      SELECT * FROM water_daily_combined wdc
      

      In the first step, I get the biggest value from every day, in the second step I evaluate usage by using over partition by and in the last step, I simply join twice the result table from the previous step to get cold and hot data to one line.

      You can see result in result.png image, the data from second table are always missing.
      It was working just fine for a long time, since I upgraded Mariad DB from 10.3.9 to 10.3.34 , because I upgraded Raspbian OS on my mini server. But the same issue is in the latest 10.8.4 on windows.

      The more interesting is that if you will repeat the same question, again and again, it will show sometimes some data in the, maybe not is this small test data set. But I have a thousand lines there.

      I did not find what is the reason, it is a combination of several factors.

      When I replace "OVER partition by" by something like "rand()" it starts working.
      But if I use the exact same "OVER partition by" but with a different table in the previous step it is also working.

      .

      Attachments

        Activity

          tprochazka Tomáš Procházka created issue -
          alice Alice Sherepa made changes -
          Field Original Value New Value
          Status Open [ 1 ] Needs Feedback [ 10501 ]
          alice Alice Sherepa made changes -
          Component/s Optimizer - CTE [ 13513 ]
          serg Sergei Golubchik made changes -
          Status Needs Feedback [ 10501 ] Open [ 1 ]
          alice Alice Sherepa made changes -
          Affects Version/s 10.3 [ 22126 ]
          Affects Version/s 10.4 [ 22408 ]
          Affects Version/s 10.5 [ 23123 ]
          Affects Version/s 10.6 [ 24028 ]
          Affects Version/s 10.7 [ 24805 ]
          Affects Version/s 10.8 [ 26121 ]
          Affects Version/s 10.9 [ 26905 ]
          Affects Version/s 10.10 [ 27530 ]
          alice Alice Sherepa made changes -
          Fix Version/s 10.3 [ 22126 ]
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.5 [ 23123 ]
          Fix Version/s 10.6 [ 24028 ]
          Fix Version/s 10.7 [ 24805 ]
          Fix Version/s 10.8 [ 26121 ]
          Fix Version/s 10.9 [ 26905 ]
          Fix Version/s 10.10 [ 27530 ]
          alice Alice Sherepa made changes -
          Status Open [ 1 ] Confirmed [ 10101 ]
          alice Alice Sherepa made changes -
          Assignee Sergei Petrunia [ psergey ]
          tprochazka Tomáš Procházka made changes -
          Description I found interesting bug introduced between 10.3.9 which was working fine 10.3.34 which is already broken. I tested it up to 10.8.4 and it is still broken.

          I have a table of water consumption measurements (attached) in regular times for hot and cold water. And I need to convert it to a daily water consumption table. I'm using this.


          {code:sql}
          WITH
          twater_daily_raw AS (
          SELECT DATE_FORMAT(`timestamp`, '%Y-%m-%d') 'date',
          EXTRACT(YEAR FROM `timestamp`) 'year',
          EXTRACT(MONTH FROM `timestamp`) 'month',
          EXTRACT(DAY FROM `timestamp`) 'day',
          `flat`, `type`,
          (SELECT `value` FROM water sw WHERE sw.`timestamp` = MAX(w.`timestamp`) AND sw.`flat` = w.`flat` AND sw.`type` = w.`type`) 'total',
          MAX(timestamp) 'last'
          FROM `test2` w
          GROUP BY `date`, `flat`, `type`
          ORDER by `date`, `flat`, `type`
          ),

          twater_daily_usage AS (
          SELECT *,
          (`total` - IFNULL(LAG(`total`, 1) OVER (partition by `type`, `flat` ORDER by `type`, `flat`, `date`),0)) 'usage'
          FROM `twater_daily_raw`
                      ORDER by `date`, `flat`, `type`
          ),

          water_daily_combined AS (
          SELECT wc.`date`, wh.`date`'date2', wc.year, wc.month, wc.day, wc.flat,
          wc.`usage` 'cold', wh.`usage` 'hot',
          wh.total 'hot_total', wc.total 'cold_total'
          FROM twater_daily_usage wc, twater_daily_usage wh
          WHERE wc.`date` = wh.`date` AND wc.`flat` = wh.`flat` AND wh.`type` = 'WATER_HOT' AND wc.`type` = 'WATER_COLD'

          )

          SELECT * FROM water_daily_combined wdc
          {code}

          In the first step, I get the biggest value from every day, in the second step I evaluate usage by using {{over partition by}} and in the last step, I simply join twice the result table from the previous step to get cold and hot data to one line.

          You can see result in result.png image, the data from second table are always missing.
          It was working just fine for a long time, since I upgraded Mariad DB from 10.3.9 to 10.3.34 , because I upgraded Raspbian OS on my mini server. But the same issue is in the latest 10.8.4 on windows.

          The more interesting is that if you will repeat the same question, again and again, it will show sometimes some data in the, maybe not is this small test data set. But I have a thousand lines there.

          I did not find what is the reason, it is a combination of several factors.

          When I replace "OVER partition by" by something like "rand()" it starts working.
          But if I use the exact same "OVER partition by" but with a different table in the previous step it is also working.


          .
          I found an interesting bug introduced between 10.3.9 which was working fine and 10.3.34 which is already broken. I tested it up to 10.8.4 and it is still broken.

          I have a table of water consumption measurements (attached) in regular times for hot and cold water. And I need to convert it to a daily water consumption table. I'm using this.


          {code:sql}
          WITH
          twater_daily_raw AS (
          SELECT DATE_FORMAT(`timestamp`, '%Y-%m-%d') 'date',
          EXTRACT(YEAR FROM `timestamp`) 'year',
          EXTRACT(MONTH FROM `timestamp`) 'month',
          EXTRACT(DAY FROM `timestamp`) 'day',
          `flat`, `type`,
          (SELECT `value` FROM water sw WHERE sw.`timestamp` = MAX(w.`timestamp`) AND sw.`flat` = w.`flat` AND sw.`type` = w.`type`) 'total',
          MAX(timestamp) 'last'
          FROM `test2` w
          GROUP BY `date`, `flat`, `type`
          ORDER by `date`, `flat`, `type`
          ),

          twater_daily_usage AS (
          SELECT *,
          (`total` - IFNULL(LAG(`total`, 1) OVER (partition by `type`, `flat` ORDER by `type`, `flat`, `date`),0)) 'usage'
          FROM `twater_daily_raw`
                      ORDER by `date`, `flat`, `type`
          ),

          water_daily_combined AS (
          SELECT wc.`date`, wh.`date`'date2', wc.year, wc.month, wc.day, wc.flat,
          wc.`usage` 'cold', wh.`usage` 'hot',
          wh.total 'hot_total', wc.total 'cold_total'
          FROM twater_daily_usage wc, twater_daily_usage wh
          WHERE wc.`date` = wh.`date` AND wc.`flat` = wh.`flat` AND wh.`type` = 'WATER_HOT' AND wc.`type` = 'WATER_COLD'

          )

          SELECT * FROM water_daily_combined wdc
          {code}

          In the first step, I get the biggest value from every day, in the second step I evaluate usage by using {{over partition by}} and in the last step, I simply join twice the result table from the previous step to get cold and hot data to one line.

          You can see result in result.png image, the data from second table are always missing.
          It was working just fine for a long time, since I upgraded Mariad DB from 10.3.9 to 10.3.34 , because I upgraded Raspbian OS on my mini server. But the same issue is in the latest 10.8.4 on windows.

          The more interesting is that if you will repeat the same question, again and again, it will show sometimes some data in the, maybe not is this small test data set. But I have a thousand lines there.

          I did not find what is the reason, it is a combination of several factors.

          When I replace "OVER partition by" by something like "rand()" it starts working.
          But if I use the exact same "OVER partition by" but with a different table in the previous step it is also working.


          .
          serg Sergei Golubchik made changes -
          Affects Version/s 10.8.2 [ 26816 ]
          Affects Version/s 10.7.3 [ 26814 ]
          Affects Version/s 10.6.7 [ 26812 ]
          Affects Version/s 10.5.15 [ 26810 ]
          Affects Version/s 10.4.24 [ 26808 ]
          Affects Version/s 10.3.34 [ 26806 ]
          Affects Version/s 10.2.43 [ 26804 ]
          Affects Version/s 10.3 [ 22126 ]
          Affects Version/s 10.4 [ 22408 ]
          Affects Version/s 10.5 [ 23123 ]
          Affects Version/s 10.6 [ 24028 ]
          Affects Version/s 10.7 [ 24805 ]
          Affects Version/s 10.8 [ 26121 ]
          Affects Version/s 10.9 [ 26905 ]
          Affects Version/s 10.8.4 [ 27503 ]
          Affects Version/s 10.10 [ 27530 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.7 [ 24805 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.3 [ 22126 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.8 [ 26121 ]
          julien.fritsch Julien Fritsch made changes -
          Fix Version/s 10.9 [ 26905 ]
          Fix Version/s 10.10 [ 27530 ]
          psergei Sergei Petrunia made changes -
          Assignee Sergei Petrunia [ psergey ] Galina Shalygina [ shagalla ]
          psergei Sergei Petrunia made changes -
          Assignee Galina Shalygina [ shagalla ] Yuchen Pei [ JIRAUSER52627 ]
          psergei Sergei Petrunia made changes -
          Assignee Yuchen Pei [ JIRAUSER52627 ] Sergei Petrunia [ psergey ]
          psergei Sergei Petrunia made changes -
          Assignee Sergei Petrunia [ psergey ] Igor Babaev [ igor ]
          igor Igor Babaev (Inactive) made changes -
          Summary Not working JOIN on virtually created tables Wrong result when joining two derived tables over the same view
          igor Igor Babaev (Inactive) made changes -
          Status Confirmed [ 10101 ] In Progress [ 3 ]
          igor Igor Babaev (Inactive) made changes -
          Assignee Igor Babaev [ igor ] Rex Johnston [ JIRAUSER52533 ]
          Status In Progress [ 3 ] In Review [ 10002 ]
          Johnston Rex Johnston made changes -
          Status In Review [ 10002 ] Stalled [ 10000 ]
          Johnston Rex Johnston made changes -
          Assignee Rex Johnston [ JIRAUSER52533 ] Igor Babaev [ igor ]
          igor Igor Babaev (Inactive) made changes -
          Status Stalled [ 10000 ] In Progress [ 3 ]
          igor Igor Babaev (Inactive) made changes -
          Component/s Optimizer [ 10200 ]
          Component/s Optimizer - CTE [ 13513 ]
          Fix Version/s 10.5.26 [ 29832 ]
          Fix Version/s 10.6.19 [ 29833 ]
          Fix Version/s 10.11.9 [ 29834 ]
          Fix Version/s 11.1.6 [ 29835 ]
          Fix Version/s 11.2.5 [ 29836 ]
          Fix Version/s 11.4.3 [ 29837 ]
          Fix Version/s 11.5.2 [ 29838 ]
          Fix Version/s 11.6.0 [ 29839 ]
          Fix Version/s 10.4 [ 22408 ]
          Fix Version/s 10.5 [ 23123 ]
          Fix Version/s 10.6 [ 24028 ]
          Resolution Fixed [ 1 ]
          Status In Progress [ 3 ] Closed [ 6 ]

          People

            igor Igor Babaev (Inactive)
            tprochazka Tomáš Procházka
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.