[MDEV-25202] JSON_TABLE: Early table reference leads to unexpected result set, server crash in st_join_table::fix_splitting or failing Assertion `join->best_read < double(1.797...) Created: 2021-03-19  Updated: 2021-04-21  Resolved: 2021-04-16

Status: Closed
Project: MariaDB Server
Component/s: JSON, Optimizer
Affects Version/s: N/A
Fix Version/s: 10.6.0

Type: Bug Priority: Critical
Reporter: Elena Stepanova Assignee: Sergei Petrunia
Resolution: Fixed Votes: 0
Labels: None

Issue Links:
PartOf
includes MDEV-25256 JSON_TABLE: Error ER_VIEW_INVALID upo... Closed
Relates
relates to MDEV-17399 Add support for JSON_TABLE Closed
relates to MDEV-25352 JSON_TABLE: Inconsistent name resolut... Closed
relates to MDEV-25254 JSON_TABLE: Inconsistent name resolut... Closed
relates to MDEV-25256 JSON_TABLE: Error ER_VIEW_INVALID upo... Closed

 Description   

In this test case JSON_TABLE references a column from t2 which is only joined later. To my understanding, it is expected to fail with ER_BAD_FIELD_ERROR (and it does in MySQL), but instead it succeeds and returns a result set. I don't know if it's a bug, and if it is, how much of a problem it is.

CREATE TABLE t1 (o INT);
INSERT INTO t1 VALUES (1),(2);
 
CREATE TABLE t2 (a INT);
INSERT INTO t2 VALUES (3),(4);
 
SELECT * FROM JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) AS jt1 NATURAL JOIN t1 JOIN t2;
 
# Cleanup
DROP TABLE t1, t2;

bb-10.6-mdev17399-hf 3530463bc

SELECT * FROM JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) AS jt1 NATURAL JOIN t1 JOIN t2;
o	a
1	3
1	4

ANALYZE

{
  "query_block": {
    "select_id": 1,
    "r_loops": 1,
    "r_total_time_ms": 0.128189085,
    "table": {
      "table_name": "t1",
      "access_type": "ALL",
      "r_loops": 1,
      "rows": 2,
      "r_rows": 2,
      "r_table_time_ms": 0.008660748,
      "r_other_time_ms": 0.025902145,
      "filtered": 100,
      "r_filtered": 100
    },
    "block-nl-join": {
      "table": {
        "table_name": "t2",
        "access_type": "ALL",
        "r_loops": 1,
        "rows": 2,
        "r_rows": 2,
        "r_table_time_ms": 0.004725865,
        "r_other_time_ms": 0.044695468,
        "filtered": 100,
        "r_filtered": 100
      },
      "buffer_type": "flat",
      "buffer_size": "65",
      "join_type": "BNL",
      "r_filtered": 100
    },
    "table": {
      "table_name": "jt1",
      "access_type": "ALL",
      "r_loops": 4,
      "rows": 40,
      "r_rows": 1,
      "r_table_time_ms": 0.005747132,
      "r_other_time_ms": 0.013326538,
      "filtered": 100,
      "r_filtered": 50,
      "table_function": "json_table",
      "attached_condition": "jt1.o = t1.o"
    }
  }
}

But the following case, which only differs from the previous one by using STRAIGHT_JOIN instead of JOIN, makes a non-debug server crash, and a debug server fail on an assertion. So, this part is undoubtedly a bug; and I presume it has the same cause as the above, so I'm filing them together.

CREATE TABLE t1 (o INT);
INSERT INTO t1 VALUES (1),(2);
 
CREATE TABLE t2 (a INT);
INSERT INTO t2 VALUES (3),(4);
 
SELECT * FROM JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) AS jt1 NATURAL JOIN t1 STRAIGHT_JOIN t2;
 
# Cleanup
DROP TABLE t1, t2;

debug build

mariadbd: /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:8845: bool greedy_search(JOIN*, table_map, uint, uint, uint): Assertion `join->best_read < double(1.79769313486231570814527423731704357e+308L)' failed.
210319 14:29:31 [ERROR] mysqld got signal 6 ;
 
#7  0x00007f6a5c9bdf36 in __GI___assert_fail (assertion=0x55b7aa9c1b60 "join->best_read < double(1.79769313486231570814527423731704357e+308L)", file=0x55b7aa9bc700 "/data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc", line=8845, function=0x55b7aa9c1b00 "bool greedy_search(JOIN*, table_map, uint, uint, uint)") at assert.c:101
#8  0x000055b7a89ff44c in greedy_search (join=0x62b00003c9c8, remaining_tables=7, search_depth=62, prune_level=1, use_cond_selectivity=4) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:8845
#9  0x000055b7a89fd035 in choose_plan (join=0x62b00003c9c8, join_tables=7) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:8402
#10 0x000055b7a89e838d in make_join_statistics (join=0x62b00003c9c8, tables_list=..., keyuse_array=0x62b00003ccb8) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:5639
#11 0x000055b7a89c53ad in JOIN::optimize_inner (this=0x62b00003c9c8) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:2270
#12 0x000055b7a89be846 in JOIN::optimize (this=0x62b00003c9c8) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:1639
#13 0x000055b7a89df2a0 in mysql_select (thd=0x62b000069288, tables=0x62b000039b60, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x62b00003c998, unit=0x62b00006d460, select_lex=0x62b0000383e0) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:4733
#14 0x000055b7a89b0822 in handle_select (thd=0x62b000069288, lex=0x62b00006d398, result=0x62b00003c998, setup_tables_done_option=0) at /data/src/bb-10.6-mdev17399-hf/sql/sql_select.cc:417
#15 0x000055b7a891a9c2 in execute_sqlcom_select (thd=0x62b000069288, all_tables=0x62b000039b60) at /data/src/bb-10.6-mdev17399-hf/sql/sql_parse.cc:6230
#16 0x000055b7a8909ca4 in mysql_execute_command (thd=0x62b000069288) at /data/src/bb-10.6-mdev17399-hf/sql/sql_parse.cc:3926
#17 0x000055b7a8925c6d in mysql_parse (thd=0x62b000069288, rawbuf=0x62b0000382a8 "SELECT * FROM JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) AS jt1 NATURAL JOIN t1 STRAIGHT_JOIN t2", length=98, parser_state=0x7f6a537ff7b0) at /data/src/bb-10.6-mdev17399-hf/sql/sql_parse.cc:8000
#18 0x000055b7a88fc6a8 in dispatch_command (command=COM_QUERY, thd=0x62b000069288, packet=0x62900024e289 "SELECT * FROM JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) AS jt1 NATURAL JOIN t1 STRAIGHT_JOIN t2", packet_length=98, blocking=true) at /data/src/bb-10.6-mdev17399-hf/sql/sql_parse.cc:1886
#19 0x000055b7a88f93e3 in do_command (thd=0x62b000069288, blocking=true) at /data/src/bb-10.6-mdev17399-hf/sql/sql_parse.cc:1397
#20 0x000055b7a8d3ae85 in do_handle_one_connection (connect=0x61100000ad48, put_in_cache=true) at /data/src/bb-10.6-mdev17399-hf/sql/sql_connect.cc:1410
#21 0x000055b7a8d3a7e2 in handle_one_connection (arg=0x61100000ac08) at /data/src/bb-10.6-mdev17399-hf/sql/sql_connect.cc:1312
#22 0x000055b7a9a4508b in pfs_spawn_thread (arg=0x616000102a08) at /data/src/bb-10.6-mdev17399-hf/storage/perfschema/pfs.cc:2201
#23 0x00007f6a5ced5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#24 0x00007f6a5caa9293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

release build

#3  <signal handler called>
#4  JOIN::fix_all_splittings_in_plan (this=this@entry=0x7fe2f8014b60) at /data/src/bb-10.6-mdev17399-hf-rel/sql/opt_split.cc:1155
#5  0x000055bc3a0113a0 in JOIN::optimize_inner (this=0x7fe2f8014b60) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_select.cc:2287
#6  0x000055bc3a011a53 in JOIN::optimize (this=this@entry=0x7fe2f8014b60) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_select.cc:1639
#7  0x000055bc3a011b17 in mysql_select (thd=0x7fe2f8000c58, tables=0x7fe2f8011de8, fields=..., conds=0x0, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=<optimized out>, result=0x7fe2f8014b38, unit=0x7fe2f8004c70, select_lex=0x7fe2f80106e8) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_select.cc:4733
#8  0x000055bc3a0125e7 in handle_select (thd=thd@entry=0x7fe2f8000c58, lex=lex@entry=0x7fe2f8004ba8, result=result@entry=0x7fe2f8014b38, setup_tables_done_option=setup_tables_done_option@entry=0) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_select.cc:417
#9  0x000055bc39fa5761 in execute_sqlcom_select (thd=0x7fe2f8000c58, all_tables=0x7fe2f8011de8) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_parse.cc:6230
#10 0x000055bc39fb36ae in mysql_execute_command (thd=0x7fe2f8000c58) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_parse.cc:3926
#11 0x000055bc39fa077d in mysql_parse (thd=0x7fe2f8000c58, rawbuf=<optimized out>, length=<optimized out>, parser_state=<optimized out>) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_parse.cc:8000
#12 0x000055bc39fac73d in dispatch_command (command=COM_QUERY, thd=0x7fe2f8000c58, packet=<optimized out>, packet_length=<optimized out>, blocking=<optimized out>) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_class.h:1318
#13 0x000055bc39fae7e6 in do_command (thd=0x7fe2f8000c58, blocking=blocking@entry=true) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_parse.cc:1397
#14 0x000055bc3a0aaa97 in do_handle_one_connection (connect=<optimized out>, put_in_cache=true) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_connect.cc:1410
#15 0x000055bc3a0aadfd in handle_one_connection (arg=arg@entry=0x55bc3c29bdb8) at /data/src/bb-10.6-mdev17399-hf-rel/sql/sql_connect.cc:1312
#16 0x000055bc3a4029c6 in pfs_spawn_thread (arg=0x55bc3bf06e48) at /data/src/bb-10.6-mdev17399-hf-rel/storage/perfschema/pfs.cc:2201
#17 0x00007fe30e006609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007fe30dbda293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95



 Comments   
Comment by Sergei Petrunia [ 2021-03-21 ]

Indeed, it crashes because there is a loop in table dependencies:

  "table_dependencies": [
    {
      "table": "jt1",
      "row_may_be_null": false,
      "map_bit": 0,
      "depends_on_map_bits": ["2"]
    },
    {
      "table": "t1",
      "row_may_be_null": false,
      "map_bit": 1,
      "depends_on_map_bits": []
    },
    {
      "table": "t2",
      "row_may_be_null": false,
      "map_bit": 2,
      "depends_on_map_bits": ["0", "1"]
    }
  ]

which means it's not possible to build a join order

Comment by Sergei Petrunia [ 2021-03-21 ]

The a in

JSON_TABLE(a, '$' COLUMNS(o FOR ORDINALITY)) 

is a forward reference, it should not have been resolved.

Comment by Elena Stepanova [ 2021-03-21 ]

Yes, that's what the first test case and the complaint about an unexpected query execution success were about.

Comment by Sergei Petrunia [ 2021-03-22 ]

The reason the query is accepted is that end_lateral_table technique fails to work for it.. Name resolution context has this TABLE_LIST object as the first table

nested_join(jt1, jt1)

it calls find_field_in_natural_join() for it, which finds nothing (and fails to stop at end_lateral_table) and proceeds with the name resolution.

Comment by Sergei Petrunia [ 2021-04-01 ]

"Monty's suggestion" - table bits (for TABLE::map) are assigned sequentially. Can one prevent forward references by not allowing references to higher bits?

(Note: this will not be able to handle RIGHT JOINs - in "t1 RIGHT JOIN t2", t1->map=1, t2->map=2).

Comment by Sergei Petrunia [ 2021-04-02 ]

The {{first_name_resolution_table -> next_name_resolution_table* }} chain looks differently in MariaDB than in MySQL.

Example with RIGHT JOIN

create table t5(a int, js varchar(32));
create table t5a as select * from t5;
 
explain 
select * 
from
  t5a,
  t5 right join
  json_table(concat('',js), 
             '$' columns ( color varchar(32) path '$.color')
             ) as JT on 1;

The context->first_name_resolution_table -> next_name_resolution_table chain:

MariaDB: t5a, t5, JT
MySQL-8.0: t5a, JT, t5

Example with NATURAL JOIN

explain
select * 
from 
  t5 
  natural join 
  json_table(concat('',js), 
             '$' columns ( a int path '$.a')
             ) as JT;

MySQL-8:  t5, JT
MariaDB: (nest_last_join)

Comment by Sergei Petrunia [ 2021-04-05 ]

Recent bugfixes that have overlap with this bug:

commit aba7884138fa649f3e1377174afbb567cf3be7af
Author: Igor Babaev <igor@askmonty.org>
Date:   Sun Mar 21 12:08:54 2021 -0700
 
    MDEV-25206 Crash with CREATE VIEW .. SELECT with non-existing field
               in ON condition
    
    The fix of the bug MDEV-25002 for 10.4 turned out to be incomplete. 
...

commit 4020e4aee0b3477f31d1fb0d797b84171b22d483
Author: Igor Babaev <igor@askmonty.org>
Date:   Thu Mar 4 23:02:47 2021 -0800
 
    MDEV-25002 ON expressions cannot contain outer references
...

Comment by Sergei Petrunia [ 2021-04-05 ]

Patch: http://lists.askmonty.org/pipermail/commits/2021-April/014535.html

Comment by Sergei Petrunia [ 2021-04-05 ]

Also pushed this into https://github.com/MariaDB/server/tree/bb-10.6-mdev17399-hf

Comment by Sergei Petrunia [ 2021-04-14 ]

Now, the complete fix is these three patches:

https://github.com/MariaDB/server/commit/6fd4ac82ce251c3428f2041983c760967d3651e8
https://github.com/MariaDB/server/commit/ded1d8b44befb545ea581aee1f4f84676eb7f254
https://github.com/MariaDB/server/commit/7cd264346f188cad6e781d87f379b1145000f28e

(I intend to squash them together during the final rebase)

Comment by Sergei Petrunia [ 2021-04-16 ]

http://lists.askmonty.org/pipermail/commits/2021-April/014558.html

Comment by Oleksandr Byelkin [ 2021-04-16 ]

OK to push after adding test for second execution.

Generated at Thu Feb 08 09:35:55 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.