[MDEV-27128] Implement JSON Schema Validation FUNCTION Created: 2021-11-26  Updated: 2023-11-27  Resolved: 2023-04-26

Status: Closed
Project: MariaDB Server
Component/s: JSON
Fix Version/s: 11.1.1

Type: Task Priority: Blocker
Reporter: Rúben Ferreira Assignee: Rucha Deodhar
Resolution: Fixed Votes: 4
Labels: CONSTRAINT, JSON, Preview_11.1, compat80

Issue Links:
Duplicate
duplicates MDEV-25934 JSON-Schema as Contraint on JSON-column Closed
Problem/Incident
causes MDEV-30287 JSON_SCHEMA_VALID returns incorrect r... Closed
causes MDEV-30677 Incorrect result for "SELECT JSON_SCH... Closed
causes MDEV-30689 JSON_SCHEMA_VALID for type=array retu... Closed
causes MDEV-30690 Server crashed on function JSON_SCHEM... Closed
causes MDEV-30703 JSON_SCHEMA_VALID : Enum array must... Closed
causes MDEV-30704 JSON_SCHEMA_VALID: multipleOf must be... Closed
causes MDEV-30705 JSON_SCHEMA_VALID: schema with multip... Closed
causes MDEV-30976 Assertion `(length % 4) == 0' failed ... Open
causes MDEV-30977 Additional key values are not validat... Closed
causes MDEV-30995 JSON_SCHEMA_VALID is not validating c... Closed
causes MDEV-31009 JSON_SCHEMA_VALID does not properly v... Closed
causes MDEV-31032 UBSAN|downcast of address X which doe... Closed
causes MDEV-31599 Assertion `0' failed in Item_param::c... Closed
Relates
relates to MDEV-17397 Implement JSON functions available in... Open
relates to MDEV-30219 Add more validation keywords for json... Open
Sub-Tasks:
Key
Summary
Type
Status
Assignee
MDEV-29899 Fix design for JSON_SCHEMA_VALID() Technical task Closed Rucha Deodhar  
MDEV-29900 Add Validation for Pattern and depend... Technical task Closed  
Epic Link: Implement JSON functions available in MySQL 8.0 but not MariaDB

 Description   

A great addition to MariaDB would be able to validate JSON schemas. MySQL already have this implemented as follows:

JSON_SCHEMA_VALID(schema,document)

Validates a JSON document against a JSON schema. Both schema and document are required. The schema must be a valid JSON object; the document must be a valid JSON document. Provided that these conditions are met: If the document validates against the schema, the function returns true (1); otherwise, it returns false (0).

https://dev.mysql.com/doc/refman/8.0/en/json-validation-functions.html#function_json-schema-valid

Since MariaDB is great and it allows the use of UDFs and deterministic functions on constraints something like this would be awesome:

CREATE TABLE t1 (a JSON CHECK JSON_SCHEMA_VALID(schema, a));

or

CREATE TABLE t1 (
a JSON,
CONSTRAINT json_validation CHECK JSON_SCHEMA_VALID(schema, a)
);

The word schema should be replaced by a string representing a valid JSON schema.



 Comments   
Comment by Federico Razzoli [ 2022-08-16 ]

Which JSON Schema version is being implemented?

Note that MySQL uses Draft 4 - actually a subset of it, as the documentation omits to mention. Draft 4 problems:

  • It's old (2013)
  • It has annoying problems that have been fixed in later versions, for example one is used id instead of $id, which led to confusion when id is used as an attribute name.
  • Not fully compatible with later versions.
Comment by Rucha Deodhar [ 2022-09-13 ]

Final Patch: https://github.com/MariaDB/server/tree/bb-11.1-MDEV-27128

Comment by Rucha Deodhar [ 2022-09-13 ]

f_razzoli We support the 2020 draft with some exception.

Comment by Alexey Botchkov [ 2022-09-18 ]

didn't agree with parts of the design. Comments sent by email.

Comment by Alexey Botchkov [ 2022-12-22 ]

ok with a couple of corrections discussed on Slack.

Comment by Lena Startseva [ 2023-02-27 ]

Some cases I'm not sure how it should be:

1. I suppose that for all cases where JSON_SCHEMA_VALID returns warning due to incorrect schema the result must be NULL. An example can be taken from MDEV-30689 or use this:

SET @schema = '{
                 "type":"array",
                  "enum": [null]}
                }';
SELECT JSON_SCHEMA_VALID(@schema, 'null');
JSON_SCHEMA_VALID(@schema, 'null')
0
Warnings:
Warning	4038	Syntax error in JSON text in argument 2 to function 'json_schema_valid' at position 86

2. With uniqueItems=true for array I guess 'null' and 'null' should be duplicate items, but that's not the case right now:

SET @schema= '{
                     "type":"array",
                     "uniqueItems":true
              }';
SELECT JSON_SCHEMA_VALID(@schema, '[null, null]');
JSON_SCHEMA_VALID(@schema, '[null, null]')
1

3. All parameters marked as "be a non-negative integer" in the schema must be checked for this condidtion. So, I suppose in cases like the following one should return null and warning or error:

SET @schema_array= '{
                      "type": "array",
                      "contains": {
                                    "type": "string"
                                  },
                      "minContains": -3,
                      "maxContains": 3
}';
SELECT JSON_SCHEMA_VALID(@schema_array, '["string1", "string2", "string3",  "string3", 1, 2, 3]');
JSON_SCHEMA_VALID(@schema_array, '["string1", "string2", "string3",  "string3", 1, 2, 3]')
0

4. Also for parameter "required":

The value of this keyword MUST be an array. Elements of this array, if any, MUST be strings, and MUST be unique.

SET @schema_required='{"type":"object","required":[1,"str1", "str1"]}';
SELECT JSON_SCHEMA_VALID(@schema_required,'{"num1":1, "str1":"abc", "arr1":[1,2,3]}');
JSON_SCHEMA_VALID(@schema_required,'{"num1":1, "str1":"abc", "arr1":[1,2,3]}')
0

After fixing bugs and rebasing on the actual tree, a little rechecking will be required (for checked options).

Comment by Alexey Botchkov [ 2023-03-06 ]

See comments to the commit
https://github.com/MariaDB/server/commit/abcd161da906bfed1a3e61ce79eae37c16332a58

Comment by Rucha Deodhar [ 2023-03-06 ]

lstartseva Thanks for the comment. You're right. I've fixed it as part of below patch

1) https://github.com/MariaDB/server/commit/357c208d67fa90b79da013c6cb7fae044af8c1b6
2), 3) and 4): https://github.com/MariaDB/server/commit/0058ac37d31767686cea2816f17b30fe59b6f60d

Comment by Alexey Botchkov [ 2023-03-15 ]

Comments on slack.
After these addressed i'm ok with making the preview out of this.

Comment by Rucha Deodhar [ 2023-03-16 ]

( Reassigned after working on review 2 for final check )

Comment by Alexey Botchkov [ 2023-03-16 ]

commit 70a7de2aed7677ef9e0a14f5734150cbc58023ae
comments on slack.
With at least keyword_map= &empty_map fix i'm ok with the patch to preview.

Comment by Ramesh Sivaraman [ 2023-04-25 ]

ok to push

Generated at Thu Feb 08 09:50:33 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.