[MDEV-24824] Can't add non-null column to versioned table Created: 2021-02-09  Updated: 2021-02-12  Resolved: 2021-02-10

Status: Closed
Project: MariaDB Server
Component/s: Versioned Tables
Affects Version/s: 10.5.8
Fix Version/s: N/A

Type: Bug Priority: Major
Reporter: Remy Fox Assignee: Sergei Golubchik
Resolution: Not a Bug Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-19655 Data truncated for column 'date_1' fo... Closed

 Description   

Evolving applications sometimes add new columns to tables. Sensible values may be added for existing rows in those tables. Versioned tables don't sufficiently support this practice.

An error 1265 shows when adding a new, non null column to a versioned table.

Example 1:

SET SESSION system_versioning_alter_history=KEEP;
CREATE TABLE abc (a INT NOT NULL) WITH SYSTEM VERSIONING;
INSERT INTO abc(a) VALUES (1), (2), (3);
ALTER TABLE abc ADD COLUMN b TEXT NULL;
UPDATE abc SET b = 'value';
ALTER TABLE abc CHANGE COLUMN b b TEXT NOT NULL;

Example 2:

SET SESSION system_versioning_alter_history=KEEP;
CREATE TABLE abc (a INT NOT NULL) WITH SYSTEM VERSIONING;
INSERT INTO abc(a) VALUES (1), (2), (3);
DELETE FROM abc;
ALTER TABLE abc ADD COLUMN b TEXT NULL;
ALTER TABLE abc CHANGE COLUMN b b TEXT NOT NULL;

Interestingly, this does not fail. Example 3:

SET SESSION system_versioning_alter_history=KEEP;
CREATE TABLE abc (a INT NOT NULL) WITH SYSTEM VERSIONING;
INSERT INTO abc(a) VALUES (1), (2), (3);
DELETE FROM abc;
ALTER TABLE abc ADD COLUMN b TEXT NOT NULL;

Example 2 and 3 do the same thing, so this is clearly a bug.

In any way, an additional ideal solution would be to allow queries that can alter historical rows, so that the historical nulls can get a sensible value too.



 Comments   
Comment by Ian Gilfillan [ 2021-02-10 ]

In SQL Server, this is handled by setting a DEFAULT value when adding (or altering) a new NOT NULL column, which is then applied to the history. See https://docs.microsoft.com/en-us/sql/relational-databases/tables/changing-the-schema-of-a-system-versioned-temporal-table?view=sql-server-ver15

But this doesn't appear to work in MariaDB. Unsure what the standard says, The standard does not permit ALTER statements to versioned tables, but it seems like it would be a useful enhancement. And the current, inconsistent behaviour is clearly a bug.

In the meantime I will document this limitation explicitly.

Comment by Sergei Golubchik [ 2021-02-10 ]

Doesn't look like a bug to me. In the examples 1 and 2 you have a column with (historical) NULL values, and your ALTER converts it to NOT NULL. This causes "1265: Data truncated for column 'b'".

Compare with

CREATE TABLE abc (a INT NOT NULL);
INSERT INTO abc(a) VALUES (1), (2), (3);
ALTER TABLE abc ADD COLUMN b TEXT;
ALTER TABLE abc CHANGE COLUMN b b TEXT NOT NULL;

No system versioning. Same error.

In your third example you add a NOT NULL column, so you don't convert NULL values to not null.

Comment by Remy Fox [ 2021-02-10 ]

@Sergei I see. In your parallel example on normal tables at least I have to option to update rows in between the two ALTER TABLE statements so as to supply sensible values. Has MariaDB planned to enable updating historical rows so as to supply sensible values?

Comment by Sergei Golubchik [ 2021-02-10 ]

No, so far we only planned to allow INSERT's for mysqldump to work.
The easiest workaround in your case would be to set sql_mode='' for the duration of the last ALTER.
Like

SET STATEMENT sql_mode='' FOR ALTER TABLE abc CHANGE COLUMN b b TEXT NOT NULL;

this will make NULL-to-empty-string conversion a warning, but not an error.

Comment by Remy Fox [ 2021-02-10 ]

What should I do if I wanted a more sophisticated historical value in a new column? E.g. a historically valid foreign key or some other sensible expression?

It is natural for data models to evolve over time and so is the practice of updating rows for a column that did not exist previously.

Example 1:
Imagine you're a business located in country A and have a table customer_addresses.
You don't store a country in the table, because all your customers are located in country A. Then one day, you decide to expand the business to a neighbouring country B. Preparing to expand to country B, you add a new column customer_address.country. You update the customer_addresses table so that all countries are filled with A. Now you would also like to update historical rows, but you can't.

Example 2:
Imagine the same table of customer_addresses. It has a country column. This is a textual column. Then one day, you decide to normalize your model. You move the textual values into a separate table called 'countries'. The column that was previously customer_addresses.country is replaced by a column customer_addresses.country_id with a foreign key. You update the customer_addresses table so as to appropriately replace the previous textual column values with references. Now you would also like to update historical rows, but you can't.

You cannot successfully make your historical selections on the historical tables any longer, after these changes.

Comment by Sergei Golubchik [ 2021-02-12 ]

there's a workaround with two ALTER's. Something like (using your first example}

SET SESSION system_versioning_alter_history=KEEP;
CREATE TABLE t1 (a INT NOT NULL) WITH SYSTEM VERSIONING;
INSERT INTO t1(a) VALUES (1), (2), (3);
ALTER TABLE t1 ADD COLUMN b TEXT NULL;
UPDATE t1 SET b = 'value';
ALTER TABLE t1 ADD COLUMN b_new TEXT NOT NULL DEFAULT (COALESCE(b, "foo"));
ALTER TABLE t1 DROP COLUMN b, CHANGE COLUMN b_new b TEXT NOT NULL DEFAULT '';

Generated at Thu Feb 08 09:32:57 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.