[MDEV-14535] Duplicate entry on ALTER TABLE ADD COLUMN Created: 2017-11-29  Updated: 2021-11-12  Resolved: 2017-12-12

Status: Closed
Project: MariaDB Server
Component/s: Data Definition - Alter Table, Storage Engine - InnoDB
Affects Version/s: 10.1.22
Fix Version/s: 10.3.2

Type: Bug Priority: Major
Reporter: Thierry Laurier Assignee: Marko Mäkelä
Resolution: Fixed Votes: 0
Labels: innodb, upstream
Environment:

debian 8
mariadb-server-10.1 10.1.22+maria-1~jessie ( version mariadb )


Issue Links:
Duplicate
duplicates MDEV-11369 Instant add column for InnoDB Closed
Relates
relates to MDEV-15250 UPSERT during ALTER-TABLE results in ... Closed

 Description   

actual_recording | CREATE TABLE `actual_recording` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `asset_manifest_id` int(10) unsigned DEFAULT NULL COMMENT 'Default to null now, but plan is to be not null when new version of transcoding platform has taken over',
  `asset_video_id` int(10) unsigned DEFAULT NULL COMMENT 'Default to null now, but plan is to be not null when new version of transcoding platform has taken over',
  `user_id` int(10) unsigned NOT NULL,
  `profile_id` int(10) unsigned NOT NULL,
  `channel_id` int(10) unsigned NOT NULL,
  `bookmark_id` int(10) unsigned DEFAULT NULL,
  `broadcast_id` int(10) unsigned NOT NULL,
  `episode_id` int(10) unsigned NOT NULL,
  `program_id` int(10) unsigned NOT NULL,
  `vbegin` int(10) unsigned NOT NULL,
  `vend` int(10) unsigned NOT NULL,
  `is_partial` tinyint(1) NOT NULL DEFAULT '0',
  `is_watched` tinyint(1) NOT NULL DEFAULT '0',
  `scheduled_at` datetime NOT NULL,
  `created_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
  `updated_at` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `deleted_at` datetime DEFAULT NULL,
  `deleted_reason` varchar(2000) DEFAULT NULL,
  `asset_manifest_file_object_id` varchar(40) DEFAULT '',
  `asset_file_file_object_id` varchar(40) DEFAULT '',
  PRIMARY KEY (`id`),
  UNIQUE KEY `are_profile_channel_episode` (`profile_id`,`channel_id`,`episode_id`),
  UNIQUE KEY `ar_broadcast_profile_id` (`broadcast_id`,`user_id`,`profile_id`,`scheduled_at`),
  KEY `ar_asset_manifest_file_object_id` (`asset_manifest_file_object_id`),
  KEY `ar_asset_file_file_object_id` (`asset_file_file_object_id`),
  KEY `are_user_id_fk` (`user_id`),
  KEY `are_episode_id_fk` (`episode_id`),
  KEY `are_bookmark_id_fk` (`bookmark_id`),
  KEY `asset_manifest_ibfk` (`asset_manifest_id`),
  KEY `asset_video_ibfk` (`asset_video_id`),
  KEY `ar_profile_id` (`profile_id`),
  KEY `are_channel_id_fk` (`channel_id`),
  KEY `are_program_id_fk` (`program_id`)
) ENGINE=InnoDB AUTO_INCREMENT=10858855 DEFAULT CHARSET=utf8 COMMENT='profile asset recordings' |
 
 
MariaDB [backend]> ALTER TABLE actual_recording ADD COLUMN IF NOT EXISTS broadcast_start_at datetime NULL;
ERROR 1062 (23000): Duplicate entry '3207644-49-1335273' for key 'are_profile_channel_episode'

Resolv a problem in add ALGORITHM=COPY

MariaDB [backend]> ALTER TABLE actual_recording ADD COLUMN IF NOT EXISTS broadcast_end_at datetime NULL, ALGORITHM=COPY;
Query OK, 8858320 rows affected (7 min 1.54 sec)
Records: 8858320  Duplicates: 0  Warnings: 0



 Comments   
Comment by Elena Stepanova [ 2017-11-29 ]

So is there the duplicate entry in the table, or not?

Comment by Thierry Laurier [ 2017-12-01 ]

Not. there 's not duplicate entry in the table.

Comment by Elena Stepanova [ 2017-12-01 ]

Was the table migrated from a previous version of MariaDB or from MySQL?
Could you please paste the output of

show table status like 'actual_recording' \G

and attach your cnf file(s)?

Comment by Thierry Laurier [ 2017-12-05 ]

Table are created on old version off mariadb. All innodb table in use Antelope file format on INNODB_SYS_TABLES.

 
 
Database changed
MariaDB [backend]> show table status like 'actual_recording' \G
*************************** 1. row ***************************
           Name: actual_recording
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 8756359
 Avg_row_length: 241
    Data_length: 2111823872
Max_data_length: 0
   Index_length: 2546843648
      Data_free: 7340032
 Auto_increment: 11266902
    Create_time: 2017-11-29 14:26:57
    Update_time: NULL
     Check_time: NULL
      Collation: utf8_general_ci
       Checksum: NULL
 Create_options:
        Comment: profile asset recordings
1 row in set (0.00 sec)

[mysqld]
#########
#read_only = 1
slave-parallel-threads=8
slave_compressed_protocol=on
# log all requests receive by slave
#log_slave_updates
#########
#
# * Basic Settings
#
user		= mysql
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
port		= 3306
basedir		= /usr
datadir		= /var/lib/mysql
tmpdir		= /tmp
secure_file_priv= /srv/files
lc-messages-dir	= /usr/share/mysql
skip-external-locking
 
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
bind-address		= 0.0.0.0
 
#
# * Fine Tuning
#
thread_stack		= 192K
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam_recover_options = BACKUP
max_connections        = 5000
back_log		= 1500
#table_cache            = 64
#thread_concurrency     = 10
 
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1
 
#
# Error log - should be very few entries.
# Use syslog
#log_error = /var/log/mysql/error.log
 
#
# Here you can see queries with especially long duration
slow_query_log_file = /var/log/mysql/mysql-slow.log
slow_query_log      = 1
long_query_time = 0.2
log_slow_verbosity = query_plan,explain
 
log_queries_not_using_indexes = 1
innodb_status_output = 1
innodb_status_output_locks = 1
 
### STATISTICS INFORMATIONS
userstat = 1
#INSTALL SONAME 'query_response_time';
query_response_time_stats = 1
 
#
# The following can be used as easy to replay backup logs or for replication.
# note: if you are setting up a replication slave, see README.Debian about
#       other settings you may need to change.
server-id		= 104
gtid-domain-id		= 10
log_bin			= /var/log/mysql/mysql-bin.log
 
expire_logs_days	= 1
max_binlog_size         = 100M
binlog-format		= row
#binlog_do_db		= include_database_name
#binlog_ignore_db	= include_database_name
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Features
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# For generating SSL certificates I recommend the OpenSSL GUI "tinyca".
#
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
 
connect_timeout         = 30
wait_timeout            = 600
max_allowed_packet      = 128M
thread_cache_size       = 80
sort_buffer_size        = 2M
bulk_insert_buffer_size = 16M
tmp_table_size          = 32M
max_heap_table_size     = 32M
 
 
key_buffer_size         = 16M
table_open_cache        = 1024
myisam_sort_buffer_size = 512M
concurrent_insert       = 2
read_buffer_size        = 2M
read_rnd_buffer_size    = 1M
 
query_cache_limit	= 128K
query_cache_size	= 16M
 
group_concat_max_len	= 1000000
 
innodb_file_format	= Barracuda
innodb_log_file_size    = 128M
innodb_log_files_in_group = 2
innodb_flush_log_at_trx_commit=2
innodb_thread_concurrency=64
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_buffer_pool_size = 50G
innodb_log_buffer_size  = 8M
innodb_file_per_table   = 1
innodb_open_files       = 4096
innodb_io_capacity      = 2000
innodb_flush_method     = O_DIRECT
innodb_buffer_pool_dump_at_shutdown = ON
innodb_buffer_pool_load_at_startup = ON
innodb_checksum_algorithm = crc32
innodb_log_compressed_pages = 0
#experimental, disabled so far
innodb_log_checksum_algorithm = crc32
innodb_large_prefix = 1
#SSD
innodb_flush_neighbors = 0
innodb_log_block_size = 4096
innodb_page_size = 8192
#Mutex contention
innodb_adaptive_hash_index_partitions = 32
innodb_buffer_pool_instances = 32
innodb_lock_wait_timeout=120
#innodb_buffer_pool_populate = 1 #### DEPRECATED

Comment by Alice Sherepa [ 2017-12-12 ]

This looks like known limitation of online ALTER operation, https://dev.mysql.com/doc/refman/5.6/en/innodb-create-index-limitations.html:

When running an online DDL operation, the thread that runs the ALTER TABLE statement applies an “online log” of DML operations that were run concurrently on the same table from other connection threads. When the DML operations are applied, it is possible to encounter a duplicate key entry error (ERROR 1062 (23000): Duplicate entry), even if the duplicate entry is only temporary and would be reverted by a later entry in the “online log”. This is similar to the idea of a foreign key constraint check in InnoDB in which constraints must hold during a transaction.

Comment by Marko Mäkelä [ 2017-12-12 ]

This is a known limitation of the online ALTER TABLE that I designed and implemented in MySQL 5.6 for InnoDB tables.

But I have good news for you: MariaDB 10.3 implements ADD COLUMN as an instantaneous operation, so that there is no logging involved.

Comment by Marko Mäkelä [ 2021-11-12 ]

You may set innodb_instant_alter_column_allowed=never to force ADD COLUMN to rebuild the table.

For any online table-rebuilding ALTER TABLE (or ADD UNIQUE INDEX) that is executed inside InnoDB, you can still hit this ‘fake duplicate’ problem, which will be tracked as MDEV-15250.

Generated at Thu Feb 08 08:14:20 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.