We are trying to dereference a NULL column, which is a virtual column in TBR-D4_min.yy:

882				ulint type = temp_index->fields[i].col->mtype;

(rr) p temp_index->fields[i]

$1 = {col = 0x0, name = {m_name = 0x61c000045a70 "col_text_g"}, 

  prefix_len = 9, fixed_len = 0}

similar, with DELETE/SELECT

row/row0mysql.cc:887(row_create_prebuilt(dict_table_t*, unsigned long))[0x55ea735dbc8e]

handler/ha_innodb.cc:5748(ha_innobase::open(char const*, int, unsigned int))[0x55ea7322b77e]

sql/handler.cc:2984(handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*))[0x55ea7270a7ec]

sql/table.cc:4218(open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*))[0x55ea722491dd]

sql/sql_base.cc:2001(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x55ea71d545f1]

sql/sql_base.cc:3801(open_and_process_table(THD*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*))[0x55ea71d5e38d]

sql/sql_base.cc:4275(open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x55ea71d60f14]

sql/sql_base.cc:5211(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x55ea71d66058]

sql/sql_base.h:507(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int))[0x55ea71cbcf40]

sql/sql_parse.cc:6229(execute_sqlcom_select(THD*, TABLE_LIST*))[0x55ea71f0879d]

sql/sql_parse.cc:4004(mysql_execute_command(THD*))[0x55ea71ef8510]

sql/sql_parse.cc:8089(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x55ea71f14518]

sql/sql_parse.cc:1892(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x55ea71eea7ab]

sql/sql_parse.cc:1370(do_command(THD*))[0x55ea71ee70d4]

sql/sql_connect.cc:1410(do_handle_one_connection(CONNECT*, bool))[0x55ea7232a608]

sql/sql_connect.cc:1314(handle_one_connection)[0x55ea72329f6c]

perfschema/pfs.cc:2203(pfs_spawn_thread)[0x55ea73036ef5]

nptl/pthread_create.c:478(start_thread)[0x7f3e508ae609]

x86_64/clone.S:97(__GI___clone)[0x7f3e50482293]

row/row0mysql.cc:887(row_create_prebuilt(dict_table_t*, unsigned long))[0x558afed1ac8e]

handler/ha_innodb.cc:5748(ha_innobase::open(char const*, int, unsigned int))[0x558afe96a77e]

sql/handler.cc:2984(handler::ha_open(TABLE*, char const*, int, unsigned int, st_mem_root*, List<String>*))[0x558afde497ec]

sql/table.cc:4218(open_table_from_share(THD*, TABLE_SHARE*, st_mysql_const_lex_string const*, unsigned int, unsigned int, unsigned int, TABLE*, bool, List<String>*))[0x558afd9881dd]

sql/sql_base.cc:2001(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x558afd4935f1]

sql/sql_base.cc:3801(open_and_process_table(THD*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*))[0x558afd49d38d]

sql/sql_base.cc:4275(open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x558afd49ff14]

sql/sql_base.cc:5211(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x558afd4a5058]

sql/sql_base.h:507(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int))[0x558afd3fbf40]

sql/sql_delete.cc:340(mysql_delete(THD*, TABLE_LIST*, Item*, SQL_I_List<st_order>*, unsigned long long, unsigned long long, select_result*))[0x558afe2f2084]

sql/sql_parse.cc:4865(mysql_execute_command(THD*))[0x558afd63d671]

sql/sql_parse.cc:8089(mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool))[0x558afd653518]

sql/sql_parse.cc:1892(dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool))[0x558afd6297ab]

sql/sql_parse.cc:1370(do_command(THD*))[0x558afd6260d4]

sql/sql_connect.cc:1410(do_handle_one_connection(CONNECT*, bool))[0x558afda69608]

sql/sql_connect.cc:1314(handle_one_connection)[0x558afda68f6c]

perfschema/pfs.cc:2203(pfs_spawn_thread)[0x558afe775ef5]

nptl/pthread_create.c:478(start_thread)[0x7fe901bf6609]

x86_64/clone.S:97(__GI___clone)[0x7fe9017ca293]

temp_index->fields[i].col was previously null'ed by dict_index_t::detach_columns called from a parallel ha_innobase::inplace_alter_table

(gdb) t

[Current thread is 16 (Thread 2910372.2911398)]

(gdb) bt

#0  0x0000563c4f040c1a in dict_index_t::detach_columns (this=0x6160002a6a20, clear=true) at /Server/10.6M/storage/innobase/include/dict0mem.h:1215

#1  0x0000563c4f0571bd in ha_innobase_inplace_ctx::clear_added_indexes (this=0x62b0000c60d8) at /Server/10.6M/storage/innobase/handler/handler0alter.cc:1029

#2  0x0000563c4f029f71 in ha_innobase::inplace_alter_table (this=0x61d0009a92b8, altered_table=0x71181748b210, ha_alter_info=0x71181748ad30) at /Server/10.6M/storage/innobase/handler/handler0alter.cc:8529

#3  0x0000563c4defe794 in handler::ha_inplace_alter_table (this=0x61d0009a92b8, altered_table=0x71181748b210, ha_alter_info=0x71181748ad30) at /Server/10.6M/sql/handler.h:4516

#4  0x0000563c4dede207 in mysql_inplace_alter_table (thd=0x62b0000bd218, table_list=0x62b0000c43f8, table=0x619000511398, altered_table=0x71181748b210, ha_alter_info=0x71181748ad30, target_mdl_request=0x71181748ae10, alter_ctx=0x71181748bc70) at /Server/10.6M/sql/sql_table.cc:8088

#5  0x0000563c4def16c4 in mysql_alter_table (thd=0x62b0000bd218, new_db=0x62b0000c1c90, new_name=0x62b0000c2090, create_info=0x71181748d030, table_list=0x62b0000c43f8, alter_info=0x71181748cf20, order_num=0, order=0x0, ignore=false, if_exists=false) at /Server/10.6M/sql/sql_table.cc:10677

#6  0x0000563c4e090ace in Sql_cmd_alter_table::execute (this=0x62b0000c4c20, thd=0x62b0000bd218) at /Server/10.6M/sql/sql_alter.cc:539

#7  0x0000563c4dc513e7 in mysql_execute_command (thd=0x62b0000bd218) at /Server/10.6M/sql/sql_parse.cc:5946

#8  0x0000563c4dc5ede7 in mysql_parse (thd=0x62b0000bd218, rawbuf=0x62b0000c4238 "ALTER TABLE t4 ADD UNIQUE ( col_text_g(9) ), LOCK = EXCLUSIVE, ALGORITHM = NOCOPY /* E_R Thread1 QNO 14 CON_ID 15 */", length=116, parser_state=0x71181748e7c0) at /Server/10.6M/sql/sql_parse.cc:7972

(gdb) cont

Continuing.

Thread 2 received signal SIGSEGV, Segmentation fault.

[Switching to Thread 2910372.2922937]

0x0000563c4f30c373 in row_create_prebuilt (table=0x618000050d20, mysql_row_len=25) at /Server/10.6M/storage/innobase/row/row0mysql.cc:882

882		for (temp_index = dict_table_get_first_index(table); temp_index;

marko It seems to be a race between inplace alter table and any concurrent open. git blame points to:

cb437417d2214474102a11f0a207bb3bc86252ee

MDEV-19114 gcol.innodb_virtual_debug: Assertion n_fields>0 failed

A race condition between a normal table open and open during purge is actually happening:

• normally, a table should be dropped while opening on insert, but this doesnt happen, because table->n_ref_count is 2:

  (gdb) bt

  #0 dict_table_try_drop_aborted_and_mutex_exit (table=0x563c4f60b68d <dict_lru_validate()+24>, try_drop=31748991368464) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:285

  #1 0x0000563c4f5f2551 in dict_table_open_on_name (table_name=0x1ce0235a45f0 "test/t4", dict_locked=0, try_drop=1, ignore_err=DICT_ERR_IGNORE_FK_NOKEY) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:1192

  #2 0x0000563c4ef71e62 in ha_innobase::open_dict_table (norm_name=0x1ce0235a45f0 "test/t4", is_partition=false, ignore_err=DICT_ERR_IGNORE_FK_NOKEY) at /Server/10.6M/storage/innobase/handler/ha_innodb.cc:5724

  #3 0x0000563c4ef6f351 in ha_innobase::open (this=0x61d0009ba0b8, name=0x61b0000b0450 "./test/t4") at /Server/10.6M/storage/innobase/handler/ha_innodb.cc:5437

  #4 0x0000563c4e453dc6 in handler::ha_open (this=0x61d0009ba0b8, table_arg=0x619000506998, name=0x61b0000b0450 "./test/t4", mode=2, test_if_locked=18, mem_root=0x0, partitions_to_open=0x0) at /Server/10.6M/sql/handler.cc:2984

  #5 0x0000563c4df9308d in open_table_from_share (thd=0x62b0000d2218, share=0x61b0000afeb8, alias=0x62b0000cb458, db_stat=33, prgflag=8, ha_open_flags=18, outparam=0x619000506998, is_create_table=false, partitions_to_open=0x0) at /Server/10.6M/sql/table.cc:4218

  #6 0x0000563c4da9ecb7 in open_table (thd=0x62b0000d2218, table_list=0x62b0000cb410, ot_ctx=0x1ce0235a5320) at /Server/10.6M/sql/sql_base.cc:2001

  #7 0x0000563c4daa8a53 in open_and_process_table (thd=0x62b0000d2218, tables=0x62b0000cb410, counter=0x1ce0235a5450, flags=0, prelocking_strategy=0x1ce0235a5580, has_prelocking_list=false, ot_ctx=0x1ce0235a5320) at /Server/10.6M/sql/sql_base.cc:3801

  #8 0x0000563c4daab5da in open_tables (thd=0x62b0000d2218, options=..., start=0x1ce0235a5460, counter=0x1ce0235a5450, flags=0, prelocking_strategy=0x1ce0235a5580) at /Server/10.6M/sql/sql_base.cc:4275

  #9 0x0000563c4dab01f4 in open_and_lock_tables (thd=0x62b0000d2218, options=..., tables=0x62b0000cb410, derived=true, flags=0, prelocking_strategy=0x1ce0235a5580) at /Server/10.6M/sql/sql_base.cc:5199

  #10 0x0000563c4da07f58 in open_and_lock_tables (thd=0x62b0000d2218, tables=0x62b0000cb410, derived=true, flags=0) at /Server/10.6M/sql/sql_base.h:507

  #11 0x0000563c4db7c460 in mysql_insert (thd=0x62b0000d2218, table_list=0x62b0000cb410, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false, result=0x0) at /Server/10.6M/sql/sql_insert.cc:756

  #12 0x0000563c4dc46982 in mysql_execute_command (thd=0x62b0000d2218) at /Server/10.6M/sql/sql_parse.cc:4519

  #13 0x0000563c4dc5ede7 in mysql_parse (thd=0x62b0000d2218, rawbuf=0x62b0000cb238 "INSERT INTO t4 (col1, col_text ) VALUES ( 243, REPEAT(SUBSTR(CAST( 243 AS CHAR),1,1), 100) ) /* E_R Thread2 QNO 371 CON_ID 16 */", length=128, parser_state=0x1ce0235a67c0) at /Server/10.6M/sql/sql_parse.cc:7972

  (gdb) p try_drop

  $10 = 1

  (gdb) p table->get_ref_count()

  $11 = 2

  (gdb) p table->drop_aborted

  $12 = 1

• previously ref_count increases 0 -> 1 in purge:

  (gdb) bt

  #0 0x0000563c4d94d9d6 in std::__atomic_base<unsigned int>::fetch_add (__m=std::memory_order_relaxed, __i=1, this=0x618000050f60) at /usr/include/c++/9/bits/atomic_base.h:541

  #1 Atomic_counter<unsigned int>::add (this=0x618000050f60, i=1) at /Server/10.6M/include/my_counter.h:27

  #2 0x0000563c4d94c310 in Atomic_counter<unsigned int>::operator++ (this=0x618000050f60) at /Server/10.6M/include/my_counter.h:36

  #3 0x0000563c4f60d531 in dict_table_t::acquire (this=0x618000050d20) at /Server/10.6M/storage/innobase/include/dict0dict.ic:1145

  #4 0x0000563c4f60ea08 in dict_sys_t::acquire (this=0x563c51dbab80 <dict_sys>, table=0x618000050d20) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:1068

  #5 0x0000563c4f5f0c47 in dict_table_open_on_id (table_id=19, dict_locked=true, table_op=DICT_TABLE_OP_NORMAL, thd=0x0, mdl=0x0) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:960

  #6 0x0000563c4f6111a9 in dict_acquire_mdl_shared<false> (table=0x618000050d20, thd=0x62b00003f218, mdl=0x61b000001ec8, table_op=DICT_TABLE_OP_NORMAL) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:878

  #7 0x0000563c4f5f0d43 in dict_table_open_on_id (table_id=19, dict_locked=false, table_op=DICT_TABLE_OP_NORMAL, thd=0x62b00003f218, mdl=0x61b000001ec8) at /Server/10.6M/storage/innobase/dict/dict0dict.cc:966

  #8 0x0000563c4f3655e3 in row_purge_parse_undo_rec (node=0x61b000001d20, undo_rec=0x6210000501a0 "", thr=0x617000000c58, updated_extern=0x71ac707da840) at /Server/10.6M/storage/innobase/row/row0purge.cc:929

  #9 0x0000563c4f366d23 in row_purge (node=0x61b000001d20, undo_rec=0x6210000501a0 "", thr=0x617000000c58) at /Server/10.6M/storage/innobase/row/row0purge.cc:1107

  #10 0x0000563c4f3670c5 in row_purge_step (thr=0x617000000c58) at /Server/10.6M/storage/innobase/row/row0purge.cc:1159

  #11 0x0000563c4f23d9cb in que_thr_step (thr=0x617000000c58) at /Server/10.6M/storage/innobase/que/que0que.cc:653

  #12 0x0000563c4f23dd73 in que_run_threads_low (thr=0x617000000c58) at /Server/10.6M/storage/innobase/que/que0que.cc:709

  #13 0x0000563c4f23df15 in que_run_threads (thr=0x617000000c58) at /Server/10.6M/storage/innobase/que/que0que.cc:729

  #14 0x0000563c4f3ea422 in srv_task_execute () at /Server/10.6M/storage/innobase/srv/srv0srv.cc:1797

  #15 0x0000563c4f3eb27c in purge_worker_callback () at /Server/10.6M/storage/innobase/srv/srv0srv.cc:1960

  ...

  dict_table_try_drop_aborted_and_mutex_exit will be called dict_table_open_on_id:

  if (!dict_locked) {

  if (thd) {

  table = dict_acquire_mdl_shared<false>( // <<< this line is in the stack trace

  table, thd, mdl, table_op);

  }

  dict_table_try_drop_aborted_and_mutex_exit(

  table, table_op == DICT_TABLE_OP_DROP_ORPHAN);

  }

  but it doesn't drop orphans, because table_op is DICT_TABLE_OP_NORMAL

In row_purge_parse_undo_rec:

try_again:

	node->table = dict_table_open_on_id(

		table_id, false, DICT_TABLE_OP_NORMAL, node->purge_thd,

		&node->mdl_ticket);

marko what do you think?
It seems, changing table_op to DICT_TABLE_OP_DROP_ORPHAN would solve this problem, but I am not proficient in the purge mechanics.

This patch reproduces the issue even without a pure thread

Index: mysql-test/suite/gcol/t/virt.test

<+>UTF-8

===================================================================

diff --git a/mysql-test/suite/gcol/t/virt.test b/mysql-test/suite/gcol/t/virt.test

new file mode 100644

--- /dev/null	(date 1617070094231)

+++ b/mysql-test/suite/gcol/t/virt.test	(date 1617070094231)

@@ -0,0 +1,24 @@

+--source include/have_innodb.inc

+

+--connect (con1,localhost,root,,test)

+--connection default

+CREATE TABLE t4 (col1 INT PRIMARY KEY, col_text TEXT,

+                 col_text_g TEXT AS (SUBSTR(col_text,1,499))

+  ) ENGINE = InnoDB ROW_FORMAT = Compact ;

+

+set debug_dbug= '+d,create_index_fail';

+

+--error ER_DUP_ENTRY

+ALTER TABLE t4 ADD UNIQUE u( col_text_g(9) ), LOCK = EXCLUSIVE , ALGORITHM = NOCOPY;

+

+set debug_sync='unlock_then_lock signal unlock wait_for finish timeout 6';

+--send

+INSERT INTO t4 (col1, col_text ) VALUES ( 512 , REPEAT(SUBSTR(CAST( 512 AS CHAR),1,1), 100) );

+--connection con1

+# row_create_prebuilt_loop_indexes

+set debug_sync='now wait_for unlock timeout 6';

+INSERT INTO t4 (col1, col_text ) VALUES ( 511 , REPEAT(SUBSTR(CAST( 512 AS CHAR),1,1), 100) );

+--connection default

+--reap

+set debug_sync='now signal read';

+DROP TABLE t4;

Index: storage/innobase/dict/dict0dict.cc

<+>UTF-8

===================================================================

diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc

--- a/storage/innobase/dict/dict0dict.cc	(revision 74281fe1fb0faf444aec3744b90995156f9f58f9)

+++ b/storage/innobase/dict/dict0dict.cc	(date 1617069764153)

@@ -294,7 +294,7 @@

 		table_id_t	table_id = table->id;

 		dict_sys.mutex_unlock();

-

+		DEBUG_SYNC(NULL, "unlock_then_lock");

 		dict_table_try_drop_aborted(table, table_id, 1);

 	} else {

 		dict_sys.mutex_unlock();

It also means that setting DICT_TABLE_OP_DROP_ORPHAN will not help

1) Alter table add index(vc) fails inside {{inplace_alter_table()}}and clears the added virtual
column via clear_added_indexes(). In rollback_inplace_alter_table() InnoDB marks the index
as DROP ABORTED and drops the index id from dictionary and retains the index stub in dict_table_t.

2) InnoDB clears the ABORTED index while opening the table in dict_table_try_drop_aborted().
But in our case, one connection is waiting to call dict_table_try_drop_aborted(). another
connection skips dict_table_try_drop_aborted() due to reference count. So it leads to
crash in row_prebuilt_create() while accessing the cleared column.

  if (try_drop

            && table != NULL

            && table->drop_aborted

            && table->get_ref_count() == 1

            && dict_table_get_first_index(table)) {

                /* Attempt to drop the indexes whose online creation

                was aborted. */

                table_id_t      table_id = table->id;

                dict_sys.mutex_unlock();

                DEBUG_SYNC(NULL, "unlock_then_lock");

                dict_table_try_drop_aborted(table, table_id, 1);

        } else {

                dict_sys.mutex_unlock();

        }

I think InnoDB should address two issues:

1) InnoDB should clear index stub objects when alter itself has exclusive lock on the table.
2) Other case related to committed index or online index abort. InnoDB should reload the
table definition when it has aborted index which has virtual column on it.

Thanks nikitamalyavin for the test case.

I think that the fix needs a thorough commit message that covers the life cycle of the changed data structures as well as mentions each added, removed or changed function.

thirumarko what was the commit? can't see it in jira

nikitamalyavin, the latest version is in bb-10.6-MDEV-24971. I suggested some cosmetic changes to make it more readable. It was based on 10.6 for test purposes, but the final one will be pushed to 10.2. thiru is now working on that.