Marko Mäkelä added a comment - 2017-10-09 11:35 I can repeat the crash on 10.2 with the attached data2.tar.gz . The crash occurs on this column: `Counter` int (11) GENERATED ALWAYS AS (`AttendDet_ID`) VIRTUAL, In 10.2, InnoDB would store virtual columns in the dictionary table SYS_VIRTUAL. There is no such table in 10.1 or in the attached data dump. So, we will have table->n_v_def==0. The question is: Why does ha_innobase::build_template() think that this column should be included in the query? There is no index on the column (because before MDEV-5800 in 10.2, indexed virtual columns were not supported in InnoDB). We do have !whole_row, so the wrong decision must have been made by build_template_needs_field(). The input parameter looks wrong: if (innobase_is_v_fld(table->field[i])) { contain = dict_index_contains_col_or_prefix( index, num_v, true ); The clustered index never really contains virtual columns, but the code incorrectly claims so: /** Returns TRUE if the index contains a column or a prefix of that column. @param[in] index index @param[in] n column number @param[in] is_virtual whether it is a virtual col @return TRUE if contains the column or its prefix */ ibool dict_index_contains_col_or_prefix( const dict_index_t* index, ulint n, bool is_virtual) { const dict_field_t* field; const dict_col_t* col; ulint pos; ulint n_fields;   ut_ad(index); ut_ad(index->magic_n == DICT_INDEX_MAGIC_N);   if (dict_index_is_clust(index)) {   return (TRUE); }

Marko Mäkelä added a comment - 2017-10-09 12:25 I fixed the InnoDB part with the patch below. But, there still is something wrong: ==11359==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6190000914d2 at pc 0x0000006b83f1 bp 0x7fffdfe94490 sp 0x7fffdfe93c40 READ of size 16736 at 0x6190000914d2 thread T27 #0 0x6b83f0 in __asan_memcpy (/mariadb/10.2/build/sql/mysqld+0x6b83f0) #1 0xc8a761 in String::append(char const*, unsigned long) /mariadb/10.2/sql/sql_string.cc:473:3 #2 0xd8416e in parse_vcol_defs(THD*, st_mem_root*, TABLE*, bool*) /mariadb/10.2/sql/table.cc:1062:14 #3 0xd8f67c in open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool) /mariadb/10.2/sql/table.cc:3176:9 #4 0x8738e2 in open_table(THD*, TABLE_LIST*, Open_table_context*) /mariadb/10.2/sql/sql_base.cc:1874:12 #5 0x87ed03 in open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) /mariadb/10.2/sql/sql_base.cc:3409:14 #6 0x87b728 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /mariadb/10.2/sql/sql_base.cc:3926:14 #7 0x8875e4 in open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /mariadb/10.2/sql/sql_base.h:235:10 #8 0x8871a8 in open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) /mariadb/10.2/sql/sql_base.cc:4745:7 #9 0xc02965 in mysqld_list_fields(THD*, TABLE_LIST*, char const*) /mariadb/10.2/sql/sql_show.cc:1422:7 #10 0x9df01a in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /mariadb/10.2/sql/sql_parse.cc:2006:5 #11 0x9e5bcc in do_command(THD*) /mariadb/10.2/sql/sql_parse.cc:1359:17 #12 0xe80b0a in do_handle_one_connection(CONNECT*) /mariadb/10.2/sql/sql_connect.cc:1354:11 #13 0xe80231 in handle_one_connection /mariadb/10.2/sql/sql_connect.cc:1260:3 #14 0x28cfec4 in pfs_spawn_thread /mariadb/10.2/storage/perfschema/pfs.cc:1862:3 #15 0x6dae52 in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) (/mariadb/10.2/build/sql/mysqld+0x6dae52) #16 0x7ffff7bc3493 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7493) #17 0x7ffff6405abe in clone (/lib/x86_64-linux-gnu/libc.so.6+0xe8abe)   0x6190000914d2 is located 98 bytes to the right of 1008-byte region [0x619000091080,0x619000091470) allocated by thread T27 here: #0 0x6cd3a0 in __interceptor_malloc (/mariadb/10.2/build/sql/mysqld+0x6cd3a0) #1 0x2a050c5 in my_malloc /mariadb/10.2/mysys/my_malloc.c:101:10 #2 0x29d67cf in alloc_root /mariadb/10.2/mysys/my_alloc.c:184:28 #3 0x29d717c in multi_alloc_root /mariadb/10.2/mysys/my_alloc.c:304:24 #4 0xd761f7 in TABLE_SHARE::init_from_binary_frm_image(THD*, bool, unsigned char const*, unsigned long) /mariadb/10.2/sql/table.cc:1620:8 #5 0xd6f890 in open_table_def(THD*, TABLE_SHARE*, unsigned int) /mariadb/10.2/sql/table.cc:669:10 #6 0x10610fb in tdc_acquire_share(THD*, TABLE_LIST*, unsigned int, TABLE**) /mariadb/10.2/sql/table_cache.cc:825:5 #7 0x872922 in open_table(THD*, TABLE_LIST*, Open_table_context*) /mariadb/10.2/sql/sql_base.cc:1742:10 #8 0x87ed03 in open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) /mariadb/10.2/sql/sql_base.cc:3409:14 #9 0x87b728 in open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /mariadb/10.2/sql/sql_base.cc:3926:14 #10 0x8875e4 in open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) /mariadb/10.2/sql/sql_base.h:235:10 #11 0x8871a8 in open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) /mariadb/10.2/sql/sql_base.cc:4745:7 #12 0xc02965 in mysqld_list_fields(THD*, TABLE_LIST*, char const*) /mariadb/10.2/sql/sql_show.cc:1422:7 #13 0x9df01a in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /mariadb/10.2/sql/sql_parse.cc:2006:5 #14 0x9e5bcc in do_command(THD*) /mariadb/10.2/sql/sql_parse.cc:1359:17 #15 0xe80b0a in do_handle_one_connection(CONNECT*) /mariadb/10.2/sql/sql_connect.cc:1354:11 #16 0xe80231 in handle_one_connection /mariadb/10.2/sql/sql_connect.cc:1260:3 #17 0x28cfec4 in pfs_spawn_thread /mariadb/10.2/storage/perfschema/pfs.cc:1862:3 #18 0x6dae52 in __asan::AsanThread::ThreadStart(unsigned long, __sanitizer::atomic_uintptr_t*) (/mariadb/10.2/build/sql/mysqld+0x6dae52)   Thread T27 created by T0 here: #0 0x630890 in pthread_create (/mariadb/10.2/build/sql/mysqld+0x630890) #1 0x28d4e9b in spawn_thread_v1(unsigned int, unsigned long*, pthread_attr_t const*, void* (*)(void*), void*) /mariadb/10.2/storage/perfschema/pfs.cc:1912:15 #2 0x713e2a in inline_mysql_thread_create(unsigned int, unsigned long*, pthread_attr_t const*, void* (*)(void*), void*) /mariadb/10.2/include/mysql/psi/mysql_thread.h:1239:11 #3 0x7230dd in create_thread_to_handle_connection(CONNECT*) /mariadb/10.2/sql/mysqld.cc:6445:15 #4 0x72517f in create_new_thread(CONNECT*) /mariadb/10.2/sql/mysqld.cc:6515:3 #5 0x72298a in handle_connections_sockets() /mariadb/10.2/sql/mysqld.cc:6790:5 #6 0x717246 in mysqld_main(int, char**) /mariadb/10.2/sql/mysqld.cc:6064:3 #7 0x70b661 in main /mariadb/10.2/sql/main.cc:25:10 #8 0x7ffff633d2e0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202e0)   SUMMARY: AddressSanitizer: heap-buffer-overflow (/mariadb/10.2/build/sql/mysqld+0x6b83f0) in __asan_memcpy Shadow bytes around the buggy address: 0x0c328000a240: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a250: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a260: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a270: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa =>0x0c328000a290: fa fa fa fa fa fa fa fa fa fa[fa]fa fa fa fa fa 0x0c328000a2a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c328000a2b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a2c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a2d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x0c328000a2e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 This happens during client startup: #5 0x00000000006b8410 in __asan_memcpy () #6 0x0000000000c8a762 in String::append (this=0x7fffdfe946e0, s=0x6190000914d2 "", size=16736) at /mariadb/10.2/sql/sql_string.cc:473 #7 0x0000000000d8416f in parse_vcol_defs (thd=0x62a0000ba208, mem_root=0x61e000041f40, table=0x61e000041488, error_reported=0x7fffdfe95180) at /mariadb/10.2/sql/table.cc:1062 #8 0x0000000000d8f67d in open_table_from_share (thd=0x62a0000ba208, share=0x61a0000702a0, alias=0x60600015fdc0 "AttendDet", db_stat=33, prgflag=8, ha_open_flags=16, outparam=0x61e000041488, ) at /mariadb/10.2/sql/table.cc:3176 #9 0x00000000008738e3 in open_table (thd=0x62a0000ba208, table_list=0x7fffdfe99010, ot_ctx=0x7fffdfe975c0) at /mariadb/10.2/sql/sql_base.cc:1874 #10 0x000000000087ed04 in open_and_process_table (thd=0x62a0000ba208, lex=0x62a0000bdce0, tables=0x7fffdfe99010, counter=0x7fffdfe97e60, flags=1024, prelocking_strategy=0x7fffdfe97e40, has_prelocking_list=false, ot_ctx=0x7fffdfe975c0) at /mariadb/10.2/sql/sql_base.cc:3409 #11 0x000000000087b729 in open_tables (thd=0x62a0000ba208, options=..., start=0x7fffdfe97e20, counter=0x7fffdfe97e60, flags=1024, prelocking_strategy=0x7fffdfe97e40) at /mariadb/10.2/sql/sql_base.cc:3926 #12 0x00000000008875e5 in open_tables (thd=0x62a0000ba208, tables=0x7fffdfe97e20, counter=0x7fffdfe97e60, flags=1024, prelocking_strategy=0x7fffdfe97e40) at /mariadb/10.2/sql/sql_base.h:235 #13 0x00000000008871a9 in open_normal_and_derived_tables (thd=0x62a0000ba208, tables=0x7fffdfe99010, flags=1024, dt_phases=35) at /mariadb/10.2/sql/sql_base.cc:4745 #14 0x0000000000c02966 in mysqld_list_fields (thd=0x62a0000ba208, table_list=0x7fffdfe99010, wild=0x604000016ef0 "") at /mariadb/10.2/sql/sql_show.cc:1422 #15 0x00000000009df01b in dispatch_command (command=COM_FIELD_LIST, thd=0x62a0000ba208, packet=0x62900008c213 "", packet_length=10, is_com_multi=false, is_next_command=false) at /mariadb/10.2/sql/sql_parse.cc:2006 #16 0x00000000009e5bcd in do_command (thd=0x62a0000ba208) at /mariadb/10.2/sql/sql_parse.cc:1359 #17 0x0000000000e80b0b in do_handle_one_connection (connect=0x6080000021a8) at /mariadb/10.2/sql/sql_connect.cc:1354 #18 0x0000000000e80232 in handle_one_connection (arg=0x6080000021a8) at /mariadb/10.2/sql/sql_connect.cc:1260 This is my InnoDB patch: diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc index 54e0115070d..b95592d43d4 100644 --- a/storage/innobase/dict/dict0dict.cc +++ b/storage/innobase/dict/dict0dict.cc @@ -912,8 +912,7 @@ dict_index_contains_col_or_prefix( ut_ad(index->magic_n == DICT_INDEX_MAGIC_N); if (dict_index_is_clust(index)) { - - return(TRUE); + return(!is_virtual); } if (is_virtual) { diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 0e9708eb151..3374f46fb24 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -8204,13 +8204,16 @@ ha_innobase::build_template( } else { ibool contain; - if (innobase_is_v_fld(table->field[i])) { - contain = dict_index_contains_col_or_prefix( - index, num_v, true); - } else { + if (!innobase_is_v_fld(table->field[i])) { contain = dict_index_contains_col_or_prefix( index, i - num_v, false); + } else if (dict_index_is_clust(index)) { + num_v++; + continue; + } else { + contain = dict_index_contains_col_or_prefix( + index, num_v, true); } field = build_template_needs_field( With the above patch, I only tested that SELECT and CHECK TABLE works. I did not test various forms of ALTER TABLE…ALGORITHM=INPLACE yet. I think that we should also test MDEV-11369 in 10.3 with the same dataset.

Marko Mäkelä added a comment - 2017-10-09 12:31 For some reason, I did not originally get the above-reported stack trace in ASAN. It appears to occur on the first access to the table. With mysql -A, it happens on the first SQL statement that refers to the table (say, CHECK TABLE AttendDet).

Marko Mäkelä added a comment - 2017-10-09 12:36 - edited I pushed the InnoDB fix to 10.2. With that fix (and when building the server without -DWITH_ASAN), the server no longer crashes on SELECT, but it does show another problem that I filed as MDEV-14046 Allow ALGORITHM=INPLACE for 10.1 tables that contain virtual columns I see that serg pushed a work-around for this into bb-10.2-serg , forcing ALGORITHM=COPY for the affected tables. That work-around is OK for now, but the commit comment should refer to MDEV-14046 .

Marko Mäkelä added a comment - 2017-10-11 08:09 The heap buffer overflow that I reported is not repeatable today. It might depend on the nondeterministic behaviour of the memory allocator. Maybe the memory alignment plays a role? I tested the current 10.2 built in the following configurations: cmake -DWITH_ASAN=1 -DCMAKE_BUILD_TYPE=Debug cmake -DWITH_ASAN=1 -DCMAKE_BUILD_TYPE=RelWithDebInfo