Marko Mäkelä added a comment - 2019-01-30 13:23 The table schema is: CREATE TABLE t1(id INT AUTO_INCREMENT PRIMARY KEY , msg VARCHAR (255), KEY msg_i(msg)) ENGINE=INNODB ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=8; The fields of the clustered index records are: (id,DB_TRX_ID,DB_ROLL_PTR,msg). AddressSanitizer claims an unknown error on the very last byte of the copied undo log record (recording the zero length of the field msg). Here are the parsed undo log records for 2 occurrences that I debugged more closely. Neither me nor kevg are able to reproduce this failure on any other system. This system is using gcc-5.real (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609 The undo log record of the first analyzed failure: 10.0 1522ee2949ae304ad9092894896a6272dc08bb39 0xe = TRX_UNDO_DEL_MARK_REC 0x80b8 = undo_no = 184 0x12 = table_id (test.t1) 0x0 = info_bits 0x0, 0x0, 0x0, 0xa, 0xd = trx_id = 2573 (trx_sys->max_trx_id=3200) 0xe0, 0x8d, 0x0, 0x0, 0x1, 0x47, 0x1, 0x10 = roll_ptr 0x4, 0x80, 0x0, 0x0, 0xd2 = primary key (id=210) 0x0, 0xa = 10 more bytes of data 0x0, 0x4, 0x80, 0x0, 0x0, 0xd2, (id=210) 0x3, 0x0, (msg='') The undo log record of the second analyzed failure: 10.0 1522ee2949ae304ad9092894896a6272dc08bb39 0xe=TRX_UNDO_DEL_MARK_REC 0x80, 0xbd = undo_no=189 0x12 = table_id (test.t1) 0x0 = info_bits 0x0, 0x0, 0x0, 0x6, 0x22 = trx_id = 1570 (trx_sys->max_trx_id=2874) 0xe0, 0xa2, 0x0, 0x0, 0x1, 0x5c, 0x1, 0x10 = roll_ptr 0x4, 0x80, 0x0, 0x4, 0xbd = primary key (id=1213) 0x0, 0xa = 10 more bytes of data 0x0, 0x4, 0x80, 0x0, 0x4, 0xbd, (id=1213) 0x3, 0x0, (msg='') If I revert commit 71e9f0d123e201141c80a240d50347ab36ce126f ( MDEV-17797 ) which improved ASAN instrumentation, then no failure is being reported. This (and the fact that ASAN builds with newer compilers do not report anything) would lead me to believe that there is something wrong with the ASAN bookkeeping.

Marko Mäkelä added a comment - 2019-02-05 07:10 - edited I encountered a similar failure in a different test on 10.3: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/2420/steps/mtr_emb/logs/stdio 10.3 7293ce0ee81f05b1ec3ac9ddcc88bfbee4030e55 CURRENT_TEST: gcol.innodb_virtual_basic Warning: /mnt/buildbot/build/mariadb-10.3.13/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-ca=/mnt/buildbot/build/mariadb-10.3.13/mysql-test/std_data/cacert.pem' Warning: /mnt/buildbot/build/mariadb-10.3.13/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-cert=/mnt/buildbot/build/mariadb-10.3.13/mysql-test/std_data/client-cert.pem' Warning: /mnt/buildbot/build/mariadb-10.3.13/libmysqld/examples/mysqltest_embedded: unknown variable 'loose-ssl-key=/mnt/buildbot/build/mariadb-10.3.13/mysql-test/std_data/client-key.pem' Warning: /mnt/buildbot/build/mariadb-10.3.13/libmysqld/examples/mysqltest_embedded: unknown option '--loose-skip-ssl' ================================================================= ==1543==ERROR: AddressSanitizer: unknown-crash on address 0x61300000d104 at pc 0x5565507a411f bp 0x7ff46efa94f0 sp 0x7ff46efa94e0 READ of size 4 at 0x61300000d104 thread T18 #0 0x5565507a411e in trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0rec.cc:577 #1 0x556550701687 in row_purge_parse_undo_rec /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1046 #2 0x556550701687 in row_purge /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1254 #3 0x556550701687 in row_purge_step(que_thr_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:1343 #4 0x55655071070a in que_thr_step /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1042 #5 0x55655071070a in que_run_threads_low /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1104 #6 0x55655071070a in que_run_threads(que_thr_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/que/que0que.cc:1144 #7 0x5565508ef308 in srv_task_execute /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0srv.cc:2449 #8 0x5565508ef308 in srv_worker_thread /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0srv.cc:2497 #9 0x7ff47d8226b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) #10 0x7ff47bdbf82c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)   0x61300000d104 is located 132 bytes inside of 368-byte region [0x61300000d080,0x61300000d1f0) allocated by thread T0 here: #0 0x7ff47dad0602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602) #1 0x5565507036be in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/mem/mem0mem.cc:269 #2 0x5565506fa945 in mem_heap_create_func /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/include/mem0mem.ic:484 #3 0x5565506fa945 in row_purge_node_create(que_thr_t*, mem_block_info_t*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/row/row0purge.cc:80 #4 0x5565504f2566 in purge_graph_build /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0purge.cc:151 #5 0x5565504f2566 in purge_sys_t::create() /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0purge.cc:166 #6 0x5565507877bb in trx_lists_init_at_db_start() /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0trx.cc:705 #7 0x55655085b5fc in srv_start(bool) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0start.cc:1973 #8 0x556550480249 in innodb_init /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/handler/ha_innodb.cc:4266 #9 0x55654f7d75bc in ha_initialize_handlerton(st_plugin_int*) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/handler.cc:523 #10 0x556550136c27 in plugin_initialize /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1432 #11 0x556550137ddc in plugin_init(int*, char**, int) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1714 #12 0x55654f6f3d47 in init_server_components /home/buildbot/buildbot/build/mariadb-10.3.13/libmysqld/../sql/mysqld.cc:5385 #13 0x55654f6fc66d in init_embedded_server /home/buildbot/buildbot/build/mariadb-10.3.13/libmysqld/lib_sql.cc:593 #14 0x55654f62c333 in main /home/buildbot/buildbot/build/mariadb-10.3.13/client/mysqltest.cc:9297 #15 0x7ff47bcd982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)   Thread T18 created by T0 here: #0 0x7ff47da6e253 in pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x36253) #1 0x556550b1ff27 in os_thread_create_func(void* (*)(void*), void*, unsigned long*) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/os/os0thread.cc:132 #2 0x556550859802 in srv_start(bool) /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/srv/srv0start.cc:2414 #3 0x556550480249 in innodb_init /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/handler/ha_innodb.cc:4266 #4 0x55654f7d75bc in ha_initialize_handlerton(st_plugin_int*) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/handler.cc:523 #5 0x556550136c27 in plugin_initialize /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1432 #6 0x556550137ddc in plugin_init(int*, char**, int) /home/buildbot/buildbot/build/mariadb-10.3.13/sql/sql_plugin.cc:1714 #7 0x55654f6f3d47 in init_server_components /home/buildbot/buildbot/build/mariadb-10.3.13/libmysqld/../sql/mysqld.cc:5385 #8 0x55654f6fc66d in init_embedded_server /home/buildbot/buildbot/build/mariadb-10.3.13/libmysqld/lib_sql.cc:593 #9 0x55654f62c333 in main /home/buildbot/buildbot/build/mariadb-10.3.13/client/mysqltest.cc:9297 #10 0x7ff47bcd982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)   SUMMARY: AddressSanitizer: unknown-crash /home/buildbot/buildbot/build/mariadb-10.3.13/storage/innobase/trx/trx0rec.cc:577 trx_undo_rec_get_pars(unsigned char*, unsigned long*, unsigned long*, bool*, unsigned long*, unsigned long*) Shadow bytes around the buggy address: 0x0c267fff99d0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 0x0c267fff99e0: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0c267fff99f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0c267fff9a00: f7 f7 f7 f7 f7 f7 fa fa fa fa fa fa fa fa fa fa 0x0c267fff9a10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x0c267fff9a20:[07]00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 0x0c267fff9a30: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 fa fa 0x0c267fff9a40: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00 0x0c267fff9a50: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0c267fff9a60: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0c267fff9a70: f7 f7 f7 f7 f7 f7 fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe ==1543==ABORTING

Marko Mäkelä added a comment - 2019-02-05 07:15 The same run shows similar errors (with different stack trace, but accessing a copied undo log record) for the tests parts.partition_alter2_1_2_innodb , parts.partition_alter2_2_2_innodb as well.

Marko Mäkelä added a comment - 2019-02-05 07:26 I am unable to repeat these locally. I would suspect a bug in an older compiler’s -fsanitize=address implementation: gcol.innodb_virtual_basic 'innodb' [ pass ] 8564 parts.partition_alter2_1_2_innodb 'innodb' [ pass ] 24834 parts.partition_alter2_2_2_innodb 'innodb' [ pass ] 24632 Separately, I ran these two test suites, with no errors whatsoever: ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto --force --retry=0 --big-test --embedded --suite=parts,gcol I am using the following compilers: clang version 7.0.1-4 (tags/RELEASE_701/final) gcc (Debian 8.2.0-16) 8.2.0 My build does catch MDEV-13942 reliably, but not this failure.

Marko Mäkelä added a comment - 2019-03-27 11:06 It seems to me that GCC 5.4 on Ubuntu Xenial is emitting invalid code WITH_ASAN for -O2 or -O3. I fixed this by appending -O1 to the compilation flags of trx0rec.cc if the compiler is GCC and older than 6.0.0 and the AddressSanitizer instrumentation is enabled.