[MDEV-13599] rocksdb.add_index_inplace_sstfilewriter failed in buildbot (AddressSanitizer failed)) Created: 2017-08-21  Updated: 2018-05-14  Resolved: 2018-05-14

Status: Closed
Project: MariaDB Server
Component/s: Storage Engine - RocksDB, Tests
Affects Version/s: 10.2
Fix Version/s: 10.2.15

Type: Bug Priority: Major
Reporter: Alice Sherepa Assignee: Sergei Petrunia
Resolution: Duplicate Votes: 0
Labels: None

Issue Links:
Relates
relates to MDEV-13904 rocksdb.add_index_inplace_sstfilewrit... Closed
relates to MDEV-13609 [OOM because of in-mem DDL] Convertin... Open

 Description   

http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest-big/builds/1158/steps/test_1/logs/stdio

==12292==ERROR: AddressSanitizer failed to allocate 0x2b003000 (721432576) bytes of LargeMmapAllocator (errno: 12)


rocksdb.add_index_inplace_sstfilewriter  w1 [ fail ]
 
        Test ended at 2017-08-19 23:57:28
 
 
 
CURRENT_TEST: rocksdb.add_index_inplace_sstfilewriter
 
mysqltest: At line 68: query 'ALTER TABLE t1 ADD INDEX kb_copy(b), ALGORITHM=COPY' failed: 2013: Lost connection to MySQL server during query
 
 
 
The result from queries just before the failure was:
 
drop table if exists t1;
 
CREATE TABLE t1(pk CHAR(5) PRIMARY KEY, a char(30), b char(30)) COLLATE 'latin1_bin';
 
set rocksdb_bulk_load=1;
 
set rocksdb_bulk_load_size=100000;
 
LOAD DATA INFILE <input_file> INTO TABLE t1;
 
set rocksdb_bulk_load=0;
 
select count(pk) from t1;
 
count(pk)
 
3000000
 
select count(a) from t1;
 
count(a)
 
3000000
 
select count(b) from t1;
 
count(b)
 
3000000
 
ALTER TABLE t1 ADD INDEX kb(b), ALGORITHM=INPLACE;
 
ALTER TABLE t1 ADD INDEX kb_copy(b), ALGORITHM=COPY;


==12292==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix.cc:121 "(("unable to mmap" && 0)) != (0)" (0x0, 0x0)
 
    #0 0x7f77b68ee631  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa0631)
 
    #1 0x7f77b68f3613 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa5613)
 
    #2 0x7f77b68fb641  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xad641)
 
    #3 0x7f77b6870c0c  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x22c0c)
 
    #4 0x7f77b68e74fe in operator new(unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x994fe)
 
    #5 0x7f77b5050498 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x11f498)
 
    #6 0x7f77b5051832 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x120832)
 
    #7 0x7f77ac075a37 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*, unsigned long) /usr/include/c++/5/bits/basic_string.h:1015
 
    #8 0x7f77ac075a37 in rocksdb::PutLengthPrefixedSlice(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, rocksdb::Slice const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/rocksdb/util/coding.h:249
 
    #9 0x7f77ac075a37 in rocksdb::WriteBatchInternal::Put(rocksdb::WriteBatch*, unsigned int, rocksdb::Slice const&, rocksdb::Slice const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/rocksdb/db/write_batch.cc:489
 
    #10 0x7f77ac075f65 in rocksdb::WriteBatch::Put(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/rocksdb/db/write_batch.cc:499
 
    #11 0x7f77ac286bb8 in rocksdb::WriteBatchWithIndex::Put(rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::Slice const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/rocksdb/utilities/write_batch_with_index/write_batch_with_index.cc:613
 
    #12 0x7f77abdc1485 in myrocks::ha_rocksdb::update_pk(myrocks::Rdb_key_def const&, myrocks::ha_rocksdb::update_row_info const&, bool const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/ha_rocksdb.cc:8360
 
    #13 0x7f77abdc24fa in myrocks::ha_rocksdb::update_indexes(myrocks::ha_rocksdb::update_row_info const&, bool const&) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/ha_rocksdb.cc:8462
 
    #14 0x7f77abdc2803 in myrocks::ha_rocksdb::update_write_row(unsigned char const*, unsigned char const*, bool) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/ha_rocksdb.cc:8527
 
    #15 0x7f77abdbe9bf in myrocks::ha_rocksdb::write_row(unsigned char*) /home/buildbot/buildbot/build/mariadb-10.2.9/storage/rocksdb/ha_rocksdb.cc:7942
 
    #16 0x5570d2801cf1 in handler::ha_write_row(unsigned char*) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/handler.cc:5965
 
    #17 0x5570d2143665 in copy_data_between_tables /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_table.cc:9905
 
    #18 0x5570d24e8d2e in mysql_alter_table(THD*, char*, char*, HA_CREATE_INFO*, TABLE_LIST*, Alter_info*, unsigned int, st_order*, bool) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_table.cc:9398
 
    #19 0x5570d25c19ac in Sql_cmd_alter_table::execute(THD*) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_alter.cc:324
 
    #20 0x5570d234659a in mysql_execute_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_parse.cc:6215
 
    #21 0x5570d235c1d8 in mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_parse.cc:7886
 
    #22 0x5570d2362d74 in dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_parse.cc:1812
 
    #23 0x5570d2364cc4 in do_command(THD*) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_parse.cc:1360
 
    #24 0x5570d25b8877 in do_handle_one_connection(CONNECT*) /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_connect.cc:1354
 
    #25 0x5570d25b8d2a in handle_one_connection /home/buildbot/buildbot/build/mariadb-10.2.9/sql/sql_connect.cc:1260
 
    #26 0x5570d2cd0a1b in pfs_spawn_thread /home/buildbot/buildbot/build/mariadb-10.2.9/storage/perfschema/pfs.cc:1862
 
    #27 0x7f77b52ba6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
 
    #28 0x7f77b496582c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10682c)



 Comments   
Comment by Sergei Petrunia [ 2017-08-21 ]

I was not able to reproduce on my machine. The test takes a lot of time ( ~ 773 seconds) but there's no failure.

Comment by Sergei Petrunia [ 2017-08-24 ]

Looking at data sizes:

var/log/rocksdb.add_index_inplace_sstfilewriter/mysqld.1/data/test_loadfile.txt has 3M rows.

The table is defined as 

CREATE TABLE t1(pk CHAR(5) PRIMARY KEY, a char(30), b char(30)) COLLATE 'latin1_bin';
 
ALTER TABLE t1 ADD INDEX kb(b), ALGORITHM=INPLACE;
 
ALTER TABLE t1 ADD INDEX kb_copy(b), ALGORITHM=COPY;

Let's see how much memory it will require

PK 
 key: 4 (indexnr) + 5 (char(5))= 9 bytes
 
 val: 4 (indexnr) + 1 (null-byte) + 30 (char(30)) + 30 (char(30)) = 61
 
 total: 70 bytes/row


SK
 
 key: 4 (indexnr) + 1 (null-byte) + 30 (char(30) + 5 (char(5)) = 40 
 val: 0 bytes
 
 total: 40 bytes/row


PK + 2*sk = 150 bytes/row

The table after the second ALTER TABLE:

3M * 150 = 450 M.

That's just the raw data size. I assume that keeping the data sorted also adds some overhead. Plus, std::string can allocate some extra as well?

It is allocating 721,432,576 bytes, which is substantially more than 450 M but still less than 2x more.

Comment by Sergei Petrunia [ 2018-05-14 ]

rocksdb.add_index_inplace_sstfilewriter was down-scaled since this bug was reported.
Memory usage is high but I don't consider that an indication of a bug. It is "natural causes".

Generated at Thu Feb 08 08:06:51 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.