Details
-
Technical task
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
Description
A 5000-row table with one INT pk and one MEDIUMTEXT field (the field is populated with ~200K values) takes about 1.6 Gb memory while it is being populated and about 1 Gb afterwards. A 10K-row table might take up to 5 Gb while it's been populated.
With InnoDB and otherwise the same flow, there is no visible growth in memory usage. See the comparative output below.
LevelDB:
CREATE TABLE t1 (pk INT PRIMARY KEY, f MEDIUMTEXT) ENGINE=LevelDB; |
# Before INSERTs
|
10723 elenst 20 0 312m 87m 9748 S 0 1.5 0:00.44 mysqld
|
BEGIN; |
# Inserting 5000 rows |
# After INSERTs, before COMMIT |
10723 elenst 20 0 1903m 1.0g 9976 S 0 17.8 0:09.02 mysqld
|
# After COMMIT and having let things settle down a bit (5 sec wait) |
10723 elenst 20 0 1268m 1.0g 9.9m S 0 17.4 0:12.64 mysqld
|
InnoDB:
CREATE TABLE t1 (pk INT PRIMARY KEY, f MEDIUMTEXT) ENGINE=InnoDB; |
# Before INSERTs
|
10625 elenst 20 0 310m 87m 9.8m S 0 1.5 0:00.46 mysqld
|
BEGIN; |
# Inserting 5000 rows |
# After INSERTs, before COMMIT |
10625 elenst 20 0 312m 95m 10m S 0 1.6 1:58.42 mysqld
|
# After COMMIT and having let things settle down a bit (5 sec wait) |
10625 elenst 20 0 312m 95m 10m S 0 1.6 1:58.46 mysqld
|
Test case (please note it's just for experimenting, not for regression suite):
let $num = 5000;
|
|
|
CREATE TABLE t1 (pk INT PRIMARY KEY, f MEDIUMTEXT) ENGINE=LevelDB; |
|
|
--echo # Before INSERTs
|
--exec top -b -n 1 | grep mysqld
|
--exec du -sk $MYSQLTEST_VARDIR
|
|
|
SELECT NOW(); |
BEGIN; |
--echo # Inserting $num rows
|
--disable_query_log
|
while ($num)
|
{
|
eval INSERT INTO t1 VALUES (10001-$num, REPEAT('a',200000)); |
dec $num; |
}
|
--echo # After INSERTs, before COMMIT
|
--exec top -b -n 1 | grep mysqld
|
--exec du -sk $MYSQLTEST_VARDIR
|
COMMIT; |
SELECT NOW(); |
|
|
--sleep 5
|
--echo # After COMMIT and having let things settle down a bit
|
--exec top -b -n 1 | grep mysqld
|
--exec du -sk $MYSQLTEST_VARDIR
|
DROP TABLE t1; |
The size of the datadir is comparable (about 1 Gb in each case).
Execution time for LevelDB is much less.
revision-id: psergey@askmonty.org-20130124165745-nm4yaxrsu8e4o2ll
|
revno: 4511
|
branch-nick: mysql-5.6-leveldb
|
Below is a smaller test case, which might not show a noticeable growth, but it's faster to execute and it still causes valgrind complaints about lost blocks:
let $num = 50;
|
|
|
CREATE TABLE t1 (pk INT PRIMARY KEY, f MEDIUMTEXT) ENGINE=LevelDB; |
|
|
while ($num)
|
{
|
eval INSERT INTO t1 VALUES (10001-$num, REPEAT('a',200000)); |
dec $num; |
}
|
--sleep 5
|
8 bytes in 1 blocks are still reachable in loss record 1 of 367
|
at 0x4C28B35: operator new(unsigned long) (vg_replace_malloc.c:261)
|
by 0x4E69DAD: leveldb::InitModule() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x58D3712: pthread_once (pthread_once.S:104)
|
by 0x4E6F288: leveldb::port::InitOnce(int*, void (*)()) (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x4E6A026: leveldb::BytewiseComparator() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x4E6EDD8: leveldb::Options::Options() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0xA5D8AC: leveldb_init_func(void*) (ha_leveldb.cc:425)
|
by 0x6148DB: ha_initialize_handlerton(st_plugin_int*) (handler.cc:658)
|
by 0x7AE3FF: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1126)
|
by 0x7AED78: plugin_init(int*, char**, int) (sql_plugin.cc:1417)
|
by 0x602B0A: init_server_components() (mysqld.cc:4746)
|
by 0x60379D: mysqld_main(int, char**) (mysqld.cc:5320)
|
by 0x5FA7E3: main (main.cc:25)
|
|
|
8 bytes in 1 blocks are definitely lost in loss record 2 of 367
|
at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
|
by 0xA3CBA5: my_malloc (my_malloc.c:38)
|
by 0xA5EBA8: ha_leveldb::create_key_defs(TABLE*, bool) (ha_leveldb.cc:793)
|
by 0xA5EDB4: ha_leveldb::create(char const*, TABLE*, st_ha_create_information*) (ha_leveldb.cc:859)
|
by 0x61C231: handler::ha_create(char const*, TABLE*, st_ha_create_information*) (handler.cc:4392)
|
by 0x61CAA5: ha_create_table(THD*, char const*, char const*, char const*, st_ha_create_information*, bool) (handler.cc:4628)
|
by 0x86801E: rea_create_table(THD*, char const*, char const*, char const*, st_ha_create_information*, List<Create_field>&, unsigned int, st_key*, handler*, bool) (unireg.cc:527)
|
by 0x804384: create_table_impl(THD*, char const*, char const*, char const*, st_ha_create_information*, Alter_info*, bool, unsigned int, bool, bool*, st_key**, unsigned int*) (sql_table.cc:4724)
|
by 0x804851: mysql_create_table_no_lock(THD*, char const*, char const*, st_ha_create_information*, Alter_info*, unsigned int, bool*) (sql_table.cc:4825)
|
by 0x80495A: mysql_create_table(THD*, TABLE_LIST*, st_ha_create_information*, Alter_info*) (sql_table.cc:4871)
|
by 0x79E53A: mysql_execute_command(THD*) (sql_parse.cc:2962)
|
by 0x7A6BE9: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6097)
|
by 0x79AA26: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1312)
|
by 0x799BE9: do_command(THD*) (sql_parse.cc:1036)
|
by 0x764FE0: do_handle_one_connection(THD*) (sql_connect.cc:969)
|
by 0x764A05: handle_one_connection (sql_connect.cc:885)
|
|
|
8 bytes in 1 blocks are definitely lost in loss record 3 of 367
|
at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
|
by 0xA3CBA5: my_malloc (my_malloc.c:38)
|
by 0xA5E909: ha_leveldb::open(char const*, int, unsigned int) (ha_leveldb.cc:744)
|
by 0x617D62: handler::ha_open(TABLE*, char const*, int, int) (handler.cc:2421)
|
by 0x853FE3: open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool) (table.cc:2152)
|
by 0x738C4C: open_table(THD*, TABLE_LIST*, Open_table_context*) (sql_base.cc:3035)
|
by 0x73B124: open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) (sql_base.cc:4510)
|
by 0x73BE0C: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) (sql_base.cc:4945)
|
by 0x73CE87: open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned int) (sql_base.cc:5633)
|
by 0x77EF15: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:712)
|
by 0x79F509: mysql_execute_command(THD*) (sql_parse.cc:3344)
|
by 0x7A6BE9: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6097)
|
by 0x79AA26: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1312)
|
by 0x799BE9: do_command(THD*) (sql_parse.cc:1036)
|
by 0x764FE0: do_handle_one_connection(THD*) (sql_connect.cc:969)
|
by 0x764A05: handle_one_connection (sql_connect.cc:885)
|
|
|
14 bytes in 1 blocks are definitely lost in loss record 4 of 367
|
at 0x4C28F9F: malloc (vg_replace_malloc.c:236)
|
by 0xA3CBA5: my_malloc (my_malloc.c:38)
|
by 0xA5E93C: ha_leveldb::open(char const*, int, unsigned int) (ha_leveldb.cc:746)
|
by 0x617D62: handler::ha_open(TABLE*, char const*, int, int) (handler.cc:2421)
|
by 0x853FE3: open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool) (table.cc:2152)
|
by 0x738C4C: open_table(THD*, TABLE_LIST*, Open_table_context*) (sql_base.cc:3035)
|
by 0x73B124: open_and_process_table(THD*, LEX*, TABLE_LIST*, unsigned int*, unsigned int, Prelocking_strategy*, bool, Open_table_context*) (sql_base.cc:4510)
|
by 0x73BE0C: open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*) (sql_base.cc:4945)
|
by 0x73CE87: open_normal_and_derived_tables(THD*, TABLE_LIST*, unsigned int) (sql_base.cc:5633)
|
by 0x77EF15: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:712)
|
by 0x79F509: mysql_execute_command(THD*) (sql_parse.cc:3344)
|
by 0x7A6BE9: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6097)
|
by 0x79AA26: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1312)
|
by 0x799BE9: do_command(THD*) (sql_parse.cc:1036)
|
by 0x764FE0: do_handle_one_connection(THD*) (sql_connect.cc:969)
|
by 0x764A05: handle_one_connection (sql_connect.cc:885)
|
|
|
64 bytes in 1 blocks are still reachable in loss record 187 of 367
|
at 0x4C28B35: operator new(unsigned long) (vg_replace_malloc.c:261)
|
by 0x4E6D012: leveldb::InitDefaultEnv() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x58D3712: pthread_once (pthread_once.S:104)
|
by 0x4E6CF56: leveldb::Env::Default() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x4E6EDEC: leveldb::Options::Options() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0xA5D8AC: leveldb_init_func(void*) (ha_leveldb.cc:425)
|
by 0x6148DB: ha_initialize_handlerton(st_plugin_int*) (handler.cc:658)
|
by 0x7AE3FF: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1126)
|
by 0x7AED78: plugin_init(int*, char**, int) (sql_plugin.cc:1417)
|
by 0x602B0A: init_server_components() (mysqld.cc:4746)
|
by 0x60379D: mysqld_main(int, char**) (mysqld.cc:5320)
|
by 0x5FA7E3: main (main.cc:25)
|
|
|
336 bytes in 1 blocks are still reachable in loss record 259 of 367
|
at 0x4C28B35: operator new(unsigned long) (vg_replace_malloc.c:261)
|
by 0x4E6CF81: leveldb::InitDefaultEnv() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x58D3712: pthread_once (pthread_once.S:104)
|
by 0x4E6CF56: leveldb::Env::Default() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x4E6EDEC: leveldb::Options::Options() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0xA5D8AC: leveldb_init_func(void*) (ha_leveldb.cc:425)
|
by 0x6148DB: ha_initialize_handlerton(st_plugin_int*) (handler.cc:658)
|
by 0x7AE3FF: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1126)
|
by 0x7AED78: plugin_init(int*, char**, int) (sql_plugin.cc:1417)
|
by 0x602B0A: init_server_components() (mysqld.cc:4746)
|
by 0x60379D: mysqld_main(int, char**) (mysqld.cc:5320)
|
by 0x5FA7E3: main (main.cc:25)
|
|
|
512 bytes in 1 blocks are still reachable in loss record 267 of 367
|
at 0x4C28B35: operator new(unsigned long) (vg_replace_malloc.c:261)
|
by 0x4E6D037: leveldb::InitDefaultEnv() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x58D3712: pthread_once (pthread_once.S:104)
|
by 0x4E6CF56: leveldb::Env::Default() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0x4E6EDEC: leveldb::Options::Options() (in /home/elenst/leveldb-1.7.0/libleveldb.so.1.7)
|
by 0xA5D8AC: leveldb_init_func(void*) (ha_leveldb.cc:425)
|
by 0x6148DB: ha_initialize_handlerton(st_plugin_int*) (handler.cc:658)
|
by 0x7AE3FF: plugin_initialize(st_plugin_int*) (sql_plugin.cc:1126)
|
by 0x7AED78: plugin_init(int*, char**, int) (sql_plugin.cc:1417)
|
by 0x602B0A: init_server_components() (mysqld.cc:4746)
|
by 0x60379D: mysqld_main(int, char**) (mysqld.cc:5320)
|
by 0x5FA7E3: main (main.cc:25)
|
|
|
LEAK SUMMARY:
|
definitely lost: 30 bytes in 3 blocks
|
indirectly lost: 0 bytes in 0 blocks
|
possibly lost: 0 bytes in 0 blocks
|
still reachable: 920 bytes in 4 blocks
|
suppressed: 67,240,557 bytes in 1,052 blocks
|
|
|
For counts of detected and suppressed errors, rerun with: -v
|
ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 312 from 312)
|
revision-id: psergey@askmonty.org-20130125200959-a7gq7phbaiajqs12
|
revno: 4519
|
branch-nick: mysql-5.6-leveldb
|