[#MDEV-12425] MariaRocks: Tests require too much disk space

Comment by Sergei Petrunia [ 2017-04-02 ]

- MyRocks
- Vanilla 10.2

Comment by Sergei Petrunia [ 2017-04-02 ]

That is, MyRocks can take up to 2.5G, while vanilla 10.2 tests use at most 550M.
Will now try to identify the biggest disk hog tests and mark them as "big tests".

Comment by Sergei Petrunia [ 2017-04-02 ]

Running MyRocks tests with --parallel=1 and checking disk space usage after each test, I get the following (note that it's not an exact metric because tests typically delete their data at the end I only see the left overs):

935120	rocksdb.bulk_load

598700	rocksdb.add_index_inplace_sstfilewriter

451032	rocksdb.rpl_statement_not_found

450768	rocksdb.rpl_statement

450732	rocksdb.rpl_savepoint

450716	rocksdb.rpl_row_stats

450700	rocksdb.rpl_row_rocksdb

348680	rocksdb.type_blob

346756	rocksdb.type_text

340104	rocksdb.rocksdb_range2

339500	rocksdb_sys_vars.all_vars

...

Comment by Sergei Petrunia [ 2017-04-02 ]

Scripts to run each test individually and watch its disk usage

~/mariadb-10.2/mysql-test$ cat watch-size.sh

#!/bin/bash

sleep 2

while true ; do

  du -s var/ >> var-size.log 2>/dev/null

  sleep 1

done

~/mariadb-10.2/mysql-test$ cat run-all.sh

#!/bin/bash

ls ../storage/rocksdb/mysql-test/rocksdb/t/*test |

while read a ; do

  testname=`basename $a`;

  testname=${testname/.test/}

  echo "TEST: $testname" >> var-size.log

  ./watch-size.sh &

  ./mysql-test-run rocksdb.$testname;

  du -s var/ >> var-size.log

  kill `jobs -p`;

  sleep 1

done

Comment by Sergei Petrunia [ 2017-04-02 ]

Conversion of script output to SQL is not fully automated...
For "rocksdb" test suite:

select

  testname, max(size) as MAX_DISK_SIZE

from var_logs

group by testname

order by MAX_DISK_SIZE desc;

+-------------------------------------------+---------------+

| testname                                  | MAX_DISK_SIZE |

+-------------------------------------------+---------------+

| bulk_load                                 |       2183400 |

| add_index_inplace_sstfilewriter           |       1298452 |

| rpl_statement_not_found                   |        239092 |

| rpl_statement                             |        238964 |

| rpl_row_rocksdb                           |        238848 |

| cardinality                               |        238832 |

| rpl_row_stats                             |        226312 |

| rpl_savepoint                             |        226312 |

| col_opt_not_null                          |        210120 |

| rocksdb                                   |        200084 |

| checkpoint                                |        198888 |

| deadlock                                  |        169684 |

| locking_issues                            |        140672 |

| statistics                                |        138736 |

| partition                                 |        138676 |

| col_opt_null                              |        138084 |

| autoinc_vars_thread_2                     |        136968 |

| concurrent_alter                          |        133460 |

| drop_table2                               |        132272 |

| commit_in_the_middle_ddl                  |        132060 |

| type_char_indexes_collation               |        131156 |

| rocksdb_datadir                           |        130732 |

| records_in_range                          |        128240 |

| rocksdb_checksums                         |        128116 |

| rocksdb_range2                            |        127484 |

| no_merge_sort                             |        127348 |

...

Comment by Sergei Petrunia [ 2017-04-02 ]

Marked bulk_load and add_index_inplace_sstfilewriter as big tests.

Comment by Sergei Petrunia [ 2017-04-04 ]

Closing as buildbot doesn't report "out of disk space" issues after this change.

[MDEV-12425] MariaRocks: Tests require too much disk space Created: 2017-04-02 Updated: 2017-04-04 Resolved: 2017-04-04
Status:	Closed
Project:	MariaDB Server
Component/s:	Storage Engine - RocksDB
Affects Version/s:	None
Fix Version/s:	N/A

[MDEV-12425] MariaRocks: Tests require too much disk space Created: 2017-04-02 Updated: 2017-04-04 Resolved: 2017-04-04