Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11101

storage/tokudb uses much more space than other engines

Details

    • Bug
    • Status: Closed (View Workflow)
    • Minor
    • Resolution: Won't Fix
    • 5.5(EOL), 10.0(EOL), 10.1(EOL), 10.2(EOL)
    • N/A
    • None

    Description

      This is from the 10.2 MariaDB branch. TokuDB uses 161M, no other engine uses more than 20M. Can we apply compression to its impact on the github repo and my time to download it?

      --- du -hs * | grep M
      1.1M    client
      1.2M    debian
      3.0M    extra
      1.5M    include
      3.3M    libmariadb
      171M    mysql-test
      1.7M    mysys
      6.9M    pcre
      2.7M    plugin
      1.2M    scripts
      18M     sql
      2.9M    sql-bench
      248M    storage
      16M     strings
      

      --- du -hs storage/* | grep M
       
      12M     connect
      13M     innobase
      3.5M    maria
      18M     mroonga
      1.6M    myisam
      1.7M    perfschema
      19M     rocksdb
      5.9M    spider
      162M    tokudb
      11M     xtradb
      

      --- du -hs storage/tokudb/* | grep M
       
      143M    mysql-test
      18M     PerconaFT
      

      --- du -hs storage/tokudb/PerconaFT/* | grep M
       
      3.9M    PerconaFT/ft
      5.1M    PerconaFT/src
      6.2M    PerconaFT/third_party
      

      --- du -hs storage/tokudb/mysql-test/* | grep M
       
      98M     mysql-test/tokudb
      18M     mysql-test/tokudb_alter_table
      13M     mysql-test/tokudb_bugs
      14M     mysql-test/tokudb_parts
      

      Attachments

        Issue Links

          Activity

            Worst 50 offenders from mysql-test. Number in first column is 1k blocks. Largest result file is 30M, many others are multi-MB. This wastes time, network and disk space for everyone using the MariaDB branch.

            find mysql-test/ -type f -exec ls -s {} \; | sort -rnk 1,1 | head -50
            30280 mysql-test/tokudb/r/savepoint-5.result
            6248 mysql-test/tokudb/t/change_column_all_1000_10.test
            6220 mysql-test/tokudb/r/change_column_all_1000_10.result
            4772 mysql-test/tokudb/t/change_column_all_1000_1.test
            4744 mysql-test/tokudb/r/change_column_all_1000_1.result
            3692 mysql-test/tokudb_bugs/r/db739_upsert.result
            3692 mysql-test/tokudb_bugs/r/db739_replace.result
            3692 mysql-test/tokudb_bugs/r/db739_insert.result
            3440 mysql-test/tokudb/r/bulk-fetch.result
            3440 mysql-test/tokudb/r/bulk-fetch2.result
            3196 mysql-test/tokudb_parts/r/partition_alter4_tokudb.result
            3056 mysql-test/tokudb/r/hotindex-del-slow.result
            3056 mysql-test/tokudb/r/hotindex-del-fast.result
            1620 mysql-test/tokudb/t/fast_update_blobs_fixed_varchar.test
            1612 mysql-test/tokudb/t/fast_update_blobs_with_varchar.test
            1608 mysql-test/tokudb/r/fast_update_blobs_fixed_varchar.result
            1600 mysql-test/tokudb/r/fast_update_blobs_with_varchar.result
            1524 mysql-test/tokudb_alter_table/r/hcad_all_add.result
            1524 mysql-test/tokudb_alter_table/r/hcad_all_add3.result
            1524 mysql-test/tokudb_alter_table/r/hcad_all_add2.result
            1520 mysql-test/tokudb_alter_table/t/hcad_all_add.test
            1520 mysql-test/tokudb_alter_table/t/hcad_all_add3.test
            1520 mysql-test/tokudb_alter_table/t/hcad_all_add2.test
            1476 mysql-test/tokudb/r/hotindex-insert-1.result
            1476 mysql-test/tokudb/r/hotindex-insert-0.result
            1472 mysql-test/tokudb/r/hotindex-update-1.result
            1472 mysql-test/tokudb/r/hotindex-update-0.result
            1368 mysql-test/tokudb/t/bulk-fetch.test
            1368 mysql-test/tokudb/t/bulk-fetch2.test
            1360 mysql-test/tokudb/r/fast_update_varchar.result
            1356 mysql-test/tokudb/r/hotindex-del-1.result
            1356 mysql-test/tokudb/r/hotindex-del-0.result
            1296 mysql-test/tokudb_parts/r/partition_alter1_2_tokudb.result
            1168 mysql-test/tokudb/t/fast_update_varchar.test
            1132 mysql-test/tokudb_parts/r/partition_basic_tokudb.result
            1064 mysql-test/tokudb_alter_table/r/hcad_all_drop.result
            1060 mysql-test/tokudb_alter_table/t/hcad_all_drop.test
            1024 mysql-test/tokudb_parts/r/partition_alter1_1_2_tokudb.result
            960 mysql-test/tokudb/t/change_column_bin.test
            944 mysql-test/tokudb/t/change_column_char.test
            888 mysql-test/tokudb/r/change_column_bin.result
            872 mysql-test/tokudb/r/change_column_char.result
            856 mysql-test/tokudb/t/fast_update_blobs.test
            848 mysql-test/tokudb/r/fast_update_blobs.result
            752 mysql-test/tokudb_parts/r/partition_max_parts_inv_tokudb.result
            736 mysql-test/tokudb/r/type_temporal_fractional.result
            728 mysql-test/tokudb_parts/r/partition_alter2_2_2_tokudb.result
            724 mysql-test/tokudb_parts/r/partition_alter2_2_1_tokudb.result
            724 mysql-test/tokudb_parts/r/partition_alter2_1_2_tokudb.result
            724 mysql-test/tokudb_parts/r/partition_alter2_1_1_tokudb.result

            mdcallag Mark Callaghan added a comment - Worst 50 offenders from mysql-test. Number in first column is 1k blocks. Largest result file is 30M, many others are multi-MB. This wastes time, network and disk space for everyone using the MariaDB branch. find mysql-test/ -type f -exec ls -s {} \; | sort -rnk 1,1 | head -50 30280 mysql-test/tokudb/r/savepoint-5.result 6248 mysql-test/tokudb/t/change_column_all_1000_10.test 6220 mysql-test/tokudb/r/change_column_all_1000_10.result 4772 mysql-test/tokudb/t/change_column_all_1000_1.test 4744 mysql-test/tokudb/r/change_column_all_1000_1.result 3692 mysql-test/tokudb_bugs/r/db739_upsert.result 3692 mysql-test/tokudb_bugs/r/db739_replace.result 3692 mysql-test/tokudb_bugs/r/db739_insert.result 3440 mysql-test/tokudb/r/bulk-fetch.result 3440 mysql-test/tokudb/r/bulk-fetch2.result 3196 mysql-test/tokudb_parts/r/partition_alter4_tokudb.result 3056 mysql-test/tokudb/r/hotindex-del-slow.result 3056 mysql-test/tokudb/r/hotindex-del-fast.result 1620 mysql-test/tokudb/t/fast_update_blobs_fixed_varchar.test 1612 mysql-test/tokudb/t/fast_update_blobs_with_varchar.test 1608 mysql-test/tokudb/r/fast_update_blobs_fixed_varchar.result 1600 mysql-test/tokudb/r/fast_update_blobs_with_varchar.result 1524 mysql-test/tokudb_alter_table/r/hcad_all_add.result 1524 mysql-test/tokudb_alter_table/r/hcad_all_add3.result 1524 mysql-test/tokudb_alter_table/r/hcad_all_add2.result 1520 mysql-test/tokudb_alter_table/t/hcad_all_add.test 1520 mysql-test/tokudb_alter_table/t/hcad_all_add3.test 1520 mysql-test/tokudb_alter_table/t/hcad_all_add2.test 1476 mysql-test/tokudb/r/hotindex-insert-1.result 1476 mysql-test/tokudb/r/hotindex-insert-0.result 1472 mysql-test/tokudb/r/hotindex-update-1.result 1472 mysql-test/tokudb/r/hotindex-update-0.result 1368 mysql-test/tokudb/t/bulk-fetch.test 1368 mysql-test/tokudb/t/bulk-fetch2.test 1360 mysql-test/tokudb/r/fast_update_varchar.result 1356 mysql-test/tokudb/r/hotindex-del-1.result 1356 mysql-test/tokudb/r/hotindex-del-0.result 1296 mysql-test/tokudb_parts/r/partition_alter1_2_tokudb.result 1168 mysql-test/tokudb/t/fast_update_varchar.test 1132 mysql-test/tokudb_parts/r/partition_basic_tokudb.result 1064 mysql-test/tokudb_alter_table/r/hcad_all_drop.result 1060 mysql-test/tokudb_alter_table/t/hcad_all_drop.test 1024 mysql-test/tokudb_parts/r/partition_alter1_1_2_tokudb.result 960 mysql-test/tokudb/t/change_column_bin.test 944 mysql-test/tokudb/t/change_column_char.test 888 mysql-test/tokudb/r/change_column_bin.result 872 mysql-test/tokudb/r/change_column_char.result 856 mysql-test/tokudb/t/fast_update_blobs.test 848 mysql-test/tokudb/r/fast_update_blobs.result 752 mysql-test/tokudb_parts/r/partition_max_parts_inv_tokudb.result 736 mysql-test/tokudb/r/type_temporal_fractional.result 728 mysql-test/tokudb_parts/r/partition_alter2_2_2_tokudb.result 724 mysql-test/tokudb_parts/r/partition_alter2_2_1_tokudb.result 724 mysql-test/tokudb_parts/r/partition_alter2_1_2_tokudb.result 724 mysql-test/tokudb_parts/r/partition_alter2_1_1_tokudb.result

            For what it is worth, TokuDB was disabled (the source code is dead weight) in MariaDB Server 10.5 and removed in 10.6 (MDEV-19780). But, I do not think that it helped reduce the size of the repository.

            marko Marko Mäkelä added a comment - For what it is worth, TokuDB was disabled (the source code is dead weight) in MariaDB Server 10.5 and removed in 10.6 ( MDEV-19780 ). But, I do not think that it helped reduce the size of the repository.

            People

              Unassigned Unassigned
              mdcallag Mark Callaghan
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.