Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-25447

MyRocks use "tmpdir" rather than using "rocksdb_tmpdir" for DDL operation

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Incomplete
    • 10.2(EOL), 10.3(EOL), 10.4(EOL), 10.5
    • N/A
    • None

    Description

      While converting storage engine from InnoDB to ROCKSDB, even after changing rocksdb_tmpdir= <new_path_location> ,its using tmpdir = /tmp location.

       
      For example :-
      Server version:         10.5.9-6-MariaDB-enterprise-log MariaDB Enterprise Server
       
      MariaDB [dev]> show session variables like '%tmpdir%';
      +-------------------+----------------+
      | Variable_name | Value |
      +-------------------+----------------+
      | innodb_tmpdir | /tmp/mysql_tmp |
      | rocksdb_tmpdir | /tmp/mysql_tmp |
      | slave_load_tmpdir | /tmp |
      | tmpdir | /tmp |
      +-------------------+----------------+
       
      MariaDB [dev]> alter table tmp engine=ROCKSDB;
      Query OK, 1200000 rows affected (3.067 sec)
      Records: 1200000 Duplicates: 0 Warnings: 0
       
      [root@master ~]# lsof +L1 /tmp
       
      mariadbd 1265 mysql 33u REG 253,0 67108864 0 16802192 /tmp/myrocksjk5rpG (deleted)
      
      

      Attachments

        Issue Links

          Activity

            serg

            Why every engine wants to use it's own tmpdir? And even as a session variable.

            Is this a rhetorical question or acknowledgment of ones' ignorance?

            I assume there must be some solid reason as different people seem to have specifically bothered about this: both MyRocks developers (see https://github.com/facebook/mysql-5.6/issues/455) and the customer that has filed the associated support ticket.

            At least, InnoDB checks that the user has FILE privilege, RocksDB doesn't do even that.

            I agree that a request to perform permission checks is reasonable. The checking function should probably be at the SQL layer.

            I think there should be only one tmpdir, in the server. We can make it modifiable, though, as that seems to be the reason for the proliferation of per-engine tmpdirs.

            Besides tmpdirs, the engines allow one to specify where the data files are placed: see innodb_data_home_dir, rocksdb_datadir, rocksdb_waldir.

            I assume that performance dictates that engine's tmpdir is located on the same volume as its datadir.

            Are we going to reject the engine's right to use its own directories?

            psergei Sergei Petrunia added a comment - serg Why every engine wants to use it's own tmpdir? And even as a session variable. Is this a rhetorical question or acknowledgment of ones' ignorance? I assume there must be some solid reason as different people seem to have specifically bothered about this: both MyRocks developers (see https://github.com/facebook/mysql-5.6/issues/455 ) and the customer that has filed the associated support ticket. At least, InnoDB checks that the user has FILE privilege, RocksDB doesn't do even that. I agree that a request to perform permission checks is reasonable. The checking function should probably be at the SQL layer. I think there should be only one tmpdir, in the server. We can make it modifiable, though, as that seems to be the reason for the proliferation of per-engine tmpdirs. Besides tmpdirs, the engines allow one to specify where the data files are placed: see innodb_data_home_dir, rocksdb_datadir, rocksdb_waldir. I assume that performance dictates that engine's tmpdir is located on the same volume as its datadir. Are we going to reject the engine's right to use its own directories?

            Acknowledgement. I really don't know why and want to understand it. My guess is (see above) that it's because @@tmpdir is read-only and engine developers wanted to be able to modify it. The original bug description seems to imply the same.

            I assume that performance dictates that engine's tmpdir is located on the same volume as its datadir.

            1. if one wants to create a new "temporary" table file and then rename it to the non-temporary table name — then yes, but those files are not created in the tmpdir.
            2. otherwise for performance it's better to have them on a separate volume

            Are we going to reject the engine's right to use its own directories?

            No, we have no way of doing it. The engine can do anything it wants, even when it's wrong or bad for the user experience. But we don't have to encourage that. So the question is — is there a good reason for engines to have their own tmpdirs? If yes — we should help them to do it. If not — we should help them not to.

            serg Sergei Golubchik added a comment - Acknowledgement. I really don't know why and want to understand it. My guess is (see above) that it's because @@tmpdir is read-only and engine developers wanted to be able to modify it. The original bug description seems to imply the same. I assume that performance dictates that engine's tmpdir is located on the same volume as its datadir. 1. if one wants to create a new "temporary" table file and then rename it to the non-temporary table name — then yes, but those files are not created in the tmpdir. 2. otherwise for performance it's better to have them on a separate volume Are we going to reject the engine's right to use its own directories? No, we have no way of doing it. The engine can do anything it wants, even when it's wrong or bad for the user experience. But we don't have to encourage that. So the question is — is there a good reason for engines to have their own tmpdirs? If yes — we should help them to do it. If not — we should help them not to.

            pramod.mahto@mariadb.com, why would anyone want to change rocksdb_tmpdir? What is the point of it, what benefits does it have?

            serg Sergei Golubchik added a comment - pramod.mahto@mariadb.com , why would anyone want to change rocksdb_tmpdir? What is the point of it, what benefits does it have?

            Note to self: study what tmp files myrocks are created for:
            1. for injection into the data directory
            2. for just temporary files.

            and then the engine should create the files eitehr at the datadir (for #1) or at the temp.dir (for #2).

            Also, for temporary files, one might want to change tmpdir for one specific query (to handle very big queries). This can be fixed by making tmpdir writable (currently, tmpdir is not writable, innodb_tmpdir is writable (with some security checks made), rocksdb_tmpdir is writable (without any checks)).

            psergei Sergei Petrunia added a comment - Note to self: study what tmp files myrocks are created for: 1. for injection into the data directory 2. for just temporary files. and then the engine should create the files eitehr at the datadir (for #1) or at the temp.dir (for #2). Also, for temporary files, one might want to change tmpdir for one specific query (to handle very big queries). This can be fixed by making tmpdir writable (currently, tmpdir is not writable, innodb_tmpdir is writable (with some security checks made), rocksdb_tmpdir is writable (without any checks)).

            moved the feature to MDEV-28485

            serg Sergei Golubchik added a comment - moved the feature to MDEV-28485

            People

              pramod.mahto@mariadb.com Pramod Mahto
              pramod.mahto@mariadb.com Pramod Mahto
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.