Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8164

Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing or alike

Details

    Description

      enc2.opt

      --plugin-load-add=file_key_management.so --file_key_management_filename=/home/elenst/git/10.1/mysql-test/std_data/keys.txt

      enc2.test

      --source include/have_innodb.inc
      SET default_storage_engine = InnoDB;
       
      CREATE TABLE t1 (pk INT PRIMARY KEY, c VARCHAR(256));
      SET GLOBAL innodb_encrypt_tables = ON;
      SET GLOBAL innodb_encryption_threads = 1;
      CREATE TABLE t2 AS SELECT * FROM t1;
      SET GLOBAL innodb_encryption_rotation_iops = 100;
      SET GLOBAL innodb_encrypt_tables = OFF;
      CREATE OR REPLACE TABLE t2 AS SELECT * FROM t1;

      Run with --repeat=N (depends on the machine, e.g. it fails for me on 1st or 2nd attempt on my local box, but on ~10th attempt or so on perro).

      Stack trace from bb-10.1-serg commit 0dcce2bf + MDEV-8045 patch

      #3  <signal handler called>
      #4  0x00007f2f0cab113a in pfs_mutex_enter_func (mutex=0x78, file_name=0x7f2f0cccaf58 "storage/innobase/fil/fil0crypt.cc", line=972) at storage/innobase/include/sync0sync.ic:245
      #5  0x00007f2f0cab389f in fil_crypt_is_closing (space=5) at storage/innobase/fil/fil0crypt.cc:972
      #6  0x00007f2f0cab4e91 in fil_crypt_rotate_page (key_state=0x7f2efd7fedb0, state=0x7f2efd7fedc0) at storage/innobase/fil/fil0crypt.cc:1759
      #7  0x00007f2f0cab541d in fil_crypt_rotate_pages (key_state=0x7f2efd7fedb0, state=0x7f2efd7fedc0) at storage/innobase/fil/fil0crypt.cc:1926
      #8  0x00007f2f0cab5af3 in fil_crypt_thread (arg=0x0) at storage/innobase/fil/fil0crypt.cc:2143
      #9  0x00007f2f15776b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
      #10 0x00007f2f1380b95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

      You might get a slighly different stack trace or an assertion failure, it happened to me. Example:

      #3  <signal handler called>
      #4  0x00007fa6396b113a in pfs_mutex_enter_func (mutex=0x78, file_name=0x7fa6398caf58 "storage/innobase/fil/fil0crypt.cc", line=1565) at storage/innobase/include/sync0sync.ic:245
      #5  0x00007fa6396b490e in fil_crypt_find_page_to_rotate (key_state=0x7fa62a3fedb0, state=0x7fa62a3fedc0) at storage/innobase/fil/fil0crypt.cc:1565
      #6  0x00007fa6396b5b31 in fil_crypt_thread (arg=0x0) at storage/innobase/fil/fil0crypt.cc:2140
      #7  0x00007fa6422e1b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
      #8  0x00007fa64037695d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

      Attachments

        Activity

          jplindst, could you please look at it?
          This is the concurrent DROP TABLE, that I've asked you about.

          serg Sergei Golubchik added a comment - jplindst , could you please look at it? This is the concurrent DROP TABLE, that I've asked you about.

          Looks very similar to MDEV-8074. Are you sure that this still is repeatable after commit 2f25c653ade1e73aa2b1aa77af9a4898bacb2330 ? At least I could not repeat with 10.1 tree (repeat = 20).

          jplindst Jan Lindström (Inactive) added a comment - Looks very similar to MDEV-8074 . Are you sure that this still is repeatable after commit 2f25c653ade1e73aa2b1aa77af9a4898bacb2330 ? At least I could not repeat with 10.1 tree (repeat = 20).

          commit 20c23048c1d2f28942f2f99e4150a58b6545c0cd
          Author: Jan Lindström <jan.lindstrom@mariadb.com>
          Date: Sun May 17 14:14:16 2015 +0300

          MDEV-8164: Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing or alike

          Analysis: Problem was that tablespaces not encrypted might not have
          crypt_data stored on disk.

          Fixed by always creating crypt_data to memory cache of the tablespace.

          MDEV-8138: strange results from encrypt-and-grep test

          Analysis: crypt_data->type is not updated correctly on memory
          cache. This caused problem with state tranfer on
          encrypted => unencrypted => encrypted.

          Fixed by updating memory cache of crypt_data->type correctly based on
          current srv_encrypt_tables value to either CRYPT_SCHEME_1 or
          CRYPT_SCHEME_UNENCRYPTED.

          jplindst Jan Lindström (Inactive) added a comment - commit 20c23048c1d2f28942f2f99e4150a58b6545c0cd Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Sun May 17 14:14:16 2015 +0300 MDEV-8164 : Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing or alike Analysis: Problem was that tablespaces not encrypted might not have crypt_data stored on disk. Fixed by always creating crypt_data to memory cache of the tablespace. MDEV-8138 : strange results from encrypt-and-grep test Analysis: crypt_data->type is not updated correctly on memory cache. This caused problem with state tranfer on encrypted => unencrypted => encrypted. Fixed by updating memory cache of crypt_data->type correctly based on current srv_encrypt_tables value to either CRYPT_SCHEME_1 or CRYPT_SCHEME_UNENCRYPTED.

          People

            jplindst Jan Lindström (Inactive)
            elenst Elena Stepanova
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Git Integration

                Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.