Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-11336

Enable defragmentation on 10.2 when tests pass

Details

    Description

      http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2-big/builds/154/steps/test_1/logs/stdio

      innodb.innodb_defrag_concurrent 'xtradb' w1 [ fail ]
              Test ended at 2016-11-22 05:19:14
       
      CURRENT_TEST: innodb.innodb_defrag_concurrent
      mysqltest: At line 85: query 'reap' failed: 2013: Lost connection to MySQL server during query
       
      The result from queries just before the failure was:
      < snip >
      select count(stat_value) > 0 from mysql.innodb_index_stats where table_name like '%t1%' and  stat_name  in ('n_pages_freed');
      count(stat_value) > 0
      0
      select count(stat_value) > 0 from mysql.innodb_index_stats where table_name like '%t1%' and  stat_name  in ('n_page_split');
      count(stat_value) > 0
      1
      select count(stat_value) > 0 from mysql.innodb_index_stats where table_name like '%t1%' and  stat_name  in ('n_leaf_pages_defrag');
      count(stat_value) > 0
      1
      connection con1;
      optimize table t1;;
      connection default;
      INSERT INTO t1 VALUES (400000, REPEAT('A', 256),300000);;
      connection con2;
      INSERT INTO t1 VALUES (500000, REPEAT('A', 256),400000);;
      connection con3;
      DELETE FROM t1 where a between 1 and 100;;
      connection con4;
      UPDATE t1 SET c = c + 1 where c between 2000 and 8000;;
      connection con1;
      

      Version: '10.2.3-MariaDB-debug-log'  socket: '/mnt/buildbot/build/mariadb-10.2.3/mysql-test/var/tmp/1/mysqld.1.sock'  port: 16000  Source distribution
      ########################################
      DEADLOCK of threads detected!
      rw-lock 0xb0cd5854 --Thread 2848327488 has waited at btr0cur.cc line 6332 for 0.00 seconds the semaphore:
      S-lock on RW-latch at 0xb0cd5854 created in file buf0buf.cc line 1504
      a writer (thread id 2792340288) has reserved it in mode  exclusive
      number of readers 0, waiters flag 1, lock_word: dfffffe4
      Last time read locked in file btr0cur.cc line 6332
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.2.3/storage/innobase/btr/btr0cur.cc line 1915
      Locked: thread 2792340288 file btr0cur.cc line 1915  X-LOCK
      rw-lock 0xb0cf892c --Thread 2792340288 has waited at btr0cur.cc line 1915 for 0.00 seconds the semaphore:
      X-lock (wait_ex) on RW-latch at 0xb0cf892c created in file buf0buf.cc line 1504
      a writer (thread id 2792340288) has reserved it in mode  wait exclusive
      number of readers 1, waiters flag 0, lock_word: efffffff
      Last time read locked in file btr0cur.cc line 6332
      Last time write locked in file /home/buildbot/buildbot/build/mariadb-10.2.3/storage/innobase/btr/btr0btr.cc line 174
      Locked: thread 2848327488 file btr0cur.cc line 6332  S-LOCK
      Locked: thread 2848327488 file btr0cur.cc line 6332  S-LOCK
      2016-11-22  5:19:04 2792340288 [ERROR] [FATAL] InnoDB: ######################################## Deadlock Detected!
      2016-11-22 05:19:04 0xa66fbb40  InnoDB: Assertion failure in thread 2792340288 in file ut0ut.cc line 949
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      161122  5:19:04 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      

      According to cross-reference, it only started happening on 10.2 tree recently (this was the first occurrence: http://buildbot.askmonty.org/buildbot/builders/kvm-fulltest2-big/builds/118)

      But it happened before on development trees – bb-10.2-jan-wlad, bb-10.2-svoj, bb-10.2-jan.

      Attachments

        Issue Links

          Activity

            BTR_INTENTION_BOTH is not available outside of btr0cur.cc but this seams to work fine with X-lock on dict_index_t::lock. I will see if gis is supported (do not very well know this code).

            jplindst Jan Lindström (Inactive) added a comment - BTR_INTENTION_BOTH is not available outside of btr0cur.cc but this seams to work fine with X-lock on dict_index_t::lock. I will see if gis is supported (do not very well know this code).

            I do not want to add GIS-tests on this commit as they need some additional work and
            we do not defragment GIS-indexes (this fact is tested).

            https://github.com/MariaDB/server/commit/8610156a337184e6bb7bd64467a8f69ae7235985

            jplindst Jan Lindström (Inactive) added a comment - I do not want to add GIS-tests on this commit as they need some additional work and we do not defragment GIS-indexes (this fact is tested). https://github.com/MariaDB/server/commit/8610156a337184e6bb7bd64467a8f69ae7235985

            The code change looks OK. But, please address my comments regarding tests and especially source code comments.

            I think that we must document the WL#6326 and WL#6363 rules in the source code.
            Currently the comment at the start of btr0btr.cc is conflicting with WL#6326.

            marko Marko Mäkelä added a comment - The code change looks OK. But, please address my comments regarding tests and especially source code comments. I think that we must document the WL#6326 and WL#6363 rules in the source code. Currently the comment at the start of btr0btr.cc is conflicting with WL#6326.

            commit a4fa940bad7f3a2c4afff776977ad5e46dfe95d3
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Thu Oct 12 12:56:20 2017 +0300

            MDEV-11336: Enable defragmentation on 10.2 when tests pass

            Problem was that we could take page latches on different
            order than wat is entitled with SX-lock. To follow the
            latching order defined in WL#6326, acquire index->lock X-latch.
            This entitles us to acquire page latches in any order for the index.

            btr0btr.cc
            Document latch rules before and after MariaDB 10.2.2

            sync0rw.cc
            Document latch compatibility rules better.

            btr_defragment_merge_pages
            Fix parameter value.

            btr_defragment_thread
            Acquire X-lock to dict_index_t::lock before restoring
            cursor position and continuing defragmentation.

            ha_innobase::optimize
            Restore defragment feature.

            Testing
            Add GIS-index and FT-index to table being defragmented.

            Defragmentation is not done to GIS-indexes and FT auxiliary
            tables.

            jplindst Jan Lindström (Inactive) added a comment - commit a4fa940bad7f3a2c4afff776977ad5e46dfe95d3 Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Thu Oct 12 12:56:20 2017 +0300 MDEV-11336 : Enable defragmentation on 10.2 when tests pass Problem was that we could take page latches on different order than wat is entitled with SX-lock. To follow the latching order defined in WL#6326, acquire index->lock X-latch. This entitles us to acquire page latches in any order for the index. btr0btr.cc Document latch rules before and after MariaDB 10.2.2 sync0rw.cc Document latch compatibility rules better. btr_defragment_merge_pages Fix parameter value. btr_defragment_thread Acquire X-lock to dict_index_t::lock before restoring cursor position and continuing defragmentation. ha_innobase::optimize Restore defragment feature. Testing Add GIS-index and FT-index to table being defragmented. Defragmentation is not done to GIS-indexes and FT auxiliary tables.

            greenman Can you document that from MariaDB 10.2.11 defragmentation feature is again available.

            jplindst Jan Lindström (Inactive) added a comment - greenman Can you document that from MariaDB 10.2.11 defragmentation feature is again available.

            People

              jplindst Jan Lindström (Inactive)
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.