Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10555

Server crashes in mysql_admin_table upon killing ANALYZE

Details

    • 10.2.5-1

    Description

      Important notes:

      • Do not add this test to the MTR suite! Create a proper one instead when the reason of the crash is known.
      • Run the test with --repeat=N
      • Do not run the test in shm, it's too fast, the chances of hitting the failure are less

      --source include/have_innodb.inc
       
      SET GLOBAL innodb_stats_persistent= ON; 
      SET GLOBAL use_stat_tables= PREFERABLY;
       
      --connect (con1,localhost,root,,)
      SET NAMES utf8;
      CREATE TABLE t1 (
      `id` int(10) unsigned NOT NULL auto_increment,
      `k` int(10) unsigned NOT NULL default '0',
      `c` char(120) NOT NULL default '',
      `pad` char(60) NOT NULL default '',
      KEY `k` (`k`),
      PRIMARY KEY (`id`)) ENGINE=InnoDB;
       
      BEGIN;
      INSERT INTO t1 VALUES (NULL,1,'foo','foo'),(NULL,2,'bar','bar');
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
      COMMIT;
       
      CREATE TABLE t2 LIKE t1;
      INSERT INTO t2 SELECT * FROM t1;
      CREATE TABLE t3 LIKE t1;
      INSERT INTO t3 SELECT * FROM t1;
      CREATE TABLE t4 LIKE t1;
      INSERT INTO t4 SELECT * FROM t1;
      CREATE TABLE t5 LIKE t1;
      INSERT INTO t5 SELECT * FROM t1;
      CREATE TABLE t6 LIKE t1;
      INSERT INTO t6 SELECT * FROM t1;
      CREATE TABLE t7 LIKE t1;
      INSERT INTO t7 SELECT * FROM t1;
      CREATE TABLE t8 LIKE t1;
      INSERT INTO t8 SELECT * FROM t1;
      CREATE TABLE t9 LIKE t1;
      INSERT INTO t9 SELECT * FROM t1;
      CREATE TABLE t10 LIKE t1;
      INSERT INTO t10 SELECT * FROM t1;
      CREATE TABLE t11 LIKE t1;
      INSERT INTO t11 SELECT * FROM t1;
      CREATE TABLE t12 LIKE t1;
      INSERT INTO t12 SELECT * FROM t1;
      CREATE TABLE t13 LIKE t1;
      INSERT INTO t13 SELECT * FROM t1;
      CREATE TABLE t14 LIKE t1;
      INSERT INTO t14 SELECT * FROM t1;
      CREATE TABLE t15 LIKE t1;
      INSERT INTO t15 SELECT * FROM t1;
      CREATE TABLE t16 LIKE t1;
      INSERT INTO t16 SELECT * FROM t1;
      CREATE TABLE t17 LIKE t1;
      INSERT INTO t17 SELECT * FROM t1;
      CREATE TABLE t18 LIKE t1;
      INSERT INTO t18 SELECT * FROM t1;
      CREATE TABLE t19 LIKE t1;
      INSERT INTO t19 SELECT * FROM t1;
      CREATE TABLE t20 LIKE t1;
      INSERT INTO t20 SELECT * FROM t1;
      --disconnect con1
       
      --connect (con1,localhost,root,,)
      #SET max_statement_time= 1;
      --let $con_id= `SELECT CONNECTION_ID()`
      send ANALYZE TABLE t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20;
       
      --connection default
      sleep 0.5;
      eval KILL QUERY $con_id;
       
      --connection con1
      --reap
      

      10.2 f2fe65106f731df48072a16eea3cda0a903d7fe5

      #3  <signal handler called>
      #4  0x00007fb00c0b984a in mysql_admin_table(THD *, TABLE_LIST *, HA_CHECK_OPT *, const char *, thr_lock_type, bool, bool, uint, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *), struct {...}, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *)) (thd=0x7fafe3816070, tables=0x7fafe3864200, check_opt=0x7fafe381aa08, operator_name=0x7fb00ca1ab57 "analyze", lock_type=TL_READ_NO_INSERT, open_for_modify=true, repair_table_use_frm=false, extra_open_options=0, prepare_func=0x0, operator_func=(int (handler::*)(handler * const, THD *, HA_CHECK_OPT *)) 0x7fb00c1d68b8 <handler::ha_analyze(THD*, st_ha_check_opt*)>, view_operator_func=0x0) at /data/src/10.2/sql/sql_admin.cc:811
      #5  0x00007fb00c0bb611 in Sql_cmd_analyze_table::execute (this=0x7fafe397ff80, thd=0x7fafe3816070) at /data/src/10.2/sql/sql_admin.cc:1284
      #6  0x00007fb00bf6e7a4 in mysql_execute_command (thd=0x7fafe3816070) at /data/src/10.2/sql/sql_parse.cc:6165
      #7  0x00007fb00bf73080 in mysql_parse (thd=0x7fafe3816070, rawbuf=0x7fafe3864088 "ANALYZE TABLE t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20", length=103, parser_state=0x7fb00d032dd0, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7839
      #8  0x00007fb00bf60c46 in dispatch_command (command=COM_QUERY, thd=0x7fafe3816070, packet=0x7fafe3858071 "", packet_length=103, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1799
      #9  0x00007fb00bf5f620 in do_command (thd=0x7fafe3816070) at /data/src/10.2/sql/sql_parse.cc:1359
      #10 0x00007fb00c0a73d2 in do_handle_one_connection (connect=0x7fb008c72130) at /data/src/10.2/sql/sql_connect.cc:1354
      #11 0x00007fb00c0a715f in handle_one_connection (arg=0x7fb008c72130) at /data/src/10.2/sql/sql_connect.cc:1260
      #12 0x00007fb00c3e5d64 in pfs_spawn_thread (arg=0x7faff63edef0) at /data/src/10.2/storage/perfschema/pfs.cc:1862
      #13 0x00007fb00b5930a4 in start_thread (arg=0x7fb00d034300) at pthread_create.c:309
      #14 0x00007fb00954087d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      

      The problem was apparently introduced by this commit:

      commit 86975e0492c03535271aea67e0b5e7e8fd0b1b78
      Author: Oleksandr Byelkin <sanja@mariadb.com>
      Date:   Mon Aug 1 19:24:55 2016 +0200
       
          MDEV-7901: re-implement analyze table for low impact
          
          Table before collecting engine independent statistics now is reopened in read mode,
          InnoDB allow write operations in this case.
      

      Attachments

        Issue Links

          Activity

            elenst Elena Stepanova created issue -
            elenst Elena Stepanova made changes -
            Field Original Value New Value
            elenst Elena Stepanova made changes -
            Fix Version/s 10.2 [ 14601 ]
            Assignee Elena Stepanova [ elenst ] Oleksandr Byelkin [ sanja ]
            Description {noformat}
            160809 16:56:50 [ERROR] mysqld got exception 0xc0000005 ;
            This could be because you hit a bug. It is also possible that this binary
            or one of the libraries it was linked against is corrupt, improperly built,
            or misconfigured. This error can also be caused by malfunctioning hardware.

            To report this bug, see https://mariadb.com/kb/en/reporting-bugs

            We will try our best to scrape up some info that will hopefully help
            diagnose the problem, but since we have already crashed,
            something is definitely wrong and this may fail.

            Server version: 10.2.2-MariaDB-debug
            key_buffer_size=1048576
            read_buffer_size=131072
            max_used_connections=7
            max_threads=1001
            thread_count=6
            It is possible that mysqld could use up to
            key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 388165 K bytes of memory
            Hope that's ok; if not, decrease some variables in the equation.

            Thread pointer: 0x0xf1899920e8
            Attempting backtrace. You can use the following information to find out
            where mysqld died. If you see no messages after this, something went
            terribly wrong...
            {noformat}
            {noformat}
            E:\buildbot\rqg/runall.pl --no-mask --seed=time --threads=5 --duration=600 --queries=100M --reporters=QueryTimeout,Backtrace,ErrorLog,Deadlock,Shutdown --redefine=conf/mariadb/redefine_random_keys.yy --redefine=conf/mariadb/redefine_set_session_vars.yy --validators=TransformerNoComparator --transformers=ConvertSubqueriesToViews,DisableOptimizations,EnableOptimizations,ExecuteAsCTE,ExecuteAsInsertSelect,ExecuteAsPreparedThrice,ExecuteAsSelectItem,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,NullIf,OrderBy,StraightJoin --grammar=conf/optimizer/optimizer_access_exp.yy --gendata=conf/optimizer/range_access.zz --mtr-build-thread=300 --basedir1=E:\buildbot\bbwin1\win-rqg-se\build\..\build-debug --vardir1=E:\buildbot\bbwin1\win-rqg-se\build\..\..\..\vardirs\10.2-2617\optim-crash-tests/current1_1
            mysqld.exe!mysql_admin_table()[sql_admin.cc:799]
            mysqld.exe!Sql_cmd_analyze_table::execute()[sql_admin.cc:1272]
            mysqld.exe!mysql_execute_command()[sql_parse.cc:6082]
            mysqld.exe!Prepared_statement::execute()[sql_prepare.cc:4277]
            mysqld.exe!Prepared_statement::execute_loop()[sql_prepare.cc:3908]
            mysqld.exe!mysql_sql_stmt_execute()[sql_prepare.cc:3037]
            mysqld.exe!mysql_execute_command()[sql_parse.cc:3365]
            mysqld.exe!mysql_parse()[sql_parse.cc:7742]
            mysqld.exe!dispatch_command()[sql_parse.cc:1793]
            mysqld.exe!do_command()[sql_parse.cc:1353]
            mysqld.exe!threadpool_process_request()[threadpool_common.cc:252]
            mysqld.exe!io_completion_callback()[threadpool_win.cc:462]
            KERNEL32.DLL!VirtualUnlock()
            ntdll.dll!RtlGetActiveActivationContext()
            ntdll.dll!RtlFreeUnicodeString()
            KERNEL32.DLL!BaseThreadInitThunk()
            ntdll.dll!RtlUserThreadStart()
            {noformat}
            {noformat}
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0xf18999ab98): ANALYZE TABLE a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p /* QUERY_NO 5199 CON_ID 10 */
            Connection ID (thread ID): 10
            Status: KILL_QUERY
            {noformat}
            {noformat}
            E:\buildbot\rqg/runall.pl --no-mask --seed=time --threads=5 --duration=600 --queries=100M --reporters=QueryTimeout,Backtrace,ErrorLog,Deadlock,Shutdown --redefine=conf/mariadb/redefine_random_keys.yy --redefine=conf/mariadb/redefine_set_session_vars.yy --validators=TransformerNoComparator --transformers=ConvertSubqueriesToViews,DisableOptimizations,EnableOptimizations,ExecuteAsCTE,ExecuteAsInsertSelect,ExecuteAsPreparedThrice,ExecuteAsSelectItem,ExecuteAsUnion,ExecuteAsUpdateDelete,ExecuteAsView,NullIf,OrderBy,StraightJoin --grammar=conf/optimizer/optimizer_access_exp.yy --gendata=conf/optimizer/range_access.zz --mtr-build-thread=300 --basedir1=E:\buildbot\bbwin1\win-rqg-se\build\..\build-debug --vardir1=E:\buildbot\bbwin1\win-rqg-se\build\..\..\..\vardirs\10.2-2617\optim-crash-tests/current1_1
            {noformat}
            _*Important notes:*_
            - _*Do not add this test to the MTR suite! Create a proper one instead when the reason of the crash is known.*_
            - _*Run the test with --repeat=N*_
            - _*Do not run the test in shm, it's too fast, the chances of hitting the failure are less*_

            {code:sql}
            --source include/have_innodb.inc

            SET GLOBAL innodb_stats_persistent= ON;
            SET GLOBAL use_stat_tables= PREFERABLY;

            --connect (con1,localhost,root,,)
            SET NAMES utf8;
            CREATE TABLE t1 (
            `id` int(10) unsigned NOT NULL auto_increment,
            `k` int(10) unsigned NOT NULL default '0',
            `c` char(120) NOT NULL default '',
            `pad` char(60) NOT NULL default '',
            KEY `k` (`k`),
            PRIMARY KEY (`id`)) ENGINE=InnoDB;

            BEGIN;
            INSERT INTO t1 VALUES (NULL,1,'foo','foo'),(NULL,2,'bar','bar');
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            INSERT INTO t1 SELECT NULL, k, c, pad FROM t1;
            COMMIT;

            CREATE TABLE t2 LIKE t1;
            INSERT INTO t2 SELECT * FROM t1;
            CREATE TABLE t3 LIKE t1;
            INSERT INTO t3 SELECT * FROM t1;
            CREATE TABLE t4 LIKE t1;
            INSERT INTO t4 SELECT * FROM t1;
            CREATE TABLE t5 LIKE t1;
            INSERT INTO t5 SELECT * FROM t1;
            CREATE TABLE t6 LIKE t1;
            INSERT INTO t6 SELECT * FROM t1;
            CREATE TABLE t7 LIKE t1;
            INSERT INTO t7 SELECT * FROM t1;
            CREATE TABLE t8 LIKE t1;
            INSERT INTO t8 SELECT * FROM t1;
            CREATE TABLE t9 LIKE t1;
            INSERT INTO t9 SELECT * FROM t1;
            CREATE TABLE t10 LIKE t1;
            INSERT INTO t10 SELECT * FROM t1;
            CREATE TABLE t11 LIKE t1;
            INSERT INTO t11 SELECT * FROM t1;
            CREATE TABLE t12 LIKE t1;
            INSERT INTO t12 SELECT * FROM t1;
            CREATE TABLE t13 LIKE t1;
            INSERT INTO t13 SELECT * FROM t1;
            CREATE TABLE t14 LIKE t1;
            INSERT INTO t14 SELECT * FROM t1;
            CREATE TABLE t15 LIKE t1;
            INSERT INTO t15 SELECT * FROM t1;
            CREATE TABLE t16 LIKE t1;
            INSERT INTO t16 SELECT * FROM t1;
            CREATE TABLE t17 LIKE t1;
            INSERT INTO t17 SELECT * FROM t1;
            CREATE TABLE t18 LIKE t1;
            INSERT INTO t18 SELECT * FROM t1;
            CREATE TABLE t19 LIKE t1;
            INSERT INTO t19 SELECT * FROM t1;
            CREATE TABLE t20 LIKE t1;
            INSERT INTO t20 SELECT * FROM t1;
            --disconnect con1

            --connect (con1,localhost,root,,)
            #SET max_statement_time= 1;
            --let $con_id= `SELECT CONNECTION_ID()`
            send ANALYZE TABLE t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20;

            --connection default
            sleep 0.5;
            eval KILL QUERY $con_id;

            --connection con1
            --reap
            {code}

            {noformat:title=10.2 f2fe65106f731df48072a16eea3cda0a903d7fe5}
            #3 <signal handler called>
            #4 0x00007fb00c0b984a in mysql_admin_table(THD *, TABLE_LIST *, HA_CHECK_OPT *, const char *, thr_lock_type, bool, bool, uint, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *), struct {...}, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *)) (thd=0x7fafe3816070, tables=0x7fafe3864200, check_opt=0x7fafe381aa08, operator_name=0x7fb00ca1ab57 "analyze", lock_type=TL_READ_NO_INSERT, open_for_modify=true, repair_table_use_frm=false, extra_open_options=0, prepare_func=0x0, operator_func=(int (handler::*)(handler * const, THD *, HA_CHECK_OPT *)) 0x7fb00c1d68b8 <handler::ha_analyze(THD*, st_ha_check_opt*)>, view_operator_func=0x0) at /data/src/10.2/sql/sql_admin.cc:811
            #5 0x00007fb00c0bb611 in Sql_cmd_analyze_table::execute (this=0x7fafe397ff80, thd=0x7fafe3816070) at /data/src/10.2/sql/sql_admin.cc:1284
            #6 0x00007fb00bf6e7a4 in mysql_execute_command (thd=0x7fafe3816070) at /data/src/10.2/sql/sql_parse.cc:6165
            #7 0x00007fb00bf73080 in mysql_parse (thd=0x7fafe3816070, rawbuf=0x7fafe3864088 "ANALYZE TABLE t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20", length=103, parser_state=0x7fb00d032dd0, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:7839
            #8 0x00007fb00bf60c46 in dispatch_command (command=COM_QUERY, thd=0x7fafe3816070, packet=0x7fafe3858071 "", packet_length=103, is_com_multi=false, is_next_command=false) at /data/src/10.2/sql/sql_parse.cc:1799
            #9 0x00007fb00bf5f620 in do_command (thd=0x7fafe3816070) at /data/src/10.2/sql/sql_parse.cc:1359
            #10 0x00007fb00c0a73d2 in do_handle_one_connection (connect=0x7fb008c72130) at /data/src/10.2/sql/sql_connect.cc:1354
            #11 0x00007fb00c0a715f in handle_one_connection (arg=0x7fb008c72130) at /data/src/10.2/sql/sql_connect.cc:1260
            #12 0x00007fb00c3e5d64 in pfs_spawn_thread (arg=0x7faff63edef0) at /data/src/10.2/storage/perfschema/pfs.cc:1862
            #13 0x00007fb00b5930a4 in start_thread (arg=0x7fb00d034300) at pthread_create.c:309
            #14 0x00007fb00954087d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            {noformat}

            The problem was apparently introduced by this commit:
            {noformat}
            commit 86975e0492c03535271aea67e0b5e7e8fd0b1b78
            Author: Oleksandr Byelkin <sanja@mariadb.com>
            Date: Mon Aug 1 19:24:55 2016 +0200

                MDEV-7901: re-implement analyze table for low impact
                
                Table before collecting engine independent statistics now is reopened in read mode,
                InnoDB allow write operations in this case.
            {noformat}
            Summary [Draft] Server crashes in mysql_admin_table upon killing ANALYZE Server crashes in mysql_admin_table upon killing ANALYZE
            elenst Elena Stepanova made changes -
            Labels 10.2-rc
            elenst Elena Stepanova made changes -
            Labels 10.2-rc 10.2-ga
            ratzpo Rasmus Johansson (Inactive) made changes -
            Sprint 10.2.5-1 [ 144 ]
            ratzpo Rasmus Johansson (Inactive) made changes -
            Rank Ranked lower
            sanja Oleksandr Byelkin made changes -
            Status Open [ 1 ] In Progress [ 3 ]

            Error is here. open_error is true and so table->table is NULL (not opened)

            open_error= open_only_one_table(thd, table,
            repair_table_use_frm, FALSE);
            thd->open_options&= ~extra_open_options;

            TABLE *tab= table->table;
            Field **field_ptr= tab->field;
            if (!lex->column_list)

            sanja Oleksandr Byelkin added a comment - Error is here. open_error is true and so table->table is NULL (not opened) open_error= open_only_one_table(thd, table, repair_table_use_frm, FALSE); thd->open_options&= ~extra_open_options; TABLE *tab= table->table; Field **field_ptr= tab->field; if (!lex->column_list)

            revision-id: 23d118899fe9fa030e61b6a9ac06df5210d36294 (mariadb-10.2.4-29-g23d1188)
            parent(s): 5da6bd7b959af4a9755556afedde8f34d95c679a
            committer: Oleksandr Byelkin
            timestamp: 2017-03-08 19:05:44 +0100
            message:

            MDEV-10555: Server crashes in mysql_admin_table upon killing ANALYZE

            Take into acount result of open table operation in mysql_admin_table.

            —

            sanja Oleksandr Byelkin added a comment - revision-id: 23d118899fe9fa030e61b6a9ac06df5210d36294 (mariadb-10.2.4-29-g23d1188) parent(s): 5da6bd7b959af4a9755556afedde8f34d95c679a committer: Oleksandr Byelkin timestamp: 2017-03-08 19:05:44 +0100 message: MDEV-10555 : Server crashes in mysql_admin_table upon killing ANALYZE Take into acount result of open table operation in mysql_admin_table. —
            sanja Oleksandr Byelkin made changes -
            Assignee Oleksandr Byelkin [ sanja ] Sergei Golubchik [ serg ]
            Status In Progress [ 3 ] In Review [ 10002 ]
            serg Sergei Golubchik made changes -
            Assignee Sergei Golubchik [ serg ] Oleksandr Byelkin [ sanja ]
            Status In Review [ 10002 ] Stalled [ 10000 ]
            sanja Oleksandr Byelkin made changes -
            Status Stalled [ 10000 ] In Progress [ 3 ]
            sanja Oleksandr Byelkin made changes -
            Fix Version/s 10.2.5 [ 22117 ]
            Fix Version/s 10.2 [ 14601 ]
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Closed [ 6 ]
            serg Sergei Golubchik made changes -
            Workflow MariaDB v3 [ 76652 ] MariaDB v4 [ 150758 ]

            People

              sanja Oleksandr Byelkin
              elenst Elena Stepanova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.