Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10535

ALTER TABLE causes standalone/wsrep cluster crash

Details

    Description

      Hello,

      While I was installing Cerberus Helpdesk our whole wsrep galera cluster crashed (Debian 8, 10.0.26+maria-1~jessie). Getting it back up was no problem and I ran the same installer on our standalone development environment (Ubuntu 14.04.5, 10.1.16+maria-1~trusty, no wsrep) and it caused the same issue: mariadb/mysqld crashed.

      All packages are installed using the mirror tool on the mariadb website.

      This is the query that causes a crash:

      ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id); ERROR 2013 (HY000): Lost connection to MySQL server during query
      

      This is the stack trace I get on our cluster:

      Aug 10 22:07:19 database1 mysqld: 160810 22:07:19 [ERROR] mysqld got signal 11 ;
      Aug 10 22:07:19 database1 mysqld: This could be because you hit a bug. It is also possible that this binary
      Aug 10 22:07:19 database1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Aug 10 22:07:19 database1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: We will try our best to scrape up some info that will hopefully help
      Aug 10 22:07:19 database1 mysqld: diagnose the problem, but since we have already crashed,
      Aug 10 22:07:19 database1 mysqld: something is definitely wrong and this may fail.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Server version: 10.0.26-MariaDB-1~jessie-wsrep
      Aug 10 22:07:19 database1 mysqld: key_buffer_size=134217728
      Aug 10 22:07:19 database1 mysqld: read_buffer_size=2097152
      Aug 10 22:07:19 database1 mysqld: max_used_connections=3
      Aug 10 22:07:19 database1 mysqld: max_threads=102
      Aug 10 22:07:19 database1 mysqld: thread_count=3
      Aug 10 22:07:19 database1 mysqld: It is possible that mysqld could use up to
      Aug 10 22:07:19 database1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759801 K  bytes of memory
      Aug 10 22:07:19 database1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Thread pointer: 0x0x7f7292f20008
      Aug 10 22:07:19 database1 mysqld: Attempting backtrace. You can use the following information to find out
      Aug 10 22:07:19 database1 mysqld: where mysqld died. If you see no messages after this, something went
      Aug 10 22:07:19 database1 mysqld: terribly wrong...
      Aug 10 22:07:19 database1 mysqld: stack_bottom = 0x7f72e2855e28 thread_stack 0x48000
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x3d)[0x7f72e2365e6d]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x31a)[0x7f72e1e93c2a]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f72e14d48d0]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x32ee26)[0x7f72e1c33e26]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x723e73)[0x7f72e2028e73]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2575)[0x7f72e1d8f065]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x60c)[0x7f72e1ddc08c]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x956)[0x7f72e1cf4296]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x24a)[0x7f72e1cfcc9a]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x3f8379)[0x7f72e1cfd379]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1875)[0x7f72e1cff285]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x325)[0x7f72e1cfff35]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x37b)[0x7f72e1dd837b]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f72e1dd83f0]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f72e14cd0a4]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72dfad387d]
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Trying to get some variables.
      Aug 10 22:07:19 database1 mysqld: Some pointers may be invalid and cause the dump to abort.
      Aug 10 22:07:19 database1 mysqld: Query (0x7f72a2b79020): is an invalid pointer
      Aug 10 22:07:19 database1 mysqld: Connection ID (thread ID): 159
      Aug 10 22:07:19 database1 mysqld: Status: NOT_KILLED
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Aug 10 22:07:19 database1 mysqld: information that should help you find out what is causing the crash.
      Aug 10 22:07:20 database1 mysqld_safe: Number of processes running now: 0
      Aug 10 22:07:20 database1 mysqld_safe: WSREP: not restarting wsrep node automatically
      Aug 10 22:07:20 database1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
      

      And here is the information from our standalone dev server:

      Aug 10 22:30:25 dev mysqld: 160810 22:30:25 [ERROR] mysqld got signal 11 ;
      Aug 10 22:30:25 dev mysqld: This could be because you hit a bug. It is also possible that this binary
      Aug 10 22:30:25 dev mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Aug 10 22:30:25 dev mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: We will try our best to scrape up some info that will hopefully help
      Aug 10 22:30:25 dev mysqld: diagnose the problem, but since we have already crashed,
      Aug 10 22:30:25 dev mysqld: something is definitely wrong and this may fail.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Server version: 10.1.16-MariaDB-1~trusty
      Aug 10 22:30:25 dev mysqld: key_buffer_size=16777216
      Aug 10 22:30:25 dev mysqld: read_buffer_size=131072
      Aug 10 22:30:25 dev mysqld: max_used_connections=14
      Aug 10 22:30:25 dev mysqld: max_threads=153
      Aug 10 22:30:25 dev mysqld: thread_count=7
      Aug 10 22:30:25 dev mysqld: It is possible that mysqld could use up to
      Aug 10 22:30:25 dev mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352425 K  bytes of memory
      Aug 10 22:30:25 dev mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Thread pointer: 0x0x7faa3c4b6008
      Aug 10 22:30:25 dev mysqld: Attempting backtrace. You can use the following information to find out
      Aug 10 22:30:25 dev mysqld: where mysqld died. If you see no messages after this, something went
      Aug 10 22:30:25 dev mysqld: terribly wrong...
      Aug 10 22:30:25 dev mysqld: stack_bottom = 0x7faa65467df0 thread_stack 0x30000
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dfa03028fe]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x55df9fe29e75]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7faa64039330]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x379749)[0x55df9fbf2749]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x801ebf)[0x55dfa007aebf]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1e3a)[0x55df9fd34e4a]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55df9fd7b5ca]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55df9fca8116]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55df9fcb073d]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55df9fcb3a80]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55df9fcb4239]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55df9fd7889a]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x55df9fd78a70]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7faa64031184]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7faa6355037d]
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Trying to get some variables.
      Aug 10 22:30:25 dev mysqld: Some pointers may be invalid and cause the dump to abort.
      Aug 10 22:30:25 dev mysqld: Query (0x7faa3e42c020): is an invalid pointer
      Aug 10 22:30:25 dev mysqld: Connection ID (thread ID): 127629
      Aug 10 22:30:25 dev mysqld: Status: NOT_KILLED
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Aug 10 22:30:25 dev mysqld: information that should help you find out what is causing the crash.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: We think the query pointer is invalid, but we will try to print it anyway.
      Aug 10 22:30:25 dev mysqld: Query: ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id)
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld_safe: Number of processes running now: 0
      Aug 10 22:30:25 dev mysqld_safe: mysqld restarted
      

      I was able to find the CREATE TABLE query for the ticket table:

      DROP TABLE IF EXISTS `ticket`;
      /*!40101 SET @saved_cs_client     = @@character_set_client */;
      /*!40101 SET character_set_client = utf8 */;
      CREATE TABLE `ticket` (
        `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
        `mask` varchar(32) NOT NULL DEFAULT '',
        `subject` varchar(255) NOT NULL DEFAULT '',
        `is_closed` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `is_deleted` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `group_id` int(10) unsigned NOT NULL DEFAULT '0',
        `bucket_id` int(10) unsigned NOT NULL DEFAULT '0',
        `first_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `created_date` int(10) unsigned DEFAULT NULL,
        `updated_date` int(10) unsigned DEFAULT NULL,
        `reopen_at` int(10) unsigned NOT NULL DEFAULT '0',
        `first_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
        `last_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
        `spam_score` decimal(4,4) NOT NULL DEFAULT '0.0000',
        `spam_training` varchar(1) NOT NULL DEFAULT '',
        `interesting_words` varchar(255) NOT NULL DEFAULT '',
        `is_waiting` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `last_action_code` varchar(1) NOT NULL DEFAULT 'O',
        `last_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `owner_id` int(10) unsigned NOT NULL DEFAULT '0',
        `org_id` int(10) unsigned NOT NULL DEFAULT '0',
        `num_messages` int(10) unsigned NOT NULL DEFAULT '0',
        `closed_at` int(10) unsigned NOT NULL DEFAULT '0',
        `first_outgoing_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `elapsed_response_first` int(10) unsigned NOT NULL DEFAULT '0',
        `elapsed_resolution_first` int(10) unsigned NOT NULL DEFAULT '0',
        PRIMARY KEY (`id`),
        KEY `first_message_id` (`first_message_id`),
        KEY `mask` (`mask`),
        KEY `is_waiting` (`is_waiting`),
        KEY `team_id` (`group_id`),
        KEY `created_date` (`created_date`),
        KEY `updated_date` (`updated_date`),
        KEY `first_wrote_address_id` (`first_wrote_address_id`),
        KEY `last_wrote_address_id` (`last_wrote_address_id`),
        KEY `is_closed` (`is_closed`),
        KEY `category_id` (`bucket_id`),
        KEY `due_date` (`reopen_at`),
        KEY `is_deleted` (`is_deleted`),
        KEY `last_action_code` (`last_action_code`),
        KEY `spam_score` (`spam_score`),
        KEY `last_message_id` (`last_message_id`),
        KEY `owner_id` (`owner_id`),
        KEY `org_id` (`org_id`)
      ) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
      /*!40101 SET character_set_client = @saved_cs_client */;
      

      And the whole schema can be found here: https://github.com/wgm/cerb/blob/stable/install/extras/automation/cerb.setup.schema.sql

      Memory is not an issue on those machines and I'm using InnoDB.

      Please let me know if you need any additional information!

      Regards,
      Daan

      Attachments

        1. cerberusweb_general_log.txt
          609 kB
        2. my.cnf
          4 kB
        3. other_mysql.cnf
          0.9 kB

        Issue Links

          Activity

            daan.vangorkum Daan van Gorkum added a comment - - edited

            Quick oberservation:

            While I explicity request for InnoDB during the install I see that MyISAM is used. And when I look at the code I don't really see what my InnoDB request actually does...so it might be related to MyISAM (while I don't understand why that affects standalone mode):

            # grep -R db_engine
            install/classes.php:    public static function saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass) {
            install/classes.php:                                            $value = $db_engine;
            install/templates/steps/step_config_file.tpl:<input type="hidden" name="db_engine" value="{$db_engine}">
            install/templates/steps/step_database.tpl:<select name="db_engine">
            install/templates/steps/step_database.tpl:      <option value="{$k}" {if $k==$db_engine}selected{/if}>{$engine}
            install/index.php:              @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string');
            install/index.php:              if(!empty($db_driver) && !empty($db_engine) && !empty($db_server) && !empty($db_name) && !empty($db_user)) {
            install/index.php:                              if(!in_array($db_engine, $discovered_engines)) {
            install/index.php:                                      $errors[] = sprintf("The '%s' storage engine is not enabled.", $db_engine);
            install/index.php:                      $tpl->assign('db_engine', $db_engine);
            install/index.php:                              $result = CerberusInstaller::saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass);
            install/index.php:              @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string');
            install/index.php:                      0 == strcasecmp($db_engine,APP_DB_ENGINE) &&
            install/index.php:                      $tpl->assign('db_engine', $db_engine);
            libs/devblocks/api/services/search.php:                 $db_engine_fulltext = 'MyISAM';
            libs/devblocks/api/services/search.php:                 $db_engine_fulltext = APP_DB_ENGINE_FULLTEXT;
            libs/devblocks/api/services/search.php:                 $db_engine_fulltext
            # grep -R -i innodb
            framework.config.php:define('APP_DB_ENGINE','innodb');
            install/index.php:                      'innodb' => 'InnoDB (Recommended)',
            libs/devblocks/api/services/search.php:         // InnoDB stop words
            

            I do see them altering tables with InnoDB in some migration scripts but nothing for the ticket table or some others. Still doesn't explain why this is an issue when not using wsrep.

            Daan

            daan.vangorkum Daan van Gorkum added a comment - - edited Quick oberservation: While I explicity request for InnoDB during the install I see that MyISAM is used. And when I look at the code I don't really see what my InnoDB request actually does...so it might be related to MyISAM (while I don't understand why that affects standalone mode): # grep -R db_engine install/classes.php: public static function saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass) { install/classes.php: $value = $db_engine; install/templates/steps/step_config_file.tpl:<input type="hidden" name="db_engine" value="{$db_engine}"> install/templates/steps/step_database.tpl:<select name="db_engine"> install/templates/steps/step_database.tpl: <option value="{$k}" {if $k==$db_engine}selected{/if}>{$engine} install/index.php: @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string'); install/index.php: if(!empty($db_driver) && !empty($db_engine) && !empty($db_server) && !empty($db_name) && !empty($db_user)) { install/index.php: if(!in_array($db_engine, $discovered_engines)) { install/index.php: $errors[] = sprintf("The '%s' storage engine is not enabled.", $db_engine); install/index.php: $tpl->assign('db_engine', $db_engine); install/index.php: $result = CerberusInstaller::saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass); install/index.php: @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string'); install/index.php: 0 == strcasecmp($db_engine,APP_DB_ENGINE) && install/index.php: $tpl->assign('db_engine', $db_engine); libs/devblocks/api/services/search.php: $db_engine_fulltext = 'MyISAM'; libs/devblocks/api/services/search.php: $db_engine_fulltext = APP_DB_ENGINE_FULLTEXT; libs/devblocks/api/services/search.php: $db_engine_fulltext # grep -R -i innodb framework.config.php:define('APP_DB_ENGINE','innodb'); install/index.php: 'innodb' => 'InnoDB (Recommended)', libs/devblocks/api/services/search.php: // InnoDB stop words I do see them altering tables with InnoDB in some migration scripts but nothing for the ticket table or some others. Still doesn't explain why this is an issue when not using wsrep. Daan

            daan.vangorkum,

            Does the table have any data by the time it was altered?
            Can you get the CREATE TABLE before the table was altered (those in the description and on github are obviously after the table was altered, since all changes from the ALTER statement have already been applied).

            Since it's reproducible for you in the development environment, the easy way to get everything in the right order would be to enable general_log on that server and run the installer again. If it's possible, please do so, and then attach the general log, or upload it to ftp.askmonty.org/private if it works better for you.

            Please also attach your cnf file(s).

            Thanks.

            elenst Elena Stepanova added a comment - daan.vangorkum , Does the table have any data by the time it was altered? Can you get the CREATE TABLE before the table was altered (those in the description and on github are obviously after the table was altered, since all changes from the ALTER statement have already been applied). Since it's reproducible for you in the development environment, the easy way to get everything in the right order would be to enable general_log on that server and run the installer again. If it's possible, please do so, and then attach the general log, or upload it to ftp.askmonty.org/private if it works better for you. Please also attach your cnf file(s). Thanks.
            daan.vangorkum Daan van Gorkum added a comment - - edited

            Hello,

            Ive attached cerberusweb_general_log.txt which shows the log from the beginning of the installer till the end (aka till it crashes, the ALTER TABLE query). CREATE TABLE IF NOT EXISTS ticket ( is at line 1922.
            I've also attached my.cnf and the config that is included from conf.d: other_mysql.cnf.

            I don't see any references in this log to MyISAM so I might've overlooked something yesterday.

            Let me know if you need more!

            Daan

            daan.vangorkum Daan van Gorkum added a comment - - edited Hello, Ive attached cerberusweb_general_log.txt which shows the log from the beginning of the installer till the end (aka till it crashes, the ALTER TABLE query). CREATE TABLE IF NOT EXISTS ticket ( is at line 1922. I've also attached my.cnf and the config that is included from conf.d: other_mysql.cnf . I don't see any references in this log to MyISAM so I might've overlooked something yesterday. Let me know if you need more! Daan
            elenst Elena Stepanova added a comment - - edited

            Thank you, I'm able to reproduce the crash by replaying your log.
            I'm not sure what happened with InnoDB vs MyISAM in your instance, but apparently it's not related to the crash, since it happens on an InnoDB table as well.

            Upd: in fact, it's only reproducible with InnoDB. Apparently, on your instance it also crashes with InnoDB, but somehow at some point it deflects to MyISAM, which works all right, that's how you've got the altered table which you first pasted.

            elenst Elena Stepanova added a comment - - edited Thank you, I'm able to reproduce the crash by replaying your log. I'm not sure what happened with InnoDB vs MyISAM in your instance, but apparently it's not related to the crash, since it happens on an InnoDB table as well. Upd: in fact, it's only reproducible with InnoDB. Apparently, on your instance it also crashes with InnoDB, but somehow at some point it deflects to MyISAM, which works all right, that's how you've got the altered table which you first pasted.
            elenst Elena Stepanova added a comment - - edited

            Crashes on 10.x.
            No crash on 5.5, MySQL 5.6.

            Test case

            --source include/have_innodb.inc
             
            CREATE TABLE IF NOT EXISTS ticket (
                id INT UNSIGNED NOT NULL AUTO_INCREMENT,
                mask VARCHAR(16) DEFAULT '' NOT NULL,
                subject VARCHAR(255)  DEFAULT '' NOT NULL,
                is_closed TINYINT(1) UNSIGNED DEFAULT 0 NOT NULL,
                is_deleted TINYINT(1) UNSIGNED DEFAULT 0 NOT NULL,
                team_id INT UNSIGNED DEFAULT 0 NOT NULL,
                category_id INT UNSIGNED DEFAULT 0 NOT NULL,
                first_message_id INT UNSIGNED DEFAULT 0 NOT NULL,
                created_date INT UNSIGNED,
                updated_date INT UNSIGNED,
                due_date INT UNSIGNED,
                first_wrote_address_id INT UNSIGNED NOT NULL DEFAULT 0,
                last_wrote_address_id INT UNSIGNED NOT NULL DEFAULT 0,
                spam_score DECIMAL(4,4) NOT NULL DEFAULT 0,
                spam_training VARCHAR(1) NOT NULL DEFAULT '',
                interesting_words VARCHAR(255) NOT NULL DEFAULT '',
                next_action VARCHAR(255) NOT NULL DEFAULT '',
                PRIMARY KEY (id)
            ) ENGINE=InnoDB;
             
            ALTER TABLE ticket 
                CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,
                CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,
                ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0, 
                ADD INDEX org_id (org_id);
            

            Stack trace from 10.0 commit b5fb2a685b6ec67d37033b020a8145d1aac1fc93

            #3  <signal handler called>
            #4  0x00007f2c3235f052 in innobase_need_rebuild (ha_alter_info=0x7f2c3bbfa270, altered_table=0x7f2c2689e470) at /data/src/10.0/storage/innobase/handler/handler0alter.cc:243
            #5  0x00007f2c3235fdd6 in ha_innodb::check_if_supported_inplace_alter (this=0x7f2c2687e888, altered_table=0x7f2c269f2070, ha_alter_info=0x7f2c3bbfa270) at /data/src/10.0/storage/innobase/handler/handler0alter.cc:578
            #6  0x00000000006fecfe in mysql_alter_table (thd=0x7f2c33a2e070, new_db=0x7f2c269a48e0 "test", new_name=0x0, create_info=0x7f2c3bbfb4f0, table_list=0x7f2c269a4300, alter_info=0x7f2c3bbfb460, order_num=0, order=0x0, ignore=false) at /data/src/10.0/sql/sql_table.cc:8815
            #7  0x0000000000766713 in Sql_cmd_alter_table::execute (this=0x7f2c269a4e58, thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_alter.cc:312
            #8  0x000000000064e9eb in mysql_execute_command (thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_parse.cc:5115
            #9  0x0000000000651e16 in mysql_parse (thd=0x7f2c33a2e070, rawbuf=0x7f2c269a4088 "ALTER TABLE ticket \nCHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,\nCHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,\nADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT"..., length=230, parser_state=0x7f2c3bbfc650) at /data/src/10.0/sql/sql_parse.cc:6575
            #10 0x0000000000644922 in dispatch_command (command=COM_QUERY, thd=0x7f2c33a2e070, packet=0x7f2c2df53071 "ALTER TABLE ticket \nCHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,\nCHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,\nADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT"..., packet_length=230) at /data/src/10.0/sql/sql_parse.cc:1309
            #11 0x0000000000643be5 in do_command (thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_parse.cc:999
            #12 0x0000000000761e66 in do_handle_one_connection (thd_arg=0x7f2c33a2e070) at /data/src/10.0/sql/sql_connect.cc:1378
            #13 0x0000000000761bd8 in handle_one_connection (arg=0x7f2c33a2e070) at /data/src/10.0/sql/sql_connect.cc:1293
            #14 0x00000000009fa3a0 in pfs_spawn_thread (arg=0x7f2c2df03670) at /data/src/10.0/storage/perfschema/pfs.cc:1860
            #15 0x00007f2c3b8430a4 in start_thread (arg=0x7f2c3bbfd700) at pthread_create.c:309
            #16 0x00007f2c399fb87d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
            

            elenst Elena Stepanova added a comment - - edited Crashes on 10.x. No crash on 5.5, MySQL 5.6. Test case --source include/have_innodb.inc   CREATE TABLE IF NOT EXISTS ticket ( id INT UNSIGNED NOT NULL AUTO_INCREMENT, mask VARCHAR (16) DEFAULT '' NOT NULL , subject VARCHAR (255) DEFAULT '' NOT NULL , is_closed TINYINT(1) UNSIGNED DEFAULT 0 NOT NULL , is_deleted TINYINT(1) UNSIGNED DEFAULT 0 NOT NULL , team_id INT UNSIGNED DEFAULT 0 NOT NULL , category_id INT UNSIGNED DEFAULT 0 NOT NULL , first_message_id INT UNSIGNED DEFAULT 0 NOT NULL , created_date INT UNSIGNED, updated_date INT UNSIGNED, due_date INT UNSIGNED, first_wrote_address_id INT UNSIGNED NOT NULL DEFAULT 0, last_wrote_address_id INT UNSIGNED NOT NULL DEFAULT 0, spam_score DECIMAL (4,4) NOT NULL DEFAULT 0, spam_training VARCHAR (1) NOT NULL DEFAULT '' , interesting_words VARCHAR (255) NOT NULL DEFAULT '' , next_action VARCHAR (255) NOT NULL DEFAULT '' , PRIMARY KEY (id) ) ENGINE=InnoDB;   ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0, CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0, ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0, ADD INDEX org_id (org_id); Stack trace from 10.0 commit b5fb2a685b6ec67d37033b020a8145d1aac1fc93 #3 <signal handler called> #4 0x00007f2c3235f052 in innobase_need_rebuild (ha_alter_info=0x7f2c3bbfa270, altered_table=0x7f2c2689e470) at /data/src/10.0/storage/innobase/handler/handler0alter.cc:243 #5 0x00007f2c3235fdd6 in ha_innodb::check_if_supported_inplace_alter (this=0x7f2c2687e888, altered_table=0x7f2c269f2070, ha_alter_info=0x7f2c3bbfa270) at /data/src/10.0/storage/innobase/handler/handler0alter.cc:578 #6 0x00000000006fecfe in mysql_alter_table (thd=0x7f2c33a2e070, new_db=0x7f2c269a48e0 "test", new_name=0x0, create_info=0x7f2c3bbfb4f0, table_list=0x7f2c269a4300, alter_info=0x7f2c3bbfb460, order_num=0, order=0x0, ignore=false) at /data/src/10.0/sql/sql_table.cc:8815 #7 0x0000000000766713 in Sql_cmd_alter_table::execute (this=0x7f2c269a4e58, thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_alter.cc:312 #8 0x000000000064e9eb in mysql_execute_command (thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_parse.cc:5115 #9 0x0000000000651e16 in mysql_parse (thd=0x7f2c33a2e070, rawbuf=0x7f2c269a4088 "ALTER TABLE ticket \nCHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,\nCHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,\nADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT"..., length=230, parser_state=0x7f2c3bbfc650) at /data/src/10.0/sql/sql_parse.cc:6575 #10 0x0000000000644922 in dispatch_command (command=COM_QUERY, thd=0x7f2c33a2e070, packet=0x7f2c2df53071 "ALTER TABLE ticket \nCHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,\nCHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,\nADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT"..., packet_length=230) at /data/src/10.0/sql/sql_parse.cc:1309 #11 0x0000000000643be5 in do_command (thd=0x7f2c33a2e070) at /data/src/10.0/sql/sql_parse.cc:999 #12 0x0000000000761e66 in do_handle_one_connection (thd_arg=0x7f2c33a2e070) at /data/src/10.0/sql/sql_connect.cc:1378 #13 0x0000000000761bd8 in handle_one_connection (arg=0x7f2c33a2e070) at /data/src/10.0/sql/sql_connect.cc:1293 #14 0x00000000009fa3a0 in pfs_spawn_thread (arg=0x7f2c2df03670) at /data/src/10.0/storage/perfschema/pfs.cc:1860 #15 0x00007f2c3b8430a4 in start_thread (arg=0x7f2c3bbfd700) at pthread_create.c:309 #16 0x00007f2c399fb87d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

            For some reason I was falsely under the impression that I used MyISAM while I never actually used it. I just saw it in their schema.sql file which is not used apparently for a fresh install. You can ignore my whole "myisam observation" thingy, I'm wrong. Still interesting what causes this though.

            daan.vangorkum Daan van Gorkum added a comment - For some reason I was falsely under the impression that I used MyISAM while I never actually used it. I just saw it in their schema.sql file which is not used apparently for a fresh install. You can ignore my whole "myisam observation" thingy, I'm wrong. Still interesting what causes this though.

            commit 9b23f8054d2f37458901b4505429c30eddc440bc
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Thu Aug 11 14:39:47 2016 +0300

            MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash

            When checking is any of the renamed columns part of the
            columns for new indexes we accessed NULL pointer if checked
            column used on index was added on same statement. Additionally,
            we tried to check too many indexes, added_index_count
            is enough here.

            jplindst Jan Lindström (Inactive) added a comment - commit 9b23f8054d2f37458901b4505429c30eddc440bc Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Thu Aug 11 14:39:47 2016 +0300 MDEV-10535 : ALTER TABLE causes standalone/wsrep cluster crash When checking is any of the renamed columns part of the columns for new indexes we accessed NULL pointer if checked column used on index was added on same statement. Additionally, we tried to check too many indexes, added_index_count is enough here.

            People

              jplindst Jan Lindström (Inactive)
              daan.vangorkum Daan van Gorkum
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.