Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-10535

ALTER TABLE causes standalone/wsrep cluster crash

    XMLWordPrintable

Details

    Description

      Hello,

      While I was installing Cerberus Helpdesk our whole wsrep galera cluster crashed (Debian 8, 10.0.26+maria-1~jessie). Getting it back up was no problem and I ran the same installer on our standalone development environment (Ubuntu 14.04.5, 10.1.16+maria-1~trusty, no wsrep) and it caused the same issue: mariadb/mysqld crashed.

      All packages are installed using the mirror tool on the mariadb website.

      This is the query that causes a crash:

      ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id); ERROR 2013 (HY000): Lost connection to MySQL server during query
      

      This is the stack trace I get on our cluster:

      Aug 10 22:07:19 database1 mysqld: 160810 22:07:19 [ERROR] mysqld got signal 11 ;
      Aug 10 22:07:19 database1 mysqld: This could be because you hit a bug. It is also possible that this binary
      Aug 10 22:07:19 database1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Aug 10 22:07:19 database1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: We will try our best to scrape up some info that will hopefully help
      Aug 10 22:07:19 database1 mysqld: diagnose the problem, but since we have already crashed,
      Aug 10 22:07:19 database1 mysqld: something is definitely wrong and this may fail.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Server version: 10.0.26-MariaDB-1~jessie-wsrep
      Aug 10 22:07:19 database1 mysqld: key_buffer_size=134217728
      Aug 10 22:07:19 database1 mysqld: read_buffer_size=2097152
      Aug 10 22:07:19 database1 mysqld: max_used_connections=3
      Aug 10 22:07:19 database1 mysqld: max_threads=102
      Aug 10 22:07:19 database1 mysqld: thread_count=3
      Aug 10 22:07:19 database1 mysqld: It is possible that mysqld could use up to
      Aug 10 22:07:19 database1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759801 K  bytes of memory
      Aug 10 22:07:19 database1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Thread pointer: 0x0x7f7292f20008
      Aug 10 22:07:19 database1 mysqld: Attempting backtrace. You can use the following information to find out
      Aug 10 22:07:19 database1 mysqld: where mysqld died. If you see no messages after this, something went
      Aug 10 22:07:19 database1 mysqld: terribly wrong...
      Aug 10 22:07:19 database1 mysqld: stack_bottom = 0x7f72e2855e28 thread_stack 0x48000
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x3d)[0x7f72e2365e6d]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x31a)[0x7f72e1e93c2a]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f72e14d48d0]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x32ee26)[0x7f72e1c33e26]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x723e73)[0x7f72e2028e73]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2575)[0x7f72e1d8f065]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x60c)[0x7f72e1ddc08c]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x956)[0x7f72e1cf4296]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x24a)[0x7f72e1cfcc9a]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x3f8379)[0x7f72e1cfd379]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1875)[0x7f72e1cff285]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x325)[0x7f72e1cfff35]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x37b)[0x7f72e1dd837b]
      Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f72e1dd83f0]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f72e14cd0a4]
      Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72dfad387d]
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Trying to get some variables.
      Aug 10 22:07:19 database1 mysqld: Some pointers may be invalid and cause the dump to abort.
      Aug 10 22:07:19 database1 mysqld: Query (0x7f72a2b79020): is an invalid pointer
      Aug 10 22:07:19 database1 mysqld: Connection ID (thread ID): 159
      Aug 10 22:07:19 database1 mysqld: Status: NOT_KILLED
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
      Aug 10 22:07:19 database1 mysqld:
      Aug 10 22:07:19 database1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Aug 10 22:07:19 database1 mysqld: information that should help you find out what is causing the crash.
      Aug 10 22:07:20 database1 mysqld_safe: Number of processes running now: 0
      Aug 10 22:07:20 database1 mysqld_safe: WSREP: not restarting wsrep node automatically
      Aug 10 22:07:20 database1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
      

      And here is the information from our standalone dev server:

      Aug 10 22:30:25 dev mysqld: 160810 22:30:25 [ERROR] mysqld got signal 11 ;
      Aug 10 22:30:25 dev mysqld: This could be because you hit a bug. It is also possible that this binary
      Aug 10 22:30:25 dev mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Aug 10 22:30:25 dev mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: We will try our best to scrape up some info that will hopefully help
      Aug 10 22:30:25 dev mysqld: diagnose the problem, but since we have already crashed,
      Aug 10 22:30:25 dev mysqld: something is definitely wrong and this may fail.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Server version: 10.1.16-MariaDB-1~trusty
      Aug 10 22:30:25 dev mysqld: key_buffer_size=16777216
      Aug 10 22:30:25 dev mysqld: read_buffer_size=131072
      Aug 10 22:30:25 dev mysqld: max_used_connections=14
      Aug 10 22:30:25 dev mysqld: max_threads=153
      Aug 10 22:30:25 dev mysqld: thread_count=7
      Aug 10 22:30:25 dev mysqld: It is possible that mysqld could use up to
      Aug 10 22:30:25 dev mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352425 K  bytes of memory
      Aug 10 22:30:25 dev mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Thread pointer: 0x0x7faa3c4b6008
      Aug 10 22:30:25 dev mysqld: Attempting backtrace. You can use the following information to find out
      Aug 10 22:30:25 dev mysqld: where mysqld died. If you see no messages after this, something went
      Aug 10 22:30:25 dev mysqld: terribly wrong...
      Aug 10 22:30:25 dev mysqld: stack_bottom = 0x7faa65467df0 thread_stack 0x30000
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dfa03028fe]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x55df9fe29e75]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7faa64039330]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x379749)[0x55df9fbf2749]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x801ebf)[0x55dfa007aebf]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1e3a)[0x55df9fd34e4a]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55df9fd7b5ca]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55df9fca8116]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55df9fcb073d]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55df9fcb3a80]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55df9fcb4239]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55df9fd7889a]
      Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x55df9fd78a70]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7faa64031184]
      Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7faa6355037d]
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Trying to get some variables.
      Aug 10 22:30:25 dev mysqld: Some pointers may be invalid and cause the dump to abort.
      Aug 10 22:30:25 dev mysqld: Query (0x7faa3e42c020): is an invalid pointer
      Aug 10 22:30:25 dev mysqld: Connection ID (thread ID): 127629
      Aug 10 22:30:25 dev mysqld: Status: NOT_KILLED
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Aug 10 22:30:25 dev mysqld: information that should help you find out what is causing the crash.
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld: We think the query pointer is invalid, but we will try to print it anyway.
      Aug 10 22:30:25 dev mysqld: Query: ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id)
      Aug 10 22:30:25 dev mysqld:
      Aug 10 22:30:25 dev mysqld_safe: Number of processes running now: 0
      Aug 10 22:30:25 dev mysqld_safe: mysqld restarted
      

      I was able to find the CREATE TABLE query for the ticket table:

      DROP TABLE IF EXISTS `ticket`;
      /*!40101 SET @saved_cs_client     = @@character_set_client */;
      /*!40101 SET character_set_client = utf8 */;
      CREATE TABLE `ticket` (
        `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
        `mask` varchar(32) NOT NULL DEFAULT '',
        `subject` varchar(255) NOT NULL DEFAULT '',
        `is_closed` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `is_deleted` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `group_id` int(10) unsigned NOT NULL DEFAULT '0',
        `bucket_id` int(10) unsigned NOT NULL DEFAULT '0',
        `first_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `created_date` int(10) unsigned DEFAULT NULL,
        `updated_date` int(10) unsigned DEFAULT NULL,
        `reopen_at` int(10) unsigned NOT NULL DEFAULT '0',
        `first_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
        `last_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
        `spam_score` decimal(4,4) NOT NULL DEFAULT '0.0000',
        `spam_training` varchar(1) NOT NULL DEFAULT '',
        `interesting_words` varchar(255) NOT NULL DEFAULT '',
        `is_waiting` tinyint(1) unsigned NOT NULL DEFAULT '0',
        `last_action_code` varchar(1) NOT NULL DEFAULT 'O',
        `last_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `owner_id` int(10) unsigned NOT NULL DEFAULT '0',
        `org_id` int(10) unsigned NOT NULL DEFAULT '0',
        `num_messages` int(10) unsigned NOT NULL DEFAULT '0',
        `closed_at` int(10) unsigned NOT NULL DEFAULT '0',
        `first_outgoing_message_id` int(10) unsigned NOT NULL DEFAULT '0',
        `elapsed_response_first` int(10) unsigned NOT NULL DEFAULT '0',
        `elapsed_resolution_first` int(10) unsigned NOT NULL DEFAULT '0',
        PRIMARY KEY (`id`),
        KEY `first_message_id` (`first_message_id`),
        KEY `mask` (`mask`),
        KEY `is_waiting` (`is_waiting`),
        KEY `team_id` (`group_id`),
        KEY `created_date` (`created_date`),
        KEY `updated_date` (`updated_date`),
        KEY `first_wrote_address_id` (`first_wrote_address_id`),
        KEY `last_wrote_address_id` (`last_wrote_address_id`),
        KEY `is_closed` (`is_closed`),
        KEY `category_id` (`bucket_id`),
        KEY `due_date` (`reopen_at`),
        KEY `is_deleted` (`is_deleted`),
        KEY `last_action_code` (`last_action_code`),
        KEY `spam_score` (`spam_score`),
        KEY `last_message_id` (`last_message_id`),
        KEY `owner_id` (`owner_id`),
        KEY `org_id` (`org_id`)
      ) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
      /*!40101 SET character_set_client = @saved_cs_client */;
      

      And the whole schema can be found here: https://github.com/wgm/cerb/blob/stable/install/extras/automation/cerb.setup.schema.sql

      Memory is not an issue on those machines and I'm using InnoDB.

      Please let me know if you need any additional information!

      Regards,
      Daan

      Attachments

        1. cerberusweb_general_log.txt
          609 kB
        2. my.cnf
          4 kB
        3. other_mysql.cnf
          0.9 kB

        Issue Links

          Activity

            People

              jplindst Jan Lindström (Inactive)
              daan.vangorkum Daan van Gorkum
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.