Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.16, 10.0(EOL), 10.1(EOL)
-
Ubuntu 14.04.5 LTS, Debian 8
Description
Hello,
While I was installing Cerberus Helpdesk our whole wsrep galera cluster crashed (Debian 8, 10.0.26+maria-1~jessie). Getting it back up was no problem and I ran the same installer on our standalone development environment (Ubuntu 14.04.5, 10.1.16+maria-1~trusty, no wsrep) and it caused the same issue: mariadb/mysqld crashed.
All packages are installed using the mirror tool on the mariadb website.
This is the query that causes a crash:
ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id); ERROR 2013 (HY000): Lost connection to MySQL server during query
|
This is the stack trace I get on our cluster:
Aug 10 22:07:19 database1 mysqld: 160810 22:07:19 [ERROR] mysqld got signal 11 ;
|
Aug 10 22:07:19 database1 mysqld: This could be because you hit a bug. It is also possible that this binary
|
Aug 10 22:07:19 database1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Aug 10 22:07:19 database1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: We will try our best to scrape up some info that will hopefully help
|
Aug 10 22:07:19 database1 mysqld: diagnose the problem, but since we have already crashed,
|
Aug 10 22:07:19 database1 mysqld: something is definitely wrong and this may fail.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Server version: 10.0.26-MariaDB-1~jessie-wsrep
|
Aug 10 22:07:19 database1 mysqld: key_buffer_size=134217728
|
Aug 10 22:07:19 database1 mysqld: read_buffer_size=2097152
|
Aug 10 22:07:19 database1 mysqld: max_used_connections=3
|
Aug 10 22:07:19 database1 mysqld: max_threads=102
|
Aug 10 22:07:19 database1 mysqld: thread_count=3
|
Aug 10 22:07:19 database1 mysqld: It is possible that mysqld could use up to
|
Aug 10 22:07:19 database1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759801 K bytes of memory
|
Aug 10 22:07:19 database1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Thread pointer: 0x0x7f7292f20008
|
Aug 10 22:07:19 database1 mysqld: Attempting backtrace. You can use the following information to find out
|
Aug 10 22:07:19 database1 mysqld: where mysqld died. If you see no messages after this, something went
|
Aug 10 22:07:19 database1 mysqld: terribly wrong...
|
Aug 10 22:07:19 database1 mysqld: stack_bottom = 0x7f72e2855e28 thread_stack 0x48000
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x3d)[0x7f72e2365e6d]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x31a)[0x7f72e1e93c2a]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f72e14d48d0]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x32ee26)[0x7f72e1c33e26]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x723e73)[0x7f72e2028e73]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2575)[0x7f72e1d8f065]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x60c)[0x7f72e1ddc08c]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x956)[0x7f72e1cf4296]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x24a)[0x7f72e1cfcc9a]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x3f8379)[0x7f72e1cfd379]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1875)[0x7f72e1cff285]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x325)[0x7f72e1cfff35]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x37b)[0x7f72e1dd837b]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f72e1dd83f0]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f72e14cd0a4]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72dfad387d]
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Trying to get some variables.
|
Aug 10 22:07:19 database1 mysqld: Some pointers may be invalid and cause the dump to abort.
|
Aug 10 22:07:19 database1 mysqld: Query (0x7f72a2b79020): is an invalid pointer
|
Aug 10 22:07:19 database1 mysqld: Connection ID (thread ID): 159
|
Aug 10 22:07:19 database1 mysqld: Status: NOT_KILLED
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Aug 10 22:07:19 database1 mysqld: information that should help you find out what is causing the crash.
|
Aug 10 22:07:20 database1 mysqld_safe: Number of processes running now: 0
|
Aug 10 22:07:20 database1 mysqld_safe: WSREP: not restarting wsrep node automatically
|
Aug 10 22:07:20 database1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
|
And here is the information from our standalone dev server:
Aug 10 22:30:25 dev mysqld: 160810 22:30:25 [ERROR] mysqld got signal 11 ;
|
Aug 10 22:30:25 dev mysqld: This could be because you hit a bug. It is also possible that this binary
|
Aug 10 22:30:25 dev mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Aug 10 22:30:25 dev mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: We will try our best to scrape up some info that will hopefully help
|
Aug 10 22:30:25 dev mysqld: diagnose the problem, but since we have already crashed,
|
Aug 10 22:30:25 dev mysqld: something is definitely wrong and this may fail.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Server version: 10.1.16-MariaDB-1~trusty
|
Aug 10 22:30:25 dev mysqld: key_buffer_size=16777216
|
Aug 10 22:30:25 dev mysqld: read_buffer_size=131072
|
Aug 10 22:30:25 dev mysqld: max_used_connections=14
|
Aug 10 22:30:25 dev mysqld: max_threads=153
|
Aug 10 22:30:25 dev mysqld: thread_count=7
|
Aug 10 22:30:25 dev mysqld: It is possible that mysqld could use up to
|
Aug 10 22:30:25 dev mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352425 K bytes of memory
|
Aug 10 22:30:25 dev mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Thread pointer: 0x0x7faa3c4b6008
|
Aug 10 22:30:25 dev mysqld: Attempting backtrace. You can use the following information to find out
|
Aug 10 22:30:25 dev mysqld: where mysqld died. If you see no messages after this, something went
|
Aug 10 22:30:25 dev mysqld: terribly wrong...
|
Aug 10 22:30:25 dev mysqld: stack_bottom = 0x7faa65467df0 thread_stack 0x30000
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dfa03028fe]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x55df9fe29e75]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7faa64039330]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x379749)[0x55df9fbf2749]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x801ebf)[0x55dfa007aebf]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1e3a)[0x55df9fd34e4a]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55df9fd7b5ca]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55df9fca8116]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55df9fcb073d]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55df9fcb3a80]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55df9fcb4239]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55df9fd7889a]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x55df9fd78a70]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7faa64031184]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7faa6355037d]
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Trying to get some variables.
|
Aug 10 22:30:25 dev mysqld: Some pointers may be invalid and cause the dump to abort.
|
Aug 10 22:30:25 dev mysqld: Query (0x7faa3e42c020): is an invalid pointer
|
Aug 10 22:30:25 dev mysqld: Connection ID (thread ID): 127629
|
Aug 10 22:30:25 dev mysqld: Status: NOT_KILLED
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Aug 10 22:30:25 dev mysqld: information that should help you find out what is causing the crash.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: We think the query pointer is invalid, but we will try to print it anyway.
|
Aug 10 22:30:25 dev mysqld: Query: ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id)
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld_safe: Number of processes running now: 0
|
Aug 10 22:30:25 dev mysqld_safe: mysqld restarted
|
I was able to find the CREATE TABLE query for the ticket table:
DROP TABLE IF EXISTS `ticket`;
|
/*!40101 SET @saved_cs_client = @@character_set_client */;
|
/*!40101 SET character_set_client = utf8 */;
|
CREATE TABLE `ticket` (
|
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
|
`mask` varchar(32) NOT NULL DEFAULT '',
|
`subject` varchar(255) NOT NULL DEFAULT '',
|
`is_closed` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`is_deleted` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`group_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`bucket_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`created_date` int(10) unsigned DEFAULT NULL,
|
`updated_date` int(10) unsigned DEFAULT NULL,
|
`reopen_at` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`last_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`spam_score` decimal(4,4) NOT NULL DEFAULT '0.0000',
|
`spam_training` varchar(1) NOT NULL DEFAULT '',
|
`interesting_words` varchar(255) NOT NULL DEFAULT '',
|
`is_waiting` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`last_action_code` varchar(1) NOT NULL DEFAULT 'O',
|
`last_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`owner_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`org_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`num_messages` int(10) unsigned NOT NULL DEFAULT '0',
|
`closed_at` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_outgoing_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`elapsed_response_first` int(10) unsigned NOT NULL DEFAULT '0',
|
`elapsed_resolution_first` int(10) unsigned NOT NULL DEFAULT '0',
|
PRIMARY KEY (`id`),
|
KEY `first_message_id` (`first_message_id`),
|
KEY `mask` (`mask`),
|
KEY `is_waiting` (`is_waiting`),
|
KEY `team_id` (`group_id`),
|
KEY `created_date` (`created_date`),
|
KEY `updated_date` (`updated_date`),
|
KEY `first_wrote_address_id` (`first_wrote_address_id`),
|
KEY `last_wrote_address_id` (`last_wrote_address_id`),
|
KEY `is_closed` (`is_closed`),
|
KEY `category_id` (`bucket_id`),
|
KEY `due_date` (`reopen_at`),
|
KEY `is_deleted` (`is_deleted`),
|
KEY `last_action_code` (`last_action_code`),
|
KEY `spam_score` (`spam_score`),
|
KEY `last_message_id` (`last_message_id`),
|
KEY `owner_id` (`owner_id`),
|
KEY `org_id` (`org_id`)
|
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
|
/*!40101 SET character_set_client = @saved_cs_client */;
|
And the whole schema can be found here: https://github.com/wgm/cerb/blob/stable/install/extras/automation/cerb.setup.schema.sql
Memory is not an issue on those machines and I'm using InnoDB.
Please let me know if you need any additional information!
Regards,
Daan
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
-
- Closed
-