Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
10.1.16, 10.0(EOL), 10.1(EOL)
-
Ubuntu 14.04.5 LTS, Debian 8
Description
Hello,
While I was installing Cerberus Helpdesk our whole wsrep galera cluster crashed (Debian 8, 10.0.26+maria-1~jessie). Getting it back up was no problem and I ran the same installer on our standalone development environment (Ubuntu 14.04.5, 10.1.16+maria-1~trusty, no wsrep) and it caused the same issue: mariadb/mysqld crashed.
All packages are installed using the mirror tool on the mariadb website.
This is the query that causes a crash:
ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id); ERROR 2013 (HY000): Lost connection to MySQL server during query
|
This is the stack trace I get on our cluster:
Aug 10 22:07:19 database1 mysqld: 160810 22:07:19 [ERROR] mysqld got signal 11 ;
|
Aug 10 22:07:19 database1 mysqld: This could be because you hit a bug. It is also possible that this binary
|
Aug 10 22:07:19 database1 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Aug 10 22:07:19 database1 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: We will try our best to scrape up some info that will hopefully help
|
Aug 10 22:07:19 database1 mysqld: diagnose the problem, but since we have already crashed,
|
Aug 10 22:07:19 database1 mysqld: something is definitely wrong and this may fail.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Server version: 10.0.26-MariaDB-1~jessie-wsrep
|
Aug 10 22:07:19 database1 mysqld: key_buffer_size=134217728
|
Aug 10 22:07:19 database1 mysqld: read_buffer_size=2097152
|
Aug 10 22:07:19 database1 mysqld: max_used_connections=3
|
Aug 10 22:07:19 database1 mysqld: max_threads=102
|
Aug 10 22:07:19 database1 mysqld: thread_count=3
|
Aug 10 22:07:19 database1 mysqld: It is possible that mysqld could use up to
|
Aug 10 22:07:19 database1 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 759801 K bytes of memory
|
Aug 10 22:07:19 database1 mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Thread pointer: 0x0x7f7292f20008
|
Aug 10 22:07:19 database1 mysqld: Attempting backtrace. You can use the following information to find out
|
Aug 10 22:07:19 database1 mysqld: where mysqld died. If you see no messages after this, something went
|
Aug 10 22:07:19 database1 mysqld: terribly wrong...
|
Aug 10 22:07:19 database1 mysqld: stack_bottom = 0x7f72e2855e28 thread_stack 0x48000
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x3d)[0x7f72e2365e6d]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x31a)[0x7f72e1e93c2a]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f72e14d48d0]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x32ee26)[0x7f72e1c33e26]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x723e73)[0x7f72e2028e73]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x2575)[0x7f72e1d8f065]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x60c)[0x7f72e1ddc08c]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x956)[0x7f72e1cf4296]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x24a)[0x7f72e1cfcc9a]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(+0x3f8379)[0x7f72e1cfd379]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1875)[0x7f72e1cff285]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x325)[0x7f72e1cfff35]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x37b)[0x7f72e1dd837b]
|
Aug 10 22:07:19 database1 mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x7f72e1dd83f0]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7f72e14cd0a4]
|
Aug 10 22:07:19 database1 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f72dfad387d]
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Trying to get some variables.
|
Aug 10 22:07:19 database1 mysqld: Some pointers may be invalid and cause the dump to abort.
|
Aug 10 22:07:19 database1 mysqld: Query (0x7f72a2b79020): is an invalid pointer
|
Aug 10 22:07:19 database1 mysqld: Connection ID (thread ID): 159
|
Aug 10 22:07:19 database1 mysqld: Status: NOT_KILLED
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
|
Aug 10 22:07:19 database1 mysqld:
|
Aug 10 22:07:19 database1 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Aug 10 22:07:19 database1 mysqld: information that should help you find out what is causing the crash.
|
Aug 10 22:07:20 database1 mysqld_safe: Number of processes running now: 0
|
Aug 10 22:07:20 database1 mysqld_safe: WSREP: not restarting wsrep node automatically
|
Aug 10 22:07:20 database1 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
|
And here is the information from our standalone dev server:
Aug 10 22:30:25 dev mysqld: 160810 22:30:25 [ERROR] mysqld got signal 11 ;
|
Aug 10 22:30:25 dev mysqld: This could be because you hit a bug. It is also possible that this binary
|
Aug 10 22:30:25 dev mysqld: or one of the libraries it was linked against is corrupt, improperly built,
|
Aug 10 22:30:25 dev mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: We will try our best to scrape up some info that will hopefully help
|
Aug 10 22:30:25 dev mysqld: diagnose the problem, but since we have already crashed,
|
Aug 10 22:30:25 dev mysqld: something is definitely wrong and this may fail.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Server version: 10.1.16-MariaDB-1~trusty
|
Aug 10 22:30:25 dev mysqld: key_buffer_size=16777216
|
Aug 10 22:30:25 dev mysqld: read_buffer_size=131072
|
Aug 10 22:30:25 dev mysqld: max_used_connections=14
|
Aug 10 22:30:25 dev mysqld: max_threads=153
|
Aug 10 22:30:25 dev mysqld: thread_count=7
|
Aug 10 22:30:25 dev mysqld: It is possible that mysqld could use up to
|
Aug 10 22:30:25 dev mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 352425 K bytes of memory
|
Aug 10 22:30:25 dev mysqld: Hope that's ok; if not, decrease some variables in the equation.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Thread pointer: 0x0x7faa3c4b6008
|
Aug 10 22:30:25 dev mysqld: Attempting backtrace. You can use the following information to find out
|
Aug 10 22:30:25 dev mysqld: where mysqld died. If you see no messages after this, something went
|
Aug 10 22:30:25 dev mysqld: terribly wrong...
|
Aug 10 22:30:25 dev mysqld: stack_bottom = 0x7faa65467df0 thread_stack 0x30000
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55dfa03028fe]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x2d5)[0x55df9fe29e75]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7faa64039330]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x379749)[0x55df9fbf2749]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(+0x801ebf)[0x55dfa007aebf]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P14HA_CREATE_INFOP10TABLE_LISTP10Alter_infojP8st_orderb+0x1e3a)[0x55df9fd34e4a]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x61a)[0x55df9fd7b5ca]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1206)[0x55df9fca8116]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x26d)[0x55df9fcb073d]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2460)[0x55df9fcb3a80]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x55df9fcb4239]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x18a)[0x55df9fd7889a]
|
Aug 10 22:30:25 dev mysqld: /usr/sbin/mysqld(handle_one_connection+0x40)[0x55df9fd78a70]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7faa64031184]
|
Aug 10 22:30:25 dev mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7faa6355037d]
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Trying to get some variables.
|
Aug 10 22:30:25 dev mysqld: Some pointers may be invalid and cause the dump to abort.
|
Aug 10 22:30:25 dev mysqld: Query (0x7faa3e42c020): is an invalid pointer
|
Aug 10 22:30:25 dev mysqld: Connection ID (thread ID): 127629
|
Aug 10 22:30:25 dev mysqld: Status: NOT_KILLED
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
|
Aug 10 22:30:25 dev mysqld: information that should help you find out what is causing the crash.
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld: We think the query pointer is invalid, but we will try to print it anyway.
|
Aug 10 22:30:25 dev mysqld: Query: ALTER TABLE ticket CHANGE COLUMN team_id group_id INT UNSIGNED NOT NULL DEFAULT 0,CHANGE COLUMN category_id bucket_id INT UNSIGNED NOT NULL DEFAULT 0,ADD COLUMN org_id INT UNSIGNED NOT NULL DEFAULT 0,ADD INDEX org_id (org_id)
|
Aug 10 22:30:25 dev mysqld:
|
Aug 10 22:30:25 dev mysqld_safe: Number of processes running now: 0
|
Aug 10 22:30:25 dev mysqld_safe: mysqld restarted
|
I was able to find the CREATE TABLE query for the ticket table:
DROP TABLE IF EXISTS `ticket`;
|
/*!40101 SET @saved_cs_client = @@character_set_client */;
|
/*!40101 SET character_set_client = utf8 */;
|
CREATE TABLE `ticket` (
|
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
|
`mask` varchar(32) NOT NULL DEFAULT '',
|
`subject` varchar(255) NOT NULL DEFAULT '',
|
`is_closed` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`is_deleted` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`group_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`bucket_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`created_date` int(10) unsigned DEFAULT NULL,
|
`updated_date` int(10) unsigned DEFAULT NULL,
|
`reopen_at` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`last_wrote_address_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`spam_score` decimal(4,4) NOT NULL DEFAULT '0.0000',
|
`spam_training` varchar(1) NOT NULL DEFAULT '',
|
`interesting_words` varchar(255) NOT NULL DEFAULT '',
|
`is_waiting` tinyint(1) unsigned NOT NULL DEFAULT '0',
|
`last_action_code` varchar(1) NOT NULL DEFAULT 'O',
|
`last_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`owner_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`org_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`num_messages` int(10) unsigned NOT NULL DEFAULT '0',
|
`closed_at` int(10) unsigned NOT NULL DEFAULT '0',
|
`first_outgoing_message_id` int(10) unsigned NOT NULL DEFAULT '0',
|
`elapsed_response_first` int(10) unsigned NOT NULL DEFAULT '0',
|
`elapsed_resolution_first` int(10) unsigned NOT NULL DEFAULT '0',
|
PRIMARY KEY (`id`),
|
KEY `first_message_id` (`first_message_id`),
|
KEY `mask` (`mask`),
|
KEY `is_waiting` (`is_waiting`),
|
KEY `team_id` (`group_id`),
|
KEY `created_date` (`created_date`),
|
KEY `updated_date` (`updated_date`),
|
KEY `first_wrote_address_id` (`first_wrote_address_id`),
|
KEY `last_wrote_address_id` (`last_wrote_address_id`),
|
KEY `is_closed` (`is_closed`),
|
KEY `category_id` (`bucket_id`),
|
KEY `due_date` (`reopen_at`),
|
KEY `is_deleted` (`is_deleted`),
|
KEY `last_action_code` (`last_action_code`),
|
KEY `spam_score` (`spam_score`),
|
KEY `last_message_id` (`last_message_id`),
|
KEY `owner_id` (`owner_id`),
|
KEY `org_id` (`org_id`)
|
) ENGINE=MyISAM AUTO_INCREMENT=2 DEFAULT CHARSET=utf8;
|
/*!40101 SET character_set_client = @saved_cs_client */;
|
And the whole schema can be found here: https://github.com/wgm/cerb/blob/stable/install/extras/automation/cerb.setup.schema.sql
Memory is not an issue on those machines and I'm using InnoDB.
Please let me know if you need any additional information!
Regards,
Daan
Attachments
Issue Links
- relates to
-
MDEV-13668 InnoDB unnecessarily rebuilds table when renaming a column and adding index
-
- Closed
-
-
MDEV-13838 Wrong result after altering a partitioned table
-
- Closed
-
Quick oberservation:
While I explicity request for InnoDB during the install I see that MyISAM is used. And when I look at the code I don't really see what my InnoDB request actually does...so it might be related to MyISAM (while I don't understand why that affects standalone mode):
# grep -R db_engine
install/classes.php: public static function saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass) {
install/classes.php: $value = $db_engine;
install/templates/steps/step_config_file.tpl:<input type="hidden" name="db_engine" value="{$db_engine}">
install/templates/steps/step_database.tpl:<select name="db_engine">
install/templates/steps/step_database.tpl: <option value="{$k}" {if $k==$db_engine}selected{/if}>{$engine}
install/index.php: @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string');
install/index.php: if(!empty($db_driver) && !empty($db_engine) && !empty($db_server) && !empty($db_name) && !empty($db_user)) {
install/index.php: if(!in_array($db_engine, $discovered_engines)) {
install/index.php: $errors[] = sprintf("The '%s' storage engine is not enabled.", $db_engine);
install/index.php: $tpl->assign('db_engine', $db_engine);
install/index.php: $result = CerberusInstaller::saveFrameworkConfig($db_driver, $db_engine, $encoding, $db_server, $db_name, $db_user, $db_pass);
install/index.php: @$db_engine = DevblocksPlatform::importGPC($_POST['db_engine'],'string');
install/index.php: 0 == strcasecmp($db_engine,APP_DB_ENGINE) &&
install/index.php: $tpl->assign('db_engine', $db_engine);
libs/devblocks/api/services/search.php: $db_engine_fulltext = 'MyISAM';
libs/devblocks/api/services/search.php: $db_engine_fulltext = APP_DB_ENGINE_FULLTEXT;
libs/devblocks/api/services/search.php: $db_engine_fulltext
# grep -R -i innodb
framework.config.php:define('APP_DB_ENGINE','innodb');
install/index.php: 'innodb' => 'InnoDB (Recommended)',
libs/devblocks/api/services/search.php: // InnoDB stop words
I do see them altering tables with InnoDB in some migration scripts but nothing for the ticket table or some others. Still doesn't explain why this is an issue when not using wsrep.
Daan