Details

    • Bug
    • Status: Closed (View Workflow)
    • Major
    • Resolution: Duplicate
    • 10.1.24
    • N/A
    • None
    • Centos Linux 3.10.0-514.21.2.el7.x86_64 x86_64 x86_64 x86_64 GNU/Linux

    Description

      I updated my MariaDB server yesterday and the server crashed twice today.
      ======================================================

      [root@centos2 mysql]# mysqld --print-defaults or SHOW VARIABLES
      mysqld would have been started with the following arguments:
      --port=3306 --socket=/var/lib/mysql/mysql.sock --skip-external-locking --key_buffer_size=8192M --max_allowed_packet=500M --table_open_cache=256 --sort_buffer_size=2M --read_buffer_size=1M --read_rnd_buffer_size=4M --myisam_sort_buffer_size=64M --thread_cache_size=8 --query_cache_size=16M --thread_concurrency=0 --symbolic-links=0 --max_connections=1024 --lower_case_table_names=1 --innodb_data_home_dir=/var/lib/mysql --innodb_data_file_path=ibdata1:10M:autoextend --innodb_log_group_home_dir=/var/lib/mysql --innodb_buffer_pool_size=16G --innodb_log_file_size=5242880 --innodb_log_buffer_size=256M --innodb_flush_log_at_trx_commit=2 --innodb_lock_wait_timeout=50
      

      ======================================================

      /var/log/messages

      Jun 23 14:40:11 centos2 mysqld: 170623 14:40:11 [ERROR] mysqld got signal 11 ;
      Jun 23 14:40:11 centos2 mysqld: This could be because you hit a bug. It is also possible that this binary
      Jun 23 14:40:11 centos2 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
      Jun 23 14:40:11 centos2 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
      Jun 23 14:40:11 centos2 mysqld: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
      Jun 23 14:40:11 centos2 mysqld: We will try our best to scrape up some info that will hopefully help
      Jun 23 14:40:11 centos2 mysqld: diagnose the problem, but since we have already crashed,
      Jun 23 14:40:11 centos2 mysqld: something is definitely wrong and this may fail.
      Jun 23 14:40:11 centos2 mysqld: Server version: 10.1.24-MariaDB
      Jun 23 14:40:11 centos2 mysqld: key_buffer_size=8589934592
      Jun 23 14:40:11 centos2 mysqld: read_buffer_size=1048576
      Jun 23 14:40:11 centos2 mysqld: max_used_connections=234
      Jun 23 14:40:11 centos2 mysqld: max_threads=1026
      Jun 23 14:40:11 centos2 mysqld: thread_count=67
      Jun 23 14:40:11 centos2 mysqld: It is possible that mysqld could use up to
      Jun 23 14:40:11 centos2 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11561556 K  bytes of memory
      Jun 23 14:40:11 centos2 mysqld: Hope that's ok; if not, decrease some variables in the equation.
      Jun 23 14:40:11 centos2 mysqld: Thread pointer: 0x7ef8d9580008
      Jun 23 14:40:11 centos2 mysqld: Attempting backtrace. You can use the following information to find out
      Jun 23 14:40:11 centos2 mysqld: where mysqld died. If you see no messages after this, something went
      Jun 23 14:40:11 centos2 mysqld: terribly wrong...
      Jun 23 14:40:11 centos2 mysqld: stack_bottom = 0x7ef8c48af0b0 thread_stack 0x48400
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f03b895433e]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x305)[0x7f03b847a0a5]
      Jun 23 14:40:11 centos2 mysqld: /lib64/libpthread.so.0(+0xf370)[0x7f03b7a92370]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(_ZN17Query_cache_block11headers_lenEv+0x0)[0x7f03b82b9b10]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(_ZN17Query_cache_block4dataEv+0x11)[0x7f03b82b9b31]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(_ZN11Query_cache6insertEP3THDP15Query_cache_tlsPKcmj+0x5c)[0x7f03b82bd52c]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(net_real_write+0x45)[0x7f03b8267cf5]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(net_flush+0x23)[0x7f03b8268033]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(net_write_command+0x3fb)[0x7f03b826879b]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(cli_advanced_command+0xe2)[0x7f03b8459d32]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(mysql_close_slow_part+0x54)[0x7f03b84572f4]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(mysql_close+0x1a)[0x7f03b845732a]
      Jun 23 14:40:11 centos2 mysqld: /usr/lib64/mysql/plugin/ha_federatedx.so(_ZN19federatedx_io_mysqlD1Ev+0x24)[0x7f03b11f8ee4]
      Jun 23 14:40:11 centos2 mysqld: /usr/lib64/mysql/plugin/ha_federatedx.so(_ZN14federatedx_txn5closeEP18st_fedrated_server+0x69)[0x7f03b11f8289]
      Jun 23 14:40:11 centos2 mysqld: /usr/lib64/mysql/plugin/ha_federatedx.so(+0xa62f)[0x7f03b11f162f]
      Jun 23 14:40:11 centos2 mysqld: /usr/lib64/mysql/plugin/ha_federatedx.so(+0xa7e5)[0x7f03b11f17e5]
      Jun 23 14:40:11 centos2 mysqld: mysys/stacktrace.c:268(my_print_stacktrace)[0x7f03b11f52b8]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_cache.cc:844(Query_cache_block::headers_len())[0x7f03b839b6c8]
      Jun 23 14:40:11 centos2 mysqld: sql-common/client.c:701(cli_advanced_command)[0x7f03b82ace26]
      Jun 23 14:40:11 centos2 mysqld: /usr/sbin/mysqld(_Z12tc_add_tableP3THDP5TABLE+0x42c)[0x7f03b842e13c]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_string.h:312(String::free())[0x7f03b82af87d]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_base.cc:4067(open_tables(THD*, DDL_options_st const&, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x7f03b82b37a0]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_base.cc:5258(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x7f03b82b3b34]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_parse.cc:5843(execute_sqlcom_select(THD*, TABLE_LIST*))[0x7f03b82ec91d]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_parse.cc:2979(mysql_execute_command(THD*))[0x7f03b82f8fbb]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_parse.cc:7340(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x7f03b82fc592]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_parse.cc:1490(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x7f03b82ffa6a]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_parse.cc:1111(do_command(THD*))[0x7f03b83002ca]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_connect.cc:1349(do_handle_one_connection(THD*))[0x7f03b83c839a]
      Jun 23 14:40:11 centos2 mysqld: sql/sql_connect.cc:1263(handle_one_connection)[0x7f03b83c8540]
      Jun 23 14:40:11 centos2 mysqld: /lib64/libpthread.so.0(+0x7dc5)[0x7f03b7a8adc5]
      Jun 23 14:40:11 centos2 mysqld: /lib64/libc.so.6(clone+0x6d)[0x7f03b5ea976d]
      Jun 23 14:40:11 centos2 mysqld: Trying to get some variables.
      Jun 23 14:40:11 centos2 mysqld: Some pointers may be invalid and cause the dump to abort.
      

      Jun 23 14:40:11 centos2 mysqld: Query (0x7f01b0a29020): SELECT u.id AS u__id, u.created_at AS u__created_at, u.created_by AS u__created_by, u.modified_at AS u__modified_at, u.modi\
      fied_by AS u__modified_by, u.record_status AS u__record_status, u.user_name AS u__user_name, u.first_name AS u__first_name, u.last_name AS u__last_name, u.email AS u__email, u.pas\
      sword AS u__password, u.md5_password AS u__md5_password, u.group_id AS u__group_id, u.referred_by AS u__referred_by FROM user u WHERE (MD5(u.id) = '8f14e45fceea167a5a36dedd4bea254\
      3' AND u.password = '4bf199124c5570a7595614c3fa361b6c') LIMIT 1
      Jun 23 14:40:11 centos2 mysqld: Connection ID (thread ID): 1260
      Jun 23 14:40:11 centos2 mysqld: Status: NOT_KILLED
      Jun 23 14:40:11 centos2 mysqld: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engin\
      e_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_\
      rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on\
      ,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=off
      Jun 23 14:40:11 centos2 mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      Jun 23 14:40:11 centos2 mysqld: information that should help you find out what is causing the crash.
      Jun 23 14:40:11 centos2 abrt-hook-ccpp: Process 20150 (mysqld) of user 27 killed by SIGSEGV - dumping core
      Jun 23 14:43:30 centos2 abrt-hook-ccpp: Failed to create core_backtrace: waitpid failed: No child processes
      Jun 23 14:43:30 centos2 systemd: mariadb.service: main process exited, code=killed, status=11/SEGV
      Jun 23 14:43:30 centos2 systemd: Unit mariadb.service entered failed state.
      Jun 23 14:43:30 centos2 systemd: mariadb.service failed.
      Jun 23 14:43:33 centos2 abrt-server: Package 'MariaDB-server' isn't signed with proper key
      Jun 23 14:43:33 centos2 abrt-server: 'post-create' on '/var/spool/abrt/ccpp-2017-06-23-14:40:11-20150' exited with 1
      Jun 23 14:43:33 centos2 abrt-server: Deleting problem directory '/var/spool/abrt/ccpp-2017-06-23-14:40:11-20150'
      Jun 23 14:43:36 centos2 systemd: mariadb.service holdoff time over, scheduling restart.
      Jun 23 14:43:36 centos2 systemd: Starting MariaDB database server...
      

      Attachments

        Issue Links

          Activity

            Could you please paste SHOW CREATE TABLE user and SHOW INDEX IN user for that table which is used in the query above?
            It is probably a FEDERATED table, in this case please do the same for the referenced local table, and specify the configuration of the referenced server.
            Thanks.

            elenst Elena Stepanova added a comment - Could you please paste SHOW CREATE TABLE user and SHOW INDEX IN user for that table which is used in the query above? It is probably a FEDERATED table, in this case please do the same for the referenced local table, and specify the configuration of the referenced server. Thanks.
            pmonev Pavel Monev added a comment - - edited

            The table in that query is NOT a federated table however I do have a federated table in that database. (ir_users)
            I also looked at subsequent crashes and they do have queries related to the federated table.
            Here is an example:

            SELECT
            	iu.username,   
            	iu.firstName,   
            	iu.lastName,   
            	if(iu.imJob_project_id in (6, 7, 8, 14, 16), 'TSM', 'TSR') as role,  
            	case     
            		when iu.imJob_project_id in (2, 6, 9) then 'Access Wireless'     
            		when iu.imJob_project_id in (5, 8, 11) then 'Budget Mobile'     
            		when iu.imJob_project_id in (4, 7, 10) then 'Life Wireless'     
            		when iu.imJob_project_id in (13, 14, 15, 16) then 'Safelink'   
            	end as mv_client 
            FROM ir_users iu 
            left join user u on iu.id = u.ir_id 
            where   
            	iu.imJob_project_id in (2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16)   
            	and iu.RecordStatus = 1   
            	and u.id is null;
            

            MariaDB [awims]> show index in ir_users;
            +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            | Table    | Non_unique | Key_name     | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
            +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            | ir_users |          0 | PRIMARY      |            1 | ID           | NULL      |        NULL |     NULL | NULL   |      | REMOTE     |         |               |
            | ir_users |          0 | UserName     |            1 | UserName     | NULL      |        NULL |     NULL | NULL   | YES  | REMOTE     |         |               |
            | ir_users |          1 | RecordStatus |            1 | RecordStatus | NULL      |        NULL |     NULL | NULL   |      | REMOTE     |         |               |
            | ir_users |          1 | GroupID      |            1 | GroupID      | NULL      |        NULL |     NULL | NULL   |      | REMOTE     |         |               |
            | ir_users |          1 | CompanyID    |            1 | CompanyID    | NULL      |        NULL |     NULL | NULL   | YES  | REMOTE     |         |               |
            +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            5 rows in set (0.02 sec)
            

            MariaDB [awims]>
            | ir_users | CREATE TABLE `ir_users` (
              `ID` int(1) unsigned NOT NULL AUTO_INCREMENT,
              `ModifiedBy` int(1) unsigned NOT NULL DEFAULT '0',
              `ModifiedDT` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
              `CreatedBy` int(1) unsigned NOT NULL DEFAULT '0',
              `CreatedDT` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
              `RecordStatus` tinyint(1) NOT NULL DEFAULT '1',
              `LastLogin` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
              `UserName` varchar(64) DEFAULT NULL,
              `Password` tinytext,
              `GroupID` int(1) unsigned NOT NULL DEFAULT '0',
              `RepType` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `MessageBoard` tinyint(1) NOT NULL DEFAULT '0',
              `CompanyID` int(1) unsigned DEFAULT '0',
              `Prefix` tinytext,
              `FirstName` tinytext,
              `MiddleName` tinytext,
              `LastName` tinytext,
              `SSN` tinytext,
              `EmployeeID` tinytext,
              `Shirt` enum('-','XS','S','M','L','XL','XXL','XXXL','TBD') DEFAULT '-',
              `Sex` enum('-','M','F') DEFAULT '-',
              `Title` tinytext,
              `CompanyEmail` tinytext,
              `PersonalEmail` tinytext,
              `ManagerID` int(1) unsigned DEFAULT '0',
              `ManagerID2` int(1) unsigned DEFAULT '0',
              `HomePhone` tinytext,
              `OfficePhone` tinytext,
              `CellPhone` tinytext,
              `Pager` tinytext,
              `HomeAddress1` tinytext,
              `HomeAddress2` tinytext,
              `HomeCity` tinytext,
              `HomeStateID` int(1) unsigned DEFAULT '0',
              `HomeZIP` tinytext,
              `HomeCountryID` int(1) unsigned DEFAULT '0',
              `MailAddress1` tinytext,
              `MailAddress2` tinytext,
              `MailCity` tinytext,
              `MailStateID` int(1) unsigned DEFAULT '0',
              `MailZIP` tinytext,
              `MailCountryID` int(1) unsigned DEFAULT '0',
              `DOB` date DEFAULT '0000-00-00',
              `StartDate` date DEFAULT '0000-00-00',
              `LastDate` date DEFAULT '0000-00-00',
              `Insurance` tinytext,
              `InsExpDate` date DEFAULT NULL,
              `ReHire` tinyint(1) DEFAULT '1',
              `Exempt` enum('Contractor','NonExempt','Exempt') DEFAULT 'Exempt',
              `FullTime` tinyint(1) DEFAULT NULL,
              `Notes` varchar(1024) DEFAULT NULL,
              `SuperUser` tinyint(1) NOT NULL DEFAULT '0',
              `Trainer` tinyint(1) NOT NULL DEFAULT '1',
              `Merchandiser` tinyint(1) NOT NULL DEFAULT '1',
              `Attribute1` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `Attribute2` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `Attribute3` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `DemoDayRep` tinyint(4) NOT NULL DEFAULT '0',
              `Hint` tinytext,
              `PasswordHintAnswer` tinytext,
              `MarketID` int(1) unsigned DEFAULT '0',
              `MarketOther` tinytext,
              `BankName` tinytext,
              `BankAccount` tinytext,
              `RoutingNumber` tinytext,
              `BranchNumber` varchar(45) DEFAULT NULL,
              `PayRoll` tinyint(1) NOT NULL DEFAULT '0',
              `OwnCompanyName` tinytext,
              `EIN` tinytext,
              `Agreement` tinyint(1) DEFAULT '0',
              `DistrictID` int(10) unsigned NOT NULL DEFAULT '0',
              `AccountType` tinytext,
              `DLicense` varchar(64) DEFAULT NULL,
              `DLicenseStateID` int(10) NOT NULL DEFAULT '0',
              `BizLic` varchar(64) DEFAULT NULL,
              `W9BusinessName` tinytext,
              `W9BusinessType` varchar(45) DEFAULT NULL,
              `W9Exempt` tinyint(3) unsigned DEFAULT NULL,
              `W9Requester` tinytext,
              `W9Account` tinytext,
              `W9Date` date DEFAULT NULL,
              `W9DifferentBuzName` tinytext,
              `TriNetStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `OasisStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `PasswordResetCode` char(32) DEFAULT NULL,
              `LatY` float DEFAULT NULL,
              `LonX` float DEFAULT NULL,
              `imJob_project_id` int(10) unsigned DEFAULT NULL,
              `imJob_posting_id` int(10) unsigned DEFAULT NULL,
              PRIMARY KEY (`ID`),
              UNIQUE KEY `UserName` (`UserName`),
              KEY `RecordStatus` (`RecordStatus`),
              KEY `GroupID` (`GroupID`),
              KEY `CompanyID` (`CompanyID`)
            ) ENGINE=FEDERATED DEFAULT CHARSET=latin1 CONNECTION='mysql://federated:XXXXXXXX@XXXXXXXXXXXX:3306/imrepqa0706/users'
            

            The actual table is residing in MariaDB Server version: 10.1.20

            Create table:
             CREATE TABLE `users` (
              `ID` int(1) unsigned NOT NULL AUTO_INCREMENT,
              `ModifiedBy` int(1) unsigned NOT NULL DEFAULT '0',
              `ModifiedDT` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
              `CreatedBy` int(1) unsigned NOT NULL DEFAULT '0',
              `CreatedDT` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
              `RecordStatus` tinyint(1) NOT NULL DEFAULT '1',
              `LastLogin` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00',
              `UserName` varchar(64) DEFAULT NULL,
              `Password` tinytext,
              `GroupID` int(1) unsigned NOT NULL DEFAULT '0',
              `RepType` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `MessageBoard` tinyint(1) NOT NULL DEFAULT '0',
              `CompanyID` int(1) unsigned DEFAULT '0',
              `Prefix` tinytext,
              `FirstName` tinytext,
              `MiddleName` tinytext,
              `LastName` tinytext,
              `SSN` tinytext,
              `EmployeeID` tinytext,
              `Shirt` enum('-','XS','S','M','L','XL','XXL','XXXL','TBD') DEFAULT '-',
              `Sex` enum('-','M','F') DEFAULT '-',
              `Title` tinytext,
              `CompanyEmail` tinytext,
              `PersonalEmail` tinytext,
              `ManagerID` int(1) unsigned DEFAULT '0',
              `ManagerID2` int(1) unsigned DEFAULT '0',
              `HomePhone` tinytext,
              `OfficePhone` tinytext,
              `CellPhone` tinytext,
              `Pager` tinytext,
              `HomeAddress1` tinytext,
              `HomeAddress2` tinytext,
              `HomeCity` tinytext,
              `HomeStateID` int(1) unsigned DEFAULT '0',
              `HomeZIP` tinytext,
              `HomeCountryID` int(1) unsigned DEFAULT '0',
              `MailAddress1` tinytext,
              `MailAddress2` tinytext,
              `MailCity` tinytext,
              `MailStateID` int(1) unsigned DEFAULT '0',
              `MailZIP` tinytext,
              `MailCountryID` int(1) unsigned DEFAULT '0',
              `DOB` date DEFAULT '0000-00-00',
              `StartDate` date DEFAULT '0000-00-00',
              `LastDate` date DEFAULT '0000-00-00',
              `Insurance` tinytext,
              `InsExpDate` date DEFAULT NULL,
              `ReHire` tinyint(1) DEFAULT '1',
              `Exempt` enum('Contractor','NonExempt','Exempt') DEFAULT 'Exempt',
              `FullTime` tinyint(1) DEFAULT NULL,
              `Notes` varchar(1024) DEFAULT NULL,
              `SuperUser` tinyint(1) NOT NULL DEFAULT '0',
              `Trainer` tinyint(1) NOT NULL DEFAULT '1',
              `Merchandiser` tinyint(1) NOT NULL DEFAULT '1',
              `Attribute1` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `Attribute2` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `Attribute3` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `DemoDayRep` tinyint(4) NOT NULL DEFAULT '0',
              `Hint` tinytext,
              `PasswordHintAnswer` tinytext,
              `MarketID` int(1) unsigned DEFAULT '0',
              `MarketOther` tinytext,
              `BankName` tinytext,
              `BankAccount` tinytext,
              `RoutingNumber` tinytext,
              `BranchNumber` varchar(45) DEFAULT NULL,
              `PayRoll` tinyint(1) NOT NULL DEFAULT '0',
              `OwnCompanyName` tinytext,
              `EIN` tinytext,
              `Agreement` tinyint(1) DEFAULT '0',
              `DistrictID` int(10) unsigned NOT NULL DEFAULT '0',
              `AccountType` tinytext,
              `DLicense` varchar(64) DEFAULT NULL,
              `DLicenseStateID` int(10) NOT NULL DEFAULT '0',
              `BizLic` varchar(64) DEFAULT NULL,
              `W9BusinessName` tinytext,
              `W9BusinessType` varchar(45) DEFAULT NULL,
              `W9Exempt` tinyint(3) unsigned DEFAULT NULL,
              `W9Requester` tinytext,
              `W9Account` tinytext,
              `W9Date` date DEFAULT NULL,
              `W9DifferentBuzName` tinytext,
              `TriNetStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `OasisStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
              `PasswordResetCode` char(32) DEFAULT NULL,
              `LatY` float DEFAULT NULL,
              `LonX` float DEFAULT NULL,
              `imJob_project_id` int(10) unsigned DEFAULT NULL,
              `imJob_posting_id` int(10) unsigned DEFAULT NULL,
              PRIMARY KEY (`ID`),
              UNIQUE KEY `UserName` (`UserName`),
              KEY `RecordStatus` (`RecordStatus`),
              KEY `GroupID` (`GroupID`),
              KEY `CompanyID` (`CompanyID`)
            ) ENGINE=MyISAM AUTO_INCREMENT=15746 DEFAULT CHARSET=latin1
            

            MariaDB [imrepqa0706]> show index in users;
            +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            | Table | Non_unique | Key_name     | Seq_in_index | Column_name  | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
            +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            | users |          0 | PRIMARY      |            1 | ID           | A         |       15738 |     NULL | NULL   |      | BTREE      |         |               |
            | users |          0 | UserName     |            1 | UserName     | A         |        NULL |     NULL | NULL   | YES  | BTREE      |         |               |
            | users |          1 | RecordStatus |            1 | RecordStatus | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
            | users |          1 | GroupID      |            1 | GroupID      | A         |           6 |     NULL | NULL   |      | BTREE      |         |               |
            | users |          1 | CompanyID    |            1 | CompanyID    | A         |          23 |     NULL | NULL   | YES  | BTREE      |         |               |
            +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
            5 rows in set (0.00 sec)
            

            Server config:

            [root@centos1 pmonev]# mysqld --print-defaults
            mysqld would have been started with the following arguments:
            --port=3306 --socket=/var/lib/mysql/mysql.sock --skip-external-locking --key_buffer_size=8192M --max_allowed_packet=500M --table_open_cache=256 --sort_buffer_size=2M --read_buffer_size=1M --read_rnd_buffer_size=4M --myisam_sort_buffer_size=64M --thread_cache_size=8 --query_cache_size=16M --thread_concurrency=8 --symbolic-links=0 --max_connections=256 --lower_case_table_names=1 --innodb_data_home_dir=/var/lib/mysql --innodb_data_file_path=ibdata1:10M:autoextend --innodb_log_group_home_dir=/var/lib/mysql --innodb_buffer_pool_size=8192M --innodb_additional_mem_pool_size=20M --innodb_log_file_size=625M --innodb_log_buffer_size=256M --innodb_flush_log_at_trx_commit=2 --innodb_lock_wait_timeout=50 --bind-address=10.101.10.38
            

            The use of the federated table in my case is not that important so I dropped it just to see if that will stop the server from crashing.

            I can bring the table back if you need more information or logs.

            pmonev Pavel Monev added a comment - - edited The table in that query is NOT a federated table however I do have a federated table in that database. (ir_users) I also looked at subsequent crashes and they do have queries related to the federated table. Here is an example: SELECT iu.username, iu.firstName, iu.lastName, if (iu.imJob_project_id in (6, 7, 8, 14, 16), 'TSM' , 'TSR' ) as role, case when iu.imJob_project_id in (2, 6, 9) then 'Access Wireless' when iu.imJob_project_id in (5, 8, 11) then 'Budget Mobile' when iu.imJob_project_id in (4, 7, 10) then 'Life Wireless' when iu.imJob_project_id in (13, 14, 15, 16) then 'Safelink' end as mv_client FROM ir_users iu left join user u on iu.id = u.ir_id where iu.imJob_project_id in (2, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16) and iu.RecordStatus = 1 and u.id is null ; MariaDB [awims]> show index in ir_users; +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | ir_users | 0 | PRIMARY | 1 | ID | NULL | NULL | NULL | NULL | | REMOTE | | | | ir_users | 0 | UserName | 1 | UserName | NULL | NULL | NULL | NULL | YES | REMOTE | | | | ir_users | 1 | RecordStatus | 1 | RecordStatus | NULL | NULL | NULL | NULL | | REMOTE | | | | ir_users | 1 | GroupID | 1 | GroupID | NULL | NULL | NULL | NULL | | REMOTE | | | | ir_users | 1 | CompanyID | 1 | CompanyID | NULL | NULL | NULL | NULL | YES | REMOTE | | | +----------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ 5 rows in set (0.02 sec) MariaDB [awims]> | ir_users | CREATE TABLE `ir_users` ( `ID` int (1) unsigned NOT NULL AUTO_INCREMENT, `ModifiedBy` int (1) unsigned NOT NULL DEFAULT '0' , `ModifiedDT` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP , `CreatedBy` int (1) unsigned NOT NULL DEFAULT '0' , `CreatedDT` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' , `RecordStatus` tinyint(1) NOT NULL DEFAULT '1' , `LastLogin` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' , `UserName` varchar (64) DEFAULT NULL , ` Password ` tinytext, `GroupID` int (1) unsigned NOT NULL DEFAULT '0' , `RepType` tinyint(3) unsigned NOT NULL DEFAULT '0' , `MessageBoard` tinyint(1) NOT NULL DEFAULT '0' , `CompanyID` int (1) unsigned DEFAULT '0' , `Prefix` tinytext, `FirstName` tinytext, `MiddleName` tinytext, `LastName` tinytext, `SSN` tinytext, `EmployeeID` tinytext, `Shirt` enum( '-' , 'XS' , 'S' , 'M' , 'L' , 'XL' , 'XXL' , 'XXXL' , 'TBD' ) DEFAULT '-' , `Sex` enum( '-' , 'M' , 'F' ) DEFAULT '-' , `Title` tinytext, `CompanyEmail` tinytext, `PersonalEmail` tinytext, `ManagerID` int (1) unsigned DEFAULT '0' , `ManagerID2` int (1) unsigned DEFAULT '0' , `HomePhone` tinytext, `OfficePhone` tinytext, `CellPhone` tinytext, `Pager` tinytext, `HomeAddress1` tinytext, `HomeAddress2` tinytext, `HomeCity` tinytext, `HomeStateID` int (1) unsigned DEFAULT '0' , `HomeZIP` tinytext, `HomeCountryID` int (1) unsigned DEFAULT '0' , `MailAddress1` tinytext, `MailAddress2` tinytext, `MailCity` tinytext, `MailStateID` int (1) unsigned DEFAULT '0' , `MailZIP` tinytext, `MailCountryID` int (1) unsigned DEFAULT '0' , `DOB` date DEFAULT '0000-00-00' , `StartDate` date DEFAULT '0000-00-00' , `LastDate` date DEFAULT '0000-00-00' , `Insurance` tinytext, `InsExpDate` date DEFAULT NULL , `ReHire` tinyint(1) DEFAULT '1' , `Exempt` enum( 'Contractor' , 'NonExempt' , 'Exempt' ) DEFAULT 'Exempt' , `FullTime` tinyint(1) DEFAULT NULL , `Notes` varchar (1024) DEFAULT NULL , `SuperUser` tinyint(1) NOT NULL DEFAULT '0' , `Trainer` tinyint(1) NOT NULL DEFAULT '1' , `Merchandiser` tinyint(1) NOT NULL DEFAULT '1' , `Attribute1` tinyint(3) unsigned NOT NULL DEFAULT '0' , `Attribute2` tinyint(3) unsigned NOT NULL DEFAULT '0' , `Attribute3` tinyint(3) unsigned NOT NULL DEFAULT '0' , `DemoDayRep` tinyint(4) NOT NULL DEFAULT '0' , `Hint` tinytext, `PasswordHintAnswer` tinytext, `MarketID` int (1) unsigned DEFAULT '0' , `MarketOther` tinytext, `BankName` tinytext, `BankAccount` tinytext, `RoutingNumber` tinytext, `BranchNumber` varchar (45) DEFAULT NULL , `PayRoll` tinyint(1) NOT NULL DEFAULT '0' , `OwnCompanyName` tinytext, `EIN` tinytext, `Agreement` tinyint(1) DEFAULT '0' , `DistrictID` int (10) unsigned NOT NULL DEFAULT '0' , `AccountType` tinytext, `DLicense` varchar (64) DEFAULT NULL , `DLicenseStateID` int (10) NOT NULL DEFAULT '0' , `BizLic` varchar (64) DEFAULT NULL , `W9BusinessName` tinytext, `W9BusinessType` varchar (45) DEFAULT NULL , `W9Exempt` tinyint(3) unsigned DEFAULT NULL , `W9Requester` tinytext, `W9Account` tinytext, `W9Date` date DEFAULT NULL , `W9DifferentBuzName` tinytext, `TriNetStatus` tinyint(3) unsigned NOT NULL DEFAULT '0' , `OasisStatus` tinyint(3) unsigned NOT NULL DEFAULT '0' , `PasswordResetCode` char (32) DEFAULT NULL , `LatY` float DEFAULT NULL , `LonX` float DEFAULT NULL , `imJob_project_id` int (10) unsigned DEFAULT NULL , `imJob_posting_id` int (10) unsigned DEFAULT NULL , PRIMARY KEY (`ID`), UNIQUE KEY `UserName` (`UserName`), KEY `RecordStatus` (`RecordStatus`), KEY `GroupID` (`GroupID`), KEY `CompanyID` (`CompanyID`) ) ENGINE=FEDERATED DEFAULT CHARSET=latin1 CONNECTION = 'mysql://federated:XXXXXXXX@XXXXXXXXXXXX:3306/imrepqa0706/users' The actual table is residing in MariaDB Server version: 10.1.20 Create table : CREATE TABLE `users` ( `ID` int (1) unsigned NOT NULL AUTO_INCREMENT, `ModifiedBy` int (1) unsigned NOT NULL DEFAULT '0' , `ModifiedDT` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP , `CreatedBy` int (1) unsigned NOT NULL DEFAULT '0' , `CreatedDT` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' , `RecordStatus` tinyint(1) NOT NULL DEFAULT '1' , `LastLogin` timestamp NOT NULL DEFAULT '0000-00-00 00:00:00' , `UserName` varchar (64) DEFAULT NULL , ` Password ` tinytext, `GroupID` int (1) unsigned NOT NULL DEFAULT '0' , `RepType` tinyint(3) unsigned NOT NULL DEFAULT '0' , `MessageBoard` tinyint(1) NOT NULL DEFAULT '0' , `CompanyID` int (1) unsigned DEFAULT '0' , `Prefix` tinytext, `FirstName` tinytext, `MiddleName` tinytext, `LastName` tinytext, `SSN` tinytext, `EmployeeID` tinytext, `Shirt` enum( '-' , 'XS' , 'S' , 'M' , 'L' , 'XL' , 'XXL' , 'XXXL' , 'TBD' ) DEFAULT '-' , `Sex` enum( '-' , 'M' , 'F' ) DEFAULT '-' , `Title` tinytext, `CompanyEmail` tinytext, `PersonalEmail` tinytext, `ManagerID` int (1) unsigned DEFAULT '0' , `ManagerID2` int (1) unsigned DEFAULT '0' , `HomePhone` tinytext, `OfficePhone` tinytext, `CellPhone` tinytext, `Pager` tinytext, `HomeAddress1` tinytext, `HomeAddress2` tinytext, `HomeCity` tinytext, `HomeStateID` int (1) unsigned DEFAULT '0' , `HomeZIP` tinytext, `HomeCountryID` int (1) unsigned DEFAULT '0' , `MailAddress1` tinytext, `MailAddress2` tinytext, `MailCity` tinytext, `MailStateID` int (1) unsigned DEFAULT '0' , `MailZIP` tinytext, `MailCountryID` int (1) unsigned DEFAULT '0' , `DOB` date DEFAULT '0000-00-00' , `StartDate` date DEFAULT '0000-00-00' , `LastDate` date DEFAULT '0000-00-00' , `Insurance` tinytext, `InsExpDate` date DEFAULT NULL , `ReHire` tinyint(1) DEFAULT '1' , `Exempt` enum( 'Contractor' , 'NonExempt' , 'Exempt' ) DEFAULT 'Exempt' , `FullTime` tinyint(1) DEFAULT NULL , `Notes` varchar (1024) DEFAULT NULL , `SuperUser` tinyint(1) NOT NULL DEFAULT '0' , `Trainer` tinyint(1) NOT NULL DEFAULT '1' , `Merchandiser` tinyint(1) NOT NULL DEFAULT '1' , `Attribute1` tinyint(3) unsigned NOT NULL DEFAULT '0' , `Attribute2` tinyint(3) unsigned NOT NULL DEFAULT '0' , `Attribute3` tinyint(3) unsigned NOT NULL DEFAULT '0' , `DemoDayRep` tinyint(4) NOT NULL DEFAULT '0' , `Hint` tinytext, `PasswordHintAnswer` tinytext, `MarketID` int (1) unsigned DEFAULT '0' , `MarketOther` tinytext, `BankName` tinytext, `BankAccount` tinytext, `RoutingNumber` tinytext, `BranchNumber` varchar (45) DEFAULT NULL , `PayRoll` tinyint(1) NOT NULL DEFAULT '0' , `OwnCompanyName` tinytext, `EIN` tinytext, `Agreement` tinyint(1) DEFAULT '0' , `DistrictID` int (10) unsigned NOT NULL DEFAULT '0' , `AccountType` tinytext, `DLicense` varchar (64) DEFAULT NULL , `DLicenseStateID` int (10) NOT NULL DEFAULT '0' , `BizLic` varchar (64) DEFAULT NULL , `W9BusinessName` tinytext, `W9BusinessType` varchar (45) DEFAULT NULL , `W9Exempt` tinyint(3) unsigned DEFAULT NULL , `W9Requester` tinytext, `W9Account` tinytext, `W9Date` date DEFAULT NULL , `W9DifferentBuzName` tinytext, `TriNetStatus` tinyint(3) unsigned NOT NULL DEFAULT '0' , `OasisStatus` tinyint(3) unsigned NOT NULL DEFAULT '0' , `PasswordResetCode` char (32) DEFAULT NULL , `LatY` float DEFAULT NULL , `LonX` float DEFAULT NULL , `imJob_project_id` int (10) unsigned DEFAULT NULL , `imJob_posting_id` int (10) unsigned DEFAULT NULL , PRIMARY KEY (`ID`), UNIQUE KEY `UserName` (`UserName`), KEY `RecordStatus` (`RecordStatus`), KEY `GroupID` (`GroupID`), KEY `CompanyID` (`CompanyID`) ) ENGINE=MyISAM AUTO_INCREMENT=15746 DEFAULT CHARSET=latin1 MariaDB [imrepqa0706]> show index in users; +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ | users | 0 | PRIMARY | 1 | ID | A | 15738 | NULL | NULL | | BTREE | | | | users | 0 | UserName | 1 | UserName | A | NULL | NULL | NULL | YES | BTREE | | | | users | 1 | RecordStatus | 1 | RecordStatus | A | 2 | NULL | NULL | | BTREE | | | | users | 1 | GroupID | 1 | GroupID | A | 6 | NULL | NULL | | BTREE | | | | users | 1 | CompanyID | 1 | CompanyID | A | 23 | NULL | NULL | YES | BTREE | | | +-------+------------+--------------+--------------+--------------+-----------+-------------+----------+--------+------+------------+---------+---------------+ 5 rows in set (0.00 sec) Server config: [root@centos1 pmonev]# mysqld --print-defaults mysqld would have been started with the following arguments: --port=3306 --socket=/var/lib/mysql/mysql.sock --skip-external-locking --key_buffer_size=8192M --max_allowed_packet=500M --table_open_cache=256 --sort_buffer_size=2M --read_buffer_size=1M --read_rnd_buffer_size=4M --myisam_sort_buffer_size=64M --thread_cache_size=8 --query_cache_size=16M --thread_concurrency=8 --symbolic-links=0 --max_connections=256 --lower_case_table_names=1 --innodb_data_home_dir=/var/lib/mysql --innodb_data_file_path=ibdata1:10M:autoextend --innodb_log_group_home_dir=/var/lib/mysql --innodb_buffer_pool_size=8192M --innodb_additional_mem_pool_size=20M --innodb_log_file_size=625M --innodb_log_buffer_size=256M --innodb_flush_log_at_trx_commit=2 --innodb_lock_wait_timeout=50 --bind-address=10.101.10.38 The use of the federated table in my case is not that important so I dropped it just to see if that will stop the server from crashing. I can bring the table back if you need more information or logs.

            pmonev, thanks for the information.
            We were able to reproduce the bug MDEV-12725, to which your stack trace appears to be identical, so hopefully we won't need you to bring the Federated table back. However, we are very interested in results of our current experiment, whether the server keeps crashing after you got rid of the federated table. I will keep the issue open until you confirm the problem is gone (in which case it will be a duplicate of MDEV-12725), or that the problem persists, in which case we will need to investigate further.

            elenst Elena Stepanova added a comment - pmonev , thanks for the information. We were able to reproduce the bug MDEV-12725 , to which your stack trace appears to be identical, so hopefully we won't need you to bring the Federated table back. However, we are very interested in results of our current experiment, whether the server keeps crashing after you got rid of the federated table. I will keep the issue open until you confirm the problem is gone (in which case it will be a duplicate of MDEV-12725 ), or that the problem persists, in which case we will need to investigate further.

            pmonev,

            Have you had any crashes since you had gotten rid of the Federated table?

            elenst Elena Stepanova added a comment - pmonev , Have you had any crashes since you had gotten rid of the Federated table?
            pmonev Pavel Monev added a comment -

            No crashes since I stopped using Federated table.

            Regards.

            pmonev Pavel Monev added a comment - No crashes since I stopped using Federated table. Regards.

            Thank you. In this case we will consider it a duplicate of MDEV-12725, and further progress will be tracked there.

            elenst Elena Stepanova added a comment - Thank you. In this case we will consider it a duplicate of MDEV-12725 , and further progress will be tracked there.

            People

              Unassigned Unassigned
              pmonev Pavel Monev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.