[MDEV-9051] mysqld got signal 11, after upgrade to 10.1.8 Created: 2015-11-01  Updated: 2015-11-18  Resolved: 2015-11-18

Status: Closed
Project: MariaDB Server
Component/s: Plugin - feedback, Plugins
Affects Version/s: 10.1.8, 5.5, 10.0, 10.1
Fix Version/s: 5.5.47, 10.0.23, 10.1.9

Type: Bug Priority: Critical
Reporter: Mohit Chandel Assignee: Sergei Golubchik
Resolution: Fixed Votes: 0
Labels: feedback
Environment:

Debian 8.2,Linux Server0 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u5 (2015-10-09) x86_64 GNU/Linux

Attachments: Text File Server 1 Stacktrace.txt     Text File Server 1 binlog 5.txt     Text File Server 1 binlog 6.txt     File Server 1 my.cnf     Text File Server 2 Stacktrace.txt     File Server2 my.cnf    
Issue Links:
Relates
relates to MDEV-9087 Assertion `(longlong) thd->status_var... Closed
relates to MDEV-9088 Server crashes on shutdown after the ... Closed
Sprint: 10.2.0-1

 Description   

After upgrading Mariadb 10.1.7 to 10.1.8 MariaDB is crashing with Signal 11 after about 24 hours of upgrade. I am using MariaDB Repo.

I am having two Servers running different applications and are not related to each other, both running MariaDB 10.1.7 and were upgraded to 10.1.8 at approx same time. After approx. 24 hours both Crashed with Signal 11 with a stack trace in the error log.

Now I have restarted both the Servers and everything seems fine. There are no signs of data corruption. Stack trace for both the Serves is nearly identical.

Before upgrade to 10.1.8, both the Servers were running perfectly fine for months together. I am not using replication on any of the servers.

Show Engines on both the Servers:
CSV
SEQUENCE
MRG_MyISAM
MyISAM
CONNECT
InnoDB
MEMORY
Aria
PERFORMANCE_SCHEMA

Attached my.cnf for both servers and binary logs of Server 1.



 Comments   
Comment by Elena Stepanova [ 2015-11-01 ]

mchandel,

I see in your config files that you have general log enabled.
Would you be able to extract the parts of the logs before the crashes happened, and paste/attach them, or if you don't want to make them public, upload them to our ftp.askmonty.org/private?

Comment by Mohit Chandel [ 2015-11-02 ]

Uploaded General Log just before the crash for both the servers to ftp.askmonty.org/private.

Log File Name:

MDEV-9051-Server 1 general_log.txt
MDEV-9051-Server 2 general_log.txt

Both the Servers were restarted at approx 01-11-2015 1300hrs.

Comment by Mohit Chandel [ 2015-11-02 ]

MariaDB on Server 1 & 2 again crashed with Signal 11 after approx 24 hours of restarting with the same stacktrace on the error log.

These queries were running fine on 10.1.7. One more thing mysql sys schema 5.6 was installed on Server 1 and logrotate is configured on both the servers for log file rotation. Event Scheduler is enabled(running 6 events) on Server 2 and Connect is also configured on Server 2 for reading data from remote Oracle Servers (2 no.) and remote Sybase Server (1 no.).

General log before the crash and error log uploaded in FTP server.

File Names:
MDEV-9051 Server 1 General log 20151102.txt
MDEV-9051 Server 1 Error log 20151102.txt
MDEV-9051 Server 2 Error log 20151102.txt
MDEV-9051 Server 1 Error log 20151101.txt

I am planning to revert to 10.1.7 within a day or two, if issue is not resolved. Till then any information required can be provided by me.

Comment by Elena Stepanova [ 2015-11-04 ]

mchandel, thank you.
From "Server 1 20151102" logs, the suspicious queries use tables caf_data, caf_scan, caf_scan_log.
Can you paste SHOW CREATE TABLE and SHOW INDEX IN for these tables? (Again, if you prefer it to stay private, you can upload it to FTP).
It will be even better if you can provide the full data dump.

Comment by Mohit Chandel [ 2015-11-04 ]

elenst

Output of the the SHOW CREATE TABLE and SHOW INDEX IN . Can't upload the Data as it contains private information.

SHOW CREATE TABLE caf_data

 
CREATE TABLE `caf_data` (
 
  `recordNo` int(10) unsigned NOT NULL AUTO_INCREMENT,
 
  `insertRecordTs` timestamp NULL DEFAULT NULL,
 
  `updateRecordTs` timestamp NULL DEFAULT NULL,
 
  `inactiveDate` date DEFAULT NULL,
 
  `msisdn` char(10) NOT NULL,
 
  `uniqueCafNo` char(10) DEFAULT NULL,
 
  `uidNo` char(14) DEFAULT NULL,
 
  `customerName` varchar(100) NOT NULL,
 
  `dateOfBirth` date DEFAULT NULL,
 
  `fatherName` varchar(100) DEFAULT NULL,
 
  `address1` varchar(100) DEFAULT NULL,
 
  `address2` varchar(100) DEFAULT NULL,
 
  `city` varchar(80) DEFAULT NULL,
 
  `district` varchar(50) DEFAULT NULL,
 
  `state` varchar(30) DEFAULT '',
 
  `pin` varchar(6) DEFAULT NULL,
 
  `permanentAddress1` tinytext,
 
  `permanentAddress2` varchar(100) DEFAULT NULL,
 
  `permanentCity` varchar(80) DEFAULT NULL,
 
  `permanentDistrict` varchar(50) DEFAULT NULL,
 
  `permanentState` varchar(30) DEFAULT NULL,
 
  `permanentPin` varchar(6) DEFAULT NULL,
 
  `altContactNo` varchar(20) DEFAULT NULL,
 
  `email` varchar(100) DEFAULT NULL,
 
  `gender` varchar(6) DEFAULT NULL,
 
  `nationality` varchar(20) DEFAULT NULL,
 
  `profession` varchar(20) DEFAULT NULL,
 
  `panNo` varchar(20) DEFAULT NULL,
 
  `poi` varchar(50) DEFAULT NULL,
 
  `poiNo` varchar(30) DEFAULT NULL,
 
  `poiIssueDate` date DEFAULT NULL,
 
  `poiIssuePlace` varchar(30) DEFAULT NULL,
 
  `poiIssueAuthority` varchar(50) DEFAULT NULL,
 
  `poa` varchar(50) DEFAULT NULL,
 
  `poaNo` varchar(30) DEFAULT NULL,
 
  `poaIssueDate` date DEFAULT NULL,
 
  `poaIssuePlace` varchar(30) DEFAULT NULL,
 
  `poaIssueAuthority` varchar(50) DEFAULT NULL,
 
  `customerType` varchar(30) DEFAULT NULL,
 
  `accountType` varchar(15) DEFAULT NULL,
 
  `imsi` char(15) DEFAULT NULL,
 
  `actDate` date DEFAULT NULL,
 
  `curStatus` varchar(8) DEFAULT NULL,
 
  `posCode` varchar(128) DEFAULT NULL,
 
  `posName` varchar(64) DEFAULT NULL,
 
  `posAddress1` varchar(120) DEFAULT NULL,
 
  `posAddress2` varchar(100) DEFAULT NULL,
 
  `posCity` varchar(80) DEFAULT NULL,
 
  `posDistrict` varchar(50) DEFAULT NULL,
 
  `posState` varchar(30) DEFAULT NULL,
 
  `posPin` varchar(6) DEFAULT NULL,
 
  `actAgent` tinytext,
 
  `printCenter` varchar(16) DEFAULT NULL,
 
  `localityType` varchar(5) DEFAULT NULL,
 
  `bsnlEmployeeCode` varchar(9) DEFAULT NULL,
 
  PRIMARY KEY (`recordNo`),
 
  KEY `idx-msisdn` (`msisdn`),
 
  KEY `idx-customerName` (`customerName`),
 
  KEY `idx-actDate` (`actDate`),
 
  KEY `idx-printCenter` (`printCenter`)
 
) ENGINE=InnoDB AUTO_INCREMENT=2619005 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;

SHOW INDEX IN caf_data

 
Table	Non_unique	Key_name	Seq_in_index	Column_name	Collation	Cardinality	Sub_part	Packed	Null	Index_type	Comment	Index_comment
 
caf_data	0	PRIMARY	1	recordNo	A	1542095	\N	\N		BTREE		
caf_data	1	idx-msisdn	1	msisdn	A	1531983	\N	\N		BTREE		
caf_data	1	idx-customerName	1	customerName	A	318134	\N	\N		BTREE		
caf_data	1	idx-actDate	1	actDate	A	4209	\N	\N	YES	BTREE		
caf_data	1	idx-printCenter	1	printCenter	A	7	\N	\N	YES	BTREE

SHOW CREATE TABLE caf_scan

 
CREATE TABLE `caf_scan` (
 
  `recordNo` int(10) unsigned NOT NULL,
 
  `cafType` char(3) DEFAULT NULL,
 
  `ssaCode` char(3) NOT NULL,
 
  `cscCode` char(3) NOT NULL,
 
  `lotNo` int(5) unsigned zerofill DEFAULT NULL,
 
  `fullLotNo` char(9) AS (IF(ISNULL(lotNo), NULL, CONCAT_WS('-',`ssaCode`,`lotNo`))) PERSISTENT,
 
  `serialNo` smallint(3) unsigned zerofill DEFAULT NULL,
 
  `scanStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
 
  `scanDoneBy` varchar(6) DEFAULT NULL,
 
  `auditCaf` char(3) NOT NULL DEFAULT 'NO',
 
  `auditDoneBy` varchar(6) DEFAULT NULL,
 
  `qcRejectReason` enum('INDEX_DATA','SCAN_QUALITY') DEFAULT NULL,
 
  `whLotCreateUser` char(9) DEFAULT NULL,
 
  `whLotCreateTs` timestamp NULL DEFAULT NULL,
 
  `whLocUpdateUser` varchar(9) DEFAULT NULL,
 
  `whLocUpdateTs` timestamp NULL DEFAULT NULL,
 
  `whAddress` tinytext,
 
  `whRack` tinyint(3) unsigned DEFAULT NULL,
 
  `whShelf` tinyint(3) unsigned DEFAULT NULL,
 
  `whPosition` tinyint(3) unsigned DEFAULT NULL,
 
  PRIMARY KEY (`recordNo`),
 
  KEY `idx-scanStatus` (`scanStatus`),
 
  KEY `idx-qcRejectReason` (`qcRejectReason`),
 
  KEY `idx-ssa-csc-scanStatus` (`ssaCode`,`cscCode`,`scanStatus`),
 
  KEY `idx-whLotCreateUser-user_data-loginId` (`whLotCreateUser`),
 
  KEY `idx-whLocUpdateUser-user_data-loginId` (`whLocUpdateUser`),
 
  CONSTRAINT `fk-recordNo-caf_data-recordNo_0` FOREIGN KEY (`recordNo`) REFERENCES `caf_data` (`recordNo`),
 
  CONSTRAINT `fk-whLocUpdateUser-user_data-loginId` FOREIGN KEY (`whLocUpdateUser`) REFERENCES `user_data` (`loginId`),
 
  CONSTRAINT `fk-whLotCreateUser-user_data-loginId` FOREIGN KEY (`whLotCreateUser`) REFERENCES `user_data` (`loginId`)
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;

SHOW INDEX IN caf_scan

 
Table	Non_unique	Key_name	Seq_in_index	Column_name	Collation	Cardinality	Sub_part	Packed	Null	Index_type	Comment	Index_comment
 
caf_scan	0	PRIMARY	1	recordNo	A	561568	\N	\N		BTREE		
caf_scan	1	idx-scanStatus	1	scanStatus	A	12	\N	\N		BTREE		
caf_scan	1	idx-qcRejectReason	1	qcRejectReason	A	439	\N	\N	YES	BTREE		
caf_scan	1	idx-ssa-csc-scanStatus	1	ssaCode	A	7	\N	\N		BTREE		
caf_scan	1	idx-ssa-csc-scanStatus	2	cscCode	A	80	\N	\N		BTREE		
caf_scan	1	idx-ssa-csc-scanStatus	3	scanStatus	A	260	\N	\N		BTREE		
caf_scan	1	idx-whLotCreateUser-user_data-loginId	1	whLotCreateUser	A	239	\N	\N	YES	BTREE		
caf_scan	1	idx-whLocUpdateUser-user_data-loginId	1	whLocUpdateUser	A	1871	\N	\N	YES	BTREE

SHOW CREATE TABLE caf_scan_log

 
CREATE TABLE `caf_scan_log` (
 
  `rowId` int(10) unsigned NOT NULL AUTO_INCREMENT,
 
  `recordNo` int(10) unsigned NOT NULL,
 
  `scanStatusUpdateTs` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
 
  `scanStatusUpdateUser` char(9) NOT NULL,
 
  `scanStatus` tinyint(3) unsigned NOT NULL,
 
  PRIMARY KEY (`rowId`),
 
  UNIQUE KEY `idx-recordNo-scanStatus` (`recordNo`,`scanStatus`),
 
  KEY `idx-scanStatusUpdateUser-user_data-loginId` (`scanStatusUpdateUser`),
 
  CONSTRAINT `fk-recordNo-caf_data-recordNo_1` FOREIGN KEY (`recordNo`) REFERENCES `caf_data` (`recordNo`),
 
  CONSTRAINT `fk-scanStatusUpdateUser-user-loginId` FOREIGN KEY (`scanStatusUpdateUser`) REFERENCES `user_data` (`loginId`)
 
) ENGINE=InnoDB AUTO_INCREMENT=5600727 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;

SHOW INDEX IN caf_scan_log

 
Table	Non_unique	Key_name	Seq_in_index	Column_name	Collation	Cardinality	Sub_part	Packed	Null	Index_type	Comment	Index_comment
 
caf_scan_log	0	PRIMARY	1	rowId	A	1662091	\N	\N		BTREE		
caf_scan_log	0	idx-recordNo-scanStatus	1	recordNo	A	561574	\N	\N		BTREE		
caf_scan_log	0	idx-recordNo-scanStatus	2	scanStatus	A	1662091	\N	\N		BTREE		
caf_scan_log	1	idx-scanStatusUpdateUser-user_data-loginId	1	scanStatusUpdateUser	A	124	\N	\N		BTREE

Comment by Elena Stepanova [ 2015-11-12 ]

mchandel, thanks for the data.
Please try to disable the FEEDBACK plugin for now, and let me know whether it helps – I expect it should.

Comment by Mohit Chandel [ 2015-11-12 ]

elenst

Set feedback = OFF in my.cnf file.

Waiting for results. If anything else is to be done pls let me know.

Comment by Elena Stepanova [ 2015-11-12 ]

Reproducible with static and dynamic FEEDBACK plugin, with sql_mode=ONLY_FULL_GROUP_BY, on 5.5, 10.0, 10.1.
The crash happens on the 2nd sending of the report, which normally happens in ~24h from the server startup. To reproduce it in reasonable time, I modified sender_thread.cc

diff --git a/plugin/feedback/sender_thread.cc b/plugin/feedback/sender_thread.cc
 
index 732118f..c3a46db 100644
 
--- a/plugin/feedback/sender_thread.cc
 
+++ b/plugin/feedback/sender_thread.cc
 
@@ -26,8 +26,8 @@
 
 
 static size_t needed_size= 20480;
 
 
-static const time_t startup_interval= 60*5;     ///< in seconds (5 minutes)
 
-static const time_t first_interval= 60*60*24;   ///< in seconds (one day)
 
+static const time_t startup_interval= 60*1;     ///< in seconds (5 minutes)
 
+static const time_t first_interval= 60*1;   ///< in seconds (one day)
 
 static const time_t interval= 60*60*24*7;       ///< in seconds (one week)

For me it's reproducible on 10.1.7 as well; if it didn't crash for mchandel, I guess it's either due to pure luck, or because something changed in the config file along with the upgrade – the feedback plugin was enabled, or sql_mode was modified.

Stack trace from 5.5 commit 7e4da9b370d032db9015adb47ad2ff585aeaea5d (with dynamic plugin)

 
#3  <signal handler called>
 
#4  0x000000000069fc35 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 5.5-bug/sql/sql_lex.h:1051
 
#5  0x00000000008010f3 in Item_field::fix_fields (this=0x7f0613c714a8, thd=0x7f0613c6a060, reference=0x7f0613c717b8) at 5.5-bug/sql/item.cc:5344
 
#6  0x000000000083ff70 in Item_func::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_func.cc:204
 
#7  0x00000000008245aa in Item_func_like::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_cmpfunc.cc:4909
 
#8  0x0000000000822c2b in Item_cond::fix_fields (this=0x7f0613c713b8, thd=0x7f0613c6a060, ref=0x7f0615ffe788) at 5.5-bug/sql/item_cmpfunc.cc:4365
 
#9  0x00007f063c9f17a6 in feedback::make_cond (thd=0x7f0613c6a060, tables=0x7f0615ffe880, filter=0x7f063cbfe420) at 5.5-bug/plugin/feedback/feedback.cc:123
 
#10 0x00007f063c9f1818 in feedback::fill_feedback (thd=0x7f0613c6a060, tables=0x7f0615ffe880, unused=0x0) at 5.5-bug/plugin/feedback/feedback.cc:207
 
#11 0x00007f063c9f4897 in feedback::send_report (when=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:215
 
#12 0x00007f063c9f4cad in feedback::background_thread (arg=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:290
 
#13 0x00007f063f4beb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
 
#14 0x00007f063ddcc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): 
Connection ID (thread ID): 1
 
Status: NOT_KILLED

Stack trace from 10.1 commit a430df3aba59c57b0756c25b1586d880d19286df (with static plugin)

 
#2  0x00007f38705ec3ba in handle_fatal_signal (sig=11) at 10.1-bug/sql/signal_handler.cc:273
 
#3  <signal handler called>
 
#4  0x00007f38704302b3 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 10.1-bug/sql/sql_lex.h:1067
 
#5  0x00007f387060cb6e in Item_field::fix_fields (this=0x7f3840119478, thd=0x7f38400fa070, reference=0x7f3840119730) at 10.1-bug/sql/item.cc:5134
 
#6  0x00007f387065035f in Item_func::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_func.cc:208
 
#7  0x00007f387062fc97 in Item_func_like::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_cmpfunc.cc:4926
 
#8  0x00007f387062e1e1 in Item_cond::fix_fields (this=0x7f3840119398, thd=0x7f38400fa070, ref=0x7f3840ffe6f8) at 10.1-bug/sql/item_cmpfunc.cc:4347
 
#9  0x00007f3870bfdf82 in feedback::make_cond (thd=0x7f38400fa070, tables=0x7f3840ffe860, filter=0x7f387155dd80) at 10.1-bug/plugin/feedback/feedback.cc:126
 
#10 0x00007f3870bfdff4 in feedback::fill_feedback (thd=0x7f38400fa070, tables=0x7f3840ffe860, unused=0x0) at 10.1-bug/plugin/feedback/feedback.cc:210
 
#11 0x00007f3870bff2c1 in feedback::send_report (when=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:216
 
#12 0x00007f3870bff724 in feedback::background_thread (arg=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:291
 
#13 0x00007f386fa70b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
 
#14 0x00007f386d8fc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x0): 
Connection ID (thread ID): 1
 
Status: NOT_KILLED

Comment by Mohit Chandel [ 2015-11-12 ]

You are spot on elenst feedback plugin was enabled during upgrade.

Generated at Thu Feb 08 07:31:46 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.