After upgrading Mariadb 10.1.7 to 10.1.8 MariaDB is crashing with Signal 11 after about 24 hours of upgrade. I am using MariaDB Repo.
I am having two Servers running different applications and are not related to each other, both running MariaDB 10.1.7 and were upgraded to 10.1.8 at approx same time. After approx. 24 hours both Crashed with Signal 11 with a stack trace in the error log.
Now I have restarted both the Servers and everything seems fine. There are no signs of data corruption. Stack trace for both the Serves is nearly identical.
Before upgrade to 10.1.8, both the Servers were running perfectly fine for months together. I am not using replication on any of the servers.
Show Engines on both the Servers:
CSV
SEQUENCE
MRG_MyISAM
MyISAM
CONNECT
InnoDB
MEMORY
Aria
PERFORMANCE_SCHEMA
Attached my.cnf for both servers and binary logs of Server 1.
mchandel, thanks for the data.
Please try to disable the FEEDBACK plugin for now, and let me know whether it helps – I expect it should.
Elena Stepanova
added a comment - mchandel , thanks for the data.
Please try to disable the FEEDBACK plugin for now, and let me know whether it helps – I expect it should.
Reproducible with static and dynamic FEEDBACK plugin, with sql_mode=ONLY_FULL_GROUP_BY, on 5.5, 10.0, 10.1.
The crash happens on the 2nd sending of the report, which normally happens in ~24h from the server startup. To reproduce it in reasonable time, I modified sender_thread.cc
-static const time_t startup_interval= 60*5; ///< in seconds (5 minutes)
-static const time_t first_interval= 60*60*24; ///< in seconds (one day)
+static const time_t startup_interval= 60*1; ///< in seconds (5 minutes)
+static const time_t first_interval= 60*1; ///< in seconds (one day)
static const time_t interval= 60*60*24*7; ///< in seconds (one week)
For me it's reproducible on 10.1.7 as well; if it didn't crash for mchandel, I guess it's either due to pure luck, or because something changed in the config file along with the upgrade – the feedback plugin was enabled, or sql_mode was modified.
Stack trace from 5.5 commit 7e4da9b370d032db9015adb47ad2ff585aeaea5d (with dynamic plugin)
#3 <signal handler called>
#4 0x000000000069fc35 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 5.5-bug/sql/sql_lex.h:1051
#5 0x00000000008010f3 in Item_field::fix_fields (this=0x7f0613c714a8, thd=0x7f0613c6a060, reference=0x7f0613c717b8) at 5.5-bug/sql/item.cc:5344
#6 0x000000000083ff70 in Item_func::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_func.cc:204
#7 0x00000000008245aa in Item_func_like::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_cmpfunc.cc:4909
#8 0x0000000000822c2b in Item_cond::fix_fields (this=0x7f0613c713b8, thd=0x7f0613c6a060, ref=0x7f0615ffe788) at 5.5-bug/sql/item_cmpfunc.cc:4365
#9 0x00007f063c9f17a6 in feedback::make_cond (thd=0x7f0613c6a060, tables=0x7f0615ffe880, filter=0x7f063cbfe420) at 5.5-bug/plugin/feedback/feedback.cc:123
#10 0x00007f063c9f1818 in feedback::fill_feedback (thd=0x7f0613c6a060, tables=0x7f0615ffe880, unused=0x0) at 5.5-bug/plugin/feedback/feedback.cc:207
#11 0x00007f063c9f4897 in feedback::send_report (when=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:215
#12 0x00007f063c9f4cad in feedback::background_thread (arg=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:290
#13 0x00007f063f4beb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#14 0x00007f063ddcc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 1
Status: NOT_KILLED
Stack trace from 10.1 commit a430df3aba59c57b0756c25b1586d880d19286df (with static plugin)
#2 0x00007f38705ec3ba in handle_fatal_signal (sig=11) at 10.1-bug/sql/signal_handler.cc:273
#3 <signal handler called>
#4 0x00007f38704302b3 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 10.1-bug/sql/sql_lex.h:1067
#5 0x00007f387060cb6e in Item_field::fix_fields (this=0x7f3840119478, thd=0x7f38400fa070, reference=0x7f3840119730) at 10.1-bug/sql/item.cc:5134
#6 0x00007f387065035f in Item_func::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_func.cc:208
#7 0x00007f387062fc97 in Item_func_like::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_cmpfunc.cc:4926
#8 0x00007f387062e1e1 in Item_cond::fix_fields (this=0x7f3840119398, thd=0x7f38400fa070, ref=0x7f3840ffe6f8) at 10.1-bug/sql/item_cmpfunc.cc:4347
#9 0x00007f3870bfdf82 in feedback::make_cond (thd=0x7f38400fa070, tables=0x7f3840ffe860, filter=0x7f387155dd80) at 10.1-bug/plugin/feedback/feedback.cc:126
#10 0x00007f3870bfdff4 in feedback::fill_feedback (thd=0x7f38400fa070, tables=0x7f3840ffe860, unused=0x0) at 10.1-bug/plugin/feedback/feedback.cc:210
#11 0x00007f3870bff2c1 in feedback::send_report (when=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:216
#12 0x00007f3870bff724 in feedback::background_thread (arg=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:291
#13 0x00007f386fa70b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#14 0x00007f386d8fc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 1
Status: NOT_KILLED
Elena Stepanova
added a comment - - edited Reproducible with static and dynamic FEEDBACK plugin, with sql_mode=ONLY_FULL_GROUP_BY , on 5.5, 10.0, 10.1.
The crash happens on the 2nd sending of the report, which normally happens in ~24h from the server startup. To reproduce it in reasonable time, I modified sender_thread.cc
diff --git a/plugin/feedback/sender_thread.cc b/plugin/feedback/sender_thread.cc
index 732118f..c3a46db 100644
--- a/plugin/feedback/sender_thread.cc
+++ b/plugin/feedback/sender_thread.cc
@@ -26,8 +26,8 @@
static size_t needed_size= 20480;
-static const time_t startup_interval= 60*5; ///< in seconds (5 minutes)
-static const time_t first_interval= 60*60*24; ///< in seconds (one day)
+static const time_t startup_interval= 60*1; ///< in seconds (5 minutes)
+static const time_t first_interval= 60*1; ///< in seconds (one day)
static const time_t interval= 60*60*24*7; ///< in seconds (one week)
For me it's reproducible on 10.1.7 as well; if it didn't crash for mchandel , I guess it's either due to pure luck, or because something changed in the config file along with the upgrade – the feedback plugin was enabled, or sql_mode was modified.
Stack trace from 5.5 commit 7e4da9b370d032db9015adb47ad2ff585aeaea5d (with dynamic plugin)
#3 <signal handler called>
#4 0x000000000069fc35 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 5.5-bug/sql/sql_lex.h:1051
#5 0x00000000008010f3 in Item_field::fix_fields (this=0x7f0613c714a8, thd=0x7f0613c6a060, reference=0x7f0613c717b8) at 5.5-bug/sql/item.cc:5344
#6 0x000000000083ff70 in Item_func::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_func.cc:204
#7 0x00000000008245aa in Item_func_like::fix_fields (this=0x7f0613c71720, thd=0x7f0613c6a060, ref=0x7f0613c718d0) at 5.5-bug/sql/item_cmpfunc.cc:4909
#8 0x0000000000822c2b in Item_cond::fix_fields (this=0x7f0613c713b8, thd=0x7f0613c6a060, ref=0x7f0615ffe788) at 5.5-bug/sql/item_cmpfunc.cc:4365
#9 0x00007f063c9f17a6 in feedback::make_cond (thd=0x7f0613c6a060, tables=0x7f0615ffe880, filter=0x7f063cbfe420) at 5.5-bug/plugin/feedback/feedback.cc:123
#10 0x00007f063c9f1818 in feedback::fill_feedback (thd=0x7f0613c6a060, tables=0x7f0615ffe880, unused=0x0) at 5.5-bug/plugin/feedback/feedback.cc:207
#11 0x00007f063c9f4897 in feedback::send_report (when=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:215
#12 0x00007f063c9f4cad in feedback::background_thread (arg=0x0) at 5.5-bug/plugin/feedback/sender_thread.cc:290
#13 0x00007f063f4beb50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#14 0x00007f063ddcc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 1
Status: NOT_KILLED
Stack trace from 10.1 commit a430df3aba59c57b0756c25b1586d880d19286df (with static plugin)
#2 0x00007f38705ec3ba in handle_fatal_signal (sig=11) at 10.1-bug/sql/signal_handler.cc:273
#3 <signal handler called>
#4 0x00007f38704302b3 in st_select_lex::set_non_agg_field_used (this=0x0, val=true) at 10.1-bug/sql/sql_lex.h:1067
#5 0x00007f387060cb6e in Item_field::fix_fields (this=0x7f3840119478, thd=0x7f38400fa070, reference=0x7f3840119730) at 10.1-bug/sql/item.cc:5134
#6 0x00007f387065035f in Item_func::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_func.cc:208
#7 0x00007f387062fc97 in Item_func_like::fix_fields (this=0x7f38401196a8, thd=0x7f38400fa070, ref=0x7f38401197f8) at 10.1-bug/sql/item_cmpfunc.cc:4926
#8 0x00007f387062e1e1 in Item_cond::fix_fields (this=0x7f3840119398, thd=0x7f38400fa070, ref=0x7f3840ffe6f8) at 10.1-bug/sql/item_cmpfunc.cc:4347
#9 0x00007f3870bfdf82 in feedback::make_cond (thd=0x7f38400fa070, tables=0x7f3840ffe860, filter=0x7f387155dd80) at 10.1-bug/plugin/feedback/feedback.cc:126
#10 0x00007f3870bfdff4 in feedback::fill_feedback (thd=0x7f38400fa070, tables=0x7f3840ffe860, unused=0x0) at 10.1-bug/plugin/feedback/feedback.cc:210
#11 0x00007f3870bff2c1 in feedback::send_report (when=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:216
#12 0x00007f3870bff724 in feedback::background_thread (arg=0x0) at 10.1-bug/plugin/feedback/sender_thread.cc:291
#13 0x00007f386fa70b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#14 0x00007f386d8fc95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0):
Connection ID (thread ID): 1
Status: NOT_KILLED
elenst
Output of the the SHOW CREATE TABLE and SHOW INDEX IN . Can't upload the Data as it contains private information.
SHOW CREATE TABLE caf_data
`permanentAddress1` tinytext,
`actAgent` tinytext,
SHOW INDEX IN caf_data
Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment
caf_data 0 PRIMARY 1 recordNo A 1542095 \N \N BTREE
caf_data 1 idx-msisdn 1 msisdn A 1531983 \N \N BTREE
caf_data 1 idx-customerName 1 customerName A 318134 \N \N BTREE
caf_data 1 idx-actDate 1 actDate A 4209 \N \N YES BTREE
caf_data 1 idx-printCenter 1 printCenter A 7 \N \N YES BTREE
SHOW CREATE TABLE caf_scan
CREATE TABLE `caf_scan` (
`recordNo` int(10) unsigned NOT NULL,
`cafType` char(3) DEFAULT NULL,
`ssaCode` char(3) NOT NULL,
`cscCode` char(3) NOT NULL,
`lotNo` int(5) unsigned zerofill DEFAULT NULL,
`fullLotNo` char(9) AS (IF(ISNULL(lotNo), NULL, CONCAT_WS('-',`ssaCode`,`lotNo`))) PERSISTENT,
`serialNo` smallint(3) unsigned zerofill DEFAULT NULL,
`scanStatus` tinyint(3) unsigned NOT NULL DEFAULT '0',
`scanDoneBy` varchar(6) DEFAULT NULL,
`auditCaf` char(3) NOT NULL DEFAULT 'NO',
`auditDoneBy` varchar(6) DEFAULT NULL,
`qcRejectReason` enum('INDEX_DATA','SCAN_QUALITY') DEFAULT NULL,
`whLotCreateUser` char(9) DEFAULT NULL,
`whLotCreateTs` timestamp NULL DEFAULT NULL,
`whLocUpdateUser` varchar(9) DEFAULT NULL,
`whLocUpdateTs` timestamp NULL DEFAULT NULL,
`whAddress` tinytext,
`whRack` tinyint(3) unsigned DEFAULT NULL,
`whShelf` tinyint(3) unsigned DEFAULT NULL,
`whPosition` tinyint(3) unsigned DEFAULT NULL,
PRIMARY KEY (`recordNo`),
KEY `idx-scanStatus` (`scanStatus`),
KEY `idx-qcRejectReason` (`qcRejectReason`),
KEY `idx-ssa-csc-scanStatus` (`ssaCode`,`cscCode`,`scanStatus`),
KEY `idx-whLotCreateUser-user_data-loginId` (`whLotCreateUser`),
KEY `idx-whLocUpdateUser-user_data-loginId` (`whLocUpdateUser`),
CONSTRAINT `fk-recordNo-caf_data-recordNo_0` FOREIGN KEY (`recordNo`) REFERENCES `caf_data` (`recordNo`),
CONSTRAINT `fk-whLocUpdateUser-user_data-loginId` FOREIGN KEY (`whLocUpdateUser`) REFERENCES `user_data` (`loginId`),
CONSTRAINT `fk-whLotCreateUser-user_data-loginId` FOREIGN KEY (`whLotCreateUser`) REFERENCES `user_data` (`loginId`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;
SHOW INDEX IN caf_scan
Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment
caf_scan 0 PRIMARY 1 recordNo A 561568 \N \N BTREE
caf_scan 1 idx-scanStatus 1 scanStatus A 12 \N \N BTREE
caf_scan 1 idx-qcRejectReason 1 qcRejectReason A 439 \N \N YES BTREE
caf_scan 1 idx-ssa-csc-scanStatus 1 ssaCode A 7 \N \N BTREE
caf_scan 1 idx-ssa-csc-scanStatus 2 cscCode A 80 \N \N BTREE
caf_scan 1 idx-ssa-csc-scanStatus 3 scanStatus A 260 \N \N BTREE
caf_scan 1 idx-whLotCreateUser-user_data-loginId 1 whLotCreateUser A 239 \N \N YES BTREE
caf_scan 1 idx-whLocUpdateUser-user_data-loginId 1 whLocUpdateUser A 1871 \N \N YES BTREE
SHOW CREATE TABLE caf_scan_log
CREATE TABLE `caf_scan_log` (
`rowId` int(10) unsigned NOT NULL AUTO_INCREMENT,
`recordNo` int(10) unsigned NOT NULL,
`scanStatusUpdateTs` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`scanStatusUpdateUser` char(9) NOT NULL,
`scanStatus` tinyint(3) unsigned NOT NULL,
PRIMARY KEY (`rowId`),
UNIQUE KEY `idx-recordNo-scanStatus` (`recordNo`,`scanStatus`),
KEY `idx-scanStatusUpdateUser-user_data-loginId` (`scanStatusUpdateUser`),
CONSTRAINT `fk-recordNo-caf_data-recordNo_1` FOREIGN KEY (`recordNo`) REFERENCES `caf_data` (`recordNo`),
CONSTRAINT `fk-scanStatusUpdateUser-user-loginId` FOREIGN KEY (`scanStatusUpdateUser`) REFERENCES `user_data` (`loginId`)
) ENGINE=InnoDB AUTO_INCREMENT=5600727 DEFAULT CHARSET=utf8 ROW_FORMAT=DYNAMIC;
SHOW INDEX IN caf_scan_log
Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment
caf_scan_log 0 PRIMARY 1 rowId A 1662091 \N \N BTREE
caf_scan_log 0 idx-recordNo-scanStatus 1 recordNo A 561574 \N \N BTREE
caf_scan_log 0 idx-recordNo-scanStatus 2 scanStatus A 1662091 \N \N BTREE
caf_scan_log 1 idx-scanStatusUpdateUser-user_data-loginId 1 scanStatusUpdateUser A 124 \N \N BTREE