[MDEV-29516] MariaDb crash with signal 11 Created: 2022-09-12  Updated: 2022-09-15

Status: Open
Project: MariaDB Server
Component/s: None
Affects Version/s: 10.7.5
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Juan Ferrer Toribio Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: crash, segfault


 Description   

MariaDB crashed with signal 11, we are not able to locate the original cause since it has occurred in a production server, seems a bug since it has been due to segmentation violation.

The internal call stack is included in the attached log, hope it helps to locate the source.

2022-09-12 14:39:59 63034 [ERROR] Failed to write to mysql.slow_log: 
220912 14:40:20 [ERROR] mysqld got signal 11 ;
 
This could be because you hit a bug. It is also possible that this binary
 
or one of the libraries it was linked against is corrupt, improperly built,
 
or misconfigured. This error can also be caused by malfunctioning hardware.
 
 
 
To report this bug, see https://mariadb.com/kb/en/reporting-bugs
 
 
 
We will try our best to scrape up some info that will hopefully help
 
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.
 
 
 
Server version: 10.7.5-MariaDB-1:10.7.5+maria~deb11-log
 
key_buffer_size=262144
 
read_buffer_size=1048576
 
max_used_connections=858
 
max_threads=1002
 
thread_count=862
 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5156811 K  bytes of memory
 
Hope that's ok; if not, decrease some variables in the equation.
 
 
 
Thread pointer: 0x7f3981df0418
 
Attempting backtrace. You can use the following information to find out
 
where mysqld died. If you see no messages after this, something went
 
terribly wrong...
 
stack_bottom = 0x7f3a6663bd58 thread_stack 0x80000
 
/usr/sbin/mariadbd(my_print_stacktrace+0x2e)[0x557e420b982e]
 
/usr/sbin/mariadbd(handle_fatal_signal+0x485)[0x557e41ba48b5]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140)[0x7f430288b140]
 
2022-09-12 14:40:27 62310 [ERROR] Failed to write to mysql.slow_log: 
2022-09-12 14:40:29 63599 [ERROR] Event Scheduler: [root@localhost][vn.ticket_doRecalc] Encontrado estancamiento (deadlock) al intentar obtener el bloqueo; intente volver a comenzar la transacción
 
2022-09-12 14:40:29 63599 [Note] Event Scheduler: [root@localhost][vn.ticket_doRecalc] En la línea 31 en vn.ticket_doRecalc
 
2022-09-12 14:40:29 63599 [Note] Event Scheduler: [root@localhost][vn.ticket_doRecalc] En la línea 1 en vn.ticket_doRecalc
 
2022-09-12 14:40:29 63599 [Note] Event Scheduler: [root@localhost].[vn.ticket_doRecalc] event execution failed.
 
/usr/sbin/mariadbd(_ZN31Multiupdate_prelocking_strategy10handle_endEP3THD+0x2d1)[0x557e41a3f5c1]
 
/usr/sbin/mariadbd(_Z11open_tablesP3THDRK14DDL_options_stPP10TABLE_LISTPjjP19Prelocking_strategy+0xce3)[0x557e418f7e83]
 
/usr/sbin/mariadbd(_Z26mysql_multi_update_prepareP3THD+0x7b)[0x557e41a4370b]
 
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x201f)[0x557e41965aef]
 
/usr/sbin/mariadbd(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x17)[0x557e418b2d37]
 
/usr/sbin/mariadbd(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x176)[0x557e418bbde6]
 
/usr/sbin/mariadbd(_ZN13sp_instr_stmt7executeEP3THDPj+0x5bc)[0x557e418bc7ac]
 
/usr/sbin/mariadbd(_ZN7sp_head7executeEP3THDb+0xa09)[0x557e418b6519]
 
/usr/sbin/mariadbd(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x45f)[0x557e418b7c6f]
 
/usr/sbin/mariadbd(+0x7d7dc7)[0x557e41958dc7]
 
/usr/sbin/mariadbd(+0x7dba88)[0x557e4195ca88]
 
/usr/sbin/mariadbd(_Z21mysql_execute_commandP3THDb+0x118a)[0x557e41964c5a]
 
/usr/sbin/mariadbd(_ZN18Prepared_statement7executeEP6Stringb+0x4aa)[0x557e4198b94a]
 
/usr/sbin/mariadbd(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x9d)[0x557e4198bb6d]
 
/usr/sbin/mariadbd(+0x80ba25)[0x557e4198ca25]
 
/usr/sbin/mariadbd(_Z19mysqld_stmt_executeP3THDPcj+0x2c)[0x557e4198cc0c]
 
/usr/sbin/mariadbd(_Z16dispatch_command19enum_server_commandP3THDPcjb+0xe89)[0x557e4196b839]
 
/usr/sbin/mariadbd(_Z10do_commandP3THDb+0x132)[0x557e4196dc82]
 
/usr/sbin/mariadbd(_Z24do_handle_one_connectionP7CONNECTb+0x3af)[0x557e41a8790f]
 
/usr/sbin/mariadbd(handle_one_connection+0x5d)[0x557e41a87c5d]
 
/usr/sbin/mariadbd(+0xc43652)[0x557e41dc4652]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8ea7)[0x7f430287fea7]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f4302496def]
 
 
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f335fca4c10): UPDATE vn.collection c
 
			JOIN (SELECT count(*) saleTotalCount , 
						sum(s.isPicked != 0) salePickedCount
 
					FROM vn.ticketCollection tc
 
						JOIN vn.sale s ON s.ticketFk = tc.ticketFk
 
						WHERE tc.collectionFk =  NAME_CONST('vCollectionFk',650866)
 
							AND s.quantity > 0
 
				) sub 
			SET 	c.saleTotalCount = sub.saleTotalCount, 
				c.salePickedCount = sub.salePickedCount
 
			WHERE c.id =  NAME_CONST('vCollectionFk',650866)
 
 
 
Connection ID (thread ID): 62651
 
Status: NOT_KILLED
 
 
 
Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on,not_null_range_scan=off
 
 
 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
 
information that should help you find out what is causing the crash.
 
Writing a core file...
 
Working directory at /mnt/mysqldata/mysql
 
Resource Limits:
 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             1030071              1030071              processes 
Max open files            524288               524288               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       1030071              1030071              signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
Core pattern: core
 
 
 
Kernel version: Linux version 5.15.35-3-pve (build@proxmox) (gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP PVE 5.15.35-6 (Fri, 17 Jun 2022 13:42:35 +0200)



 Comments   
Comment by Alice Sherepa [ 2022-09-12 ]

Could you please add the output of "SHOW CREATE TABLE" for tables `collection`,` ticketCollection`, `sale` ?

Comment by Juan Ferrer Toribio [ 2022-09-13 ]

Hi alice, here you have the tables structure.

SHOW CREATE TABLE vn.collection;

collection | CREATE TABLE `collection` (
 
  `id` int(11) NOT NULL AUTO_INCREMENT,
 
  `created` timestamp NOT NULL DEFAULT current_timestamp(),
 
  `workerFk` int(10) unsigned DEFAULT NULL,
 
  `stateFk` tinyint(3) unsigned DEFAULT NULL,
 
  `itemPackingTypeFk` varchar(1) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
 
  `saleTotalCount` int(11) NOT NULL DEFAULT 0,
 
  `salePickedCount` int(11) NOT NULL DEFAULT 0,
 
  `trainFk` int(11) NOT NULL DEFAULT 1,
 
  `sectorFk` int(11) DEFAULT NULL,
 
  `wagons` int(11) DEFAULT NULL,
 
  PRIMARY KEY (`id`),
 
  KEY `ticketCollection_idx` (`workerFk`),
 
  KEY `collection_id2_idx` (`stateFk`),
 
  KEY `collection_FK` (`itemPackingTypeFk`),
 
  KEY `collectionTrain_Fk` (`trainFk`),
 
  KEY `collectionSector_FK` (`sectorFk`),
 
  CONSTRAINT `collectionSector_FK` FOREIGN KEY (`sectorFk`) REFERENCES `sector` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
 
  CONSTRAINT `collectionTrain_Fk` FOREIGN KEY (`trainFk`) REFERENCES `train` (`id`) ON UPDATE CASCADE,
 
  CONSTRAINT `collection_id2` FOREIGN KEY (`stateFk`) REFERENCES `state` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
 
  CONSTRAINT `ticketCollection` FOREIGN KEY (`workerFk`) REFERENCES `worker` (`id`) ON DELETE SET NULL ON UPDATE CASCADE
 
) ENGINE=InnoDB AUTO_INCREMENT=651496 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci

SHOW CREATE TABLE vn.ticketCollection;

ticketCollection | CREATE TABLE `ticketCollection` (
 
  `id` int(11) NOT NULL AUTO_INCREMENT,
 
  `ticketFk` int(11) NOT NULL,
 
  `collectionFk` int(11) NOT NULL,
 
  `created` timestamp NOT NULL DEFAULT current_timestamp(),
 
  `level` int(11) DEFAULT NULL,
 
  `wagon` int(11) NOT NULL DEFAULT 0,
 
  `smartTagFk` varchar(12) CHARACTER SET utf8mb3 DEFAULT NULL,
 
  `usedShelves` int(11) DEFAULT NULL COMMENT 'número de bandejas que ocupa un ticket en el carro',
 
  `itemCount` int(11) DEFAULT NULL COMMENT 'número de productos distintos en el pedido',
 
  `liters` int(11) DEFAULT NULL COMMENT 'volumen del pedido en litros',
 
  PRIMARY KEY (`id`),
 
  UNIQUE KEY `ticketCollection_UN` (`ticketFk`,`collectionFk`),
 
  KEY `ticketCollection_fk1_idx` (`collectionFk`),
 
  KEY `ticketCollection_fk2_idx` (`ticketFk`),
 
  KEY `ticketCollection_smartTagFk_IDX` (`smartTagFk`) USING BTREE,
 
  KEY `ticketCollection_created_IDX` (`created`) USING BTREE,
 
  CONSTRAINT `ticketCollection_FK` FOREIGN KEY (`smartTagFk`) REFERENCES `smartTag` (`code`) ON UPDATE CASCADE,
 
  CONSTRAINT `ticketCollection_fk1` FOREIGN KEY (`collectionFk`) REFERENCES `collection` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
 
  CONSTRAINT `ticketCollection_fk2` FOREIGN KEY (`ticketFk`) REFERENCES `ticket` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
 
) ENGINE=InnoDB AUTO_INCREMENT=4893088 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci

SHOW CREATE TABLE vn.sale;

sale  | CREATE TABLE `sale` (
 
  `id` int(11) NOT NULL AUTO_INCREMENT,
 
  `itemFk` int(11) NOT NULL,
 
  `ticketFk` int(11) NOT NULL,
 
  `concept` varchar(50) COLLATE utf8mb3_unicode_ci DEFAULT NULL,
 
  `quantity` decimal(10,2) NOT NULL DEFAULT 0.00,
 
  `originalQuantity` double(9,1) DEFAULT NULL,
 
  `price` decimal(10,2) DEFAULT 0.00,
 
  `discount` tinyint(2) unsigned NOT NULL DEFAULT 0,
 
  `priceFixed` decimal(10,2) NOT NULL DEFAULT 0.00,
 
  `reserved` tinyint(1) NOT NULL DEFAULT 0,
 
  `isPicked` tinyint(1) NOT NULL DEFAULT 0,
 
  `isPriceFixed` tinyint(1) NOT NULL DEFAULT 0,
 
  `created` timestamp NOT NULL DEFAULT current_timestamp(),
 
  `isAdded` tinyint(1) NOT NULL DEFAULT 0,
 
  PRIMARY KEY (`id`),
 
  KEY `Id_Ticket` (`ticketFk`),
 
  KEY `itemFk_ticketFk` (`itemFk`,`ticketFk`),
 
  CONSTRAINT `movement_ticket_id` FOREIGN KEY (`ticketFk`) REFERENCES `ticket` (`id`) ON DELETE CASCADE ON UPDATE CASCADE,
 
  CONSTRAINT `sale_ibfk_1` FOREIGN KEY (`itemFk`) REFERENCES `item` (`id`) ON UPDATE CASCADE
 
) ENGINE=InnoDB AUTO_INCREMENT=36833410 DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_unicode_ci

Comment by Alice Sherepa [ 2022-09-14 ]

In UPDATE query there is NAME_CONST function - did the crash happened on a replica then? And if yes, then was there a crash on master also or not?

Comment by Juan Ferrer Toribio [ 2022-09-15 ]

alice, the crash happened on the master, replica was not affected. That query is inside a stored procedure and does not use the NAME_CONST() function, it seems that it is added internally by mariadb.

Generated at Thu Feb 08 10:09:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.