[MDEV-21315] Server 10.4 sporadically crashes when issuing simple INSERT commands Created: 2019-12-13  Updated: 2019-12-19  Resolved: 2019-12-16

Status: Closed
Project: MariaDB Server
Component/s: Server
Affects Version/s: 10.4.10
Fix Version/s: N/A

Type: Bug Priority: Critical
Reporter: Christophe Migliorini Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Environment:

Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux

AMD Epyc, 256Go RAM, 3xNVMe disks, software RAID
Linux


Attachments: HTML File 191211-16-11-02     HTML File 191212-13-31-05     HTML File 191213-11-49-20     HTML File 191213-12-14-01     HTML File 191213-12-54-02     HTML File 191213-14-11-47     HTML File 191215 12-37-28    
Issue Links:
Duplicate
duplicates MDEV-20290 Server crash in st_select_lex::cleanu... Closed
Relates
relates to MDEV-16529 [Draft] Server crashes in cleanup_emp... Closed
relates to MDEV-20803 IF NOT EXISTS AND NOT EXISTS crash DB... Closed

 Description   

Since we upgraded to 10.4.10 less than a week ago, we encountered 6 crashes, all occurred during a simple INSERT statement, all seemingly crashing in _ZN4JOIN7destroyEv (we don't have debug symbols so this is all we have).

The problem NEVER occurred on our preproduction server, which has Intel Xeon processor and SSD SATA hard drives, but the load on that one is several orders of magnitude smaller.

Stack traces attached. 5 crashed during an INSERT, 1 during an INSERT IGNORE

Also we had a dozen records inserted in the course of last week, and we were able to execute some more without crashing the server, and as we got only 6 crashes we are rather confident that not all queries crash.

On the other hand that's about 50% of them !

 Comments   
Comment by Christophe Migliorini [ 2019-12-14 ]

this is a production server, hence our evaluation that this is "critical". It takes the application down for 5 minutes until Binlog is replayed.

Comment by Christophe Migliorini [ 2019-12-16 ]

After the first crashes, we found out that libc/libpthread had been upgraded as well as the kernel but the new kernel was not installed (no reboot since upgrade), so we restarted the server.
We had another crash, which trace is attached here 191215 12-37-28

Also perhaps worth noting: after 6th crash, we changed SET GLOBAL thread_pool_max_threads=16384, only live, not in the my.cnf file. Original value was 65536, whereas crash dump says max_threads=65340.

Then, after 2019-12-15 crash, we upgraded to 10.4.11 hoping for the best, and this time set thread_pool_max_threads=16384 both dynamically and in my.cnf. Here we stand for now...

Comment by Alice Sherepa [ 2019-12-16 ]

It seems to be the same problem as MDEV-20290, which is not fixed yet.
I will close the report as a duplicate, but please reopen if it turns out to still happen after MDEV-20290 fix

Comment by Christophe Migliorini [ 2019-12-16 ]

OK, thank you. Let's hope you're right!

Comment by Christophe Migliorini [ 2019-12-18 ]

@alice, I appreciate that my knowledge of MariaDB's internals is nil, still I'm wondering, looking at issues that were marked as related to this one: others fail on rather complex queries and our case only fails on a simple INSERT, without any JOIN or anything.

Comment by Alice Sherepa [ 2019-12-18 ]

c_migliorini, I compared stacktraces, e.g. from 191212-13-31-05 :

/usr/sbin/mysqld(_ZN4JOIN7destroyEv+0x20)[0x558b3a5b52f0]
 
/usr/sbin/mysqld(_ZN13st_select_lex7cleanupEv+0x6d)[0x558b3a61530d]
 
/usr/sbin/mysqld(_ZN30subselect_single_select_engine7prepareEP3THD+0x3c)[0x558b3a838bdc]
 
/usr/sbin/mysqld(_ZN14Item_subselect10fix_fieldsEP3THDPP4Item+0x143)[0x558b3a8385b3]
 
/usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x153)[0x558b3a7ed103]
 
/usr/sbin/mysqld(_ZN13Item_func_not10fix_fieldsEP3THDPP4Item+0x1c6)[0x558b3a7c1076]
 
/usr/sbin/mysqld(_ZN9Item_cond10fix_fieldsEP3THDPP4Item+0x11e)[0x558b3a7c187e]
 
/usr/sbin/mysqld(_ZN3THD16sp_fix_func_itemEPP4Item+0x90)[0x558b3a4d9b90]
 
/usr/sbin/mysqld(_ZN3THD20sp_prepare_func_itemEPP4Itemj+0xf)[0x558b3a4d9baf]
 
/usr/sbin/mysqld(_ZN20sp_instr_jump_if_not9exec_coreEP3THDPj+0x1e)[0x558b3a4d9bfe]
 
/usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x152)[0x558b3a4dfc32]
 
/usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0xa2c)[0x558b3a4db77c]
 
/usr/sbin/mysqld(_ZN7sp_head15execute_triggerEP3THDPK25st_mysql_const_lex_stringS4_P13st_grant_info+0x23c)[0x558b3a4dc45c]
 
/usr/sbin/mysqld(_ZN19Table_triggers_list16process_triggersEP3THD14trg_event_type20trg_action_time_typeb+0x105)[0x558b3a60df35]
 
/usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0xcf)[0x558b3a53e47f]
 
/usr/sbin/mysqld(_ZN13select_insert9send_dataER4ListI4ItemE+0xf9)[0x558b3a53f129]
 
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa56)[0x558b3a5cf056]
 
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x558b3a5cf433]
 
/usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x558b3a5cd806]
 
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x558b3a5ce156]
 
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7c5f)[0x558b3a57874f]
 
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x558b3a57985a]
 
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1349)[0x558b3a57bdd9]
 
/usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558b3a57d66c]
 
/usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0xe8)[0x558b3a75afd8]
 
/usr/sbin/mysqld(+0x9f1ad0)[0x558b3a910ad0]
 
/usr/sbin/mysqld(+0xd86551)[0x558b3aca5551]
 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f98014c24a4]
 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f97ffc20d0f]
 
Trying to get some variables.
 
Some pointers may be invalid and cause the dump to abort.
 
Query (0x7f637c15a650): INSERT IGNORE INTO user_lnk_contact (user_id, client_id, contact_id, is_created)   SELECT 132561    , (SELECT client_id FROM sf_guard_user AS u WHERE u.id=132561)   , 44028334   , 1

and the one from the first comment in MDEV-20290. It seems like the server crashes after executing trigger before/after insert.

Comment by Christophe Migliorini [ 2019-12-18 ]

You're right for this one, although this is the only INSERT IGNORE of the set. Others are simple INSERT and there is no trigger on this table, only foreign keys.
Here is the DDL, maybe there's something I'm not seeing?

CREATE TABLE `user_lnk_contact` (
 
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
 
  `client_id` bigint(20) DEFAULT NULL,
 
  `user_id` bigint(20) DEFAULT NULL,
 
  `contact_id` bigint(20) DEFAULT NULL,
 
  `is_created` tinyint(1) NOT NULL DEFAULT 0,
 
  `created_at` timestamp NULL DEFAULT current_timestamp(),
 
  `updated_at` timestamp NULL DEFAULT current_timestamp(),
 
  PRIMARY KEY (`id`),
 
  UNIQUE KEY `ixClientIdUserIdContactId` (`client_id`,`user_id`,`contact_id`),
 
  UNIQUE KEY `ixUserIdContactId` (`user_id`,`contact_id`),
 
  KEY `ixUserId` (`user_id`),
 
  KEY `ixContactId` (`contact_id`),
 
  KEY `xUserIdIsCreated` (`user_id`,`is_created`),
 
  KEY `ixContactIsCreatedCreatedAt` (`contact_id`,`is_created`,`created_at`),
 
  KEY `ixClientContact` (`client_id`,`contact_id`),
 
  KEY `ixClient` (`client_id`),
 
  CONSTRAINT `user_lnk_contact_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `sf_guard_user` (`id`) ON DELETE CASCADE,
 
  CONSTRAINT `user_lnk_contact_ibfk_2` FOREIGN KEY (`contact_id`) REFERENCES `contact` (`id`) ON DELETE CASCADE,
 
  CONSTRAINT `user_lnk_contact_ibfk_3` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON DELETE CASCADE
 
) ENGINE=InnoDB DEFAULT CHARSET=utf8

Comment by Alice Sherepa [ 2019-12-18 ]

could you please add the output of "show triggers like 'user_lnk_contact';"

Comment by Christophe Migliorini [ 2019-12-18 ]

sure

MariaDB []> show triggers like 'user_lnk_contact';
 
--------------
 
show triggers like 'user_lnk_contact'
 
--------------
 
 
 
Empty set (0.003 sec)

Comment by Christophe Migliorini [ 2019-12-19 ]

@Alice Sherepa, I'm sorry there is a trigger AFTER INSERT, I was not connected as root so I could not see it – n00b ...

Is there a non-public server I can upload it to? I would rather not publish this piece of code.

Generated at Thu Feb 08 09:06:12 UTC 2024 using Jira 8.20.16#820016-sha1:9d11dbea5f4be3d4cc21f03a88dd11d8c8687422.