Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-21315

Server 10.4 sporadically crashes when issuing simple INSERT commands

Details

    • Bug
    • Status: Closed (View Workflow)
    • Critical
    • Resolution: Duplicate
    • 10.4.10
    • N/A
    • Server
    • None
    • Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux

      AMD Epyc, 256Go RAM, 3xNVMe disks, software RAID
      Linux

    Description

      Since we upgraded to 10.4.10 less than a week ago, we encountered 6 crashes, all occurred during a simple INSERT statement, all seemingly crashing in _ZN4JOIN7destroyEv (we don't have debug symbols so this is all we have).

      The problem NEVER occurred on our preproduction server, which has Intel Xeon processor and SSD SATA hard drives, but the load on that one is several orders of magnitude smaller.

      Stack traces attached. 5 crashed during an INSERT, 1 during an INSERT IGNORE

      Also we had a dozen records inserted in the course of last week, and we were able to execute some more without crashing the server, and as we got only 6 crashes we are rather confident that not all queries crash.

      On the other hand that's about 50% of them !

      Attachments

        1. 191211-16-11-02
          6 kB
        2. 191212-13-31-05
          6 kB
        3. 191213-11-49-20
          6 kB
        4. 191213-12-14-01
          6 kB
        5. 191213-12-54-02
          6 kB
        6. 191213-14-11-47
          6 kB
        7. 191215 12-37-28
          4 kB

        Issue Links

          Activity

            alice Alice Sherepa added a comment -

            c_migliorini, I compared stacktraces, e.g. from 191212-13-31-05 :

            /usr/sbin/mysqld(_ZN4JOIN7destroyEv+0x20)[0x558b3a5b52f0]
            /usr/sbin/mysqld(_ZN13st_select_lex7cleanupEv+0x6d)[0x558b3a61530d]
            /usr/sbin/mysqld(_ZN30subselect_single_select_engine7prepareEP3THD+0x3c)[0x558b3a838bdc]
            /usr/sbin/mysqld(_ZN14Item_subselect10fix_fieldsEP3THDPP4Item+0x143)[0x558b3a8385b3]
            /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x153)[0x558b3a7ed103]
            /usr/sbin/mysqld(_ZN13Item_func_not10fix_fieldsEP3THDPP4Item+0x1c6)[0x558b3a7c1076]
            /usr/sbin/mysqld(_ZN9Item_cond10fix_fieldsEP3THDPP4Item+0x11e)[0x558b3a7c187e]
            /usr/sbin/mysqld(_ZN3THD16sp_fix_func_itemEPP4Item+0x90)[0x558b3a4d9b90]
            /usr/sbin/mysqld(_ZN3THD20sp_prepare_func_itemEPP4Itemj+0xf)[0x558b3a4d9baf]
            /usr/sbin/mysqld(_ZN20sp_instr_jump_if_not9exec_coreEP3THDPj+0x1e)[0x558b3a4d9bfe]
            /usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x152)[0x558b3a4dfc32]
            /usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0xa2c)[0x558b3a4db77c]
            /usr/sbin/mysqld(_ZN7sp_head15execute_triggerEP3THDPK25st_mysql_const_lex_stringS4_P13st_grant_info+0x23c)[0x558b3a4dc45c]
            /usr/sbin/mysqld(_ZN19Table_triggers_list16process_triggersEP3THD14trg_event_type20trg_action_time_typeb+0x105)[0x558b3a60df35]
            /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0xcf)[0x558b3a53e47f]
            /usr/sbin/mysqld(_ZN13select_insert9send_dataER4ListI4ItemE+0xf9)[0x558b3a53f129]
            /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa56)[0x558b3a5cf056]
            /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x558b3a5cf433]
            /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x558b3a5cd806]
            /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x558b3a5ce156]
            /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7c5f)[0x558b3a57874f]
            /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x558b3a57985a]
            /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1349)[0x558b3a57bdd9]
            /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558b3a57d66c]
            /usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0xe8)[0x558b3a75afd8]
            /usr/sbin/mysqld(+0x9f1ad0)[0x558b3a910ad0]
            /usr/sbin/mysqld(+0xd86551)[0x558b3aca5551]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f98014c24a4]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f97ffc20d0f]
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7f637c15a650): INSERT IGNORE INTO user_lnk_contact (user_id, client_id, contact_id, is_created)   SELECT 132561    , (SELECT client_id FROM sf_guard_user AS u WHERE u.id=132561)   , 44028334   , 1
            

            and the one from the first comment in MDEV-20290. It seems like the server crashes after executing trigger before/after insert.

            alice Alice Sherepa added a comment - c_migliorini , I compared stacktraces, e.g. from 191212-13-31-05 : /usr/sbin/mysqld(_ZN4JOIN7destroyEv+0x20)[0x558b3a5b52f0] /usr/sbin/mysqld(_ZN13st_select_lex7cleanupEv+0x6d)[0x558b3a61530d] /usr/sbin/mysqld(_ZN30subselect_single_select_engine7prepareEP3THD+0x3c)[0x558b3a838bdc] /usr/sbin/mysqld(_ZN14Item_subselect10fix_fieldsEP3THDPP4Item+0x143)[0x558b3a8385b3] /usr/sbin/mysqld(_ZN9Item_func10fix_fieldsEP3THDPP4Item+0x153)[0x558b3a7ed103] /usr/sbin/mysqld(_ZN13Item_func_not10fix_fieldsEP3THDPP4Item+0x1c6)[0x558b3a7c1076] /usr/sbin/mysqld(_ZN9Item_cond10fix_fieldsEP3THDPP4Item+0x11e)[0x558b3a7c187e] /usr/sbin/mysqld(_ZN3THD16sp_fix_func_itemEPP4Item+0x90)[0x558b3a4d9b90] /usr/sbin/mysqld(_ZN3THD20sp_prepare_func_itemEPP4Itemj+0xf)[0x558b3a4d9baf] /usr/sbin/mysqld(_ZN20sp_instr_jump_if_not9exec_coreEP3THDPj+0x1e)[0x558b3a4d9bfe] /usr/sbin/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x152)[0x558b3a4dfc32] /usr/sbin/mysqld(_ZN7sp_head7executeEP3THDb+0xa2c)[0x558b3a4db77c] /usr/sbin/mysqld(_ZN7sp_head15execute_triggerEP3THDPK25st_mysql_const_lex_stringS4_P13st_grant_info+0x23c)[0x558b3a4dc45c] /usr/sbin/mysqld(_ZN19Table_triggers_list16process_triggersEP3THD14trg_event_type20trg_action_time_typeb+0x105)[0x558b3a60df35] /usr/sbin/mysqld(_Z12write_recordP3THDP5TABLEP12st_copy_info+0xcf)[0x558b3a53e47f] /usr/sbin/mysqld(_ZN13select_insert9send_dataER4ListI4ItemE+0xf9)[0x558b3a53f129] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa56)[0x558b3a5cf056] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x33)[0x558b3a5cf433] /usr/sbin/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x186)[0x558b3a5cd806] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0xf6)[0x558b3a5ce156] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x7c5f)[0x558b3a57874f] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_statebb+0x23a)[0x558b3a57985a] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0x1349)[0x558b3a57bdd9] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x558b3a57d66c] /usr/sbin/mysqld(_Z11tp_callbackP13TP_connection+0xe8)[0x558b3a75afd8] /usr/sbin/mysqld(+0x9f1ad0)[0x558b3a910ad0] /usr/sbin/mysqld(+0xd86551)[0x558b3aca5551] /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7f98014c24a4] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f97ffc20d0f] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f637c15a650): INSERT IGNORE INTO user_lnk_contact (user_id, client_id, contact_id, is_created) SELECT 132561 , (SELECT client_id FROM sf_guard_user AS u WHERE u.id=132561) , 44028334 , 1 and the one from the first comment in MDEV-20290 . It seems like the server crashes after executing trigger before/after insert.
            c_migliorini Christophe Migliorini added a comment - - edited

            You're right for this one, although this is the only INSERT IGNORE of the set. Others are simple INSERT and there is no trigger on this table, only foreign keys.
            Here is the DDL, maybe there's something I'm not seeing?

            CREATE TABLE `user_lnk_contact` (
              `id` bigint(20) NOT NULL AUTO_INCREMENT,
              `client_id` bigint(20) DEFAULT NULL,
              `user_id` bigint(20) DEFAULT NULL,
              `contact_id` bigint(20) DEFAULT NULL,
              `is_created` tinyint(1) NOT NULL DEFAULT 0,
              `created_at` timestamp NULL DEFAULT current_timestamp(),
              `updated_at` timestamp NULL DEFAULT current_timestamp(),
              PRIMARY KEY (`id`),
              UNIQUE KEY `ixClientIdUserIdContactId` (`client_id`,`user_id`,`contact_id`),
              UNIQUE KEY `ixUserIdContactId` (`user_id`,`contact_id`),
              KEY `ixUserId` (`user_id`),
              KEY `ixContactId` (`contact_id`),
              KEY `xUserIdIsCreated` (`user_id`,`is_created`),
              KEY `ixContactIsCreatedCreatedAt` (`contact_id`,`is_created`,`created_at`),
              KEY `ixClientContact` (`client_id`,`contact_id`),
              KEY `ixClient` (`client_id`),
              CONSTRAINT `user_lnk_contact_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `sf_guard_user` (`id`) ON DELETE CASCADE,
              CONSTRAINT `user_lnk_contact_ibfk_2` FOREIGN KEY (`contact_id`) REFERENCES `contact` (`id`) ON DELETE CASCADE,
              CONSTRAINT `user_lnk_contact_ibfk_3` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON DELETE CASCADE
            ) ENGINE=InnoDB DEFAULT CHARSET=utf8
            

            c_migliorini Christophe Migliorini added a comment - - edited You're right for this one, although this is the only INSERT IGNORE of the set. Others are simple INSERT and there is no trigger on this table, only foreign keys. Here is the DDL, maybe there's something I'm not seeing? CREATE TABLE `user_lnk_contact` ( `id` bigint( 20 ) NOT NULL AUTO_INCREMENT, `client_id` bigint( 20 ) DEFAULT NULL, `user_id` bigint( 20 ) DEFAULT NULL, `contact_id` bigint( 20 ) DEFAULT NULL, `is_created` tinyint( 1 ) NOT NULL DEFAULT 0 , `created_at` timestamp NULL DEFAULT current_timestamp(), `updated_at` timestamp NULL DEFAULT current_timestamp(), PRIMARY KEY (`id`), UNIQUE KEY `ixClientIdUserIdContactId` (`client_id`,`user_id`,`contact_id`), UNIQUE KEY `ixUserIdContactId` (`user_id`,`contact_id`), KEY `ixUserId` (`user_id`), KEY `ixContactId` (`contact_id`), KEY `xUserIdIsCreated` (`user_id`,`is_created`), KEY `ixContactIsCreatedCreatedAt` (`contact_id`,`is_created`,`created_at`), KEY `ixClientContact` (`client_id`,`contact_id`), KEY `ixClient` (`client_id`), CONSTRAINT `user_lnk_contact_ibfk_1` FOREIGN KEY (`user_id`) REFERENCES `sf_guard_user` (`id`) ON DELETE CASCADE, CONSTRAINT `user_lnk_contact_ibfk_2` FOREIGN KEY (`contact_id`) REFERENCES `contact` (`id`) ON DELETE CASCADE, CONSTRAINT `user_lnk_contact_ibfk_3` FOREIGN KEY (`client_id`) REFERENCES `client` (`id`) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8
            alice Alice Sherepa added a comment -

            could you please add the output of "show triggers like 'user_lnk_contact';"

            alice Alice Sherepa added a comment - could you please add the output of "show triggers like 'user_lnk_contact';"

            sure

            MariaDB []> show triggers like 'user_lnk_contact';
            --------------
            show triggers like 'user_lnk_contact'
            --------------
             
            Empty set (0.003 sec)
            

            c_migliorini Christophe Migliorini added a comment - sure MariaDB []> show triggers like 'user_lnk_contact'; -------------- show triggers like 'user_lnk_contact' --------------   Empty set (0.003 sec)

            @Alice Sherepa, I'm sorry there is a trigger AFTER INSERT, I was not connected as root so I could not see it – n00b ...

            Is there a non-public server I can upload it to? I would rather not publish this piece of code.

            c_migliorini Christophe Migliorini added a comment - @Alice Sherepa, I'm sorry there is a trigger AFTER INSERT , I was not connected as root so I could not see it – n00b ... Is there a non-public server I can upload it to? I would rather not publish this piece of code.

            People

              Unassigned Unassigned
              c_migliorini Christophe Migliorini
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.