Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-34521

Running mysql_install_db on 10.11 results in crash by assert failure at log0log.cc

Details

    Description

      Attempt to set up a new database fails on MacOS by hitting the assert . Stack trace is below

      frame #7: 0x000000010d30a77f mariadbd`unsigned long long log_t::write_buf<true>(this=0x000000010e971480) at log0log.cc:863:7
         860 	    else
         861 	    {
         862 	      const size_t new_buf_free{length & write_size_1};
      -> 863 	      ut_ad(new_buf_free == ((lsn - first_lsn) & write_size_1));
         864 	      buf_free.store(new_buf_free, std::memory_order_relaxed);
         865
         866 	      if (new_buf_free)
      (lldb) p new_buf_free
      (const size_t) $0 = 183
      (lldb) p lsn
      (const lsn_t) $1 = 53970
      (lldb) p first_lsn
      (lsn_t) $2 = 12288
      (lldb) p lsn - first_lsn
      (unsigned long long) $3 = 41682
      (lldb) p write_size_1
      (const size_t) $4 = 511
      (lldb) p length
      (size_t) $5 = 2743
      (lldb) bt
      * thread #1
          frame #0: 0x00007ff8131adca6 libsystem_kernel.dylib`__kill + 10
          frame #1: 0x000000010c50dccf mariadbd`::handle_fatal_signal(sig=6) at signal_handler.cc:370:3
          frame #2: 0x00007ff8131f9e2d libsystem_platform.dylib`_sigtramp + 29
          frame #3: 0x00007ff8131ae113 libsystem_kernel.dylib`__pthread_kill + 11
          frame #4: 0x00007ff8131e4233 libsystem_pthread.dylib`pthread_kill + 263
          frame #5: 0x00007ff813130d10 libsystem_c.dylib`abort + 123
          frame #6: 0x00007ff8131300be libsystem_c.dylib`__assert_rtn + 314
        * frame #7: 0x000000010d30a77f mariadbd`unsigned long long log_t::write_buf<true>(this=0x000000010e971480) at log0log.cc:863:7
          frame #8: 0x000000010d308cc7 mariadbd`log_write_up_to(lsn=53970, durable=true, callback=0x0000000000000000) at log0log.cc:992:51
          frame #9: 0x000000010d144707 mariadbd`fil_ibd_create(space_id=4, name=(m_name = "mysql/gtid_slave_pos"), path="./mysql/gtid_slave_pos.ibd", flags=21, size=4, mode=FIL_ENCRYPTION_DEFAULT, key_id=1, err=0x00007ff7b3adae0c) at fil0fil.cc:1928:2
          frame #10: 0x000000010d0bcf31 mariadbd`dict_create_index_space(node=0x00007f93af210ff8) at dict0crea.cc:1153:17
          frame #11: 0x000000010d0bc19c mariadbd`dict_create_index_step(thr=0x00007f93af885a10) at dict0crea.cc:1207:9
          frame #12: 0x000000010d3bd3a7 mariadbd`que_thr_step(thr=0x00007f93af885a10) at que0que.cc:598:9
          frame #13: 0x000000010d3bb081 mariadbd`que_run_threads_low(thr=0x00007f93af885a10) at que0que.cc:644:25
          frame #14: 0x000000010d3bae24 mariadbd`que_run_threads(thr=0x00007f93af885a10) at que0que.cc:664:2
          frame #15: 0x000000010d451c55 mariadbd`row_create_index_for_mysql(index=0x00007f93af211cd8, trx=0x00007f9390249740, field_lengths=0x00006000036bc268, mode=FIL_ENCRYPTION_DEFAULT, key_id=1) at row0mysql.cc:2177:3
          frame #16: 0x000000010d20763e mariadbd`create_index(trx=0x00007f9390249740, form=0x00007ff7b3adc2e8, table=0x00007f93af85bc98, key_num=0) at ha_innodb.cc:11037:3
          frame #17: 0x000000010d204ac7 mariadbd`create_table_info_t::create_table(this=0x00007ff7b3adb400, create_fk=true) at ha_innodb.cc:12728:16
          frame #18: 0x000000010d20983c mariadbd`ha_innobase::create(this=0x00007f93af85aca0, name="./mysql/gtid_slave_pos", form=0x00007ff7b3adc2e8, create_info=0x00007ff7b3ade018, file_per_table=true, trx=0x00007f9390249740) at ha_innodb.cc:13205:17
          frame #19: 0x000000010d209fb7 mariadbd`ha_innobase::create(this=0x00007f93af85aca0, name="./mysql/gtid_slave_pos", form=0x00007ff7b3adc2e8, create_info=0x00007ff7b3ade018) at ha_innodb.cc:13249:10
          frame #20: 0x000000010c525129 mariadbd`handler::ha_create(this=0x00007f93af85aca0, name="./mysql/gtid_slave_pos", form=0x00007ff7b3adc2e8, info_arg=0x00007ff7b3ade018) at handler.cc:5588:14
          frame #21: 0x000000010c526e3b mariadbd`ha_create_table(thd=0x00007f93b0054c88, path="./mysql/gtid_slave_pos", db="mysql", table_name="gtid_slave_pos", create_info=0x00007ff7b3ade018, frm=0x00007ff7b3add750, skip_frm_file=false) at handler.cc:6055:22
          frame #22: 0x000000010cac135e mariadbd`create_table_impl(thd=0x00007f93b0054c88, ddl_log_state_create=0x00007ff7b3addba0, ddl_log_state_rm=0x00007ff7b3addb80, orig_db=0x00007f93a02320f0, orig_table_name=0x00007f93a0232100, db=0x00007f93a02320f0, table_name=0x00007f93a0232100, path=0x00007ff7b3add770, options=(m_options = OPT_IF_NOT_EXISTS), create_info=0x00007ff7b3ade018, alter_info=0x00007ff7b3addea8, create_table_mode=0, is_trans=0x00007ff7b3addb77, key_info=0x00007ff7b3add790, key_count=0x00007ff7b3add78c, frm=0x00007ff7b3add750) at sql_table.cc:4658:11
          frame #23: 0x000000010cac00e5 mariadbd`mysql_create_table_no_lock(thd=0x00007f93b0054c88, ddl_log_state_create=0x00007ff7b3addba0, ddl_log_state_rm=0x00007ff7b3addb80, create_info=0x00007ff7b3ade018, alter_info=0x00007ff7b3addea8, is_trans=0x00007ff7b3addb77, create_table_mode=0, table_list=0x00007f93a02320d8) at sql_table.cc:4760:8
          frame #24: 0x000000010cadf6d3 mariadbd`mysql_create_table(thd=0x00007f93b0054c88, create_table=0x00007f93a02320d8, create_info=0x00007ff7b3ade018, alter_info=0x00007ff7b3addea8) at sql_table.cc:4986:7
          frame #25: 0x000000010cadd6d3 mariadbd`Sql_cmd_create_table_like::execute(this=0x00007f93a0232060, thd=0x00007f93b0054c88) at sql_table.cc:12770:12
          frame #26: 0x000000010c97f566 mariadbd`mysql_execute_command(thd=0x00007f93b0054c88, is_called_from_prepared_stmt=true) at sql_parse.cc:6126:26
          frame #27: 0x000000010c9cb4a1 mariadbd`Prepared_statement::execute(this=0x00007f93b0042e88, expanded_query=0x00007ff7b3adf9d0, open_cursor=false) at sql_prepare.cc:5273:14
          frame #28: 0x000000010c9c628b mariadbd`Prepared_statement::execute_loop(this=0x00007f93b0042e88, expanded_query=0x00007ff7b3adf9d0, open_cursor=false, packet=0x0000000000000000, packet_end=0x0000000000000000) at sql_prepare.cc:4674:10
          frame #29: 0x000000010c9c5e0f mariadbd`mysql_sql_stmt_execute(thd=0x00007f93b0054c88) at sql_prepare.cc:3700:16
          frame #30: 0x000000010c974ac1 mariadbd`mysql_execute_command(thd=0x00007f93b0054c88, is_called_from_prepared_stmt=false) at sql_parse.cc:4004:5
          frame #31: 0x000000010c96acb4 mariadbd`mysql_parse(thd=0x00007f93b0054c88, rawbuf="EXECUTE stmt;", length=13, parser_state=0x00007ff7b3ae0f88) at sql_parse.cc:8143:18
          frame #32: 0x000000010c96a610 mariadbd`bootstrap(file=0x000000010e8f3d20) at sql_parse.cc:1081:5
          frame #33: 0x000000010c75f5d8 mariadbd`mysqld_main(argc=16, argv=0x00007f93af0044b8) at mysqld.cc:5980:26
          frame #34: 0x000000010c424132 mariadbd`main(argc=11, argv=0x00007ff7b3ae1448) at main.cc:34:10
      

      frame #35: 0x000000011eb414fe dyld`start + 462

      Attachments

        Issue Links

          Activity

            sitano Ivan Prisiazhnyi added a comment - - edited

            on Arch Linux x86_64 I was unable to reproduce it https://jira.mariadb.org/secure/ViewProfile.jspa?name=marko whereas on MBP m3 it pretty stably hits. I also have MBP x86_64, if you see sense I can try to verify if it hits there.

            sitano Ivan Prisiazhnyi added a comment - - edited on Arch Linux x86_64 I was unable to reproduce it https://jira.mariadb.org/secure/ViewProfile.jspa?name=marko whereas on MBP m3 it pretty stably hits. I also have MBP x86_64, if you see sense I can try to verify if it hits there.

            I don't have much time to debug it but it looked to me that there are not too much entry points in the code that change LSN vs Buffer write length. Thus it must be simple to add the asserts into those code points so to catch the stacktrace that caused the invariant violation if its not (probably not) a data race.

            sitano Ivan Prisiazhnyi added a comment - I don't have much time to debug it but it looked to me that there are not too much entry points in the code that change LSN vs Buffer write length. Thus it must be simple to add the asserts into those code points so to catch the stacktrace that caused the invariant violation if its not (probably not) a data race.

            I pushed a couple of experiments to test this on our CI. One common denominator between AIX and macOS is the lack of MDEV-26476. FreeBSD could be reasonably similar to those two, and maybe enabling SUX_LOCK_GENERIC (disabling the futex like system call) would make things fail there.

            marko Marko Mäkelä added a comment - I pushed a couple of experiments to test this on our CI. One common denominator between AIX and macOS is the lack of MDEV-26476 . FreeBSD could be reasonably similar to those two, and maybe enabling SUX_LOCK_GENERIC (disabling the futex like system call) would make things fail there.
            Gosselin Dave Gosselin added a comment - MDEV-34422

            Thank you, Gosselin. I see that my experiment was only successful on Windows, causing a build failure, in the step that tries to bootstrap a database:

            ntdll.dll!RtlEnterCriticalSection()         
            server.dll!pthread_mutex_wrapper<0>::wr_lock()[srw_lock.h:67]
            server.dll!log_t::append_prepare<0,0>()[mtr0mtr.cc:1042]
            server.dll!mtr_t::finish_writer<0,0>()[mtr0mtr.cc:1320]
            server.dll!mtr_t::finish_write()[mtr0mtr.h:724]
            server.dll!mtr_t::commit_files()[mtr0mtr.cc:786]
            

            The MemorySanitizer build on Linux did not catch the uninitialized mutex that https://github.com/MariaDB/server/pull/3433 is fixing. For some reason, I did not find any FreeBSD build.

            marko Marko Mäkelä added a comment - Thank you, Gosselin . I see that my experiment was only successful on Windows, causing a build failure, in the step that tries to bootstrap a database: ntdll.dll!RtlEnterCriticalSection() server.dll!pthread_mutex_wrapper<0>::wr_lock()[srw_lock.h:67] server.dll!log_t::append_prepare<0,0>()[mtr0mtr.cc:1042] server.dll!mtr_t::finish_writer<0,0>()[mtr0mtr.cc:1320] server.dll!mtr_t::finish_write()[mtr0mtr.h:724] server.dll!mtr_t::commit_files()[mtr0mtr.cc:786] The MemorySanitizer build on Linux did not catch the uninitialized mutex that https://github.com/MariaDB/server/pull/3433 is fixing. For some reason, I did not find any FreeBSD build.

            People

              Gosselin Dave Gosselin
              shulga Dmitry Shulga
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.