Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-35802

Race condition between log_t::persist() and log_t::write_checkpoint() on log resizing

    XMLWordPrintable

Details

    Description

      The function log_t::persist() is not acquiring or holding log_sys.latch. Therefore, the pmem_persist() calls in it can read an unprotected value of log_sys.buf, and the buffer can be switched when log resizing completes in log_t::write_checkpoint(). mleich produced got an rr replay trace of this race condition:

      Thread 5 hit Hardware watchpoint 1: -location log_sys.buf
       
      Old value = (byte *) 0x69c82d4c1000 <error: Cannot access memory at address 0x69c82d4c1000>
      New value = (byte *) 0x51d15d243000 "Phys"
      log_t::write_checkpoint (this=this@entry=0x55e831a23f80 <log_sys>, end_lsn=end_lsn@entry=4728132) at /data/Server/10.11-MDEV-35699A/storage/innobase/buf/buf0flu.cc:1887
      1887	        set_buf_free(START_OFFSET + (get_lsn() - resizing));
      (rr) thr a 3 bt
       
      Thread 3 (Thread 3154073.3161052 (mariadbd)):
      #0  pmem_clwb (buf=<optimized out>, size=7157) at /data/Server/10.11-MDEV-35699A/storage/innobase/sync/cache.cc:50
      #1  0x000055e8300a1cea in log_t::persist (this=this@entry=0x55e831a23f80 <log_sys>, lsn=lsn@entry=4593150, holding_latch=holding_latch@entry=false) at /data/Server/10.11-MDEV-35699A/storage/innobase/log/log0log.cc:914

      In this trace, we would hit SIGSEGV because munmap() had been called on the old log_sys.buf pointee:

      Dump of assembler code for function pmem_clwb(void const*, size_t):
         0x000055e8302978be <+0>:	endbr64 
         0x000055e8302978c2 <+4>:	mov    %rdi,%rax
         0x000055e8302978c5 <+7>:	and    $0xffffffffffffffbf,%rax
         0x000055e8302978c9 <+11>:	add    %rsi,%rdi
         0x000055e8302978cc <+14>:	jmp    0x55e8302978d6 <pmem_clwb(void const*, size_t)+24>
      => 0x000055e8302978ce <+16>:	clwb   (%rax)
         0x000055e8302978d2 <+20>:	add    $0x40,%rax
         0x000055e8302978d6 <+24>:	cmp    %rdi,%rax
         0x000055e8302978d9 <+27>:	jb     0x55e8302978ce <pmem_clwb(void const*, size_t)+16>
         0x000055e8302978db <+29>:	sfence 
         0x000055e8302978de <+32>:	ret    
      

      This only affects memory-mapped log writes (using mount -o dax file system or /dev/shm).

      Attachments

        Issue Links

          Activity

            People

              marko Marko Mäkelä
              marko Marko Mäkelä
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Git Integration

                  Error rendering 'com.xiplink.jira.git.jira_git_plugin:git-issue-webpanel'. Please contact your Jira administrators.