As part of this work, the function log_buffer_extend() will be removed.

As part of MDEV-14425 the recovery logic should be improved so that when a redo log block is corrupted, only the mini-transaction(s) that are (partly or fully) contained in the block will be skipped. This would augment MDEV-12699, which is about improving the recovery of corrupted data pages.

The function log_free_check() will have to be replaced.
With the original circularly-written InnoDB redo log, the function seeked to prevent a situation where the tail of the log is overwriting the head before the head is logically truncated by a redo log checkpoint. If such overwriting happens, InnoDB will be unable to recover from a crash. The situation would be normalized by a redo log checkpoint.

With these append-only log files, the overwriting issue is replaced with another one: running out of space in the file system. So, we will continue to need a function similar to log_free_check(). If the file system of the current thread’s log file is about to run out of space, the replacement of log_free_check() would return an ‘out of space’ error, which would be returned all the way up the call stack. If the ultimate caller is a client connection, the error would be reported as ER_DISK_FULL.

Running out of space should only be an issue when log archiving is enabled (innodb_log_max_size is overridden from its default value).

For the record: monty asked me to compare the relative performance of writing to a preallocated file and appending to a file.
On my HP ZBook 15u G3 laptop, I ran the test programs append.c and preallocate.c to write a 2GiB file on ext4fs, and the results are clear.
I used 2 SSD devices: 220GiB NVMe that is encrypted by dm-crypt, and a 450GiB SATA SSD that is not encrypted. With 4 programs running in parallel like this:

time ./append foo1&time ./append foo2&time ./append foo3&time ./append foo4

the reported real time was like this:

It would be interesting to test other relevant file systems as well as HDD.

Some server hardware stats:

Power Systems S822LC "Firestone", Power8 - 10 cores
Ubuntu-16.04 userspace, Kernel 4.14.0-27552-gadb89ad
nvme - IBM PCIe3 1.6TB NVMe Flash Adapter
disk - ST1000NX0313, 7200rpm - SATA 3.1, 2 disks in software raid1 (changed to 4.15.0-10668-g3527799 kernel - managed to hit kernel thread that hung)

ix10 is the i variable in the source * 10. Felt the size was too small at ~6 seconds.

error margin seems about 5% based on repetition (most results in the batch of 4 where identical however between batches varied). Assumed to be effects of journalling.

So ext3 is worse at prealloc. Multiple runs did occur. otherwise comparable.

Your preallocate code doesn't reflect the reality well. Since one log file can be written many times, they contains therandom data instead of just falloc. Your code simulate the situation when first time the log file get writes (brand new server). Please consider adding a command line parameter to open the existing file from previous run and skip falloc to simulate when the log file gets overwrite.

danblack, thanks for your benchmarks!

junsu, thank you for your comment. Another problem with the test program that there is only a fdatasync() call at the end, while in the reality there would be some fdatasync() whenever we need to make something durable.

I am not actively working on this task yet. I would welcome improved versions of the test programs, as well as more benchmarks, especially ones that would seem to indicate that preallocating is faster than appending.

In addition to the proposed parameter innodb_log_max_size there perhaps also should be some time limit after which the excessive old logs will be discarded. Perhaps innodb_log_min_age?
To ease maintenance and backups, maybe we should have an option to have strictly append-only log files, and should revise the design so that log files are never renamed. Most notably, there should be no header in the partitioned log files; the checkpoint block offsets would be written into the single-block control file ib_logfile0 only.

On a related note, last Tuesday I gave a high-level view of the InnoDB internals in a M18 talk Deep Dive: InnoDB Transactions and Write Paths (video). The slides with the diagram on mini-transactions and describing the log checkpoints could be useful background information.

Marko,

This is a very interesting idea. I believe we should think of redo log as nothing but a bunch of ordered changes to the pages. This implies that if for a given page we have all the changes available we should be able to deal with the gaps in the log (obviously, as long as we do flush all log files at checkpoint).

How about we map to log partition based on space_id:page_no i.e.: all changes to a page must always go to same logfile. If this invariant is maintained then we don't need to flush anything on mtr_commit(). As a trx touches different pages it will keep track of which logfiles it needs to flush. For small trxs like a single row DML it will be a single file. This calculation can be done at mtr_commit(). At trx_commit() we only flush the relevant files.

I think with above scheme we can allow gaps in LSN. Conceptually this design feels intuitive. Redo log records changes to pages. Redo logs are partitioned based on page number. Log buffer can also be partitioned extending the same logic.

As an aside, if we redesign the format may be we should add information during mtr_commit() about back pointer to last redo record that changed the page. If we are able to follow the chain of changes to a page that might actually be quite helpful. There might not be an immediate use case for MariaDB but then log format changes are cumbersome to orchestrate.

inaamrana, thank you for the valuable feedback.

Your suggestion to flush a subset of the log files at transaction commit and to ignoring gaps in the log recovery seems to imply a partial ordering of events. With the added invariant that modifications of a certain page must always go to a certain log file, I cannot see any obvious correctness problem. We would only need an additional field "total number of log files" for the mini-transaction, so that recovery can reject a mini-transaction that was not fully written to all log files. What if this is combined with physical replication? We should only replicate each mini-transaction log up to the latest log flush. Here it could be tricky to guarantee the atomicity of the replicated mini-transactions without flushing all logs up to our mini-transaction commit LSN.

This architecture would seem to require some scatter-gather operation or partitioning of the local mini-transaction buffer, so that the log for pages are written to the appropriate log files. Maybe the easiest way to arrange that would be by copying log snippets from local mini-transaction buffers to a per-log global buffer. This would also imply that the mini-transaction log for a given mini-transaction (identified by logical LSN) can exist in multiple log files.

For maximum commit concurrency, I think that in this scheme, there should be 1 redo log for each rollback segment (now that with MDEV-15132 and MDEV-15158 commit only writes to the rollback segment header and undo log header pages, not the TRX_SYS page). For maximal concurrency, we could experiment with dedicated redo logs for transaction metadata, and separate redo logs for data file changes.

This is definitely worth trying. I think that we should prototype and benchmark both approaches before committing to a solution.

The back-pointer to the last record that changed the page sounds like a good idea to me, and certainly useful for troubleshooting. When the full log is archived, this could allow faster point-in-time recovery. In your suggested scheme, this could be a byte offset from the start of the file. In my original scheme, it should probably be LSN, and some searching would be needed to find the record among the log files.

In my opinion, there are issues on Inaam's idea. The below is a use case.
1. Mini-transaction m1 updates page A via log file 1, and updates page B via log file 2.
2. Mini-transaction m2 updates page A via log file 1.
3. All write to log file 1 completes.
4. m2 is in the user transaction tr2. tr2 commits successfully.
5. But at this time, the write to log file 2 for m1 hasn't been finished
6. The system is killed.
7. Recover sees the commit of m2. The change of page A made by m2 is replayed.
8. Recover doesn't see the completed log events of m1 in log file 2. m1 isn't completed. m1 is discarded. The change of page A which is made by m1 is ignored. By this, page A may be inconsistent.

In short, there might be uncommitted transactions which modify the same page of our transaction. The mini transactions of such uncommitted ones may write additional log file which is other than the ones written by our transaction. So to avoid the above issue, when commit our transaction, beside of the redo log file the current transaction writes, we also need to flush all redo log files written by such uncommitted transactions.

Because of this, Inaam's idea might not perform well in practice.

I agree with micai that likely the only practical solution for preventing the scenario is to flush all redo log files whenever state change needs to be made durable in the database. This would seem to remove any performance benefits of partitioning the log file.

inaamrana’s idea sould still work in special cases, such as when each mini-transaction modifies its private set of pages, or if a (short) user transaction keeps page locks for the whole duration of the transaction. Maybe we could consider having separate groups of undo-redo-log or rseg-redo-log files, and allowing some level of partial ordering among related mini-transaction commits.

I was also thinking about extending the checkpoint information. Perhaps we should store all checkpoints in a separate sequential log file, pointing to the individual log files that contain changes since the start of the checkpoint. Perhaps all MLOG_FILE_ entries should be written to the checkpoint log file, while the log record files would only contain page-level log.

micai you are right. I haven't thought it through enough. Back to the drawing board

Regarding the preallocate.c vs append.c, today I found out that there still exist file systems where writing to a preallocated file is faster than appending to a file. This means that we should continue to offer an option where the log file is preallocated and written in circular fashion, instead of being written in append-only mode.

I also ran a modified version of preallocate.c that would omit the O_CREAT flag and the posix_fallocate() call. On ext4, fallocate -l 2g file completed in virtually no time, and using the modified test program to write to the preallocated file took around 25 seconds on the same hardware where I previously reported 24.2 seconds.

MDEV-15914 (and its main fix) showed that a small change to the redo log volume can have a huge impact on performance.
I think that the redo log record format must allow multiple byte strings to be written to the same page without repeating the tablespace identifier or page number.

The VCDIFF format implemented in Xdelta could be a good starting point for a new redo log format.

The bsdiff format has a different design goal: using a lot of RAM and CPU, create a minimal "binary patch" that can be distributed to a large number of clients. In InnoDB, there usually is at most 1 "consumer" of the redo log: the InnoDB crash recovery.

Related to this work, MDEV-18115 will stop creating a fil_space_t object and using the fil_io() and fil_flush(SRV_LOG_SPACE_FIRST_ID) interfaces for writing to the redo log files. This should reduce contention on fil_system.mutex.

While implementing this, we should ensure that the latest redo log block is never being overwritten. InnoDB is currently doing that, and mariabackup is compensating it by re-reading the latest redo log block if it was not completely filled. Rewriting the latest redo log block feels crash-unsafe: if the server is killed during the write, you could end up with a corrupted log block, and lose a few redo log records. Worst case, you would lose a already durable transaction commit, or some pages would have been already flushed with the LSN of the mini-transaction that was lost because of the log block corruption.

I think that it should be simplest to exclusively use synchronous I/O for the redo log. Currently, the log checkpoint write uses asynchronous I/O.

My reading of man 2 write suggests that whether or not the individual redo log files are append-only or written in a circular fashion, we should not need any mutex to guard concurrent writes from multiple threads to a file through a shared file descriptor:

For a seekable file (i.e., one to which lseek(2) may be applied, for example, a regular file) writing takes place at the file offset, and the file offset is incremented by the number of bytes actually written. If the file was open(2)ed with O_APPEND, the file offset is first set to the end of the file before writing. The adjustment of the file offset and the write operation are performed as an atomic step.
…
BUGS

According to POSIX.1-2008/SUSv4 Section XSI 2.9.7 ("Thread Interactions with Regular File Operations"):

"All of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2008 when they operate on regular files or symbolic links: ..."

Among the APIs subsequently listed are write() and writev(2). And among the effects that should be atomic across threads (and processes) are updates of the file offset. However, on Linux before version 3.14, this was not the case: if two processes that share an open file description (see open(2)) perform a write() (or writev(2)) at the same time, then the I/O op‐ erations were not atomic with respect updating the file offset, with the result that the blocks of data output by the two processes might (incorrectly) overlap. This problem was fixed in Linux 3.14.

The mentioned Linux kernel bug should not affect InnoDB, because InnoDB would be writing to the log files from multiple threads of the same process.

The key seems to be to invoke write() or similar function that uses and updates the current position of the file descriptor. pwrite() would require the caller to keep track of a position, and we would do not want that. All the log of the mini-transaction should be written with a single system call. We will probably want some framing with explicit length and checksum around each mini-transaction log snippet, instead of forcing the log to be structured as blocks.

Edit: An open problem with multiple concurrent threads writing to a file is that each write can be truncated into a partial write if the write is interrupted by a signal. If such an interrupting signal can be sent only to a subset of the file-writing threads, the log could easily be corrupted. Having the very last write truncated due to the server being killed is tolerable, but continuing writes to the log after a truncated write is not.

Hence, I believe that it could be cleaner to have O_DIRECT synchronous write requests from a single thread, writing full, aligned blocks. Partly filled blocks would only be written on log_write_up_to(), just like it is now. The main difference would be that log could be written into multiple files in parallel.

We might also employ some form of Lempel-Ziv compression on the log data that is going to be written. This would require identifying some ‘restart points’ for parsing the log.

MDEV-12353 will improve the format of individual redo log records. This task will implement some framing around them, such as compression, division into blocks, LSN assignment. I think that for now, we will write the entire log of a single mini-transaction into a single log file.

Some ideas for improving the redo log block format:

• Use a variable block size, so that a mini-transaction will never be split between blocks. Write the size at the start of the block and the checksum at the end. In this way, we can make the collection of log records point directly to the parse buffer, and also remove recv_data_copy_to_buf(). (If log blocks are compressed, decompression would output to the parse buffer.)
• Each time a new block is seen, it will mark the start of a new mini-transaction. In this way, the log blocks can avoid encoding the LSN (which would grow by one per mini-transaction commit).
• Allow NUL-padding of short blocks to the physical block size, so that if a log flush is needed, read-modify-write on the file system can be avoided.

When it comes to file operations and log checkpoints, I think that it could be worthwhile to have a separate sequential log file that keeps a number of latest checkpoints as well as all the file operations. The checkpoint information in the checkpoint log file would point to the data log files, which would only contain page-level redo log records. This would remove the need to call the equivalent of fil_names_clear() at log checkpoint. Only when emptying or rotating the checkpoint log file we would write the equivalent of MLOG_FILE_NAME records to the new checkpoint log file.

I think that we must abandon the idea of partitioning the redo log. I would go with 3 files:

• mostly dummy ib_logfile0 to identify the file format
• an append-only, binlog-style-rotated file for checkpoints and file-level operations (create, delete, rename, modify)
• a page-level redo log file, 2 variants: block-oriented circular, or append-only
• LSN will be logical, incremented by 1 on mtr_t::commit() when redo log records were generated.

Even in the circular log file format, page-level redo log records will never be interrupted by log block trailers or headers. That is, we will write variable-size blocks, with the LSN at the start of the block (to detect the end of "new" log).

In the byte-oriented append-only log file format, if a persistent write is requested (on user transaction commit), we will write an extra record that contains a checksum of the bytes that were written since the previous persistent write. The payload of the stream could even be encrypted.

To alleviate the log_sys.mutex and log_sys.write_mutex bottleneck, we will introduce a dedicated log writing task, which will:

• Get mtr_t::log records from and encode them to a private buffer of this task
• In case of a circular log file, issue a log checkpoint if the log tail would overwrite the head. (Other calls to log_free_check() can be removed!)
• Before a log checkpoint is issued, any background crash recovery (MDEV-14481) must be finished.
• If persistence is requested, issue a synchronous write of the data to the log file.

Because page-level log records of any single mini-transaction will be continuous streams of bytes in the log file, on recovery we can avoid copying log record snippets to recv_sys.pages. Instead, we can simply attach pointers to log_sys.buf that was read from the page-level redo log file. This should automatically fix MDEV-19176 when using the MDEV-12353 format. Note: compressing the log record stream could prevent such optimization, so we will not introduce any compression for now.

Currently, on recovery, redo log records are being copied twice from memory to memory:

1. from redo log file blocks to contiguous strings of bytes
2. from contiguous strings of bytes to recv_t (in limited-size chunks that are allocated from the buffer pool)

We have a 2MiB recv_sys.buf for the initial buffering. The minimum size of log_sys.buf would be 16MiB, and that buffer should be practically unused during recovery. If the buffer pool size is measured in gigabytes, it would indeed make sense to use the buffer pool for the recovered log records.

I updated MDEV-19176 with a suggested design how to improve the buffer pool utilization during crash recovery. It is independent of the redo log record or file format.

When innodb_scrub_log=ON, log checkpoint should clear the unused part of the redo log, simply by invoking fallocate(fd, FALLOC_FL_PUNCH_HOLE, offset, len) or its Windows equivalent. There is no need for a separate log_scrub_thread. The hole-punching should work with both circular and append-only log files.

Removing the log_scrub_thread should fix MDEV-18370, MDEV-20474, MDEV-20475.

This is my test program which was created to test performance of writing to circular file and appending to file. I've run test on my laptop with HDD and ext4.

Results without fsync():

File size 524288 Kb

Writing 4194304 Kb to it

Simple cyclic file

Took 6 seconds 714 milliseconds

Mmapped cyclic file

Took 0 seconds 532 milliseconds

O_APPEND append file

Took 25 seconds 260 milliseconds

Simple append file

Took 24 seconds 623 milliseconds

Results with fsync():

File size 1024 Kb

Writing 4096 Kb to it

Simple cyclic file

Took 59 seconds 111 milliseconds

O_DSYNC cyclic file

Took 59 seconds 267 milliseconds

Mmapped cyclic file

Took 58 seconds 516 milliseconds

O_DIRECT|O_DSYNC cyclic file

Took 107 seconds 6 milliseconds

O_APPEND append file

Took 202 seconds 492 milliseconds

Simple append file

Took 249 seconds 301 milliseconds

And the program itself:

#include <sys/types.h>

#include <sys/mman.h>

#include <sys/stat.h>

#include <fcntl.h>

#include <unistd.h>

#include <cerrno>

#include <cassert>

#include <cstdlib>

#include <cstring>

#include <array>

#include <chrono>

#include <string>

#include <iostream>

static const size_t kWriteTotal = 4 * 1024 * 1024;

static const size_t kFileSize = 1 * 1024 * 1024;

static const std::array<unsigned char, 1024> kBuf = { 33 };

static const std::array<unsigned char, 1024> kAlignedBuf alignas(512) = { 33 };

static_assert(kFileSize % kBuf.size() == 0, "");

static const std::string kPath = "test_file";

void ShowErrnoAndExit(std::string function_name) {

  std::cerr << function_name << " returned errno " << strerror(errno) << "\n";

  std::exit(EXIT_FAILURE);

}

struct File {

  File(std::string path, int additional_flags = 0) : path_(std::move(path)) {

    fd_ = open(path_.c_str(),

	       O_CREAT | O_TRUNC | O_RDWR | additional_flags,

	       S_IRUSR | S_IWUSR);

    if (fd_ == -1)

      ShowErrnoAndExit("open()");

  }

  void Resize(off_t length) {

    assert(fd_ != -1);

    if (int ret = posix_fallocate(fd_, 0, length)) {

      errno = ret;

      ShowErrnoAndExit("posix_fallocate()");

    }

  }

  void Write(const void *buf, size_t count) {

    assert(fd_ != -1);

    ssize_t ret = write(fd_, buf, count);

    if (ret == -1)

      ShowErrnoAndExit("write()");

    if (static_cast<size_t>(ret) != count) {

      std::cerr << "write() partial write\n";

      std::exit(EXIT_FAILURE);

    }

  }

  void PWrite(const void *buf, size_t count, off_t offset) {

    assert(fd_ != -1);

    ssize_t ret = pwrite(fd_, buf, count, offset);

    if (ret == -1)

      ShowErrnoAndExit("pwrite()");

    if (static_cast<size_t>(ret) != count) {

      std::cerr << "pwrite() partial write\n";

      std::exit(EXIT_FAILURE);

    }

  }

  void Fsync() {

    assert(fd_ != -1);

    if (fsync(fd_) == -1)

      ShowErrnoAndExit("fsync()");

  }

  void Fdatasync() {

    assert(fd_ != -1);

    if (fdatasync(fd_) == -1)

      ShowErrnoAndExit("fdatasync()");

  }

  size_t Size() {

    assert(fd_ != 1);

    struct stat s;

    if (fstat(fd_, &s) == -1)

      ShowErrnoAndExit("fstat()");

    return s.st_size;

  }

  void Mmap() {

    assert(fd_ != -1);

    size_t size = Size();

    mapped_ = mmap(nullptr, size,

		   PROT_READ | PROT_WRITE,

		   MAP_SHARED | MAP_POPULATE,

		   fd_, 0);

    if (mapped_ == MAP_FAILED)

      ShowErrnoAndExit("mmap()");

  }

  unsigned char *GetMmappedRegion() {

    assert(fd_ != -1);

    assert(mapped_);

    return static_cast<unsigned char *>(mapped_);

  }

  void Msync(void *addr, size_t len) {

    if (msync(addr, len, MS_SYNC) == -1) {

      ShowErrnoAndExit("msync()");

    }

  }

  void Munmap() {

    assert(fd_ != -1);

    assert(mapped_ != nullptr);

    if (munmap(mapped_, Size()) == -1)

      ShowErrnoAndExit("munmap()");

    mapped_ = nullptr;

  }

  ~File() {

    if (mapped_)

      Munmap();

    if (fd_ != -1 && close(fd_) == -1)

      ShowErrnoAndExit("close()");

    if (unlink(path_.c_str()) == -1)

      ShowErrnoAndExit("unlink()");

  }

  int fd_{-1};

  void *mapped_{nullptr};

  std::string path_;

};

struct SimpleCyclicWriter {

  SimpleCyclicWriter(std::string path) : file_(path) {

    file_.Resize(kFileSize);

    file_.Fsync();

  }

  static std::string Name()  { return "Simple cyclic file"; }

  void Write() {

    const off_t offset = offset_ % kFileSize;

    file_.PWrite(kBuf.data(), kBuf.size(), offset);

    offset_ += kBuf.size();

  }

  void Flush() { file_.Fdatasync(); }

  File file_;

  off_t offset_{0};

};

struct DSyncCyclicWriter {

  DSyncCyclicWriter(std::string path) : file_(path, O_DSYNC) {

    file_.Resize(kFileSize);

    file_.Fsync();

  }

  static std::string Name()  { return "O_DSYNC cyclic file"; }

  void Write() {

    const off_t offset = offset_ % kFileSize;

    file_.PWrite(kBuf.data(), kBuf.size(), offset);

    offset_ += kBuf.size();

  }

  void Flush() { }

  File file_;

  off_t offset_{0};

};

struct ODirectODsyncCyclicWriter {

  ODirectODsyncCyclicWriter(std::string path) :

      file_(path, O_DIRECT | O_DSYNC) {

    file_.Resize(kFileSize);

    file_.Fsync();

  }

  static std::string Name()  { return "O_DIRECT|O_DSYNC cyclic file"; }

  void Write() {

    const off_t offset = offset_ % kFileSize;

    file_.PWrite(kAlignedBuf.data(), kAlignedBuf.size(), offset);

    offset_ += kAlignedBuf.size();

  }

  void Flush() { }

  File file_;

  off_t offset_{0};

};

struct MmappedCyclicWriter {

  MmappedCyclicWriter(std::string path) : file_(path) {

    file_.Resize(kFileSize);

    file_.Fsync();

    file_.Mmap();

  }

  static std::string Name()  { return "Mmapped cyclic file"; }

  void Write() {

    const off_t offset = offset_ % kFileSize;

    memcpy(file_.GetMmappedRegion() + offset, kBuf.data(), kBuf.size());

    prev_offset_ = offset;

    offset_ += kBuf.size();

  }

  void Flush() {

    auto *start = file_.GetMmappedRegion() + prev_offset_;

    auto *end = start + kBuf.size();

    start = start - reinterpret_cast<std::uintptr_t>(start) % page_size_;

    assert(start >= file_.GetMmappedRegion());

    assert(end <= file_.GetMmappedRegion() + file_.Size());

    file_.Msync(start, end - start);

  }

  File file_;

  uintptr_t page_size_{static_cast<uintptr_t>(sysconf(_SC_PAGE_SIZE))};

  off_t prev_offset_{0};

  off_t offset_{0};

};

struct OAppendAppendWriter {

  OAppendAppendWriter(std::string path) : file_(path, O_APPEND) {}

  static std::string Name()  { return "O_APPEND append file"; }

  void Write() { file_.Write(kBuf.data(), kBuf.size()); }

  void Flush() { file_.Fsync(); }

  File file_;

};

struct SimpleAppendWriter {

  SimpleAppendWriter(std::string path) : file_(path) {}

  static std::string Name()  { return "Simple append file"; }

  void Write() { file_.Write(kBuf.data(), kBuf.size()); }

  void Flush() { file_.Fsync(); }

  File file_;

};

struct Timer {

  using Clock = std::chrono::steady_clock;

  ~Timer() {

    auto duration = Clock::now() - now_;

    auto ms = std::chrono::duration_cast<std::chrono::milliseconds>(duration);

    auto s = std::chrono::duration_cast<std::chrono::seconds>(ms);

    ms = ms - s;

    std::cout << "Took " << s.count() << " seconds " << ms.count() <<

        " milliseconds\n";

  }

  Clock::time_point now_ = Clock::now();

};

template <class Writer>

void Test() {

  std::cout << Writer::Name() << "\n";

  Writer file_(kPath);

  {

    Timer timer;

    size_t write_total = kWriteTotal;

    while (write_total) {

      file_.Write();

      file_.Flush();

      write_total -= kBuf.size();

    }

  }

  std::cout << "\n";

}

int main() {

  std::cout << "File size " << kFileSize / 1024 << " Kb\n";

  std::cout << "Writing " << kWriteTotal / 1024 << " Kb to it\n";

  std::cout << "\n";

  Test<SimpleCyclicWriter>();

  Test<DSyncCyclicWriter>();

  Test<MmappedCyclicWriter>();

  Test<ODirectODsyncCyclicWriter>();

  Test<OAppendAppendWriter>();

  Test<SimpleAppendWriter>();

  return EXIT_SUCCESS;

}

So, writing to circular file is clearly much faster than appending to file.

junsu danblack could you please run my program on some server hardware? I have no access to anything but my not so up to date laptop.

The biggest questing is to decide what's faster: appending to file or writing to a circular file. Sorry, no CLI interface for my program, but you tweak file size and amount of data to write by changing globals.

Hey guys. I have done almost the same work. change the redo log from circular file to appending to new file..
Of course writing to circular file is much faster than appending to file. Since when appending to file, it need to modify the inode and allocate an extent to the file, which need about 8 times than writing to circular file. I have show the result in this slide https://www.slideshare.net/baotiao/polardb-percona19 from page 18.
However, writing to circular file need to solve "read-on-write" issue, but appending to file don't need it.

so the way we use in our environment is that when there is no stale redo log file, we allocate a new redo log file and filling the file with zero. And in the background, if we don't need some stale redo log file, we don't delete directly, we rename the stale file as new redo log file. And in InnoDB, we also padding the write to 4k to avoid the "read-on-write" issue.

static const size_t kFileSize = 1 * 128 * 1024 * 1024;

static const size_t kWriteTotal = 4 * kFileSize;

$ ~/write_test

File size 131072 Kb

Writing 524288 Kb to it

Simple cyclic file

Took 132 seconds 538 milliseconds

O_DSYNC cyclic file

Took 61 seconds 86 milliseconds

Mmapped cyclic file

Took 248 seconds 543 milliseconds

O_DIRECT|O_DSYNC cyclic file

pwrite() returned errno Invalid argument

O_APPEND append file

Took 959 seconds 463 milliseconds

Simple append file

Took 770 seconds 324 milliseconds

./write_test 

File size 131072 Kb

Writing 524288 Kb to it

Simple cyclic file

Took 98 seconds 359 milliseconds

O_DSYNC cyclic file

Took 99 seconds 659 milliseconds

Mmapped cyclic file

Took 100 seconds 107 milliseconds

O_DIRECT|O_DSYNC cyclic file

Took 42 seconds 208 milliseconds

O_APPEND append file

Took 300 seconds 383 milliseconds

Simple append file

Took 303 seconds 216 milliseconds

Looking why NVMe was slow - seems to be 4k LBA:

smartctl 6.6 2016-05-31 r4324 [ppc64le-linux-5.3.0] (local build)

Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

Model Number:                       PCIe3 1.6TB NVMe Flash Adapter

Serial Number:                      CJH0010003EE

Firmware Version:                   KMIPP107

PCI Vendor ID:                      0x1c58

PCI Vendor Subsystem ID:            0x1014

IEEE OUI Identifier:                0x000cca

Controller ID:                      1269

Number of Namespaces:               1

Namespace 1 Size/Capacity:          1,600,321,314,816 [1.60 TB]

Namespace 1 Formatted LBA Size:     4096

Local Time is:                      Wed Jan  8 13:22:03 2020 AEDT

Firmware Updates (0x08):            4 Slots

Optional Admin Commands (0x0006):   Format Frmw_DL

Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat

Supported Power States

St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat

 0 +    25.00W       -        -    0  0  0  0    15000   15000

 1 +    20.00W       -        -    1  1  1  1    15000   15000

 2 +    15.00W       -        -    2  2  2  2    15000   15000

 3 +    10.00W       -        -    3  3  3  3    15000   15000

 4 -    10.00W       -        -    3  3  3  3    15000   15000

Supported LBA Sizes (NSID 0x1)

Id Fmt  Data  Metadt  Rel_Perf

 0 +    4096       0         0

 1 -    4096       8         1

=== START OF SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0x1)

Critical Warning:                   0x00

Temperature:                        36 Celsius

Available Spare:                    100%

Available Spare Threshold:          10%

Percentage Used:                    0%smartctl 6.6 2016-05-31 r4324 [ppc64le-linux-5.3.0] (local build)

Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===

Model Number:                       PCIe3 1.6TB NVMe Flash Adapter

Serial Number:                      CJH0010003EE

Firmware Version:                   KMIPP107

PCI Vendor ID:                      0x1c58

PCI Vendor Subsystem ID:            0x1014

IEEE OUI Identifier:                0x000cca

Controller ID:                      1269

Number of Namespaces:               1

Namespace 1 Size/Capacity:          1,600,321,314,816 [1.60 TB]

Namespace 1 Formatted LBA Size:     4096

Local Time is:                      Wed Jan  8 13:22:03 2020 AEDT

Firmware Updates (0x08):            4 Slots

Optional Admin Commands (0x0006):   Format Frmw_DL

Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat

Supported Power States

St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat

 0 +    25.00W       -        -    0  0  0  0    15000   15000

 1 +    20.00W       -        -    1  1  1  1    15000   15000

 2 +    15.00W       -        -    2  2  2  2    15000   15000

 3 +    10.00W       -        -    3  3  3  3    15000   15000

 4 -    10.00W       -        -    3  3  3  3    15000   15000

Supported LBA Sizes (NSID 0x1)

Id Fmt  Data  Metadt  Rel_Perf

 0 +    4096       0         0

 1 -    4096       8         1

=== START OF SMART DATA SECTION ===

SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0x1)

Critical Warning:                   0x00

Temperature:                        36 Celsius

Available Spare:                    100%

Available Spare Threshold:          10%

Percentage Used:                    0%

Data Units Read:                    3,643,068 [1.86 TB]

Data Units Written:                 6,647,820 [3.40 TB]

Host Read Commands:                 83,729,047

Host Write Commands:                81,415,498

Controller Busy Time:               841

Power Cycles:                       1,569

Power On Hours:                     22,345

Unsafe Shutdowns:                   516

Media and Data Integrity Errors:    0

Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 63 entries)

No Errors Logged

Data Units Read:                    3,643,068 [1.86 TB]

Data Units Written:                 6,647,820 [3.40 TB]

Host Read Commands:                 83,729,047

Host Write Commands:                81,415,498

Controller Busy Time:               841

Power Cycles:                       1,569

Power On Hours:                     22,345

Unsafe Shutdowns:                   516

Media and Data Integrity Errors:    0

Error Information Log Entries:      0

Error Information (NVMe Log 0x01, max 63 entries)

No Errors Logged

sudo dumpe2fs -h /dev/nvme0n1p1 | more

dumpe2fs 1.44.1 (24-Mar-2018)

Filesystem volume name:   scratch

Last mounted on:          /scratch

Filesystem UUID:          356af640-5d8e-4256-9dce-a0983c9e0e43

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum

Filesystem flags:         unsigned_directory_hash 

Default mount options:    user_xattr acl

Filesystem state:         clean

Errors behavior:          Continue

Filesystem OS type:       Linux

Inode count:              65052672

Block count:              260208384

Reserved block count:     13010419

Free blocks:              132171164

Free inodes:              62683469

First block:              0

Block size:               4096

Fragment size:            4096

Reserved GDT blocks:      961

Blocks per group:         32768

Fragments per group:      32768

Inodes per group:         8192

Inode blocks per group:   512

Flex block group size:    16

Filesystem created:       Wed Nov 22 15:34:07 2017

Last mount time:          Tue Jan  7 20:38:12 2020

Last write time:          Tue Jan  7 20:38:12 2020

Mount count:              293

Maximum mount count:      -1

Last checked:             Wed Nov 22 15:34:07 2017

Check interval:           0 (<none>)

Lifetime writes:          20 TB

Reserved blocks uid:      0 (user root)

Reserved blocks gid:      0 (group root)

First inode:              11

Inode size:	          256

Required extra isize:     32

Desired extra isize:      32

Journal inode:            8

Default directory hash:   half_md4

Directory Hash Seed:      b68c7dc9-1e4b-46d2-b6d2-6f2d0128b12a

Journal backup:           inode blocks

Checksum type:            crc32c

Checksum:                 0xb0abf812

Journal features:         journal_incompat_revoke journal_checksum_v3

Journal size:             1024M

Journal length:           262144

Journal sequence:         0x0050f6ac

Journal start:            153997

Journal checksum type:    crc32c

Journal checksum:         0xc52e4213

static const size_t kFileSize = 1 * 128 * 1024 * 1024; 

static const size_t kWriteTotal = 4 * kFileSize;

static const size_t kBuffSize = 4096;

static const std::array<unsigned char, kBuffSize> kBuf = { 33 };

static const std::array<unsigned char, kBuffSize> kAlignedBuf alignas(kBuffSize) = { 33 };

File size 131072 Kb

Writing 524288 Kb to it

Simple cyclic file

Took 49 seconds 300 milliseconds

O_DSYNC cyclic file

Took 57 seconds 536 milliseconds

Mmapped cyclic file

Took 75 seconds 627 milliseconds

O_DIRECT|O_DSYNC cyclic file

Took 64 seconds 600 milliseconds

O_APPEND append file

Took 231 seconds 47 milliseconds

Simple append file

Took 229 seconds 604 milliseconds

File size 131072 Kb

Writing 524288 Kb to it

Simple cyclic file

Took 16 seconds 5 milliseconds

O_DSYNC cyclic file

Took 15 seconds 664 milliseconds

Mmapped cyclic file

Took 17 seconds 846 milliseconds

O_DIRECT|O_DSYNC cyclic file

Took 14 seconds 823 milliseconds

O_APPEND append file

Took 48 seconds 328 milliseconds

Simple append file

Took 48 seconds 207 milliseconds

xfs_info /var

meta-data=/dev/mapper/ka4_disks-ka4_var isize=512    agcount=4, agsize=73119744 blks

         =                       sectsz=4096  attr=2, projid32bit=1

         =                       crc=1        finobt=1 spinodes=0 rmapbt=0

         =                       reflink=0

data     =                       bsize=4096   blocks=292478976, imaxpct=5

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal               bsize=4096   blocks=142812, version=2

         =                       sectsz=4096  sunit=1 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

Seems this was setup for 4k too.

@Marko why did you abandon the work of partitioned the redo log. Since we know that AWS aurora must have done this work. Otherwise, they can't partitioned the data by space_id:page_id into multi storage node. The redo log which modify the page must stay in the same partition with the page. This is the basic design that then can apply the redo log and crash recovery parallel..

I really think that partitioned redo log is a good idea, since in the compute-storage separation architecture, InnoDB need to support much more larger data size. such as 20T or 100T. POLARDB has meet this case, If we don't partitioned the redo log and data page, then we can't parallel well..

baotiao thank you for your answer! I think to solve `read-on-write` we can use posix_fadvise() or posix_mavdise(). Did you try that? Now as I understand you fill file with zeroes to force OS to cache it. Can you instead preread it somehow? At first glance that looks less invasive than writing.

NO, The root cause is that if the read size isn't aligned to 4k block, os need to read the whole 4k block, and modify the data you want. The write operation need another read operation

The fill file with zero operation solve the allocation of extent when append to file. If the address in file hasn't beed written before then it need allocate extent from file system.

danblack hi. We become interested in write optimization FUA (https://bobsql.com/sql-server-on-linux-forced-unit-access-fua-internals/) It's implemented at least on XFS. And you performed benchmarks on it where `O_DIRECT|O_DSYNC` was the fastest thing. Do you have that FUA enabled? You probably can check it like this:

$ dmesg | grep -i fua

[    1.549434] sd 3:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

O_DIRECT|O_DSYNC cyclic file

pwrite() returned errno Invalid argument

Do you think it was a bug in a testing program?

Also, maybe you know what `fdatasync()` does for file descriptors opened with `O_DSYNC`? In my understanding it whould be a no-op.

pwrite() returned errno Invalid argument - I assume this was writing a 512 block when the underlying layer was 4k as changing to 4k got a result for this. fstat on the fs and use `st_blksize` probably avoids this.

Seem FUA is a standard part of nvme (which I used) and the linux nvme driver has some concept of it based on its codebase.

Appears to be nothing special with `O_DSYNC` (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/sync.c?h=v5.5#n196) nothing seemingly special at ext4/xfs implementation either.

baotiao, sorry, I missed your question.

I think that if we eliminate the undo log pages and the TRX_SYS page and write all user transaction data only to the redo log, we will remove an artificial synchronization point between independent transactions. I have been toying with the idea, ever since junsu challenged me to think how to make more efficient use of NVDIMM or PMEM (byte-addressable persistent storage). The idea would be to write undo log records into the redo log, like many databases do it. The MDEV-12353 redo log format does allow this easily. We could even do memory-mapped I/O and let the DB_ROLL_PTR be a direct pointer to the redo log, to speed up MVCC and ROLLBACK. I have not come up with any good solution for the redo log checkpointing, though. In this scheme, an old read view or active transaction can prevent a log checkpoint from being made. (Alternatively, we would have to append old undo log information to the redo log and patch all the DB_ROLL_PTR that are pointing to them.)

If transactions are truly independent due to not sharing any undo log pages, then I think that the partitioned log should work.

The current design idea is as follows:

Checkpoint information & file operations

There will be a separate file that contains information about log checkpoints and data file names. This file can contain information about multiple checkpoints. (The old ib_logfile0 only has room for 2 checkpoints.)

The checkpoint log file allows to construct the mapping between numeric tablespace identifiers and file names.
The checkpoint log file is never encrypted. This allows mariabackup --backup to work without having access to encryption keys. Because file names are not encrypted in the file system either, and because LSNs appear unencrypted in the diagnostic output, encrypting this file does not offer any security benefits.

A checkpoint log record comprises the checkpoint LSN and a byte offset in the circular log file, pointing to the log right after the LSN. It will also include the value of the sequence_bit that is described below.

The circular log

If any parameters of the log change, the redo log will be rebuilt:

• innodb_log_file_size
• innodb_encrypt_log, or the encryption key (key rotation will require the log to be rebuilt)
• innodb_log_checksums (if we choose to revive this deprecated parameter and implement a variant that lacks checksums)

There will be no fixed block structure in the circular log file. The LSN will count InnoDB mini-transactions, not bytes. This allows some flexibility: For example, a future version of mariabackup --backup --incremental could inject records to the backup of the main log file, instead of writing separate .delta files.

The circular log file will consist of length-tagged sequences of bytes:

byte *append_log(byte *log, const void *payload, size_t size, bool skip_bit, bool sequence_bit)

{

  size_t length= size;

  if (!skip_bit && innodb_log_checksums)

    length+= 4; /* CRC-32C at the end of the payload */

  byte * const start= log;

  log= mlog_encode_varint(log, length << 2 | skip_bit << 1 | sequence_bit);

  if (!skip_bit)

    memcpy(log, payload, size);

  log+= size;

  if (!skip_bit && innodb_log_checksums)

  {

    /* Always compute the checksum without the sequence_bit. */

    log[-size - 1]&= 0xfe;

    mach_write_to_4(log, ut_crc32(start, log - start));

    log[-size - 1]|= sequence_bit;

    log+= 4;

  }

  return log;

}

Explanation:

• If encryption is enabled, the payload will have been encrypted before the log is written. The length and the checksum will not be encrypted.
• The sequence_bit will be toggled whenever the write position jumps from the end of the circular log file to the beginning.
• The skip_bit allows us to write a partially filled log block of any size. If we need to persist the log (due to user transaction commit) and we are L bytes into a N-byte block (this depends on the underlying storage!), we can write a special record to say ‘skip the next N-L bytes’. There is no need to initialize (memset()) any skipped garbage bytes.
• We assume that the log ends when we get a CRC-32C mismatch or the sequence_bit of the next record differs from what we expect. (The last log record could end exactly at a byte offset where a log record before the last wrap-around had been stored, and that record would have a valid checksum.)
• Note: due to the skip_bit and the lack of memset(), it may be necessary to always store checksums, to reliably detect the end of the log.

@markoMarko Mäkelä

The design right now is that the partition is still based on space_id:page_no, and then if the mtr modify multi page, then we need flush all the redo log that id modified.. Is that right? Please correct me if I misunderstanding something..
And even though this solution looks like too rought, I think it is a practical solution, since most of the mtr only modify single page, most of the mtr modify multi page is the smo operation and undo operation. And we can write the undo log in the redo log directly, then the undo log will stay the same file with the redo log. And the smo operation is useless..

However, how can we keep the operation of flushing multi file atomic?

> However, how can we keep the operation of flushing multi file atomic?

I don't think we have such a problem at all. Only one file with redo data will exists. Writing to it looks like this: thread which owns mtr_t prepares buffer to write to a file (prepends it's size, computes crc32 and appends it to the end), then takes log mutex to write() buffer, releases mutex and performs fsync(). And that's it. Then, after some LSN is flushed to redo we can write(O_DIRECT|O_APPEND) corresponding checkpoint to a another file. Writes to log file and checkpoint files are not required to be atomical, thus, it's safe to crash right after flushing redo log, and before writing a checkpoint.

No, If the mtr is a smo operation, then this mtr will include multi page operation..
Then when mtr commit, it need to change more than one data page, then these two page may exist two different redo files as your partitioned by space_id:page_no. When we commit the trx, then we need to promise that the write to two redo logs is atomic. Since in the origin version, we only need to write to one file. It is easy to guarantee that the write is atomic. We don't have this problem.

Sorry, I think you have some misunderstanding. There are no several redo log files partitioned by space_it::page_no. Before 10.5 it was possible to have several redo log files, but they're used as one logical circular file. In current 10.5 it's already impossible to have several log files. And current design for new file format still assumes just one circular redo log file.

Sorry, I saw the design document
"The idea: Partition the log into append-only, truncate-the-start files"

I suppose we are talking about partition the redo log into multi redo log file..

baotiao this initial design was changed. You can find the recent version in comments for this issue. We decided to the the simplest possible thing: metadata file with creator version and such info is a separate file, checkpoints + file operations (create, delete, rename) is a separate append-only file, and actual redo log data is a third separate circular file.

@Eugene Kosov Ok, I got it..

let me summary changes are:
1. separate the chekpoint information from ib_logfile
2. add the file operation in the checkpoint file
3. the undo data in the redo log as comment by marko
4. lsn count the number of mtr, not byte

is it right?

However, I really interested in seperate the data into multi redo log files, since in the Architeture like aurora and PolarDB, there will exist about 100T large btree, if we don't partition it, there will only one redo log file, we can't make use of the under layer storage of fs..

baotiao, I have been thinking of the following format:

• Maintain a 512-byte header in ib_logfile0. Add information needed by innodb_encrypt_log there.
• After the first 512 bytes, write an append-only log consisting only of file name records (similar to the MDEV-12353 FILE_ records) and fixed-length checkpoint records. Each record will be followed by a CRC-32C checksum. The ib_logfile0 will never be encrypted. (File names and LSNs appear unencrypted in the file system and in the logs anyway.)
• We write a separate circular file ib_logdata that may be encrypted. The checkpoint records point to a byte offset within this file. This file can support any underlying physical sector size.
• LSN will be in bytes, just like before. But, encryption might no longer use LSN as part of the initialization vector, so that we can encrypt mtr_t::m_log before acquiring any mutex.
• Rebuilding the redo log file will not affect the LSN.
• In the future, mariabackup --backup --incremental could get rid of .delta files and instead write the information to the ib_logfile0 file. Any amount of data can be written to that file without affecting the LSN.

The circular log file could technically be split to multiple files, but we did not see a need for that. I think 128 TiB should suffice for quite some time in the future. The log checkpoint record would be 1+8+6+4=19 bytes.

At this point, we will not write undo log data to the redo log. I do not know if we will ever do that. I only mentioned the idea and challenges around it.

I did some research on writing to a file from different threads. My testing code is in https://github.com/kevgs/redo/
Here are the results for single thread for commit 0400fe0cc3829177c05341413e555c1f09f81b54 on my weak laptop with HDD:

File size: 134217728, threads: 1, duration: 20s

Circular file:

RedoSyncTLSBuffer handled 841 commits

RedoSync handled 368 commits

RedoSyncBuffer handled 801 commits

RedoODirectSparse handled 390 commits

RedoODirectBuffer handled 513 commits

RedoODirectTwoBuffers handled 747 commits

RedoOverlappedFsync handled 670 commits

RedoOverlappedMsync handled 3235 commits

RedoGroupCommit handled 744 commits

Append-only file:

RedoSyncTLSBuffer handled 758 commits

RedoSync handled 361 commits

RedoSyncBuffer handled 823 commits

RedoODirectSparse handled 365 commits

RedoODirectBuffer handled 756 commits

RedoODirectTwoBuffers handled 753 commits

RedoOverlappedFsync handled 786 commits

RedoGroupCommit handled 766 commits

And for 64 threads:

File size: 134217728, threads: 64, duration: 20s

Circular file:

RedoSyncTLSBuffer handled 825 commits

RedoSync handled 434 commits

RedoSyncBuffer handled 823 commits

RedoODirectSparse handled 487 commits

RedoODirectBuffer handled 851 commits

RedoODirectTwoBuffers handled 863 commits

RedoOverlappedFsync handled 11242 commits

RedoOverlappedMsync handled 58714 commits

RedoGroupCommit handled 1035 commits

Append-only file:

RedoSyncTLSBuffer handled 879 commits

RedoSync handled 514 commits

RedoSyncBuffer handled 855 commits

RedoODirectSparse handled 477 commits

RedoODirectBuffer handled 830 commits

RedoODirectTwoBuffers handled 896 commits

RedoOverlappedFsync handled 11318 commits

RedoGroupCommit handled 1361 commits

In multithreaded environment append-only file has the same performance as circular one.