I believe we can get pretty far on both this ticket and MCOL-3459 by using atomic-appearing writes. By writing to a uniquely-named temp file in the same location (guaranteeing that it's on the same filesystem), then renaming it when done (guaranteed by posix to be 'atomic'), no readers or writers can interfere while the write is happening. This should let us greatly reduce the use of locks across SM.

Another source of lock reduction is having writers write to unique journal files instead of having them all append to the same journal file for an object. Then, the only point of contention is on the .meta file. For this mod it may be necessary to record the journal files for each object in the .meta file.

I haven't fully explored either one of these ideas wrt SM, they are just starting points.

The key will be remembering the guarantees of the posix file i/o interface. It lets readers and writers step all over each other, it really only guarantees that it's own metadata for the file and filesystem are consistent. That's the only thing that SM needs to guarantee as well. It doesn't need to prevent the callers from doing stupid things, that's their problem.