Hi Andrew, there's just one writer (which can tunnel multiple TCP connections).
Arbitration was indeed one of the trickiest bits. Originally I pre-reallocated the full file size (10 MB). Then used an integer at the beginning of the file to signal to the other side that a block was ready. The other side repeatedly read that int, and read the corresponding part of the file. But writing twice (once for the data, once for the int) had a significant performance impact.
In the end, what worked best was not pre-allocating the file. Rather letting the file grow whenever the writer writes to it. The reader knows when data is available by doing a PeekChar() in a tight loop. It's surprisingly fast, and accurately reflects the state of the file.
Thanks J :) Absolutely! I did try various flags to optimise performance. For CreateFile (which in .NET is wrapped by the FileStream class) I tried FILE_WRITE_THROUGH (which is FileOptions.WriteThrough in .NET), and found it impacted performance quite a lot.
The key to high performance as you rightly pointed out was preventing flushing. In the end, what worked best was reducing the number of writes to disk (which is the bottleneck). I did that by buffering 10-50 ms worth of TCP data, coupled with only flushing explicitly (using a large buffer so that neither BinaryWriter or FileStream flush automatically).
https://github.com/fiddyschmitt/SimpleKVM