What does not being in C have anything to do with that? Whatever tricks Twisted and Node.js and friends use can be done in C, just with a different syntax, but at the end of the day it's the same machine code. I don't know how Twistd handles file I/O but Node.js uses libeio which performs all file I/O in a thread pool. It's no better than what you can do in C.
rbranson said: "but it doesn't seem as if it's really practically possible (in C at least) to make mmap() I/O work in a cooperative concurrency model."
In that context, it ought to make sense.
Further, there's a set of technologies that are good, but range from difficult to effectively impossible to use in C, and the list is growing, not shrinking. We can't afford to bind ourselves to what will work in C forever. For instance: Garbage collection, possible but hard and certainly a bit inelegant, you're fighting C. Software transactional memory: Don't even think about doing it in C. But it may still be a useful thing in some places. And so on, for a mix of things. C is what C is, but "it doesn't practically work in C" can't be allowed to be a fatal objection if we want to progress.
Sure, you can type anything you want in C. But you won't. And I know it. Don't even try to argue that you will, it's obvious that you won't. Given the current state of the programming language environment, it is an impossibly small window to sail through to claim that Java, Erlang, Haskell, Javascript, C#, C++, SQL, etc, all have no advantages over C because we can always do the same thing in C (and I deliberately picked a wide range), yet C has some sort of advantage over assembler that makes it worth using. I reject that C is a Platonic default language that all others must justify their existence against; it's merely one that got some popular OSes written in it for certain good reasons, but that doesn't gold-plate it against criticism. This argument hasn't been sensible for about 30 years now, and now it's just gibberish; we know better. Cost factors matter a lot.
Definitely agree with you that abstraction matters. I use C as an example because it's the canonical way in which we as developers can reasonably converse about directly dealing with the machine. C is still important and will continue to be important because it's arguably the best abstraction we have to write code directly against the machine. GC and STM would be orthogonal to the purpose of C.
It's really sort of pointless to argue what's possible in environments lacking real, unsafe pointers, because the whole idea with mmap()'d I/O is that it lets apps access files as if they are a region of memory. Java wraps up the behavior in FileChannel, and most other platforms have a way to access mmap() in some fashion, albeit the practical use cases are vanishingly small. In C, it's done mostly for squeezing out bits of extra performance by shrinking memory requirements/allocation cost and avoiding copies. Much of the safety mechanisms used by VM environments invalidate or make unavailable the shortcuts that are used in tandem with mapped I/O to achieve higher performance.
In the end, it's all still triggering the virtual memory interrupt that will load the page from disk if it's not in RAM, and the kernel is still going to suspend the calling thread while it's happening. No way around that. Alleviating this through thread pools would just re-introduce the issues of inter-thread locking and context switch overhead, and present an additional bottleneck; in the end just piling on extra complexity. I don't see that as a net win at all.
While the post has some great information on mapped I/O, it seems as if the author is dogmatic about the use of mapped I/O and is trying to find a way to use this hammer in a situation where it might be inappropriate.
EDIT: To the author who replied: awesome, awesome. Ultimately this is great stuff to talk about and these types of posts are what HN should really be about, even though it's turned into startup TMZ.
Broad agreement in general, but I would observe there are still use cases for non-bare-metal languages for mmap. It doesn't conflict with all VMs. Haskell could use it (it actually has surprising interfacing capabilities on this front), Google says OCaml can use mmap though AFAIK it has a less useful threading story, Lua can probably productively use it though again I'm unclear on the threading. Erlang can't use it out-of-the-box but conceptually it could be modelled as a port, though again whether you could get a performance win I don't know. Mono and the JVM could use it, though again, primitive threading story. Python and Perl have interfaces to them but you sacrifice so much performance simply by using them that yeah, it doesn't much matter. But at least in theory there are VMs that can productively use it.
Good point about C. You misunderstand the intent of my post though.
First, as a small point, I was not seriously suggesting using thread pools to implement mlock_eventfd. That's just a way that we could play around with it short of a kernel implementation, which would probably be more efficient.
Second, I don't think I'm dogmatic about mmap. It's just a cool interface that would be nice to see more widely used, and I was suggesting that it might be more usable with a small addition. Commenters here, at my blog and on Reddit have pointed out other problems, and I'm happy to learn about these. If mmap isn't good and we should keep using DIO, then I think there are still possible improvements that we could make to the interface to allow better cooperation with the kernel.
