Here's an actual example, instead of all this pontificating. It's not that impressive but it illustrates the point.
I had to write an external "chat history" module (i.e., always send the last N lines of a conversation to a client as soon as client joins) for an IRC server, in python. (it's a long story)
Your standard RDBMS solution would probably involve the ircd logging directly or indirectly to a database, with a python script handling requests and doing selects out of that db. On a network with thousands of users, this will load a db machine down with IO operations. This also has to happen quickly, so that new text is added to the history log in near real-time. Batching is probably out of the question.
My solution was to have the ircd pass off its strings to another program (in this case, I used python) via a named pipe (this could also have been done via a socket, to aggregate multiple servers or do other neat tricks), and then have the logger app load these into memory - in this case, a giant python dictionary keyed on channel name which pointed to a constantly-updated ring buffer class for each channel. This ring buffer held the last N lines of conversation.
Writes were simple - h[c].write('text')
Lookups were simple - h[c].get()
Fast writes, fast reads, and disk i/o was batched for efficient logging to disk, at which point you deal with the data in batches and maybe store it into a db if you want to do something more advanced like with SQL.
Of course this could all be done with C within the ircd itself as well, using a hash table and ring buffers as well.
It was actually much faster (and logically simpler) than writing a DB solution, and I'm certainly no master hacker.
I had to write an external "chat history" module (i.e., always send the last N lines of a conversation to a client as soon as client joins) for an IRC server, in python. (it's a long story)
Your standard RDBMS solution would probably involve the ircd logging directly or indirectly to a database, with a python script handling requests and doing selects out of that db. On a network with thousands of users, this will load a db machine down with IO operations. This also has to happen quickly, so that new text is added to the history log in near real-time. Batching is probably out of the question.
My solution was to have the ircd pass off its strings to another program (in this case, I used python) via a named pipe (this could also have been done via a socket, to aggregate multiple servers or do other neat tricks), and then have the logger app load these into memory - in this case, a giant python dictionary keyed on channel name which pointed to a constantly-updated ring buffer class for each channel. This ring buffer held the last N lines of conversation.
Writes were simple - h[c].write('text')
Lookups were simple - h[c].get()
Fast writes, fast reads, and disk i/o was batched for efficient logging to disk, at which point you deal with the data in batches and maybe store it into a db if you want to do something more advanced like with SQL.
Of course this could all be done with C within the ircd itself as well, using a hash table and ring buffers as well.
It was actually much faster (and logically simpler) than writing a DB solution, and I'm certainly no master hacker.