pg: I have fair bit of lisp dev experience. If, as a weekend project, I modified the HN src to use postgres and memcache would you consider using it in production? Obviously, I don't expect carte blanche prior agreement, but I wouldn't want to invest the time unless I thought it was plausible the work could actually help.
I would expect it to solve most of your performance problems for the foreseeable future (at the very least, by letting you scale horizontally and move the DB, frontends, and memcaches to separate boxes - plus ending memory leaks/etc by moving most of the data off the MzScheme heap).
The obvious downside is that it would use your (or someone at YC's) time. First to merge the changes I make to http://ycombinator.com/arc/arc3.tar into the production code, then to buy/setup some extra boxes and do the migration. We're probably talking, roughly, a day. It also has the unfortunate side effect of costing HN's src some of its pedagogical value, since it adds external dependencies and loses 'purity'.
Been looking for an excuse to learn arc for a while now ...
There's a great chapter in "Surely your joking Mr. Feynman" about mental arithmetic.
Here's an excerpt, from when Feynman was eating at a cafe and an abacus salesman walked in and challenged Feynman to an arithmetic contest:
"He writes down a number on some paper— any old number— and I still remember it: 1729.03. He starts working on it, mumbling and grumbling: "Mmmmmmagmmmmbrrr"— he's working like a demon! He's poring away, doing this cube root.
Meanwhile I'm just sitting there.
One of the waiters says, "What are you doing?".
I point to my head. "Thinking!" I say. I write down 12 on the paper. After a little while I've got 12.002.
The man with the abacus wipes the sweat off his forehead: "Twelve!" he says.
"Oh, no!" I say. "More digits! More digits!" I know that in taking a cube root by arithmetic, each new digit is even more work that the one before. It's a hard job.
He buries himself again, grunting "Rrrrgrrrrmmmmmm ...," while I add on two more digits. He finally lifts his head to say, "12.01!"
The waiter are all excited and happy. They tell the man, "Look! He does it only by thinking, and you need an abacus! He's got more digits!"
He was completely washed out, and left, humiliated. The waiters congratulated each other.
How did the customer beat the abacus?
The number was 1729.03. I happened to know that a cubic foot contains 1728 cubic inches, so the answer is a tiny bit more than 12. The excess, 1.03 is only one part in nearly 2000, and I had learned in calculus that for small fractions, the cube root's excess is one-third of the number's excess. So all I had to do is find the fraction 1/1728, and multiply by 4 (divide by 3 and multiply by 12). So I was able to pull out a whole lot of digits that way. "
My hope is that it will only take a day or so to deploy it, once it's ready.