I know what I am going to say is borderline hubris in this web 2.0 world we live in, but bear with me.
A lot of problems this site faces are due to the large influx of new, anonymous users. So I was wondering, how would this website be different if there was a small (say $5/yr) subscription needed, at least for posting. For one thing, it would have less (active) users, but we know that the sweet spot for social news sites is when they have a relatively small amount of users. Also, these users would be easier to identify and perminently ban if necessary. Finally, these users would most likely be those who are more positive in their interaction with the community. I find it hard to imagine a troll paying.
There are of course negative side-effects, not least of which is the locking out (from posting) of hackers who may not have a paypal account/credit card. Also, some may feel a sense of entitlement, an expectation of service-level. On the plus side, if there is anything left after the transaction processing, it may go towards covering server costs and whatnot. Even if pg/YC was to make some money off of this, I doubt us raging capitalists would mind, given the value we get out of HN. It might also go some way towards proving (if successful of course) that users will pay for quality and that ad-supported/freeminum is not the only way.
So, do you think such an alternative-reality HN would be desirable?
That doesn't mean it's a good idea, though. I think it almost certainly isn't. It sends entirely the wrong message, it's a much bigger pain to implement than it's worth, it might work all too well at cutting the volume of submissions and comments (if you think a crowded HN is bad, fork HN -- the code is open source! -- put it up on your server, and see how exciting that is)... and it won't work. You really think a troll won't pay five bucks a year? I think you severely underestimate the entertainment value that a troll derives from trolling.
If, in fact, HN eventually declines to the Reddit level, the solution will be the same as ever: Some of us will head off into small invite-only groups (note that invite-only is distinct from for pay), and the rest of us will migrate to a handful of other new social news sites which will thrive for a while until they succumb to the same problem, or to an entirely new problem, after which the cycle begins anew. As someone has said in the past, it's like restaurants, or music clubs. They flare up, grow, shrink, and die out all the time.