Automated web testing tools, such as Watir and Selenium, are also pretty good options. I'm especially surprised Watir hasn't been mentioned yet in the comments.
Indeed - or WatiN, the .NET port of WatiR. I've done some pretty heavy duty scraping and automation with WatiN, which included building a OO framework that trivialized writing scripts. Good stuff.
