I use Playwright to intercept all requests and responses and have Claude Code navigate to a website like YouTube and click and interact with all the elements and inputs while recording all the requests and responses associated with each interaction. Then it creates a detailed strongly typed API to interact with any website using the underlying API.
Yes, I know it likely breaks everybody's terms of service but at the same time I'm not loading gigabytes of ads, images, markup, to accomplish things.
If anyone is interested I can take some time and publish it this week.
I also do this. My primary use case is for reproducing page layout and styling at any given tree in the dom. So, capturing various states of a component etc.
I also use it to automatically retrieve page responsiveness behavior in complex web apps. It uses playwright to adjust the width and monitor entire trees for exact changes which it writes structured data that includes the complete cascade of styles relevant with screenshots to support the snapshots.
There are tools you can buy that let you do this kind of inspection manually, but they are designed for humans. So, lots of clickety-clackety and human speed results.
---
My first reaction to seeing this FP was why are people still releasing MCPs? So far I've managed to completely avoid that hype loop and went straight to building custom CLIs even before skills were a thing.
I think people are still not realizing the power and efficiency of direct access to things you want and skills to guide the AI in using the access effectively.
Maybe I'm missing something in this particular use case?
> There are tools you can buy that let you do this kind of inspection manually, but they are designed for humans.
You should try my SnipCSS Claude Code plugin. It still uses MCP as skill (haven't converted to CLI yet), but it does exactly what you want for reproducing designs in Tailwind/CSS at AI speeds.
> My first reaction to seeing this FP was why are people still releasing MCPs?
MCPs are more difficult to use. You need to use an agent to use the tools, can't do it manually easily. I wonder if some people see that friction as a feature.
Yes please, maybe there will be some solution that will fit the problem better! I recently released something similar, and because of the small API, I'm more comfortable using it.
>I'm in so deep that Claude Code can predict the stock market.
“What?”, more polite than “yeah right” :)
(oh I guess obviously it would have a chance at nailing it for weeks in a row, and have more good years than bad—since actively managed funds can pull that off until, universally, they can’t [beat the market])
I'm curious, have you developed your own reasoning system for how Claude can predict the stock market? Or have you trained it on past data combined with news sources?
Yes, it's pretty good! I've also written API harnesses for bot-based browser automation so that you can detect fields to fill in, remember where they are for next time you need them, and then if the webpage changes, re-explore and rewrite the tags to remember for the new form fields.
Spoiler: this is to automate ticket submission to my landlord's half-baked web portal, not some kind of nefarious captcha breaking thing.
yt-dlp is relatively stable, but still occasionally breaks for long periods. I get the sense YouTube is becoming increasingly adversarial to yt-dlp as well.
I don't know the details, but it doesn't seem like yt-dlp is running the entire YouTube JS+DOM environment. Something like a real headless browser seems like it would break less often, but be much heavier weight. And Youtube might have all sorts of other mitigations against this approach.
> yt-dlp is running the entire YouTube JS+DOM environment
IIRC they maintain a minimal execution environment that is able to run just the JS needed to pass a few checks but this breaks too often enough that they're planning to make Node.js or another JS interpreter a hard requirement (possibly already happened).
It would probably help people who want to go to a concert and have a chance to beat the scalpers cornering the market on an event in 30 seconds hitting the marketplace services with 20,000 requests.
I can try to see if can bypass yt-dlp. But that is always a cat and mouse game.
To clarify - yt-dlp is a command line tool for downloading youtube videos, but it's in a constant arms race with the youtube website because they are constantly changing things in a way that blocks yt-dlp.
Exactly, it is an agent skill that interacts pressing buttons and stuff with a webpage capturing and documenting all the API requests the page makes using Playwright's request / response interception methods. It creates and strongly typed well documented API at the end.
Sounds awesome. I've been using mitmproxy's --mode local to intercept with a separate skill to read flow files dumped from it, but interactive is even better.
It turns any authenticated browser session into a fully typed REST API proxy — exposing discovered endpoints as local Hono routes that relay requests through the browser, so cookies and auth are automatic.
The point is that it creates an API proxy in code that a Typescript server calls directly. The AI runs for about 10 minutes with codegen. The rest of the time it is just API calls to a service. Remove the endpoint for "Delete Account" and that API endpoint never gets called.
This 100% breaks everyone's terms of service. I would not recommend nor encourage using.
100% I'll response to this by Friday with link to Github.
I use Patchright + Ghostery and I have a cleaver tool that uses web sockets to pass 1 second interval screenshots to the a dashboard and pointer / keyboard events to the server which allow interacting with websites so that a user can create authentication that is stored in the chrome user profile with all the cookies, history, local storage, ect.. in the cloud on a server.
Can you list some websites that don't require subscription that you would like to me to test against? I used this for Robinhood and I think Linked in would be a good example for people to use.
I love how HN is loving this idea when it's the exact same thing Anthropic and OpenAi (and every other llm maker) did.
It's God's gift to them when it lets them bypass ads and dl copyrighted material. But it's Satan's curse on humanity when the Zuck does it to train his llm and dl copyrighted material.
So you’re that Hal Jordan then? Why would a Green Lantern feel the need to defend either? I feel like the Guardians would not accept your arguments as soon as you got to Oa, poozer. I guess what I am saying is don’t have a famous name. Seems obvious.
You conflate web crawling for inference with web crawling for training.
Web crawling for training is when you ingest content on a mass scale, usually indiscriminately, usually with a dumb crawler for scale's sake, for the purposes of training an LLM. You don't really care whether one particular website is in the dataset (unless it's the size of Reddit), you just want a large, diverse, high-quality data mix.
Web crawling for inference is when a user asks a targeted question, you do a web search, and fetch exactly those resources that are likely to be relevant to that search. Nothing ends up in the training data, it's just context enrichment.
People have a much larger issue with crawling for training than for inference (though I personally think both are equally ok).
Yes, I know it likely breaks everybody's terms of service but at the same time I'm not loading gigabytes of ads, images, markup, to accomplish things.
If anyone is interested I can take some time and publish it this week.