Reverse engineering APIs through network requests is one of the most fun things you can do with Claude Code to automate tasks..
SO many websites are impossible to navigate "deterministically" via the DOM (or through screenshots).
So, I just point Claude Code to use browser_harness by
@browser_use (or vanilla playwright) and ask it to sniff network requests on the pages that I'm trying to get info on.
And, I just keep clicking around on the sections of data that I want. And, then Claude Code is generally able to go through the logs to figure out what is the right structure for these APIs and what kind of auth do they need (most are cookie based). We also determine what kind of rate limits exist based on trial and error.
I'm able to use that to construct jobs that allow me to get that data programmatically. There are many use cases for this besides scraping. I use this for random side projects (like the travel CLI), for monitoring websites (for intel), and for many many other use cases.
Every website will soon need to be headless, and we'll need to figure out mechanisms for how we have our agents pay these websites programmatically as well.
Just as we have llms.txt for data and structure, we'll soon need tools.txt for agents to determine what tools exist that can be leveraged.