>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
Various ISPs give out either a /64, a /56 or a /48. Anything else is very unusual. A normal approach is to limit by /64 at first, limit a /56 to 3-4 times the rate limit of a /64 and a /48 to 3-4 times that again. Someone who has a /48 gets to enjoy 16 times the rate limit of someone who has a /64 but that's not too bad, and you don't have to tune anything per ISP.
Anyone who has bigger than /48 is no longer an individual user. They could be an ISP set up solely for scraping, though that comes with some fees and requirements.
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)