>>If I’m using a small website, there’s a chance that my little Python script could take down their service because it’s doing way more requests than they’re able to handle. So when I’m doing this I try to be respectful and not make too many requests too quickly.
What is a reasonable rate to send requests? I've done a little scraping and I wanted to do the same thing but I realized I had no idea what would be considered acceptable use and what would be unacceptable. If anyone has a heuristic they like to use I'm all ears.
If you're going to be doing it manually regardless (if not automated), then as far as I'm concerned, you could definitely just use a "normal clicking speed" rate - so a second or two between clicks is probably just fine and non-parallel requests. Usually if it's likely to overload the server, it's probably slower to return, too, so the server itself will slow the requests down naturally if you're not using parallel requests.
Usually you're automating these things not to get the job done that much faster, but instead just to do it without all the tedium, so a slow but asynchronous scrape is fine.
What is a reasonable rate to send requests? I've done a little scraping and I wanted to do the same thing but I realized I had no idea what would be considered acceptable use and what would be unacceptable. If anyone has a heuristic they like to use I'm all ears.