Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why can't crawling be crowd sourced? It would solve ip rotation and spread the load



Too bad it doesn't support android. It is much more energy efficient than anything else I can spare (for 100% uptime contribution)


That’s how residential proxies work, in a perverse way


Common crawl sort of serves this function. I use it. It's a really good foundation.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: