Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.

This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: