Hacker News new | past | comments | ask | show | jobs | submit login

How do you deal with the legality of scraping?

I was hired to scrape some economic related pages and build an excel file and email it, I got that done but not sure if I should try to host this and turn it into a service or just set it up for the client and let them deal with it. It's just personal use on their part.




I would not equate scraping with periodic searching for a keyword.

The problematic part is when you scrape data off of websites, and the owners don't want you to do it; as in, they would not even be happy if you manually copied that stuff into an excel file for fun or profit.


I can see that, although automating to decrease views of a page(from checking by refresh)... I suppose the end result is sign up in this case.

So for me I'll set it up for them and them run it.


> How do you deal with the legality of scraping?

How did Google deal with it, when they started their search engine business?


My counter to this is, without Google's "algorithm" the search results would not be great/find what you want. And in this case websites want to volunteer/agree to index their site for visibility but then you still have to visit the site because conveniently The site summary ends before the content that you want. I don't know how I would find websites unless referred on some website.

I kinda felt the same way about how Netflix started out but I think they probably had a deal to pay a portion of revenue from dvd's that they rented out.


robots.txt





Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: