I'm glad that all of the sites you target want your scraper to access them. The ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		mdaniel on Aug 5, 2014 \| parent \| context \| favorite \| on: Python web scraping resources I'm glad that all of the sites you target want your scraper to access them. The goal in many cases where one would use a scraper is to access information not provided in an API or otherwise encased in HTML. Most of their robots.txt are "User-Agent: *\nDisallow: /\n"

preinheimer on Aug 11, 2014 [–]

Then we have no right to scrape that content.

Why is there an implied right to scrape?

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact