Gotcha - it's times like that that I go with an old-fashioned raw HTTP request and parse it with a DOM parsing engine. Sometimes the "web-facing API" (so-to-speak) is better than the official API.
Yes, an API that is isomorphic to the website itself would be much easier to use. Our plan is for the next version of HN's API to simply serve a JSON version of every page.
In the meantime, if you scrape the HTML, please respect https://news.ycombinator.com/robots.txt and don't do it more than once per 30 seconds. HN's app server runs on a single core, so we don't have a lot of performance to spare! This is in the works to change as well.
> HN's app server runs on a single core, so we don't have a lot of performance to spare!
That's interesting. Is there any more information about HN's tech stack? I know it's built on top of the Lisp dialect Arc, but I wonder about the server specs and would be happy to be pointed at some resource.
Although I experienced HN being down, I guess the overall uptime and availability is pretty decent. Keep up the good work!
Super cool project!