Programmer Builds Unofficial Hacker News API

Adam DuVander
Sep. 23 2010, 12:00AM EDT

The favorite news aggregation site of programmers now has an API. But it didn't come directly from YCombinator, the company that runs the site. Instead, it was built by a community member who published it while working on a side project. The data is scraped from Hacker News and essentially has the blessing to operate as an unofficial API.

Ronnie Roller built the Hacker News API while working on iHackerNews, his mobile version of the site for reading the latest links and discussion of interest to programmers. While his site does not run off the API, Roller noted that the two tools share a codebase.

The API provides access to tops news, as well as commenting and voting. It has write options, as well, which requires login credentials. OAuth is not an option, as it would have to be supported by the main site, which has no API of its own.

Roller announced the API on Hacker News and it was met with mixed comments. While most feedback was positive, he says, some felt he should not be scraping the site or exposing some features that are not easily accessible via the standard site. "There's a good collection of really smart people and industry leaders on the site," Roller said, and some are worried about the community becoming too big.

There's also the potential that Roller's API might put stress on the Hacker News server. So, Roller again took to Hacker News to determine an appropriate request rate. YCombinator founder Paul Graham replied with what amounts to an endorsement of scraping the site to feed the API.

Roller is a programmer at a financial company by day, where he doesn't get the opportunity to build these sorts of projects. In the evening he works on sites like iHackerNews and a sort of sister project, ViewText, that frees a page's content from its presentation. That one also has an API.

It's surprising that we haven't seen more of these scrape-to-API sorts of services for sites that don't already have an API of their own. Scraping is a controversial process, but developers often need access to the raw data to be able to combine it with other sources to create something new. Hopefully sites with unofficial APIs will learn from Graham's reaction and embrace the developer--and take the service as a request to launch an official API itself.

Adam DuVander Hi! I'm Developer Communications Director for SendGrid and former Executive Editor of ProgrammableWeb. I currently serve as a Contributing Editor. If you have API news, or are interested in writing for ProgrammableWeb, please contact editor@programmableweb.com Though I'm a fan of anything API-related, my particular interest is in mapping. I've published a how-to book, Map Scripting 101, to get anyone started making maps on websites. In a not-so-distant past life I wrote for Wired and Webmonkey.

Comments

Comments(4)

Andrea, thanks for your comments and for sharing your Ruby gem to access Hacker News. The reason that this was newsworthy is that it takes a popular site and makes it available via API to anyone, using any programming language. That's the sort of stuff we champion here.