Oh, About that API that Yahoo! Killed Without Warning Developers

Across the API economy, deactivating an API without first warning the developers who are using it is regarded as one of the best ways to destroy your reputation as a reliable API provider. So, when reports first surfaced on Hacker News that Yahoo obliterated one of its stock data APIs without warning, you can imagine how surprised we were. After all Yahoo knows the API ropes pretty well. But, upon further inspection, it not only appears as though there may have been a rush to judgement, there’s also a solid lesson that developers can learn when it comes to the perils of working with free and open APIs. 

Although it technically isn’t an official REST API, Yahoo Finance appears to have killed a more than decade-old service that offered public stock data as a batch-downloadable CSV file. Across Hacker News and Yahoo’s own forums, much of that disappointment is associated with the company's failure to issue a warning, a mistake that’s more commonly associated with rookie API providers.

However, looking across the various posts, it appears as though the “service” may not have been intended for developer consumption in the first place. Rather, it appears as though developers originally discovered the capability through an export-to-CSV feature that Yahoo Finance was offering to regular users of its Web site.  

The incident once again draws attention to the best practices of both Web site operators who offer data in machine readable form without requiring authentication, as well as developers that accept the risk of working with that data.

For example, as shown below, if the page is used to view summary data for the IBM and Amazon stocks, the same page displays an export button to retrieve the data as a CSV file (often done for import into a spreadsheet). 


Developers used the Export to CSV functionality on this Yahoo Finance Web Page as an API

Judging by the dearth of developer documentation from Yahoo on how to consume the “API,” and the abundance of third-party documentation (searchable via Google) on how to take advantage of it, it’s relatively clear that Yahoo wasn’t publishing the service with developers in mind. In fact, the error message (shown below) from Yahoo supports the theory that pretty much every developer consuming the CSV via machine was doing so in violation of the company’s terms of service and that the company was willing to take the entire service down to thwart continued consumption.
 

However, whether the company knows it or not, deactivation of the CSV download capability also resulted in deactivation of the Export button on its Web page; perhaps an unintended consequence. Meanwhile, the developers wondering why there was no warning about the service’s discontinuation will have to consider how they expected to receive that warning if Yahoo didn’t know who they were since the majority of them were probably accessing the CSV capability anonymously (in other words, they weren’t logged-in to Yahoo!). 

Lessons Learned? 

There’s no telling what Yahoo’s motivations for the shut-down were. Maybe the company discovered overly excessive utilization of its servers as a result of the number of automated downloads. But given the popularity of the service with developers (something it could have discovered through its analytics), maybe it also should have looked for a mutually beneficial way to continue the service.  In terms of API economy best practices, when it comes to retiring services (be they APIs or not), deactivating a official service that developers depend on is widely regarded as a big fat no-no. As far as deactivating an unofficial API, it’s hard to frown on such a practice. 

As can be told from the various forum posts, there is perhaps no better way to establish yourself as an unreliable API provider than to terminate a service, official or not, without warning. Developers are a finicky (and communicative) bunch and once word gets out, they might easily be frightened away from other APIs and services offered by the same API provider. 

Ultimately however, when you think about it, what choice did Yahoo really have? The CSV download was openly available without a login. Developers should know that for open APIs and services that don’t require registration or an API key of any sort, there is always the risk of the service disappearing without notice because the API provider has no way of knowing who is using the service or how to contact them.

David Berlind is the editor-in-chief of ProgrammableWeb.com. You can reach him at david.berlind@programmableweb.com. Connect to David on Twitter at @dberlind or on LinkedIn, put him in a Google+ circle, or friend him on Facebook.
 

Comments

Comments(4)

Stephane-Dubois

This API has been around for way more than 10 years. I used it already at least 15 years ago.

The probable reason for termination and the reason for the quietness about it is that Yahoo, was under pressure from their data vendors that this API was a huge "backdoor" for businesses to acquire the data illegally (i.e. for business use without paying for it). Rather than paying the huge fees vendors would have required to pay to keep the APIp around, they shut it down.

It is true that anyone using that API for business use was doing this in violation of the Yahoo terms of use. Individual investors could have used it without problem. The numbers of businesses using that API illegally was indeed staggering. 

david_berlind

Thanks for the comment Stephane. Yes. My sense from speaking to a few developers is that not only were there a lot of developers using the API, but that they were hitting it very frequently. After all, it was free with no limits. The tax on Yahoo's systems (now Verizon's) must have been significant. I mean, VERY significant. I'm suprised though. Given that degree of popularity, if it were my "API," I would have looked for a way to try to keep some of those developers. I have another article coming that discusses some options that the entire API economy might consider for in-band developer notification. Thanks again for leaving the comment.

Andrew-Evans

Whats worse than a csv export? A couple million scrapers behind proxies and TOR.

david_berlind

So true Andrew.  What do you think the answer is? How do the scrapers behind proxies and TOR work? For example, does every "scrape" come from a different IP?  Thanks for commenting. 

David