Learning from Experience: Deprecating an API at The New York Times
Smaller API providers may look at the approaches of API giants when looking for models of what to do (or not to do) when seeking to inform their own developer community about changes to their APIs.
Scott Feinberg, API Architect at the New York Times, oversees a diverse collection of APIs (such as the NYTimes Newswire API), which have successfully grown into a suite of API products and has led to the NYTimes APIs being some of the most popular on API automation aggregator service, IFTTT. The NYTimes API developer portal sees an average of 25,000 monthly visits (according to data from Web site statistics site SimilarWeb).
“So we recently deprecated our Districts API. Since we launched it four years ago, Google launched a much better one with US-wide coverage that we now use internally,” explained Feinberg. The NYTimes Districts API provided a way to determine voting districts based on a latitude and longitude and showed information about the elected officials in that district, originally built to power certain NYTimes interactive voting tools. The API was in use by various campaigning organizations prior to Google's launch of their Civic API.
Feinberg said the API was not widely used, unlike other NYTimes APIs, but developers needed to be informed through community-wide mechanisms as the The Times was unaware who the specific users were:
We know who our large users are, but some of our smaller users who are making a few calls a day, we don’t much about what they do or how they’re using our API. We try to strike a balance between giving people the ability to use our APIs, explore our data, and build interesting things with an understanding of how our data is being used outside of our platform. When traffic starts to increase from an API consumer, we reach out and learn more about how they’re using our API.
Nowadays, Feinberg says, to improve developer communications around issues like deprecation, more formal registration is usually arranged:
To start using an API, you need to provide a URL, an App Name, an email, and a short description. In the future, we’ll likely decrease our API limits for new API consumers to give our consumers more of an opportunity to share what they’re building with us (as they will need to reach out to us for API limit increases). We like to promote our consumers, including small apps that are doing interesting things (like public libraries, visualizations, and academics). Active API users building cool stuff that reach out to us about our APIs are often invited to stop by the office and share their experience and meet the teams that built those APIs, it’s something we love to do.
Trends in Deprecating APIs
The APIs being deprecated this year show some key themes emerging in how API providers are making decisions to sunset their APIs and how they are communicating those changes. The above deprecated APIs demonstrate four key reasons API providers retire their APIs:
1. Changes to Platform Business Model
The deprecation strategy of both LinkedIn and Instagram point to how the providers are redefining their platform business model. Both list a clear set of minimal use cases for which building new businesses on their APIs is permitted. The approach is reminiscent of the ‘quadrant grid’ first presented by Twitter when it was realigning access to its APIs and aims to clarify the sorts of platform partners that it wants to work with.
Other API providers often do this the other way around: they start with a preferred partner program that asks developer teams from third parties to submit an application detailing their use case and their business model prior to entering into an agreement. Instagram, LinkedIn and Twitter went instead for a strategy that first sought to leverage growth by making a public API accessible, and then reined it in once the API had helped the provider reach new markets.
Some see this as a short-sighted or limited view of what a platform model can be. HackerNews user waterlesscloud commented “Twitter has insight into what everyone on the planet is thinking RIGHT NOW, and they're using it to sell ads for t-shirts. That's just a fundamental lack of imagination. There's all kinds of things that can be built on that, if you can rely on the platform to remain available to you.”
Feinberg from The Times is also cautionary around limiting opportunity of leveraging a platform play: “By limiting your platform, you’re limiting yourself to the good ideas that people sitting inside your office have. We have a lot of really smart people here, but there are a lot of good ideas out there and I want to see what ideas the people who actually use our product come up with.”
2. Lack of use
Low usage levels of an API are the ultimate in user experience feedback: if API usage is not high, perhaps there is not a need for the API. that’s what Feinberg found with the NYTimes Districts API, and is why Dropbox dropped its Datastore API and why Instagram didn’t see any potential in continuing their Feeds API. Maintaining an unused API is still a resource constraint for API providers who must prioritize what to focus on.
3. Closing Backdoors
Two undocumented but widely used “public” APIs were closed this year. Twitter’s share count and Google’s Autocomplete features were, both say, not intended for developer consumer use but once opened were allowed to exist.. until this year. Cynical developers see Twitter’s move as an attempt to monetize access to this data through their Gnip offerings, while Google says that there are alternatives for users looking for a private Web site search engine, and that autocomplete was always intended as part of the overall search experience.
4. Removing Security Risks
Google’s Earth and Asana’s API keys were both closed this year to avoid security risks with now-outdated technologies.
In most cases, APIs were deprecated as part of a versioning strategy that sought to move developers wholesale to a more updated version of the API. Yelp, Facebook, and eBay all stressed the availability of migration pathways to help existing developer consumers make the switch. Yelp and Facebook also sweetened the deal by providing new tools that allowed developers to see how the new API would work in a developer’s app before they made the switch.
Best practices for API Providers in Deprecating APIs
Pedro recommends five best practice standards for API providers who are looking to deprecate their APIs:
1. Announce the deprecation via email, blog post, via 'add a notice' on docs, and in a header to API responses:
Pedro points to a best practice blog post by Alex McCaw of Clearbit API, which spells out a three pronged communication strategy to alerting users to API deprecation and versioning changes. Initially, Clearbit alerts users via email, but recognizes that developers may auto archive or not read an email delivery of this nature. On their API docs, Clearbit then adds a bright blue colorful banner to alert users they are using an older version of the API and pointing them to migration resources. Finally, Clearbit adds an X-API-Warn header to API response requests for the API version being deprecated so that this appears in server logs.
“In addition, writing a blog post on your engineering blog is another good communication strategy. If you are using an X-API-Warn header like Clearbit suggests, it allows developer to automate their server logs so they are alerted when a header like this is received,” suggests Pedro.
2. Always offer a migration path:
Pedro points out the best practice examples of eBay and Yelp in helping developers understand what has changed when moving to a new version or alternate API.
“Migration pathways should list all resources of an API in the version being deprecated and show how that can be accessed in the new version or alternative API. Sometimes it may be necessary to say an alternative for this functionality is not available, but that is better than leaving developers scratching their heads around what how the changes impact on their own use case,” says Pedro.
3. Allow a long lead time for change:
Both Feinberg and Pedro are in agreement that API deprecation requires a long lead time: “I’d leave as much lead time as possible for people, both when versioning and deprecating,” says Feinberg. Pedro agrees: “A year should be the standard for deprecating and like Yelp did, when that year is up, it is nice to then offer another three months before completely shutting it down. Keep up the flow of communication via header responses, details on your developer portal and tweets back to your blog post announcement,” Pedro recommends.
4. Explain what the platform use cases are and offer a certified partner program:
5. Have a deprecation policy:
Pedro says all of these strategies can be explained in the terms of service. “Include a statement around how an API may be deprecated in the future, how that will be communicated to developer consumers, and what timelines will be given,” suggests Pedro. Best practice leaders will want to offer tools or tutorials on how to integrate the deprecation policy into DevOps: “A tutorial or doc page on how to set up an automated logging alert to track your response headers for API-X-Warn notices would be an example of going that extra mile for your developer community,” suggests Pedro.
Key Lessons for API Consumers
Pedro also makes several suggestions for API developer consumers to better manage the risk that comes with using an external API in business processes or to enhance the features of an end-user product. Of course, following the APIs on ProgrammableWeb and subscribing to alerts in APIChangelog can increase the likelihood of coming across the news as it is happening, meaning less likelihood that a developer consumer will become aware of the broken API only when one of their end users has an issue.
“APIs are a supply material that adds value to your business. Develop a risk management strategy for your APIs, like you would with any supply good that is fundamental to maintaining your business strategy,” Pedro recommends to API developer consumers. “Start with a list of each API you use, track what resources from each API you are using and look around at what the alternative APIs offer a similar type of resource capability. Like with any of your suppliers, have an annual review of the value you are getting from the APIs you consume and compare how reliable your APIs are with the alternatives you have identified.”
The growth of API deprecation is a sign of the growth and maturity of the API economy, and it is to be expected. APIs are incredibly powerful tools, but are also brittle to change and once in place, are difficult to alter too much. But as new features become available and tech stacks change, versioning becomes essential alongside decisions to pivot or refocus the role of APIs in a business strategy. Developer consumers do acknowledge that, as long as they are kept informed and are recognized as legitimate businesses who need long-term confidence in the API providers ability to continue providing their services. This year has seen a growing sophistication in how API providers are handling the deprecation process.
Thanks to James Higginbotham, Kin Lane, John Sheehan, Tyler Singletary and Sheera Gendzel for background material for this article.