ProgrammableWeb Updates API Directory to Better Support Multiple API Versions

This year (2019) marks a special year in the 15-year history of ProgrammableWeb, the Web's journal of the API economy. By the time you read this, ProgrammableWeb's API directory will have eclipsed over 23,000 unique API profiles. But, in all honesty, this number barely scratches the surface of the number of networkable APIs that have been published as a part of the API economy. Furthermore, ProgrammableWeb's own data model for recording and tracking API profiles is now, in the Fall 2019, about to go through some significant changes that are likely to impact the total number of APIs that we list.

That number will increase based on an immutable truth that we've learned over the years: the best example of something that uniquely identifies an API is its endpoint. This truth also represented a thorn of inelegance in the data model behind our API directory. For example, if an API supported multiple architectural styles (eg: REST and RPC), then there were invariably multiple endpoints associated with those styles. Our old data model permitted the assignment of multiple architectural styles to an API, but not multiple endpoints (one for each style).

As any good database designer knows, the last thing you want to do is modify your record structure with new fields like "Endpoint #2" and "Endpoint #3" to address a one-to-many problem. Sooner or later, we would have to bite the bullet, redesign our data tables to reflect these one-to-many relationships, and split our records to fill them. An API that supported two architectural styles would need to split into two records. Three records for three architectural styles, and so on. Arriving at a new data model was one part of the challenge. Migrating our existing data to that model was another part. And then came the adjustments to ProgrammableWeb's user interface; not just how we display this data, but also the forms used to to input and validate the data.

These changes to our data model have been in the works for over two years now. They're also the foundational element to two other strategic initiatives for ProgrammableWeb. The first of these is to launch a great, faceted, power search feature; one that allows you to slice and dice our directory in myriad ways to find exactly what you're looking for. We could have launched a power search feature on the old data model. But, we would have wasted precious engineering resources on something that we knew we'd have to replace anyway. Now, with our new data model in place, our power search won't be far behind. The second of these initiatives is to finally offer an API of our own. For years we've been asked why we, the journal of the API economy, do not offer an API (the proverbial API of APIs). Talk about ironies. To be honest, it was mainly a case of the cobbler's children having no shoes. And, like the faceted search, it would have been wasteful to devote engineering resources to something we'd have to throw away because of how tied to the data model an API would be. Soon after the new search feature, we'll begin work on the API.

Old Versus New

When ProgrammableWeb was first founded, its data model was predicated on a common (and highly developer-unfriendly) API versioning practice that existed throughout the API economy. Simply put, when a new version number of an API was introduced, the provider of that API would immediately retire the old version. The old API's endpoint — the Web-based URI to which applications would address their API calls — would literally disappear, often causing applications that depended on it to break.

This approach to versioning made things easy for us here at ProgrammableWeb. When there was a new version of an API, we'd update that API's existing entry in our directory with all the particulars of the new API, usually wiping out the old version's data.

Today, however, things are wildly different. As evidenced by the partial screenshot below of Facebook's Graph API and the schedule Facebook is maintaining for the different versions, most API providers know never to suddenly deactivate one version of an API just because a newer version has been put into production.

Partial screenshot of Facebook's Graph API and the schedule Facebook is maintaining for the different versions.

As a result, in an act of graceful and gradual deprecation, it's not uncommon for API providers to simultaneously offer multiple versions of the same API — each of which is addressable at a unique URI endpoint --- even though only one of them (usually the most recent) is the current recommended version. Additionally, there could even be an experimental beta version. After years of running the API directory and observing all sorts of versions in the wild, our new data model will reflect the typical lifecycle of a modern API by supporting multiple version numbers of the same API, each of which can be classified according to one of the following designations:

Pre-Release
This version is one that's currently active but that has not yet ascended to being a recommended production version. It could be a beta or alpha release and may not be available to all developers. Pre-release versions are typically non-supported versions (at least not supported in the same way that the active, recommended version is supported).

Recommended
A "Recommended" version is an active, supported, production version of the API for a given architectural style. It is the most current version that you, as an API provider, want developers to use in their applications. There can be multiple recommended versions when an API provider supports multiple architectural styles. For example, if the API is available as both a REST API and as an RPC-based API, there could be a recommended version of each. Each of those versions will appear as separate entries in our directory.

Active
An "Active" version is also a supported, production version of an API. It's just not the recommended version. Like Facebook above, some API providers are likely to have one or more active versions that overlap their recommended version. This happens when the API provider decides to release a new version but is not ready to fully deprecate an older version at the same time (this is usually to assist developers who are running behind in migrating to the recommended version).

Retired
A "Retired" version of an API is one where the API is still active (as with the previous state), but it is no longer supported through official channels. Although it increases their operational costs and complexity, API providers sometimes leave retired versions of APIs in an active state as a courtesy to a few remaining developers who've been unable to migrate their applications to the newer active versions of the API. However, those developers are not guaranteed of ongoing technical support from the API provider.

Deactivated
A "Deactivated" version is one whose endpoint is no longer accessible. ProgrammableWeb keeps a record of this version for historical and research purposes but will conceal the metadata at the API provider's request.

As can be seen from the partial screenshot below, we've added a new field to our Add API form that makes an allowance for this new "Version Status" field.


ProgrammableWeb's Add and Maintain API forms have a new "Version Status" field that helps identify an API version's current lifecycle state. API providers can set the status when an API is first entered into the ProgrammableWeb directory, and then, over time, change that status multiple times until the API is eventually deactivated.

We realize that this is an opinionated breakdown of an API's lifecycle. For example, we've purposely avoided the oft-used word "deprecated." Whereas we've seen some API providers refer to deprecated APIs as simply those APIs that they no longer favor (even though they may still be active), we've seen other API providers use the term to refer to versions of APIs that have been deactivated. In other words, their endpoints are dead. Across the API economy, there's no commonly accepted definition for "deprecate" or "deprecated."

Therefore, in our view, the act of deprecating an API is the act of advancing an API one or more steps towards its final resting place; a deactivated state. For example, when the current recommended version of an API is replaced by a new recommended version, the original recommended version has essentially been deprecated to its next phase of life (active and supported, but not recommended in our parlance) as it begins its journey towards end-of-life. The same is true when an active, supported version (but not necessarily a recommended version) of an API is deprecated to become either retired or deactivated.

As can be seen from the graphic below, whereas multiple versions of the same API can be designated according to the above structure in a way that developers can easily explore a particular API version's full history, separate versions can also be designated according to their architectural style. For example, Github offers two endpoints for the same API; one is REST-based and the other is GraphQL-based. With very few exceptions, we consider these two APIs as distinctly separate versions of the same API. And, as you might suspect, there can be multiple versions (1.0, 1.1, 2.0, etc) of each.

ProgrammableWeb's new data model makes allowances for multiple versions of the same API both in terms of architectural styles and version numbers within those architectural styles.

ProgrammableWeb's new data model makes allowances for multiple versions of the same API both in terms of architectural styles and version numbers within those architectural styles.

Realizing that this is a bit more complicated than our previous model, especially for those of you who want to maintain the various versions of your APIs in our directory, we've added a special Clone API feature to ProgrammableWeb that makes it dirt simple to add a new version of an existing API. The way the feature works is simple. After first starting the process for adding a new API (by clicking the blue "Add APIs & More" button found at the top of every page on ProgrammableWeb), you'll be asked in YES/NO fashion if the API already exists in our directory or not (as shown in the partial screenshot below). Select YES to start the cloning process.

The process for adding a new API to ProgrammableWeb's API directory has a new starting point at which the user can choose between cloning an existing API (for ease of input of the new API) and adding a brand new API for which there is no pre-existing version.

The process for adding a new API to ProgrammableWeb's API directory has a new starting point at which the user can choose between cloning an existing API (for ease of input of the new API) and adding a brand new API for which there is no pre-existing version.

Next, you'll have the opportunity to pick one of the existing versions of your API that's already listed in ProgrammableWeb's directory. Pick the version you wish to clone and click the Clone Version button as shown below in a partial screenshot of the mockup we used to prototype the feature.

Once you click the clone button, ProgrammableWeb will automatically create a new version of that API with all of the fields pre-populated from the version that you cloned. It's like a big fat copy and paste operation. The difference is that you're copying the entire contents of an existing database record into a new database record instead of copying and pasting unstructured text. From there, all you need to do is edit a few of the new version's fields (eg: API endpoint, Description File URL, etc.) that are specific to the new version and submit it for publication.

To add a new version of an API that already exists in ProgrammableWeb's API directory, simply pick the version that most closely resembles the new version, and click the ''Clone Version'' button. Behind the scenes, ProgrammableWeb will create a new version of the API that's nearly identical to the version you cloned and from there, you just have to edit a few fields before submitting the API for publication.

To add a new version of an API that already exists in ProgrammableWeb's API directory, simply pick the version that most closely resembles the new version, and click the "Clone Version" button. Behind the scenes, ProgrammableWeb will create a new version of the API that's nearly identical to the version you cloned and from there, you just have to edit a few fields before submitting the API for publication.

How GraphQL Informed Our Data Model

Speaking of GraphQL, GraphQL is one of those API technologies that challenged our thinking about the API directory. When the first public GraphQL APIs arrived on the scene, the ProgrammableWeb team wasn't sure how exactly to catalog them in the directory. Across the various fields that we use to classify an API, GraphQL was so new that it did not appear as a choice in the drop-down menus. We knew we had to add it as a choice, but to what menu? Was it a new alternative architectural style to the existing styles like REST, RPC, and streaming? Or, did it belong in one of the payload formatting menus?

Prior to the November 2019 update of ProgrammableWeb's data model, GraphQL was listed as one of the choices for an API's architectural style in ProgrammableWeb's New API Input Form. As a side note, if you are ever confused about what that form is asking for, just click the question mark to the right of the field for a detailed explanation.

Prior to the November 2019 update of ProgrammableWeb's data model, GraphQL was listed as one of the choices for an API's architectural style in ProgrammableWeb's New API Input Form. As a side note, if you are ever confused about what that form is asking for, just click the question mark to the right of the field for a detailed explanation.

As shown above, in the partial screenshot of ProgrammableWeb's New API Input form, we originally added GraphQL as a choice of architectural style. After debating the issue, however, we reconsidered and grudgingly decided that, for now, GraphQL would be classified under the Remote Procedure Call (RPC) architectural style of APIs. For more on the rationale behind this decision, please read GraphQL APIs are Most Definitely Not RESTful. So, What are They Then? We may eventually change our mind. But the good news is that it really doesn't impact the final outcome. In terms of facilitating great power searches (which was the main motivation for updating our data model), it won't matter where exactly within our model to put GraphQL. What matters is that you'll be able to search and filter for GraphQL APIs just the same way you can search for REST APIs and always get great results.

Could we be overthinking the architectural style? The TL;DR is "yes," it's possible. We realize the inputs aren't nearly as important as the outputs. Whether it's our forthcoming power search feature or the API to our directory (the so-called API of the APIs), we know, based on history, ProgrammableWeb's users just want to run sophisticated queries of our directory. As a means to this end, the architectural style is not nearly as important as the API's most important pattern. In some cases — for example, RESTful APIs — the primary pattern is that they are RESTful APIs. It doesn't matter that REST is an architectural style. It's just that people categorize it as a RESTful API.

But in the case of GraphQL APIs, the primary pattern is GraphQL. People will always view a GraphQL API as a GraphQL API, even though GraphQL itself is not an architectural style. The solution, which we are already experimenting with, is to forget about architectural style and simply go with the API's primary pattern, whatever that may be. Whatever we do, it won't be a huge change, nor will it be disruptive to the usability of the site.

How Streaming APIs are Addressed by ProgrammableWeb's New Data Model If there's a category of APIs to pay really close attention to, then Streaming APIs (sometimes called Push APIs or Publish & Subscribe APIs) might be it (for an in-depth guide to streaming APIs, see How and Why to Provide Event-Driven Streaming APIs). With a typical API (like a RESTful API), the client-side software (like a Web or mobile application) that consumes the API will issue a call to that API's endpoint whenever the application thinks it needs an update. For example, when a user clicks on a stock's ticker symbol in a financial management application, the application will make a call to an API to retrieve the latest stock price and other information that's relevant to that ticker symbol.

With streaming APIs, things basically go the other direction. First, the client indicates to the API that it wants to be updated without having to constantly make API calls. In other words, it wants to subscribe to updates in a way that those updates are sent or "streamed" back to the client as soon as there's new information. Using the stock example again, the user of the financial management app might indicate an interest in the symbol AAPL (Apple Computer, Inc.) and, then, from that point forward, a price update is pushed out (aka "streamed") to the app each time there's a new event. In other words, each time the price of Apple's stock changes. Architecturally, from the client-side point of view, this sort of "evented" approach is way more efficient because the app doesn't have to keep making API calls to get its updates. It subscribes once and waits for updates to come to it.

Seems great, right? Why don't all APIs work this way?

For most of the last decade, the streaming architectural style (officially "Push/Streaming" in the old ProgrammableWeb data model, see above graphic) was like a bastard step-child of the API economy. For starters, among the dozen or so technological approaches to building streaming APIs, none of them was a clear favorite across the API economy. Not only has there been a lack of widespread agreement on how to do it, until very recently, there were very few off-the-shelf solutions that demystified the implementation details. Even the most modern API management solutions lacked provisions for streaming APIs.

But things are starting to change as more and more constituents across the API economy are taking a closer look at streaming APIs. Off the shelf solutions are finally here and some of the newest API technologies — namely GraphQL and gRPC — have built-in provisions for streaming (something that REST officially lacks).

With the status quo for streaming about to shift into hyperspace, it was a good time for ProgrammableWeb to revisit how streaming was addressed in its data model. Like with GraphQL, it came as no surprise to us that it isn't that easy to put streaming APIs into a neat little box and be done with them. As can be seen from the previous graphic "Push/Streaming" has been one of the choices for an API's architectural style and it will remain that way under the name "Streaming." However, starting with the new model, we have some additional fields that, taken together, will fuel some powerful query-ability.

Once an API has been primarily designated as a streaming API (more on primary and secondary designations in a moment), that API can be further classified according to the specific streaming technology it uses. For example, Webhooks, Websockets, GraphQL Subscriptions, and HTTP/2 Streams (the streaming technology behind gRPC) just to name a few. Additionally, there are two more fields to further describe the directionality of the streams. This was a decision that was primarily informed by the bidirectional capabilities of gRPC (again, based on HTTP/2's inherent streaming capabilities), XMPP, and Websockets. The first of these two fields indicate whether the streaming API was built to be unidirectional or bidirectional. And the second of these two fields indicates the actual direction of the stream when the streaming API is unidirectional. The choices are "client to server" or "server to client."

Lastly, when it comes to streaming APIs, as we thought more about the great query-ability that we wanted to enable, we realized that some streaming APIs actually conform to two architectural styles at once. For example, many API providers offer streaming APIs based on Webhooks. Webhook APIs are always unidirectional and most if not all Webhook APIs stream from the server to the client. So, their primary function is most definitely to stream data to the client and therefore, they should be classified under the streaming architectural style. But then we realized that it's not that cut and dry.

The way a Webhook works, the client software indicates what stream(s) it wants to subscribe to. There's no standard way to invoke this subscription. Some API providers require the client to register its subscription using an API call. Other API providers require the user to activate the subscription from somewhere within the API provider's developer portal. Although there's no standard for subscribing to a Webhooks-based stream, the stream itself is implemented through a relatively standard approach whereby the client provisions an HTTP-based API that the server calls every time it has an update for the client. In other words, the server is essentially making a RESTful call to the client. Therefore, it could be argued that when a streaming API relies on Webhooks as its streaming technology, it's architectural style is both streaming and REST.

A similar situation arises in the case of GraphQL Subscriptions — the streaming mechanism that's built into the GraphQL specification. As discussed earlier, GraphQL's architectural style is designated as RPC under ProgrammableWeb's new data model. However, if it's a GraphQL Subscription, shouldn't it be a streaming API?

As said earlier, if you build a box for this, chances are nothing will ever fit perfectly and neatly into it. The bottom line is that a Webhooks API should show up in a search for Streaming APIs as well as in a search for RESTful APIs because technically, it's RESTful. Similarly, a GraphQL subscription API should show up in a search for Streaming APIs as well as in a search for RPC APIs. To accommodate these edge cases (there's only a few), our new data model offers a secondary architectural style field that's only activated in some instances (like these). And whenever streaming is one of two architectural styles, it is always the primary style since the API's most interesting feature is that it streams.

Conclusion

Our goal for ProgrammableWeb's API directory is to be the world's most comprehensive and searchable resource for developers who are deciding which APIs are the best fit for them and the next applications they're building. The directory's new data model represents a significant advancement over our prior model, enabling not just our forthcoming power search, but eventually, an API of our own. Also, In our view, our simple 5-stage framework for version status is a framework that API providers should consider using in their own developer portals instead of the old, highly ambiguous "deprecated." We're always looking for feedback and suggestions. Comments should be directed to our central inbox at editor@ProgrammableWeb.com.

Be sure to read the next API article: ProgrammableWeb's Most Clicked, Shared and Talked About APIs of 2019

 

Comments (0)