A few weeks back, I reported on the developer outrage that was provoked by Yahoo!’s sudden and unexpected shutdown of a service that, for well-more than a decade, was being consumed as an API by an untold number of apps. All of those applications broke. Never ones to be shy, developers voiced their displeasure.
But the service in question was not an official API. It was a feature of Yahoo! Finance — an everyday HTML button -- that was not intended for developers to consume in their applications. It was really intended for ordinary Web users who might want to download a CSV file containing stock data for subsequent import and analysis with their spreadsheets. But, in a move that was tantamount to screen-scraping, developers jumped on it like it was an API when they discovered more than a decade ago that Yahoo! wasn’t requiring identification or authentication of any sort to use the service.
Without identifying themselves, developers could emulate a click of that button once per day, or once per second. It was the equivalent of a free, near real-time stock quote API; an incredibly valuable service to a great many applications and users. Especially since there are a great many API providers out there charging hard cash for the exact same thing. I’m guessing that Yahoo!’s new owners (Verizon) spotted a giant resource pig in their log files, traced it to a bunch of machines “clicking” the button with absolute wreckless abandon, and thought “WTF!?”
Off went the service. Even the button intended for ordinary Web users broke.
Whether the “APIs” are official or not, there are hundreds if not thousands of APIs out there that, like that download button, are wide open. In other words, no form of authentication or identity is required to use them. For some API providers, the thinking is that API adoption is inversely proportional to any friction in access. But as I pointed out in my second article, if developers can access an API or service without identifying themselves, the service provider has no sure-fire way of contacting those developers to warn them of something like an upcoming service disruption.
Even if Yahoo! wanted to contact the developers whose apps were consuming the CSV download, it had no way of doing so (whether it really wanted to make contact with them — I certainly would have if I was involved — is another question altogether).
The situation got me to wondering whether it was time for Web APIs — and all Web traffic in general — to be capable of bearing warnings along with normal payloads. Or, what I think of as an in-band warning. Sort of like when a bill arrives in the mail with a special indicator on it that the previous month's payment is overdue.
In researching the options, I came across a possibility on Stack Overflow that I also wrote about. The bad news is that I misunderstood the implementation details and so what I wrote — that the approach relies on extending the HTTP status code that comes with every Web response — was technically inaccurate. The good news is that even with the correct technical details, the idea of doing in-band notifications not only remains unchanged, it can apply no matter what the HTTP status code is. In other words, an in-band service warning can essentially be appended to an HTTP response of any type and status.
So, in this article, I will not only offer the correct technical details, I will also offer a tutorial that anyone can try in order to see it in action for themselves.
First, the correct technical details.
Originally, I thought the idea was based on the notion that any 3-digit HTTP status with a 2 in the first digit (eg: 200, 201, 299, etc.) was still viewed as as much of an HTTP success as an actual HTTP status of 200. So, my original explanation suggested that when a Web service (an API, a site, etc.) wanted to issue a warning as a part of an otherwise successful response, that it could flip the status to 299 (instead of 200) and append the status with some text that offered additional warning details.
Instead of relying on the HTTP status, the actual implementation relies on a special HTTP header known as the Warning header. The origins of this warning Header can be found in IETF RFC 7234, the original intention of which is to define "HTTP caches and the associated header fields that control cache behavior or indicate cacheable response messages.” But the Internet is full of RFCs and specs that were originally intended for one thing and then got extended to support other ideas. And rather than propose an entirely new HTTP specification, maybe it’s easiest just to take advantage of what’s already there.
A bit deeper into RFC 7234 is section 5.5.7 which describes the aforementioned optional “Warning: 299” HTTP header. The Mozilla document offers some easy to read implementation details, noting that the syntax is as follows:
Warning: <warn-code> <warn-agent> <warn-text> [<warn-date>]
The Stack Overflow post showed some examples of what this warning might look like:
Warning: 299 - "Deprecated API”
Slightly More Complex:
Warning: 299 api.blazingFrog.com "Deprecated API : use betterapi.blazingFrog.com instead. Old API maintained until 2015-06-02”
In the first example, a hyphen is substituted for the warn-agent parameter (this is allowed) and the warn-text (always in quotes) is very simple. But the second example includes the warn-agent along with a much more detailed explanation in the warn-text parameter.
My idea is for the API industry to work together to come up with a standard vocabulary for the warn-text that all API and site operators could use in order to package a spectrum of service warnings into HTTP responses. For example, much the same way there’s a fixed set of numerical codes that can be offered as the aforementioned warn-code (110,111,112, 214, 299, etc.), the API industry could agree to a standard vocabulary that could be provided as the warn-text. For example. a machine-consumable comma delimited string that involves a sub-code (150), a brief explanation (Service Retirement Pending), and a URI that the machine can crawl for more information:
Warning: 299 - “150, Service Retirement Pending, http://yoursite.com/details/150.htm”
Earlier this week, I not only proposed this idea in a presentation given to the December edition of the monthly Washington, DC API Meetup, I mocked its functionality using my MacBook, Node.js, Express.js, and ngrok. The idea was simply to show what it might look like. Here’s how I did it (and how you can do it as well).
Then, install Express.js. Express.js is a framework for Node.js that eliminates a lot of the pain that’s associated with coding Web applications or writing code that provisions APIs from scratch.