A Python console program called mitmproxy that is used to intercept and examine HTTP traffic has become a go-to tool for individuals looking to reverse engineer APIs thought to be private and secure when those APIs are called by a client-side Web or mobile application (for example, an iOS or Android app on a smartphone). This trend, along with the recent security vulnerabilities revealed within the Moonpig API authentication strategy, highlight a need for increased API security analysis.
A developer recently published an explanation of how he used mitmproxy to reverse engineer the private API used by the mobile app of travel service Kayak. And this week, another developer revealed how, when told that a popular service he liked, Couchsurfing, would not be offering a public API, he used mitmproxy to reverse engineer the company's private API.
Aldo Cortesi, CEO of Nullcube, a security consultancy, founded the mitmproxy project. He believes that making APIs hard to inspect is generally counterproductive. "There’s no way to be 100% sure your API can’t be inspected, so you can’t rely on keeping it private," he told me. "The consequence is that APIs, like crypto implementations, are at their best when they’re developed openly and widely scrutinized. There’s also an ethical dimension. Apps have deep access to our lives, and I believe that it’s a user’s right to know what an app communicates upstream. Without the ability to inspect API traffic, we’d never have known about many serious privacy issues."
Balancing Transparency and Security
Mitmproxy works using the man-in-the-middle principle to intercept HTTP communications. To intercept and examine Web API communications, mitmproxy typically runs on a machine or in a process thats physically positioned to intercept the HTTP or encrypted HTTPS traffic between the client application and the server that hosts the API endpoint (usually, somewhere across the Web). It comes with functionality that allows that machine or "man-in-the-middle" to be viewed by the client application as a trusted certificate authority (CA) and generate dummy trusted certificates on the fly. Once mitmproxy has established trust with the client application, it is able to decrypt and inspect supposedly-secure API traffic before passing it on to the endpoint it was originally intended for. In other words, despite their transmission via HTTPS (the encrypted version of the Web's core protocol), mitmproxy makes confidential API keys, OAuth tokens, and sensitive user information, all thought to be secure by both the application's user and API provider, immediately discoverable.
Tools like mitmproxy can be somewhat thwarted through the use of "locked down" mobile platforms like iOS and a technique called certificate pinning. Certificate pinning associates a known host with a certificate, essentially instructing the client application not to trust any CA but the "pinned" host (the actual API provider's server). To beat certificate pinning, "an analyst basically has to root the client device (ie: the smartphone) and install a shim, or decrypt the app and modify it to disable pinning. There are other, rarer scenarios like the recent goto fail vulnerability where pinning fails, but we can’t rely on these being present," Cortesi stated.
Cortesi said use of certificate pinning may be on the rise:
I come across [it] increasingly often in consulting. A good example here is the suite of official Twitter applications. Twitter definitely has good reason to pin, because they are a legitimate target for powerful actors (like governments) that definitely have the ability to subvert the certificate authority system. However, they do not allow their pins to be overridden by locally installed root CAs. Why is this? Well, my guess is that they are partly trying to protect their API. The official Twitter apps have access to a range of functionality that other applications don’t (like getting a complete user timeline that includes favorites, retweets and @-mentions). Access to these features is protected by the official Twitter OAuth credentials, which would be visible if traffic could be inspected.
So is certificate pinning a good idea? "I think any company that deals with sensitive user data should at least consider certificate pinning," Cortesi said. "Done right, it lets us shore up the weaknesses of the current CA system, while still being transparent to analysts." He points to Google as an example of a company using certificate pinning in an appropriate fashion, as the search giant's apps permit locally installed root CAs to override the pinning.
Unfortunately, certificate pinning is not without some complexity. It requires developers to manage their own certificate trust system and depending on how this is done, a certificate leak, for instance, could require a developer to release an app update. "One common way to cope with this is to pin against at least two certificates, one of which is kept as an offline, locked-down reserve. If your primary cert leaks, you can then move to a secondary one without interruption," Cortesi explained.
Obscurity Doesn't Guarantee Security
The use of mitmproxy to reverse engineer high-profile APIs demonstrates an obvious fact: As far as private APIs are concerned, security by obscurity is an increasingly difficult proposition, because the tools available to prying eyes are only becoming more sophisticated and accessible.
Companies utilizing private APIs will need to become familiar with security techniques like certificate pinning and employ them where appropriate. But using these techniques with the goal of keeping third parties from reverse engineering APIs may be misguided and, if history is any indication, ultimately unsuccessful.