Anatomy of A Real World API Security Breach

It seems like at least once a week we hear about another huge security breach. And that’s only when it happens to huge Fortune-500 companies. It’s happening so often that we’re starting to zone it out. In fact, the millennials are known as the generation that simply doesn’t care about privacy—we live out our lives online and we simply accept the privacy and security trade-offs ofthat lifestyle.

Problem is, we’ve only seen the tip of the iceberg that us, aboard the virtual Titanic, are heading for. If the security and privacy of how we connect isn’t dealt with soon, we are going to see disasters like we’ve never imagined.
And that’s where the API comes in. The Application Programming Interface is not only the lynchpin of how everything connects online, but, as ProgrammableWeb’s editor in chief David Berlind said last year, API security “is fundamental to the security of the entire Internet,” and, thus, the connected lives we lead.

Berlind first broached this topic at an APIcon panel last year: “Anatomy of a Real-World API Attack and What to Do About It.” This panel included John Bradley of Ping Identity, Adam Dawes of Google, and Sunil Sadasivan of Buffer. Together they dissected the multi-layered October 2013 security breach of Buffer’s infrastructure.

Back in 2013, around 400,000 Twitter and Facebook OAuth access tokens for Buffer's social media service were compromised, filling those two networks with magical fruit weight-loss ads on behalf of more than 30,000 users in only ten minutes.

This piece works to tell the story of how one business dealt with its API security being compromised and what they’ve learned since.

You’ve Been Hacked, but How?

Sadasivan, Buffer’s CTO, reflected on the shock felt by his then eight-person, three developer team: “We had no idea how this occurred. We were completely baffled. We triggered the kill switch. And we were kind of lost.”

Two days later, his team began to understand when they received an email from their Platform as a Service (PaaS) provider MongoHQ after one of its employees was duped into giving up his administrative login credentials. These unknown hackers weren’t simply doing a basic hacking, but had automated the paging of a display through the MongoHQ database which included information from Buffer and numerous other clients. For about four days, the hackers were able to scrape user information and access tokens off the screen.

And the sophistication of the attack didn’t just end there. Twitter uses client-side Authentication for each Twitter request, so you need to prove you have a client secret in order to post to Twitter. But Buffer was storing the plain text version of its client secret within its GitHub source Code Repository. One of the developers had a shared password, which led to another breach of the Buffer GitHub Repository.

Both MongoHQ and GitHub provided Buffer with the ability to take the sad look back at the logged movement of the hackers. It was clear that these hackers were searching for anything they could use to reproduce, share, or sell, including their Amazon Web Services keys. “They could have done a lot more and that’s what seems like they were after,” Sadasivan said.

You’ve Been Hacked, What Now?

“After going through this breach, we extracted anything we could think of from our repository. We reset credentials. We made security way more of a priority. We think about it every day now, Sadasivan said. “Going through a hack really changes your perspective on things.”

After you’ve put a stop to the first attack, what can you do from preventing it from happening again? For Buffer, it’s very clear that they shifted their priorities, to make security a greater concern. The team adopted two-factor authentication that was previously avoided in favor of maintaining a faster development speed by a minute or two.

One of the major eye-openers of this particular attack was that they were able to gain access via the users’ Twitter and Facebook OAuth tokens. However, while the hackers got these tokens, by using OAuth the user names and passwords were luckily still protected.

“The idea behind OAuth when we first created it was to stop people from having to give all of these usernames and passwords to all of these sites that were enabling API access to these valuable services that were popping up like Twitter,” Bradley, one of the founders of OAuth explained. He went on to point out that because of OAuth, Buffer was able to stop the attack within ten minutes, “and they were also able to recover from that in a fairly simple way.”

The increasing focus for business is how to take that OAuth identification and API security to the next level.

That’s where identity guru Dawes jumped into the conversation, talking about his role at Google: “It’s really our mission to lower the friction around authentication on the Internet so that’s both increasing the security of a given session through things like two-factor [authentication], making that as easy as possible, as good a UX [ User Experience] as possible. And then as users travel across the Internet, really minimizing the amount of log-in friction that they have to go through. Clicking a button, doing a federated login is really easier than a username and password.”

Google theorizes that the safest Internet is one where everyone logs in simply via one login or even eventually none.

Dawes continued that “Unless you really have hundreds of people managing your security—of your log-in system, of your API system, all that—you’ve probably got quite a few holes that you’ve got to take care of.” He argues that the more you can rely on standards, federations and the well-tested, well-managed infrastructure of others, the safer you and your users will be.

More than a year later, lessons learned from Buffer’s worst ten minutes continue to be that OAuth and two-factor authentication are both still two of the standards to live by if you want to maintain as much API security as possible.

The Importance of Two-Factor Authentication

Known for transparently toting a learn-from-your-mistakes culture, Buffer looked at this security breach as an opportunity to beef up safety and security, as well as a way to transparently to share its experience with others. Instead of passing the blame, Buffer admitted its role in the breach, pointing specifically to how it hadn’t encrypted access tokens.

As reported on its blog: “For the past few weeks, we have been focusing on making Buffer the safest, most secure way for you to manage and publish to your social media accounts. We have a number of awesome things to show you. The most important step in this process is a feature we’re announcing today: 2-Step Login.”

Since human error—aka passwords like Obama’s 123457—and repeated passwords that come along with it are usually the biggest threat to API security, two-factor authentication provides two means of identification, without having to force people to use long, complex passwords that they’ll just forget. In Buffer’s case, this two-step login is a password and then a code sent via SMS or the Buffer app. Working both for personal and team accounts, it helps verify you are really the person trying to access your account.

The teams of MongoHQ, Facebook and GitHub worked together forensically to identify one IP address for one member of what’s assumed to be a team that hacked Buffer. Ironically, after all Buffer employees introduced two-factor authentication, the person behind that IP address then tweeted via his or her own Buffer account, asking how to get around that. This is a sign of two-factor authentication being much more secure, as well as the hacker’s hubris perhaps leading to his downfall, as the FBI took over the investigation.

What’s the Moral of this Story?

As Berlind said, every time you login in somewhere new, “You really have no idea what best practices are in place at these different security providers. You could be giving a password to them that’s a shared password and they could go off and do something else with it. You have no idea what their true agenda is. It’s almost pure blind faith that almost all users of the Internet are acting on.” Essentially, we are waking up and realizing that we aren’t going to want to provide a username and password for everything we do.

As a security standard, OAuth is good but imperfect. It is still lacking in areas that are currently being addressed by the Internet Engineering Task Force. Meanwhile, two-factor authentication a la what Google has to offer is far from ubiquitous. But what can we do in the meantime? Well, first and foremost Don’t Use the Same Password.

But as Sadasivan and his team at Buffer learned the hard way, you just can’t sleep. Until that perfect solution is discovered, API security must be the absolute highest priority of any business.

Have you had a similar hacker API security violation? Tell us about how you discovered, halted it and prevented it from happening again below!

And if you want to learn more about being hacked through the eyes of a CTO, you can’t miss Sunil Sadasivan’s revealing personal take on it all.

Be sure to read the next Security article: Google Offers Bounty for Android Bugs