One API for the Web, Oh the Possibilities

Guest Author
Dec. 14 2011, 12:00AM EST

Shion Deysarkar is the CEO of Datafiniti, the first search engine for data and recently launched newco. In a previous life, Shion ran a predictive modeling firm. He enjoys playing poker and basketball, but is only good at one of them.

The API landscape is an extremely well chronicled and championed space. Every product has its own API nearly, and developers still get excited about digging into the newest, greatest data on the market. It’s the developer equivalent of having to have the newest Apple gadget, right when it becomes available.

In our zeal to access more and more data though, we’ve forgotten what working with data is supposed to be all about.

Let’s take social activity as an example. When you think of social data APIs, you probably think of Twitter, Facebook, and a few others like Google+. Now, each of these API providers would have you think that the data they provide is a complete, 100% full-proof representation of what you want to track, whether it’s influence, purchasing intent or something else.

Based on the way the market has materialized over the years, we developers have been trained to look at data in terms of siloed, single-source APIs. The problem with it though is that it’s like trying to see the entire world through a keyhole.

Not to mention, when the collective developer community does this, we get locked into platforms and are left combating and generally ranting about fragmentation in our data and applications.

It doesn’t have to be this way. The truth is the ability to see an entire world of data isn’t unreachable. We could have access to it all. It’s out there right now, on the web.

The issue becomes then how do we get it. Today, there is no easy way to go about this.

What we’re lacking is a single API for the Web – a veritable master key to access them all, in one place. You’ve probably seen what we’re doing at Datafiniti even. This is exactly the future we’d love to live in. But outside my own company and ambitions, I want to explain the logic behind why exactly this should exist.

My initial answer – just ask any developer if they’d like access to more types of data and less APIs bogging them down with time-consuming integrations. You’ll likely get a big “hell yes!”

The Door Opens: Now, What’s Possible?

It’s a little mundane to say, but a lot of possibilities are suddenly in grasp when data is not only more accessible but put into close contact. Today, by contrast, you have to go to a number of sources to get everything you need.

When it’s all whittled down to a single API source, we can forget about bearing the brunt of normalizing all data sources into a single stream. Just go ahead and throw that step out the door.

In other words, instead of having to provide siloed access points for individual data types, a single API would provide just one access point for all your data, cutting out a lot of time and headaches. A rather large side benefit of this is that developers are offered a lot more flexibility as far as licensing too, which means hopefully a stronger ecosystem would be able to take hold and develop without as much backlash and feuding as exists today.

Thinking more conceptually, developers have not necessarily been crafting worlds of data built on an accurate representation of reality either. Twitter does not an entire social profile make.

For instance, take the phone number for your doctor. It might be shown differently or be out of date on some websites or through certain APIs. Normalization across different source completely removes these problems and ensures you’re operating with the most accurate and up-to-date information, a hugely important issue when data quality is how you pay the bills.

There’s also just a lot of day-to-day, practical benefits that accompany a single API for the Web.

For one, rather than having fragmented developer communities, split among the different API silos, a single API would need to bring with it a large community to provide strong guidance for features, improvements and support. This would essentially be the entire developer ecosystem working together to improve the basis for innovations. A bit of a dream admittedly, but a good dream no less.

Bringing new data types into an application would also just become easier in and of themselves. For example, let’s say you wanted to start bringing in product data into a daily deal application you had built.  Rather than trying to find and learn a new product API (or build a web crawler even), you would just run a different query on the same API for all web data.

The bigger hope is that a singular API would give way to more imaginative and robust applications that we can all really benefit from -- allowing us to do all sorts of things like reference an infinite amount of data, trace from location check-ins to people that frequent that spot, to identify what brands interest those people, to name what products those brands sell and so on.

Suddenly everything becomes much simpler.

But Can We Do It Really?

A single API for the Web definitely sounds compelling, but there will be challenges, of course.  In my honest opinion, the two biggest are implementation and licensing.

For data collection and implementation, you’d need a massive data collection and storage effort to get it off the ground.  Building out the normalization process alone is highly difficult, regardless of how simple the concept may seem at the surface.  And dealing with licenses proves an even greater challenge.

Right now, we’re working on solving these problems at Datafiniti and I can say from experience, that most of the data we’d like to include in our own API is almost always semi-protected by Terms of Service. Overtime, it is our hope that TOS agreements and the company’s that write them become more willing to opening up their data and see that building another API is not the only solution on the table.

The good news is that a single API is becoming less of a question of if it will ever happen, but when. We at Datafiniti are working hard to solve this problem and the feedback that what we’re doing is sorely needed has been much.

There can be a better way without simply adding another hundred APIs to the pot. You can have your data cake and eat it too. It just takes effort and dedication on part of the entire community.

Photo credit: Shappy Seasholtz

Guest Author

Comments

Comments(3)

User HTML

  • Allowed HTML tags: <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • Lines and paragraphs break automatically.
  • Web page addresses and e-mail addresses turn into links automatically.
simon

As part of a university project, I created a generic social-networking API to test the viability of such a system. The results were positive. Most developers appreciated the desire for such a system. My implementation however still required some refining.

I do think however that the answer to this problem is Linked Data: the web as a database. Although there are still some issues (such as speed of traversing it), it is definitely the way forward!