Web Crawling Powerhouse Datafiniti To Provide Web's Datasets via SQL

Garrett Wilkin
Oct. 11 2011, 02:00PM EDT

If there’s a new dawn in the age of search, Datafiniti is the sun. They are the first to take a new and profound approach to search in years. Most search engines return a list of links to web pages, but Datafiniti has much bigger plans. Instead of links, they return a set of data. A search for a burrito in Texas would give you a list of restaurants complete with goodies like reviews in addition to the important bits like name and address. The dataset itself has been out there for years, but its the aggregartion and presentation that’s different. With the Datafiniti API, it is now possible for developers to easily integrate that data into web applications.

This is incredibly profound because the task of normalizing and serving data sets is a big one.  For any given subject area, there are multiple APIs to learn and master in order to collect just the most relevant information.  This is a real problem for programmers and Datafiniti is making an earnest effort tp solve it.  They provide one simple API call that consists of a token and an SQL query.  The results can be provided in JSON or CSV. This makes it possible for programmers to use Datafiniti as a great simplifying normalization point, but it also makes the data accessible to the moderately tech savvy spreadsheet user.  Datafiniti will serve copy and pastable datasets that will likely find their way into slide show presentations.  There’s a page dedicated toward helping people up the syntax learning curve and a screencast to walk you through some typical queries.

An important aspect of this service is that its built on a powerful web crawling architecture developed by 80legs which is now a part of Datafiniti. In a recent conversation CEO Shion Deysarkar described the crawling system as “completely transient and ephemeral cloud computing grid” which allows Datafiniti to operate nimbly and cheaply. It also gives their service a true competitive advantage. They’ve got a serious engine under the hood that should keep the data set healthy on all metrics of coverage, recency, and accuracy.  A high level of quality is Datafiniti’s stategy for adoption.

This team has experience, tried and tested technology, and a game changing idea. I can’t wait to see where they take it. Datafiniti is one of more than 200 search APIs, but they really stand head and shoulders above the rest.

Garrett Wilkin

Comments

Comments(5)

[...] realtime open movement information and a book trade village service. In further we lonesome Datafiniti’s efforts to yield a web’s datasets around SQL queries. Below are some-more sum on any of these new [...]

[...] realtime public transit information and a book trading community service. In addition we covered Datafiniti’s efforts to provide the web’s datasets via SQL queries. Below are more details on each of these new [...]