Splunk Digs Deep into Big Data Application Development

While working with Big Data affords a lot of potential business value the complexity of building applications that manipulate all that information can be nothing short of daunting. Not only do most of the currently popular approaches require developers to master arcane interfaces such as MapReduce, the performance of the application tends to suffer under the weight of all the data that needs to be processed.

Starting today the folks at Splunk want to change the equation with the release of a software development kit of Splunk Enterprise, an indexing engine that allows organizations to sort and analyze massive amounts of Big Data in real time.

According to Jon Rooney, the basic idea is to allow developers to more easily invoke Splunk indexing engines to process queries across massive amounts of Big Data using an SDK for Python, Java, JavaScript and soon PHP. That approach eliminates the need to master interfaces such as MapReduce, while at the same time leveraging the performance attributes of Splunk to run queries across Big Data files quickly using a set of RESTful APIs.

It was that specific capability that attracted Socialize, a provider of a set of tools that makes it easier for developers of mobile computing applications to add social networking features to their applications, to the Splunk Platform. According to Socialize CTO Isaac Mosquera, the company was looking to accomplish two primary goals. The first was to process queries about the results of on-line advertising programs in real time it offers based on the number of page views that its developer partners generate on the social network. The idea is allow customers to optimize their ad spending by making decisions based on data being generated in real time. The results of those queries are then displayed in dashboards that Socialize presents to its customers.

The second goal was to allow the internal IT staff to more easily discover and analyze performance issues that might be adversely affecting the Socialize web site. Rather than having to acquire the expertise required to deploy and manage something like Hadoop, Splunk allows Socialize engineers to query a repository of Big Data generated by the site using a standard set of management tools and workflow processes they already know.

Both those tasks would have required a lot of work, says Mosquera, especially when it comes to supporting mobile computing devices access Socialize dashboards. The Splunk SDK reduced the effort associated with delivering those kinds of capabilities by a factor of 10, says Mosquera.

Splunk’s Rooney says the company envisions developers taking advantage of the Splunk SDK to run queries against data residing in not only specific devices and applications, but entire sets of data that may be generated by social networks or machine-to-machine (M2M) applications. The goal isn’t necessarily to replace Hadoop as much as it is to over an alternative approach to accessing Big Data in way that is more accessible to average developers that can now create Big Data applications in as little as five hours, says Rooney. In the near future, Splunk expects to add support for Ruby on Rails and C# in addition to ODBC and Odata application Integration formats, says Rooney.

There’s obviously a massive amount of interest in building Big Data applications that span multiple use cases. Right now, however, the assumption is that working with Hadoop and either MapReduce or a hybrid SQL interface is the only way to make that happen. The Splunk SDK provides an alternative that doesn’t require a data scientist to configure, set up and then manage. And perhaps best of all, it’s based on a set of indexing engines already used in a number of systems management applications that have already stood the test of time.

Be sure to read the next News Services article: New Google App Engine Graduates Several Experimental Features