Thanks to cheap storage and a new generation of data stores designed specifically for collecting massive amounts of information, more and more businesses are sitting on vast troves of data. The challenge now: finding ways to extract value from all that data.
The need to turn raw data into actionable knowledge is fueling the growth of an ecosystem of young companies aiming to develop solutions that can literally turn data into dollars. One such company is Prelert. Its core offering, Anomaly Detective, uses unsupervised machine learning to establish what constitutes normal behavior patterns in massive amounts of data so that customers can spot anomalous activity in real time.
Using the Anomaly Detective engine, Prelert customers can, for instance, monitor their networks and be alerted to traffic and usage that is abnormal and may signal a security threat. Unlike other kinds of solutions that don't rely on machine learning, Prelert's technology does not require customers to set arbitrary rules, limits or thresholds. Instead, it identifies the meaningful relationships and correlations in the data.
To help companies leverage all of their data with its offering, Prelert last week unveiled an API that allows its customers to perform real-time and batch analysis with Anomaly Detective using a REST interface. In conjunction with this, Prelert has made available Java and Python clients for its API, as well as a set of connectors that allow data to be forwarded from common data sources such as MySQL and PostgreSQL databases as well as HTTP access logs.
Lots of data sources and broad applications necessitate an API
The applications for the kind of anomaly detection Prelert's technology enables are diverse. "We have users uncovering the fingerprints of advanced hackers in order to make sure they are not the next company to suffer the high profile loss of tens of millions of credit card records. Anomaly Detective is at work in dozens of sites preventing the next major service outage by spotting performance problems and their cause as soon as they start developing. We have users optimizing dating sites and other web based services. We even have customers diagnosing traffic congestion in major metropolitan areas," Mark Jaffe, Prelert's CEO, told me.
"Prior to Prelert you could only access this technology if you were a data scientist or if it was a feature embedded in a pre-existing app like a security firewall or network monitoring device," he stated. Prelert is trying to change that and has an ambitious goal: "democratize" anomaly detection and "put it in the hands of millions of knowledge workers."
Releasing an open API is a big part of realizing that goal. As the amount of data companies are collecting has exploded, so too have the number of technologies companies are using to house the data. The development of an open API ensures that companies will be able to use Anomaly Detective whether they're using Hadoop, a traditional SQL database or any one of the many popular NoSQL data stores that have emerged in recent years.
"There are simply more and more environments in IT or other operations realms where the volume of data sources simply exceeds the users' ability to try to monitor them using performance thresholds and rules. So we have a large number of enterprises, service providers, application developers and product vendors that are interested in our anomaly detection engine. The best way to meet this demand is to provide an Open API that is designed for download, quick evaluation and ease of deployment," Jaffe explained.
As companies use Prelert's API to learn the normal patterns so that outliers can be spotted, Jaffe expects that the applications for anomaly detection technology will only continue to grow in number. "[It] will drive a large number of applications in consumer trends, buying patterns, cyber security, asset optimization and other future business apps," he predicted. And, if Jaffe has his way, his company's engine and API will be working behind the scenes to support all of those applications.