SwiftIQ Announces Innovative Data Mining API

SwiftIQ has released a first-of-its-kind data mining API aimed at uncovering the deep associations previously hidden in large datasets. The Frequent Pattern Mining (FPM) API has wide potential for use across major sectors, government, and healthcare, with the ability to speed up Big Data analysis and identify the opportunities that "connect the dots" for suppliers and service providers across any number of industries. SwiftIQ -- which offers a range of predictive algorithm APIs that customers use in conjunction with their own datasets -- is making the FPM API available on a case-by-case basis at present. Jason Lobel, CEO and Co Founder of SwiftIQ, spoke with ProgrammableWeb to announce the API.

"Frequent Pattern Mining is a new algorithm in the SWIFT Predictions catalog," Lobel told ProgrammableWeb. "It is an algorithm for mining big data to quickly discover all the permutations that might occur within a large dataset. It runs on Hadoop in the cloud as an on-demand service. FPM results are then stored and can be searched across tens of millions of records instantly. You can even visualize and embed the results via our d3-based JavaScript Library for simple interpretation. These insights extend the realm of human capabilities by unearthing complex relationships as needed."

IMAGE: SwiftIQ uses d3 to visualize products with the greatest affinity from large datasets of supermarket purchases.

Big Data Mining via API

The engineering team incorporated the FPM algorithm into their full cloud-based big data infrastructure with the aim of enabling customers to ingest, process, and deliver complex pattern insights at high volumes far more easily than was possible before. The end-to-end data preparation, algorithm, and output search capabilities are now available in the SwiftIQ interface and as a REST API. The algorithm is currently being tested by some of SwiftIQ's enterprise clients. However, Lobel indicated that he is open to more individual-use cases if developers want to contact the data mining start-up, which won Top Innovator for its API Infrastructure at DataWeek 2013. Lobel commented that "eventually we may make the API self-service, but right now we are making sure it is scalable for large datasets."

The REST API for the FPM output is available in JSON, which exposes the permutations. The results are stored in a distributed querying engine that also has an available API.

Although Lobel has been in discussion with larger companies -- retailers, supermarkets, consumer packaged goods, medical device companies, and others -- he is eager to support technology companies that want quick access to the data mining infrastructure when designing solutions for their customer markets.

Industry Use Cases: Six Practical Examples

Supermarkets: Lobel sees supermarkets and packaged goods retailers as one of the most industry-ready sectors able to take advantage of the SwiftIQ FPM API. With the FPM API, supermarkets can analyze which combinations of products are most often purchased together and can set weekly in-store promotions; or, they can determine store layout based on big data fact-based evidence of how their customers are shopping.

For example, a supermarket could review the differences in shopping basket affiliation between particular branded goods. Here, you can see how different brands of light beer appeal to very different consumers, which has implications for how supermarkets could market to each shopping cohort.

IMAGE: FPM analysis of basket purchases at a supermarket shows that one type of light beer is enjoyed with snacks (and maybe with a homemade Italian dish?), while the other is more popular among health-conscious buyers.

Quantified Self: Life-hacker and The 4-Hour Chef author, Timothy Ferriss, is currently working with the Lift app and UC Berkeley to study dietary habits using quantified self data. Having something like SwiftIQ's FPM API sitting behind the collection of large quantified self datasets could help reveal the minimal effort required (and therefore the activities most likely to be maintained) to effect positive behavior change and improve health outcomes. "If variables like weather, mood, activities, and diet are tracked by these apps, the FPM API provided by SwiftIQ can identify relationships between these behaviors," said Lobel.

IMAGE: Using FPM with large quantified self datasets could reveal the path of least effort to positive behavior change.

Retail: "The immediate use is in merchandising, store layout, and upsells, but we are finding even for those who sell electronics over the phone, or high-end furniture for example, sales agents can literally type in the product while on the phone call, and it will tell them what other customers bought at the same time," Lobel explained. "For e-commerce, retailers can see if for some common orders whether deliveries are being delayed because of waiting on one product from a supplier, so they can mine these patterns to make better stock decisions and improve customer satisfaction."

Healthcare: Hospitals collect large amounts of data, for example, on patient healthcare at the time of hospital discharge. Identifying risk factors that might lead to readmission to the hospital can be difficult to identify within such large population datasets. Adverse reactions that might occur between prescribed medications, social factors influencing rehabilitation, or the therapeutic impact of new medical devices can remain obscured in large population datasets where lots of variables are attached to each individual case. Unearthing common patterns within these big datasets is one of the greatest difficulties for advancing evidence-based public health. "Research companies and healthcare providers have contacted us to explore how to identify patterns in patient outcomes, such as reasons for readmission. This form of pattern mining is good for detecting relationships with high variety. It is not predictive, Machine Learning. It is all fact based and could be applied to analyze those 100 reasons for hospital readmission from all the actual patient records," Lobel said.

IMAGE: FPM could analyze individual records at hospitals with the highest levels of readmissions to identify the most common "hidden" factors (demographic, medical, social, etc.) among readmitted patients.

Travel: One of the immediate uses of FPM for businesses is the ability to create choice and recommendation engines based on data mining of hundreds of other customer purchases. In an experienced-based economy like travel, businesses could suggest excursions, package deals, restaurants, and day activities based on all previous travelers' itineraries. "There are numerous opportunities: The things we could do with a database such as TripAdvisor would be incredible to explore," Lobel said.

Smart Cities: More data on the interplay between local citizens and the areas where they live, work, and play would help city governments make better urban-planning decisions with less policy intervention. Cities would be able to tweak the margins where greatest risk occurs. For example, in areas with an active nightlife, cities could use FPM to determine whether adding more nonalcoholic activities at night has an impact on reducing the amount of alcohol-related nighttime violence, whether investing in more late-night transportation is a better use of public spending, or whether the heavy regulatory hand of cutting back on venue opening hours has the greatest impact. FPM could help reveal more effective ways to curtail traffic congestion, reduce crime, better manage waste collection, or activate economic development opportunities. With increasing sensor data being collected by smart cities, FPM could help identify ways in which cities could improve their operations by focusing on the opportunities with the greatest potential for reducing population and land pressures.

IMAGE: FPM could analyze each individual crime record in high-risk areas to identify common place-making elements that could be addressed to improve neighborhood safety. (Map from Rutgers, State University of New Jersey, Office of the Vice President for Research and Economic Development)

SwiftIQ published the announcement about the new FPM API on their blog this week.

By Mark Boyd. Mark is a freelance writer focusing on how we use technology to connect and interact. He writes regularly about API business models, open data, smart cities, Quantified Self, and e-commerce. He can be contacted via e-mail, on Twitter, or on Google+.

Be sure to read the next Database article: Orchestrate Eases Complex Database Tasks