I recently had my second conversation with Josh Williams, CSO of Kontagent, a leader in the world of big data. In our previous conversation we touched on the potential for PhD level mathematics students to apply their knowledge to the challenges in the big data space. By the end of this second conversation it became apparent to me that there is a huge opportunity to be seized by companies that can leverage big data systems.
This opportunity has been created by the recent mastery of new levels of analytical complexity in large scale information systems. The ability of a company to understand the power of big data can lead directly into better quality products and services in the technology sector.
I invite you to reflect, for a moment, on the success or failure of websites, mobile apps, video games, or other media forms. Up until now whether or not a particular site or app succeeds appears to be a bit of skill, magic, and luck. When it fails, you're left wondering what went wrong. When it succeeds, how can you tell what components influenced its success? Everyone can offer their subjective view, but big data services offer the possibility for us to more deeply understand the influence of design in the success of our applications. The best part is that big data offers hard numbers and metrics that augment our subjective understandings with impartial numbers. It's the marriage of art and science!
Kontagent will routinely work with social game studios to help them gain insight into how user's are experiencing their game. There are many aspects of the user experience to understand when you consider Farmville, to take a familiar example. This game, and others like it, seeks to expand its reach and promote high levels of user engagement through social sharing. Each new achievement that's unlocked, or any significant event in the game play becomes an opportunity to "share" with your friends.
Now imagine that the creator of this game wants to be able to get a handle on how an individual user's sharing activity has affected the reach of the game and the engagement level of its users. The average Facebook user has approximately 150 friends. Starting with a typical user who initially shares with 150 friends involves staggering numbers at just few degrees of separation. At the third level of separation we are already approaching 3.4 million users, but being able to identify the users is just the starting point. We then want to understand how those users interacted with the shared messages from their friends in the game. We might like to know how many game invite messages they saw before finally deciding to join. We might also like to enable the game company to perform multivariate testing in order to establish which invitation messages are the most successful. There are a million different questions that could now be asked because Kontagent and other big data providers have developed the ability to work with these large data sets.
This brings me to perhaps my most significant discovery about big data thus far: it is a misnomer. The idea that we can store big data set is commendable for certain, but it is not the storage of this data that presents the compelling opportunity. Instead the opportunity is in the new class of questions that we can ask of the systems that we are building. Big data is really about big complexity and developing information architecture that can provide a tight feedback loop into our design processes.
It's now possible to have every user action generate a small bit of data which is transmitted, and analyzed for better understanding, through API of course. Just a few years ago the prospect of analyzing the log activity of a chatty application was just not feasible at scale. Now that the scale problem has been solved and we have systems that can handle the aggregate data flow, things will get interesting.
If this approach was incorporated into Photoshop, for instance, Adobe would be able to understand what actions a user takes before creating that first PSD file. Being able to know exactly which buttons we click, which menu options were explored, and it what order, can provide a lot of insight. Taking it a step further and comparing users to each other would allow application developers to reach out to their expert users or the new users. It could even be possible to detect that a user is suffering from multiple failed attempts to complete a particular task.
What big data can provide is a new level of granularity of insight into the user experience. That new insight will allow organizations to innovate and improve more quickly and with a more accurate understanding of the problems in their products.