How Google's Now On Tap Probes Android Apps For Context

Google Now Director Arpana Chennapragada Announcing Now-On-Top

At Google I/O 2015: Google Now director Aparna Chennapragada announces Now On Tap.

While the Google Photos announcement at Google I/O 2015 was big (and relevant to developers), another announcement — that of Google's Now On Tap — was bigger. Now On Tap could just have easily been called "Google Now On Steroids," or better put, "Now On Contextual Steroids."

Google Now has always been envisioned as the ultimate personal digital assistant (I say this with true reverence for the original PDA from the '90s, the Palm Pilot). By studying whatever personal signals Google can find (a.k.a. your personal context), such as your location at certain times of the day, the service attempts to anticipate what information you'll need and when you'll need it. It then delivers that information to your Android smartphone in the form of a digital index card that you can use to take further action (i.e., it might have some links). Or you can dismiss it. For example, once Google Now figures out where you work and live and when you typically begin your commute, it aspires to surface a card that gets you from one place to the other as quickly as possible, taking current traffic conditions into consideration, and it does so in proximity to the time you normally begin your commute.

From the Google I/O keynote stage, implying that the current Google Now isn't enough, Google Now director Aparna Chennapragada asked, "Why can't it tell you when and where to park your car? Why can't it tell you to pick up the milk that your wife reminded you about?" To answer these sorts of questions, Google Now needs to know more about what's going on with you in the moment. Where are you exactly? What are you on your way to? What time do you have to be there? What else do you have to get done?

Enter Google's Now On Tap.

Now On Tap takes Google Now to the next level by tapping into more signal sources in an effort to better determine personal context, which should result in improved digital assistance. Chennapragada asked attendees to imagine that they were planning a trip to Disney. You'd want to know how long it takes to get there, when it opens, what the most popular rides are, how long the lines are when you're at the park, what you're likely to need when you get there and so on.

Or imagine you have to catch a flight, but you have to return your rental car with a full tank of gas first. Ideally, a truly personal digital assistant would be like Chloe O'Brian on the TV show 24, wirelessly feeding Jack Bauer with all the just-in-time information he needs to get his job done as he's getting it done. To do this, Google Now needs to know where you are, what time it is in your time zone, what time your flight is, that you have a rental car, where it needs to be returned, which gas stations it makes the most sense to tank up at, how long the security lines are at the airport, and how long it will ultimately take you to get to the gate.

Today, not all of these "signals" — some of which aren't necessarily personal to you — are available to Google Now. But with Now On Tap, they will be. For example, if you use OpenTable to book a dinner reservation and your spouse reminds you via text to pick up the laundry before the dry cleaner closes, Now On Tap might deliver a card that not only reminds you of the dinner reservation, how to get there (with an opportunity to order transportation), what's on the menu and what your friends have to say about the food, it will prompt you to set a reminder to pick up the laundry too.

But in order to collect these additional personal signals and optimize the assistance it can give you, Now On Tap needs deep access to the apps you use: the app you use to text with your spouse, the app you use to book your reservations (e.g., OpenTable), to book your transportation (e.g., Uber or FlyWheel), Yelp and so on.

The big question, of course — aside from whether, in the name of privacy, Google has or should have access to all this information — is what developers must do to their code to enable this capability. After all, it's being announced at Google I/O, right? The answer, according to Chennapragada on the keynote stage: nothing. It just works.

Wait. Surely, there must be something the developer must do. Or was this yet another Google I/O announcement that was ironically devoid of developer relevance? I decided to dig deeper, making my first stop at one of the many "sandboxes" that makes Google I/O special. The sandboxes are basically chalk talk stations where attendees can request personalized ad hoc instruction from Google's best and brightest engineers. Sandboxes are organized by subject matter, and the one that seemed closest in relevance to my question was the Sandbox for deep application indexing.

Google advertises app indexing as the technology that makes it possible for content that's inside an application to appear in search results. As evidence of how Google is taking the fight to Apple, app indexing works on both Android and iOS apps. For example (see image below), if you are using Google to look for real estate from your mobile device, a search result from Trulia might offer you the opportunity to view the listing from inside Trulia's app instead of on its website (the way search results normally link to websites). If you don't have the Trulia app on your device, the search result also gives you the option of installing the app from the Google Play store.

While developers don't have to do anything specific to the code in order to enable app indexing, some other adjustments must be made to both the app and its corresponding website.

On first blush, the app indexing sandbox seemed like a natural place to begin my inquiry. I assumed that Google's ability to discover stuff about us from the apps we use had something to do with app indexing. The engineers I encountered were coy. When I asked how they were getting at that data — suggesting the app index or Screen Scraping as possibilities — their answer implied that it was a deeply guarded secret that they couldn't disclose.

Not satisfied, I tracked down Chennapragada, who didn't waste any time telling me that Now On Tap was, in her estimation, the biggest step forward for Android coming out of Google I/O. "Now On Tap is very strategic for us," Chennapragada said. "The idea that context can invoke an assistant to help you with something is pretty powerful."

Chennapragada confirmed to me that app indexing was key to making Now On Tap work and that no changes to code are required. "The destination app has to be indexed by Google. This makes it possible for us to deep link into an app like OpenTable. Developers don't have to change their code."

She was also more forthcoming about how Google is able to mine personal context out of the apps we use: "No, we don't do any screen scraping. It uses Android's view hierarchy." The view hierarchy is a system-level technology that many developers will recognize as one of Google's more powerful tools for debugging and optimizing their applications.

In the world of Android development, a view is the most basic of building blocks for an app's User Interface. For example, a full screen in an Android app (one view) is often broken down into smaller, separately controlled rectangles (each of which is also a view), all of which can be broken down into more views and so on. Thus, a hierarchy. To the extent that these views are responsible for all screen drawing, event handling, buttons, text fields, etc., having access to the view hierarchy is like having the keys to the kingdom — exactly what Now On Tap apparently needs to mine personal context out of the apps you use.

As you can imagine, giving any application (Now On Tap or otherwise) carte blanche access to the internal ongoings of other apps is sensitive stuff from both a security and privacy point of view. Though I haven't seen the entire end-to-end User Experience, Chennapragada described to me how Now On Tap has to clear two end-user-controlled hurdles before it is allowed to see a bird's-eye view of someone's digital life. The first of these, she said, "is that the end user must designate Now On Tap as an assistant on the phone." After interviewing her, I realized how this also suggests that a user might be able to designate some other application (perhaps one that's not from Google) as an assistant. Second, the end user has to deliberately grant Now On Tap the right to snoop on everything you do in your applications.

So, while it's true that developers don't have to do anything in their code in order to enable Now On Tap, they still must optimize their content for app indexing (if developers want that content to surface on a Google Now card), and users must still grant Now On Tap the permissions it needs to work its magic.

The next question for users who enable Now On Tap is the extent to which the bulk of their interactions will take place through the Now On Tap interface versus the applications themselves (potentially pushing the application layer into the fabric as a tier that does little more than channel information between Web services and Google Now). With unbridled access to the view hierarchy, Google Now should be able to create events in third-party apps as easily as it is able read them. When it reminds you that it's time to head to the restaurant and presents you with the option to order transportation from Uber, you may never even see Uber's app in the course of accepting that offer (even though Uber's app is on your phone).

On the heels of Google I/O, Wired summarized Google On Tap as Google's ingenious plan to make apps obsolete. Some very simple apps, perhaps. For example, I could imagine Google Now commoditizing the totality of the "special offer" workflow found in apps like Groupon. There's a list of offerings, each with some metadata, and a very limited number of calls to action. But, where app workflows involve myriad possibilities and branches, each of which involves human interaction, the native app is likely to provide the better user experience, with Google Now serving more as a reminder that it's time to use that app, rather than taking over its user experience.

Be sure to read the next Personal Information Management article: Evernote Is Dead? I Don't Think So