Image recognition software typically works by leveraging Machine Learning to identify content from images. While it can be used to identify things like famous landmarks, colours and objects, the technology is more commonly used to identify faces to make it quick and easy to tag friends in photos on Facebook.
Tagging friends in photos is nothing new, but the more recent Facebook feature that pops up asking ‘Do you want to tag X?’ when hovering over an image got developer Narendra Rajput wondering how Facebook identifies who the person is. In this recent post, Rajput explained how he figured out the undocumented API.
Using the Chrome Dev Tools, the author found various div blocks for defining the boxes around faces, with another block of HTML with the names of identified users. Since this data is only loaded after clicking on the photo, he switched to the network tab to try and identify an Ajax call to the servers. After trawling through hundreds of requests, Rajput found "/ajax/pagelet/generic. PHP/PhotoViewerInitPagelet".
Rajput removed unnecessary response params, leaving just fbid (the photo ID) and 2-3 other constants. He used a Chrome feature to replicate the request as CURL to retrieve HTML data with information about the users, then tried applying the API to images that aren’t uploaded on Facebook.
He uploaded a test photo using the FB Graph API with options "temporary"=> true and "published"=> false to upload the images temporarily without publishing them. A simple app on FB Photo Friend Finder was used to upload the photo and the API returns the photo ID which is passed to the PhotoViewerInitPagelet API mentioned above to return HTML.
The API is limited to suggesting only people already in your Facebook friends list, but Mr Rajput wrote a small Ruby script that needs just a path to the image as an input param and automates the rest of the process.