How userstack Delivers a Scalable, Real Time Cloud-based User-Agent API

Continued from page 1.

apilayer is a Web entrepreneur's friend.

Working with userstack's User-Agent API

Let's dive into the userstack User-Agent sniffing, detection API and crawler, spyder and bot blocker API in more detail.

The Single User-Agent String Lookup

Let's first learn to use the API in its single User-Agent string lookup mode.

Example userstack API request for single User-Agent string lookup. Request shown above, list of request parameters and definitions shown below.

Example userstack API request for single User-Agent string lookup. Request shown above, list of request parameters and definitions shown below.

Here's the response in JSON from the request above. You can run it live from the documentation by clicking on the "Run API Request" button shown above:

{
    "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36",
    "type": "browser",
    "brand": "Apple",
    "name": "Mac",
    "url": "https://www.google.com/about/company/",
    "os": {
        "name": "macOS 10.14 Mojave",
        "code": "macos_10_14",
        "url": "https://en.wikipedia.org/wiki/MacOS_Mojave",
        "family": "macOS",
        "family_code": "macos",
        "family_vendor": "Apple Inc.",
        "icon": "https://assets.userstack.com/icons/os/macosx.png",
        "icon_large": "https://assets.userstack.com/icons/os/macosx_big.png"
    },
    "device": {
        "is_mobile_device": false,
        "type": "desktop",
        "brand": "Apple",
        "brand_code": "apple",
        "brand_url": "http://www.apple.com/",
        "name": "Mac"
    },
    "browser": {
        "name": "Chrome",
        "version": "71.0.3578.98",
        "version_major": "71",
        "engine": "WebKit/Blink"
    },
    "crawler": {
        "is_crawler": false,
        "category": null,
        "last_seen": null
    }
}

Now, let's try customizing the request to act as a browser detection API which returns the output as XML. I'll restrict the fields argument to browser.

My Safari User-Agent string is:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15

Here's my request paraphrased:

https://api.userstack.com/api/detect?access_key=xxxxxxxxxxxxxxxx
  ua= Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15
   output => xml 
   fields => browser

Here's the response in XML:

<result>
<browser>
<name>Safari</name>
<version>12.0</version>
<version_major>12</version_major>
<engine>WebKit</engine>
</browser>
</result>

This API makes browser detection pretty simple. In fact, the userstack API is very simple to use even for broader device, operating system parsing and crawler detection.

Customizing Response Objects for User-Agent Parsing

You can optimize your use of the API and performance by customizing the fields you request. Here is a portion of the table of Response Objects you can choose from:

Summary of userstack API Response Objects to customize with your API requests. Available textually in the documentation.

Summary of userstack API Response Objects to customize with your API requests. Available textually in the documentation.

When shown above visually with a ">" and sub-parameter such as "os > icon", you can request those as a field parameter using a period. For example:

fields = os.name,device.name // returns both the "type" and "device" > "name" objects

Crawler detection is shown below:

Crawler detection

In other words, the response object crawler.is_crawler indicates that the User-Agent string is a crawler. The crawler categories are as follows:

  • search-engine
  • monitoring
  • screenshot-service
  • scraper
  • security-scanner
  • unknown

JSONP Callbacks

Here's more detail from the userstack documentation of JSONP callbacks. Basically you set the callback parameter to the name of your coding language's function for processing the userstack API response:

Screenshot of documentation for JSONP callbacks, available in text form from the documentation. Top is a call with callback function name. Bottom shows an example result.

Screenshot of documentation for JSONP callbacks, available in text form from the documentation. Top is a call with callback function name. Bottom shows an example result.

The following is quoted from a now missing page of Cameron Spear's blog and now at StackOverflow:

"JSONP, which stands for 'JSON with Padding' (and JSON stands for JavaScript Object Notation), is a way to get data from another domain that bypasses CORS (Cross Origin Resource Sharing) rules. CORS is a set of 'rules,' about transferring data between sites that have a different domain name from the client."

As they show the CALLBACK_FUNCTION could be implemented as JavaScript as shown below:

function localJsonpCallback(json) {
   //You can process response from here.
}

I also wrote an embedded PHP example which demonstrates JSONP callbacks, where a framework generates my HTML and JavaScript via PHP, in the Introduction to the ipapi Geolocation API, another apilayer service.

Now, let's learn about bulk User-Agent detection and User-Agent parsing.

Bulk Lookup of User-Agent Strings

Here's a screenshot from the documentation of parsing User-Agent strings at once using bulk lookup:

Example userstack API request for bulk User-Agent string lookups. Top portion shows the example request and parameters. Below, defines each the parameter array of User-Agent Strings and GET request parameters.

Example userstack API request for bulk User-Agent string lookups. Top portion shows the example request and parameters. Below, defines each the parameter array of User-Agent Strings and GET request parameters.

Bulk requests require at least the Business level plan at $49.99 per month, or $39.99 per month when billed annually.

The ua_batch array of User-Agent strings is sent to the userstack endpoint as a POSTFIELD array in the body of the POST operation. Below is a brief cURL script in PHP for a bulk User-Agent string API detection request:

$requestArray = [
  'ua_batch' => [
       'Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1',
       'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; FunWebProducts)',
       'Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko)',
       ],
  ];
$query = http_build_query($requestArray);
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'http://api.userstack.com/detect?access_key=############&fields=os.name,browser.name&output=xml');
curl_setopt($ch,CURLOPT_POST, 1);
curl_setopt($ch,CURLOPT_POSTFIELDS, $query);
$result = curl_exec($ch);
echo $result;
curl_close($ch);

Here's the response in XML as I requested. I deliberately requested only two fields to simplify the response which includes multiple results:

<?xml version="1.0" encoding="UTF-8"?>
<batch>
  <result index="0">
    <os>
      <name>Windows XP</name>
    </os>
    <browser>
      <name>Firefox</name>
    </browser>
  </result>
  <result index="1">
    <os>
      <name>Windows 2000</name>
    </os>
    <browser>
      <name>IE</name>
    </browser>
  </result>
  <result index="2">
    <os>
      <name>iOS 8</name>
    </os>
    <browser>
      <name>Apple Mail</name>
    </browser>
  </result>
</batch>

Language Compatibility

Again, as a REST API, the userstack User-Agent parser is usable in any platform that supports HTTP. userstack has examples in Node.js, JQuery, PHP, Python, Ruby and Go.

You can browse all of the code examples in the documentation. Here's Python for example:

import requests
params = {
    'access_key': 'YOUR_ACCESS_KEY',
    'ua': 'Mozilla/5.0 (iPad; CPU OS 9_3_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13F69 Safari/601.1'
}
api_result = requests.get('http://api.userstack.com/detect', params)
if api_result.json()['device']['type'] == 'tablet':
    print 'It\'s a tablet';

And here's Go:

package main
import (
    "encoding/json"
    "fmt"
    "net/http"
)
type Device struct {
  Type string
}
type Response struct {
  Success bool
  Device Device
}
func main() {
  httpClient := http.Client{}
    req, err := http.NewRequest("GET", "http://api.userstack.com/detect", nil)
  if err != nil {
        panic(err)
    }
  q := req.URL.Query()
  q.Add("access_key", "YOUR_ACCESS_KEY")
  q.Add("ua", "Mozilla/5.0 (iPad; CPU OS 9_3_2 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13F69 Safari/601.1")
  req.URL.RawQuery = q.Encode()
  res, err := httpClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
  response := Response{}
  json.NewDecoder(res.Body).Decode(&response)
  if(response.Device.Type == "tablet") {
    fmt.Printf("It's a tablet.\n")
  }
}

Keeping Your API Key Secure

If your API key is ever at risk, you can reset it in the dashboard:

Example of userstack User-Agent API Access Key on Dashboard (redacted) with a Reset button at right. Useful if your API key may be compromised.

Example of userstack User-Agent API Access Key on Dashboard (redacted) with a Reset button at right. Useful if your API key may be compromised.

While this is the type of single request API commonly deployed in JavaScript, the language can make itvery difficult to conceal API keys and requests.

You could obscure your key from casual observers by placing your userstack API code within an encrypted JavaScript file. But, developers know how to observe the network calls. And, the smart ones can even decrypt your code, often.

The only secure way from JavaScript is to write your own request to your host server which then makes the request to the userstack API itself and returns it via AJAX to the browser.

Something like this JavaScript excerpt:

$.ajax({
   url: 'https://yourwebsite.com/js/request',
   data: {ua: window.navigator.userAgent,
          fields: browser,
    },
   success: function(data) {
    browserObj=data;
   }
});

Then, your js/request endpoint must make its own request to userstack, concealing your userstack API key.

In Closing

I hope you've had some fun learning about the userstack User-Agent API for browser and device and parsing and crawler detection. I'm considering integrating it into my blog which has far too many crawlers and bots as fans.

To recap, userstack is easy for startups and small companies as well as large enterprise level corporations to integrate into their existing products. I'm glad it's a newly integrated API service at apilayer, all of whose services are similarly constructed to simplify adoption and development.

Learn more about their suite of products:

Screenshot of portfolio of apilayer companies which offer related web-based data APIs.

Screenshot of portfolio of apilayer companies which offer related web-based data APIs.

apilayer and userstack appreciate your questions, comments and feedback. You can also follow them on Twitter @apilayernet and the apilayer Facebook page.

About apilayer

userstack is the most recent solution from apilayer, an established leader in product APIs. apilayer aims to help developers and businesses automate and outsource complex processes by serving them with dedicated and handy programming interfaces.

Three other products by apilayer include ipstack, a real-time geolocation & reverse IP lookup REST API, streetlayer, an international address validation service and eversign, an electronic document signing, approval and delivery service. You can find my articles about them at ProgrammableWeb.

Be sure to read the next Browsers article: Google's Planned Update to Chrome Extensions API Causes Brouhaha

 

Comments (0)