How to Use the cURL Command to Examine the Contents of a Web API's Header

I was inspired to write this How-To article because, sometimes, in their API documentation, API providers fail to tell you what sort of data their APIs return and how that data is formatted. Yes, I know. It's hard to believe how, more than a decade after Web APIs first hit the scene, API providers still omit this all-important detail. But, it happens. So, this article is a quick introduction -- especially for those who are new to working with APIs -- on how to discover this basic bit of information. It should be fun for anyone -- especially beginners -- looking to dip their toes into the world of Web APIs. One note: this is not a comprehensive overview. It's just a very quick and dirty hands-on exercise to get your feet wet (hands, feet, toes? OK, enough of that!).

When any software -- your Web browser for instance -- communicates with a Web server, that communication typically starts with a collection of information known as the HTTP header that gives the other end some idea of what's going on. It's called an "HTTP header" because communication of this nature with a Web server typically happens over the HTTP protocol (hypertext transfer protocol); the official protocol of the World Wide Web. In fact, if you wanted, you could also call a "Web server" an "HTTP server."  The word "header" naturally means the head-end of the communication, or what comes first. One of the fields of data that's typically included in an HTTP header is the "content-type" and, confusingly, the information that's provided in that field is referred to as the "media type."  If you're wishing they just kept the language consistent, well, me too.  The content-type field in the HTTP header is the field that gives you some idea of the way that the data returned by an API (or any Web URL) is formatted. 

So, let's take a look at an example involving an actual Web API. By the way, one reason they're called "Web APIs" is that, in the same way that a Web server will send a Web page to your browser when your browser visits that Web server (aka "Web site"), that same Web server might send a differently formatted response when some other software like a mobile application visits its API. In fact, this is where the content-type field in the browser plays a big role. Either way, the information is being delivered to the consumer of that data (your browser, a mobile application, etc.) over the Web's protocol; HTTP. 

Let's take compare and contrast some example responses from one Web site: the Weather Underground (important note: the Weather Underground is NOT one of those sites whose documentation does a poor job documenting how its API formats its data. It actually does a pretty good job.)

Even though it's the root domain (the ".com" part) of the Weather Underground's API, if you visit http://api.wunderground.com with your browser, you'll probably get a Web page that's the same as Weather Underground's home page. Like any response from a Web server, this Web page was preceded by an HTTP header and if you use the following cURL command from the terminal app on your Mac or Linux machine (or using one of many cURL utilities you can download and install on Windows) to visit the same URL, you'll see what the header looks like:
 

curl -I http://api.wunderground.com/

The -I option (that's an upper-case "i") is interchangeable with the "--head" option and tells cURL to fetch the just the header from the specified URL. In the case of http://api.wunderground.com, the output will look something like this (to keep it simple, I've redacted some cookie data):
 

HTTP/1.1 200 OK

Server: Apache/2.2.15 (CentOS)

Access-Control-Allow-Origin: http://www.wunderground.com

Access-Control-Allow-Credentials: true

X-CreationTime: 0.156

Content-Type: text/html; charset=UTF-8

Expires: Mon, 29 Aug 2016 12:40:52 GMT

Cache-Control: max-age=0, no-cache

Pragma: no-cache

Date: Mon, 29 Aug 2016 12:40:52 GMT

Connection: keep-alive

Notice the sixth line down? It says:
 

Content-Type: text/html; charset=UTF-8

And that makes perfect sense, doesn't it? After all, if you visit that URL with your Web browser, you get a Web page back (and a Web page is a mixture of text and HTML; the markup language that makes Web pages look pretty). So, the content type is naturally "text/html."

But now, let's actually cURL the API with a real API request.  To cURL the Weather Underground's API, you'll need an API key from Weather Underground. To get this, created a registered user account on wunderground.com. Then, visit the API pricing page and "purchase" an API for the Stratus Plan (don't worry, it's free).  You'll have to fill out a form (don't worry, it doesn't have to be too official) and then, when your free purchase is complete, you'll see your API key (it'll be a long string of characters). 

This time, let's enter a slightly different cURL command:
 

curl -I http://api.wunderground.com/api/<<your API key goes here with no brackets>>/conditions/q/94107.json

Here's what it might look like with an actual API key. This API key is fake so it won't work if you try to cut and paste it. Be sure to use your own:
 

curl -I http://api.wunderground.com/api/1b02bc3d1ab6c044/conditions/q/94107.json

OK, in fairness, for the Weather Underground's API, we really don't need to go through this exercise because the URL itself ends with "json." It's a dead give away that the data that will be returned by this URL is formatted with JSON. But, we're just having fun, right?  This time, the output will look something like this:
 

HTTP/1.1 200 OK

Server: Apache/2.2.15 (CentOS)

Access-Control-Allow-Origin: *

Access-Control-Allow-Credentials: true

X-CreationTime: 0.154

Last-Modified: Mon, 29 Aug 2016 14:12:32 GMT

Content-Type: application/json; charset=UTF-8

Expires: Mon, 29 Aug 2016 14:12:32 GMT

Cache-Control: max-age=0, no-cache

Pragma: no-cache

Date: Mon, 29 Aug 2016 14:12:32 GMT

Connection: keep-alive

Notice the seventh line down? This time, the Content-Type field is different. Right? What does it say?
 

Content-Type: application/json; charset=UTF-8

It's telling us that the data that will follow (if we didn't retrieve just the header) is formatted with JSON.  Sure enough, let's take a look at the data you'd get if actually called the API.  To do this, point your browser to the same URL you just fed to the cURL command and, in response, you should get a whole package of data formatted as JSON that looks something like this:
 

{

  "response": {

  "version":"0.1",

  "termsofService":"http://www.wunderground.com/weather/api/d/terms.html",

  "features": {

  "conditions": 1

  }

    }

  ,    "current_observation": {

        "image": {

        "url":"http://icons.wxug.com/graphics/wu2/logo_130x80.png",

        "title":"Weather Underground",

        "link":"http://www.wunderground.com"

        },

        "display_location": {

        "full":"San Francisco, CA",

        "city":"San Francisco",

        "state":"CA",

        "state_name":"California",

        "country":"US",

        "country_iso3166":"US",

        "zip":"94107",

        "magic":"1",

        "wmo":"99999",

        "latitude":"37.76834106",

        "longitude":"-122.39418793",

        "elevation":"5.00000000"

        },

        "observation_location": {

        "full":"Potrero Hill, San Francisco, California",

        "city":"Potrero Hill, San Francisco",

        "state":"California",

        "country":"US",

        "country_iso3166":"US",

        "latitude":"37.769039",

        "longitude":"-122.402245",

        "elevation":"20 ft"

        },

        "estimated": {

        },

        "station_id":"KCASANFR880",

        "observation_time":"Last Updated on August 29, 7:23 AM PDT",

        "observation_time_rfc822":"Mon, 29 Aug 2016 07:23:10 -0700",

        "observation_epoch":"1472480590",

        "local_time_rfc822":"Mon, 29 Aug 2016 07:23:14 -0700",

        "local_epoch":"1472480594",

        "local_tz_short":"PDT",

        "local_tz_long":"America/Los_Angeles",

        "local_tz_offset":"-0700",

        "weather":"Overcast",

        "temperature_string":"57.9 F (14.4 C)",

        "temp_f":57.9,

        "temp_c":14.4,

        "relative_humidity":"83%",

        "wind_string":"From the West at 6.5 MPH Gusting to 9.8 MPH",

        "wind_dir":"West",

        "wind_degrees":281,

        "wind_mph":6.5,

        "wind_gust_mph":"9.8",

        "wind_kph":10.5,

        "wind_gust_kph":"15.8",

        "pressure_mb":"1019",

        "pressure_in":"30.10",

        "pressure_trend":"0",

        "dewpoint_string":"53 F (12 C)",

        "dewpoint_f":53,

        "dewpoint_c":12,

        "heat_index_string":"NA",

        "heat_index_f":"NA",

        "heat_index_c":"NA",

        "windchill_string":"NA",

        "windchill_f":"NA",

        "windchill_c":"NA",

        "feelslike_string":"57.9 F (14.4 C)",

        "feelslike_f":"57.9",

        "feelslike_c":"14.4",

        "visibility_mi":"10.0",

        "visibility_km":"16.1",

        "solarradiation":"30",

        "UV":"0.0","precip_1hr_string":"0.00 in ( 0 mm)",

        "precip_1hr_in":"0.00",

        "precip_1hr_metric":" 0",

        "precip_today_string":"0.00 in (0 mm)",

        "precip_today_in":"0.00",

        "precip_today_metric":"0",

        "icon":"cloudy",

        "icon_url":"http://icons.wxug.com/i/c/k/cloudy.gif",

        "forecast_url":"http://www.wunderground.com/US/CA/San_Francisco.html",

        "history_url":"http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KCASANFR880",

        "ob_url":"http://www.wunderground.com/cgi-bin/findweather/getForecast?query=37.769039,-122.402245",

        "nowcast":""

    }

}

OK, we're almost done! Just for kicks, let's do one more experiment with the Weather Underground's API.

David Berlind is the editor-in-chief of ProgrammableWeb.com. You can reach him at david.berlind@programmableweb.com. Connect to David on Twitter at @dberlind or on LinkedIn, put him in a Google+ circle, or friend him on Facebook.

Comments