NASA has launched an API portal as part of its ongoing attempts to encourage reuse of the significant data sets available from the U.S. space agency’s missions and other ongoing projects. The API Portal complements other initiatives, including the recently launched NASA Data Portal, open source code libraries and GitHub account.
“We originally set up data.nasa.gov as our open data Library, but we found that it is not such a good place to catalog our APIs and manage those resources,” says API portal lead Jason Duley. “So Dan Hammer set up our API docs GitHub pages as a quick way to get them online, and then we built api.nasa.gov as the central hub for our APIs.”
A Lean Approach to NASA’s Open Data Strategy
Given the tight budget NASA is operating within, it is taking a minimum viable product and lean approach to the Developer Portal. NASA is a large and disparate conglomeration of subagencies, so even finding what APIs the agency has published has been a task in itself, says Beth Beck, manager of NASA’s Open Innovation program, which operates out of NASA’s CIO office.
To start, NASA worked on collecting as many of its data sets as possible and published them under the data.nasa.gov catalog, which uses Socrata’s open data Platform. APIs are listed on the home page of that catalog, and in addition, developers are referred to Socrata’s Open Data API tools, which enable users to wrap any of NASA’s data sets in an API.
The API portal release means api.nasa.gov works as the API management layer for the agency. Developers can register for just one API Key that can be used for multiple APIs owned by NASA. This portal also creates a central system so that usage rates can be tracked, and NASA can gain better insight into developer needs.
“What api.nasa.gov does is it Builds a management layer across those APIs, so that NASA developers don’t have to worry about registering for multiple keys, caching, etc.,” says Duley. “Moving forward, we have also identified staff across NASA that are building data sets, so we are now offering to publish their API endpoints so that devs have a structured programming interface to access them.”
According to Beck, this is also a legislative requirement of the agency: “One of our mandates is that we must report on who is using our data. So our API portal will allow us to do that better, as well as be able to dedicate more resources to those with greatest use.”
NASA’s API Portal
The APIs available via the API portal include:
- Landsat imagery: Endpoints include satellite images of the Earth for any given location, and the date on which any location last had an image taken.
- Astronomy Picture of the Day (APOD): This popular data set allows programmatic access to NASA’s space image of the day, alongside concept tags that categorize the image.
- Earth temperature anomalies: Endpoints provide data on any temperature anomalies for a specific location and date range.
- Patents: Endpoints include NASA’s extensive patent library and details of each project and technology for which a patent is held.
- Space sounds: Endpoints are audio files of sounds collected during various space missions that can be filtered by particular mission or other text terms.
Other APIs listed on the data.nasa.gov website include the Digital Universe API (published in partnership with the American Museum of Natural History), which shows stars, constellations and exoplanets; the Global Imagery Browse Services (GIBS) API; Mars rover data sets; TechPort data; and the MODIS land products Web service.
“What we are trying to do is as quickly as possible give these tools to citizens,” says Duley. Beck confirms: “Our mandate is to put all of NASA’s inventory online and open. We want people to innovate and use this data."
In the past, NASA’s APIs have been unwieldy and difficult to locate, let alone use, according to some developers.
For example, NASA’s annual Space Apps Challenge has seen more than 900 application submissions this year, but that includes only around a dozen entries that made use of NASA’s APIs. Software developer Quintessence looked to NASA’s API resources last year, with the goal of building some space science applications:
I did my undergrad in astronomy and maths and physics, so I wanted to design a junior engineering project and thought how cool would it be to use the NASA API.
I first tried pulling the solar data. I thought it would be good to pull space weather and make a mini-app to do that. I couldn’t even get the data set to pull, let alone analyze the data. I tried one other data set to see if anything else was working, and then I figured it was a more global issue than solar specific.
Since last week, Quintessence has taken another look at the API portal from NASA:
On the bright side, what it does do appears to work! But still on the downside, there is not that much I can do with it. I do like the APOD data, because that it is available as a quick data source. I see the digital universe API is now available. You know those various sky maps? I thought it would be neat to make my own Web-based program where you can see the planets and stars at your location, so the availability of this API is really cool, and I can see that I could pull that data from what is already there.
What I would like to do eventually is write software for space science applications, so this is very encouraging.
But it would be nice if it is a little easier to search. From the data.nasa.gov portal, it requires you to know the name of the project, so you might want to be able to search solar data, but you have to know to search for the Helioviewer.
But overall, Quintessence is optimistic about the new portal and sees a definite role in the agency letting more developers know what they are doing: “I follow various NASA social media accounts, and they should make sure everyone knows that this portal exists. People need to know they did something awesome!”
Quintessence’s struggles with API discoverability are something Duley and Beck acknowledge is a continuing problem on the open data site, but they see that as part of the trade-off of opening data on a limited budget.
One of the biggest difficulties with the design of Socrata’s platform is that it is not always easy to identify the latest and most relevant version of a data set. This has been an issue in city deployments where people can add their own user-generated data sets to sit alongside city-published data. In theory, this is citizen engagement at its best: In many cases, citizen-contributed data may even be more user friendly or have more useful data endpoints than the officially published version. But the result for a catalog platform can be that multiple items of one data set are shown, making it difficult to know which one to use or how to compare data sets with the same name.
For NASA, the problem is that when one data set is updated, both versions remain in the catalog with the same name. Duley explains:
There are approximately 16,000 data sets on our portal, and every quarter we add data there. Basically, we are rolling up the data and providing a uniform interface. So we are getting data sets from earth sciences and TechPort and other subagencies, and we need to report that data. We are responsible to report our data to citizens, but we don’t have the funding or resources to go in and create a thorough understanding for the data dictionary, or to make the data sets easily digestible. So there is a lot of data set redundancy; they are not duplicates, they are multiple data sets. There might be six different releases of the same data. So unfortunately, there is so much data, it ends up not being very user friendly. So one of our priorities is to identify which data sets are high value.
In the new release of data.nasa.gov, we put in a suggestion engine. Folks have been using it quite heavily, which allows citizens to tell NASA what data sets they are interested in. So we have 100 or so requests asking for specific data — for example, patent data, project data, celestial body positions — and we are allocating some of our resources to doing some fact finding and working with data owners to make that more usable. And based on that feedback, we are prioritizing it that way.
Keith Prifte from Socrata has confirmed that they are currently in the process of supporting NASA to add a "Recently added" drop down feature. This would allow users of data.nasa.gov to more clearly identify the most recent version of any dataset in the catalog, and to filter searches by the date the dataset was last updated.
Using Landsat APIs
Open source mapping software Mapbox has been using many NASA APIs for some time. Mapbox imagery specialist Charlie Loyd says NASA’s APIs have been crucial to many Mapbox projects:
We use a variety of NASA APIs (and relatedly, U.S. Geological Survey APIs) to acquire imagery data both for our basemap and one-off projects and demos. One particularly powerful interface for us has been GIBS, which allows for sizable data transfers at speed.
Since NASA provides such a wealth of data sets, the main barrier for practical use of the APIs is often simply figuring out what’s available and how. Once you find the right API and have identified the data set you’re after (or subset thereof), the processes for acquiring data are generally well-designed and scalable. If you’re doing something at the terabyte scale, it never hurts to get in touch with the technical folks at NASA, who are always informative and accommodating.
The MODIS imagery data set is a particular favorite of ours for demos. It’s constantly updating and gives you an easily intelligible layer with global coverage. It’s available through GIBS and also in subsets via HTTP.
In March, Mapbox imagery specialist Camilla Mahon announced the company's Landsat-live product, which provides a constantly refreshed world map using the latest available satellite imagery. Loyd explains:
To help provide clear APIs for accessing NASA/USGS data, we’ve recently been involved with the Landsat public data set on AWS. This public data set makes it truly easy for anyone with some basic skills to Fetch large quantities of near-real-time Landsat data. We’d like it to be the kind of API that’s so simple it doesn’t even feel like an API.
In addition to being available as endpoints on NASA’s new API portal, NASA’s Landsat data is also used by Esri, Development Seed, Planet Labs and others, and has been instrumental in enabling a more thorough understanding of our impacts on land. Landsat data has been used to prevent forest thinning of old-growth redwood trees, enable better understanding of forests’ contribution to global carbon cycles, and significantly increase the evidence base showing impacts of human development on natural environments.
NASA’s API-Related Policy Goals
NASA’s moves to open its data sets via API through Socrata’s platform — and now its more recent work to create a central landing point for developers wanting to access all its APIs — reflect policy goals from the Office of the President through to NASA agency-wide policy, and even trickle down to subagency policy goals.
At the national level, the president’s digital strategy — launched three years ago this week — required government agencies to make their data “open and machine-readable by default.” As part of this initiative, NASA noted in its annual performance report last year that the open innovation program has been tasked as the agency-wide data management team “to ensure that new datasets adhere to information architecture standards, including open format and the use of metadata. The agency continues to encourage, and will soon require, missions to publish non-sensitive data and to periodically update the data inventory.” (NASA’s response to the president’s digital strategy is available in machine-readable JSON and XML formats.)
At the agency level, NASA’s Open Government Plan articulates the importance of the data.nasa.gov platform and recognizes that “developers, technologists, entrepreneurs, citizen scientists and many others can contribute directly to the exploration of space and Earth by helping to create new ways of looking at this data.” In addition, by making the data open, NASA hopes internal efficiencies and learnings from re-examining the data from within NASA’s own ranks can also be heightened.
Alongside this, other policy initiatives such as the plan for accelerating technology transfer can benefit from leveraging the potential of NASA’s APIs. At present, NASA has released a series of technology road maps and is seeking feedback until June 10. The goal of these road maps is to enable strategic investment in new technologies that will help NASA achieve its core missions. As technology projects are prioritized and progressed, information about the technologies is also made available to the public and industry so that others can begin to consider how to incorporate the NASA tech into commercial and community opportunities. Part of this process leads to the technologies being documented through a patents database, which is accessible via the Patents API. But the technology road map documents never actually refer to the Patents API, nor to the potential that enabling programmatic access to NASA’s technology investments might generate.
So the potential of NASA’s open data initiatives may be stymied — or at least not optimized — because of a lack of connection with NASA’s technology development initiatives.
Beck sees this as part of the challenge of an agency like NASA and is hopeful that over time the value of open data and better management of NASA APIs will become more recognized across the agency’s operations. She explains:
When Congress appropriates funds, it is to do missions, and our mission is to send humans out to the far reaches of the universe and let others know what we have found. So our data is an output of a mission. So our priority within NASA is to do the mission. What is intriguing for us is to see what citizen-generated innovation can tell us about our data that we didn’t know, what incredible insights are embedded in our data that we didn’t know. What is the second, third, fourth, fifth use of this data so that the value continually increases? People here are super busy, so if they can show us where the data is, we will point to it so others can find it so it is the least disruption to their day. Then we show them what new themes have come out of their data. This is the first step in this conversation, as we learn, we can take that back to the data owner. We are generating some interesting follow-up conversations.
Duley is hopeful that this conversation can help leverage a better understanding of the value of APIs and open data alongside the technology agenda at NASA: “One of the reasons why the Patents API was done is because the patent data is available and it is one of those data sets that is out there and not very usable. It was something identified by Dan Hammer as high value.”
Developers and NASA: A Reciprocal Relationship
Software engineering has long had a reciprocal relationship with NASA. Computer scientist Margaret Hamilton, for example, is credited with hand coding the software that guided the Apollo space program, which resulted in the successful landing of Apollo 11 on the moon in 1969.
In return, it is where Hamilton also conceptualized some of the now-foundational approaches common to today’s coding environments, including Asynchronous software, priority scheduling and end-to-end testing.
With the new API portal, developers are again being invited into a reciprocal relationship with NASA to participate in using open data and APIs to contribute to the work of NASA, and to use NASA’s APIs to apply the space agency’s data and technological insights to external coding projects.