Lessons Learned From Building APIs For the US Federal Government

As a part of ProgrammableWeb's ongoing series of on-demand re-broadcasts of presentations that are given at the monthly Washington, DC-Area API meetup (anyone can attend), this article offers a video recording, and audio-only podcast, and a full transcript of the Dec 5, 2019 discussion given by Oddball.io CTO Rob Wilkinson. Based on his experience (which includes gigs with the Veterans Administration and the Department of Defense), Wilkinson believes he's discovered some valuable lessons when it comes to launching APIs not just for the federal government, but for pretty much any enterprise.

Tops on his list of must-haves is a clear understanding of the API's value proposition (and we here at ProgrammableWeb.com strongly agree with that establishing that clear understanding). Wilkinson has seen to many APIs launched under the assumption that "if we build it, they (the developers) will come." That's also true based on ProgrammableWeb.com's observations of the API economy. However, more often than not, they don't come.  

Wilkinson then covers more ground regarding tracking the growth of your APIs, self-service developer accounts, your path to production and so much more. In an effort to please all of ProgrammableWeb's users by providing content in whatever form they prefer, Wilkinson's full presentation is embedded below in video form, as audio-only, and as a full-text transcription.

The DC-Area API Meetup almost always takes place on the first Tuesday of every month. The attendees consist of API enthusiasts and practitioners from all around the federal government as well as businesses and organizations that are local to the DC Metro area. There is no charge to attend and attendees get free pizza and beer, compliments of the sponsors. The meetup is always looking for great speakers and sustaining sponsors. If you're interested in either opportunity, please contact David Berlind at David.Berlind@programmableweb.com. If you're interested in attending, just visit the the meetup page and RSVP one of the upcoming meetups. It's that simple.

Lessons Learned From Building APIs For the US Federal Government

Editor's Note: This and other original video content (interviews, demos, etc.) from ProgrammableWeb can also be found on ProgrammableWeb's YouTube Channel.

Audio-Only Version

Editor's note: ProgrammableWeb has started a podcast called ProgrammableWeb's Developers Rock Podcast. To subscribe to the podcast with an iPhone, go to ProgrammableWeb's iTunes channel. To subscribe via Google Play Music, go to our Google Play Music channel. Or point your podcatcher to our SoundCloud RSS feed or tune into our station on SoundCloud.

Tune into the ProgrammableWeb Radio Podcast on Google Play Music  Tune into the ProgrammableWeb Radio Podcast on Apple iTunes  Tune into the ProgrammableWeb Radio Podcast on SoundCloud

Transcript of: Lessons Learned From Building APIs For the US Federal Government

Rob Wilkinson: Yeah, as Gray [Brooks] said, my name is Rob. I'm going to talk a little bit tonight. You know what an API is, he's kind of introduced it. I'm going to talk a little bit on an API Platform and some lessons we've learned in the federal space doing it. I'm assuming we're in DC. Some people here work for government or contact government and know government is a little different than Silicon Valley in a lot of reasons. So, we're going to talk a little bit about that today.

But who am I? My name is Rob Wilkinson. I'm a developer. I've done dev-ops. I've been a backend developer. I've been a front end developer. I'm the CTO of a company called Oddball. We do a lot of digital services for the federal government, at the VA, at CMS, and at DOD. I really like APIs, and the reason I like APIs is because I used to teach it at coding bootcamp, and there's nothing better than grabbing someone who maybe has been writing code for two weeks and saying, "You can build a Pokédex, and you don't have to understand a database because all you need to understand is JQuery and HTML", and they can ping an API, and they can load data, and they can make magic happen.

My go-to, in case you can't tell, whenever I'm learning a new Framework is to build a Pokédex using the PokéAPI. Okay, if you guys are familiar with that one go to PokeAPI.co. Amazing API. Free, easy to use. Done a lot of dev-ops stuff, and as of January of this year somewhat of a newly minted DC resident. Which the funny story is I came here to this meetup in June, introduced myself to Gray, I said what I was doing. He said, "When are you speaking?" I said, "Well I guess I am now." So took me a little while, but now we're here.

What are we going to talk about today? We're going to talk about an API program, some of the lessons we've learned in putting it into practice, and how to actually do it yourself or recommend to people and stuff like that, so hopefully you can come away with a little bit of a learning.

I broke this down into the most simplest way I could in what an API Portal, an API program within the government, might look like or in commercial as well, and it's exposing data and systems and enabling developers to access them in a documented, secure way. Where you have all of these APIs that you've built, you can't just deploy them in a cloud and say, "Anybody can use them, and we don't ever want to track it, monitor it, log it, or do an insecure way." Right, if you're in the government you might have PII there, PHI, a whole bunch of stuff. So, Documentation and security is important. So there's a few things that I think are really important to do.

You've got to have a value proposition. There's, if you look around government you can find many examples of programs where people have built things and launched them with the idea that, "If we build it, they will come." That's not necessarily the case. Unless people want to use it, they won't use it. Right? So you need to have a value proposition. Whatever API that you're publishing, does somebody actually care about it? Does somebody want to use it? And are they willing to invest time? Consuming APIs and writing code to do it matters and takes time and takes money for those teams. If there's something for them at the end of that tunnel, they'll do it. If there's nothing, they might not. Right? So it's really important to make sure you have a value proposition.

We've seen this a bunch of different ways. Like, we're working with the VA right now, and one of our things is veteran verification. Well, who wants to verify if somebody's a veteran? A lot of businesses who want to offer veteran discounts or maybe they want to do veteran loans, things like that right? And that's been really important to sort of work on those and sort of drive along those channels.

The other thing is track your growth over time. Don't tack on your logs and your sort of growth at the very end. You want to see, "This is where we were. This is where we are now," and make sure that you're taking that into account from the very beginning. So you can prove that this is successful. You want, if you like APIs and you like what they do and the services they provide and you can show that they've grown, hopefully you can get more providers within the platform, and you can get more consumers and have great examples and sort of get this beautiful feedback cycle going. If you don't track that growth or you do it after you've already been running for two years, you might not really have the value that you want.

The other thing is self-service developer accounts. So what does self-service developer account means? What it means is if you go to Square.com and you open up an API account you can within minutes or hours be accessing and pinging their Sandbox API and be doing things with their production one and all sorts of stuff. It's important that we can kind of make that happen on the government sense. Like originally when we first started doing, we had this idea where you would open up a GitHub issue and some stuff, and now we have the ability where you can go to developer.va.gov right now and within minutes or a certain amount of time, you can have an account, be able to ping and access sandbox... not sandbox... but, like test data and stuff like that.

Then the last key component is path to production. Right? If there's no way people using these APIs can ever access production data and they don't see how they can access production data and see a clear, well defined path to that then they're sort of just putting their time, investing their energy using these APIs, but they don't really know that they're going to get to production.

Right? So what's your path to production look like? Well, if you're talking about PII or PHI, maybe you're working with a federal agency, that's not as easy as just flipping a switch, right? You have different things you need to figure out, and you're consuming this. What are you doing with it? I'm giving you someone's birthdate and address, or social security, or whatever it is. So, how do you escalate and allow for consumers to go and use real production data and make sure that can kind of happen? Defining that and being honest with your users and sort of figuring that out is really important as well.

These are four things I think are pretty key here. And once you sort of have an idea of that, you actually kind of have to get it done. These are how you sort of do this. Within any sort of API portal, an API program, you're going to have governance, tech, and marketing. I'll cover some of those in here.

So what is API governance? It's a lot of things, as you can see. It's kind of how things are documented, where that documentation is stored. How do you handle Versioning? How do you handle updates and releases and communicate that to people? How do you name things? What do you use as your consistent sort of unique identifiers? Is it like an integer ID? Is it like a UUID? Is it someone's social security number? Is it their veteran ID or their TIN or their MPI or whatever depending on what you're using within a specific agency? How do you deprecate APIs, right? Once six months have gone, you've got people using your version 1, and now you're on version 2. What do you do? Do you care about them? Do you not? Those are all important things to kind of think about.

Working agreements, like if you're working with a provider that you don't control, it's a downstream service, and they crash in the middle of the night and you have production users that are using it, who do you call? How fast do they fix it? You need to have these sort of incident response and working agreements for these downstream people. And it'll also tell your consumers that so they kind of know that their business is in your hands and they can feel safe about it.

Then understanding principles and learning from other teams and programs that are doing this, like, we couldn't have done it or learned any of the things we have unless we looked at some great examples from the private sector and some really good examples from the federal sector as well.

Just gave a bunch of things, but be careful, because these are not a list of strict and firm rules. Like don't say, "This is exactly how we do versioning," and, "We just started this program a month ago and we're never ever, changing." That's a bad way to make your program grow and do well and get users and stuff like that.

Be flexible with these guidelines and be willing to change them. We found that sometimes we're working with commercial users and they may want something a little different than we currently have. Is it worth it to flex a little and get a big client? That's something you've got to understand as well. And build good relationships. You'll find in the federal space that you'll have users, and your users will be big companies. Why would you not have a relationship with these big companies who are using your APIs? This isn't like armies of anonymous devs who are using this. Who wants access to federal data? It's usually companies that want to use it to improve outcomes and things like that. That's a little about governance.

On the topic of tech, so we were kind of an interesting space when we started our program with the VA called Lighthouse. We were originally on the team that built the APIs, and then we were on the team that built the API portal. So you kind of see at the top the sort of life cycle. We built APIs in 2017. It became va.gov on that team, and then we became the developer.va.gov which sort of is the gateway that empowers third party developers to use those same APIs.

So this is sort of a double-edged sword, right? We know the people who built the APIs, and we're also people building the portal, so we kind of have an inside track. Eventually that kind of changed, but it really was an advantage for us in the beginning to be able to sort of do that and kind of cross that bridge.

Now that we've been doing it for a while, we have a lot of APIs that we proxy to that are like downstream services within the agencies and stuff like that. If you have built those APIs, definitely use that to your advantage, but be open to ones that you haven't worked on and kind of proxy to them as well.

Focus on open source and free tools for flexibility and extensibility. If you're just getting started and you don't know what you're doing, it's probably not a good idea to lock into like a five year enterprise license with a big product. It might not be what you want to do in three years. It's better to be flexible and extensible.

We ended up using Kong's Community Edition. It's great. I highly recommend people who use Kong, it's worked really well. It's an API gateway, and you can sort of define rules and set up it for third-party access and developer management and account management and all that stuff to proxy to RESTful APIs and RPC calls and GRPC and even GraphQL stuff now. So there's a whole bunch of cool stuff happening with that, and it has not led us wrong. A lot of cool stuff happening in that space, too.

Then human centered design. So you build the Developer Portal. Developers are humans, and they're users. We're not robots, and we really spent some time to actually do research, talk to our developers, and see what they wanted to see and how they could navigate things, and what worked, and what didn't. Rather than just like throwing them into Swagger docs. If anyone's familiar with Swagger or OpenAPI or anything like that, how do you navigate and sort of handle support? All that stuff has been interesting as well. So [it's] pretty important to focus on design. You may think it's just an API program, what does design matter? It does. Your developers are humans. They're users.

Status page is pretty important too. Demonstrate and show that you're reliable to your users so they can trust you, put their business in your hands. So, statuspage.io is not an incredibly complicated or difficult thing to do. You can go and set one up pretty easily and kind of make it work.

Then lastly is outreach. You've sort of got governance, you've got your tech approach set, and how do you actually get people to use your API? We've kind of taken a three-prong approach has been pretty good, and I recommend it to people who are maybe getting started. Focus on business development, marketing, and customer success. It's sort of a three-pronged approach. You've got to focus on high profile relationships. If you're in the federal space, there's going to be big companies who want to use these APIs. Make sure that you work with them and have a strong relationship with those sort of channel partners. Marketing, get regular developers to use it too. Whether they're veterans or just coding students that are interested in working with these things, or professors who are interested in health APIs. There's people who want to use APIs for fun. Then take care of your current customers as well. Make sure you get a good MPS from your current customers and they're recommending your APIs and talking about it to people.

One thing I've noticed is accounts versus usage. If you have like square.com, you probably have like a million and a half accounts, and some of those people have used it maybe like a few times or for a couple days. What you'll probably have in the federal space is less accounts, but those accounts use it a whole lot. Right? Because you have one account that a consumer is using, and all of their servers are using that account. So they're really just pounding your API. To be able to track not only your accounts but the usage per account and things like that.

And listen to your users. Originally we had like a customer support flow that was very GitHub focused, and we changed so now we're using Salesforce to manage a lot of our user issues and triage that and things like that to sort of improve their outcomes as well. Because not everyone who uses APIs is like an expert at GitHub. Maybe they're used to other tools and things like that. That's been important as well.

In kind of closing, I have this first one which I like is like, "Focus on a process and stick to it." I think that a lot of people when they're standing up new programs they don't really do enough iterations before they start changing things. Figure out somewhat what works and try it out once or twice. A couple consumers or a couple providers. Don't change everything all at once because then you don't really know what worked and what didn't.

Second one is pretty generic, "Identify pain points and find innovative solutions.", kind of a no brainer there, but listen to your users around the pain points and make that happen.

Third one's an important one. Learn to say ,"no". Not every system is a candidate for your API platform that you want to put out there. If someone has a legacy SOAP service that no one actually cares about or wants to use but they just think it would be cool to put on the API portal, you sort of have to ask yourself, "Is this a real candidate for the program? Is this not? Are third parties going to actually want to care about this?" Or are you just doing it for some other reason?

Then the other thing is that if you're working on this and you're a developer and you're building a program, you're really building a tool for yourself. There's not many cases where you get to do that. So [ask], "Do you x research with other developers with the company or people that you know that are developers?", and kind of see what they think about it. That can kind of give you some great wins as well.

If you want to know more about any of this stuff, we got our newsletter. You can go to developer.va.gov and sort of sign up for that. You can make a user account at developer.va.gov. You can go to my company's site, Oddball. We're hiring full Stack developers and looking for people. So you can come talk to me or any of the people who work with me. And you can follow me on Twitter or GitHub if you want as well. So, yeah, thank you.

Be sure to read the next DC-Area API Meetup article: March's DC-Area API Meetup to Feature Talks on Serverless, Big Parser APIs, and APIs 101 (Part 3)