What's Next in API Testing

This will be the last piece in our series on API testing. This time around, we talk about what's coming up in the near future that will change how we test and deliver new APIs.

Architectural Addendums

The current future of APIs and API testing seems to be in technologies that sit on top of existing architectures like REST or Web services. These technologies provide more sophisticated and powerful ways of manipulating data than the traditional REST conventions of GET, POST, PUT, and DELETE.

GraphQL, for example, is a query language for APIs that's neither REST nor RPC in its architectural style. If I'm testing a REST API and want to verify that a piece of data was created, perhaps a new movie in IMDB, I would have to perform a GET using that movie's ID. When the GET completed, I would typically have a blob of JSON, some portion of which would be incorporated into the next GET.

GraphQL on the other hand will typically retrieve the entire "graph" of data in a single call. Relative to RESTful APIs, its newcomer status (it was invented by Facebook) means it will take some time before the various testing tools can test GraphQL endpoints. When RESTful APIs are described using API description specifications like the Open API Specification and RAML, certain testing solutions can auto-generate test harnesses from those descriptions. However, as there currently is no OAS or RAML equivalent specification for GraphQL, there's no way to auto-generate a test suite for GraphQL-based APIs.

gRPC is another architectural style for building APIs that is gaining in popularity. This is based on Remote Procedure Call (RPC) technology. Rather than using a test payload, such as JSON or XML, gRPC relies on protocols such as XHTTP with data that's serialized in binary form for maximum performance. Like GraphQL however, people building APIs using the gRPC architectural style will find a dearth of viable testing tools. Grpcc is available on GitHub as a command line interface tool for gRPC. gRPC also contains built-in tooling for performance benchmarking. Outside of that, teams using gRPC will likely be building their own testing tools.


Description languages like the Open API Specification, RAML, and API Blueprint are changing the landscape of API testing. Traditionally, API testing has been done one call at a time. A tester might have a wiki with some information about required headers, keys, and data types for an endpoint, make a call using cURL, and get an error. After an hour of hunting down a developer and talking, they discover that the wiki was missing a required header, and the data type was incorrect for one of the keys. Description tools take a big step toward solving this problem.

Rather than developers hand writing documentation for each API change, they can now use a tool such as Apiary or Restlet Studio. These are combination API design, specification readers with documentation generation, and test stubbing tools. On the design side, a developer might design an API that conforms to one of the popular API specification languages (OAS, RAML), and then output a description file. Some newer API testing tools will do basic configuration and build a small set of tests based on generated documentation and specification files. If you have an endpoint with POST, PUT, GET, and DELETE functions, an API testing tool might generate a test for each of those functions along with permutations for required or optional data and headers based on the specification. Testing shouldn't end there, of course. You'll want to change data and add tests for specific scenarios that only a person on your team would know are important, but it's a good start.

My colleague, Bob Reselman says about the future of APIs and API testing:

As API's become more prominent on the architectural landscape, it becomes more important than ever to design an API with testing in mind. For example, when specifying the input and output for a POST method on a given endpoint, just declaring the data input/output data as object is sure to cause pain down the line. Test designers have to guess at what structures to create to test your API. Rather, you will do well to specify each attribute of the request and response objects, as well as the datatype of the given attribute therein. Then, in some cases, testers can use automated test generation tools to create the test. Without a detailed specification to the attribute level, testers will incur a good deal of cost not only having figure out the data structures to use in a test, but also the data to assign to the given structure as well.

In short, when designing an API, make sure you design with testing in mind.

The future of API is tooling and architectural overlays that have testing in mind. Some tooling will help make more powerful and targeted queries. Some will help developers unify different APIs onto different platforms so that making a product that works takes less code. Lastly, oher tools will help testers specifically by both providing timely and correct data about the guts of an API and the ability to create something more powerful than a test stub.

What do the testing and monitoring solution providers see in their crystal balls?

ProgrammableWeb asked the founders of many of the solutions that are mentioned in this series what they thought was coming next for API testing and monitoring. Here's what they had to say:

David O'Neill Founder, API Metrics: "APIS are going to get smarter. We envisage the emergence of the Artificially Intelligent Programming Interface as AI is increasingly applied to APIs. For instance, AIPIs will know when something is wrong with their inputs or outputs or overall behavior and will be able to inform their users and try to do something about it."

John Musser, Founder, API Science: "API testing now seen as one core piece of the API lifecycle, all the way from requirements and design, through testing, and through operations and monitoring. It's all about API Ops, the next wave of DevOps.The future is APIs for APIs: APIs to design APIs, APIs for testing APIs, APIs for creating API wrappers like libraries and SDKs, and APIs for API management and monitoring. It's APIs all the way down."

Ole Lensmar, CTO, SmartBear: "I see two major trends in the API testing space; virtualization and shift-left. Virtualization is helping teams evolve individual APIs in complex architectures with many dependencies, both internal and external. Shift-left moves the testing of APIs upstream in the lifecycle; tests are written and managed as code by both developers and testers and are tightly integrated into the CI/CD pipeline."

Patrick Poulin, founder, API Fortress: "Testless testing. Imagine a world where you do not need to create a test. Instead, a platform listens to live API calls, and after a few days or weeks, it has enough information to know what is expected within a 99.9% level of confidence. The evolution from automatic test creation (what we do now) to no test needed will be much bigger than people realize."

Sven Hammer, co-founder Apica: "By 2020 IoT will be a tsunami for the Internet across all industries. Assume one million cars reporting data on speed, connectivity and external environmental factors such as road conditions. In less than a second, 10,000 cars will detect icy road conditions. How long will it take to report back to the surrounding vehicles to avoid crashes? What if this needs to scale to four continents and 10 million cars?

The new challenge then is simultaneously testing small devices and the connected mobile apps and/or IoT platforms. Both tools and the testing paradigm need to change to emulate APIs, and in many aspects, new asynchronous message functionality. Scalability becomes a priority for performance testing."

Justin Rohrman Justin has been a professional software tester in various capacities since 2005. In his current role, Justin is a consulting software tester and writer working with Excelon Development. Outside of work, he is currently serving on the Association For Software Testing Board of Directors as President helping to facilitate and develop various projects such as conferences, the WHOSE skill book, and BBST software testing courses.

Comments (0)