APIs are the critical backbone of many platforms today. Yet, they only get a fraction of the quality assurance (QA) attention a Web site or app gets. There is a hesitance to test upstream, and it is costing companies money and users.
There is a misconception that an API issue would be catastrophic and clear to everyone. Unfortunately, that is not the case. There are countless subtle problems that sneak through the cracks. This is where there is a lack of awareness.
I have compiled four major, but surprisingly subtle, issues we saw in 2016. Since the holiday shopping season is approaching, they are e-commerce focused. With that said, these examples can apply to any company. I have kept the details anonymous to protect the privacy of our customers.
1. NULL Costs $75k Per Day
Most e-commerce platforms have an endpoint that lists all the products they sell. It contains thousands of products, each with unique and important details. Sometimes, one of those details is off and no one catches it.
One company’s product listing endpoint has 50,100 products. Every time it was tested, there were over 3,000 errors. The problem was an object named ‘recipient.’ It is what they use to categorize products, and should be one of 17 options including men, women, babies, etc... In this endpoint thousands of products had ‘NULL’ for the category. That means “nothing” in API speak.
That meant 3,000 items with no category associated. The category option is how the website navigation can find and show you products under Men > Accessories > Scarves. That is thousands in potentially lost sales because of an API that never caused them concern before.
The proper QA of an API means more than monitoring its speed and uptime. Verify the logic of the entire payload to make sure thousands are not being lost for something as simple as a missing category.
2. Cache Me If You Can
API managers are a valuable tool for many successful API programs. One company had a product listing endpoint, similar to the e-commerce company above. The APIs were passing individual tests. We then built an integration test, which emulates the path end users would experience. The results from the listing endpoint were used to inform the subsequent call that dove into the details of each product. On occasion, a product would return with a 404 (not found) error. This was confusing given that we were only looking into the products that the listing endpoint gave us.
After some investigation the culprit was found. It was the API manager. It cached the listing endpoint, which is usually a good practice, but when improperly configured can lead to hours of outdated results.
Merely exercising a singular endpoint is not enough. Always aim for integration testing, which reproduces an entire flow that consumers would experience. Check everything for everything, and you will be able to uncover hard to catch issues such as this one.
3. Out With The New, In With The Old
An API program is built, maintained, and used by different teams. This makes communication among stakeholders pivotal when making any changes. An obvious, but too often, poorly executed fact.
This company’s analytics team noticed a significant problem in sales and asked us for help. Specifically, new earrings being listed on their Web site were not selling, but older listings saw no dip in sales. We reviewed the API and quickly discovered that earrings were being listed as both ‘earrings’ and ‘jewelry.’ We spoke with the API development team and learned that a few weeks prior a developer had made a small improvement to the ‘type’ of products available. This was thrown into a deluge of release notes.
The Web team had a stringent process where the navigation was specifically curated to maximize conversions. This meant the new ‘earring’ type was not built for, and therefore these products were not being listed on the Web site’s navigation.
There are two here. First, API blueprinting and mocking platforms are available today. They can help product teams plan how an API should function. That plan should be stuck to and confirmed against. There are various ways today to evaluate an API against its definition file (Swagger, RAML, API Blueprint, etc...). Use the original plan to confirm it still conforms.
Second, there are no small API changes. Create clear, bulleted notes. Communicate the information in advance, and confirm it has been read and understood by key decision makers.
4. This App Is Terrible
Many companies have an expansive and varied catalog of items they sell. This can make their API unique in that it has to be flexible enough to provide details that can vary greatly between products. This specific company can offer everything from car tires to apples.
One day, a keen-eyed QA noticed something interesting. Every item had “mounting” instructions. It seemed funny but was a huge issue. Every product users searched, once clicked, would contain detailed mounting information. Imagine being a customer in this scenario. The app seems to be failing, and you would hesitate to purchase seemingly buggy items leading to more lost sales.
Choosing the right QA methodology is critical. Due to the varied nature of this company’s API, intelligent tests have to be developed. They should look at the product category and IF the category is a picture frame, then the mounting instructions should exist, if it is a shoe, then it should only contain shoe related information. This seems routine to developers, but is rarely taken advantage of during testing.
APIs power the platforms you depend on, and yet there is often too little effort spent planning and properly implementing a QA strategy. There is a lot of “good enough for now” and “we will look at it next quarter.” Put an hour into your schedule tomorrow morning to do it.
Be the office hero.