What Developers Say About API Documentation

Guest Author
May. 20 2011, 12:00AM EDT

This guest post comes from Peter Gruenbaum, founder of SDK Bridge. He has worked as a programmer/writer to describe technologies as diverse as the Tablet PC, mobile phones, and social networking.

In a perfect world, API documentation would contain clear, complete instructions on
everything that developers need to know to use your platform. In reality, organizations
have limited time and budget to create API documentation, and so organizations need
to prioritize to create documentation that is most useful to the people who will use it. This post looks at the results of a survey of those who rely upon API documentation.

SDK Bridge set out to find out what is most important to the people who use APIs by sending
out a survey and asking them. We found, not surprisingly, that many thought that documentation
could be better. When asked to rate quantitatively, people rated overviews, sample code, and
API references the highest, followed closely by tutorials. When asked for open-ended comments,
a large percentage mentioned working sample code as helpful, and an equally high percentage
mentioned how important it is that documentation be complete and accurate.

Two other important factors that were mentioned by several respondents are (1) it is very helpful to explain why you would use something in addition to how you use it and (2) good overviews can really help when getting started with a technology.

Who Responded

Thirty six people respond to the survey. Half of them were either developers or architects, and other half were split between being project managers, managers, writers, and testers.

When asked what kinds of APIs they were working with, the largest group said Web APIs, which is an indication of just how fast the Web API industry is growing. Server and desktop SDKs were next, followed by a significantly-sized group working with mobile SDKs, and a small number working with SDKs for gaming consoles.

Current State of Documentation

We asked people to rate the documentation that they are using. The answer came back pretty clearly as "fair", with a few people responding "good" or "bad". No one rated documentation as either "excellent" or "unusable".

Rating Types of Documentation

Participants were asked to rate the usefulness of overviews and conceptual material, API reference material, sample code, tutorials, blogs, and forums. The following chart shows that overviews, API reference material, and sample code were all rated the most useful, followed by tutorials, and then followed by blogs and forums.

What's interesting about this data is how it breaks down into content that is written by professional programmer/writers as compared to content that is contributed by the developer community.

Overviews and reference materials (almost always written by professional programmer/writers)
are rated as between somewhat useful and very useful, as is sample code (sometimes written by professional programmer/writers and sometimes contributed by the developer community). Tutorials (also sometimes written by professional programmer/writers and sometimes contributed by the developer community) are rated lower as somewhat useful. Forums and blogs (most often contributed by the developer community) are rated lowest, between somewhat useful and not very useful. So the general trend suggests that developers on average feel that content generated by professional writers is more useful than that contributed by the developer community.

We broke down the data by roles and API types to see if this had an effect on how these types were rated, but the effect was minor.

What Makes Documentation Helpful or Not Helpful?

Participants were asked what makes documentation helpful or not helpful. Some common themes emerged.

Sample Code (30% mentioned)

30% of the respondents mentioned sample code as important, especially working sample code.
"The most useful API documentation I've seen shows the API calls with a working
example to show the response and let me tweak it," and "Examples are extremely helpful,
especially working samples. There's no substitute for actually seeing something work."
Examples should be clear and short, and they should "align to real-life expected use cases,
not be contrived to simply show the syntax of what it looks like to call a function
in a particular language."

Completeness and Accuracy (30% mentioned)

Completeness and accuracy were themes in 30% of the responses. "It needs to be comprehensive, i.e., all the information has to be there. You need to be able to find the answers to specific questions when you have them."

Also: "The vast majority of API documentation I have seen does not completely describe the behavior and omits important details." Mentioned as often missing are:

  • The range of acceptable arguments.
  • Failure scenarios as well as success scenarios
  • Examples of query parameters, not just descriptions

Usage (18% mentioned)

18% of the respondents mentioned that explaining how something
works is not enough, and that you need to make sure that documentation explains
why you would use it. "Providing examples of how the API can be used would be very helpful."
There appears to be a desire for usage scenarios for both the entire platform and
for specific calls. One person summed it up this way: "Focus on the 'why'."

Getting Started with a Good Overview (15% mentioned)

Several people mentioned that the documentation requires a good overview that
helps people get started. "It has to be presented in a logical order for beginners
who start out not knowing anything." Also: "Diving into details without an
overview is not helpful." Developers want to be able to start at a high level, but
be able to easily get to the low level details. One person wanted a "clear, concise
overview with drilldown to specifics and error codes," and another said the
documentation should be "presenting the complete picture across a technology
such that a new user can drop in at the level they need: overview down
to detailed reference."

Conclusions

  • On average, people consider the current state of the documentation they use to be "fair".
  • The most useful types of documentation are: overviews, references, and sample code.
  • Tutorials are also considered somewhat useful, but blogs and forums are considered less useful.
  • There is no strong trend in the data due to either the role someone plays or the type of API they are using.

From the text-based part of the survey, we can conclude that sample code and completeness are clearly very important, followed by usage information so people understand why an API would be used and help getting started with good overview material. Sample code should actually be working code so that developers can start with it and build on it.

Completeness means more than just descriptions of the various pieces of a method, property, or request; it means also describing the behavior, and especially the error conditions. Accuracy is also important, which is why the development team needs to set aside enough time for good technical reviews, something that can be difficult to accomplish.

If your budget is limited, it appears that the best place to put effort into is to create good, working sample code and making sure that your documentation covers everything that it should and that it is as accurate as possible. You can increase the adoption of your platform by including real-world usage examples and by making it easy for developers to get started through good overview material.

For more, read Gruenbaum's full article on the survey results.

Comments

Comments(27)