How to Measure the Effectiveness of Your Developer Program

This is the third part of our series How To Build a Strong Developer Community. In part 2, we looked at how to focus on the right developers to achieve your product goals.

Metrics can be a murky area for many developer programs. There is no shortage of things to measure, including (but certainly not limited to) API calls, revenue generated by developers, signups, Web visitors and event attendees. The disconnect tends to occur when programs either aren’t measuring the right things or aren’t making effective use of the metrics they do collect.

The key to developer program metrics is alignment--namely, alignment with business goals and objectives. Developer program metrics should be aligned with these objectives. In short, it’s how the developer program will contribute to the success of the overall business.

Developer program metrics

When properly aligned with business goals, program metrics allow us to:

  • Communicate priorities to and motivate developer program teams, and show them how they contribute to the overall business
  • Measure progress toward goals and objectives
  • Forecast if those goals and objectives can be reached
  • Make improvements to strategy, tactics and resources

Results vs. Activities

These program metrics are then aligned with activity metrics and community metrics.

A common mistake organizations make is focusing program metrics solely on activities--things like number of events participated in, blog posts written, and so on. Program metrics should focus on results and outcomes that contribute to the overall business, such as:

  • Number of API calls or SDK downloads
  • Number of active developers
  • Number of apps/services successfully integrating or using an API
  • Developer satisfaction/net promoter score.

Program metrics reflect our developer program strategy: Is it sound? Does it deliver value to our business?

Activity and community metrics lie beneath program metrics. Activity metrics allow you to track the things you are doing to achieve program goals and metrics, while community metrics track progress in building a developer community. Combined, they show the effectiveness of your tactics, and how they are contributing to the program metrics.

Activity metrics take on several different forms, all focused on the things a program and its team does:

  • Functional metrics around certain job roles, such as promotion or evangelism
  • Deliverable metrics, such as number of support articles written or hackathons attended
  • Developer cycle stage metrics, such as Time To First Hello World or the percentage of apps that pass a certification process

Similarly, there are many potential community metrics that illustrate the size and depth of your developer community and how your program interacts with and supports it. These can include:

  • Metrics that show developer adoption, such as number of mentions on Stack Overflow and number GitHub project starts, or number stars/forks for company-hosted GitHub projects
  • Metrics that show community support, such as percentage of answered/unanswered questions on Stack Overflow

When applied correctly and taken together, your metrics should help you see how your program strategy is working and contributing to the overall business goals (in the program metrics), and whether your tactics are contributing toward your program’s success (in the activity and community metrics).

Keep in mind there is no one specific correct set of metrics for every program; the best fit differs from program to program based on the overall business, its strategy, program resources and more. But every metric should pass the “So what?” test. Ask “So what?” after every metric. What does the number tell us? What does it mean? Does it tell us something we can act on, or show how what’s being measured contributes to our overall success? If you can’t answer these questions, chances are the metric isn’t relevant.

Metrics in Action

Beyond simply showing that your developer program is doing “something,” metrics can be extremely useful in helping to make adjustments to your strategy and tactics.

Take, for example, the chart below. It shows the number of developers who progress through the various stages of a developer program:
Developer program stages
The chart shows the number of developers that enter this process (1,000), and some examples of the metrics that could be used to measure success at each stage. Notice that as the developer travels further through the process, the metrics evolve from activity- or community-driven to higher-level program metrics. This is that alignment we’ve discussed in action.

By having the right metrics in place at each stage, we can also see where we’re losing developers in their journey, or where our program and its tactics need enhancement. The chart below shows some general problems in each stage, and the metrics that can help illustrate them. Problems in developer program stages

Rather than simply shooting in the dark when trying to make adjustments to your program and activities--or just moving to a shotgun approach to maximize the number of developers coming into the funnel and not worrying about how many fall out--this data (and organization of that data) will help you get a better grasp on exactly where and how your program is succeeding and where it needs to improve.

This is part of our series How To Build a Strong Developer Community. In part 4, we will look at how to take control of the onboarding process by minimizing the hurdles developers will face when they decide to move forward with your product.

Be sure to read the next Developer Relations article: 7 Steps to Successful Developer Onboarding