Software engineering productivity: How do you measure it?

Happy Research Monday 🔬. In a previous edition, we used research to help defined software engineering productivity. In this edition, we look into an even greater challenge - how do we measure it?

The context:

Today, developer productivity is one of the biggest topics on managers’ minds. In fact, a Forrester study found that 87% of engineering managers rated improving developer productivity as a “high” or “critical” priority, the highest of any process. Managers have the ability to pull data from tools like Github, JIRA, CircleCI and view countless metrics for their team, but this has led to a general sense of overwhelm. In a sea of information, which metric (or combination of metrics) actually matters?

Researchers have been studying this concept for decades. The evidence has become overwhelming that there is no single measure of productivity in software engineering. Rather, a team can measure productivity using a combination of metrics that span both system and perception metrics.

But what is that combination? In 2021, researchers finally found the answer.

The research:

In 2021, researchers at Microsoft published the SPACE framework, a groundbreaking framework that captures the most important dimensions of developer productivity. Since its publishing it has grown in popularity, with consensus across the engineering community that this constellation of metrics adequately captures the tension of measuring productivity in software engineering.

The SPACE acronym stands for:

Satisfaction and well-being

• What is it: Satisfaction is how fulfilled developers feel with their work, team, tools, or culture. Well-being covers how happy they are, and how their work impacts it.
• Example metrics: developer efficacy, perception of burnout, developer satisfaction

Performance

• What is it: The outcome of the work a developer team does. This measure looks at whether the team’s output does what it was meant to do.
• Example metrics: Reliability of system, number of bugs logged, customer satisfaction

Activity

• What is it: The actions or outputs completed in the course of performing work. Activity provides a high-level view of what type of work developers spend their time on.
• Example metrics: Pull request velocity, deployment frequency, on-call participation

Communication and Collaboration

• What is it: How people and teams communicate and work together. This includes information sharing, dependability, transparency and awareness of work, and effectiveness of problem solving.
• Example metrics: Documentation discoverability, onboarding time, pull request cycle time

Efficiency and Flow

• What is it: The ability to complete work or make progress with minimal interruptions or delay.
• Example metrics: lead time for changes, number of interruptions, wait time for code reviews

The application:

This paper provides a great framework for measuring productivity in software engineers, but it’s also fair to say that it takes a few steps to implement. To put this framework to use, here are some useful tips:

1. When implementing this framework, start with a goal - what do you want your team to focus on?
2. Then, pick a combination of metrics that span at least 3 categories of SPACE, mixing both system and perception metrics. Be mindful of how many metrics you pick - too many metrics can lead to confusion and lower motivation.
3. To capture perception metrics, create a developer survey to share out on a regular basis. Treat perception metrics as leading indicators of change, and system metrics as quantifiable evidence.
4. Talk to your team on a regular basis, because these metrics matter to more than just the manager. When productivity is more comprehensively measured, engineers will see it as a team sport to work on together. Conversely, when productivity is not comprehensively measured (ie without perception metrics), this can make improving productivity more of a “top down” initiative and engineers might not feel fairly represented.

As a reminder from our last post, productivity is a measure of outputs (effectiveness) divided by inputs (efficiency). In the SPACE model, effectiveness is related to satisfaction and performance. Efficiency is driven by activity, communication/collaboration, and efficiency/flow.

With a clear definition and framework for developer productivity, your teams are ready to realize their best potential 🎉.

—

To receive weekly editions of research-driven insights, subscribe to our newsletter, Research-Driven Engineering Leadership.