Metrics are important for Agile teams. It is wiser to focus on a handful of simple balanced metrics rather than hoard of metrics that does not give much value to the team or organization. In fact, it creates more noise.
A good team creates the metrics to understand how it is doing, what it needs to improve and makes the metrics available to the entire organization in the spirit of visibility and transparency. Stakeholders keep an eye on the metrics to see how the team is performing on its commitments. The leaders should review the metrics from time to time. Any question on an odd looking metric should be followed-up with a sense of concern. Their focus should be on helping the team, empowering the team and supporting it to move towards excellence rather than judging, discriminating or criticizing. The Servant Leadership role, such as Scrum Master or an Agile Champion, is needed to bring in effectiveness by understanding the metrics and using the data to help the team grow in effectiveness.
An ideal dashboard should kindle curiosity, emotions and questions. If any stakeholder questions the relevancy of data, the Scrum Master and the team should review the feedback objectively and update the dashboard as appropriate.
While there are many attractive metrics available, care should be taken to choose a few meaningful ones, especially when a new team has just begun its journey.
Here are some considerations to make while deciding your set of metrics.
Value - Are we giving high value to our end users and customers? How can we show or display this on our dashboard?
Predictability - What is the ability to deliver on commitments?
Productivity - Consistently delivering same or more amount of work in the timeboxed period (iteration).
Quality - How free from defects are the products we build?
Sustainable Pace - The teams should continue to work at a sustainable pace, indefinitely.
Happiness (Growth) - Are the people and teams learning, growing and happy?
Let us now dive a little deeper into what metrics can represent the above themes.
Value can only be measured after the product has been released in the market. Did we meet our desired outcomes? If yes, then we got high value from our product. If no, what are we doing about meeting our desired outcomes in the upcoming sprints? Does our current/future work reflect the feedback that will help us reach or move towards our desired outcomes?
Predictability should be measured as a trend. We should measure the last Sprint Predictability and also the Average Predictability from the last 3, 5 and 10 Sprints. This will show the trend. If the trend is consistent or slowly becoming better, it’s a good thing. If it’s going down, or it’s random all the time, that should be of concern to the team and for the leaders to see what the team needs. Maybe the team needs coaching and understanding on how to deliver to the commitment.
It could also be that the product owner and the business side need coaching, as they could be putting too much pressure on the team during the sprint to take on additional or non-committed work after the sprint starts, thereby disrupting the rhythm. The Scrum Master may need coaching if he/she is commanding the team to self-organize, when the team is still in the forming/storming stage and may benefit more from direction than just delegation (at the current time).
Productivity is measured in terms of Velocity. We recommend that as in case of predictability, the team should display its last sprint velocity and also the average velocity for past 3,5 and 10 sprints. Velocity should also be reported as two numbers. Most people are familiar with velocity as the total number of points delivered in a sprint. However, working with many teams over the years, I have come to realize that while the total points delivered per sprint may be close to the committed points, the actual points delivered were not the same as the committed points or the points committed at the beginning of the sprint. This shows that either the planning is not robust or there are impediments in the system or the culture that disrupts the planned work more often than not. As a result, even high performing teams may be exchanging their committed scope to deliver a different set of work than they planned for. True velocity numbers, can uncover a lot of hidden problems in teams and how they work. It could also mean that the backlog items are not being groomed properly. Working with unhealthy and incomplete backlog items, teams often find they are unable to proceed with a story they started, forcing them to exchange the incomplete story with another one in the backlog. Velocity numbers are never to be compared between teams, although a team can benefit from comparing its own velocity between sprints.
Quality: As a team becomes effective and displays Agile Values and Principles in its behaviors (example: collaborating, storming, swarming, cross-functionally diverse, carrying a musketeer’s attitude, etc.), they take pride in their work and that reflects not only in them meeting their commitments but delivering work with a high quality. What this means is that the production defects should continue to decline, and should ideally drop close to zero with time. Metrics such as “Defects found in the past 3, 6 and 10 sprints” should reflect exactly that. “Meantime to resolve defects”, especially P1 and P2, should be minimized, which also shows proactiveness of an Agile team in resolving past defects and creating less technical debt.
Code-complexity is another important metric that can be measured. There are tools that help measure cyclomatic complexity, which actually tells the risk inherent in any build, determining how complex the code is. A team should also display a Sprint Report for the last completed sprint where some of these numbers are displayed in a table format; example total points committed, points added after sprint started, points removed from the sprint, total points completed, actual committed points completed, etc. - give a hoard of information on how the team is doing and uncovers hidden problems that are not visible by just a single number.
It is a good practice for a team to also display their top three or four retrospective improvement items that they identified in the last sprint (retrospective) showing progress for each item (example: completed, in-progress, disowned or debunked). This tells the stakeholders and Agile champions for the team how the team owns its continuous improvement and how dedicated it is towards improving its behaviors and processes.
All the metrics mentioned above reflects on the past team performance. It tells you how much the team completed, how much percentage of work it completes every sprint, quality of work, etc., but the one and the only metric that is forward-looking is the Team Happiness. As Jeff Sutherland explains, this can reflect on how the team will do in the future. The diagram below shows a lower team happiness can impact other metrics negatively (in the near future).
How do you measure team happiness? Happiness is one of the things you cannot measure so easily. You cannot quantify happiness in a very direct manner. It has to be measured by taking inputs from the team, and by observing their actions, behaviors, body-language, communication, etc. This is what the Servant Leader Scrum Master or Agile Champions in the bigger team have to look out for. They have to connect with members one-on-one and with the team as a whole, continuously working towards increasing and sustaining team happiness. While working with my Scrum teams for years, I would ask every team member at the end of retrospectives to write a number (on a flip-chart) from 1-10 indicating how happy they are working as a team. Whenever this number would go below seven, I would take personal care to connect and understand what could be bothering the team or the team members and would work towards bringing the happiness back up organically. I always secured a budget from the leadership to celebrate with the team. This would often result in a small celebration either at the work or outside and would include things like food, drinks, movies, team building outings, etc.
Here is a snapshot of the dashboard.
Conclusion:
You can improve only what you can measure. Meaningful and useful metrics uncover problems that reduce the team effectiveness, and are therefore, needed for continuous improvement.
I recommend starting with the dashboard and metrics explained in this article. The dashboard above also includes the sprint burndown chart and a few other things that I found useful. These metrics are not all readily available from any one given tool. While working with industry standard tools like Jira, Rally and others, we were able to extract a few of the metrics, while others had to be created using third-party tools such as easyBI. Metrics such as retrospective items and Team Happiness were input manually by the Scrum Master.
Amitabh (Amit) Sinha is a servant leader entrepreneur, visionary, mentor, trainer and coach. Amit is highly passionate about Agile, its principles, values, and the human side. Amit is a people champion and strives to bring out the best in his teams. Amit leverages his expertise in Agile, Scrum, Kanban and people skills to increase team effectiveness and happiness. See more
Amit,
Great article. I really liked the idea of team happiness and retrospective items. Securing a budget to celebrate is another good idea but my boss says completing committed work in a sprint is compensated by paycheck. So, why should we have a budget for celebration ? Any thoughts ?
John,
Thanks for your comments. Will it help to explain to your Manager that while what he/she says is not incorrect, it might be useful to see the celebration budget as an investment on happiness rather than an expense?
Amit