… i.e. KPI in SCRUM.

I was inspired to write this text by a three-hour (recruitment) interview with the Scrum Master. Half of that time we spent on one topic — KPI (Key Performance Indicators). The conversation began with the question:

How to recognize a good Scrum Master from his/her results? How to show to the management which SM is better and which carries out its tasks worse?

My professional life has always been embedded in business and IT. For several years I took turns in the roles of PM and analyst in which measuring progress and drawing indexes was obvious to me from the very first project. From the very beginning of my work in scrum, I have been serious about tracking work and progress and … today I think it was a mistake. I believed the results too much.

In my first project, a detailed progress tracking distracted me from key issues: customer satisfaction surveys and team enthusiasm check. After all, everything looked fine on paper, so why should the client be dissatisfied? It was only thanks to the reaction of a more experienced Scrum Master that I was able to direct the project to the right path.

In the next project, the team accidentally caused an even bigger problem. They gradually changed their approach to estimations, slightly raising them. It was not visible, and even if we started a conversation on this topic, neither the Scrum Master nor the Product Owner has the right to press the team for changing the estimations. As a result, at constant capacity, the team delivered less, more time was devoted to the individual stories, there were fewer bugs, increased predictability, but after some time, customer satisfaction decreased. Note that all KPIs (except for customer satisfaction) were rising or remaining at the same level but the project was going to collapse. Value in a single sprint decreased, but it is natural since the most valuable work was done at the beginning. The only things that changed were the early estimations in the backlog that grew as they were refined and the Burn Down Chart of the product that stretched. Pulling ourselves together came only when the client announced that he was tired, that he had expected the next release to be completed a few weeks earlier, that he could see the team working well, but the results disappointed him. The team sped up, the estimations returned to original, and high quality remained.

I learned my lessons from both situations:

• not to blindly believe KPIs, because they are just numbers,
• the most important thing is customer satisfaction because he decides to pay for increments,
• the charts can only give a clue as to where the problem might be located, but they do not convey specific information.

In this way, I didn’t use to blindly believe in measures depending on:

• velocity (dependent on Product Backlog Item estimation),
• yesterday’s weather (depending on previous PBI estimations),
• capacity (dependent on PBI estimation),
• RPI (depending on identified PBI problems).

Indicators showing the quality of the product depend on time allocated to the PBIs (i.e. defects per release or technical debt). They also can indirectly depend on the estimation of individual PBIs. Of course, they indicate the state of the product, but not the quality of the Scrum Master’s work as such. Thus, the answer to the posed question How to evaluate the work of a Scrum Master based on the results? is Not based on KPIs, because they can show nonsense results.

At the same time, I leave at the top of the list those measures that should never be underestimated, i.e. customer satisfaction, team enthusiasm, and time to learn, meaning the agility of the entire organization, about which I will write another time.