m&ms are not just for eating, it thought me something about establishing SCRUM metrics.

As part of my Operations Management module at MBS, we studied the concept of performance and quality measurement. We played a game whereby a team of four separated into two quality inspectors and two operators. I was an operator. The inspectors were told to develop two measures of product quality. The product? m&ms.

m&m's are not just for eating

The M&Ms were laid out and given the quality metrics: fail any m&m if the 'm' was less than 50% visible and if there exists any visible cracks. I diligently went through each m&m, examining the quality of the printed 'm', its opacity, the shape of the 'm'. I then did a 360 degree check on the m&m checking for uniformity in the shell. This was repeated for my colleague and we each repeated the check for consistency.

In the end I failed 40% of the m&ms. So did my colleague. The quality inspectors only failed 5%.

This example illustrates the difficulty in establishing performance and quality metrics that furthers and organizations goals. As an operator, I wanted to do the best job that I can to the established metrics. I also let pass m&ms that had pinhole defects, but did not fail them as they were not specified in the metrics, although they were clearly defects (still as yummy though). The danger in metrics is that they may create an imbalance between performance and quality, in this case, the number of sellable m&ms and clear defects (e.g. missing 'm').

SCRUM performance and quality metrics

This made me wonder as to the effect of scrum's performance and quality metrics have on a team. Increasing and maintaining the velocity is a performance metric. That's the amount of work a team is able to accomplish during a sprint. If a scrum team's only key metric is velocity, the team could overly focus on cutting code, thereby increasing velocity. The cost would then be defects introduced. The performance metric of velocity would then have to be balanced by the number of defects introduced, this would promote creation of unit tests and developer testing.

The metric of defects introduced would have to be denominated by velocity (defects/velocity) to provide an apple-to-apple metric, whereby a team's velocity increases, but so does the defects introduced as the team is producing more per sprint.

There are a couple of challenges with the defects introduced metrics. Firstly, it is a lagging indicator. Defects introduced would only be found beyond the current sprint, thus would reflect on past work done. Secondly, if there are testers on the SCRUM team, then there is a conflict of interest for testers to report found defects. Perhaps a relevant control metric would be number of test cases executed against the story size (via points).

Unfortunately, defects, unlike m&ms, are not as tasty.