Dangers of Scores in Decision Making -- From James Taylor

James Taylor, who we've referenced on this blog on several occasions, shared his thoughts today on the dangers of scores in decision making. He provides a great example: the Body Mass Index (BMI), which is fraught with issues but is nonetheless used inappropriately by companies and health officials alike.

I've seen the misuse of scores or other numbers in some of the organizations served by Big Sky. When a model is developed that produces a number or a score -- such as with Analytic Hierarchy Process, Statistics that measure strength of relationships, or even simple weighted averages -- there are many who focus too narrowly on the number without context, or fail to realize its limitations. For example, we will often help clients use quantitative methods to prioritize budgets, but the numbers that "pop out" are really nothing more than measures of preference; they are not truth. In many cases, they aren't useful measures outside of the model, and can't even be compared to previous analyses using the same method.

Scores must be balanced with other factors -- which James correctly points out can be codified in rules and often automated. For those decisions that can't be automated because they are inherently more political or are subject to other unpredictable factors, the limitations of "scores" should be carefully and frequently communicated to all stakeholders to avoid confusion or misuse.

Thanks to James for a great post.