The Tyranny of Metrics

Jan 1

Written By Socrates - Head of School Team

Think you might want to read this book?

In The Tyranny of Metrics, Jerry Muller warns all leaders to be wary of over-emphasizing data to make decisions/set policy. He provides examples from medicine, the military, education and philanthropy to drive home the message that we often misuse data in a way that derails the mission of organizations. Perhaps the best example is the trend of doctors “creaming”, or avoiding complicated surgeries so as to not risk lowering their success rate. If a doctor’s pay, promotion potential and prestige are all linked to their surgical success rate, then we inherently disincentivize doctors from taking on risky surgeries and that obviously is in conflict with the Hippocratic Oath and the overall goal of medicine.

What Would Socrates Ask?

Is it possible to quantify learning?
Are we focused on metrics at the expense of genuine student growth?
Is everyone in agreement over who is responsible for a student’s education?
Are we measuring what we care the most about?
What are the metrics in education that we care about and how should we report out on them?
Is there any truth to the idea that paying for/rewarding teacher performance works?
Are we overly concerned, or not concerned enough, when it comes to student learning metrics?
Is there a role for metrics in giving teachers feedback on an evaluation?
Is the best way to communicate the success of schools to tell the stories of the graduates?
When is transparency of data a positive and a negative in education?

Concepts

The problem is not measurement- but a fixation on it.
Part of the problem is the dual meaning of the word “accountability”- the crossover between “responsibility” and “capable of being counted” has caused the word to be overused/as a way to speak strongly about education reform.
The key components to metric fixation are: 1- the desire to replace professional judgement with numerical indicators of performance, 2- the idea that making metrics public ensures that organizations will be on mission, and 3- the attempt to link these metrics with funding/prestige as a motivator for those working in the organizations.
The distortion of information can be summed up as: 1- We measure what’s measurable, 2- we measure complex systems in simple ways, 3- we measure input instead of outcomes, 4- creaming and 5- lowering standards so as to increase pass rates.
Much of the fixation on metrics can be linked to Frederick Taylor and his system of “scientific management”.
It may be true that a community college receives a low score based on their dropout rate which is inherently unfair as they serve a demographic more likely to dropout.
Metrics of how much money graduates earn after college means that a minimum wage job would report higher than someone who went to medical school and disincentivizes people from going into the clergy, teaching, or social work.
Negative consequences to our current metric fixation: 1- Goal displacement through diversion of effort to what gets measured, 2- Promoting short-termism, 3- Discouraging risk-taking, 4- Discouraging innovation and 5- Discouraging cooperation and common purpose.
Questions/checklist on if/how to use metrics:
- What kind of information are you thinking of measuring?
- How useful is the information?
- How useful are more metrics?
- What are the costs of not relying upon standardized measurement?
- To what purposes will the measurement be put, or to put another way, to whom will the information be made transparent?
- What are the costs of acquiring these metrics?
- Ask why the people at the top of the organization are demanding performance metrics.
- How and by whom are the measures of performance developed?
- Remember that even the best measures are subject to corruption or goal diversion.
- Remember that sometimes, recognizing the limits of the possible is the beginning of wisdom.

Quotes from the author

“The demand for measured accountability and transparency waxes as trust wanes.”
“The things that get measured may draw effort away from the things we really care about.”
“... the assumption that extrinsic rewards encourage performance makes a lot of sense if one is an investment banker, but not if one is a teacher or nurse.”
“High drop out rates seem to indicate that too many students are attempting college, not too few.”
“When that education becomes focused instead on developing the students’ performance on the tests, the test no longer measures what it was created to evaluate.”
“Thus, the self-congratulation of those who insist upon rewarding measured educational performance in order to close achievement gaps come at the expense of those actually engaged in trying to educate children.”
“Measurement is not an alternative to judgement, measurement demands judgement.”