I ran into this episode of Science Friday and I loved how the guest, Philosopher C. Thi Nguyen, spoke about metrics. I am not typically a fan of metrics, I feel they often do a disservice to understanding whole stories. This guest explained this nature of metrics as a weakness that gives it its strength – a strength and weakness that go hand-in-hand – which is that metrics allow information to be shared across contexts (strength), but to do that the information has to be decontextualized (weakness):
“…metrics are both incredibly powerful and incredibly limited and that their power and their limitations are part and parcel of their core function. They offer us clear, simple, objective, unbiased counts, but that there’s also an incredible price to that.
One way to put it is that metrics are incredibly good at capturing the kinds of things that we can all count together easily. And I think the kinds of things they miss systematically are the kinds of things that people are going to count differently, that require some kind of discretion, some kind of judgment, some kind of sensitivity….
Theodore Porter puts it this way…qualitative knowledge is rich and sensitive and open ended and dynamic, but it travels really badly between contexts….
So my favorite example, as a professor and a teacher, is the evaluations I write on my students’ essays. But what I write is not going to be comprehensible to a business school professor or a CS professor. And what they write on their students won’t be comprehensible to me. So what Porter says is that when we make quantitative data, we pick a chunk that’s going to be steady and stable across contexts. We find a chunk that everyone can understand, no matter their background, across the institution or across the world.
So in education, that’s letter grades, GPA. And I think you can get a glimpse of what’s going on here by just thinking about how different and how rich a qualitative evaluation is and how hard it is to understand at a distance, and how quick and easy a letter grade is to understand and how thin and simplified it needs to be.
I felt like this was my, like, Matrix, ‘the veil is ripped from my eyes’ moment, was that metrics are powerful because they’re designed to be stable across contexts so we can all understand them and we can all collect into them. And that’s what creates this kind of cross-cutting, massive, shareable piece of information. But to make that, we had to cut out the context. That is actually the design feature and the design bug in one. They are powerful because they’re decontextualized.”
-C. Thi Nguyen