Sketchy Metrics

‘Tis the season. No, not that season. I mean the season when high school seniors and their parents are getting down to business about choosing a college or university. Many of those “you’re accepted” letters have arrived, and next comes the sorting process. If you don’t think higher ed is a competitive marketplace, just spend a little time looking at all the things colleges do to tout their excellence and entice students to enroll. When I was young, only private colleges did much in the way of marketing, but now that states are disinvesting in higher education at a breakneck pace, public universities (if that term still has any meaning) are out there selling themselves as aggressively as anyone.

For some time now, the most popular ranking of colleges and universities has been that published annually by U.S. News and World Reports. On the basis of a set of weighted criteria, the Powers That Be at USNWR tell us where our institutions “rank” against our supposed peers. The U.S. News rankings have been amply criticized by any number of educational authorities, and I won’t rehash the critiques here except to say that if we wish to know whether students have powerful and effective learning experiences at an institution, we will find no answers at all in these rankings. That notwithstanding, many colleges and universities fall all over themselves to move up in the rankings, even adopting policies and processes explicitly conceived to score points in U.S. News.

But all this folderol about rankings and measurement does raise an important question: how do we measure the quality of the teaching and learning that takes place in our institutions? Or should we measure it at all? Or can we measure it? And by the way: what constitutes success in our work with students? Higher-education administrators today find themselves swimming in a vast pool of numbers. We inform, or perhaps mollify, our stakeholders by showing them our “metrics,” by which we mean some plausibly quantifiable measure of our progress toward our goals. [n.b.: I am told by some mathematicians that this is a very poor use of the term “metric,” but it has become the coin of the realm.] The list of these measurements is well-known to those of us in the biz; it includes retention (usually talking about the percentage of first-year students who go on to the second year), persistence (a broader measure of continuing enrollment), graduation rates (four, five and six year!), grade point averages, and so on.

What do these common metrics really measure? After all, we tend to represent them to our constituents as measures of “student success.” How would most of us define student success? The minimal definition would include achieving one’s academic goals, whether that means a degree, preparation for transfer, obtaining a set of skills, or any number of possible ends. A more expansive definition might include deep learning, the ability to think critically and challenge one’s own assumptions and those of others, preparation for engaged citizenship, and so on.

Once we reflect a bit on learning and success, we quickly come to see that the most widely-valued metrics offer, at best, very indirect indicators. And some of them have, at bottom, nothing whatever to do with teaching, learning, or the fulfillment of student goals and aspirations. First-to-second year retention rates tell us only that a student who can “fog the mirror” and has not been suspended for academic or conduct reasons has returned for a second year. Graduation rates tell us that someone has managed to maintain a 2.0 g.p.a. and complete all formal requirements for a degree. A grade point average tells us that a student has maintained a given quantified level of performance in meeting course criteria. None of these measures tells us one whit about whether meaningful, deep, transformative learning took place during a student’s college career. They do not touch upon, except in the vaguest and most indirect manner, any of the central expectations most of us would harbor for high-quality teaching and learning.

Our accustomed metrics for institutional quality are the bluntest of instruments. It is hard to imagine that an automobile company would measure its quality without reference to how well the car performs in relation to direct criteria of automotive quality. Imagine Mercedes Benz arguing for its quality on the basis of how quickly cars move through the manufacturing process, or how much it spends on parts, or how many units are produced per year. But much of higher education—oddly enough in an effort to make itself analogous to business enterprises—moves along quite nicely doing precisely such a thing.

What is the answer? Learning is a qualitative enterprise. A university must demonstrate its success on the basis of a qualitative measurement of learning. Learning can be measured, against a set of clearly articulated outcomes. Can learning be measured perfectly? No; of course not. Anyone who has taught has a healthy respect for the ineffable mystery of teaching and learning. Anyone who has learned knows that frequently the “penny drops” only years, or decades, after the experience. But we must be careful not to lapse into the perfectionist fallacy, namely, the notion that if a thing cannot be done perfectly it should not be done at all.

We should certainly not award quality points to colleges and universities for the mere act of recruiting bright, well-prepared students who can cruise through a college curriculum in a timely manner. To return to Mercedes, imagine them receiving autos 95% assembled from a subcontractor, then adding some chrome and a paint job, then claiming full credit for its successful “manufacturing process.”  This is certainly the ordained path to the top of the U.S. News rankings; look at the top ten national universities and you will find institutions whose “inputs” are so well assembled that they only need a little buffing to get through the process. If we really want to measure our success should insist without compromise on the quality question: what does the learning curve look like for our students? How far do we take them by providing deep, engaged learning experiences? Only when we begin to frame our questions in this way will we generate measurements adequate to the mission of American higher education.