The cost of counting
Few workplaces have embraced measurement as wholeheartedly as the modern research university. A scientist's career now generates a trail of figures: papers published, times those papers are cited, the prestige ranking of the journals that accepted them, and an index that combines several of these into a single number. Administrators find such figures irresistible, and it is easy to see why. Deciding whether a physicist's work on unstable particles is better than a botanist's catalogue of mountain grasses is a task for which almost no one has the expertise, whereas comparing two integers requires none at all.
The difficulty is that a measure adopted as a convenience tends to become a target, and a target soon ceases to measure anything reliably. This principle, familiar to economists for decades, has a predictable effect on research. When promotion depends on the number of papers produced, researchers divide one substantial study into several thin ones; when citations count, they cite colleagues generously and expect the courtesy to be returned. None of this requires dishonesty in any ordinary sense. It is simply what intelligent people do when the rules of a game are announced in advance. The numbers rise; whether knowledge rises with them is another matter.
Consider a hypothetical department that is told its funding will depend on average citations per member. The sensible response is not to improve the research but to reshape the department: to recruit people working in fields where papers attract many citations, and to quietly discourage those whose subjects are small and specialised. Such decisions are made without any ill will and are described in meeting minutes as strategic alignment. The result, over a decade, is a department that scores well and covers less.
Defenders of the system are quick to point out that the alternative is not a golden age of disinterested inquiry but the old patronage of senior professors, in which appointments were decided by reputation, friendship and sometimes prejudice. They have a point. Metrics did open doors for gifted researchers from unfashionable institutions, and a figure at least can be contested in public in a way that a whispered opinion cannot. Yet to concede this is not to concede that the present arrangement is the best available, still less that it should be extended to every corner of academic life.
A more subtle objection concerns what gets measured at all. Slow projects, those that take ten years to yield a single result, are poorly suited to an annual review. So are negative findings, the careful demonstration that a promising treatment does not work, which are valuable to other researchers but unlikely to be cited. Most worrying of all is the effect on the questions scientists choose to ask. A young researcher who knows that a safe, incremental problem will produce three papers in two years, while a bold one may produce none, is not being cowardly in choosing the former. The system has made caution the rational choice, and rational people respond accordingly.
The pressure extends to the journals themselves. Editors know that articles on fashionable subjects attract citations, and that a journal's own ranking depends on them; it is hardly surprising, then, that fashionable subjects fill their pages. Reviewers, too, have been known to favour work that resembles what they have already read. None of this is sinister, but together these habits make a field more conservative than the individuals within it would choose to be.
Some institutions have begun to experiment. One approach asks applicants for promotion to submit only a small number of their best works, accompanied by a short statement of why each matters, so that the panel reads rather than counts. Early reports suggest that panels find the process considerably more time-consuming, and it would be surprising if they did not; reading is slower than arithmetic. Whether overstretched committees will tolerate the burden once the novelty has worn off remains to be seen. The risk is that such schemes become a ritual, in which the statements are written to a formula and read by no one.
It would be comforting to conclude with a neat remedy, but the honest conclusion is less tidy. Any system that rewards people according to visible criteria will be gamed by them, and any system that dispenses with criteria invites favouritism. The sensible aim is not to find an incorruptible measure but to keep measures in their place, treating them as a clue to be weighed alongside judgement rather than a verdict that replaces it. That requires something metrics were invented to avoid: people willing to take responsibility for a decision that cannot be defended by pointing to a number.