Measuring Impact

Goodhart's Law is the principle that when a measure becomes a target, it ceases to be a good measure. We can already see it at work in the existing prestige economy: evaluating scientists by their publication count pushes them to publish a large number of insignificant articles, and the measures used to evaluate grant proposals are similarly imperfect, and are inevitably gamed by good and bad actors alike.

So in crafting a reward system for impactful science, we have to be careful about the metrics we choose. It is impossible to foresee every workaround that will emerge to corrupt a given scheme, but that does not invalidate the use of incentives in the first place. The assumption behind this whole proposal is that it is easier for the government and foundations to identify what science is impactful after it has been published than to identify which research programs will be impactful before they are funded. That is the easier judgment, and the system only has to make it better than the current one predicts.

There are several ways to measure impact, each with its own failure modes.

Scientific review boards

Panels of scientists could identify the most important papers in a field, much like prize-awarding committees but operating at a larger scale. The strength of the approach is judgment, since a panel can recognize a paper that quietly reframed a field. Its weaknesses are corruptibility and scope. Science is vast, the board members have to be chosen by someone, and any standing committee of academics is vulnerable to capture by intellectual fashion and to political litmus tests that have nothing to do with the quality of the work.

Paying for publication in important journals

The government could pay the owners of articles that appear in established journals. But this raises the question of which journals count, it further incentivizes excessive publishing, and it intensifies the corrupting pressure on journals, some of which would resort to pay-to-play.

Paying for citations

Citations are more measurable, and they track per-article impact rather than mere publication. The danger is that paying a flat rate for them rewards meaningless over-citation and circular citation among friends and colleagues, which already happens for prestige alone.

The remedy is to weight a citation by the measured impact of the article doing the citing, so that a citation from an influential paper is worth far more than one from a paper nobody reads. This is roughly how a search engine ranks a page by who links to it. It also changes the economics of fraud, since spinning up a pile of obscure papers to cite your own work accomplishes little when those papers carry almost no weight to confer. To raise an article's reward, you would need citations from work that itself has impact.

Democratic vouchers

Alongside citations, every working scientist in a field could be given a budget to allocate, by secret ballot, to the papers that most influenced their own work, with rewards distributed accordingly. A secret ballot is harder to pressure or trade than a citation printed in a reference list. Citations record which papers were built upon, while vouchers record which papers scientists actually value, and the two can be gamed only in different ways.

A new kind of journal

The most natural home for all of this is a new kind of journal, or an existing one converted to the purpose. Such journals would be free to read and fully transparent about their data and their peer review. Funding would flow to the articles they cite. A journal might reserve five to ten percent of its citation budget to pay peer reviewers, which finally makes reviewing a compensated job and aligns everyone toward publishing work that lasts.

The arithmetic is straightforward. Say a journal in a field the government wants to advance receives $5 million a year and has $1 million in overhead, and that each quarterly issue carries ten articles, so forty a year. That leaves $100,000 of citation reward per article. If an article cites twenty sources, the owners of each receive $5,000. An author who cites carelessly spreads the money thinly, while one who cites the work that genuinely informed theirs rewards it well.

Catching fraud

Introducing a financial incentive on top of the existing incentive to publish influential papers does raise the temptation to cut corners and massage data. But the market can address this too. Much of the bad science that survives does so because there is insufficient financial or reputational incentive to investigate it. A journal could commit to paying anyone who produces evidence of fraud strong enough that the journal retracts the article, with the cost falling on the editors and reviewers who approved it. Investigating fraud in high-value papers would become a viable line of work rather than a thankless one.

Two choices for the law

Two further choices matter enough to fix in the law as goals, while leaving the details to the agency that administers them. The first is that rewards should be pooled and normalized by field, since citation rates differ enormously between, say, cell biology and mathematics, and a single common pool would simply favor the fields that cite most heavily. The second is that these per-field pools also give the government a means of directing effort: by adjusting how much each field's pool holds, it can encourage work in areas of national priority without any official choosing individual projects. Because investors anticipate such adjustments, even a credible announcement that a field will be better funded draws capital toward it ahead of the money itself.

None of these measures is incorruptible, and none should be fixed permanently in statute. The methods should be published openly, so that manipulation can be seen, and revised as new forms of gaming emerge. The aim is not a perfect measure, which is unattainable, but a process for revising imperfect ones before they are corrupted.

Next: From Here to There