The problem is that we want "quality science" (whatever that means!), but we don't know how to quantify that, or indeed really define it. So a quantitative metric will necessarily be measuring some sort of proxy or set of proxies for "quality", and then you will get people optimizing those proxies, not "quality". To the extent that those proxies miss something important, it will be underinvested in.
Unfortunately, I don't have a better proposal, perhaps short of taking the warnings in http://mcadams.posc.mu.edu/ike.htm to heart (the ones that are NOT about the military-industrial complex). Doing that might change the general funding climate sufficiently that the need for deciding "quality" like we do now may simply become less critical.
I disagree, because of https://en.wikipedia.org/wiki/Goodhart%27s_law
The problem is that we want "quality science" (whatever that means!), but we don't know how to quantify that, or indeed really define it. So a quantitative metric will necessarily be measuring some sort of proxy or set of proxies for "quality", and then you will get people optimizing those proxies, not "quality". To the extent that those proxies miss something important, it will be underinvested in.
Unfortunately, I don't have a better proposal, perhaps short of taking the warnings in http://mcadams.posc.mu.edu/ike.htm to heart (the ones that are NOT about the military-industrial complex). Doing that might change the general funding climate sufficiently that the need for deciding "quality" like we do now may simply become less critical.