Goodhart's law is an adage named after British economist Charles Goodhart, who advanced the idea in a 1975 article on monetary policy in the United Kingdom, Problems of Monetary Management: the U.K. Experience:[1][2]
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.
It was later used to criticize the British Thatcher government for trying to conduct monetary policy on the basis of targets for broad and narrow money.[3]
Generalization by Marilyn Strathern
In a paper published in 1997, anthropologist Marilyn Strathern generalized Goodhart's law beyond statistics and control to evaluation more broadly. The phrase commonly referred to as Goodhart's law comes from Strathern's paper, not from any of Goodhart's writings:
When a measure becomes a target, it ceases to be a good measure.[4]
One way in which this can occur is individuals trying to anticipate the effect of a policy and then taking actions that alter its outcome.[5]
Priority and background
There are numerous concepts related to this idea, at least one of which predates Goodhart's statement of 1975.[6] Notably, Campbell's law likely has precedence, as Jeff Rodamar has argued, since various formulations date to 1969.[7] Other academics had similar insights during this time period. Jerome Ravetz's 1971 book Scientific Knowledge and Its Social Problems[8] also predates Goodhart, though it does not formulate the same law. He discusses how systems in general can be gamed, focuses on cases where the goals of a task are complex, sophisticated, or subtle. In such cases, the persons possessing the skills to execute the tasks properly are instead able to achieve their own goals to the detriment of the assigned tasks. When the goals are instantiated as metrics, this could be seen as equivalent to Goodhart and Campbell's claim.
Shortly after Goodhart's publication, others suggested closely related ideas, including the Lucas critique (1976). As applied in economics, the law is also implicit in the idea of rational expectations, a theory in economics that states that those who are aware of a system of rewards and punishments will optimize their actions within that system to achieve their desired results. For example, if an employee is rewarded by the number of cars sold each month, they will try to sell more cars, even at a loss.
While it originated in the context of market responses, the law has profound implications for the selection of high-level targets in organizations.[2] Jon Danielsson quotes the law as "Any statistical relationship will break down when used for policy purposes", and suggests a corollary to the law for use in financial risk modelling: "A risk model breaks down when used for regulatory purposes."[9] Mario Biagioli has related the concept to consequences of using citation impact measures to estimate the importance of scientific publications:[10]
All metrics of scientific evaluation are bound to be abused. Goodhart's law [...] states that when a feature of the economy is picked as an indicator of the economy, then it inexorably ceases to function as that indicator because people start to game it.
The law is illustrated in the 2018 book The Tyranny of Metrics by Jerry Z. Muller.[11]
See also
- Campbell's law – "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures"
- Cobra effect
- Lucas critique – it is naive to try to predict the effects of a change in economic policy entirely on the basis of relationships observed in historical data
- McNamara fallacy – involves making a decision based solely on quantitative observations (or metrics) and ignoring all others
- Overfitting
- Reflexivity (social theory)
- Reification (fallacy)
References
- ^ Goodhart, Charles (1975). "Problems of Monetary Management: The U.K. Experience". Papers in Monetary Economics. 1. Sydney: Reserve Bank of Australia.
- ^ a b Goodhart, Charles (1975). "Problems of Monetary Management: The U.K. Experience". In Courakis, Anthony S. (ed.). Inflation, Depression, and Economic Policy in the West. Totowa, New Jersey: Barnes and Noble Books (published 1981). p. 116. ISBN 0-389-20144-8.
- ^ Smith, David (1987). The Rise And Fall of Monetarism. London: Penguin.
- ^ Strathern, Marilyn (1997). "'Improving ratings': audit in the British University system". European Review. John Wiley & Sons. 5 (3): 305–321. doi:10.1002/(SICI)1234-981X(199707)5:3<05::AID-EURO184>3.0.CO;2-4.
- ^ Manheim, David; Garrabrant, Scott (2018). "Categorizing Variants of Goodhart's Law". arXiv:1803.04585 [cs.AI].
- ^ Manheim, David (29 September 2016). "Overpowered Metrics Eat Underspecified Goals". ribbonfarm. Retrieved 26 January 2017.
- ^ Rodamar, Jeffery (28 November 2018). "There ought to be a law! Campbell versus Goodhart". Significance. 15 (6): 9. doi:10.1111/j.1740-9713.2018.01205.x.
- ^ Ravetz, Jerome R. (1971). Scientific knowledge and its social problems. New Brunswick, New Jersey: Transaction Publishers. pp. 295–296. ISBN 1-56000-851-2. OCLC 32779931.
- ^ Daníelsson, Jón (July 2002). "The Emperor has no Clothes: Limits to Risk Modelling". Journal of Banking & Finance. 26 (7): 1273–1296. CiteSeerX 10.1.1.27.3392. doi:10.1016/S0378-4266(02)00263-7.
- ^ Biagioli, Mario (12 July 2016). "Watch out for cheats in citation game" (PDF). Nature. 535 (7611): 201. Bibcode:2016Natur.535..201B. doi:10.1038/535201a. PMID 27411599.
- ^ Muller, Jerry Z. (2018). The Tyranny of Metrics. Princeton University Press. ISBN 978-0-691-19126-3.
Further reading
- Chrystal, K. Alec; Mizen, Paul D. (12 November 2001). "Goodhart's Law: Its Origins, Meaning and Implications for Monetary Policy" (PDF). Retrieved 3 July 2020.
- Manheim, David (9 June 2016). "Goodhart's Law and Why Measurement is Hard". ribbonfarm. Retrieved 3 July 2020.
- Malone, Kenny; Gonzalez, Sarah; Horowitz-Ghazi, Alexi; Goldmark, Alex (21 November 2018). "The Laws Of The Office". Planet Money (Podcast). NPR. Retrieved 3 July 2020.