"When a measure becomes a target, it ceases to be a good measure", this adage is known as Goodhart's law. This paper formally investigates this law and proves that it critically depends on the tail distribution of the discrepancy between the true goal and the measure that is optimized. Discrepancies with long-tail distributions favor a Goodhart's law, that is, the optimization of the measure can have a counter-productive effect on the goal. A formal setting is provided to assess Goodhart's law by studying the asymptotic behavior of the correlation between the goal and the measure, as the measure is optimized. Moreover, a distinction is introduced between a weak Goodhart's law, when over-optimizing the metric is useless for the true goal, and a strong Goodhart's law, when over-optimizing the metric is harmful for the true goal. A distinction proven to depend on the tail distribution. The implications of this result are stressed for large-scale decision making and for policies that are (and have to be) based on metrics, and numerous research directions are proposed to better assess the safety of such policies in general, and to the particularly concerning case where these policies are automated with algorithms. Because deployed algorithms modify their environment and create feedback loops that can generate long-tailed discrepancies, the alignment problem - designing the objective functions to be maximized — may be harder than expected, and a mere correlation between a given measure and a goal should not count as a strong argument for the safety of maximizing that measure. The paper therefore calls for investigation into the design of measures that are resilient to Goodhart's law.
Develop algorithms that are resilient to counter-productive over-optimization. Mathematically prove the numerical finding of a “double descent” of the goal, whose expected value first increases, then decreases, and then increases again, under log-normal discrepancy.