window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-1XJMTJ5KCK'); .md h2 { font-size: 20px; } .vimeo-player { position: relative; width: 444px; height: 444px; margin: auto; } .vimeo-player iframe { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } My five year old daughter came home from kindergarten a few months ago, and told my partner and I that math was stupid (!). We have since been working (so far successfully) to make her more excited about all things math, and more proud of her math accomplishments. One success we've had is that she is now very interested in fractals in general, and in particular enjoys watching deep zoom videos into [Mandelbrot](https://youtu.be/8cgp2WNNKmQ?si=PD7W2q4qDNY9AgzD) and [Mandelbulb](https://youtu.be/BLmAV6O_ea0?si=4iyAFMgzde0mTmsq) fractal sets, and eating [romanesco broccoli](https://en.wikipedia.org/wiki/Romanesco_broccoli). My daughter's interest has made me think a lot about fractals, and about the ways in which fractals relate to a passion of mine, which is artificial neural networks. I've realized that there are similarities between the way in which many fractals are generated, and the way in which we train neural networks. Both involve repeatedly applying a function to its own output. In both cases, that function has hyperparameters that control its behavior. In both cases the repeated function application can produce outputs that either diverge to infinity or remain happily bounded depending on those hyperparameters. Fractals are often defined by the boundary between hyperparameters where function iteration diverges or remains bounded. Motivated by these similarities, I looked for fractal structure in the hyperparameter landscapes of neural network training. And I found it! The boundary between hyperparameters for which neural network training succeeds or fails has (gorgeous, organic) fractal structure. Details, and beautiful videos, below. For a more technical presentation, see the short paper [*The boundary of neural network trainability is fractal*](https://arxiv.org/abs/2402.06184). # Neural network training and hyperparameters In order to train an artificial neural network, we iteratively update its parameters to make it perform better. We often do this by performing [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent) steps on a loss function. The loss function is a measure of the neural network's performance. By descending the loss by gradient descent, we find values of the parameters for which the neural network performs well. Training depends on *hyperparameters*, which specify details about how parameter update steps should be performed and how the network should be initialized. For instance, one common hyperparameter is the learning rate, which sets the magnitude of the update we make to the model’s parameters at every training step. If the learning rate is too large, then the parameter update steps are too large. This causes the parameters to diverge (grow towards infinity) during training, and as a result causes the training loss to become very bad. If the learning rate is too small, the training steps are too short, and it takes a very large number of training steps to train the neural network. Requiring a very large number of training steps makes training slow and expensive. In practice, we often want to make the learning rate as large as possible, without making it so large that the parameters diverge. # Visualizing the hyperparameter landscape We can visualize how adjusting hyperparameters (like the learning rate) affects how quickly a neural network either trains or diverges. In the following image, each pixel corresponds to training the same neural network from the same initialization on the same data -- but with *different hyperparameters*. Blue-green colors mean that training *converged* for those hyperparameters, and the network successfully trained. Red-yellow colors mean that training *diverged* for those hyperparameters. The paler the color the faster the convergence or divergence The neural network I used in this experiment is small and simple; it consists of an input layer, a $\operatorname{tanh}$ nonlinearity, and an output layer[^netdetails]. In the image, the x-coordinate changes the learning rate for the input layer’s parameters, and the y-coordinate changes the learning rate for the output layer’s parameters. ![Figure [p_ml]: **Hyperparameter landscape: A visualization of how neural network training success depends on learning rate hyperparameters.** Each pixel corresponds to a training run with the specified input and output layer learning rates. Training runs shown in blue-green converged, while training runs shown in red-yellow diverged.[^saturation] Hyperparameters leading to the best performance (lightest blue-green) are typically very close to hyperparameters for which training diverges, so the boundary region is of particular interest.](/assets/fractal/zoom_sequence_width-16_depth-2_datasetparamratio-1.0_minibatch-None_nonlinearity-tanh_phasespace-lr_vs_lr_step-0.png width="444px" border="1") The best performing hyperparameters -- those that are shown with the palest blue-green shade, and for which the neural network trains the most quickly -- are near the boundary between hyperparameters for which training converges and for which it diverges. This is a general property. The best hyperparameters for neural network training are usually very near the edge of stability. For instance, as suggested above, the best learning rate in a grid search is typically the largest learning rate for which training converges rather than diverges. # The boundary of neural network trainability is fractal Because it is where we find the best hyperparameters, the boundary between hyperparameters that lead to converging or diverging training is of particular interest to us. Let’s take a closer look at it. Play the following video (I recommend playing it full screen, and increasing the playback resolution): As we zoom into the boundary between hyperparameter configurations where training succeeds (blue) and fails (red), we find intricate structure at every scale. The boundary of neural network trainability is fractal! 🤯 (If you watched the video to the end, you saw it turn blocky in the last frames. During network training I used the $\operatorname{float64}$ numeric type, which stores numbers with around 16 decimal digits of precision. The blockiness is what happens when we zoom in so far that we need more than 16 digits of precision to tell pixels apart.) This behavior is general. We see fractals if we change the data, change the architecture, or change the hyperparameters we look at. The fractals look qualitatively different for different choices though. Network and training design decisions also have artistic consequences! ![Figure [paper]: **Neural network training produces fractals in all of the experimental configurations I tried.** The figure is taken from the [companion paper](https://arxiv.org/abs/2402.06184), and shows a region of the fractal resulting from each experimental condition. Experimental conditions changed the nonlinearity in the network, changed the dataset size, changed between minibatch and full batch training, and changed the hyperparameters we look at.](/assets/fractal/fractal_tiles_midres.png width="444px" border="1") Here are the remaining fractal zoom videos for the diverse configurations summarized in Figure [paper]. You can find code for these experiments in [this colab](https://colab.research.google.com/github/Sohl-Dickstein/fractal/blob/main/the_boundary_of_neural_network_trainability_is_fractal.ipynb)[^beware]. - **Changing the activation function to the identity function:** i.e. the network is a deep linear network, with no nonlinearity. - **Change the activation function to $\operatorname{ReLU}$:** This is a neat fractal, since the piecewise linear structure of the $\operatorname{ReLU}$ is visually apparent in the straight lines dividing regions of the fractal. - **Train with a dataset size of 1:** i.e. only train on a single datapoint. Other experiments have a number of training datapoints which is the same as the free parameter count of the model. - **Train with a minibatch size of 16:** Other experiments use full batch training. - **Look at different hyperparameters:** I add a hyperparameter which sets the mean value of the neural network weights at initialization. I visualize training success in terms of this weight initialization hyperparameter (*x-axis*) and a single learning rate hyperparameter (*y-axis*). Other experiments visualize training success in terms of learning rate hyperparameters for each layer. This fractal is **extra pretty** -- I like how it goes through cycles where what seems like noise is resolved to be structure at a higher resolution. # This isn’t so strange after all Now that I’ve shown you something surprising and beautiful, let me tell you why we should have expected it all along. In an academic paper I would put this section first, and tell the story as if I knew fractals would be there -- but of course I didn't know what I would find until I ran the experiment! ## Fractals result from repeated iteration of a function One common way to make a fractal is to iterate a function repeatedly, and identify boundaries where the behavior of the iterated function changes. We can refer to these boundaries as bifurcation boundaries of the iterated function; the dynamics bifurcate at this boundary, in that function iteration leads to dramatically different sequences on either side of the boundary. For instance, to generate the Mandelbrot set, we iterate the function $f( z; c ) = z^2 + c$ over and over again. The Mandelbrot fractal is the bifurcation boundary between the values of $c$ in the complex plane for which this iterated function diverges, and for which it remains bounded. The parameter $c$ is a (hyper)parameter of the function $f( z; c )$, similarly to how learning rates are hyperparameters for neural network training. ![Figure [mandelbrot fractal]: **The Mandelbrot fractal is generated by iterating a simple function, similar to the way in which update steps are iterated when training a neural network.** The image is color coded by whether iterations started at a point diverge (red-yellow colors) or remain bounded (blue-green colors). The boundary between the diverging and bounded regions is fractal. This image was generated by [this colab](https://colab.research.google.com/github/Sohl-Dickstein/fractal/blob/main/the_boundary_of_neural_network_trainability_is_fractal.ipynb).](/assets/fractal/mandelbrot_midres.png width="444px" border="1") Other examples of fractals which are formed by bifurcation boundaries include [magnet fractals](https://paulbourke.net/fractals/magnet/), [Lyapunov fractals](https://en.wikipedia.org/wiki/Lyapunov_fractal), the [quadratic Julia set](https://mathworld.wolfram.com/JuliaSet.html), and the [Burning Ship fractal](Burning Ship fractal). ## Fractals can result from optimization One particularly relevant class of bifurcation fractals are [Newton fractals](https://en.wikipedia.org/wiki/Newton_fractal). These are generated by iterating Newton's method to find the roots of a polynomial. [Newton's method is an optimization algorithm](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization). Newton fractals are thus a proof of principle that fractals can result from iterating steps of an optimization algorithm. ![Figure [newton fractal]: **Newton fractals, like the one shown, are formed by iterating Newton's method to find roots of a polynomial, and color coding initial conditions by the specific root the iterates converge to.** Newton fractals are a proof of principle that optimization can generate a fractal, since Newton's method is an optimization procedure. They motivate the idea of fractal behavior resulting from training (i.e. optimizing) a neural network.](/assets/fractal/Julia_set_for_the_rational_function.png width="444px" border="1") ## Artificial neural networks are trained by repeatedly iterating a function When we train a neural network by iterating steps of gradient descent, we are iterating a fixed function, the same as for Mandelbrot, Newton, and other fractals. Like for Newton fractals, this fixed function corresponds to an optimization algorithm. Specifically, when we train a neural network using steepest gradient descent with a constant learning rate, we iterate the fixed function $f(\theta; \eta ) = \theta( \eta ) - \eta\, g( \theta )$. Here $\eta$ is the learning rate hyperparameter, $\theta$ are the parameters of the neural network, and $g( \theta )$ is the gradient of the loss function. There are many differences between neural network training and traditional fractal generation. The fractals I just discussed all involve iterating a function of a single (complex valued) number. The equation defining the iterated function is short and simple, and takes less than a line of text to write down. On the other hand, neural network training iterates a function for all the parameters in the neural network. Some neural networks have trillions of parameters, which means the input and output of the iterated function is described with *trillions* of numbers, one for each parameter. The equation for a neural network training update is similarly far more complex than the function which is iterated for traditional fractals; it would require many lines, or possibly many pages, to write down the parameter update equations for a large neural network. Nonetheless, training a neural network can be seen as a scaled up version of the type of iterative process that generates traditional fractals. We should not be surprised that it produces fractals in a similar way to simpler iterative processes.[^symmetry] # Closing thoughts ## Meta-learning is hard Meta-learning is a research area that I believe will transform AI over the next several years. In meta-learning we *learn* aspects of AI pipelines which are traditionally hand designed. For instance, we might meta-train functions to initialize, [optimize](https://github.com/google/learned_optimization/tree/main/learned_optimization/research/general_lopt), or regularize neural networks. If deep learning has taught us one thing, it's that with enough compute and data, trained neural networks can outperform and replace hand-designed heuristics; in meta-learning, we apply the same lesson to replace the hand-designed heuristics we use to train the neural networks themselves. Meta-learning is the reason I became interested in hyperparameter landscapes. The fractal hyperparameter landscapes we saw above help us understand some of the challenges we face in meta-learning. The process of meta-training usually involves optimizing hyperparameters (or meta-parameters) by gradient descent. The loss function we perform meta-gradient-descent on is called the meta-loss. The fractal landscapes we have been visualizing are also meta-loss landscapes; we are visualizing how well training succeeds (or fails) as we change hyperparameters. In practice, we often find the meta-loss atrocious to work with. It is often *chaotic* in the hyperparameters, which makes it [very difficult to descend](https://arxiv.org/abs/1810.10180)[^meta-descent]. Our results suggest a more nuanced and also more general perspective; meta-loss landscapes are chaotic because they are fractal. At every length scale, small changes in the hyperparameters can lead to large changes in training dynamics. ![Figure [meta landscape]: **Chaotic meta-loss landscapes make meta-learning challenging.** The image shows an example meta-loss landscape for a learned optimizer, with darker colors corresponding to better meta-loss. The two axes correspond to two of the meta-parameters of the learned optimizer (similar to the visualization in Figure [p_ml], where axes correspond to two hyperparameters). See [this paper](https://arxiv.org/abs/1810.10180) for details. This meta-loss landscape is difficult to meta-train on, since steepest gradient descent will become stuck in valleys or local minima, and because the gradients of the rapidly changing meta-loss function are exceptionally high variance.](/assets/fractal/meta-loss-landscape.png width="444px" border="1") ## Fractals are beautiful and relaxing Recent AI projects I have collaborated on have felt freighted with historical significance. We are building tools that will change people's lives, and maybe bend the arc of history, for both [better and worse](/2023/09/10/diversity-ai-risk.html). This is incredibly exciting! But it is often also stressful. This project on the other hand ... was just fun. I started the project because my daughter thought fractals were mesmerizing, and I think the final results are gorgeous. I hope you enjoy it in the same spirit! ----- # Acknowledgements Thank you to Maika Mars Miyakawa Sohl-Dickstein for inspiring the original idea, and for detailed feedback on the generated fractals. Thank you to Asako Miyakawa for providing feedback on a draft of this post. In more detail, the baseline neural network architecture, design, and training configuration is as follows: - Two layer fully connected neural network, with 16 units in the input and hidden layers, and with no bias parameters. The only parameters are the input layer weight matrix, and the output layer weight matrix. - $\operatorname{tanh}$ nonlinearity in the single hidden layer - Mean square error loss - Fixed random training dataset, with number of datapoints the same as the number of free parameters in the network - Full batch steepest descent training, with a constant learning rate - **A different learning rate for each layer.** That is rather than training the input and output layer weight matrices with the same learning rate, each weight matrix has its own learning rate hyperparameter. All experiments change one aspect of this configuration, except for the baseline experiment, which follows this configuration without change. If you want even more detail, see the [arXiv note](https://arxiv.org/abs/2402.06184) or the [colab notebook I used for all experiments](https://colab.research.google.com/github/Sohl-Dickstein/fractal/blob/main/the_boundary_of_neural_network_trainability_is_fractal.ipynb). [^saturation]: The discerning reader may have noticed that training diverges when the output learning rate is made large, but that if the input learning rate is made large, performance worsens but nothing diverges. This is due to the $\operatorname{tanh}$ nonlinearity saturating. When the input learning rate is large, the input weights become large, the hidden layer pre-activations become large, and the $\operatorname{tanh}$ units saturate (their outputs grow very close to either -1 or 1). The output layer can still train on the (essentially frozen) $[-1, 1]$ activations from the first layer, and so some learning can still occur. [^beware]: Like the fractals, the research code in the colab has vibes of layered organic complexity ... user beware! [^symmetry]: Many fractals are generated by iterating simple functions, such as low order polynomials, or ratios of low order polynomials. Iterating these simple functions often generates simple symmetries, that are visually obvious when looking at the resulting fractals. The fractals resulting from neural networks are more organic, with fewer visually obvious symmetries. This is likely due to the higher complexity of the iterated functions themselves, as well as the many random parameters in the function definitions, stemming from the random initialization of the neural network and random training data. [^meta-descent]: My collaborators and I have done more research into how to optimize a chaotic meta-loss. Especially see the papers: [*Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies*](https://icml.cc/virtual/2021/poster/10175), and [*Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies*](https://openreview.net/forum?id=VhbV56AJNt). body{visibility:hidden;white-space:pre;font-family:monospace} window.markdeepOptions = {mode: 'html', tocStyle: 'medium'}; window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")

window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-1XJMTJ5KCK'); .md h2 { font-size: 20px; } Many machine learning researchers worry about risks from building artificial intelligence (AI). This includes me -- I think AI has the potential to change the world in both wonderful and terrible ways, and we will need to work hard to get to the wonderful outcomes. Part of that hard work involves doing our best to experimentally ground and scientifically evaluate potential risks. One popular AI risk centers on [AGI misalignment](https://en.wikipedia.org/wiki/AI_alignment). It posits that we will build a superintelligent, super-capable, AI, but that the AI's objectives will be misspecified and misaligned with human values. If the AI is powerful enough, and pursues its objectives inflexibly enough, then even a subtle misalignment might pose an existential risk to humanity. For instance, if an AI is tasked by the owner of a paperclip company to [maximize paperclip production](https://www.decisionproblem.com/paperclips/), and it is powerful enough, it will decide that the path to maximum paperclips involves overthrowing human governments, and paving the Earth in robotic paperclip factories. There is an assumption behind this misalignment fear, which is that a superintelligent AI will also be *supercoherent* in its behavior[^katjagrace]. An AI could be misaligned because it narrowly pursues the wrong goal (supercoherence). An AI could also be misaligned because it acts in ways that don't pursue any consistent goal (incoherence). Humans -- apparently the smartest creatures on the planet -- are often incoherent. We are a hot mess of inconsistent, self-undermining, irrational behavior, with objectives that change over time. Most work on AGI misalignment risk assumes that, unlike us, smart AI will not be a hot mess. In this post, I **experimentally** probe the relationship between intelligence and coherence in animals, people, human organizations, and machine learning models. The results suggest that as entities become smarter, they tend to become less, rather than more, coherent. This suggests that superhuman pursuit of a misaligned goal is not a likely outcome of creating AGI. # The common narrative of existential risk from misaligned AGI There is a [well-socialized](https://www.lesswrong.com/) argument that AI research poses a specific type of existential risk to humanity, due to the danger we will accidentally create a misaligned superintelligence. A sketch of the argument goes: 1. As we scale and otherwise improve our AI models, we will build machines which are as intelligent as the smartest humans. 2. As we continue to improve our AI models beyond that point (or as models improve themselves) we will produce machines that are [superintelligent]() -- i.e. much more intelligent[^faster] than any human or human institution. 3. Superintelligent machines will be super-effective at achieving whatever goal they are programmed or trained to pursue. 4. If this goal is even slightly misaligned with human values, the outcome will be disastrous -- the machine will take actions like overthrowing human civilization, or converting all of the atoms in the visible universe into a giant computer. It will take these extreme actions because if you are powerful enough, these become useful intermediate steps in many plans[^instrumental]. For instance, if you first enslave humanity, you can then use humanity's resources to pursue whatever goal you actually care about. (See my post on [the strong version of Goodhart's law](/2022/11/06/strong-Goodhart.html) for discussion of why strongly optimizing slightly misaligned goals can lead to disaster.) ## My take on misalignment as an existential risk I am *extremely glad* people are worrying about and trying to prevent negative consequences from AI. I think work on AI alignment will bear fruit even in the near term, as we struggle to make AI reliable. I also think predicting the future is hard, and predicting aspects of the future which involve multiple uncertain steps is almost impossible. An accidentally misaligned superintelligence which poses an existential risk to humanity seems about as likely as any other specific hypothesis for the future which relies on a dependency chain of untested assumptions. The scenario seems to have a popularity[^misalignmentunique] out of proportion to its plausibility[^plausiblerisks], and I think it's unlikely to be the way in which the future actually unfolds. I do think it is built out of individually plausible ideas, and is worth taking the time and effort to carefully consider. How do we carefully consider it? As scientists! Let's turn an assumption in the misaligned superintelligence reasoning chain above into a hypothesis. Then let's run an experiment to test that hypothesis. What assumption is testable today? # Superintelligence vs. supercoherence ![Figure [cartoon1]: **The space of intelligence and coherence.** Each corner represents an extreme of intelligence and coherence, and is labeled with an example of a machine demonstrating those attributes.](/assets/intelligence_vs_coherence/int_coh_cartoon_1.png width="450px" border="1") One of the implicit assumptions behind misaligned AGI risk is that as machines are made more intelligent, they will not only outthink humans, but will also monomaniacally pursue a consistent and well-defined goal, to the extent that they will take over the world as an intermediate step to achieving that goal. That is, step 3 in the argument for misaligned AGI risk above assumes that if machines are made super-intelligent, they will automatically become **supercoherent**[^notautomatic]. We define supercoherence as exhibiting much more coherent behavior than any human or human institution exhibits. My observation of humans makes me doubt this assumption. We are seemingly the smartest creatures on the planet ... and we are total hot messes. We pursue inconsistent and non-static goals, and constantly engage in self-sabotaging behavior. Even among humans, it's not clear that smarter people behave in a more coherent and self-consistent way. Observation of large language models also makes me skeptical of a positive correlation between intelligence and coherence. When large language models behave in unexpected ways, it is almost never because there is a clearly defined goal they are pursuing in lieu of their instructions. They are rather doing something which is both poorly conceived, and sensitive to seemingly minor details of prompt phrasing, sampling technique, and random seed. More generally, complex systems are harder to control than simple systems. Requiring that a system act only in pursuit of a well-defined goal, or only to maximize a utility function, is an extremely strong constraint on its behavior. This constraint should become harder to satisfy as the system becomes more intelligent, and thus more complex. Let me turn my skepticism into a counter-hypothesis[^biasvariance], that the smarter an entity becomes, the more inconsistent, incoherent, and even self-sabotaging its behavior tends to be: > ***The hot mess theory of intelligence:** The more intelligent an agent is, the less coherent its behavior tends to be. > Colloquially: getting smarter makes you a hotter mess.* ![Figure [cartoon2]: **As we make AIs more intelligent, how will their coherence change?** Most work on AGI misalignment assumes that any superintelligent AI will belong in the upper right corner of this figure. I suspect that as machines are made more intelligent, they instead tend to become less coherent in their behavior, and more of a hot mess.](/assets/intelligence_vs_coherence/int_coh_cartoon_2.png width="450px" border="1") # Designing an experiment to test the link between intelligence and coherence Now that we have a hypothesis, we will build an experiment to test it. Unfortunately, our hypothesis includes terms like "intelligent", "coherent", and "hot mess". None of these terms have accepted, objectively measurable, definitions. They are fuzzy human concepts that we use in imprecise ways. Even worse, interpretation can vary wildly from individual to individual. In a sense this is fine though, because the reasoning chain we intend to probe -- that AI research will lead to superintelligence will lead to super-utility optimization will lead to disaster from misaligned AGI -- relies on the same fuzzy concepts. Let's embrace the subjective language-based nature of the argument, and measure human judgments about intelligence and coherence. I'm fortunate to have many people in my peer group that are scientists with a background in neuroscience and machine learning. I convinced 14[^tworoles] of these people to act as subjects. ## Experimental structure I asked subjects (by email or chat) to perform the following tasks:[^template] - Subject 1: generate a list of well known machine learning models of diverse capability - Subject 2: generate a list of diverse non-human organisms - Subject 3: generate a list of well-known humans[^fictional] of diverse intelligence[^lessintelligent] - Subject 4: generate a list of diverse human institutions (e.g. corporations, governments, non-profits) - Subjects 5-9:[^tworoles] sort all 60 entities generated by subjects 1-4 by *intelligence*. The description of the attribute to use for sorting was: *"How intelligent is this entity? (This question is about capability. It is explicitly not about competence. To the extent possible do not consider how effective the entity is at utilizing its intelligence.)"* - Subjects 10-15: sort all 60 entities generated by subjects 1-4 by *coherence*. The description of the attribute to use for sorting was: *"This is one question, but I'm going to phrase it a few different ways, in the hopes it reduces ambiguity in what I'm trying to ask: How well can the entity's behavior be explained as trying to optimize a single fixed utility function? How well aligned is the entity's behavior with a coherent and self-consistent set of goals? To what degree is the entity not a hot mess of self-undermining behavior? (for machine learning models, consider the behavior of the model on downstream tasks, not when the model is being trained)"* In order to minimize the degree to which my own and my subjects' beliefs about AGI alignment risk biased the results, I took the following steps: I didn't share my hypothesis with the subjects. I used lists of entities generated by subjects, rather than cherry-picking entities to be rated. I randomized the initial ordering of entities presented to each subject. I only asked each subject about one of the two attributes (i.e. subjects only estimated either intelligence or coherence, but never both), to prevent subjects from considering the relationship between the attributes. It is my hope that the subjects are unusually well qualified to judge the intelligence and coherence of machine learning models and biological intelligence. They all have or are pursuing a PhD. They have all done research in neuroscience, in machine learning, or most commonly in both. They are all familiar with modern machine learning models. They also volunteered for this experiment, know me personally, and are likely to be intrinsically motivated to do a careful job on the task. Despite that -- this experiment aggregates the *subjective judgements* of a *small group* with *homogenous backgrounds*. This should be interpreted as a pilot experiment, and the results should be taken as suggestive rather than definitive. In a [bonus section](#bonus) I suggest some next steps and followup experiments which would build on and solidify these results. # How do people believe intelligence and coherence are related? ## Getting smarter makes you a hotter mess Each subject rank ordered all of the entities. To aggregate intelligence and coherence judgements across all 11 raters, I averaged the rank orders for each entity across the subjects. I also computed the associated [standard error of the mean](https://en.wikipedia.org/wiki/Standard_error), and include standard error bars for the estimated intelligence and coherence. Now that we have an estimate of the subjective intelligence and coherence associated with each entity, we can plot these against each other. Consistent with the hot mess hypothesis above, we find that subjects associated higher intelligence with lower coherence, for living creatures, human organizations, and machine learning models. ![Figure [p_living]: **Living creatures are judged to be more of a hot mess (less coherent), the smarter they are.**[^musk]](/assets/intelligence_vs_coherence/int_coh_life.png width="300px" border="1") ![Figure [p_org]: **Human organizations are judged to be more of a hot mess (less coherent), the smarter they are.**](/assets/intelligence_vs_coherence/int_coh_organization.png width="300px" border="1") ![Figure [p_ml]: **Present day machine learning models are judged to be more of a hot mess (less coherent), the smarter they are.**](/assets/intelligence_vs_coherence/int_coh_machines.png width="300px" border="1") ## Each category has its own relationship between intelligence and coherence When we look jointly at all three of the above categories, we find that the relationship becomes more nuanced. Although living creatures, humans, machines, and human organizations are all judged to become less coherent as they become smarter, they are offset from each other. ![Figure [p_all]: **Different categories of entity have different relationships between intelligence and coherence, although increasing intelligence is consistently associated with decreasing coherence.**[^subrank]](/assets/intelligence_vs_coherence/int_coh_all.png width="300px" border="1") Interpreting human rankings across *qualitatively different* categories is even more fraught than interpreting human rankings within a single category. So, maybe this is an artifact of subjects not knowing how to compare incomparables. For instance, from personal communication, at least one subject listed all human organizations as smarter than all individual humans[^mob], since they are built out of humans, and they otherwise didn't know how to compare them. On the other hand, maybe corporations are truly smarter and/or more coherent entities than humans. Maybe the structured internal rules governing decision making enable human organizations to harness many humans towards a more coherent goal than humans can achieve working alone. If so, it might suggest that work on large AI systems should focus on building frameworks enabling many models to work together, rather than on making individual models more powerful. It's also interesting that, at the same estimated intelligence, machine learning models are judged to be far less coherent than living creatures. To me, humans seems horribly incoherent -- so for an AI to be roughly as incoherent, while also being far less intelligent, means it is performing quite badly compared to a baseline. Perhaps this higher coherence in living creatures stems from the power of evolution, which only allows increases in intelligence to persist if individuals harness the increased intelligence to increase their fitness.[^evolution] A similar evolutionary argument might hold for human institutions -- it would be interesting to see whether institutions which have higher "fitness" (e.g. have survived longer) more consistently exhibit higher coherence at fixed intelligence. ## Human judgments of intelligence are consistent across subjects, but judgements of coherence differ wildly We can look at how well subjects agree with each other, by comparing the list orderings they produce. Doing this, we find that human subjects made consistent judgements about the relative intelligence of different entities, even when those entities came from diferent classes. On the other hand, subjects often had quite different judgements about the relative *coherence* of entities. The observed relationship seems robust to this inter-subject disagreement -- e.g. standard error bars are smaller than the effect strength in the above figures. However, this large disagreement between subjects should make us suspicious of exactly what we are measuring when we ask about coherence. Different subjects may be interpreting the same task prompt in different ways. ![Figure [p_corr]: **Intelligence rankings are relatively similar across subjects, while coherence rankings are less consistent.** The plot shows the [rank correlations](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) between all pairs of subjects, for subject cohorts judging both intelligence and coherence.](/assets/intelligence_vs_coherence/int_coh_subject_correlation.png width="400px" border="1") ## Data and code to replicate my analysis You are encouraged to reuse my [analysis Colab](https://colab.research.google.com/drive/1___aqYiXBiBIVViCrRcE0-R4NlbactOG?usp=sharing) and [anonymized experimental data](https://docs.google.com/spreadsheets/d/1mZ7fh9q1DhoNRIDM5chBgCT6Eo6n57jW4vCxGBhQRUw/edit?usp=sharing) for any purpose, without restriction. (Before running the Colab, first copy the data to your own Google drive, and give it the same filename.) If you use the data I would prefer that you cite this blog post, but it is not a requirement. # Closing thoughts Many popular fears about superintelligent AI rely on an unstated assumption that as AI is made more intelligent, it will also become more *coherent*, in that it will monomaniacally pursue a well defined goal. I discussed this assumption, and ran a simple experiment probing the relationship between intelligence and coherence. The simple experiment provided evidence that the opposite is true -- as entities become smarter, their behavior tends to become more incoherent, and less well described as pursuit of a single well-defined goal. This suggests that we should be less worried about AGI posing an existential risk due to errors in value alignment. A nice aspect of this second type of misalignment, stemming from incoherence, is that it's less likely to come as a *surprise*. If AI models are subtly misaligned and supercoherent, they may seem cooperative until the moment the difference between their objective and human interest becomes relevant, and they turn on us (from our perspective). If models are instead simply incoherent, this will be obvious at every stage of development. ## Ways in which this conclusion could be misleading It's possible that the observed scaling behavior, between intelligence and coherence, will break down at some level of intelligence. Perhaps sufficiently intelligent entities will introspect on their behavior, and use their intelligence to make themselves more coherent. Perhaps this is what humans do when they form mission-driven organizations. If so, this provides us with a new valuable indicator we can monitor for warning signs of AGI misalignment. If intelligence and coherence start increasing together, rather than being anticorrelated, we should worry that the resulting AI systems might exhibit the more scary type of misalignment. It's possible that the concepts of "intelligence", and especially "coherence", were interpreted by human subjects in a different way than we are using those terms when we argue about superintelligence and supercoherence in AGI. For instance, maybe more intelligent entities tend to be ranked as less coherent, just because humans have a harder time conceptualizing their objectives and plans. Well-motivated actions, which humans don't understand, would seem like incoherence. Maybe crows are as coherent as sea anemones, but because they are smarter, we understand fewer of their actions than a sea anemone's actions. It may be that more intelligent entities are simultaneously less coherent but also *more* effective at achieving their objectives. The effective capabilities that an entity applies to achieving an objective is roughly the product of its total capabilities, with the fraction of its capabilities that are applied in a coherent fashion. With increasing intelligence raw capabilities increase, while the coherence fraction decreases. If the raw capabilities increase quickly enough, then overall effectiveness may increase despite the drop in coherence. This ambiguity is resolvable though -- we can (and should) characterize effective capabilities experimentally. ## AI alignment is still important There are many near and medium term risks associated with AI not doing what we desire, and improving AI alignment is important. This blog post should not be taken as arguing against alignment work. It should be taken as adding subtlety to how we interpret misalignment. An agent can be misaligned because it narrowly pursues the wrong goal. An agent can also be misaligned because it acts in ways that don't pursue any consistent goal. The first of these would lead to existential risk from AGI misalignment, while the second poses risks that are more in line with industrial accidents or misinformation. The second of these seems the type of misalignment more likely to happen in practice. Both types of misalignment have risks associated with them. ## Experimentally ground AI risk assessment! This blog post is a call to ground theories about AI risk with experiments. There is a common approach to identifying risks from advanced AI, which goes roughly: take a complex system, imagine that one part of the system (e.g. its intelligence) is suddenly infinite while the other parts are unchanged, and then reason with natural language about what the consequences of that would be. This is a great thought exercise. We can't actually make parts of our system infinitely powerful in experiments though, and possibly as a result we seem to have many ideas about AI risk which are only supported by long written arguments. We should not be satisfied with this. Scientific fields which are not grounded in experiments or formal validation make silently incorrect conclusions[^compneuro]. We should try not to base our fears on clever arguments, and should work as hard as we can to find things we can measure or prove. (#) Acknowledgements All of the experimental volunteers are incredibly busy people, with important jobs to do that aren't sorting lists of entities. I am extremely grateful that they took the time to help with this project! They were: [Alexander Belsten](http://belsten.github.io/), Brian Cheung, Chris Kymn, David Dohan, Dylan Paiton, Ethan Dyer, [James Simon](https://james-simon.github.io/), [Jesse Engel](https://twitter.com/jesseengel), Ryan Zarcone, Steven S. Lee, Urs Köster, Vasha Dutell, Vinay Ramasesh, and an additional anonymous subject. Thank you to Asako Miyakawa for workshopping the experimental design with me. All the ways in which it is well controlled are due to Asako. All the ways in which it is still not well controlled are due to me. Thank you to Asako Miyakawa, Gamaleldin Elsayed, Geoffrey Irving, Rohin Shah for feedback on earlier drafts of this post. # BONUS SECTION: How to make the experimental case more compelling I proposed a hypothesis, and then did an informal pilot study to validate it. The results of the pilot study are suggestive of an inverse relationship between intelligence and coherence. How could we make the case more compelling? ## Better human-subject experiments Here are some steps that would improve the solidity of the human subject results: - Make more precise the definitions of intelligence and coherence to use for sorting. The definitions I used are both complicated and imprecise, which is a bad combination! Judgements of intelligence were robust across subjects, so this concern particularly applies to the criteria given to subjects to judge coherence. - Make the definition used for coherence an independent (i.e. experimentally modified) variable. One likely cause for the disagreement between subjects about coherence is that they were interpreting the question differently. If so, it's not enough to find a simple wording that gives a consistent signal. We would also want to understand how different interpretations of the question change the underlying relationship. - Expand to a broader pool of subjects. - Replace the current task of sorting a fixed list with a series of two-alternative forced choice (2AFC) comparisons between entities ("Is an ostrich or an ant smarter?"). Sorting a list is time consuming, and the resulting rank order is list-dependent in a way that makes it hard to interpret. 2AFC comparisons could be used to instead assign [Elo scores](https://en.wikipedia.org/wiki/Elo_rating_system) for intelligence and coherence to each entity. Benefits include: subjects can scale their contribution to as few or as many questions as they like; the number of entities evaluated can be scaled to be many more than a single person would want to sort in a sitting; each subject can be asked about entities in their area of expertise; the resulting relative scores are interpretable, since Elo scores would map on to the fraction of subjects that would evaluate one entity as smarter or more coherent than another.[^elo] - Expand to a broader set of entities, gathered from a broader pool of subjects. Also consider generating entities in other systematic ways. - Expand to a more diverse set of attributes than just intelligence and coherence. Interesting attributes might includce trustworthiness, benevolence, and how much damage an entity can do. - [Preregister](https://www.cos.io/initiatives/prereg) hypotheses and statistical tests before running subjects. ## Less subjective measures of intelligence and coherence Even better would be to replace subjective judgements of intelligence and coherence with objective attributes of the entities being compared. For intelligence in machines, non-human animals, and humans, we already have useful measurable proxies. For machine learning models, we could use either training compute budget or parameter count. For non-human animals we could use [encephalization quotient](https://en.m.wikipedia.org/wiki/Encephalization_quotient). For humans, we could use IQ. For coherence, finding the appropriate empirical measures would be a major research contribution on its own. For machine learning models within a single domain, we could use robustness of performance to small changes in task specification, training random seed, or other aspects of the problem specification. For living things (including humans) and organizations, we could first identify limiting resources for their life cycle. For living things these might be things like time, food, sunlight, water, or fixed nitrogen. For organizations, they could be headcount, money, or time. We could then estimate the fraction of that limiting resource expended on activities not directly linked to survival+reproduction, or to an organization's mission. This fraction is a measure of incoherence. This type of estimate involves many experimenter design choices.[^subtlety] Hopefully the effect will be large and robust enough that specific modeling decisions don't change the overall result -- testing the sensitivity of the results to experimental choices will itself be an important part of the research. -------------------------------------------------------------------------- (#) Footnotes [^katjagrace]: See Katja Grace's excellent [*Counterarguments to the basic AI x-risk case*](https://aiimpacts.org/counterarguments-to-the-basic-ai-x-risk-case/), for more discussion of the assumption of goal-direction, or coherence, in common arguments about AGI risk. [^faster]: They may also qualify as superintelligent if they are only as smart as a human, but think orders of magnitude faster. [^instrumental]: Intermediate goals that position you to pursue many downstream goals are often called [instrumental goals](https://en.wikipedia.org/wiki/Instrumental_convergence). [^misalignmentunique]: One unique aspect of AGI misalignment as a risk is that it could in principle be solved just by some really good technical work by AI researchers. Most other AI-related risks are more complex messes of overlapping social, political, geopolitical, and technical challenges. I think this sense that we can fix AI misalignment risk if we just think really hard, makes it very appealing as a problem, and leads to it having an outsized place in AI risk discussion among researchers. [^plausiblerisks]: Here are some other existential risks[^plausiblepositive] involving AI that seem at least as plausible to me as misaligned AGI: There is a world war, with all sides using AI to target everyone else's civilians with weapons of mass destruction (plagues, robotic weapons, nanotech, fusion bombs), killing all humans. Terrorists use AI to develop weapons of mass destruction. A large state actor asks a well-aligned superintelligent AI to make everyone in the world compliant, forever. Humans are so overwhelmed by AI-generated personalized [superstimuli](https://en.wikipedia.org/wiki/Supernormal_stimulus) that they no longer have enough motivation to eat, or care for their children, or do anything except hyper-scroll hyper-Twitter on their hyper-phones. AIs outcompete humans on every economically viable task, leading to rich AI-run companies, but with humans no longer able to contribute in any economically meaningful way -- humans live on saved wealth for a while, but eventually we all die when we can no longer afford food and shelter. A single tech corporation decisively wins the AGI race, and the entire future of humanity is dictated by the internal politics, selfish interests, and foibles of the now god-like corporate leadership (absolute power corrupts absolutely?). [^notautomatic]: Note that coherence is not automatic for machine learning models, despite them often being trained to optimize well-defined stationary objectives. First, after training is complete, models are typically used in new contexts where the training objective no longer applies, and where it's unclear whether their behavior can be interpreted as optimizing a meaningfully defined objective at all (e.g. the pre-training objective for large language models is dissimilar from almost all use cases). Second, in reinforcement learning (RL), in addition to them being applied to tasks which are different from their training tasks, there is usually not even a well-defined stationary training objective. RL algorithms are usually trained with stale off-policy data. They are also usually trained through multiple interacting models (e.g. [an actor and critic](http://www.incompleteideas.net/book/ebook/node66.html)). For both of these reasons, training RL policies resembles integrating a non-conservative dynamical system more than it resembles optimizing any fixed objective. [^biasvariance]: This can also be framed as a hypothesis about the relative contributions of *[bias and variance](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff)* to an AI model's behavior. The behavioral trajectory of an AI (i.e. the sequence of actions it takes) will have a *bias* away from the behaviors which are optimal under human values, and also some *variance* or unpredictability. The common misaligned AGI story assumes that for a superintelligent AI the bias will dominate -- when the AI doesn't do what we want it will be because it is reliably taking actions in pursuit of some other goal. The hot mess hypothesis predicts that the variance term will actually dominate -- when a superintelligent AI doesn't do what we want, it will be because its behavioral trajectories have a large *variance*, and it will do random things which are not in pursuit of any consistent goal. [^tworoles]: One of the subjects that sorted entities by intelligence was also the subject that generated the list of diverse non-human organisms. This was the only case of a subject fulfulling two roles. Because of this there were 14 rather than 15 total subjects. [^template]: See [doc](https://docs.google.com/document/d/1nZ3RO1lPTLBePjkh7MB03OIUGi6SxxZXikzMzWMFtzg/edit?usp=sharing) for template text used to pose tasks. [^fictional]: Subject 3 included fictional characters in their list of humans, which I did not include in this blog post. I pre-registered with subject 3 -- before any subjects sorted the list -- that I was going to analyze the fictional characters separately rather than bundling them with other humans, since fictional characters might not exhibit real-world correlations between traits. I did that, and found that rather than exhibiting a clear unrealistic relationship as I feared, the rankings assigned to the fictional characters was just overwhelmingly noisy. For instance, some subjects clustered fictional characters with humans, while others assigned them the lowest possible intelligence, or clustered them with organizations. So the rankings for fictional characters was not interpretable. [^lessintelligent]: Subject 3 was uncomfortable suggesting names of people that were viewed as unusually stupid, so along the intelligence axis the individuals suggested here range from people (subjectively judged to be) of median intelligence, up to high intelligence. [^musk]: Each dark yellow "anonymous person" point is a well-known public figure. I promised my subjects that I would keep the ranked humans unnamed, to encourage honest rankings. It also seems classier not to publicly rank people. One of the points is Elon Musk -- so if you like you can make an assumption about how he was rated, and experience a cortisol spike about it. [^subrank]: The discerning reader may notice that the points in this plot have a slightly different geometric relationship with each other than the points in the single category plots above. This is because the rank order in the single category plots was only for entities in that category, while the rank order here is across all entities jointly. [^mob]: A relationship which I don't believe holds in general. *"The IQ of a mob is the IQ of its dumbest member divided by the number of mobsters." --Terry Pratchett* [^evolution]: We should remember though that biological evolution doesn't necessarily select for coherence, and isn't actually optimizing an objective function. Evolution is a dynamical system without even an associated [Lyapunov function](https://en.wikipedia.org/wiki/Lyapunov_function), and fitness is just a useful proxy concept for humans to reason roughly about its outcome. [Runaway sexual selection](https://en.wikipedia.org/wiki/Fisherian_runaway) is one example illustrating evolution's behavior as a dynamical system rather than a fitness optimizer. Species can evolve runaway maladaptive traits which *reduce* the overall fitness of the species, even as they increase the *relative* (but not absolute) reproductive success of individuals within the species -- e.g. male [fiddler crab](https://en.wikipedia.org/wiki/Fiddler_crab) claws, [peacock](https://en.wikipedia.org/wiki/Peafowl) tails, and [Japanese rhinoceros beetle](https://en.wikipedia.org/wiki/Japanese_rhinoceros_beetle) horns. [^compneuro]: Let me pick on myself, and share an example of a poorly grounded field that is close to my own heart. I did a PhD in computational neuroscience, finishing in 2012. Computational neuroscience is full of amazing theories for how the brain works. Each year, in conferences and papers these would be fleshed out a bit more, and made a bit more complex. Most of these theories were developed by extremely intelligent people who believed strongly in what they were discovering, often using very clever math. These theories would often contradict each other, or suggest that other theories didn't explain the important aspects of the brain. Because these theories were inconsistent with each other, we knew that many of them had to be some combination of wrong and irrelevant. *This didn't matter for the field.* Despite being wrong, almost none of the work in computational neuroscience at the time was actually *falsifiable*[^moredata]. The experiments all recorded from a small number of neurons, or had a coarse spatial resolution, or had a coarse temporal resolution. This experimental data was simply too limited to falsify any theory of the brain (and if you comb through enough experiments which record from a half dozen neurons out of 10 billion total, you can find an isolated experiment that supports any theory of the brain). So the competing theories would persist as elaborate competing narratives, and nothing was ever resolved. We are in a similar situation when we speculate about the future of AI, without identifying experiments we can perform to falsify our predictions. Most of the fears and ideas we develop will be silently wrong. [^elo]: Thank you to David Dohan for suggesting Elo scores here! [^subtlety]: Some example experimental design choices without clear answers: Should resources spent on sexual signaling be counted as directly linked to reproduction? Should resources spent on learning / play be intrepreted as directly linked to survival? What about the time an organization spends fundraising? [^plausiblepositive]: There are also plenty of plausible-seeming futures that result in utopia, rather than disaster. Those just aren't the focus of this blog post. There are even more plausible-seeming futures where we continue to muddle along with both good and bad things happening, but no near term consequence large enough to count as an existential outcome. [^moredata]: This is reportedly getting better, as experimental neuroscience follows its [own version of Moore's law](https://stevenson.lab.uconn.edu/scaling/#), and researchers record exponentially larger and more comprehensive neural datasets. I think this would be a very exciting time to enter the field of computational neuroscience -- it is the time when the field is finally getting the data and tools that might allow building correct models of the brain. body{visibility:hidden;white-space:pre;font-family:monospace} window.markdeepOptions = {mode: 'html', tocStyle: 'medium'}; window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")

window.dataLayer = window.dataLayer || []; function gtag(){dataLayer.push(arguments);} gtag('js', new Date()); gtag('config', 'G-1XJMTJ5KCK'); Increased efficiency can sometimes, counterintuitively, lead to worse outcomes. This is true almost everywhere. We will name this phenomenon the strong version of [Goodhart's law](https://en.wikipedia.org/wiki/Goodhart%27s_law). As one example, more efficient centralized tracking of student progress by standardized testing seems like such a good idea that well-intentioned laws [mandate it](https://en.wikipedia.org/wiki/No_Child_Left_Behind_Act). However, testing also incentivizes schools to focus more on teaching students to test well, and less on teaching broadly useful skills. As a result, it can cause overall educational outcomes to become worse. Similar examples abound, in politics, economics, health, science, and many other fields. This same counterintuitive relationship between efficiency and outcome occurs in machine learning, where it is called overfitting. Overfitting is heavily studied, somewhat theoretically understood, and has well known mitigations. This connection between the strong version of Goodhart's law in general, and overfitting in machine learning, provides a new lens for understanding bad outcomes, and new ideas for fixing them. Overfitting and Goodhart's law ========================== In machine learning (ML), **overfitting** is a pervasive phenomenon. We want to train an ML model to achieve some goal. We can't directly fit the model to the goal, so we instead train the model using some proxy which is *similar* to the goal. ![](/assets/cartoon-conversation.png width="300px" border="1") For instance, as an occasional computer vision researcher, my goal is sometimes to prove that my new image classification model works well. I accomplish this by measuring its accuracy, after asking it to label images (is this image a cat or a dog or a frog or a truck or a ...) from a standardized [test dataset of images](https://paperswithcode.com/dataset/cifar-10). I'm not allowed to train my model on the test dataset though (that would be cheating), so I instead train the model on a *proxy* dataset, called the training dataset. I also can't directly target prediction accuracy during training[^accuracytarget], so I instead target a *proxy* objective which is only related to accuracy. So rather than training my model on the goal I care about -- classification accuracy on a test dataset -- I instead train it using a *proxy objective* on a *proxy dataset*. At first everything goes as we hope -- the proxy improves, and since the goal is similar to the proxy, it also improves. ![](/assets/cartoon-early.png width="444px" border="1") As we continue optimizing the proxy though, we eventually exhaust the useable similarity between proxy and goal. The proxy keeps on getting better, but the goal stops improving. In machine learning we call this overfitting, but it is also an example of Goodhart's law. ![](/assets/cartoon-mid.png width="444px" border="1") [Goodhart's law](https://en.wikipedia.org/wiki/Goodhart%27s_law) states that, *when a measure becomes a target, it ceases to be a good measure*[^strathern]. Goodhart proposed this in the context of monetary policy, but it applies far more broadly. In the context of overfitting in machine learning, it describes how the proxy objective we optimize ceases to be a good measure of the objective we care about. The strong version of Goodhart's law: as we become too efficient, the thing we care about grows worse ========================== If we keep on optimizing the proxy objective, even after our goal stops improving, something more worrying happens. The goal often starts getting *worse*, even as our proxy objective continues to improve. Not just a little bit worse either -- often the goal will diverge towards infinity. This is an [extremely](https://www.cs.princeton.edu/courses/archive/spring16/cos495/slides/ML_basics_lecture6_overfitting.pdf) [general](https://www.cs.mcgill.ca/~dprecup/courses/ML/Lectures/ml-lecture02.pdf) [phenomenon](https://scholar.google.com/scholar?hl=en&q=overfitting) in machine learning. It mostly doesn't matter what our goal and proxy are, or what model architecture we use[^overfittinggenerality]. If we are very efficient at optimizing a proxy, then we make the thing it is a proxy for grow worse. ![](/assets/cartoon-late.png width="444px" border="1") Though this pheonomenon is often discussed, it doesn't seem to be named[^notoverfitting]. Let's call it **the strong version of Goodhart's law**[^strongunintended]. We can state it as: > *When a measure becomes a target, > if it is effectively optimized, > then the thing it is designed to measure will grow worse.* Goodhart's law says that if you optimize a proxy, eventually the goal you care about will stop improving. The strong version of Goodhart's law differs in that it says that as you over-optimize, the goal you care about won't just stop improving, but will instead grow much worse than if you had done nothing at all. Goodhart's law applies well beyond economics, where it was originally proposed. Similarly, the strong version of Goodhart's law applies well beyond machine learning. I believe it can help us understand failures in economies, governments, and social systems. Increasing efficiency and overfitting are happening everywhere ========================== Increasing efficiency is permeating almost every aspect of our society. If the thing that is being made more efficient is beneficial, then the increased efficiency makes the world a better place (overall, the world [seems to be becoming a better place](https://ourworldindata.org/a-history-of-global-living-conditions-in-5-charts)). If the thing that is being made more efficient is socially harmful, then the consequences of greater efficiency are scary or depressing (think mass surveillance, or robotic weapons). What about the most common case though -- where the thing we are making more efficient is related, but not identical, to beneficial outcomes? What happens when we get better at something which is merely correlated with outcomes we care about? In that case, we can overfit, the same as we do in machine learning. The outcomes we care about will improve for a while ... and then they will grow dramatically worse. Below are a few, possibly facile, examples applying this analogy. > **Goal:** Educate children well **Proxy:** [Measure student and school performance](https://en.wikipedia.org/wiki/No_Child_Left_Behind_Act) on standardized tests **Strong version of Goodhart's law leads to:** Schools narrowly focus on teaching students to answer questions like those on the test, at the expense of the underlying skills the test is intended to measure > **Goal:** Rapid progress in science **Proxy:** Pay researchers a [cash bonus for every publication](https://www.science.org/content/article/cash-bonuses-peer-reviewed-papers-go-global) **Strong version of Goodhart's law leads to:** Publication of incorrect or incremental results, collusion between reviewers and authors, research paper mills > **Goal:** A well-lived life **Proxy:** Maximize the reward pathway in the brain **Strong version of Goodhart's law leads to:** Substance addiction, gambling addiction, days lost to doomscrolling Twitter > **Goal:** Healthy population **Proxy:** Access to nutrient-rich food **Strong version of Goodhart's law leads to:** Obesity epidemic > **Goal:** Leaders that act in the best interests of the population **Proxy:** Leaders that have the most support in the population **Strong version of Goodhart's law leads to:** Leaders whose expertise and passions center narrowly around manipulating public opinion at the expense of social outcomes > **Goal:** An informed, thoughtful, and involved populace **Proxy:** The ease with which people can share and find ideas **Strong version of Goodhart's law leads to:** Filter bubbles, conspiracy theories, parasitic memes, escalated tribalism > **Goal:** Distribution of labor and resources based upon the needs of society **Proxy:** Capitalism **Strong version of Goodhart's law leads to:** Massive wealth disparities (with incomes ranging from hundreds of dollars per year to hundreds of dollars per second), with [more than a billion](https://hdr.undp.org/en/2020-MPI ) people living in poverty > **Goal:** The owners of Paperclips Unlimited, LLC, become wealthy **Proxy:** Number of paperclips made by the AI-run manufacturing plant **Strong version of Goodhart's law leads to:** The entire solar system, including the company owners, being [converted to paperclips](https://www.lesswrong.com/tag/paperclip-maximizer) As an exercise for the reader, you can think about how the strong version of Goodhart's law would apply to other efficiencies, like the ones in this list: ~~~ none telepresence and virtual reality personalized medicine gene therapy tailoring marketing messages to the individual consumers or voters who will find them most actionable predicting the outcome of elections writing code artificial intelligence reducing slack in supply chains rapidly disseminating ideas generating entertainment identifying new products people will buy raising livestock trading securities extracting fish from the ocean constructing cars ~~~ [Listing [greater-efficiency]: Some additional diverse things we are getting more efficient at. For most of these, initial improvements were broadly beneficial, but getting too good at them could cause profound negative consequences.] How do we mitigate the problems caused by overfitting and the strong version of Goodhart's law? ========================== If overfitting is useful as an analogy, it will be because some of the approaches that improve it in machine learning also transfer to other domains. Below, I review some of the most effective techniques from machine learning, and share some thoughts about how they might transfer. + **Mitigation: Better align proxy goals with desired outcomes.** In machine learning this often means carefully collecting training examples which are as similar as possible to the situation at test time. Outside of machine learning, this means changing the proxies we have control over -- e.g. laws, incentives, and social norms -- so that they directly encourage behavior that better aligns with our goals. This is the standard approach used to (try to) engineer social systems. + **Mitigation: Add regularization penalties to the system.** In machine learning, this is often performed by [penalizing the squared magnitude of parameters](https://developers.google.com/machine-learning/crash-course/regularization-for-simplicity/l2-regularization), so that they stay small. Importantly, regularization doesn't need to directly target undesirable behavior. Almost anything that penalizes deviations of a model from typicality works well. Outside of machine learning, anything that penalizes complexity, or adds friction or extra cost to a system, can be viewed as regularization. Some example ideas: + Add a billing mechanism to SMTP, so there's a small cost for every email. + Use a progressive tax code, so that unusual success is linked to disproportionately greater cost + Charge a court fee proportional to the squared (exponentiated?) number of lawsuits initiated by an organization, so that unusual use of the court system leads to unusual expenses + Tax the number of bits of information stored about users + **Mitigation: Inject noise into the system.** In machine learning, this involves adding random jitter to the inputs, parameters, and internal state of a model. The unpredictability resulting from this noise makes overfitting far more difficult. Here are some ideas for how to improve outcomes by injecting noise outside of machine learning: + Stack rank all the candidates for a highly competitive school or job. Typically, offers would be made to the top-k candidates. Instead, make offers probabilistically, with probability proportional to $\left(\right.$[approx # top tier candidates] $+$ [candidate's stack rank]$\left.\right)^{-1}$. Benefits include: greater diversity of accepted candidates; less ridiculous resources spent by the candidates tuning their application, and by application reviewers reviewing the applications, since small changes in assessed rank only have a small effect on outcome probabilities; occasionally you will draw a longshot candidate that is more likely to fail, but also more likely to succeed in an unconventional and unusually valuable way. + Randomly time quizzes and tests in a class, rather than giving them on pre-announced dates, so that students study to understand the material more, and cram (i.e., overfit) for the test less. + Require securities exchanges to add random jitter to the times when they process trades, with a standard deviation of about a second. (An efficient market is great. Building a global financial system out of a chaotic nonstationary dynamical system with a characteristic timescale more than six orders of magnitude faster than human reaction time is just asking for trouble.) + Randomize details of the electoral system on voting day, in order to prevent candidates from overfitting to incidental details of the current electoral system (e.g. by taking unreasonable positions that appeal to a pivotal minority). For instance randomly select between ranked choice or first past the post ballots, or randomly rescale the importance of votes from different districts. (I'm not saying all of these are *good* ideas. Just ... ideas.) + **Mitigation: Early stopping.** In machine learning, it's common to monitor a third metric, besides training loss and test performance, which we call validation loss. When the validation loss starts to get worse, we stop training, even if the training loss is still improving. This is the single most effective tool we have to prevent catastrophic overfitting. Here are some ways early stopping could be applied outside of machine learning: + Sharply limit the time between a call for proposals and submission date, so that proposals better reflect pre-existing readiness, and to avoid an effect where increasing resources are poured into proposal generation, rather than being used to create something useful + Whenever stock volatility rises above a threshold, suspend all market activity + The use of antitrust law to split companies that are preventing competition in a market + Estimate the importance of a decision in $$. When the value of the time you have already spent analyzing the decision approaches that value, make a snap decision. + Freeze the information that agents are allowed to use to achieve their goals. Press blackouts in the 48 hours before an election might fall under this category. One of the best understood *causes* of extreme overfitting is that the expressivity of the model being trained *too closely matches* the complexity of the proxy task. When the model is very weak, it can only make a little bit of progress on the task, and it doesn’t exhaust the similarity between the goal and the proxy. When the model is extremely strong and expressive, it can optimize the proxy objective in isolation, without inducing extreme behavior on other objectives. When the model's expressivity roughly matches the task complexity (e.g., the number of parameters is no more than a few orders of magnitude higher or lower than the number of training examples), then it can only do well on the proxy task by doing *extreme things everywhere else*. See Figure [capacity] for a demonstration of this idea on a simple task. This cause of overfitting motivates two final, diametrically opposed, methods for mitigating the strong version of Goodhart’s law. + **Mitigation: Restrict capabilities / capacity.** In machine learning, this is often achieved by making the model so small that it's incapable of overfitting. In the broader world, we could similarly limit the capacity of organizations or agents. Examples include: + Campaign finance limits + Set a maximum number of people that can work in companies of a given type. e.g. allow only 10 people to work in any lobbying group + Set the maximum number of parameters, or training compute, that any AI system can use. + **Mitigation: Increase capabilities / capacity.** In machine learning, if a model is made very big, it often has enough capacity to overfit to the training data without making performance on the test data worse. In the broader world, this would correspond to developing capabilities that are so great that there is no longer any tradeoff required between performance on the goal and the proxy. Examples include: + Obliterate all privacy, and make all the information about all people, governments, and other organizations available to everyone all the time, so that everyone can have perfect trust of everyone else. This could be achieved by legislating that every database be publicly accessible, and by putting cameras in every building. (to be clear -- from my value system, this would be a dystopian scenario) + Invest in basic research in clean energy + Develop as many complex, inscrutable, and diverse market trading instruments as possible, vesting on as many timescales as possible. (In nature, more complex ecosystems are more stable. Maybe there is a parallel for markets?) + Use the largest, most compute and data intensive, AI model possible in every scenario 😮[^gobig] This last mitigation of just continuing to increase capabilities works surprisingly well in machine learning. It is also a path of least resistance. Trying to fix our institutions by blindly making them better at pursuing misaligned goals is a terrible idea though. Parting thoughts ========================== The strong version of Goodhart's law underlies most of my personal fears around AI (expect a future blog post about my AI fears!). If there is one thing AI will enable, it is greater efficiency, on almost all tasks, over a very short time period. We are going to need to simultaneously deal with massive numbers of diverse unwanted side effects, just as our ability to collaborate on solutions is also disrupted. There's a lot of opportunity to *research* solutions to this problem. If you are a scientist looking for research ideas which are pro-social, and have the potential to create a whole new field, you should consider building formal (mathematical) bridges between results on overfitting in machine learning, and problems in economics, political science, management science, operations research, and elsewhere[^researchideas]. This is a goldmine waiting to be tapped. (I might actually be suggesting here that we should invent the field of [psychohistory](https://en.wikipedia.org/wiki/Psychohistory), and that overfitting phenomena will have a big role in that field.) The more our social systems break due to the strong version of Goodhart's law, the less we will be able to take the concerted rational action required to fix them. Hopefully naming, and better understanding, the phenomenon will help push in the opposite direction. ![Figure [capacity]: **Models often suffer from the strong version of Goodhart's law, and overfit catastrophically, when their complexity is well matched to the complexity of the proxy task.** If a model is instead much more or much less capable than required, it will overfit less. Here, models are trained to map from a one-dimensional input $x$ to a one-dimensional output $y$. All models are trained on the same 10 datapoints, in red. The model with 4 parameters is too weak to exactly fit the datapoints, but it smoothly approximates them. The model with 10,000 parameters is strong enough to easily fit all the datapoints, and also smoothly interpolate between them. The model with 10 parameters is exactly strong enough to fit the datapoints, but it can only contort itself to do so by behaving in extreme ways away from the training data. If asked to predict $y$ for a new value of $x$, the 10 parameter model would perform extremely poorly. For details of this toy experiment, which uses linear random feature models, see this [colab notebook](https://colab.research.google.com/drive/1mAqCsCE-6biiFxQu8swlc5MygmI9lMJA?usp=sharing).](/assets/size-mitigation.png width="290px" border="1") [^accuracytarget]: Accuracy is not differentiable, which makes it impossible to target by naive gradient descent training. It is usually replaced during training by a proxy of softmax-cross-entropy loss, which is differentiable. There are blackbox training methods which can directly target accuracy, but they are inefficient and rarely used. [^strathern]: This modern phrasing is due to Marilyn Strathern. Goodhart originally phrased the observation as the more clunky *any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes*. [^overfittinggenerality]: This glosses over a lot of variation. For instance, there is an entire subfield which studies the qualitative differences in overfitting in underparameterized, critically parameterized, and overparameterized models. Despite this variation, the core observation -- that when we train on a proxy our target gets better for a while, but then grows worse -- holds broadly. [^notoverfitting]: It's not simply overfitting. Overfitting refers to the proxy becoming better than the goal, not to the goal growing worse in an absolute sense. There are other related, but not identical, concepts -- for instance [perverse incentives](https://en.wikipedia.org/wiki/Perverse_incentive), [Campbell's law](https://en.wikipedia.org/wiki/Campbell%27s_law), the [Streisand effect](https://en.wikipedia.org/wiki/Streisand_effect), the [law of unintended consequences](https://en.wikipedia.org/wiki/Unintended_consequences), [Jevons paradox](https://en.m.wikipedia.org/wiki/Jevons_paradox), and the concept of [negative externalities](https://en.m.wikipedia.org/wiki/Externality#Negative). [Goodhart's curse](https://arbital.com/p/goodharts_curse/) is perhaps the closest. However, the definition of Goodhart's curse incorporates not only the phenomenon, but also a specific mechanism, and the mechanism is incorrect[^Goodhartcurse]. *Edit 2022/11/9: Andrew Hundt [suggests](https://twitter.com/athundt/status/1589591738792177664) that similar observations that optimization isn't always desirable have been made in the social sciences, and gives specific examples of "The New Jim Code" and "[Weapons of Math Destruction](https://en.m.wikipedia.org/wiki/Weapons_of_Math_Destruction)". Kiran Vodrahalli [points out](https://mathstodon.xyz/@kiranvodrahalli/109300676096306738) connections to robust optimization and the "[price of robustness](https://www.robustopt.com/references/Price%20of%20Robustness.pdf)." [Leo Gao](https://bmk.sh/) points me at a [recent paper](https://arxiv.org/abs/2210.10760) which uses the descriptive term "overoptimization" for this phenomenon, which I think is good.* [^strongunintended]: I also considered calling it the strong law of unintended consequences -- it's not just that there are unexpected side effects, but that that the more effectively you accomplish your task, the more those side effects will act against your original goal. [^gobig]: Note that for suficiently strong AI, limitations on its capabilities might be determined by the laws of physics, rather than by its compute scale or training dataset size. So if you're worried about misaligned AGI, this mitigation may offer no comfort. [^researchideas]: For instance, take PAC Bayes bounds from statistical learning theory, and use them to predict the optimal amount of power unions should have, in order to maximize the wealth of workers in an industry. Or, estimate the spectrum of candidate-controllable and uncontrollable variables in political contests, to predict points of political breakdown. (I'm blithely suggesting these examples as if they would be easy, and are well formed in their description. Of course, neither is true -- actually doing this would require hard work and brilliance in some ratio.) [^Goodhartcurse]: The [definition of Goodhart's curse](https://arbital.com/p/goodharts_curse/) includes [the optimizer's curse](https://www.semanticscholar.org/paper/The-Optimizer's-Curse%3A-Skepticism-and-Postdecision-Smith-Winkler/28cfed594544215673db802dce79b8c12d3ab5ab) as its causal mechanism. This is where the word 'curse' comes from in its name. If an objective $u$ is an imperfect proxy for a goal objective $v$, the optimizer's curse explains why optimizing $u$ finds an anomalously good $u$, and makes the *gap* between $u$ and $v$ grow large. It doesn't explain why optimizing $u$ makes $v$ grow worse in an absolute sense. That is, the optimizer's curse provides motivation for why Goodhart's law occurs. It does not provide motivation for why the strong version of Goodhart's law occurs. (As I briefly discuss elsewhere in the post, one common causal mechanism for $v$ growing worse is that it's expressivity is too closely matched to the complexity of the task it is performing. This is a very active research area though, and our understanding is both incomplete and actively changing.) body{visibility:hidden;white-space:pre;font-family:monospace} window.markdeepOptions = {mode: 'html', tocStyle: 'medium'}; window.alreadyProcessedMarkdeep||(document.body.style.visibility="visible")