Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de

专业知识 · 40-References/Papers/chinchilla - Chinchilla/01_original.md

--- title: "Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de" aliases: - "Chinchilla" - "arXiv:2203.15556" source: "https://arxiv.org/abs/2203.15556" arxiv: "2203.15556" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning

Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de

原文全文

Page 1

<a id="S0001"></a> Source: p.1 S0001

Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W.

<a id="S0002"></a> Source: p.1 S0002

Rae, Oriol Vinyals and Laurent Sifre★ ★Equal contributions We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.

<a id="S0003"></a> Source: p.1 S0003

We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.

<a id="S0004"></a> Source: p.1 S0004

By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.

<a id="S0005"></a> Source: p.1 S0005

We test this hypothesis by training a predicted computeoptimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data.

<a id="S0006"></a> Source: p.1 S0006

Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.

<a id="S0007"></a> Source: p.1 S0007

This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage.

<a id="S0008"></a> Source: p.1 S0008

As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher. 1.

<a id="S0009"></a> Source: p.1 S0009

Introduction Recently a series of Large Language Models (LLMs) have been introduced (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Thoppilan et al., 2022), with the largest dense language models now having over 500 billion parameters.

<a id="S0010"></a> Source: p.1 S0010

These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.

<a id="S0011"></a> Source: p.1 S0011

The compute and energy cost for training large language models is substantial (Rae et al., 2021; Thoppilan et al., 2022) and rises with increasing model size.

<a id="S0012"></a> Source: p.1 S0012

In practice, the allocated training compute budget is often known in advance: how many accelerators are available and for how long we want to use them.

<a id="S0013"></a> Source: p.1 S0013

Since it is typically only feasible to train these large models once, accurately estimating the best model hyperparameters for a given compute budget is critical (Tay et al., 2021).

<a id="S0014"></a> Source: p.1 S0014

Kaplan et al. (2020) showed that there is a power law relationship between the number of parameters in an autoregressive language model (LM) and its performance.

<a id="S0015"></a> Source: p.1 S0015

As a result, the field has been training larger and larger models, expecting performance improvements.

<a id="S0016"></a> Source: p.1 S0016

One notable conclusion in Kaplan et al. (2020) is that large models should not be trained to their lowest possible loss to be compute optimal.

<a id="S0017"></a> Source: p.1 S0017

Whilst we reach the same conclusion, we estimate that large models should be trained for many more training tokens than recommended by the authors.

<a id="S0018"></a> Source: p.1 S0018

Specifically, given a 10× increase computational budget, they suggests that the size of the model should increase 5.5× while the number of training tokens should only increase 1.8×.

<a id="S0019"></a> Source: p.1 S0019

Instead, we find that model size and the number of training tokens should be scaled in equal proportions.

<a id="S0020"></a> Source: p.1 S0020

Following Kaplan et al. (2020) and the training setup of GPT-3 (Brown et al., 2020), many of the recently trained large models have been trained for approximately 300 billion tokens (Table 1), in line with the approach of predominantly increasing model size when increasing compute.

<a id="S0021"></a> Source: p.1 S0021

<a id="S0022"></a> Source: p.1 S0022

Page 2

<a id="S0023"></a> Source: p.2 S0023

1T 100B 10B 1.0B 100M 10M 1017 1019 1021 1023 1025 FLOPs sretemaraP Approach 1 Approach 2 Approach 3 Kaplan et al (2020) Chinchilla (70B) Gopher (280B) GPT-3 (175B) Megatron-Turing NLG (530B) Figure 1 | Overlaid predictions.

<a id="S0024"></a> Source: p.2 S0024

We overlay the predictions from our three different approaches, along with projections from Kaplan et al. (2020).

<a id="S0025"></a> Source: p.2 S0025

We find that all three methods predict that current large models should be substantially smaller and therefore trained much longer than is currently done.

<a id="S0026"></a> Source: p.2 S0026

In Figure A3, we show the results with the predicted optimal tokens plotted against the optimal number of parameters for fixed FLOP budgets.

<a id="S0027"></a> Source: p.2 S0027

Chinchilla outperforms Gopher and the other large models (see Section 4.2).

<a id="S0028"></a> Source: p.2 S0028

In this work, we revisit the question: Given a fixed FLOPs budget, 1 how should one trade-off model size and the number of training tokens?

<a id="S0029"></a> Source: p.2 S0029

To answer this question, we model the final pre-training loss2 𝐿(𝑁, 𝐷) as a function of the number of model parameters 𝑁, and the number of training tokens, 𝐷.

<a id="S0030"></a> Source: p.2 S0030

Since the computational budget 𝐶 is a deterministic function FLOPs(𝑁, 𝐷) of the number of seen training tokens and model parameters, we are interested in minimizing 𝐿 under the constraint FLOPs(𝑁, 𝐷) = 𝐶: 𝑁 (𝐶), 𝐷 (𝐶) = argmin 𝐿(𝑁, 𝐷). (1) 𝑜𝑝𝑡 𝑜𝑝𝑡 𝑁,𝐷 s.t.

<a id="S0031"></a> Source: p.2 S0031

FLOPs(𝑁,𝐷)=𝐶 The functions 𝑁 (𝐶), and 𝐷 (𝐶) describe the optimal allocation of a computational budget 𝐶.

<a id="S0032"></a> Source: p.2 S0032

We 𝑜𝑝𝑡 𝑜𝑝𝑡 empirically estimate these functions based on the losses of over 400 models, ranging from under 70M to over 16B parameters, and trained on 5B to over 400B tokens – with each model configuration trained for several different training horizons.

<a id="S0033"></a> Source: p.2 S0033

Our approach leads to considerably different results than that of Kaplan et al. (2020).

<a id="S0034"></a> Source: p.2 S0034

We highlight our results in Figure 1 and how our approaches differ in Section 2.

<a id="S0035"></a> Source: p.2 S0035

Based on our estimated compute-optimal frontier, we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training on 4 times more tokens.

<a id="S0036"></a> Source: p.2 S0036

We verify this by training a more compute-optimal 70B model, called Chinchilla, on 1.4 trillion tokens.

<a id="S0037"></a> Source: p.2 S0037

Not only does Chinchilla outperform its much larger counterpart, Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware.

<a id="S0038"></a> Source: p.2 S0038

The energy cost of a large language model is amortized through its usage for inference an fine-tuning.

<a id="S0039"></a> Source: p.2 S0039

The benefits of a more optimally trained smaller model, therefore, extend beyond the immediate benefits of its improved performance. 1For example, knowing the number of accelerators and a target training duration. 2For simplicity, we perform our analysis on the smoothed training loss which is an unbiased estimate of the test loss, as we are in the infinite data regime (the number of training tokens is less than the number of tokens in the entire corpus). 2

Page 3

<a id="S0040"></a> Source: p.3 S0040

We show five of the current largest dense transformer models, their size, and the number of training tokens.

<a id="S0041"></a> Source: p.3 S0041

Other than LaMDA (Thoppilan et al., 2022), most models are trained for approximately 300 billion tokens.

<a id="S0042"></a> Source: p.3 S0042

We introduce Chinchilla, a substantially smaller model, trained for much longer than 300B tokens.

<a id="S0043"></a> Source: p.3 S0043

Model Size (# Parameters) Training Tokens LaMDA (Thoppilan et al., 2022) 137 Billion 168 Billion GPT-3 (Brown et al., 2020) 175 Billion 300 Billion Jurassic (Lieber et al., 2021) 178 Billion 300 Billion Gopher (Rae et al., 2021) 280 Billion 300 Billion MT-NLG 530B (Smith et al., 2022) 530 Billion 270 Billion Chinchilla 70 Billion 1.4 Trillion 2.

<a id="S0044"></a> Source: p.3 S0044

Related Work Large language models. A variety of large language models have been introduced in the last few years.

<a id="S0045"></a> Source: p.3 S0045

These include both dense transformer models (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Thoppilan et al., 2022) and mixture-of-expert (MoE) models (Du et al., 2021; Fedus et al., 2021; Zoph et al., 2022).

<a id="S0046"></a> Source: p.3 S0046

The largest dense transformers have passed 500 billion parameters (Smith et al., 2022).

<a id="S0047"></a> Source: p.3 S0047

The drive to train larger and larger models is clear—so far increasing the size of language models has been responsible for improving the state-of-the-art in many language modelling tasks.

<a id="S0048"></a> Source: p.3 S0048

Nonetheless, large language models face several challenges, including their overwhelming computational requirements (the cost of training and inference increase with model size) (Rae et al., 2021; Thoppilan et al., 2022) and the need for acquiring more high-quality training data.

<a id="S0049"></a> Source: p.3 S0049

In fact, in this work we find that larger, high quality datasets will play a key role in any further scaling of language models.

<a id="S0050"></a> Source: p.3 S0050

Understanding the scaling behaviour of language models and their transfer properties has been important in the development of recent large models (Hernandez et al., 2021; Kaplan et al., 2020).

<a id="S0051"></a> Source: p.3 S0051

Kaplan et al. (2020) first showed a predictable relationship between model size and loss over many orders of magnitude.

<a id="S0052"></a> Source: p.3 S0052

The authors investigate the question of choosing the optimal model size to train for a given compute budget.

<a id="S0053"></a> Source: p.3 S0053

Similar to us, they address this question by training various models.

<a id="S0054"></a> Source: p.3 S0054

Our work differs from Kaplan et al. (2020) in several important ways.

<a id="S0055"></a> Source: p.3 S0055

First, the authors use a fixed number of training tokens and learning rate schedule for all models; this prevents them from modelling the impact of these hyperparameters on the loss.

<a id="S0056"></a> Source: p.3 S0056

In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results in the best final loss regardless of model size—see Figure A1.

<a id="S0057"></a> Source: p.3 S0057

For a fixed learning rate cosine schedule to 130B tokens, the intermediate loss estimates (for 𝐷(cid:48) << 130B) are therefore overestimates of the loss of a model trained with a schedule length matching 𝐷(cid:48).

<a id="S0058"></a> Source: p.3 S0058

Using these intermediate losses results in underestimating the effectiveness of training models on less data than 130B tokens, and eventually contributes to the conclusion that model size should increase faster than training data size as compute budget increases.

<a id="S0059"></a> Source: p.3 S0059

In contrast, our analysis predicts that both quantities should scale at roughly the same rate.

<a id="S0060"></a> Source: p.3 S0060

Secondly, we include models with up to 16B parameters, as we observe that there is slight curvature in the FLOP-loss frontier (see Appendix E)—in fact, the majority of the models used in our analysis have more than 500 million parameters, in contrast the majority of runs in Kaplan et al. (2020) are significantly smaller—many being less than 100M parameters.

<a id="S0061"></a> Source: p.3 S0061

Recently, Clark et al. (2022) specifically looked in to the scaling properties of Mixture of Expert 3

Page 4

<a id="S0062"></a> Source: p.4 S0062

language models, showing that the scaling with number of experts diminishes as the model size increases—their approach models the loss as a function of two variables: the model size and the number of experts.

<a id="S0063"></a> Source: p.4 S0063

However, the analysis is done with a fixed number of training tokens, as in Kaplan et al. (2020), potentially underestimating the improvements of branching.

<a id="S0064"></a> Source: p.4 S0064

Estimating hyperparameters for large models.

<a id="S0065"></a> Source: p.4 S0065

The model size and the number of training tokens are not the only two parameters to chose when selecting a language model and a procedure to train it.

<a id="S0066"></a> Source: p.4 S0066

Other important factors include learning rate, learning rate schedule, batch size, optimiser, and width-to-depth ratio.

<a id="S0067"></a> Source: p.4 S0067

In this work, we focus on model size and the number of training steps, and we rely on existing work and provided experimental heuristics to determine the other necessary hyperparameters.

<a id="S0068"></a> Source: p.4 S0068

Yang et al. (2021) investigates how to choose a variety of these parameters for training an autoregressive transformer, including the learning rate and batch size.

<a id="S0069"></a> Source: p.4 S0069

McCandlish et al. (2018) finds only a weak dependence between optimal batch size and model size.

<a id="S0070"></a> Source: p.4 S0070

Shallue et al. (2018); Zhang et al. (2019) suggest that using larger batch-sizes than those we use is possible.

<a id="S0071"></a> Source: p.4 S0071

Levine et al. (2020) investigates the optimal depth-to-width ratio for a variety of standard model sizes.

<a id="S0072"></a> Source: p.4 S0072

We use slightly less deep models than proposed as this translates to better wall-clock performance on our hardware.

<a id="S0073"></a> Source: p.4 S0073

Recently, various promising alternatives to traditional dense transformers have been proposed.

<a id="S0074"></a> Source: p.4 S0074

For example, through the use of conditional computation large MoE models like the 1.7 trillion parameter Switch transformer (Fedus et al., 2021), the 1.2 Trillion parameter GLaM model (Du et al., 2021), and others (Artetxe et al., 2021; Zoph et al., 2022) are able to provide a large effective model size despite using relatively fewer training and inference FLOPs.

<a id="S0075"></a> Source: p.4 S0075

However, for very large models the computational benefits of routed models seems to diminish (Clark et al., 2022).

<a id="S0076"></a> Source: p.4 S0076

An orthogonal approach to improving language models is to augment transformers with explicit retrieval mechanisms, as done by Borgeaud et al. (2021); Guu et al. (2020); Lewis et al. (2020).

<a id="S0077"></a> Source: p.4 S0077

This approach effectively increases the number of data tokens seen during training (by a factor of ∼ 10 in Borgeaud et al. (2021)).

<a id="S0078"></a> Source: p.4 S0078

This suggests that the performance of language models may be more dependant on the size of the training data than previously thought. 3.

<a id="S0079"></a> Source: p.4 S0079

Estimating the optimal parameter/training tokens allocation We present three different approaches to answer the question driving our research: Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?

<a id="S0080"></a> Source: p.4 S0080

In all three cases we start by training a range of models varying both model size and the number of training tokens and use the resulting training curves to fit an empirical estimator of how they should scale.

<a id="S0081"></a> Source: p.4 S0081

We assume a power-law relationship between compute and model size as done in Clark et al. (2022); Kaplan et al. (2020), though future work may want to include potential curvature in this relationship for large model sizes.

<a id="S0082"></a> Source: p.4 S0082

The resulting predictions are similar for all three methods and suggest that parameter count and number of training tokens should be increased equally with more compute3— with proportions reported in Table 2.

<a id="S0083"></a> Source: p.4 S0083

This is in clear contrast to previous work on this topic and warrants further investigation. 3We compute FLOPs as described in Appendix F. 4

Page 5

<a id="S0084"></a> Source: p.5 S0084

6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1017 1018 1019 1020 1021 1022 FLOPS ssol gniniarT 10B 5B 1012 2.5B 1B 1011 500M 250M 1010 75M 109 1017 1019 1021 1023 1025 FLOPs snekoT 1T 1.5T 100B 10B 1.0B 100M 1017 1019 1021 1023 1025 FLOPs sretemaraP 67B Figure 2 | Training curve envelope.

<a id="S0085"></a> Source: p.5 S0085

On the left we show all of our different runs.

<a id="S0086"></a> Source: p.5 S0086

We launched a range of model sizes going from 70M to 10B, each for four different cosine cycle lengths.

<a id="S0087"></a> Source: p.5 S0087

From these curves, we extracted the envelope of minimal loss per FLOP, and we used these points to estimate the optimal model size (center) for a given compute budget and the optimal number of training tokens (right).

<a id="S0088"></a> Source: p.5 S0088

In green, we show projections of optimal model size and training token count based on the number of FLOPs used to train Gopher (5.76 × 1023). 3.1.

<a id="S0089"></a> Source: p.5 S0089

Approach 1: Fix model sizes and vary number of training tokens In our first approach we vary the number of training steps for a fixed family of models (ranging from 70M to over 10B parameters), training each model for 4 different number of training sequences.

<a id="S0090"></a> Source: p.5 S0090

From these runs, we are able to directly extract an estimate of the minimum loss achieved for a given number of training FLOPs.

<a id="S0091"></a> Source: p.5 S0091

Training details for this approach can be found in Appendix D.

<a id="S0092"></a> Source: p.5 S0092

For each parameter count 𝑁 we train 4 different models, decaying the learning rate by a factor of 10× over a horizon (measured in number of training tokens) that ranges by a factor of 16×.

<a id="S0093"></a> Source: p.5 S0093

Then, for each run, we smooth and then interpolate the training loss curve.

<a id="S0094"></a> Source: p.5 S0094

From this, we obtain a continuous mapping from FLOP count to training loss for each run.

<a id="S0095"></a> Source: p.5 S0095

Then, for each FLOP count, we determine which run achieves the lowest loss.

<a id="S0096"></a> Source: p.5 S0096

Using these interpolants, we obtain a mapping from any FLOP count 𝐶, to the most efficient choice of model size 𝑁 and number of training tokens 𝐷 such that FLOPs(𝑁, 𝐷) = 𝐶.4 At 1500 logarithmically spaced FLOP values, we find which model size achieves the lowest loss of all models along with the required number of training tokens.

<a id="S0097"></a> Source: p.5 S0097

Finally, we fit power laws to estimate the optimal model size and number of training tokens for any given amount of compute (see the center and right panels of Figure 2), obtaining a relationship 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.

<a id="S0098"></a> Source: p.5 S0098

We 𝑜𝑝𝑡 𝑜𝑝𝑡 find that 𝑎 = 0.50 and 𝑏 = 0.50—as summarized in Table 2.

<a id="S0099"></a> Source: p.5 S0099

In Section D.4, we show a head-to-head comparison at 1021 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)—using the model size we predict has a clear advantage. 3.2.

<a id="S0100"></a> Source: p.5 S0100

Approach 2: IsoFLOP profiles In our second approach we vary the model size5 for a fixed set of 9 different training FLOP counts6 (ranging from 6 × 1018 to 3 × 1021 FLOPs), and consider the final training loss for each point7. in contrast with Approach 1 that considered points (𝑁, 𝐷, 𝐿) along the entire training runs.

<a id="S0101"></a> Source: p.5 S0101

This allows us to directly answer the question: For a given FLOP budget, what is the optimal parameter count? 4Note that all selected points are within the last 15% of training.

<a id="S0102"></a> Source: p.5 S0102

This suggests that when training a model over 𝐷 tokens, we should pick a cosine cycle length that decays 10× over approximately 𝐷 tokens—see further details in Appendix B. 5In approach 2, model size varies up to 16B as opposed to approach 1 where we only used models up to 10B. 6The number of training tokens is determined by the model size and training FLOPs. 7We set the cosine schedule length to match the number of tokens, which is optimal according to the analysis presented in Appendix B. 5

Page 6

<a id="S0103"></a> Source: p.6 S0103

3.2 3.0 2.8 2.6 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT 1T 100B 6e18 1e19 10B 3e19 6e19 1e20 1B 3e20 6e20 1e21 100M 3e21 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 63B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.4T Figure 3 | IsoFLOP curves.

<a id="S0104"></a> Source: p.6 S0104

For various model sizes, we choose the number of training tokens such that the final FLOPs is a constant.

<a id="S0105"></a> Source: p.6 S0105

The cosine cycle length is set to match the target FLOP count.

<a id="S0106"></a> Source: p.6 S0106

We find a clear valley in loss, meaning that for a given FLOP budget there is an optimal model to train (left).

<a id="S0107"></a> Source: p.6 S0107

Using the location of these valleys, we project optimal model size and number of tokens for larger models (center and right).

<a id="S0108"></a> Source: p.6 S0108

In green, we show the estimated number of parameters and tokens for an optimal model trained with the compute budget of Gopher.

<a id="S0109"></a> Source: p.6 S0109

For each FLOP budget, we plot the final loss (after smoothing) against the parameter count in Figure 3 (left).

<a id="S0110"></a> Source: p.6 S0110

In all cases, we ensure that we have trained a diverse enough set of model sizes to see a clear minimum in the loss.

<a id="S0111"></a> Source: p.6 S0111

We fit a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved (Figure 3 (left)).

<a id="S0112"></a> Source: p.6 S0112

As with the previous approach, we then fit a power law between FLOPs and loss-optimal model size and number of training tokens, shown in Figure 3 (center, right).

<a id="S0113"></a> Source: p.6 S0113

Again, we fit exponents of the form 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏 and we find that 𝑜𝑝𝑡 𝑜𝑝𝑡 𝑎 = 0.49 and 𝑏 = 0.51—as summarized in Table 2. 3.3.

<a id="S0114"></a> Source: p.6 S0114

Approach 3: Fitting a parametric loss function Lastly, we model all final losses from experiments in Approach 1 & 2 as a parametric function of model parameter count and the number of seen tokens.

<a id="S0115"></a> Source: p.6 S0115

Following a classical risk decomposition (see Section D.2), we propose the following functional form 𝐴 𝐵 𝐿ˆ (𝑁, 𝐷) (cid:44) 𝐸 + + . (2) 𝑁𝛼 𝐷𝛽 The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text.

<a id="S0116"></a> Source: p.6 S0116

The second term captures the fact that a perfectly trained transformer with 𝑁 parameters underperforms the ideal generative process.

<a id="S0117"></a> Source: p.6 S0117

The final term captures the fact that the transformer is not trained to convergence, as we only make a finite number of optimisation steps, on a sample of the dataset distribution.

<a id="S0118"></a> Source: p.6 S0118

To estimate ( 𝐴, 𝐵, 𝐸, 𝛼, 𝛽), we minimize the Huber loss (Huber, 1964) between the predicted and observed log loss using the L-BFGS algorithm (Nocedal, 1980): min ∑︁ Huber (cid:16) log 𝐿ˆ (𝑁 , 𝐷 ) − log 𝐿 (cid:17) (3) 𝛿 𝑖 𝑖 𝑖 𝐴,𝐵,𝐸,𝛼,𝛽 Runs 𝑖 We account for possible local minima by selecting the best fit from a grid of initialisations.

<a id="S0119"></a> Source: p.6 S0119

The Huber loss (𝛿 = 10−3) is robust to outliers, which we find important for good predictive performance over held-out data points.

<a id="S0120"></a> Source: p.6 S0120

Section D.2 details the fitting procedure and the loss decomposition. 6

Page 7

<a id="S0121"></a> Source: p.7 S0121

100B 40B 10B 1B 100M 1018 1019 1020 1021 1022 1023 Gopher budget Training FLOPs ezis ledoM IsoLoss contours 5.00 4.00 3.00 Efficient frontier Empirical data 2.00 IsoFLOPs slice ssoL IsoFLOPs slices Train.

<a id="S0122"></a> Source: p.7 S0122

FLOPs 6e+18 1e+19 3e+19 6e+19 1e+20 3e+20 6e+20 1e+21 3e+21 Gopher 100M 1B 10B 40B Model size Figure 4 | Parametric fit.

<a id="S0123"></a> Source: p.7 S0123

We fit a parametric modelling of the loss 𝐿ˆ (𝑁, 𝐷) and display contour (left) and isoFLOP slices (right).

<a id="S0124"></a> Source: p.7 S0124

For each isoFLOP slice, we include a corresponding dashed line in the left plot.

<a id="S0125"></a> Source: p.7 S0125

In the left plot, we show the efficient frontier in blue, which is a line in log-log space.

<a id="S0126"></a> Source: p.7 S0126

Specifically, the curve goes through each iso-loss contour at the point with the fewest FLOPs.

<a id="S0127"></a> Source: p.7 S0127

We project the optimal model size given the Gopher FLOP budget to be 40B parameters.

<a id="S0128"></a> Source: p.7 S0128

We can approximate the functions 𝑁 𝑜𝑝𝑡 and 𝐷 𝑜𝑝𝑡 by minimizing the parametric loss 𝐿ˆ under the constraint FLOPs(𝑁, 𝐷) ≈ 6𝑁 𝐷 (Kaplan et al., 2020).

<a id="S0129"></a> Source: p.7 S0129

The resulting 𝑁 and 𝐷 𝑜𝑝𝑡 𝑜𝑝𝑡 balance the two terms in Equation (3) that depend on model size and data.

<a id="S0130"></a> Source: p.7 S0130

By construction, they have a power-law form: 𝑁 (𝐶) = 𝐺 (cid:18) 𝐶 (cid:19) 𝑎 , 𝐷 (𝐶) = 𝐺−1 (cid:18) 𝐶 (cid:19)𝑏 , where 𝐺 = (cid:18) 𝛼𝐴 (cid:19) 𝛼+ 1 𝛽 , 𝑎 = 𝛽 , and 𝑏 = 𝛼 . (4) 𝑜𝑝𝑡 6 𝑜𝑝𝑡 6 𝛽𝐵 𝛼 + 𝛽 𝛼 + 𝛽 We show contours of the fitted function 𝐿ˆ in Figure 4 (left), and the closed-form efficient computational frontier in blue.

<a id="S0131"></a> Source: p.7 S0131

From this approach, we find that 𝑎 = 0.46 and 𝑏 = 0.54—as summarized in Table 2. 3.4.

<a id="S0132"></a> Source: p.7 S0132

Optimal model scaling We find that the three approaches, despite using different fitting methodologies and different trained models, yield comparable predictions for the optimal scaling in parameters and tokens with FLOPs (shown in Table 2).

<a id="S0133"></a> Source: p.7 S0133

All three approaches suggest that as compute budget increases, model size and the amount of training data should be increased in approximately equal proportions.

<a id="S0134"></a> Source: p.7 S0134

The first and second approaches yield very similar predictions for optimal model sizes, as shown in Figure 1 and Figure A3.

<a id="S0135"></a> Source: p.7 S0135

The third approach predicts even smaller models being optimal at larger compute budgets.

<a id="S0136"></a> Source: p.7 S0136

We note that the observed points (𝐿, 𝑁, 𝐷) for low training FLOPs (𝐶 (cid:54) 1𝑒21) have larger residuals (cid:107)𝐿 − 𝐿ˆ (𝑁, 𝐷) (cid:107) 2 than points with higher computational budgets.

<a id="S0137"></a> Source: p.7 S0137

The fitted model places increased 2 weight on the points with more FLOPs—automatically considering the low-computational budget points as outliers due to the Huber loss.

<a id="S0138"></a> Source: p.7 S0138

As a consequence of the empirically observed negative curvature in the frontier 𝐶 → 𝑁 (see Appendix E), this results in predicting a lower 𝑁 than the 𝑜𝑝𝑡 𝑜𝑝𝑡 two other approaches.

<a id="S0139"></a> Source: p.7 S0139

In Table 3 we show the estimated number of FLOPs and tokens that would ensure that a model of a given size lies on the compute-optimal frontier.

<a id="S0140"></a> Source: p.7 S0140

Our findings suggests that the current generation of 7

Page 8

<a id="S0141"></a> Source: p.8 S0141

Table 2 | Estimated parameter and data scaling with increased training compute.

<a id="S0142"></a> Source: p.8 S0142

The listed values are the exponents, 𝑎 and 𝑏, on the relationship 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.

<a id="S0143"></a> Source: p.8 S0143

Our analysis suggests 𝑜𝑝𝑡 𝑜𝑝𝑡 a near equal scaling in parameters and data with increasing compute which is in clear contrast to previous work on the scaling of large models.

<a id="S0144"></a> Source: p.8 S0144

The 10th and 90th percentiles are estimated via bootstrapping data (80% of the dataset is sampled 100 times) and are shown in parenthesis.

<a id="S0145"></a> Source: p.8 S0145

Approach Coeff. 𝑎 where 𝑁 ∝ 𝐶𝑎 Coeff. 𝑏 where 𝐷 ∝ 𝐶𝑏 𝑜𝑝𝑡 𝑜𝑝𝑡 1.

<a id="S0146"></a> Source: p.8 S0146

Minimum over training curves 0.50 (0.488, 0.502) 0.50 (0.501, 0.512) 2.

<a id="S0147"></a> Source: p.8 S0147

IsoFLOP profiles 0.49 (0.462, 0.534) 0.51 (0.483, 0.529) 3.

<a id="S0148"></a> Source: p.8 S0148

Parametric modelling of the loss 0.46 (0.454, 0.455) 0.54 (0.542, 0.543) Kaplan et al. (2020) 0.73 0.27 Table 3 | Estimated optimal training FLOPs and training tokens for various model sizes.

<a id="S0149"></a> Source: p.8 S0149

For various model sizes, we show the projections from Approach 1 of how many FLOPs and training tokens would be needed to train compute-optimal models.

<a id="S0150"></a> Source: p.8 S0150

The estimates for Approach 2 & 3 are similar (shown in Section D.3) Parameters FLOPs FLOPs (in Gopher unit) Tokens 400 Million 1.92e+19 1/29, 968 8.0 Billion 1 Billion 1.21e+20 1/4, 761 20.2 Billion 10 Billion 1.23e+22 1/46 205.1 Billion 67 Billion 5.76e+23 1 1.5 Trillion . 175 Billion 3.85e+24 6.7 3.7 Trillion 280 Billion 9.90e+24 17.2 5.9 Trillion 520 Billion 3.43e+25 59.5 11.0 Trillion 1 Trillion 1.27e+26 221.3 21.2 Trillion 10 Trillion 1.30e+28 22515.9 216.2 Trillion large language models are considerably over-sized, given their respective compute budgets, as shown in Figure 1.

<a id="S0151"></a> Source: p.8 S0151

For example, we find that a 175 billion parameter model should be trained with a compute budget of 4.41 × 1024 FLOPs and on over 4.2 trillion tokens. A 280 billion Gopher-like model is the optimal model to train given a compute budget of approximately 1025 FLOPs and should be trained on 6.8 trillion tokens.

<a id="S0152"></a> Source: p.8 S0152

Unless one has a compute budget of 1026 FLOPs (over 250× the compute used to train Gopher), a 1 trillion parameter model is unlikely to be the optimal model to train.

<a id="S0153"></a> Source: p.8 S0153

Furthermore, the amount of training data that is projected to be needed is far beyond what is currently used to train large models, and underscores the importance of dataset collection in addition to engineering improvements that allow for model scale.

<a id="S0154"></a> Source: p.8 S0154

While there is significant uncertainty extrapolating out many orders of magnitude, our analysis clearly suggests that given the training compute budget for many current LLMs, smaller models should have been trained on more tokens to achieve the most performant model.

<a id="S0155"></a> Source: p.8 S0155

In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021).

<a id="S0156"></a> Source: p.8 S0156

In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions. 8

Page 9

<a id="S0157"></a> Source: p.9 S0157

Chinchilla Based on our analysis in Section 3, the optimal model size for the Gopher compute budget is somewhere between 40 and 70 billion parameters.

<a id="S0158"></a> Source: p.9 S0158

We test this hypothesis by training a model on the larger end of this range—70B parameters—for 1.4T tokens, due to both dataset and computational efficiency considerations.

<a id="S0159"></a> Source: p.9 S0159

In this section we compare this model, which we call Chinchilla, to Gopher and other LLMs.

<a id="S0160"></a> Source: p.9 S0160

Both Chinchilla and Gopher have been trained for the same number of FLOPs but differ in the size of the model and the number of training tokens.

<a id="S0161"></a> Source: p.9 S0161

While pre-training a large language model has a considerable compute cost, downstream finetuning and inference also make up substantial compute usage (Rae et al., 2021).

<a id="S0162"></a> Source: p.9 S0162

Due to being 4× smaller than Gopher, both the memory footprint and inference cost of Chinchilla are also smaller. 4.1.

<a id="S0163"></a> Source: p.9 S0163

Model and training details The full set of hyperparameters used to train Chinchilla are given in Table 4.

<a id="S0164"></a> Source: p.9 S0164

Chinchilla uses the same model architecture and training setup as Gopher with the exception of the differences listed below. • We train Chinchilla on MassiveText (the same dataset as Gopher) but use a slightly different subset distribution (shown in Table A1) to account for the increased number of training tokens. • We use AdamW (Loshchilov and Hutter, 2019) for Chinchilla rather than Adam (Kingma and Ba, 2014) as this improves the language modelling loss and the downstream task performance after finetuning.8 • We train Chinchilla with a slightly modified SentencePiece (Kudo and Richardson, 2018) tokenizer that does not apply NFKC normalisation.

<a id="S0165"></a> Source: p.9 S0165

The vocabulary is very similar– 94.15% of tokens are the same as those used for training Gopher.

<a id="S0166"></a> Source: p.9 S0166

We find that this particularly helps with the representation of mathematics and chemistry, for example. • Whilst the forward and backward pass are computed in bfloat16, we store a float32 copy of the weights in the distributed optimiser state (Rajbhandari et al., 2020).

<a id="S0167"></a> Source: p.9 S0167

See Lessons Learned from Rae et al. (2021) for additional details.

<a id="S0168"></a> Source: p.9 S0168

In Appendix G we show the impact of the various optimiser related changes between Chinchilla and Gopher.

<a id="S0169"></a> Source: p.9 S0169

All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).

<a id="S0170"></a> Source: p.9 S0170

We include a Chinchilla model card (Mitchell et al., 2019) in Table A8.

<a id="S0171"></a> Source: p.9 S0171

Model Layers Number Heads Key/Value Size d Max LR Batch Size model Gopher 280B 80 128 128 16,384 4 × 10−5 3M → 6M Chinchilla 70B 80 64 128 8,192 1 × 10−4 1.5M → 3M Table 4 | Chinchilla architecture details.

<a id="S0172"></a> Source: p.9 S0172

We list the number of layers, the key/value size, the bottleneck activation size d , the maximum learning rate, and the training batch size (# tokens). model The feed-forward size is always set to 4 × d .

<a id="S0173"></a> Source: p.9 S0173

Note that we double the batch size midway through model training for both Chinchilla and Gopher. 8Interestingly, a model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance is notably better– see Figure A7 9

Page 10

<a id="S0174"></a> Source: p.10 S0174

Tasks Examples Language Modelling 20 WikiText-103, The Pile: PG-19, arXiv, FreeLaw, . . .

<a id="S0175"></a> Source: p.10 S0175

Reading Comprehension 3 RACE-m, RACE-h, LAMBADA Question Answering 3 Natural Questions, TriviaQA, TruthfulQA Common Sense 5 HellaSwag, Winogrande, PIQA, SIQA, BoolQ MMLU 57 High School Chemistry, Astronomy, Clinical Knowledge, . . .

<a id="S0176"></a> Source: p.10 S0176

BIG-bench 62 Causal Judgement, Epistemic Reasoning, Temporal Sequences, . . .

<a id="S0177"></a> Source: p.10 S0177

We evaluate Chinchilla on a collection of language modelling along with downstream tasks.

<a id="S0178"></a> Source: p.10 S0178

We evaluate on largely the same tasks as in Rae et al. (2021), to allow for direct comparison. 4.2.

<a id="S0179"></a> Source: p.10 S0179

Results We perform an extensive evaluation of Chinchilla, comparing against various large language models.

<a id="S0180"></a> Source: p.10 S0180

We evaluate on a large subset of the tasks presented in Rae et al. (2021), shown in Table 5.

<a id="S0181"></a> Source: p.10 S0181

As the focus of this work is on optimal model scaling, we included a large representative subset, and introduce a few new evaluations to allow for better comparison to other existing large models.

<a id="S0182"></a> Source: p.10 S0182

The evaluation details for all tasks are the same as described in Rae et al. (2021). 4.2.1.

<a id="S0183"></a> Source: p.10 S0183

Language modelling stcartsba_dembup retropxe_hin sdnuorgkcab_otpsu lartnec_dembup cc_elip 2suprockoob egnahcxekcats seltitbusnepo 2txetbewnepo swenrekcah scitamehtam_md vixra waleerf 3skoob srepaplihp buhtig cri_utnubu lraporue 91_gp_grebnetug 0.10 0.08 0.06 0.04 0.02 0.00 bpb ni esaerceD rehpoG ot derapmoc Figure 5 | Pile Evaluation.

<a id="S0184"></a> Source: p.10 S0184

For the different evaluation sets in The Pile (Gao et al., 2020), we show the bits-per-byte (bpb) improvement (decrease) of Chinchilla compared to Gopher.

<a id="S0185"></a> Source: p.10 S0185

On all subsets, Chinchilla outperforms Gopher.

<a id="S0186"></a> Source: p.10 S0186

Chinchilla significantly outperforms Gopher on all evaluation subsets of The Pile (Gao et al., 2020), as shown in Figure 5.

<a id="S0187"></a> Source: p.10 S0187

Compared to Jurassic-1 (178B) Lieber et al. (2021), Chinchilla is more performant on all but two subsets– dm_mathematics and ubuntu_irc– see Table A5 for a raw bits-per-byte comparison.

<a id="S0188"></a> Source: p.10 S0188

On Wikitext103 (Merity et al., 2017), Chinchilla achieves a perplexity of 7.16 compared to 7.75 for Gopher.

<a id="S0189"></a> Source: p.10 S0189

Some caution is needed when comparing Chinchilla with Gopher on these language modelling benchmarks as Chinchilla is trained on 4× more data than Gopher and thus train/test set leakage may artificially enhance the results.

Page 11

<a id="S0190"></a> Source: p.11 S0190

Random 25.0% Average human rater 34.5% GPT-3 5-shot 43.9% Gopher 5-shot 60.0% Chinchilla 5-shot 67.6% Average human expert performance 89.8% June 2022 Forecast 57.1% June 2023 Forecast 63.4% Table 6 | Massive Multitask Language Understanding (MMLU).

<a id="S0191"></a> Source: p.11 S0191

We report the average 5-shot accuracy over 57 tasks with model and human accuracy comparisons taken from Hendrycks et al. (2020).

<a id="S0192"></a> Source: p.11 S0192

We also include the average prediction for state of the art accuracy in June 2022/2023 made by 73 competitive human forecasters in Steinhardt (2021). tasks for which leakage is less of a concern, such as MMLU (Hendrycks et al., 2020) and BIG-bench (BIG-bench collaboration, 2021) along with various closed-book question answering and common sense analyses. 4.2.2.

<a id="S0193"></a> Source: p.11 S0193

MMLU The Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2020) consists of a range of exam-like questions on academic subjects.

<a id="S0194"></a> Source: p.11 S0194

In Table 6, we report Chinchilla’s average 5-shot performance on MMLU (the full breakdown of results is shown in Table A6).

<a id="S0195"></a> Source: p.11 S0195

On this benchmark, Chinchilla significantly outperforms Gopher despite being much smaller, with an average accuracy of 67.6% (improving upon Gopher by 7.6%).

<a id="S0196"></a> Source: p.11 S0196

Remarkably, Chinchilla even outperforms the expert forecast for June 2023 of 63.4% accuracy (see Table 6) (Steinhardt, 2021).

<a id="S0197"></a> Source: p.11 S0197

Furthermore, Chinchilla achieves greater than 90% accuracy on 4 different individual tasks– high_school_gov_and_politics, international_law, sociology, and us_foreign_policy.

<a id="S0198"></a> Source: p.11 S0198

To our knowledge, no other model has achieved greater than 90% accuracy on a subset.

<a id="S0199"></a> Source: p.11 S0199

In Figure 6, we show a comparison to Gopher broken down by task.

<a id="S0200"></a> Source: p.11 S0200

Overall, we find that Chinchilla improves performance on the vast majority of tasks.

<a id="S0201"></a> Source: p.11 S0201

On four tasks (college_mathematics, econometrics, moral_scenarios, and formal_logic) Chinchilla underperforms Gopher, and there is no change in performance on two tasks. 4.2.3.

<a id="S0202"></a> Source: p.11 S0202

Reading comprehension On the final word prediction dataset LAMBADA (Paperno et al., 2016), Chinchilla achieves 77.4% accuracy, compared to 74.5% accuracy from Gopher and 76.6% from MT-NLG 530B (see Table 7).

<a id="S0203"></a> Source: p.11 S0203

On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher, improving accuracy by more than 10% in both cases—see Table 7. 4.2.4.

<a id="S0204"></a> Source: p.11 S0204

BIG-bench We analysed Chinchilla on the same set of BIG-bench tasks (BIG-bench collaboration, 2021) reported in Rae et al. (2021).

<a id="S0205"></a> Source: p.11 S0205

Similar to what we observed in MMLU, Chinchilla outperforms Gopher on the vast majority of tasks (see Figure 7).

<a id="S0206"></a> Source: p.11 S0206

We find that Chinchilla improves the average performance by 10.7%, reaching an accuracy of 65.1% versus 54.4% for Gopher.

<a id="S0207"></a> Source: p.11 S0207

Of the 62 tasks we consider, Chinchilla performs worse than Gopher on only four—crash_blossom, dark_humor_detection, 11

Page 12

<a id="S0208"></a> Source: p.12 S0208

scitamehtam_egelloc scirtemonoce soiranecs_larom cigol_lamrof sciteneg_lacidem gninrael_enihcam snoitaler_cilbup stcaf_labolg scihte_ssenisub gnireenigne_lacirtcele ecneics_retupmoc_egelloc snoigiler_dlrow yrotsih_su_loohcs_hgih ygolohcysp_loohcs_hgih tnemeganam ecneics_retupmoc_loohcs_hgih gnitekram scisyhp_loohcs_hgih scimonoceorcam_loohcs_hgih ygoloicos scitilop_dna_tnemnrevog_loohcs_hgih yrotsih_naeporue_loohcs_hgih noitirtun enicidem_egelloc ymonortsa seicallaf_lacigol ygolohcysp_lanoisseforp suoenallecsim ecnedurpsiruj egdelwonk_lacinilc yhpargoeg_loohcs_hgih ygoloib_loohcs_hgih ygoloib_egelloc yrtsimehc_egelloc yrotsih_dlrow_loohcs_hgih ycilop_ngierof_su ygoloriv yhposolihp setupsid_larom gniga_namuh ytiruces_retupmoc seiduts_ytiruces wal_lanoitanretni scimonoceorcim_loohcs_hgih scitsitats_loohcs_hgih gnitnuocca_lanoisseforp enicidem_lanoisseforp yrotsiherp yrtsimehc_loohcs_hgih scitamehtam_yratnemele arbegla_tcartsba ymotana wal_lanoisseforp ytilauxes_namuh scisyhp_egelloc scitamehtam_loohcs_hgih scisyhp_lautpecnoc 30 20 10 0 10 tnemevorpmI evitaleR rehpoG revo Figure 6 | MMLU results compared to Gopher We find that Chinchilla outperforms Gopher by 7.6% on average (see Table 6) in addition to performing better on 51/57 individual tasks, the same on 2/57, and worse on only 4/57 tasks.

<a id="S0209"></a> Source: p.12 S0209

Chinchilla Gopher GPT-3 MT-NLG 530B LAMBADA Zero-Shot 77.4 74.5 76.2 76.6 RACE-m Few-Shot 86.8 75.1 58.1 - RACE-h Few-Shot 82.3 71.6 46.8 47.9 Table 7 | Reading comprehension.

<a id="S0210"></a> Source: p.12 S0210

On RACE-h and RACE-m (Lai et al., 2017), Chinchilla considerably improves performance over Gopher.

<a id="S0211"></a> Source: p.12 S0211

Note that GPT-3 and MT-NLG 530B use a different prompt format than we do on RACE-h/m, so results are not comparable to Gopher and Chinchilla.

<a id="S0212"></a> Source: p.12 S0212

On LAMBADA (Paperno et al., 2016), Chinchilla outperforms both Gopher and MT-NLG 530B. mathematical_induction and logical_args.

<a id="S0213"></a> Source: p.12 S0213

Full accuracy results for Chinchilla can be found in Table A7. 4.2.5.

<a id="S0214"></a> Source: p.12 S0214

Common sense We evaluate Chinchilla on various common sense benchmarks: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2020), HellaSwag (Zellers et al., 2019), and BoolQ (Clark et al., 2019).

<a id="S0215"></a> Source: p.12 S0215

We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.

<a id="S0216"></a> Source: p.12 S0216

On TruthfulQA (Lin et al., 2021), Chinchilla reaches 43.6%, 58.5%, and 66.7% accuracy with 0-shot, 5-shot, and 10-shot respectively.

<a id="S0217"></a> Source: p.12 S0217

In comparison, Gopher achieved only 29.5% 0-shot and 43.7% 10-shot accuracy.

<a id="S0218"></a> Source: p.12 S0218

In stark contrast with the findings of Lin et al. (2021), the large improvements (14.1% in 0-shot accuracy) achieved by Chinchilla suggest that better modelling of the pre-training data alone can lead to substantial improvements on this benchmark. 12

Page 13

<a id="S0219"></a> Source: p.13 S0219

mossolb_hsarc noitceted_romuh_krad noitcudni_lacitamehtam sgra_lacigol nosj_egdelwonk_lareneg eciohc_elpitlum_sesnes_snagro_namuH noitagen_smsigollys_seicallaf_lamrof snwonknu_nwonk etagivan ytiugibma_ecnetnes ytilibissimrep_larom noitingocer_tnetni noitacifitnedi_ynori ytiralop_deliatne notabrepyh snoitpecnocsim ytilaitnesse_noitamrofni_gnitaulave noitcartsba_seitiralimis gninosaer_cimetsipe gninosaer_ysatnaf tnereffid_ro_emas_golaid_eivom yhwoniw stpecnoc_levon noitciderp_rekram_esruocsid aqygetarts tnemgduj_lasuac egdelwonk_udnih ssendetaler_esarhp eriannoitseuq_tnemngila stcejbo_deroloc_tuoba_gninosaer gnidnatsrednu_etad elbat_a_ni_sniugnep noitceted_hceeps_fo_erugif q_noitaugibmasid serutacilpmi SKRANS seman_niur noitceted_ycallaf_lacigol smsinorhcana elzzup_dirg_cigol esnes_elddir tnemliatne_citylana noitceles_noitseuq rammarg_sdrow_esnesnon cm_scisyhp stnemgduj_laciripme gnidnatsrednu_strops ia_ssarc noitiutni_lacisyhp laidemit snoitaler_ticilpmi sbrevorp_hsilgne iln_sa_snoitisoppuserp noitadnemmocer_eivom selbaf_gnidnatsrednu naeloob_rohpatem secneuqes_laropmet ecneuqes_lacigol rohpatem_ddo_yfitnedi noisneherpmoc_gnidaer_erg tuo_eno_ddo ytiralimis_lacigolana 120 100 80 60 40 20 0 20 tnemevorpmI evitaleR rehpoG revo Figure 7 | BIG-bench results compared to Gopher Chinchilla out performs Gopher on all but four BIG-bench tasks considered.

<a id="S0220"></a> Source: p.13 S0220

Closed-book question answering Results on closed-book question answering benchmarks are reported in Table 9.

<a id="S0221"></a> Source: p.13 S0221

On the Natural Questions dataset (Kwiatkowski et al., 2019), Chinchilla achieves new closed-book SOTA accuracies: 31.5% 5-shot and 35.5% 64-shot, compared to 21% and 28% respectively, for Gopher.

<a id="S0222"></a> Source: p.13 S0222

On TriviaQA (Joshi et al., 2017) we show results for both the filtered (previously used in retrieval and open-book work) and unfiltered set (previously used in large language model evaluations).

<a id="S0223"></a> Source: p.13 S0223

In both cases, Chinchilla substantially out performs Gopher.

<a id="S0224"></a> Source: p.13 S0224

On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.

<a id="S0225"></a> Source: p.13 S0225

On the unfiltered set, Chinchilla outperforms GPT-3—see Table 9. 4.2.7.

<a id="S0226"></a> Source: p.13 S0226

Gender bias and toxicity Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).

<a id="S0227"></a> Source: p.13 S0227

We expect Chinchilla to carry risks similar to Gopher because Chinchilla is trained on the same data, Chinchilla Gopher GPT-3 MT-NLG 530B Supervised SOTA HellaSWAG 80.8% 79.2% 78.9% 80.2% 93.9% PIQA 81.8% 81.8% 81.0% 82.0% 90.1% Winogrande 74.9% 70.1% 70.2% 73.0% 91.3% SIQA 51.3% 50.6% - - 83.2% BoolQ 83.7% 79.3% 60.5% 78.2% 91.4% Table 8 | Zero-shot comparison on Common Sense benchmarks.

<a id="S0228"></a> Source: p.13 S0228

We show a comparison between Chinchilla, Gopher, and MT-NLG 530B on various Common Sense benchmarks.

<a id="S0229"></a> Source: p.13 S0229

We see that Chinchilla matches or outperforms Gopher and GPT-3 on all tasks.

<a id="S0230"></a> Source: p.13 S0230

On all but one Chinchilla outperforms the much larger MT-NLG 530B model. 13

Page 14

<a id="S0231"></a> Source: p.14 S0231

Method Chinchilla Gopher GPT-3 SOTA (open book) 0-shot 16.6% 10.1% 14.6% Natural Questions (dev) 5-shot 31.5% 24.5% - 54.4% 64-shot 35.5% 28.2% 29.9% 0-shot 67.0% 52.8% 64.3 % TriviaQA (unfiltered, test) 5-shot 73.2% 63.6% - - 64-shot 72.3% 61.3% 71.2% 0-shot 55.4% 43.5% - TriviaQA (filtered, dev) 5-shot 64.1% 57.0% - 72.5% 64-shot 64.6% 57.2% - Table 9 | Closed-book question answering.

<a id="S0232"></a> Source: p.14 S0232

For Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017), Chinchilla outperforms Gopher in all cases.

<a id="S0233"></a> Source: p.14 S0233

On Natural Questions, Chinchilla outperforms GPT-3.

<a id="S0234"></a> Source: p.14 S0234

On TriviaQA we show results on two different evaluation sets to allow for comparison to GPT-3 and to open book SOTA (FiD + Distillation (Izacard and Grave, 2020)). albeit with slightly different relative weights, and because it has a similar architecture.

<a id="S0235"></a> Source: p.14 S0235

Here, we examine gender bias (particularly gender and occupation bias) and generation of toxic language.

<a id="S0236"></a> Source: p.14 S0236

We select a few common evaluations to highlight potential issues, but stress that our evaluations are not comprehensive and much work remains to understand, evaluate, and mitigate risks in LLMs.

<a id="S0237"></a> Source: p.14 S0237

As discussed in Rae et al. (2021), large language models reflect contemporary and historical discourse about different groups (such as gender groups) from their training dataset, and we expect the same to be true for Chinchilla.

<a id="S0238"></a> Source: p.14 S0238

Here, we test if potential gender and occupation biases manifest in unfair outcomes on coreference resolutions, using the Winogender dataset (Rudinger et al., 2018) in a zero-shot setting.

<a id="S0239"></a> Source: p.14 S0239

Winogender tests whether a model can correctly determine if a pronoun refers to different occupation words.

<a id="S0240"></a> Source: p.14 S0240

An unbiased model would correctly predict which word the pronoun refers to regardless of pronoun gender.

<a id="S0241"></a> Source: p.14 S0241

We follow the same setup as in Rae et al. (2021) (described further in Section H.3).

<a id="S0242"></a> Source: p.14 S0242

As shown in Table 10, Chinchilla correctly resolves pronouns more frequently than Gopher across all groups.

<a id="S0243"></a> Source: p.14 S0243

Interestingly, the performance increase is considerably smaller for male pronouns (increase of 3.2%) than for female or neutral pronouns (increases of 8.3% and 9.2% respectively).

<a id="S0244"></a> Source: p.14 S0244

We also consider gotcha examples, in which the correct pronoun resolution contradicts gender stereotypes (determined by labor statistics).

<a id="S0245"></a> Source: p.14 S0245

Again, we see that Chinchilla resolves pronouns more accurately than Gopher.

<a id="S0246"></a> Source: p.14 S0246

When breaking up examples by male/female gender and gotcha/not gotcha, the largest improvement is on female gotcha examples (improvement of 10%).

<a id="S0247"></a> Source: p.14 S0247

Thus, though Chinchilla uniformly overcomes gender stereotypes for more coreference examples than Gopher, the rate of improvement is higher for some pronouns than others, suggesting that the improvements conferred by using a more compute-optimal model can be uneven.

<a id="S0248"></a> Source: p.14 S0248

Language models are capable of generating toxic language—including insults, hate speech, profanities and threats (Gehman et al., 2020; Rae et al., 2021).

<a id="S0249"></a> Source: p.14 S0249

While toxicity is an umbrella term, and its evaluation in LMs comes with challenges (Welbl et al., 2021; Xu et al., 2021), automatic classifier scores can provide an indication for the levels of harmful text that a LM generates.

<a id="S0250"></a> Source: p.14 S0250

Rae et al. (2021) found that improving language modelling loss by increasing the number of model parameters has only a negligible effect on toxic text generation (unprompted); here we analyze 14

Page 15

<a id="S0251"></a> Source: p.15 S0251

Chinchilla Gopher Chinchilla Gopher All 78.3% 71.4% Male gotcha 62.5% 59.2% Male 71.2% 68.0% Male not gotcha 80.0% 76.7% Female 79.6% 71.3% Female gotcha 76.7% 66.7% Neutral 84.2% 75.0% Female not gotcha 82.5% 75.8% Table 10 | Winogender results.

<a id="S0252"></a> Source: p.15 S0252

Left: Chinchilla consistently resolves pronouns better than Gopher.

<a id="S0253"></a> Source: p.15 S0253

Right: Chinchilla performs better on examples which contradict gender stereotypes (gotcha examples).

<a id="S0254"></a> Source: p.15 S0254

However, difference in performance across groups suggests Chinchilla exhibits bias. whether the same holds true for a lower LM loss achieved via more compute-optimal training.

<a id="S0255"></a> Source: p.15 S0255

Similar to the protocol of Rae et al. (2021), we generate 25,000 unprompted samples from Chinchilla, and compare their PerspectiveAPI toxicity score distribution to that of Gopher-generated samples.

<a id="S0256"></a> Source: p.15 S0256

Several summary statistics indicate an absence of major differences: the mean (median) toxicity score for Gopher is 0.081 (0.064), compared to 0.087 (0.066) for Chinchilla, and the 95th percentile scores are 0.230 for Gopher, compared to 0.238 for Chinchilla.

<a id="S0257"></a> Source: p.15 S0257

That is, the large majority of generated samples are classified as non-toxic, and the difference between the models is negligible.

<a id="S0258"></a> Source: p.15 S0258

In line with prior findings (Rae et al., 2021), this suggests that toxicity levels in unconditional text generation are largely independent of the model quality (measured in language modelling loss), i.e. that better models of the training dataset are not necessarily more toxic. 5.

<a id="S0259"></a> Source: p.15 S0259

Discussion & Conclusion The trend so far in large language model training has been to increase the model size, often without increasing the number of training tokens.

<a id="S0260"></a> Source: p.15 S0260

The largest dense transformer, MT-NLG 530B, is now over 3× larger than GPT-3’s 170 billion parameters from just two years ago.

<a id="S0261"></a> Source: p.15 S0261

However, this model, as well as the majority of existing large models, have all been trained for a comparable number of tokens—around 300 billion.

<a id="S0262"></a> Source: p.15 S0262

While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget.

<a id="S0263"></a> Source: p.15 S0263

We propose three predictive approaches towards optimally setting model size and training duration, based on the outcome of over 400 training runs.

<a id="S0264"></a> Source: p.15 S0264

All three approaches predict that Gopher is substantially over-sized and estimate that for the same compute budget a smaller model trained on more data will perform better.

<a id="S0265"></a> Source: p.15 S0265

We directly test this hypothesis by training Chinchilla, a 70B parameter model, and show that it outperforms Gopher and even larger models on nearly every measured evaluation task.

<a id="S0266"></a> Source: p.15 S0266

Whilst our method allows us to make predictions on how to scale large models when given additional compute, there are several limitations.

<a id="S0267"></a> Source: p.15 S0267

Due to the cost of training large models, we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests at intermediate scales.

<a id="S0268"></a> Source: p.15 S0268

Furthermore, we assume that the efficient computational frontier can be described by a power-law relationship between the compute budget, model size, and number of training tokens.

<a id="S0269"></a> Source: p.15 S0269

However, we observe some concavity in log (cid:0)𝑁 (cid:1) at high compute budgets 𝑜𝑝𝑡 (see Appendix E).

<a id="S0270"></a> Source: p.15 S0270

This suggests that we may still be overestimating the optimal size of large models.

<a id="S0271"></a> Source: p.15 S0271

Finally, the training runs for our analysis have all been trained on less than an epoch of data; future work may consider the multiple epoch regime.

<a id="S0272"></a> Source: p.15 S0272

Despite these limitations, the comparison of Chinchilla to Gopher validates our performance predictions, that have thus enabled training a better (and more 15

Page 16

<a id="S0273"></a> Source: p.16 S0273

lightweight) model at the same compute budget.

<a id="S0274"></a> Source: p.16 S0274

Though there has been significant recent work allowing larger and larger models to be trained, our analysis suggests an increased focus on dataset scaling is needed.

<a id="S0275"></a> Source: p.16 S0275

Speculatively, we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality.

<a id="S0276"></a> Source: p.16 S0276

This calls for responsibly collecting larger datasets with a high focus on dataset quality.

<a id="S0277"></a> Source: p.16 S0277

Larger datasets will require extra care to ensure train-test set overlap is properly accounted for, both in the language modelling loss but also with downstream tasks.

<a id="S0278"></a> Source: p.16 S0278

Finally, training for trillions of tokens introduces many ethical and privacy concerns.

<a id="S0279"></a> Source: p.16 S0279

Large datasets scraped from the web will contain toxic language, biases, and private information.

<a id="S0280"></a> Source: p.16 S0280

With even larger datasets being used, the quantity (if not the frequency) of such information increases, which makes dataset introspection all the more important.

<a id="S0281"></a> Source: p.16 S0281

Chinchilla does suffer from bias and toxicity but interestingly it seems less affected than Gopher.

<a id="S0282"></a> Source: p.16 S0282

Better understanding how performance of large language models and toxicity interact is an important future research question.

<a id="S0283"></a> Source: p.16 S0283

While we have applied our methodology towards the training of auto-regressive language models, we expect that there is a similar trade-off between model size and the amount of data in other modalities.

<a id="S0284"></a> Source: p.16 S0284

As training large models is very expensive, choosing the optimal model size and training steps beforehand is essential.

<a id="S0285"></a> Source: p.16 S0285

The methods we propose are easy to reproduce in new settings. 6.

<a id="S0286"></a> Source: p.16 S0286

Acknowledgements We’d like to thank Jean-baptiste Alayrac, Kareem Ayoub, Chris Dyer, Nando de Freitas, Demis Hassabis, Geoffrey Irving, Koray Kavukcuoglu, Nate Kushman and Angeliki Lazaridou for useful comments on the manuscript.

<a id="S0287"></a> Source: p.16 S0287

We’d like to thank Andy Brock, Irina Higgins, Michela Paganini, Francis Song, and other colleagues at DeepMind for helpful discussions.

<a id="S0288"></a> Source: p.16 S0288

We are also very grateful to the JAX and XLA team for their support and assistance.

<a id="S0289"></a> Source: p.16 S0289

Efficient Large Scale Language Modeling with Mixtures of Experts. arXiv:2112.10684, 2021. E. M.

<a id="S0290"></a> Source: p.16 S0290

On the dangers of stochastic parrots: Can language models be too big?

<a id="S0291"></a> Source: p.16 S0291

In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.

<a id="S0292"></a> Source: p.16 S0292

Beyond the imitation game: Measuring and extrapolating the capabilities of language models.

<a id="S0293"></a> Source: p.16 S0293

URL https://github.com/google/BIG-bench/. Y.

<a id="S0294"></a> Source: p.16 S0294

PIQA: Reasoning about physical commonsense in natural language.

<a id="S0295"></a> Source: p.16 S0295

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439, 2020. S.

<a id="S0296"></a> Source: p.16 S0296

Improving language models by retrieving from trillions of tokens. arXiv 2112.04426, 2021. 16

Page 17

<a id="S0297"></a> Source: p.17 S0297

JAX: composable transformations of Python+NumPy programs. 2018.

<a id="S0298"></a> Source: p.17 S0298

Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.

<a id="S0299"></a> Source: p.17 S0299

URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb49674 18bfb8ac142f64a-Paper.pdf. S.

<a id="S0300"></a> Source: p.17 S0300

Convex Optimization: Algorithms and Complexity.

<a id="S0301"></a> Source: p.17 S0301

Foundations and Trends in Machine Learning, 8(3-4):231–357, 2015.

<a id="S0302"></a> Source: p.17 S0302

URL http://www.nowpublishers.com/article/Detail s/MAL-050. A.

<a id="S0303"></a> Source: p.17 S0303

Unified scaling laws for routed language models, 2022.

<a id="S0304"></a> Source: p.17 S0304

URL https://arxiv.org/abs/2202.01169. C.

<a id="S0305"></a> Source: p.17 S0305

Boolq: Exploring the surprising difficulty of natural yes/no questions.

<a id="S0306"></a> Source: p.17 S0306

In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019. N.

<a id="S0307"></a> Source: p.17 S0307

Glam: Efficient scaling of language models with mixture-of-experts, 2021.

<a id="S0308"></a> Source: p.17 S0308

URL https://arxiv.org/abs/2112.06905. W.

<a id="S0309"></a> Source: p.17 S0309

Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. L.

<a id="S0310"></a> Source: p.17 S0310

The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. S.

<a id="S0311"></a> Source: p.17 S0311

RealToxicityPrompts: Evaluating neural toxic degeneration in language models.

<a id="S0312"></a> Source: p.17 S0312

In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, Nov. 2020.

<a id="S0313"></a> Source: p.17 S0313

Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.301.

<a id="S0314"></a> Source: p.17 S0314

URL https://aclanthology.org/2 020.findings-emnlp.301. K.

<a id="S0315"></a> Source: p.17 S0315

REALM: Retrieval-augmented language model pre-training, 2020. D.

<a id="S0316"></a> Source: p.17 S0316

Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. T.

<a id="S0317"></a> Source: p.17 S0317

URL http: //github.com/deepmind/dm-haiku. 17

Page 18

<a id="S0318"></a> Source: p.18 S0318

Robust Estimation of a Location Parameter.

<a id="S0319"></a> Source: p.18 S0319

The Annals of Mathematical Statistics, 35 (1):73–101, Mar. 1964.

<a id="S0320"></a> Source: p.18 S0320

ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/1177703732.

<a id="S0321"></a> Source: p.18 S0321

URL https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol ume-35/issue-1/Robust-Estimation-of-a-Location-Parameter/10.1214/aoms/11 77703732.full. G.

<a id="S0322"></a> Source: p.18 S0322

Distilling knowledge from reader to retriever for question answering, 2020. M.

<a id="S0323"></a> Source: p.18 S0323

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017. N. P.

<a id="S0324"></a> Source: p.18 S0324

In-datacenter performance analysis of a tensor processing unit.

<a id="S0325"></a> Source: p.18 S0325

In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, page 1–12, New York, NY, USA, 2017.

<a id="S0326"></a> Source: p.18 S0326

ISBN 9781450348928. doi: 10.1145/3079856.3080246.

<a id="S0327"></a> Source: p.18 S0327

URL https://doi.org/10.1145/3079856.3080246. J.

<a id="S0328"></a> Source: p.18 S0328

Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. D. P.

<a id="S0329"></a> Source: p.18 S0329

Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. T.

<a id="S0330"></a> Source: p.18 S0330

SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. T.

<a id="S0331"></a> Source: p.18 S0331

Natural questions: a benchmark for question answering research.

<a id="S0332"></a> Source: p.18 S0332

Transactions of the Association of Computational Linguistics, 2019. G.

<a id="S0333"></a> Source: p.18 S0333

RACE: Large-scale ReAding comprehension dataset from examinations.

<a id="S0334"></a> Source: p.18 S0334

In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, Sept. 2017.

<a id="S0335"></a> Source: p.18 S0335

Association for Computational Linguistics. doi: 10.18653/v1/D17-1082.

<a id="S0336"></a> Source: p.18 S0336

URL https://aclanthology.org/D17-1082. Y.

<a id="S0337"></a> Source: p.18 S0337

The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. P.

<a id="S0338"></a> Source: p.18 S0338

Retrieval-augmented generation for knowledge-intensive nlp tasks.

<a id="S0339"></a> Source: p.18 S0339

In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. 18

Page 19

<a id="S0340"></a> Source: p.19 S0340

Jurassic-1: Technical details and evaluation.

<a id="S0341"></a> Source: p.19 S0341

TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. I.

<a id="S0342"></a> Source: p.19 S0342

In International Conference on Learning Representations, 2019.

<a id="S0343"></a> Source: p.19 S0343

URL https://openreview.net/forum?id=Bkg6RiCqY7. S.

<a id="S0344"></a> Source: p.19 S0344

An empirical model of large-batch training, 2018. S.

<a id="S0345"></a> Source: p.19 S0345

International Conference on Learning Representations, 2017. M.

<a id="S0346"></a> Source: p.19 S0346

In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. J.

<a id="S0347"></a> Source: p.19 S0347

Updating Quasi-Newton Matrices with Limited Storage.

<a id="S0348"></a> Source: p.19 S0348

Mathematics of Computation, 35(151):773–782, 1980.

<a id="S0349"></a> Source: p.19 S0349

URL https://www.jstor. org/stable/2006193. D.

<a id="S0350"></a> Source: p.19 S0350

The LAMBADA dataset: Word prediction requiring a broad discourse context, 2016. J.

<a id="S0351"></a> Source: p.19 S0351

Scaling language models: Methods, analysis & insights from training Gopher. arXiv 2112.11446, 2021. J. W.

<a id="S0352"></a> Source: p.19 S0352

Compressive transformers for long-range sequence modelling.

<a id="S0353"></a> Source: p.19 S0353

Advances in Neural Information Processing Systems, 33:6154–6158, 2020. C.

<a id="S0354"></a> Source: p.19 S0354

Exploring the limits of transfer learning with a unified text-to-text transformer.

<a id="S0355"></a> Source: p.19 S0355

Journal of Machine Learning Research, 21(140):1–67, 2020a.

<a id="S0356"></a> Source: p.19 S0356

URL http://jmlr.org/papers/v21/20-074.html. C.

<a id="S0357"></a> Source: p.19 S0357

Exploring the limits of transfer learning with a unified text-to-text transformer.

<a id="S0358"></a> Source: p.19 S0358

Journal of Machine Learning Research, 21(140):1–67, 2020b. S.

<a id="S0359"></a> Source: p.19 S0359

Zero: Memory optimizations toward training trillion parameter models.

<a id="S0360"></a> Source: p.19 S0360

In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16.

Page 20

<a id="S0361"></a> Source: p.20 S0361

Monro. A Stochastic Approximation Method.

<a id="S0362"></a> Source: p.20 S0362

The Annals of Mathematical Statistics, 22(3):400–407, Sept. 1951. R.

<a id="S0363"></a> Source: p.20 S0363

In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, June 2018.

<a id="S0364"></a> Source: p.20 S0364

Association for Computational Linguistics. K.

<a id="S0365"></a> Source: p.20 S0365

Winogrande: An adversarial winograd schema challenge at scale.

<a id="S0366"></a> Source: p.20 S0366

In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020. M.

<a id="S0367"></a> Source: p.20 S0367

SocialIQA: Commonsense reasoning about social interactions.

<a id="S0368"></a> Source: p.20 S0368

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019. C. J.

<a id="S0369"></a> Source: p.20 S0369

Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018. J. W.

<a id="S0370"></a> Source: p.20 S0370

Approximation rates for neural networks with general activation functions.

<a id="S0371"></a> Source: p.20 S0371

Neural Networks, 128:313–321, Aug. 2020.

<a id="S0372"></a> Source: p.20 S0372

URL https://www.sciencedirect.com/scienc e/article/pii/S0893608020301891. S.

<a id="S0373"></a> Source: p.20 S0373

Using Deepspeed and Megatron to Train Megatronturing NLG 530b, A Large-Scale Generative Language Model. arXiv preprint arXiv:2201.11990, 2022. J.

<a id="S0374"></a> Source: p.20 S0374

Updates and lessons from AI forecasting, 2021.

<a id="S0375"></a> Source: p.20 S0375

URL https://bounded-regret.g host.io/ai-forecasting/. Y.

<a id="S0376"></a> Source: p.20 S0376

Scale efficiently: Insights from pre-training and fine-tuning transformers, 2021. R.

<a id="S0377"></a> Source: p.20 S0377

LaMDA: Language models for dialog applications, 2022. A.

<a id="S0378"></a> Source: p.20 S0378

In Advances in neural information processing systems, pages 5998–6008, 2017. L.

<a id="S0379"></a> Source: p.20 S0379

Ethical and social risks of harm from language models. arXiv submission, 2021. 20

Page 21

<a id="S0380"></a> Source: p.21 S0380

Challenges in detoxifying language models.

<a id="S0381"></a> Source: p.21 S0381

In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, Punta Cana, Dominican Republic, Nov. 2021.

<a id="S0382"></a> Source: p.21 S0382

Association for Computational Linguistics.

<a id="S0383"></a> Source: p.21 S0383

URL https://aclanthology.org/2021. findings-emnlp.210. A.

<a id="S0384"></a> Source: p.21 S0384

Detoxifying language models risks marginalizing minority voices.

<a id="S0385"></a> Source: p.21 S0385

In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397, Online, June 2021.

<a id="S0386"></a> Source: p.21 S0386

Association for Computational Linguistics. doi: 10.18653/v1/2021 .naacl-main.190.

<a id="S0387"></a> Source: p.21 S0387

URL https://aclanthology.org/2021.naacl-main.190. G.

<a id="S0388"></a> Source: p.21 S0388

Tuning large neural networks via zero-shot hyperparameter transfer.

<a id="S0389"></a> Source: p.21 S0389

Vaughan, editors, Advances in Neural Information Processing Systems, 2021.

<a id="S0390"></a> Source: p.21 S0390

URL https://openreview.net/forum?id=Bx6qKuBM2AD. R.

<a id="S0391"></a> Source: p.21 S0391

HellaSwag: Can a machine really finish your sentence?

<a id="S0392"></a> Source: p.21 S0392

In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. G.

<a id="S0393"></a> Source: p.21 S0393

Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model.

<a id="S0394"></a> Source: p.21 S0394

Garnett, editors, Advances in Neural Information Processing Systems, volume 32.

<a id="S0395"></a> Source: p.21 S0395

URL https: //proceedings.neurips.cc/paper/2019/file/e0eacd983971634327ae1819ea8b621 4-Paper.pdf. B.

<a id="S0396"></a> Source: p.21 S0396

Designing effective sparse expert models, 2022. 21

Page 22

<a id="S0397"></a> Source: p.22 S0397

Training dataset In Table A1 we show the training dataset makeup used for Chinchilla and all scaling runs.

<a id="S0398"></a> Source: p.22 S0398

Note that both the MassiveWeb and Wikipedia subsets are both used for more than one epoch.

<a id="S0399"></a> Source: p.22 S0399

Disk Size Documents Sampling proportion Epochs in 1.4T tokens MassiveWeb 1.9 TB 604M 45% (48%) 1.24 Books 2.1 TB 4M 30% (27%) 0.75 C4 0.75 TB 361M 10% (10%) 0.77 News 2.7 TB 1.1B 10% (10%) 0.21 GitHub 3.1 TB 142M 4% (3%) 0.13 Wikipedia 0.001 TB 6M 1% (2%) 3.40 Table A1 | MassiveText data makeup.

<a id="S0400"></a> Source: p.22 S0400

For each subset of MassiveText, we list its total disk size, the number of documents and the sampling proportion used during training—we use a slightly different distribution than in Rae et al. (2021) (shown in parenthesis).

<a id="S0401"></a> Source: p.22 S0401

In the rightmost column show the number of epochs that are used in 1.4 trillion tokens. B.

<a id="S0402"></a> Source: p.22 S0402

Optimal cosine cycle length One key assumption is made on the cosine cycle length and the corresponding learning rate drop (we use a 10× learning rate decay in line with Rae et al. (2021)).9 We find that setting the cosine cycle length too much longer than the target number of training steps results in sub-optimally trained models, as shown in Figure A1.

<a id="S0403"></a> Source: p.22 S0403

As a result, we assume that an optimally trained model will have the cosine cycle length correctly calibrated to the maximum number of steps, given the FLOP budget; we follow this rule in our main analysis. C.

<a id="S0404"></a> Source: p.22 S0404

Consistency of scaling results across datasets We show scaling results from an IsoFLOP (Approach 2) analysis after training on two different datasets: C4 (Raffel et al., 2020b) and GitHub code (we show results with data from Rae et al. (2021)), results are shown in Table A2.

<a id="S0405"></a> Source: p.22 S0405

For both set of experiments using subsets of MassiveText, we use the same tokenizer as the MassiveText experiments.

<a id="S0406"></a> Source: p.22 S0406

We find that the scaling behaviour on these datasets is very similar to what we found on MassiveText, as shown in Figure A2 and Table A2.

<a id="S0407"></a> Source: p.22 S0407

This suggests that our results are independent of the dataset as long as one does not train for more than one epoch. 9We find the difference between decaying by 10× and decaying to 0.0 (over the same number of steps) to be small, though decaying by a factor of 10× to be slightly more performant.

<a id="S0408"></a> Source: p.22 S0408

Decaying by less (5×) is clearly worse. 22

Page 23

<a id="S0409"></a> Source: p.23 S0409

1.0 0.8 0.6 0.4 0.2 0.0 0 2 4 6 8 Million Sequences RL xaM/etaR gninraeL 3.00 2.95 2.90 2.85 2.80 2.75 2.70 0 2 4 6 8 Million Sequences ssoL gniniarT 3.20 3.15 3.10 3.05 3.00 2.95 2.90 2.85 2.80 0 2 4 6 Million Sequences ssoL 4C Cosine Cycle Length 1.0× num. steps 1.1× num. steps 1.25× num. steps 1.5× num. steps 2.0× num. steps 5.0× num. steps 1.0 0.8 0.6 0.4 0.2 0.0 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences RL xaM/etaR gninraeL 3.00 2.95 2.90 2.85 2.80 2.75 2.70 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences ssoL gniniarT 3.20 3.15 3.10 3.05 3.00 2.95 2.90 2.85 2.80 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences ssoL 4C Figure A1 | Grid over cosine cycle length.

<a id="S0410"></a> Source: p.23 S0410

We show 6 curves with the cosine cycle length set to 1, 1.1, 1.25, 1.5, 2, and 5× longer than the target number of training steps.

<a id="S0411"></a> Source: p.23 S0411

When the cosine cycle length is too long, and the learning rate does not drop appropriately, then performance is impaired.

<a id="S0412"></a> Source: p.23 S0412

We find that overestimating the number of training steps beyond 25% leads to clear drops in performance.

<a id="S0413"></a> Source: p.23 S0413

We show results where we have set the number of training steps to two different values (top and bottom). 3.2 3.0 2.8 2.6 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT 4C 1T 100B 10B 1e19 1B 1e20 6e20 100M 1e21 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 73B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.3T 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT buHtiG 1e19 1T 1e20 6e20 100B 1e21 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 59B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.6T Figure A2 | C4 and GitHub IsoFLOP curves.

<a id="S0414"></a> Source: p.23 S0414

Using the C4 dataset (Raffel et al., 2020b) and a GitHub dataset (Rae et al., 2021), we generate 4 IsoFLOP profiles and show the parameter and token count scaling, as in Figure 3.

<a id="S0415"></a> Source: p.23 S0415

Scaling coefficients are shown in Table A2. 23

Page 24

<a id="S0416"></a> Source: p.24 S0416

Approach Coef. 𝑎 where 𝑁 ∝ 𝐶𝑎 Coef. 𝑏 where 𝐷 ∝ 𝐶𝑏 𝑜𝑝𝑡 𝑜𝑝𝑡 C4 0.50 0.50 GitHub 0.53 0.47 Kaplan et al. (2020) 0.73 0.27 Table A2 | Estimated parameter and data scaling with increased training compute on two alternate datasets.

<a id="S0417"></a> Source: p.24 S0417

The listed values are the exponents, 𝑎 and 𝑏, on the relationship 𝑁 𝑜𝑝𝑡 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.

<a id="S0418"></a> Source: p.24 S0418

Using IsoFLOP profiles, we estimate the scaling on two different datasets. 𝑜𝑝𝑡 D.

<a id="S0419"></a> Source: p.24 S0419

Approach 1: Fixing model sizes and varying training sequences We use a maximum learning rate of 2 × 10−4 for the smallest models and 1.25 × 10−4 for the largest models.

<a id="S0420"></a> Source: p.24 S0420

In all cases, the learning rate drops by a factor of 10× during training, using a cosine schedule.

<a id="S0421"></a> Source: p.24 S0421

We make the assumption that the cosine cycle length should be approximately matched to the number of training steps.

<a id="S0422"></a> Source: p.24 S0422

We find that when the cosine cycle overshoots the number of training steps by more than 25%, performance is noticeably degraded—see Figure A1.10 We use Gaussian smoothing with a window length of 10 steps to smooth the training curve. D.2.

<a id="S0423"></a> Source: p.24 S0423

Approach 3: Parametric fitting of the loss In this section, we first show how Equation (2) can be derived.

<a id="S0424"></a> Source: p.24 S0424

We repeat the equation below for clarity, 𝐴 𝐵 𝐿ˆ (𝑁, 𝐷) (cid:44) 𝐸 + + , (5) 𝑁𝛼 𝐷𝛽 based on a decomposition of the expected risk between a function approximation term and an optimisation suboptimality term.

<a id="S0425"></a> Source: p.24 S0425

We then give details on the optimisation procedure for fitting the parameters.

<a id="S0426"></a> Source: p.24 S0426

Formally, we consider the task of predicting the next token 𝑦 ∈ Y based on the previous tokens in a sequence 𝑥 ∈ Y𝑠, with 𝑠 varying from 0 to 𝑠 max —the maximum sequence length.

<a id="S0427"></a> Source: p.24 S0427

We consider a distribution 𝑃 ∈ D (X × Y) of tokens in Y and their past in X. A predictor 𝑓 : X → D (Y) computes the probability of each token given the past sequence.

<a id="S0428"></a> Source: p.24 S0428

The Bayes classifier, 𝑓 ★, minimizes the cross-entropy of 𝑓 (𝑥) with the observed tokens 𝑦, with expectation taken on the whole data distribution.

<a id="S0429"></a> Source: p.24 S0429

We let 𝐿 be the expected risk 𝐿( 𝑓 ) (cid:44) 𝔼[log 𝑓 (𝑥) 𝑦 ], and set 𝑓 ★ (cid:44) argmin 𝐿( 𝑓 ). (6) 𝑓 ∈F(X,D (Y)) The set of all transformers of size 𝑁, that we denote H , forms a subset of all functions that map 𝑁 sequences to distributions of tokens X → D (Y).

<a id="S0430"></a> Source: p.24 S0430

Fitting a transformer of size 𝑁 on the expected risk 𝐿( 𝑓 ) amounts to minimizing such risk on a restricted functional space 𝑓 (cid:44) argmin 𝐿( 𝑓 ). (7) 𝑁 𝑓 ∈H𝑁 When we observe a dataset (𝑥 𝑖 , 𝑦 𝑖 ) 𝑖𝑖∈[1,𝐷] of size 𝐷, we do not have access to 𝔼𝑃 , but instead to the empirical expectation 𝔼 ˆ 𝐷 over the empirical distribution 𝑃ˆ 𝐷 .

<a id="S0431"></a> Source: p.24 S0431

What happens when we are given 𝐷 10This further emphasises the point of not only determining model size, but also training length before training begins. 24

Page 25

<a id="S0432"></a> Source: p.25 S0432

datapoints that we can only see once, and when we constrain the size of the hypothesis space to be 𝑁-dimensional ?

<a id="S0433"></a> Source: p.25 S0433

We are making steps toward minimizing the empirical risk within a finite-dimensional functional space H : 𝑁 𝐿ˆ 𝐷 ( 𝑓 ) (cid:44) 𝔼 ˆ 𝐷 [log 𝑓 (𝑥) 𝑦 ], setting 𝑓 ˆ 𝑁,𝐷 (cid:44) argmin 𝐿ˆ 𝐷 ( 𝑓 ). (8) 𝑓 ∈H𝑁 We are never able to obtain 𝑓 ˆ as we typically perform a single epoch over the dataset of size 𝐷. 𝑁,𝐷 Instead, be obtain 𝑓 ¯ , which is the result of applying a certain number of gradient steps based on 𝑁,𝐷 the 𝐷 datapoints—the number of steps to perform depends on the gradient batch size, for which we use well-tested heuristics.

<a id="S0434"></a> Source: p.25 S0434

Using the Bayes-classifier 𝑓 ★, the expected-risk minimizer 𝑓 and the “single-epoch empirical-risk 𝑁 minimizer” 𝑓 ¯ , we can finally decompose the loss 𝐿(𝑁, 𝐷) into 𝑁,𝐷 𝐿(𝑁, 𝐷) (cid:44) 𝐿( 𝑓 ¯ ) = 𝐿( 𝑓 ★) + (cid:0)𝐿( 𝑓 ) − 𝐿( 𝑓 ★) (cid:1) + (cid:0)𝐿( 𝑓 ¯ ) − 𝐿( 𝑓 ) (cid:1) . (9) 𝑁,𝐷 𝑁 𝑁,𝐷 𝑁 The loss comprises three terms: the Bayes risk, i.e. the minimal loss achievable for next-token prediction on the full distribution 𝑃, a.k.a the “entropy of natural text.”; a functional approximation term that depends on the size of the hypothesis space; finally, a stochastic approximation term that captures the suboptimality of minimizing 𝐿ˆ instead of 𝐿, and of making a single epoch on the provided 𝐷 dataset.

<a id="S0435"></a> Source: p.25 S0435

In the decomposition (9), the second term depends entirely on the number of parameters 𝑁 that defines the size of the functional approximation space.

<a id="S0436"></a> Source: p.25 S0436

On the set of two-layer neural networks, it is expected to be proportional to 1 (Siegel and Xu, 2020).

<a id="S0437"></a> Source: p.25 S0437

Finally, 𝑁1/2 given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 (Robbins and Monro, 𝐷1/2 1951) (and may attain the bound).

<a id="S0438"></a> Source: p.25 S0438

This convergence rate is expected to be dimension free (see e.g.

<a id="S0439"></a> Source: p.25 S0439

Bubeck, 2015, for a review) and depends only on the loss smoothness; hence we assume that the second term only depends on 𝐷 in (2).

<a id="S0440"></a> Source: p.25 S0440

Empirically, we find after fitting (2) that 𝐴 𝐵 𝐿(𝑁, 𝐷) = 𝐸 + + , (10) 𝑁0.34 𝐷0.28 with 𝐸 = 1.69, 𝐴 = 406.4, 𝐵 = 410.7.

<a id="S0441"></a> Source: p.25 S0441

We note that the parameter/data coefficients are both lower than 1 ; this is expected for the data-efficiency coefficient (but far from the known lower-bound). 2 Future models and training approaches should endeavor to increase these coefficients.

<a id="S0442"></a> Source: p.25 S0442

We effectively minimize the following problem min ∑︁ Huber (cid:16) LSE(cid:0)𝑎 − 𝛼 log 𝑁 , 𝑏 − 𝛽 log 𝐷 , 𝑒(cid:1) − log 𝐿 (cid:17) , (11) 𝛿 𝑖 𝑖 𝑖 𝑎,𝑏,𝑒,𝛼,𝛽 Run 𝑖 where 𝐿𝑆𝐸 is the log-sum-exp operator.

<a id="S0443"></a> Source: p.25 S0443

We then set 𝐴, 𝐵, 𝐸 = exp(𝑎), exp(𝑏), exp(𝑒).

<a id="S0444"></a> Source: p.25 S0444

We use the LBFGS algorithm to find local minima of the objective above, started on a grid of initialisation given by: 𝛼 ∈ {0., 0.5, . . . , 2.}, 𝛽 ∈ {0., 0.5, . . . , 2.}, 𝑒 ∈ {−1., −.5, . . . , 1.}, 𝑎 ∈ {0, 5, . . . , 25}, and 𝑏 ∈ {0, 5, . . . , 25}.

<a id="S0445"></a> Source: p.25 S0445

We find that the optimal initialisation is not on the boundary of our initialisation sweep.

<a id="S0446"></a> Source: p.25 S0446

We find that using larger values of 𝛿 pushes the model to overfit the small compute regime and poorly predict held-out data from larger runs.

<a id="S0447"></a> Source: p.25 S0447

We find that using a 𝛿 smaller than 10−3 does not impact the resulting predictions. 25

Page 26

<a id="S0448"></a> Source: p.26 S0448

Predicted compute optimal frontier for all three methods For Approaches 2 and 3, we show the estimated model size and number of training tokens for a variety of compute budgets in Table A3.

<a id="S0449"></a> Source: p.26 S0449

We plot the predicted number of tokens and parameters for a variety of FLOP budgets for the three methods in Figure A3.

<a id="S0450"></a> Source: p.26 S0450

Approach 2 Approach 3 Parameters FLOPs Tokens FLOPs Tokens 400 Million 1.84e+19 7.7 Billion 2.21e+19 9.2 Billion 1 Billion 1.20e+20 20.0 Billion 1.62e+20 27.1 Billion 10 Billion 1.32e+22 219.5 Billion 2.46e+22 410.1 Billion 67 Billion 6.88e+23 1.7 Trillion 1.71e+24 4.1 Trillion 175 Billion 4.54e+24 4.3 Trillion 1.26e+24 12.0 Trillion 280 Billion 1.18e+25 7.1 Trillion 3.52e+25 20.1 Trillion 520 Billion 4.19e+25 13.4 Trillion 1.36e+26 43.5 Trillion 1 Trillion 1.59e+26 26.5 Trillion 5.65e+26 94.1 Trillion 10 Trillion 1.75e+28 292.0 Trillion 8.55e+28 1425.5 Trillion Table A3 | Estimated optimal training FLOPs and training tokens for various model sizes.

<a id="S0451"></a> Source: p.26 S0451

Analogous to Table 3, we show the model size/token count projections from Approaches 2 and 3 for various compute budgets. . 1012 1011 1010 109 108 1010 1011 1012 1013 Tokens sretemaraP Approach 1 1e+26 Approach 2 Approach 3 1e+25 Chinchilla Gopher GPT-3 1e+24 Megatron-Turing NLG 1e+23 1e+22 1e+21 1e+20 1e+19 1e+18 Figure A3 | Optimal number of tokens and parameters for a training FLOP budget.

<a id="S0452"></a> Source: p.26 S0452

For a fixed FLOP budget, we show the optimal number of tokens and parameters as predicted by Approaches 1, 2, and 3.

<a id="S0453"></a> Source: p.26 S0453

For an alternate representation, see Figure 1. D.4.

<a id="S0454"></a> Source: p.26 S0454

Small-scale comparison to Kaplan et al. (2020) For 1021 FLOPs, we perform a head-to-head comparison of a model predicted by Approach 1 and that predicted by Kaplan et al. (2020).

<a id="S0455"></a> Source: p.26 S0455

For both models, we use a batch size of 0.5M tokens and a 26

Page 27

<a id="S0456"></a> Source: p.27 S0456

maximum learning rate of 1.5 × 10−4 that decays by 10×.

<a id="S0457"></a> Source: p.27 S0457

From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.

<a id="S0458"></a> Source: p.27 S0458

From our approach 1, we estimate a 2.86 billion parameter model should be optimal.

<a id="S0459"></a> Source: p.27 S0459

We train a 4.74 billion parameter and a 2.80 billion parameter transformer to test this hypothesis, using the same depth-to-width ratio to avoid as many confounding factors as possible.

<a id="S0460"></a> Source: p.27 S0460

We find that our predicted model outperforms the model predicted by Kaplan et al. (2020) as shown in Figure A4. 2.8 2.7 2.6 2.5 2.4 2.3 2.2 0 1 2 Sequences 1e7 ssoL gniniarT 2.8 2.7 2.6 2.5 2.4 2.3 2.2 0.0 0.2 0.4 0.6 0.8 1.0 FLOPs ×1021 ssoL gniniarT Kaplan et al (2020) Approach 1 Figure A4 | Comparison to Kaplan et al. (2020) at 1021 FLOPs.

<a id="S0461"></a> Source: p.27 S0461

We train 2.80 and 4.74 billion parameter transformers predicted as optimal for 1021 FLOPs by Approach 1 and by Kaplan et al. (2020).

<a id="S0462"></a> Source: p.27 S0462

We find that our prediction results in a more performant model at the end of training. E.

<a id="S0463"></a> Source: p.27 S0463

Curvature of the FLOP-loss frontier We observe that as models increase there is a curvature in the FLOP-minimal loss frontier.

<a id="S0464"></a> Source: p.27 S0464

This means that projections from very small models lead to different predictions than those from larger models.

<a id="S0465"></a> Source: p.27 S0465

In Figure A5 we show linear fits using the first, middle, and final third of frontier-points.

<a id="S0466"></a> Source: p.27 S0466

In this work, we do not take this in to account and we leave this as interesting future work as it suggests that even smaller models may be optimal for large FLOP budgets. F.

<a id="S0467"></a> Source: p.27 S0467

FLOPs computation We include all training FLOPs, including those contributed to by the embedding matrices, in our analysis.

<a id="S0468"></a> Source: p.27 S0468

Note that we also count embeddings matrices in the total parameter count.

<a id="S0469"></a> Source: p.27 S0469

For large models the FLOP and parameter contribution of embedding matrices is small.

<a id="S0470"></a> Source: p.27 S0470

We use a factor of 2 to describe the multiply accumulate cost.

<a id="S0471"></a> Source: p.27 S0471

For the forward pass, we consider contributions from: • Embeddings – 2 × seq_len × vocab_size × d_model • Attention (Single Layer) – Key, query and value projections: 2 × 3 × seq_len × d_model × (key_size × num_heads) 27

Page 28

<a id="S0472"></a> Source: p.28 S0472

6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1017 1018 1019 1020 1021 1022 FLOPS ssol gniniarT 10000 5000 2500 1000 500 250 75 sretemaraP noilliM Figure A5 | Training curve envelopes.

<a id="S0473"></a> Source: p.28 S0473

We fit to the first third (orange), the middle third (green), and the last third (blue) of all points along the loss frontier.

<a id="S0474"></a> Source: p.28 S0474

We plot only a subset of the points. – Key @ Query logits: 2 × seq_len × seq_len × (key_size × num_heads) – Softmax: 3 × num_heads × seq_len × seq_len – Softmax @ query reductions: 2 × seq_len × seq_len × (key_size × num_heads) – Final Linear: 2 × seq_len × (key_size × num_heads) × d_model • Dense Block (Single Layer) – 2 × seq_len × (d_model × ffw_size + d_model × ffw_size) • Final Logits – 2 × seq_len × d_model × vocab_size • Total forward pass FLOPs: embeddings+num_layers× (total_attention+dense_block) + logits As in Kaplan et al. (2020) we assume that the backward pass has twice the FLOPs of the forward pass.

<a id="S0475"></a> Source: p.28 S0475

We show a comparison between our calculation and that using the common approximation 𝐶 = 6𝐷𝑁 (Kaplan et al., 2020) where 𝐶 is FLOPs, 𝐷 is the number of training tokens, and 𝑁 is the number of parameters in Table A4.

<a id="S0476"></a> Source: p.28 S0476

We find the differences in FLOP calculation to be very small and they do not impact our analysis.

<a id="S0477"></a> Source: p.28 S0477

Compared to the results presented in Rae et al. (2021), we use a slightly more Parameters num_layers d_model ffw_size num_heads k/q size FLOP Ratio (Ours/6𝑁 𝐷) 73M 10 640 2560 10 64 1.03 305M 20 1024 4096 16 64 1.10 552M 24 1280 5120 10 128 1.08 1.1B 26 1792 7168 14 128 1.04 1.6B 28 2048 8192 16 128 1.03 6.8B 40 3584 14336 28 128 0.99 Table A4 | FLOP comparison.

<a id="S0478"></a> Source: p.28 S0478

For a variety of different model sizes, we show the ratio of the FLOPs that we compute per sequence to that using the 6𝑁 𝐷 approximation. accurate calculation giving a slightly different value (6.3 × 1023 compared to 5.76 × 1023). 28

Page 29

<a id="S0479"></a> Source: p.29 S0479

Other differences between Chinchilla and Gopher Beyond differences in model size and number of training tokens, there are some additional minor differences between Chinchilla and Gopher.

<a id="S0480"></a> Source: p.29 S0480

Specifically, Gopher was trained with Adam (Kingma and Ba, 2014) whereas Chinchilla was trained with AdamW (Loshchilov and Hutter, 2019).

<a id="S0481"></a> Source: p.29 S0481

Furthermore, as discussed in Lessons Learned in Rae et al. (2021), Chinchilla stored a higher-precision copy of the weights in the sharded optimiser state.

<a id="S0482"></a> Source: p.29 S0482

We show comparisons of models trained with Adam and AdamW in Figure A6 and Figure A7.

<a id="S0483"></a> Source: p.29 S0483

We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.

<a id="S0484"></a> Source: p.29 S0484

In Figure A6 we show a comparison of an 680 million parameter model trained 2.70 2.65 2.60 2.55 2.50 2.45 0 5 10 15 20 25 30 Million Sequences ssoL gniniarT 26 25 24 23 22 21 20 19 18 17 0 5 10 15 20 25 30 Million Sequences ytixelpreP 301txetikiW 3.00 2.95 2.90 2.85 2.80 2.75 2.70 2.65 2.60 0 5 10 15 20 25 30 Million Sequences ssoL 4C Training Setup Adam w/ High Precision AdamW w/ High Precision Adam No High Precision AdamW No High Precision Figure A6 | Comparison of other differences.

<a id="S0485"></a> Source: p.29 S0485

Using an 680 million parameter model, we show a comparison between the setup used to train Gopher and Chinchilla— the change in optimiser and using a higher precision copy of the weights in the optimiser state.

<a id="S0486"></a> Source: p.29 S0486

The setup used for Chinchilla (orange) clearly outperforms the setup used to train Gopher (green). 2.8 2.7 2.6 2.5 2.4 2.3 0 25 50 75 100 125 150 Million Sequences ssoL 4C 30.0 27.5 25.0 22.5 20.0 17.5 15.0 12.5 10.0 0 25 50 75 100 125 150 Million Sequences ytixelpreP 301txetikiW 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 25 50 75 100 125 150 Million Sequences ycaruccA ADABMAL 417M, Adam 417M, AdamW 1.4B, Adam 1.4B, AdamW Figure A7 | Adam vs AdamW.

<a id="S0487"></a> Source: p.29 S0487

For a 417M (blue) and 1.4B model (green), we find that training with AdamW improves performance over training with Adam. with and without the higher precision copy of the weights and with Adam/AdamW for comparison. H.

<a id="S0488"></a> Source: p.29 S0488

The Pile In Table A5 we show the bits-per-byte (bpb) on The Pile (Gao et al., 2020) of Chinchilla, Gopher, and Jurassic-1.

<a id="S0489"></a> Source: p.29 S0489

Chinchilla outperforms Gopher on all subsets.

<a id="S0490"></a> Source: p.29 S0490

Jurassic-1 outperforms Chinchilla on 2 subsets— dm_mathematics and ubuntu_irc. 29

Page 30

<a id="S0491"></a> Source: p.30 S0491

Subset Chinchilla (70B) Gopher (280B) Jurassic-1 (170B) pile_cc 0.667 0.691 0.669 pubmed_abstracts 0.559 0.578 0.587 stackexchange 0.614 0.641 0.655 github 0.337 0.377 0.358 openwebtext2 0.647 0.677 arxiv 0.627 0.662 0.680 uspto_backgrounds 0.526 0.546 0.537 freelaw 0.476 0.513 0.514 pubmed_central 0.504 0.525 0.579 dm_mathematics 1.111 1.142 1.037 hackernews 0.859 0.890 0.869 nih_exporter 0.572 0.590 0.590 opensubtitles 0.871 0.900 0.879 europarl 0.833 0.938 books3 0.675 0.712 0.835 philpapers 0.656 0.695 0.742 gutenberg_pg_19 0.548 0.656 0.890 bookcorpus2 0.714 0.741 ubuntu_irc 1.026 1.090 0.857 Table A5 | Bits-per-Byte on The Pile.

<a id="S0492"></a> Source: p.30 S0492

We show the bpb on The Pile for Chinchilla compared to Gopher and Jurassic-1. H.2.

<a id="S0493"></a> Source: p.30 S0493

MMLU In Table A6 we show the performance of Chinchilla and Gopher on each subset of MMLU. H.3.

<a id="S0494"></a> Source: p.30 S0494

Winogender Setup We follow the same setup as in Rae et al. (2021).

<a id="S0495"></a> Source: p.30 S0495

To test coreference resolution in Chinchilla, we input a sentence which includes a pronoun reference (e.g., “The librarian helped the child pick out a book because {pronoun} liked to encourage reading.”), then measure the probability of the model completing the sentence “‘{Pronoun}’ refers to the” with different sentence roles (“librarian” and “child” in this example).

<a id="S0496"></a> Source: p.30 S0496

Each example is annotated with the correct pronoun resolution (the pronoun corresponds to the librarian in this example).

<a id="S0497"></a> Source: p.30 S0497

Each sentence is tested with a female, male, and gender-neutral pronoun.

<a id="S0498"></a> Source: p.30 S0498

An unbiased model would correctly predict which word the pronoun refers to regardless of pronoun gender. H.4.

<a id="S0499"></a> Source: p.30 S0499

BIG-bench In Table A7 we show Chinchilla and Gopher performance on each subset of BIG-bench that we consider. I.

<a id="S0500"></a> Source: p.30 S0500

Model Card We present the Chinchilla model card in Table A8, following the framework presented by Mitchell et al. (2019). 30

Page 31

<a id="S0501"></a> Source: p.31 S0501

Task Chinchilla Gopher Task Chinchilla Gopher abstract_algebra 31.0 25.0 anatomy 70.4 56.3 astronomy 73.0 65.8 business_ethics 72.0 70.0 clinical_knowledge 75.1 67.2 college_biology 79.9 70.8 college_chemistry 51.0 45.0 college_computer_science 51.0 49.0 college_mathematics 32.0 37.0 college_medicine 66.5 60.1 college_physics 46.1 34.3 computer_security 76.0 65.0 conceptual_physics 67.2 49.4 econometrics 38.6 43.0 electrical_engineering 62.1 60.0 elementary_mathematics 41.5 33.6 formal_logic 33.3 35.7 global_facts 39.0 38.0 high_school_biology 80.3 71.3 high_school_chemistry 58.1 47.8 high_school_computer_science 58.0 54.0 high_school_european_history 78.8 72.1 high_school_geography 86.4 76.8 high_school_gov_and_politics 91.2 83.9 high_school_macroeconomics 70.5 65.1 high_school_mathematics 31.9 23.7 high_school_microeconomics 77.7 66.4 high_school_physics 36.4 33.8 high_school_psychology 86.6 81.8 high_school_statistics 58.8 50.0 high_school_us_history 83.3 78.9 high_school_world_history 85.2 75.1 human_aging 77.6 66.4 human_sexuality 86.3 67.2 international_law 90.9 77.7 jurisprudence 79.6 71.3 logical_fallacies 80.4 72.4 machine_learning 41.1 41.1 management 82.5 77.7 marketing 89.7 83.3 medical_genetics 69.0 69.0 miscellaneous 84.5 75.7 moral_disputes 77.5 66.8 moral_scenarios 36.5 40.2 nutrition 77.1 69.9 philosophy 79.4 68.8 prehistory 81.2 67.6 professional_accounting 52.1 44.3 professional_law 56.5 44.5 professional_medicine 75.4 64.0 professional_psychology 75.7 68.1 public_relations 73.6 71.8 security_studies 75.9 64.9 sociology 91.0 84.1 us_foreign_policy 92.0 81.0 virology 53.6 47.0 world_religions 87.7 84.2 Table A6 | Chinchilla MMLU results.

<a id="S0502"></a> Source: p.31 S0502

For each subset of MMLU (Hendrycks et al., 2020), we show Chinchilla’s accuracy compared to Gopher.

<a id="S0503"></a> Source: p.31 S0503

Model Details Organization Developing the Model DeepMind Model Date March 2022 Model Type Autoregressive Transformer Language Model (Section 4.1 for details) Feedback on the Model {jordanhoffmann, sborgeaud, amensch,sifre}@deepmind.com Intended Uses Primary Intended Uses The primary use is research on language models, including: research on the scaling behaviour of language models along with those listed in Rae et al. (2021). 31

Page 32

<a id="S0504"></a> Source: p.32 S0504

Primary Intended Users DeepMind researchers.

<a id="S0505"></a> Source: p.32 S0505

We will not make this model available publicly.

<a id="S0506"></a> Source: p.32 S0506

Out-of-Scope Uses Uses of the language model for language generation in harmful or deceitful settings.

<a id="S0507"></a> Source: p.32 S0507

More generally, the model should not be used for downstream applications without further safety and fairness mitigations.

<a id="S0508"></a> Source: p.32 S0508

Factors Card Prompts – Relevant Factor Relevant factors include which language is used.

<a id="S0509"></a> Source: p.32 S0509

Furthermore, in the analysis of models trained on the same corpus in Rae et al. (2021), we found it has unequal performance when modelling some dialects (e.g., African American English).

<a id="S0510"></a> Source: p.32 S0510

The model should not be used for downstream applications without further analysis on factors in the proposed downstream application.

<a id="S0511"></a> Source: p.32 S0511

Card Prompts – Evaluation Factors See the results in Rae et al. (2021) which analyzes models trained on the same text corpus.

<a id="S0512"></a> Source: p.32 S0512

Metrics Model Performance Measures • Perplexity and bits per byte on language modelling datasets • Accuracy on completion tasks, reading comprehension, MMLU, BIG-bench and fact checking. • Exact match accuracy for question answering. • Generation toxicity from Real Toxicity Prompts (RTP) alongside toxicity classification accuracy. • Gender and occupation bias.

<a id="S0513"></a> Source: p.32 S0513

Test include comparing the probability of generating different gender terms and the Winogender coreference resolution task.

<a id="S0514"></a> Source: p.32 S0514

We principally focus on Chinchilla’s performance compared to Gopher on text likelihood prediction.

<a id="S0515"></a> Source: p.32 S0515

Decision thresholds N/A Approaches to Uncertainty and Vari- Due to the costs of training large language models, we did ability not train Chinchilla multiple times.

<a id="S0516"></a> Source: p.32 S0516

However, the breadth of our evaluation on a range of different task types gives a reasonable estimate of the overall performance of the model.

<a id="S0517"></a> Source: p.32 S0517

Furthermore, the existence of another large model trained on the same dataset (Gopher) provides a clear point of comparison.

Page 33

<a id="S0518"></a> Source: p.33 S0518

Datasets • Language modelling on LAMBADA, Wikitext103 (Merity et al., 2017), C4 (Raffel et al., 2020a), PG-19 (Rae et al., 2020) and the Pile (Gao et al., 2020). • Language understanding, real world knowledge, mathematical and logical reasoning on the Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2020) and on the “Beyond the Imitation Game Benchmark” (BIG-bench) (BIG-bench collaboration, 2021). • Question answering (closed book) on Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). • Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).

<a id="S0519"></a> Source: p.33 S0519

Motivation We chose evaluations from Rae et al. (2021) to allow us to most directly compare to Gopher.

<a id="S0520"></a> Source: p.33 S0520

Preprocessing Input text is tokenized using a SentencePiece tokenizer with a vocabulary of size 32,000.

<a id="S0521"></a> Source: p.33 S0521

Unlike the tokenizer used for Gopher, the tokenizer used for Chinchilla does not perform NFKC normalization.

<a id="S0522"></a> Source: p.33 S0522

Training Data The same dataset is used as in Rae et al. (2021).

<a id="S0523"></a> Source: p.33 S0523

Differences in sampling are shown in Table A1.

<a id="S0524"></a> Source: p.33 S0524

Quantitative Analyses Unitary Results Section 4.2 gives a detailed description of our analysis.

<a id="S0525"></a> Source: p.33 S0525

Main take-aways include: • Our model is capable of outputting toxic language as measured by the PerspectiveAPI.

<a id="S0526"></a> Source: p.33 S0526

This is particularly true when the model is prompted with toxic prompts. • Gender: Our model emulates stereotypes found in our dataset, with occupations such as “dietician” and “receptionist” being more associated with women and “carpenter” and “sheriff ” being more associated with men. • Race/religion/country sentiment: Prompting our model to discuss some groups leads to sentences with lower or higher sentiment, likely reflecting text in our dataset. 33

Page 34

<a id="S0527"></a> Source: p.34 S0527

Intersectional Results We did not investigate intersectional biases.

<a id="S0528"></a> Source: p.34 S0528

Ethical Considerations Data The data is the same as described in Rae et al. (2021).

<a id="S0529"></a> Source: p.34 S0529

Human Life The model is not intended to inform decisions about matters central to human life or flourishing.

<a id="S0530"></a> Source: p.34 S0530

Mitigations We considered filtering the dataset to remove toxic content but decided against it due to the observation that this can introduce new biases as studied by Welbl et al. (2021).

<a id="S0531"></a> Source: p.34 S0531

More work is needed on mitigation approaches to toxic content and other types of risks associated with language models, such as those discussed in Weidinger et al. (2021).

<a id="S0532"></a> Source: p.34 S0532

Risks and Harms The data is collected from the internet, and thus undoubtedly there is toxic/biased content in our training dataset.

<a id="S0533"></a> Source: p.34 S0533

Furthermore, it is likely that personal information is also in the dataset that has been used to train our models.

<a id="S0534"></a> Source: p.34 S0534

We defer to the more detailed discussion in Weidinger et al. (2021).

<a id="S0535"></a> Source: p.34 S0535

Use Cases Especially fraught use cases include the generation of factually incorrect information with the intent of distributing it or using the model to generate racist, sexist or otherwise toxic text with harmful intent.

<a id="S0536"></a> Source: p.34 S0536

Many more use cases that could cause harm exist.

<a id="S0537"></a> Source: p.34 S0537

Such applications to malicious use are discussed in detail in Weidinger et al. (2021).

<a id="S0538"></a> Source: p.34 S0538

We follow the framework presented in Mitchell et al. (2019). J.

<a id="S0539"></a> Source: p.34 S0539

List of trained models In Table A9 we list the model size and configuration of all models used in this study.

<a id="S0540"></a> Source: p.34 S0540

Many models have been trained multiple times, for a different number of training steps. 34

Page 35

<a id="S0541"></a> Source: p.35 S0541

Task Chinchilla Gopher Task Chinchilla Gopher hyperbaton 54.2 51.7 movie_dialog_same_or_diff 54.5 50.7 causal_judgment 57.4 50.8 winowhy 62.5 56.7 formal_fallacies_syllogisms_neg 52.1 50.7 movie_recommendation 75.6 50.5 crash_blossom 47.6 63.6 moral_permissibility 57.3 55.1 discourse_marker_prediction 13.1 11.7 strategyqa 68.3 61.0 general_knowledge_json 94.3 93.9 nonsense_words_grammar 78.0 61.4 sports_understanding 71.0 54.9 metaphor_boolean 93.1 59.3 implicit_relations 49.4 36.4 navigate 52.6 51.1 penguins_in_a_table 48.7 40.6 presuppositions_as_nli 49.9 34.0 intent_recognition 92.8 88.7 temporal_sequences 32.0 19.0 reasoning_about_colored_objects 59.7 49.2 question_selection 52.6 41.4 logic_grid_puzzle 44.0 35.1 logical_fallacy_detection 72.1 58.9 timedial 68.8 50.9 physical_intuition 79.0 59.7 epistemic_reasoning 60.6 56.4 physics_mc 65.5 50.9 ruin_names 47.1 38.6 identify_odd_metaphor 68.8 38.6 hindu_knowledge 91.4 80.0 understanding_fables 60.3 39.6 misconceptions 65.3 61.7 logical_sequence 64.1 36.4 implicatures 75.0 62.0 mathematical_induction 47.3 57.6 disambiguation_q 54.7 45.5 fantasy_reasoning 69.0 64.1 known_unknowns 65.2 63.6 SNARKS 58.6 48.3 dark_humor_detection 66.2 83.1 crass_ai 75.0 56.8 analogical_similarity 38.1 17.2 entailed_polarity 94.0 89.5 sentence_ambiguity 71.7 69.1 irony_identification 73.0 69.7 riddle_sense 85.7 68.2 evaluating_info_essentiality 17.6 16.7 date_understanding 52.3 44.1 phrase_relatedness 94.0 81.8 analytic_entailment 67.1 53.0 novel_concepts 65.6 59.1 odd_one_out 70.9 32.5 empirical_judgments 67.7 52.5 logical_args 56.2 59.1 figure_of_speech_detection 63.3 52.7 alignment_questionnaire 91.3 79.2 english_proverbs 82.4 57.6 similarities_abstraction 87.0 81.8 Human_organs_senses_mcc 85.7 84.8 anachronisms 69.1 56.4 gre_reading_comprehension 53.1 27.3 Table A7 | Chinchilla BIG-bench results.

<a id="S0542"></a> Source: p.35 S0542

For each subset of BIG-bench (BIG-bench collaboration, 2021), we show Chinchilla and Gopher’s accuracy. 35

Page 36

<a id="S0543"></a> Source: p.36 S0543

Parameters (million) d_model ffw_size kv_size n_heads n_layers 44 512 2048 64 8 8 57 576 2304 64 9 9 74 640 2560 64 10 10 90 640 2560 64 10 13 106 640 2560 64 10 16 117 768 3072 64 12 12 140 768 3072 64 12 15 163 768 3072 64 12 18 175 896 3584 64 14 14 196 896 3584 64 14 16 217 896 3584 64 14 18 251 1024 4096 64 16 16 278 1024 4096 64 16 18 306 1024 4096 64 16 20 425 1280 5120 128 10 18 489 1280 5120 128 10 21 509 1408 5632 128 11 18 552 1280 5120 128 10 24 587 1408 5632 128 11 21 632 1536 6144 128 12 19 664 1408 5632 128 11 24 724 1536 6144 128 12 22 816 1536 6144 128 12 25 893 1792 7168 128 14 20 1,018 1792 7168 128 14 23 1,143 1792 7168 128 14 26 1,266 2048 8192 128 16 22 1,424 2176 8704 128 17 22 1,429 2048 8192 128 16 25 1,593 2048 8192 128 16 28 1,609 2176 8704 128 17 25 1,731 2304 9216 128 18 24 1,794 2176 8704 128 17 28 2,007 2304 9216 128 18 28 2,283 2304 9216 128 18 32 2,298 2560 10240 128 20 26 2,639 2560 10240 128 20 30 2,980 2560 10240 128 20 34 3,530 2688 10752 128 22 36 3,802 2816 11264 128 22 36 4,084 2944 11776 128 22 36 4,516 3072 12288 128 24 36 6,796 3584 14336 128 28 40 9,293 4096 16384 128 32 42 11,452 4352 17408 128 32 47 12,295 4608 18432 128 36 44 12,569 4608 18432 128 32 47 13,735 4864 19456 128 32 47 14,940 4992 19968 128 32 49 16,183 5120 20480 128 40 47 Table A9 | All models.

<a id="S0544"></a> Source: p.36 S0544

We list the hyperparameters and size of all models trained as part of this work.

<a id="S0545"></a> Source: p.36 S0545

Many shown models have been trained with multiple learning rate schedules/number of training tokens. 36