Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B.
--- title: "Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B." aliases: - "Scaling Law" - "arXiv:2001.08361" source: "https://arxiv.org/abs/2001.08361" arxiv: "2001.08361" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning
Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B.
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B.
<a id="S0002"></a> Source: p.1 S0002
Brown Benjamin Chess Rewon Child OpenAI OpenAI OpenAI OpenAI henighan@openai.com tom@openai.com bchess@openai.com rewon@openai.com Scott Gray Alec Radford Jeffrey Wu Dario Amodei OpenAI OpenAI OpenAI OpenAI scott@openai.com alec@openai.com jeffwu@openai.com damodei@openai.com Abstract We study empirical scaling laws for language model performance on the cross-entropy loss.
<a id="S0003"></a> Source: p.1 S0003
The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.
<a id="S0004"></a> Source: p.1 S0004
Other architectural details such as network width or depth have minimal effects within a wide range.
<a id="S0005"></a> Source: p.1 S0005
Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size.
<a id="S0006"></a> Source: p.1 S0006
These relationships allow us to determine the optimal allocation of a fixed compute budget.
<a id="S0007"></a> Source: p.1 S0007
Larger models are significantly more sampleefficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence. ∗Equal contribution.
<a id="S0008"></a> Source: p.1 S0008
Contributions: Jared Kaplan and Sam McCandlish led the research.
<a id="S0009"></a> Source: p.1 S0009
Tom Henighan contributed the LSTM experiments.
<a id="S0010"></a> Source: p.1 S0010
Tom Brown, Rewon Child, and Scott Gray, and Alec Radford developed the optimized Transformer implementation.
<a id="S0011"></a> Source: p.1 S0011
Jeff Wu, Benjamin Chess, and Alec Radford developed the text datasets.
<a id="S0012"></a> Source: p.1 S0012
Dario Amodei provided guidance throughout the project. 0202 naJ 32 ]GL.sc[ 1v16380.1002:viXra
Page 2
<a id="S0013"></a> Source: p.2 S0013
Contents 1 Introduction 2 2 Background and Methods 6 3 Empirical Results and Basic Power Laws 7 4 Charting the Infinite Data Limit and Overfitting 10 5 Scaling Laws with Model Size and Training Time 12 6 Optimal Allocation of the Compute Budget 14 7 Related Work 18 8 Discussion 18 Appendices 20 A Summary of Power Laws 20 B Empirical Model of Compute-Efficient Frontier 20 C Caveats 22 D Supplemental Figures 23 1 Introduction Language provides a natural domain for the study of artificial intelligence, as the vast majority of reasoning tasks can be efficiently expressed and evaluated in language, and the world’s text provides a wealth of data for unsupervised learning via generative modeling.
<a id="S0014"></a> Source: p.2 S0014
Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY+19, LOG+19, RSR+19] approaching human-level performance on many specific tasks [WPN+19], including the composition of coherent multiparagraph prompted text samples [RWC+19].
<a id="S0015"></a> Source: p.2 S0015
One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this training process.
<a id="S0016"></a> Source: p.2 S0016
In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP+17, LSP+18].
<a id="S0017"></a> Source: p.2 S0017
The high ceiling and low floor for performance on language tasks allows us to study trends over more than seven orders of magnitude in scale.
<a id="S0018"></a> Source: p.2 S0018
Throughout we will observe precise power-law scalings for performance as a function of training time, context length, dataset size, model size, and compute budget. 1.1 Summary Our key findings for Transformer language models are are as follows: 2Here we display predicted compute when using a sufficiently small batch size.
<a id="S0019"></a> Source: p.2 S0019
See Figure 13 for comparison to the purely empirical data. 2
Page 3
<a id="S0020"></a> Source: p.3 S0020
Compute Dataset Size Parameters PF-days, non-embedding tokens non-embedding ssoL tseT Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute2 used for training.
<a id="S0021"></a> Source: p.3 S0021
For optimal performance all three factors must be scaled up in tandem.
<a id="S0022"></a> Source: p.3 S0022
Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
<a id="S0023"></a> Source: p.3 S0023
Performance depends strongly on scale, weakly on model shape: Model performance depends most strongly on scale, which consists of three factors: the number of model parameters N (excluding embeddings), the size of the dataset D, and the amount of compute C used for training.
<a id="S0024"></a> Source: p.3 S0024
Within reasonable limits, performance depends very weakly on other architectural hyperparameters such as depth vs. width. (Section 3) Smooth power laws: Performance has a power-law relationship with each of the three scale factors N, D, C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1).
<a id="S0025"></a> Source: p.3 S0025
We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3) Universality of overfitting: Performance improves predictably as long as we scale up N and D in tandem, but enters a regime of diminishing returns if either N or D is held fixed while the other increases.
<a id="S0026"></a> Source: p.3 S0026
The performance penalty depends predictably on the ratio N 0.74/D, meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty. (Section 4) Universality of training: Training curves follow predictable power-laws whose parameters are roughly independent of the model size.
<a id="S0027"></a> Source: p.3 S0027
By extrapolating the early part of a training curve, we can roughly predict the loss that would be achieved if we trained for much longer. (Section 5) Transfer improves with test performance: When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss – in other words, transfer to a different distribution incurs a constant penalty but otherwise improves roughly in line with performance on the training set. (Section 3.2.2) Sample efficiency: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4).
<a id="S0028"></a> Source: p.3 S0028
Convergence is inefficient: When working within a fixed compute budget C but without any other restrictions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3).
<a id="S0029"></a> Source: p.3 S0029
Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C0.27 with training compute. (Section 6) Optimal batch size: The ideal batch size for training these models is roughly a power of the loss only, and continues to be determinable by measuring the gradient noise scale [MKAT18]; it is roughly 1-2 million tokens at convergence for the largest models we can train. (Section 5.1) Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute.
<a id="S0030"></a> Source: p.3 S0030
We expect that larger language models will perform better and be more sample efficient than current models. 3
Page 4
<a id="S0031"></a> Source: p.4 S0031
Larger models require fewer samples The optimal model size grows smoothly to reach the same performance with the loss target and compute budget Line color indicates Test Loss 10 10 number of parameters 103 106 109 8 8 103 Params 6 6 Compute-efficient 109 Params training stops far short of convergence 4 4 107 109 1011 10-9 10-6 10-3 100 Tokens Processed Compute (PF-days) Figure 2 We show a series of language model training runs, with models ranging in size from 103 to 109 parameters (excluding embeddings). M inc in re im as u e m s s n e e r g ia li l g s ib te ly ps <1 1 0 0 x 0 S x e B ri a a t l c S h t e S p iz s e D gr a o t w a r r e e q la u t i i r v e e m ly e s n l t o s w ly >1,000,000x Model Size O in p cr t e im as a e l s m v o e d r e y l q s u iz ic e k ly Figure 3 As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps.
<a id="S0032"></a> Source: p.4 S0032
We illustrate this for a billion-fold increase in compute.
<a id="S0033"></a> Source: p.4 S0033
For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse.
<a id="S0034"></a> Source: p.4 S0034
Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required. 1.2 Summary of Scaling Laws The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters N , the dataset size D, or the optimally allocated compute budget C (see Figure 1): min 1.
<a id="S0035"></a> Source: p.4 S0035
For models with a limited number of parameters, trained to convergence on sufficiently large datasets: L(N ) = (N /N )αN ; α ∼ 0.076, N ∼ 8.8 × 1013 (non-embedding parameters) (1.1) c N c 2.
<a id="S0036"></a> Source: p.4 S0036
For large models trained with a limited dataset with early stopping: L(D) = (D /D)αD ; α ∼ 0.095, D ∼ 5.4 × 1013 (tokens) (1.2) c D c 3.
<a id="S0037"></a> Source: p.4 S0037
When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal3 use of compute): L(C ) = (cid:0) Cmin/C (cid:1)αm C in ; αmin ∼ 0.050, Cmin ∼ 3.1 × 108 (PF-days) (1.3) min c min C c 3We also observe an empirical power-law trend with the training compute C (Figure 1) while training at fixed batch size, but it is the trend with C that should be used to make predictions.
<a id="S0038"></a> Source: p.4 S0038
They are related by equation (5.5). min 4
Page 5
<a id="S0039"></a> Source: p.5 S0039
4.5 4.0 3.5 3.0 2.5 107 108 109 1010 Tokens in Dataset ssoL Loss vs Model and Dataset Size 4.4 4.0 Params 708M 3.6 302M 85M 3.2 3M 25M 393.2K 2.8 2.4 104 105 Estimated Smin ssoL Loss vs Model Size and Training Steps 108 107 106 )debme-non( sretemaraP Figure 4 Left: The early-stopped test loss L(N, D) varies predictably with the dataset size D and model size N according to Equation (1.5).
<a id="S0040"></a> Source: p.5 S0040
Right: After an initial transient period, learning curves for all model sizes N can be fit with Equation (1.6), which is parameterized in terms of S , the number of steps when min training at large batch size (details in Section 5.1).
<a id="S0041"></a> Source: p.5 S0041
These relations hold across eight orders of magnitude in C , six orders of magnitude in N , and over two min orders of magnitude in D.
<a id="S0042"></a> Source: p.5 S0042
They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [RWC+19].
<a id="S0043"></a> Source: p.5 S0043
The power laws α , α , αmin specify the degree of performance improvement N D C expected as we scale up N , D, or C ; for example, doubling the number of parameters yields a loss that min is smaller by a factor 2−αN = 0.95.
<a id="S0044"></a> Source: p.5 S0044
The precise numerical values of N c , C c min, and D c depend on the vocabulary size and tokenization and hence do not have a fundamental meaning.
<a id="S0045"></a> Source: p.5 S0045
The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ([MKAT18]), also roughly obeys a power law in L: B B (L) = ∗ , B ∼ 2 · 108 tokens, α ∼ 0.21 (1.4) crit L1/αB ∗ B Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to D ∝ N α α N D ∼ N 0.74.
<a id="S0046"></a> Source: p.5 S0046
In fact, we find that there is a single equation combining (1.1) and (1.2) that governs the simultaneous dependence on N and D and governs the degree of overfitting: (cid:34)(cid:18) (cid:19) αN (cid:35)αD L(N, D) = N c αD + D c (1.5) N D with fits pictured on the left in figure 4.
<a id="S0047"></a> Source: p.5 S0047
We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.
<a id="S0048"></a> Source: p.5 S0048
When training a given model for a finite number of parameter update steps S in the infinite data limit, after an initial transient period, the learning curves can be accurately fit by (see the right of figure 4) (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L(N, S) = c + c (1.6) N S (S) min where S ≈ 2.1 × 103 and α ≈ 0.76, and S (S) is the minimum possible number of optimization steps c S min (parameter updates) estimated using Equation (5.4).
<a id="S0049"></a> Source: p.5 S0049
When training within a fixed compute budget C, but with no other constraints, Equation (1.6) leads to the prediction that the optimal model size N , optimal batch size B, optimal number of steps S, and dataset size D should grow as N ∝ Cαm C in/αN , B ∝ Cαm C in/αB , S ∝ Cαm C in/αS , D = B · S (1.7) with αmin = 1/ (1/α + 1/α + 1/α ) (1.8) C S B N which closely matches the empirically optimal results N ∝ C0.73, B ∝ C0.24, and S ∝ C0.03.
<a id="S0050"></a> Source: p.5 S0050
As the min min min computational budget C increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size (see Figure 3).
<a id="S0051"></a> Source: p.5 S0051
This also implies that as models grow larger, they become increasingly sample efficient.
<a id="S0052"></a> Source: p.5 S0052
In practice, researchers typically train smaller models for longer than would 5
Page 6
<a id="S0053"></a> Source: p.6 S0053
be maximally compute-efficient because of hardware constraints.
<a id="S0054"></a> Source: p.6 S0054
Optimal performance depends on total compute as a power law (see Equation (1.3)).
<a id="S0055"></a> Source: p.6 S0055
We provide some basic theoretical motivation for Equation (1.5), an analysis of learning curve fits and their implications for training time, and a breakdown of our results per token.
<a id="S0056"></a> Source: p.6 S0056
We also make some brief comparisons to LSTMs and recurrent Transformers [DGV+18]. 1.3 Notation We use the following notation: • L – the cross entropy loss in nats.
<a id="S0057"></a> Source: p.6 S0057
Typically it will be averaged over the tokens in a context, but in some cases we report the loss for specific tokens within the context. • N – the number of model parameters, excluding all vocabulary and positional embeddings • C ≈ 6N BS – an estimate of the total non-embedding training compute, where B is the batch size, and S is the number of training steps (ie parameter updates).
<a id="S0058"></a> Source: p.6 S0058
We quote numerical values in PF-days, where one PF-day = 1015 × 24 × 3600 = 8.64 × 1019 floating point operations. • D – the dataset size in tokens • B – the critical batch size [MKAT18], defined and discussed in Section 5.1.
<a id="S0059"></a> Source: p.6 S0059
Training at the crit critical batch size provides a roughly optimal compromise between time and compute efficiency. • C – an estimate of the minimum amount of non-embedding compute to reach a given value of min the loss.
<a id="S0060"></a> Source: p.6 S0060
This is the training compute that would be used if the model were trained at a batch size much less than the critical batch size. • S – an estimate of the minimal number of training steps needed to reach a given value of the loss. min This is also the number of training steps that would be used if the model were trained at a batch size much greater than the critical batch size. • α X – power-law exponents for the scaling of the loss as L(X) ∝ 1/XαX where X can be any of N, D, C, S, B, Cmin. 2 Background and Methods We train language models on WebText2, an extended version of the WebText [RWC+19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n = 50257.
<a id="S0061"></a> Source: p.6 S0061
We optimize the autoregresvocab sive log-likelihood (i.e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric.
<a id="S0062"></a> Source: p.6 S0062
We record the loss on the WebText2 test distribution and on a selection of other text distributions.
<a id="S0063"></a> Source: p.6 S0063
We primarily train decoder-only [LSP+18, RNSS18] Transformer [VSP+17] models, though we also train LSTM models and Universal Transformers [DGV+18] for comparison. 2.1 Parameter and Compute Scaling of Transformers We parameterize the Transformer architecture using hyperparameters n (number of layers), d (dilayer model mension of the residual stream), d (dimension of the intermediate feed-forward layer), d (dimension of ff attn the attention output), and n (number of attention heads per layer).
<a id="S0064"></a> Source: p.6 S0064
We include n tokens in the input heads ctx context, with n = 1024 except where otherwise noted. ctx We use N to denote the model size, which we define as the number of non-embedding parameters N ≈ 2d n (2d + d ) model layer attn ff = 12n d2 with the standard d = d /4 = d (2.1) layer model attn ff model where we have excluded biases and other sub-leading terms.
<a id="S0065"></a> Source: p.6 S0065
Our models also have n d parameters vocab model in an embedding matrix, and use n d parameters for positional embeddings, but we do not include ctx model these when discussing the ‘model size’ N ; we will see that this produces significantly cleaner scaling laws.
<a id="S0066"></a> Source: p.6 S0066
Evaluating a forward pass of the Transformer involves roughly C ≈ 2N + 2n n d (2.2) forward layer ctx model add-multiply operations, where the factor of two comes from the multiply-accumulate operation used in matrix multiplication. A more detailed per-operation parameter and compute count is included in Table 1. 6
Page 7
<a id="S0067"></a> Source: p.7 S0067
Operation Parameters FLOPs per Token Embed (n + n ) d 4d vocab ctx model model Attention: QKV n d 3d 2n d 3d layer model attn layer model attn Attention: Mask — 2n n d layer ctx attn Attention: Project n d d 2n d d layer attn model layer attn embd Feedforward n 2d d 2n 2d d layer model ff layer model ff De-embed — 2d n model vocab Total (Non-Embedding) N = 2d n (2d + d ) C = 2N + 2n n d model layer attn ff forward layer ctx attn Table 1 Parameter counts and compute (forward pass) estimates for a Transformer model.
<a id="S0068"></a> Source: p.7 S0068
Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted.
<a id="S0069"></a> Source: p.7 S0069
For contexts and models with d > n /12, the context-dependent computational cost per token is a model ctx relatively small fraction of the total compute.
<a id="S0070"></a> Source: p.7 S0070
Since we primarily study models where d (cid:29) n /12, model ctx we do not include context-dependent terms in our training compute estimate.
<a id="S0071"></a> Source: p.7 S0071
Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C ≈ 6N floating point operators per training token. 2.2 Training Procedures Unless otherwise noted, we train models with the Adam optimizer [KB14] for a fixed 2.5 × 105 steps with a batch size of 512 sequences of 1024 tokens.
<a id="S0072"></a> Source: p.7 S0072
Due to memory constraints, our largest models (more than 1B parameters) were trained with Adafactor [SS18].
<a id="S0073"></a> Source: p.7 S0073
We experimented with a variety of learning rates and schedules, as discussed in Appendix D.6.
<a id="S0074"></a> Source: p.7 S0074
We found that results at convergence were largely independent of learning rate schedule.
<a id="S0075"></a> Source: p.7 S0075
Unless otherwise noted, all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero. 2.3 Datasets We train our models on an extended version of the WebText dataset described in [RWC+19].
<a id="S0076"></a> Source: p.7 S0076
The original WebText dataset was a web scrape of outbound links from Reddit through December 2017 which received at least 3 karma.
<a id="S0077"></a> Source: p.7 S0077
In the second version, WebText2, we added outbound Reddit links from the period of January to October 2018, also with a minimum of 3 karma.
<a id="S0078"></a> Source: p.7 S0078
The karma threshold served as a heuristic for whether people found the link interesting or useful.
<a id="S0079"></a> Source: p.7 S0079
The text of the new links was extracted with the Newspaper3k python library.
<a id="S0080"></a> Source: p.7 S0080
In total, the dataset consists of 20.3M documents containing 96 GB of text and 1.62 × 1010 words (as defined by wc).
<a id="S0081"></a> Source: p.7 S0081
We then apply the reversible tokenizer described in [RWC+19], which yields 2.29 × 1010 tokens.
<a id="S0082"></a> Source: p.7 S0082
We reserve 6.6 × 108 of these tokens for use as a test set, and we also test on similarlyprepared samples of Books Corpus [ZKZ+15], Common Crawl [Fou], English Wikipedia, and a collection of publicly-available Internet Books. 3 Empirical Results and Basic Power Laws To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding parameters) • Dataset size (ranging from 22 million to 23 billion tokens) • Shape (including depth, width, attention heads, and feed-forward dimension) • Context length (1024 for most runs, though we also experiment with shorter contexts) • Batch size (219 for most runs, but we also vary it to measure the critical batch size) 7
Page 8
<a id="S0083"></a> Source: p.8 S0083
10% 8% 6% 4% 2% 0% Feed-Fo 5 r 0 w M a r P d a R ra a m ti e o te ( r d s ff / dmodel) Aspect Ratio (dmodel / nlayer) Attention He 2 a 5 d M D P im ara e m ns e i t o e n rs (dmodel / nhead) esaercnI ssoL A wide range of architectures achieve similar performance 22% additional compute compensates for 1% loss increase Figure 5 Performance depends very mildly on model shape when the total number of non-embedding parameters N is held fixed.
<a id="S0084"></a> Source: p.8 S0084
The loss varies only a few percent over a wide range of shapes.
<a id="S0085"></a> Source: p.8 S0085
Small differences in parameter counts are compensated for by using the fit to L(N ) as a baseline.
<a id="S0086"></a> Source: p.8 S0086
Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (n , d ) = (6, 4288) reaches a layer model loss within 3% of the (48, 1600) model used in [RWC+19]. 7 6 5 4 3 2 106 107 108 109 Parameters (with embedding) ssoL tseT 7 6 5 0 Layer 4 1 Layer 2 Layers 3 Layers 3 6 Layers > 6 Layers 2 103 104 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 1 Layer 2 Layers 3 Layers 6 Layers > 6 Layers Figure 6 Left: When we include embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters.
<a id="S0087"></a> Source: p.8 S0087
Right: When we exclude embedding parameters, the performance of models with different depths converge to a single trend.
<a id="S0088"></a> Source: p.8 S0088
Only models with fewer than 2 layers or with extreme depth-to-width ratios deviate significantly from the trend.
<a id="S0089"></a> Source: p.8 S0089
In this section we will display data along with empirically-motivated fits, deferring theoretical analysis to later sections. 3.1 Approximate Transformer Shape and Hyperparameter Independence Transformer performance depends very weakly on the shape parameters n , n , and d when we hold layer heads ff the total non-embedding parameter count N fixed.
<a id="S0090"></a> Source: p.8 S0090
To establish these results we trained models with fixed size while varying a single hyperparameter.
<a id="S0091"></a> Source: p.8 S0091
When varying n , heads layer we simultaneously varied d while keeping N ≈ 12n d2 fixed.
<a id="S0092"></a> Source: p.8 S0092
Similarly, to vary d at fixed model layer model ff model size we also simultaneously varied the d parameter, as required by the parameter counts in Table model 1.
<a id="S0093"></a> Source: p.8 S0093
Independence of n would follow if deeper Transformers effectively behave as ensembles of shallower layers models, as has been suggested for ResNets [VWB16].
<a id="S0094"></a> Source: p.8 S0094
The results are shown in Figure 5. 3.2 Performance with Non-Embedding Parameter Count N In Figure 6 we display the performance of a wide variety of models, ranging from small models with shape (n , d ) = (2, 128) through billion-parameter models, ranging in shape from (6, 4288) through layer model (207, 768).
<a id="S0095"></a> Source: p.8 S0095
Here we have trained to near convergence on the full WebText2 dataset and observe no overfitting (except possibly for the very largest models).
<a id="S0096"></a> Source: p.8 S0096
As shown in Figure 1, we find a steady trend with non-embedding parameter count N , which can be fit to the first term of Equation (1.5), so that (cid:18) N (cid:19)αN L(N ) ≈ c (3.1) N 8
Page 9
<a id="S0097"></a> Source: p.9 S0097
Transformers asymptotically outperform LSTMs LSTM plateaus after <100 tokens due to improved use of long contexts Transformer improves through the whole context Test Loss 5.4 Per-token Test Loss 6 4.8 4.2 LSTMs 4 Parameters: 3.6 400K 1 Layer 5 400K 2 Layers 2M 3.0 Transformers 4 Layers 3M 3 200M 2.4 300M 2 105 106 107 108 109 101 102 103 Parameters (non-embedding) Token Index in Context Figure 7 To observe these trends it is crucial to study performance as a function of N ; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6).
<a id="S0098"></a> Source: p.9 S0098
This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].
<a id="S0099"></a> Source: p.9 S0099
Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasets is also a power-law in N with nearly identical power, as shown in Figure 8. 3.2.1 Comparing to LSTMs and Universal Transformers In Figure 7 we compare LSTM and Transformer performance as a function of non-embedding parameter count N .
<a id="S0100"></a> Source: p.9 S0100
The LSTMs were trained with the same dataset and context length.
<a id="S0101"></a> Source: p.9 S0101
We see from these figures that the LSTMs perform as well as Transformers for tokens appearing early in the context, but cannot match the Transformer performance for later tokens.
<a id="S0102"></a> Source: p.9 S0102
We present power-law relationships between performance and context position Appendix D.5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.
<a id="S0103"></a> Source: p.9 S0103
We also compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17 in the appendix.
<a id="S0104"></a> Source: p.9 S0104
These models re-use parameters, and so perform slightly better as a function of N , at the cost of additional compute per-parameter. 3.2.2 Generalization Among Data Distributions We have also tested our models on a set of additional text data distributions.
<a id="S0105"></a> Source: p.9 S0105
The test loss on these datasets as a function of model size is shown in Figure 8; in all cases the models were trained only on the WebText2 dataset.
<a id="S0106"></a> Source: p.9 S0106
We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2.
<a id="S0107"></a> Source: p.9 S0107
We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence.
<a id="S0108"></a> Source: p.9 S0108
We also observe no dependence on model depth (see Appendix D.8). 3.3 Performance with Dataset Size and Compute We display empirical trends for the test loss as a function of dataset size D (in tokens) and training compute C in Figure 1.
<a id="S0109"></a> Source: p.9 S0109
For the trend with D we trained a model with (n , n ) = (36, 1280) on fixed subsets of the WebText2 layer embd dataset.
<a id="S0110"></a> Source: p.9 S0110
We stopped training once the test loss ceased to decrease.
<a id="S0111"></a> Source: p.9 S0111
We see that the resulting test losses can be fit with simple power-law (cid:18) D (cid:19)αD L(D) ≈ c (3.2) D in the dataset size.
<a id="S0112"></a> Source: p.9 S0112
The total amount of non-embedding compute used during training can be estimated as C = 6N BS, where B is the batch size, S is the number of parameter updates, and the factor of 6 accounts for the forward and backward passes.
<a id="S0113"></a> Source: p.9 S0113
Thus for a given value of C we can scan over all models with various N to find the model 9
Page 10
<a id="S0114"></a> Source: p.10 S0114
7 6 5 4 3 104 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 5.0 WebText2 (Test) Internet Books 4.5 Books Wikipedia 4.0 Common Crawl 3.5 3.0 2.5 5.0 4.5 4.0 3.5 3.0 2.5 Test Loss on Training Distribution noitubirtsiD rehtO no ssoL Books during training Wikipedia during training Books at convergence Wikipedia at convergence Figure 8 Left: Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution.
<a id="S0115"></a> Source: p.10 S0115
Right: Generalization performance depends only on training distribution performance, and not on the phase of training.
<a id="S0116"></a> Source: p.10 S0116
We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains. with the best performance on step S = C .
<a id="S0117"></a> Source: p.10 S0117
Note that in these results the batch size B remains fixed for 6BS all models, which means that these empirical results are not truly optimal.
<a id="S0118"></a> Source: p.10 S0118
We will account for this in later sections using an adjusted C to produce cleaner trends. min The result appears as the heavy black line on the left-hand plot in Figure 1.
<a id="S0119"></a> Source: p.10 S0119
It can be fit with (cid:18) C (cid:19)αC L(C) ≈ c (3.3) C The figure also includes images of individual learning curves to clarify when individual models are optimal.
<a id="S0120"></a> Source: p.10 S0120
We will study the optimal allocation of compute more closely later on.
<a id="S0121"></a> Source: p.10 S0121
The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix. 4 Charting the Infinite Data Limit and Overfitting In Section 3 we found a number of basic scaling laws for language modeling performance.
<a id="S0122"></a> Source: p.10 S0122
Here we will study the performance of a model of size N trained on a dataset with D tokens while varying N and D simultaneously.
<a id="S0123"></a> Source: p.10 S0123
We will empirically demonstrate that the optimally trained test loss accords with the scaling law of Equation (1.5).
<a id="S0124"></a> Source: p.10 S0124
This provides guidance on how much data we would need to train models of increasing size while keeping overfitting under control. 4.1 Proposed L(N, D) Equation We have chosen the parameterization (1.5) (repeated here for convenience): (cid:34)(cid:18) (cid:19) αN (cid:35)αD L(N, D) = N c αD + D c (4.1) N D using three principles: 1.
<a id="S0125"></a> Source: p.10 S0125
Changes in vocabulary size or tokenization are expected to rescale the loss by an overall factor.
<a id="S0126"></a> Source: p.10 S0126
The parameterization of L(N, D) (and all models of the loss) must naturally allow for such a rescaling. 2.
<a id="S0127"></a> Source: p.10 S0127
Fixing D and sending N → ∞, the overall loss should approach L(D).
<a id="S0128"></a> Source: p.10 S0128
Conversely, fixing N and sending D → ∞ the loss must approach L(N ). 3.
<a id="S0129"></a> Source: p.10 S0129
L(N, D) should be analytic at D = ∞, so that it has a series expansion in 1/D with integer powers.
<a id="S0130"></a> Source: p.10 S0130
Theoretical support for this principle is significantly weaker than for the first two.
<a id="S0131"></a> Source: p.10 S0131
Our choice of L(N, D) satisfies the first requirement because we can rescale N , D with changes in the c c vocabulary.
<a id="S0132"></a> Source: p.10 S0132
This also implies that the values of N , D have no fundamental meaning. c c 10
Page 11
<a id="S0133"></a> Source: p.11 S0133
4.5 4.0 3.5 3.0 2.5 106 107 108 109 Params (non-embed) ssoL tseT Data Size Bottleneck 0.5 Data Size 0.4 21M 43M 86M 0.3 172M 344M 0.2 688M 1.4B 22.0B 0.1 0.0 10 4 10 3 10 2 10 1 N N/ D/D 1 ) =D(L/L Overfitting Data Size 21M 43M 86M 172M 344M 688M 1.4B 22.0B Figure 9 The early-stopped test loss L(N, D) depends predictably on the dataset size D and model size N according to Equation (1.5).
<a id="S0134"></a> Source: p.11 S0134
Left: For large D, performance is a straight power law in N .
<a id="S0135"></a> Source: p.11 S0135
For a smaller fixed D, performance stops improving as N increases and the model begins to overfit. (The reverse is also true, αN see Figure 4.) Right: The extent of overfitting depends predominantly on the ratio N αD /D, as predicted in equation (4.3).
<a id="S0136"></a> Source: p.11 S0136
Since we stop training early when the test loss ceases to improve and optimize all models in the same way, we expect that larger models should always perform better than smaller models.
<a id="S0137"></a> Source: p.11 S0137
But with fixed finite D, we also do not expect any model to be capable of approaching the best possible loss (ie the entropy of text).
<a id="S0138"></a> Source: p.11 S0138
Similarly, a model with fixed size will be capacity-limited.
<a id="S0139"></a> Source: p.11 S0139
These considerations motivate our second principle.
<a id="S0140"></a> Source: p.11 S0140
Note that knowledge of L(N ) at infinite D and L(D) at infinite N fully determines all the parameters in L(N, D).
<a id="S0141"></a> Source: p.11 S0141
The third principle is more speculative.
<a id="S0142"></a> Source: p.11 S0142
There is a simple and general reason one might expect overfitting to scale ∝ 1/D at very large D.
<a id="S0143"></a> Source: p.11 S0143
Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as 1/D.
<a id="S0144"></a> Source: p.11 S0144
This expectation should hold for any smooth loss function, since we expect to be able to expand the loss about the D → ∞ limit.
<a id="S0145"></a> Source: p.11 S0145
However, this argument assumes that 1/D corrections dominate over other sources of variance, such as the finite batch size and other limits on the efficacy of optimization.
<a id="S0146"></a> Source: p.11 S0146
Without empirical confirmation, we would not be very confident of its applicability.
<a id="S0147"></a> Source: p.11 S0147
Our third principle explains the asymmetry between the roles of N and D in Equation (1.5).
<a id="S0148"></a> Source: p.11 S0148
Very similar symmetric expressions4 are possible, but they would not have a 1/D expansion with integer powers, and would require the introduction of an additional parameter.
<a id="S0149"></a> Source: p.11 S0149
In any case, we will see that our equation for L(N, D) fits the data well, which is the most important justification for our L(N, D) ansatz. 4.2 Results We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing.
<a id="S0150"></a> Source: p.11 S0150
The results are displayed in Figure 9, including a fit to the four parameters α , α , N , D in N D c c Equation (1.5): Parameter α α N D N D c c Value 0.076 0.103 6.4 × 1013 1.8 × 1013 Table 2 Fits to L(N, D) We obtain an excellent fit, with the exception of the runs where the dataset has been reduced by a factor of 1024, to about 2 × 107 tokens.
<a id="S0151"></a> Source: p.11 S0151
With such a small dataset, an epoch consists of only 40 parameter updates.
<a id="S0152"></a> Source: p.11 S0152
Perhaps such a tiny dataset represents a different regime for language modeling, as overfitting happens very early in training (see Figure 16).
<a id="S0153"></a> Source: p.11 S0153
Also note that the parameters differ very slightly from those obtained in Section 3, as here we are fitting the full L(N, D) rather than just L(N, ∞) or L(∞, D).
<a id="S0154"></a> Source: p.11 S0154
To chart the borderlands of the infinite data limit, we can directly study the extent of overfitting.
<a id="S0155"></a> Source: p.11 S0155
For all but the largest models, we see no sign of overfitting when training with the full 22B token WebText2 dataset, so we can take it as representative of D = ∞.
<a id="S0156"></a> Source: p.11 S0156
Thus we can compare finite D to the infinite data limit by 4For example, one might have used L(N, D) = (cid:2)(cid:0) Nc (cid:1)αN + (cid:0) Dc (cid:1)αD (cid:3)β , but this does not have a 1/D expansion. N D 11
Page 12
<a id="S0157"></a> Source: p.12 S0157
106 105 104 103 101 6 × 100 4 × 100 3 × 100 WebText2 Train Loss )snekoT( eziS hctaB lacitirC Critical Batch Size vs.
<a id="S0158"></a> Source: p.12 S0158
Performance Empirical Bcrit, N = 3M Empirical Bcrit, N = 85M Bcrit = 2.1 × 108 tokens L 4.8 Noise Scale Measurement Figure 10 The critical batch size B follows a power law in the loss as performance increase, and does crit not depend directly on the model size.
<a id="S0159"></a> Source: p.12 S0159
We find that the critical batch size approximately doubles for every 13% decrease in loss. B is measured empirically from the data shown in Figure 18, but it is also roughly crit predicted by the gradient noise scale, as in [MKAT18]. defining L(N, D) δL(N, D) ≡ − 1 (4.2) L(N, ∞) and studying it as a function of N, D.
<a id="S0160"></a> Source: p.12 S0160
In fact, we see empirically that δL depends only a specific combination of N and D, as shown in Figure 16.
<a id="S0161"></a> Source: p.12 S0161
This follows from the scaling law of Equation (1.5), which implies (cid:32) (cid:18) (cid:19) αN (cid:33)αD δL ≈ 1 + N αD D c − 1 (4.3) N D c Note that at large D this formula also has a series expansion in powers of 1/D.
<a id="S0162"></a> Source: p.12 S0162
We estimate that the variation in the loss with different random seeds is roughly 0.02, which means that to avoid overfitting when training to within that threshold of convergence we require D (cid:38) (5 × 103) N 0.74 (4.4) With this relation, models smaller than 109 parameters can be trained with minimal overfitting on the 22B token WebText2 dataset, but our largest models will encounter some mild overfitting.
<a id="S0163"></a> Source: p.12 S0163
More generally, this relation shows that dataset size may grow sub-linearly in model size while avoiding overfitting.
<a id="S0164"></a> Source: p.12 S0164
Note however that this does not typically represent maximally compute-efficient training.
<a id="S0165"></a> Source: p.12 S0165
We should also emphasize that we have not optimized regularization (eg the dropout probability) while varying dataset and model size. 5 Scaling Laws with Model Size and Training Time In this section we will demonstrate that a simple scaling law provides a good description for the loss as a function of model size N and training time.
<a id="S0166"></a> Source: p.12 S0166
First we will explain how to use the results of [MKAT18] to define a universal training step S , which accounts for the fact that most of our models have not been min trained at an optimal batch size.
<a id="S0167"></a> Source: p.12 S0167
Then we will demonstrate that we can fit the model size and training time dependence of the loss using Equation (1.6).
<a id="S0168"></a> Source: p.12 S0168
Later we will use these results to predict the optimal allocation of training compute between model size and training time, and then confirm that prediction. 5.1 Adjustment for Training at B (L) crit A simple empirical theory for the batch size dependence of training was developed in [MKAT18] (see also [SLA+18, ZLN+19]).
<a id="S0169"></a> Source: p.12 S0169
It was argued that there is a critical batch size B for training; for B up to B crit crit the batch size can be increased with very minimal degradation in compute-efficiency, whereas for B > B crit increases in B result in diminishing returns.
<a id="S0170"></a> Source: p.12 S0170
It was also argued that the gradient noise scale provides a simple 12
Page 13
<a id="S0171"></a> Source: p.13 S0171
prediction for B , and that neither depends directly on model size except through the value of the loss that crit has been attained.
<a id="S0172"></a> Source: p.13 S0172
These results can be used to predict how training time and compute will vary with the batch size.
<a id="S0173"></a> Source: p.13 S0173
To utilize both training time and compute as effectively as possible, it is best to train with a batch size B ≈ B .
<a id="S0174"></a> Source: p.13 S0174
Training at B (cid:29) B minimizes the number of training steps, while B (cid:28) B minimizes crit crit crit the use of compute.
<a id="S0175"></a> Source: p.13 S0175
More specifically, it was demonstrated that for a wide variety of neural network tasks, the number of training steps S and the number of data examples processed E = BS satisfy the simple relation (cid:18) (cid:19) (cid:18) (cid:19) S E − 1 − 1 = 1 (5.1) S E min min when training to any fixed value of the loss L.
<a id="S0176"></a> Source: p.13 S0176
Here S is the minimum number of steps necessary to reach min L, while E is the minimum number of data examples that must be processed. min We demonstrate the relation (5.1) for Transformers in Figure 18 in the appendix.
<a id="S0177"></a> Source: p.13 S0177
This relation defines the critical batch size E B (L) ≡ min (5.2) crit S min which is a function of the target value of the loss.
<a id="S0178"></a> Source: p.13 S0178
Training at the critical batch size makes a roughly optimal time/compute tradeoff, requiring 2S training steps and processing E = 2E data examples. min min In Figure 10 we have plotted the critical batch size and gradient noise scale5 as a function of training loss for two different models.
<a id="S0179"></a> Source: p.13 S0179
We see that B (L) is independent of model size, and only depends on the loss L.
<a id="S0180"></a> Source: p.13 S0180
So crit the predictions of [MKAT18] continue to hold for Transformer language models.
<a id="S0181"></a> Source: p.13 S0181
The critical batch size can be fit with a power-law in the loss B B (L) ≈ ∗ (5.3) crit L1/αB where B ≈ 2 × 108 and α ≈ 0.21. ∗ B We have chosen this parameterization for B (L) because as the loss approaches its minimum value L , crit min the gradient noise scale is expected to diverge, and we expect B to track this noise scale.
<a id="S0182"></a> Source: p.13 S0182
We do not crit know L , as we see no sign that our models are approaching it, but L > 0 since the entropy of natural min min language is non-zero.
<a id="S0183"></a> Source: p.13 S0183
Since apparently L is much smaller than the values of L we have achieved, we used min a parameterization where B diverges as L → 0. crit We will use B (L) to estimate the relation between the number of training steps S while training at batch crit size B = 219 tokens and the number of training steps while training at B (cid:29) B .
<a id="S0184"></a> Source: p.13 S0184
This is simply crit S S (S) ≡ (minimum steps, at B (cid:29) B ) (5.4) min 1 + B (L)/B crit crit for any given target value L for the loss.
<a id="S0185"></a> Source: p.13 S0185
This also defines a critical value of the compute needed to train to L with a model of size N if we were to train at B (cid:28) B (L).
<a id="S0186"></a> Source: p.13 S0186
This is crit C C (C) ≡ (minimum compute, at B (cid:28) B ) (5.5) min 1 + B/B (L) crit crit where C = 6N BS estimates the (non-embedding) compute used at batch size B. 5.2 Results for L(N, S ) and Performance with Model Size and Compute min Now we will use S defined in Equation (5.4) to obtain a simple and universal fit for the dependence of the min loss on model size and training time in the infinite data limit.
<a id="S0187"></a> Source: p.13 S0187
We will fit the stable, Adam-optimized training runs using Equation (1.6), repeated here for convenience: (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L(N, S ) = c + c (5.6) min N S min for the loss.
<a id="S0188"></a> Source: p.13 S0188
We include all training steps after the warmup period of the learning rate schedule, and find a fit to the data with the parameters: 5Although the critical batch size roughly matches the gradient noise scale, we are using a direct measurements of B from Figures 18 and 10 for all our later analyses. crit 13
Page 14
<a id="S0189"></a> Source: p.14 S0189
8 7 6 5 4 3 2 104 106 108 Parameters (non-embedding) ssoL tseT Performance vs Compute Budget 100 10 1 10 2 10 3 10 4 10 5 ssyad-FP 5.4 4.8 4.2 3.6 3.0 2.4 106 107 108 109 Parameters (non-embedding) ssoL tseT Performance vs Steps 105 104 spetS Figure 11 When we hold either total compute or number of training steps fixed, performance follows L(N, S) from Equation (5.6).
<a id="S0190"></a> Source: p.14 S0190
Each value of compute budget has an associated optimal model size that maximizes performance.
<a id="S0191"></a> Source: p.14 S0191
Mediocre fits at small S are unsurprising, as the power-law equation for the learning curves breaks down very early in training.
<a id="S0192"></a> Source: p.14 S0192
Parameter α α N S N S c c Value 0.077 0.76 6.5 × 1013 2.1 × 103 Table 3 Fits to L(N, S) With these parameters, we obtain the learning curve fits in Figure 4.
<a id="S0193"></a> Source: p.14 S0193
Though the fits are imperfect, we believe they are quite compelling given the simplicity of Equation (5.6).
<a id="S0194"></a> Source: p.14 S0194
The data and fits can be visualized in a different and more interesting way, as shown in Figure 11.
<a id="S0195"></a> Source: p.14 S0195
There we study the test loss as a function of model size while fixing either the total non-embedding compute C used in training, or the number of steps S.
<a id="S0196"></a> Source: p.14 S0196
For the fits we use Equation (5.5) and (5.4) along with the parameters above and Equation (5.6).
<a id="S0197"></a> Source: p.14 S0197
The power-law dependence of the loss on S reflects the interplay of optimizer dynamics and the loss min landscape.
<a id="S0198"></a> Source: p.14 S0198
Since the fits are best late in training, when the loss may be approximately quadratic, the powerlaw should provide information about the spectrum of the Hessian of the loss.
<a id="S0199"></a> Source: p.14 S0199
Its universality suggests that the Hessian eigenvalue density is roughly independent of model size. 5.3 Lower Bound on Early Stopping Step The results for L(N, S ) can be used to derive a lower-bound (and rough estimate) of the step at which min early stopping should occur when training is data limited.
<a id="S0200"></a> Source: p.14 S0200
It is motivated by the idea that finite and infinite D learning curves for a given model will be very similar until we reach S ≈ S .
<a id="S0201"></a> Source: p.14 S0201
Thus overfitting should min stop be proportional to the correction from simply ending training at S .
<a id="S0202"></a> Source: p.14 S0202
This will underestimate S , because stop stop in reality the test loss will decrease more slowly when we have a finite D, and therefore we will require more training steps to reach the optimal test loss at finite D.
<a id="S0203"></a> Source: p.14 S0203
This line of reasoning leads to the inequality S S (N, D) (cid:38) c (5.7) stop [L(N, D) − L(N, ∞)]1/αS where L(N, ∞) is the converged loss, evaluated with infinite available data.
<a id="S0204"></a> Source: p.14 S0204
This inequality and its comparison to the empirical data is displayed in Figure 16 in the appendix.
<a id="S0205"></a> Source: p.14 S0205
In that figure, the values of S stop and L(N, D) are empirical (though S is adjusted to mimic training at B (cid:29) B ), while L(N, ∞) is stop crit computed from the fit to L(N, D) evaluated at D = ∞. 6 Optimal Allocation of the Compute Budget We displayed the empirical trend of performance as a function of the computation used during training in the top-right of Figure 1.
<a id="S0206"></a> Source: p.14 S0206
However, this result involved training at a fixed batch size B, whereas we know 14
Page 15
<a id="S0207"></a> Source: p.15 S0207
Smaller models require more steps to train, while larger models require fewer Models between 0.6x and 2.2x the optimal size can be trained with a 20% larger compute budget Our framework does not capture early training dynamics Figure 12 Left: Given a fixed compute budget, a particular model size is optimal, though somewhat larger or smaller models can be trained with minimal additional compute.
<a id="S0208"></a> Source: p.15 S0208
Right: Models larger than the computeefficient size require fewer steps to train, allowing for potentially faster training if sufficient additional parallelism is possible.
<a id="S0209"></a> Source: p.15 S0209
Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve, after initial transient effects. 7 6 5 4 3 2 10 8 10 6 10 4 10 2 100 Compute (PF-days), non-embedding ssoL tseT L = (C min/2.3 108) 0.050 L = (C/2.0 107) 0.057 Figure 13 When adjusting performance to simulate training far below the critical batch size, we find a somewhat altered power law for L(C ) when compared with the fully empirical results.
<a id="S0210"></a> Source: p.15 S0210
The conspicuous min lump at 10−5 PF-days marks the transition from 1-layer to 2-layer networks; we exclude 1-layer networks in the power-law fits.
<a id="S0211"></a> Source: p.15 S0211
It is the L(C ) trend that we expect to provide a reliable extrapolation for larger min compute. that in fact we could train more efficiently6 by training at the batch size B discussed in Section 5.1. crit Large and small values of the loss could have been achieved with fewer samples or fewer steps, respectively, and correcting for this inefficiency by standardizing to the critical batch size results in cleaner and more predictable trends.
<a id="S0212"></a> Source: p.15 S0212
In this section we will adjust for this oversight.
<a id="S0213"></a> Source: p.15 S0213
More importantly, we will use the results of Section 5 to determine the optimal allocation of compute between model size N and the quantity of data processed during training, namely 2B S .
<a id="S0214"></a> Source: p.15 S0214
We will determine this allocation both empirically and theoretically, by crit min using the equation for L(N, S ), and we will demonstrate that these methods agree. min 6.1 Optimal Performance and Allocations Let us first study the loss as a function of the optimally allocated compute from Equation (5.5).
<a id="S0215"></a> Source: p.15 S0215
The result is plotted in Figure 13, along with a power-law fit.
<a id="S0216"></a> Source: p.15 S0216
We see that as compared to the compute plot of Figure 1, the new fit with C is somewhat improved. min Given L(C ), it is natural to ask for the optimal model size N (C ) that provides the minimal loss with a min min given quantity of training compute.
<a id="S0217"></a> Source: p.15 S0217
The optimal model size is shown in Figure 14.
<a id="S0218"></a> Source: p.15 S0218
We observe that N (C ) min 6One might ask why we did not simply train at B in the first place.
<a id="S0219"></a> Source: p.15 S0219
The reason is that it depends not only on the crit model but also on the target value of the loss we wish to achieve, and so is a moving target. 15
Page 16
<a id="S0220"></a> Source: p.16 S0220
107 105 103 10 7 10 5 10 3 10 1 Compute (PF-days), non-embedding )gniddebme-non( sretemaraP N = (1.3 109) C0.73 min N = (1.6 109) C0.88 15000 10000 5000 0 10 7 10 5 10 3 10 1 Compute (PF-days), excluding embeddings spetS S (adjusted) min S min = (5.4 103) C m 0. i 0 n 3 S (fixed-batch) Figure 14 Left: Each value of the compute budget C has an associated optimal model size N .
<a id="S0221"></a> Source: p.16 S0221
Optimal min model size grows very rapidly with C , increasing by 5x for each 10x increase in compute.
<a id="S0222"></a> Source: p.16 S0222
The number min of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
<a id="S0223"></a> Source: p.16 S0223
Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes. can be fit very well with a power-law N (C ) ∝ (C )0.73. (6.1) min min In Figure 12, we show the effect of training models of sub-optimal sizes (see Appendix B.4).
<a id="S0224"></a> Source: p.16 S0224
By definition C ≡ 6N B S, and so we can use N (C ) to extract further results.
<a id="S0225"></a> Source: p.16 S0225
In particular, since min crit min prior fits show B ∝ L−4.8 and L ∝ C−0.05, we can conclude that B ∝ C0.24.
<a id="S0226"></a> Source: p.16 S0226
This leads us to conclude min crit min that the optimal number of steps will only grow very slowly with compute, as S ∝ (C )0.03, (6.2) min min matching the empirical results in Figure 14.
<a id="S0227"></a> Source: p.16 S0227
In fact the measured exponent is sufficiently small that our results may even be consistent with an exponent of zero.
<a id="S0228"></a> Source: p.16 S0228
Thus we conclude that as we scale up language modeling with an optimal allocation of computation, we should predominantly increase the model size N , while simultaneously scaling up the batch size via B ∝ B with negligible increase in the number of serial steps.
<a id="S0229"></a> Source: p.16 S0229
Since compute-efficient training uses relatively crit few optimization steps, additional work on speeding up early training dynamics may be warranted. 6.2 Predictions from L(N, S ) min The results for L(C ) and the allocations can be predicted from the L(N, S ) equation obtained in min min Section 5.
<a id="S0230"></a> Source: p.16 S0230
Given our equation for L(N, S min ), we can substitute S min = C 6N m B in and then find the minimum of the loss as a function of N , while fixing the training compute.
<a id="S0231"></a> Source: p.16 S0231
We carry out this procedure in detail in Appendix B, where we also provide some additional predictions.
<a id="S0232"></a> Source: p.16 S0232
For the loss as a function of training compute, we predict that (cid:18) Cmin (cid:19)αm C in L(C ) = c (6.3) min C min where 1 αmin ≡ ≈ 0.054 (6.4) C 1/α + 1/α + 1/α S B N in excellent agreement with the exponent of Figure 13.
<a id="S0233"></a> Source: p.16 S0233
We also predict that N (C min ) ∝ (C min )αm C in/αN ≈ (C min )0.71 (6.5) which also matches the scaling of Figure 14 to within a few percent.
<a id="S0234"></a> Source: p.16 S0234
Our scaling laws provide a predictive framework for the performance of language modeling. 16
Page 17
<a id="S0235"></a> Source: p.17 S0235
The intersection point is sensitive to the precise power-law parameters Figure 15 Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(C ) and L(D) due to the slow growth of data needed for compute-efficient training.
<a id="S0236"></a> Source: p.17 S0236
The intersection min marks the point before which we expect our predictions to break down.
<a id="S0237"></a> Source: p.17 S0237
The location of this point is highly sensitive to the precise exponents from our power-law fits. 6.3 Contradictions and a Conjecture We observe no signs of deviation from straight power-law trends at large values of compute, data, or model size.
<a id="S0238"></a> Source: p.17 S0238
Our trends must eventually level off, though, since natural language has non-zero entropy.
<a id="S0239"></a> Source: p.17 S0239
Indeed, the trends for compute-efficient training described in this section already contain an apparent contradiction.
<a id="S0240"></a> Source: p.17 S0240
At scales several orders of magnitude above those documented here, the performance predicted by the L(C ) scaling law decreases below what should be possible given the slow growth in training data with min compute.
<a id="S0241"></a> Source: p.17 S0241
This implies that our scaling laws must break down before this point, but we conjecture that the intersection point has a deeper meaning: it provides an estimate of the point at which Transformer language models reach maximal performance.
<a id="S0242"></a> Source: p.17 S0242
Since the amount of data used by compute-efficient training grows slowly with the compute budget, the performance predicted by L(C ) eventually hits a lower bound set by the L(D) power law (see Figure 15). min Let us work this out in more detail.
<a id="S0243"></a> Source: p.17 S0243
To keep overfitting under control, the results of Section 4 imply that we should scale the dataset size as D ∝ N 0.74 ∝ C0.54 (6.6) min where we have used the compute-efficient N (C ) from Figure 14. min Let us compare this to the data requirements of compute-efficient training.
<a id="S0244"></a> Source: p.17 S0244
If we train at the critical batch size (i.e. C = 2C ) and never re-use data during training, we find that data usage grows with compute as min D(C ) = 2C min ≈ (cid:0) 4 × 1010 tokens (cid:1) (C /PF-Day)0.26 (6.7) min 6N (C ) min min This is the maximum rate at which the dataset size can productively grow with compute, since it means that we are only training for a single epoch.
<a id="S0245"></a> Source: p.17 S0245
But it grows the dataset much more slowly than in Equation (6.6).
<a id="S0246"></a> Source: p.17 S0246
It appears to imply that compute-efficient training will eventually run into a problem with overfitting, even if the training process never re-uses any data!
<a id="S0247"></a> Source: p.17 S0247
According to Figure 1, we expect that when we are bottlenecked by the dataset size (ie by overfitting), the loss should scale as L(D) ∝ D−0.095.
<a id="S0248"></a> Source: p.17 S0248
This implies that the loss would scale with compute as L(D(C )) ∝ min C−0.03 once we are data-limited.
<a id="S0249"></a> Source: p.17 S0249
Once again, we have a contradiction, as this will eventually intersect with min our prediction for L(C ) from Figure 13, where we found a scaling L(C ) ∝ C−0.050. min min min The intersection point of L(D(C )) and L(C ) occurs at min min C∗ ∼ 104 PF-Days N ∗ ∼ 1012 parameters, D∗ ∼ 1012 tokens, L∗ ∼ 1.7 nats/token (6.8) though the numerical values are highly uncertain, varying by an order or magnitude in either direction depending on the precise values of the exponents from the power-law fits.
<a id="S0250"></a> Source: p.17 S0250
The most obvious interpretation is that our scaling laws break down at or before we reach this point, which is still many orders of magnitude away in both compute and model size. 17
Page 18
<a id="S0251"></a> Source: p.18 S0251
One might also conjecture that this intersection point has a deeper meaning.
<a id="S0252"></a> Source: p.18 S0252
If we cannot increase the model size beyond N ∗ without qualitatively different data requirements, perhaps this means that once we reach C∗ and N ∗, we have extracted all of the reliable information available in natural language data.
<a id="S0253"></a> Source: p.18 S0253
In this min interpretation, L∗ would provide a rough estimate for the entropy-per-token7 of natural language.
<a id="S0254"></a> Source: p.18 S0254
In this scenario, we would expect the loss trend to level off at or before L∗.
<a id="S0255"></a> Source: p.18 S0255
We can guess at the functional form of L(C ) as it levels off by considering a version of our training min dataset with added noise.
<a id="S0256"></a> Source: p.18 S0256
For example, we could append a random string of tokens to each context shown to the model to artificially boost the loss by a constant additive factor.
<a id="S0257"></a> Source: p.18 S0257
Then, the distance from the noise floor L − L would be a more meaningful performance metric, with even a small decrease in this distance noise potentially representing a significant boost in qualitative performance.
<a id="S0258"></a> Source: p.18 S0258
Since the artificial noise would affect all of our trends equally, the critical point of 6.8 would not change (aside from the absolute value of L∗), and may be meaningful even if it occurs after the leveling off. 7 Related Work Power laws can arise from a wide variety of sources [THK18].
<a id="S0259"></a> Source: p.18 S0259
Power-law scalings with model and dataset size in density estimation [Was06] and in random forest models [Bia12] may be connected with our results.
<a id="S0260"></a> Source: p.18 S0260
These models suggest that power-law exponents may have a very rough interpretation as the inverse of the number of relevant features in the data.
<a id="S0261"></a> Source: p.18 S0261
Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.
<a id="S0262"></a> Source: p.18 S0262
More recent work [HNA+17, HAD19] also investigated scaling between model size and data size; their work is perhaps the closest to ours in the literature8.
<a id="S0263"></a> Source: p.18 S0263
Note, however, that [HNA+17] found super-linear scaling of dataset size with model size, whereas we find a sub-linear scaling.
<a id="S0264"></a> Source: p.18 S0264
There are some parallels between our findings on optimal allocation of compute and [Kom19], including power-law learning curves.
<a id="S0265"></a> Source: p.18 S0265
EfficientNets [TL19] also appear to obey an approximate power-law relation between accuracy and model size.
<a id="S0266"></a> Source: p.18 S0266
Very recent work [RRBS19b] studies scaling with both dataset size and model size for a variety of datasets, and fits an ansatz similar to ours.
<a id="S0267"></a> Source: p.18 S0267
EfficientNet [TL19] advocates scaling depth and width exponentially (with different coefficients) for optimal performance of image models, resulting in a power-law scaling of width as a function of depth.
<a id="S0268"></a> Source: p.18 S0268
We find that for language models this power should be roughly one when scaling up (as width/depth should remain fixed).
<a id="S0269"></a> Source: p.18 S0269
But more importantly, we find that the precise architectural hyperparameters are unimportant compared to the overall scale of the language model.
<a id="S0270"></a> Source: p.18 S0270
In [VWB16] it was argued that deep models can function as ensembles of shallower models, which could potentially explain this finding.
<a id="S0271"></a> Source: p.18 S0271
Earlier work [ZK16] has compared width and depth, and found that wide ResNets can outperform deep ResNets on image classification.
<a id="S0272"></a> Source: p.18 S0272
Some studies fix computation per data example, which tends to scale in proportion to the number of model parameters, whereas we investigate scaling with both model size and the quantity of training computation.
<a id="S0273"></a> Source: p.18 S0273
Various works [AS17, BHMM18] have investigated generalization in highly overparameterized models, finding a “jamming transition” [GJS+19] when the model size reaches the dataset size (this may require training many orders of magnitude beyond typical practice, and in particular does not use early stopping).
<a id="S0274"></a> Source: p.18 S0274
We do not observe such a transition, and find that the necessary training data scales sublinearly in the model size.
<a id="S0275"></a> Source: p.18 S0275
Expansions in the model size, particularly at large width [JGH18, LXS+19], may provide a useful framework for thinking about some of our scaling relations.
<a id="S0276"></a> Source: p.18 S0276
Our results on optimization, such as the shape of learning curves, can likely be explained using a noisy quadratic model, which can provide quite accurate predictions [ZLN+19] in realistic settings.
<a id="S0277"></a> Source: p.18 S0277
Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18]. 8 Discussion We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N , dataset size D, and optimized training computation C , as encapsulated in Equations (1.5) and min (1.6).
<a id="S0278"></a> Source: p.18 S0278
Conversely, we find very weak dependence on many architectural and optimization hyperparameters.
<a id="S0279"></a> Source: p.18 S0279
Since scalings with N, D, C are power-laws, there are diminishing returns with increasing scale. min 7Defining words using the wc utility, the WebText2 dataset has 1.4 tokens per word and 4.3 characters per token. 8After this work was completed, [RRBS19a] also appeared, which makes similar predictions for the dependence of loss on both model and dataset size. 18
Page 19
<a id="S0280"></a> Source: p.19 S0280
We were able to precisely model the dependence of the loss on N and D, and alternatively on N and S, when these parameters are varied simultaneously.
<a id="S0281"></a> Source: p.19 S0281
We used these relations to derive the compute scaling, magnitude of overfitting, early stopping step, and data requirements when training large language models.
<a id="S0282"></a> Source: p.19 S0282
So our scaling relations go beyond mere observation to provide a predictive framework.
<a id="S0283"></a> Source: p.19 S0283
One might interpret these relations as analogues of the ideal gas law, which relates the macroscopic properties of a gas in a universal way, independent of most of the details of its microscopic consituents.
<a id="S0284"></a> Source: p.19 S0284
It is natural to conjecture that the scaling relations will apply to other generative modeling tasks with a maximum likelihood loss, and perhaps in other settings as well.
<a id="S0285"></a> Source: p.19 S0285
To this purpose, it will be interesting to test these relations on other domains, such as images, audio, and video models, and perhaps also for random network distillation.
<a id="S0286"></a> Source: p.19 S0286
At this point we do not know which of our results depend on the structure of natural language data, and which are universal.
<a id="S0287"></a> Source: p.19 S0287
It would also be exciting to find a theoretical framework from which the scaling relations can be derived: a ‘statistical mechanics’ underlying the ‘thermodynamics’ we have observed.
<a id="S0288"></a> Source: p.19 S0288
Such a theory might make it possible to derive other more precise predictions, and provide a systematic understanding of the limitations of the scaling laws.
<a id="S0289"></a> Source: p.19 S0289
In the domain of natural language, it will be important to investigate whether continued improvement on the loss translates into improvement on relevant language tasks.
<a id="S0290"></a> Source: p.19 S0290
Smooth quantitative change can mask major qualitative improvements: “more is different”.
<a id="S0291"></a> Source: p.19 S0291
For example, the smooth aggregate growth of the economy provides no indication of the specific technological developments that underwrite it.
<a id="S0292"></a> Source: p.19 S0292
Similarly, the smooth improvements in language model loss may hide seemingly qualitative changes in capability.
<a id="S0293"></a> Source: p.19 S0293
Our results strongly suggest that larger models will continue to perform better, and will also be much more sample efficient than has been previously appreciated.
<a id="S0294"></a> Source: p.19 S0294
Big models may be more important than big data.
<a id="S0295"></a> Source: p.19 S0295
In this context, further investigation into model parallelism is warranted.
<a id="S0296"></a> Source: p.19 S0296
Deep models can be trained using pipelining [HCC+18], which splits parameters depth-wise between devices, but eventually requires increased batch sizes as more devices are used.
<a id="S0297"></a> Source: p.19 S0297
Wide networks on the other hand are more amenable to parallelization [SCP+18], since large layers can be split between multiple workers with less serial dependency.
<a id="S0298"></a> Source: p.19 S0298
Sparsity [CGRS19, GRK17] or branching (e.g. [KSH12]) may allow for even faster training of large networks through increased model parallelism.
<a id="S0299"></a> Source: p.19 S0299
And using methods like [WRH17, WYL19], which grow networks as they train, it might be possible to remain on the compute-efficient frontier for an entire training run.
<a id="S0300"></a> Source: p.19 S0300
Acknowledgements We would like to thank Shan Carter, Paul Christiano, Jack Clark, Ajeya Cotra, Ethan Dyer, Jason Eisner, Danny Hernandez, Jacob Hilton, Brice Menard, Chris Olah, and Ilya Sutskever for discussions and for feedback on drafts of this work. 19
Page 20
<a id="S0301"></a> Source: p.20 S0301
Appendices A Summary of Power Laws For easier reference, we provide a summary below of the key trends described throughout the paper.
<a id="S0302"></a> Source: p.20 S0302
Parameters Data Compute Batch Size Equation N ∞ ∞ Fixed L (N ) = (N /N )αN c ∞ D Early Stop Fixed L (D) = (D /D)αD c Optimal ∞ C Fixed L (C) = (C /C)αC (naive) c N D C B (cid:28) B L (C ) = (cid:0) Cmin/C (cid:1)αm C in opt opt min crit min c min N D Early Stop Fixed L (N, D) = (cid:20) (cid:0) Nc (cid:1) α α N D + Dc (cid:21)αD N D N ∞ S steps B L (N, S) = (cid:0) Nc (cid:1)αN + (cid:16) Sc (cid:17)αS N Smin(S,B) Table 4 The empirical fitted values for these trends are: Power Law Scale (tokenization-dependent) α = 0.076 N = 8.8 × 1013 params (non-embed) N c α = 0.095 D = 5.4 × 1013 tokens D c α = 0.057 C = 1.6 × 107 PF-days C c αmin = 0.050 Cmin = 3.1 × 108 PF-days C c α = 0.21 B = 2.1 × 108 tokens B ∗ α = 0.76 S = 2.1 × 103 steps S c Table 5 The optimal parameters for compute efficient training are given by: Compute-Efficient Value Power Law Scale N = N · CpN p = 0.73 N = 1.3 · 109 params opt e min N e B (cid:28) B crit = L1 B /α ∗ B = B e C m pB in p B = 0.24 B e = 2.0 · 106 tokens S = S · CpS (lower bound) p = 0.03 S = 5.4 · 103 steps min e min S e D = D · CpD (1 epoch) p = 0.27 D = 2 · 1010 tokens opt e min D e Table 6 B Empirical Model of Compute-Efficient Frontier Throughout this appendix all values of C, S, and α are adjusted for training at the critical batch size B . C crit We have left off the ‘adj’ label to avoid cluttering the notation. B.1 Defining Equations The power-law fit to the learning curves implies a simple prescription for compute-efficient training.
<a id="S0303"></a> Source: p.20 S0303
In this appendix, we will derive the optimal performance, model size, and number of training steps as a function of 20
Page 21
<a id="S0304"></a> Source: p.21 S0304
We start with the Equation (1.6), repeated here for convenience: (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L (N, S) = c + c . (B.1) N S Here, S represents the number of parameter updates when training at the critical batch size [MKAT18], which was defined in Equation (5.2)9: B B (L) = ∗ . (B.2) L1/αB We would like to determine optimal training parameters for a fixed compute budget, so we replace S = C/ (6N B (L)), where C is the number of FLOPs used in the training run: (cid:18) N (cid:19)αN (cid:18) N (cid:19)αS L (N, C) = c + 6B S . (B.3) N ∗ c L1/αB C (cid:12) Now, we set ∂ N L(cid:12) C = 0 to find the condition for optimality: ∂L (cid:12) 0 = (cid:12) ∂N C = − α N N (cid:18) N N c (cid:19)αN + α N S (cid:18) 6B ∗ S c L1/ N αB C (cid:19)αS (cid:18) 1 − 5 N L (cid:26)∂ ∂ N L (cid:26)(cid:12) (cid:12) (cid:26) C (cid:19) α (cid:18) N (cid:19)αN (cid:18) N (cid:19)αS =⇒ N c = 6B S (B.4) α S N ∗ c L1/αB C Equation (B.3) and (B.4) together determine the compute-efficient frontier. B.2 Efficient Training Now we assemble the implications of (B.3) and (B.4).
<a id="S0305"></a> Source: p.21 S0305
First, note that inserting (B.4) into (B.3) yields (cid:18) (cid:19) α L (N (C) , C) = 1 + N L (N , ∞) , (B.5) eff α eff S which implies that for compute-efficient training, we should train to a fixed percentage αN ≈ 10% above αS the converged loss.
<a id="S0306"></a> Source: p.21 S0306
Next, let’s determine how the optimal loss depends on the compute budget.
<a id="S0307"></a> Source: p.21 S0307
Eliminating N yields a power-law dependence of performance on compute: (cid:18) C (cid:19)αC L (C) = c (B.6) C where we defined α = 1/ (1/α + 1/α + 1/α ) ≈ 0.052 (B.7) C S B N (cid:18) α (cid:19)1/αS+1/αN (cid:18) α (cid:19)1/αS C = 6N B S 1 + N S . (B.8) c c ∗ c α α S N Similarly, we can eliminate L to find N (C): N (C) (cid:18) C (cid:19)αC/αN (cid:18) α (cid:19)1/αN = 1 + N (B.9) N C α c c S and C (cid:18) α (cid:19)−1/αN (cid:18) C (cid:19)αC/αS S (C) = c 1 + N (B.10) 6N B α C c ∗ S c 9There is a slight ambiguity here: we can imagine training either at a constant batch size B (L ), or we could target instead train at a variable batch size B˜ (L), where B˜ is the instantaneous critical batch size (as opposed to B, which is the averaged version).
<a id="S0308"></a> Source: p.21 S0308
These two prescriptions result in the same number of steps, so we can ignore this subtlety (see [MKAT18]). 21
Page 22
<a id="S0309"></a> Source: p.22 S0309
B.3 Comparison to Inefficient Typically, researchers train models until they appear to be close to convergence.
<a id="S0310"></a> Source: p.22 S0310
In this section, we compare the efficient training procedure described above to this more typical setup.
<a id="S0311"></a> Source: p.22 S0311
We define a the convergence factor f as the percent deviation from the converged loss: L (N, C) = (1 + f ) L (N, ∞) . (B.11) For compute-efficient training we have f = α /α ≈ 10% from the previous section, but researchers N S typically use a much smaller value.
<a id="S0312"></a> Source: p.22 S0312
Here, we choose f (cid:48) = 2% as an estimate.
<a id="S0313"></a> Source: p.22 S0313
For a fixed value of the loss, we predict: N (cid:18) 1 + f (cid:19)1/αN f = ≈ 2.7 (B.12) N 1 + f (cid:48) f(cid:48) S (cid:32) 1 + 1 (cid:33)1/αS f = f ≈ 0.13 (B.13) S 1 + 1 f(cid:48) f(cid:48) C N S f = f f ≈ 0.35 (B.14) C N S f(cid:48) f(cid:48) f(cid:48) So that compute-efficient training uses 7.7x fewer parameter updates, 2.7x more parameters, and 65% less compute to reach the same loss. B.4 Suboptimal Model Sizes We can solve A.1 to find an expression for the amount of compute needed to reach a given value of the loss L with a model of size N : (cid:18) N (cid:19) (cid:18) (cid:18) N (cid:19)αN (cid:19)−1/αS C (N, L) = 6B S L − c . (B.15) ∗ c L1/αB N Using A.6 and A.9, we can eliminate L in favor of N (L), the model size which reaches L most efficiently. eff From there, we find an expression for the excess compute needed as a consequence of using a suboptimal model size: C (N, N ) N (cid:20) α (cid:18) (cid:18) N (cid:19)αN (cid:19)(cid:21)−1/αS eff = 1 + S 1 − eff . (B.16) C (N , N ) N α N eff eff eff N The result is shown in Figure X.
<a id="S0314"></a> Source: p.22 S0314
Models between 0.6x and 2.2x the optimal size can be used with only a 20% increase in compute budget.
<a id="S0315"></a> Source: p.22 S0315
Using a smaller model is useful when accounting for the cost inference. A larger model can be trained the the same level of performance in fewer steps, allowing for more parallelism and faster training if sufficient harware is available (see Figure Y): S (N, N ) (cid:20) α (cid:18) (cid:18) N (cid:19)αN (cid:19)(cid:21)−1/αS eff = 1 + S 1 − eff . (B.17) S (N , N ) α N eff eff N A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute.
<a id="S0316"></a> Source: p.22 S0316
Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve after initial transient effects. C Caveats In this section we list some potential caveats to our analysis. • At present we do not have a solid theoretical understanding for any of our proposed scaling laws.
<a id="S0317"></a> Source: p.22 S0317
The scaling relations with model size and compute are especially mysterious.
<a id="S0318"></a> Source: p.22 S0318
It may be possible to understand scaling at very large D holding model size fixed [AS17], and also the shape of learning curves late in training, by modeling the loss with a noisy quadratic.
<a id="S0319"></a> Source: p.22 S0319
But the scaling with D at very large model size still remains mysterious.
<a id="S0320"></a> Source: p.22 S0320
Without a theory or a systematic understanding of the corrections to our scaling laws, it’s difficult to determine in what circumstances they can be trusted. 22
Page 23
<a id="S0321"></a> Source: p.23 S0321
105 104 103 103 104 105 Sc × [L(N, D) L(N, )] 1/ S S pots 6 Early Stopping Step 5 Data Size 4 21M 43M 86M 172M 3 344M 688M 1.4B 2 103 104 105 Step ssoL Test Loss 1010 Train Loss 109 108 )snekoT( eziS tesataD Figure 16 Left: We characterize the step on which early stopping occurs, as a function of the extent of overfitting.
<a id="S0322"></a> Source: p.23 S0322
The red line indicates a lower bound for early stopping that is derived in Section 5.3.
<a id="S0323"></a> Source: p.23 S0323
Right: We display train and test loss for a series of 300M parameter models trained on different sized dataset subsamples.
<a id="S0324"></a> Source: p.23 S0324
The test loss typically follows that of a run done with unrestricted data until diverging.
<a id="S0325"></a> Source: p.23 S0325
Note that the degree of overfitting (as compared to the infinite data limit) is significantly overestimated by L − L test train (denoted by a black bar for each run). • We are not especially confident in the prediction of B (L) for values of the loss far outside the crit range we have explored.
<a id="S0326"></a> Source: p.23 S0326
Changes in B could have a significant impact on trade-offs between crit data parallelism and the number of serial training steps required, which would have a major impact on training time. • We did not thoroughly investigate the small data regime, and our fits for L(N, D) were poor for the smallest values of D (where an epoch corresponded to only 40 steps).
<a id="S0327"></a> Source: p.23 S0327
Furthermore, we did not experiment with regularization and data augmentation.
<a id="S0328"></a> Source: p.23 S0328
Improvements in these could alter our results, quantitatively or qualitatively. • We used the estimated training compute C ≈ 6N BS, which did not include contributions proportional to n (see Section 2.1).
<a id="S0329"></a> Source: p.23 S0329
So our scalings with compute may be confounded in practice in the ctx regime of very large n , specifically where n (cid:38) 12d . ctx ctx model • We tuned learning rates, and we experimented with learning rate schedules.
<a id="S0330"></a> Source: p.23 S0330
But we may have neglected to tune some hyperparameter (e.g. intialization scale or momentum) that have an important effect on scaling. • The optimal choice of learning rate is sensitive to the target loss.
<a id="S0331"></a> Source: p.23 S0331
When training close to convergence, it may be necessary to use a smaller learning rate to avoid divergences.
<a id="S0332"></a> Source: p.23 S0332
But when conducting a short training run (eg due to compute limitations), it may be possible to use a larger learning rate.
<a id="S0333"></a> Source: p.23 S0333
We did not experiment with higher learning rates for training runs that did not proceed to convergence. D Supplemental Figures D.1 Early Stopping and Test vs Train In section 5.3 we described the result shown in Figure 16, which provides a prediction for a lower bound on the early stopping step.
<a id="S0334"></a> Source: p.23 S0334
We also show the train and test loss for a given model size when training on different sized datasets. D.2 Universal Transformers We compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17.
<a id="S0335"></a> Source: p.23 S0335
These models re-use parameters, and so perform slightly better as a function of N , but slightly worse as a function of compute C.
<a id="S0336"></a> Source: p.23 S0336
We include several different different possibilities for parameter re-use. D.3 Batch Size We measure the critical batch size using the data displayed in figure 18.
<a id="S0337"></a> Source: p.23 S0337
This made it possible to estimate B (L) in figure 10. crit 23
Page 24
<a id="S0338"></a> Source: p.24 S0338
4.5 4.0 3.5 3.0 2.5 105 106 107 108 109 Parameters, including reuse (non-embedding) ssoL tseT 4.5 4.0 3.5 2x Reuse 3.0 4x Reuse 8x Reuse Non-recurrent Models 2.5 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 2x Reuse 4x Reuse 8x Reuse Non-recurrent Models Figure 17 We compare recurrent Transformers [DGV+18], which re-use parameters, to standard Transformers.
<a id="S0339"></a> Source: p.24 S0339
Recurrent Transformers perform slightly better when comparing models with equal parameter count, but slightly worse when accounting for reuse and comparing per FLOP. 1011 1010 109 108 107 106 102 103 104 105 Step dessecorP snekoT Batch Size Scan - 3M Params 10 8 6 4 ssoL tseT 1010 108 106 101 102 103 104 105 Step dessecorP snekoT Batch Size Scan - 85M Params 10 8 6 4 ssoL tseT Figure 18 These figures demonstrate fits to Equation (5.1) for a large number of values of the loss L, and for two different Transformer model sizes.
<a id="S0340"></a> Source: p.24 S0340
These fits were used to measure B (L) for Figure 10. crit D.4 Sample Efficiency vs Model Size It is easy to see from figure 2 that larger models train faster, and are therefore more sample efficient.
<a id="S0341"></a> Source: p.24 S0341
We provide another way of looking at this phenomenon in figure 19, which shows when different models reach various fixed values of the loss. 105 104 103 106 107 108 Parameters (non-embedding) ) S( spetS muminiM nim 5.5 5.0 4.5 4.0 3.5 3.0 2.5 ssoL 1011 1010 109 108 106 107 108 Parameters (non-embedding) ) E( selpmaxE muminiM nim 5.5 5.0 4.5 4.0 3.5 3.0 2.5 ssoL Figure 19 The number of minimum serial steps needed to reach any fixed value of the test loss decreases precipitously with model size.
<a id="S0342"></a> Source: p.24 S0342
Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to a very large one. 24
Page 25
<a id="S0343"></a> Source: p.25 S0343
8 7 6 5 4 3 100 101 102 103 Token Index ssoL tseT nekoT-reP 4.0+3.2 T 0.47 3.4+4.0 T 0.56 2 2 . . 9 7 + + 4 4 . . 5 9 T T 0 0 . . 5 6 6 0 108 2.4+5.1 T 0.61 2.3+5.4 T 0.62 107 106 sretemaraP ledoM 10 8 6 4 2 101 103 105 Step ssoL tseT Per-token Loss (774M Params) 103 102 101 100 xednI nekoT Figure 20 This figure provides information about the performance per token as a function of model size and training time.
<a id="S0344"></a> Source: p.25 S0344
Left: Loss per token as a function of its position T in the 1024-token context.
<a id="S0345"></a> Source: p.25 S0345
Loss scales predictably as a power-law in T .
<a id="S0346"></a> Source: p.25 S0346
Right: Test loss per token as a function of training step. 7.5 6.0 4.5 3.0 104 105 106 107 108 109 Parameters (excl. embedding) ssoL tseT Token 1/1024 Token 2/1024 Token 4/1024 Token 8/1024 Token 16/1024 Token 64/1024 Token 256/1024 Token 1024/1024 Token 1/8 Token 2/8 Token 4/8 Token 8/8 Figure 21 In addition to the averaged loss, individual tokens within the 1024-token context also improve smoothly as model size increases.
<a id="S0347"></a> Source: p.25 S0347
Training runs with shorter context n = 8 (dashed lines) perform better ctx on early tokens, since they can allocate all of their capacity to them. D.5 Context Dependence The trends for loss as a function of model size are displayed for different tokens in the context in Figure 21.
<a id="S0348"></a> Source: p.25 S0348
We see that models trained on n = 1024 show steady improvement with model size on all but the first ctx token.
<a id="S0349"></a> Source: p.25 S0349
Fixing model size, it appears that the loss scales as a power-law as a function of position T in the context, see Figure 20.
<a id="S0350"></a> Source: p.25 S0350
This may be a consequence of underlying power-law correlations in language [EP94, ACDE12, LT16], or a more general feature of the model architecture and optimization.
<a id="S0351"></a> Source: p.25 S0351
It provides some suggestion for the potential benefits (or lack thereof) from training on larger contexts.
<a id="S0352"></a> Source: p.25 S0352
Not only do larger models converge to better performance at T = 1024, but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting patterns with less contextual information.
<a id="S0353"></a> Source: p.25 S0353
In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step.
<a id="S0354"></a> Source: p.25 S0354
The model begins by learning short-range information, and only learns longer-range correlations later in training.
<a id="S0355"></a> Source: p.25 S0355
We have also included models trained with a tiny context n = 8 in order to compare with our longer ctx context models.
<a id="S0356"></a> Source: p.25 S0356
Even modestly sized models trained on n = 8 can dominate our largest n = 1024 ctx ctx models on very early tokens.
<a id="S0357"></a> Source: p.25 S0357
This also suggests that further improvements should be possible with much larger models trained on large contexts. D.6 Learning Rate Schedules and Error Analysis We experimented with a variety of learning rates and schedules. A host of schedules and resulting test performances for a small language model are plotted in Figure 22.
<a id="S0358"></a> Source: p.25 S0358
We conclude that the choice of learning rate schedule is mostly irrelevant, as long as the total summed learning rate is sufficiently large, and the schedule includes a warmup period and a final decay to near-vanishing learning rate.
Page 26
<a id="S0359"></a> Source: p.26 S0359
0.0010 0.0008 0.0006 0.0004 0.0002 0.0000 0 50000 100000 150000 200000 250000 Step etaR gninraeL 3.90 3.85 3.80 3.75 3.70 3.65 50 100 150 200 250 LR Summed Over Steps ssoL Figure 22 We test a variety of learning rate schedules including cosine decay, linear decay, as well as other faster/slower decays schedules on a 3 million parameter model, shown on the left.
<a id="S0360"></a> Source: p.26 S0360
For these experiments we do not decay to zero, since we find that this tends to give a fixed improvement close to the end of training.
<a id="S0361"></a> Source: p.26 S0361
We find that, as long as the learning rate is not too small and does not decay too quickly, performance does not depend strongly on learning rate.
<a id="S0362"></a> Source: p.26 S0362
Run-to-run variation is at the level of 0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level. 6 5 4 3 2 104 105 106 107 108 109 Parameters (non-embedding) )ecnegrevnoc ta( ssoL tseT L = (N/8.8 1013) 0.076 L = 0.25log(N/7.1 1012) Figure 23 The trend for performance as a function of parameter count, L(N ), is fit better by a power law than by other functions such as a logarithm at a qualitative level. schedules appear to be statistical noise, and provide a rough gauge for the scale of variation between different training runs.
<a id="S0363"></a> Source: p.26 S0363
Experiments on larger models suggest that the variation in the final test loss between different random seeds is roughly constant in magnitude for different model sizes.
<a id="S0364"></a> Source: p.26 S0364
We found that larger models require a smaller learning rate to prevent divergence, while smaller models can tolerate a larger learning rate.
<a id="S0365"></a> Source: p.26 S0365
To implement this, the following rule of thumb was used for most runs: LR(N ) ≈ 0.003239 + −0.0001395 log(N ) (D.1) We expect that this formula could be improved.
<a id="S0366"></a> Source: p.26 S0366
There may be a dependence on network width, likely set by the initialization scale.
<a id="S0367"></a> Source: p.26 S0367
The formula also breaks down for N > 1010 parameters.
<a id="S0368"></a> Source: p.26 S0368
Nevertheless, we found that it works sufficiently well for the models we considered. D.7 Fit Details and Power Law Quality We experimented with a number of functional forms for the fits to L(N ), L(C), and L(D); the power-law fits were qualitatively much more accurate than other functions such as logarithms (see Figure 23).
<a id="S0369"></a> Source: p.26 S0369
For L(C), we do not include small models with only 1 layer in the fit, as the transition from 1 to 2 layers causes a noticable lump in the data.
<a id="S0370"></a> Source: p.26 S0370
For L(N ) we also do not include very small models with only 1 layer in the fit, and we exclude the largest models that have not trained fully to convergence.
<a id="S0371"></a> Source: p.26 S0371
Fit parameters change marginally if we do include them, and the trend extrapolates well in both directions regardless. D.8 Generalization and Architecture In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed.
<a id="S0372"></a> Source: p.26 S0372
It seems to depend only on the performance on the training distribution. 26
Page 27
<a id="S0373"></a> Source: p.27 S0373
2.8 2.7 2.6 2.5 2.4 2.3 101 102 Depth ssoL tseT Wikipedia Books Internet Books Common Crawl WebText2 (Train) WebText2 (Test) Figure 24 We show evaluations on a series of datasets for models with approximately 1.5 Billion parameters.
<a id="S0374"></a> Source: p.27 S0374
We observe no effect of depth on generalization; generalization performance depends primarily on training distribution performance.
<a id="S0375"></a> Source: p.27 S0375
The 12-layer model overfit the Internet Books dataset and we show the early-stopped performance; we have not seen this surprising result in other experiments.
<a id="S0376"></a> Source: p.27 S0376
List of Figures 1 Summary of simple power laws. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Illustration of sample efficiency and compute efficiency. . . . . . . . . . . . . . . . . . . . . 4 3 How to scale up model size, batch size, and serial steps . . . . . . . . . . . . . . . . . . . . 4 4 Performance when varying model and data size, or model and training steps, simultaneously 5 5 Weak dependence of performance on hyperparameter tuning . . . . . . . . . . . . . . . . . 8 6 Comparison of performance trend when including or excluding embeddings . . . . . . . . . 8 7 LSTM and Transformer performance comparison . . . . . . . . . . . . . . . . . . . . . . . 9 8 Generalization to other test datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 9 Universality of overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 10 Critical batch size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 11 Performance versus compute budget or number of parameter updates . . . . . . . . . . . . . 14 12 Training on suboptimal models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 13 Comparison between empirical and adjusted compute trends . . . . . . . . . . . . . . . . . 15 14 Optimal model size and serial number of steps versus compute budget . . . . . . . . . . . . 16 15 Contradiction between compute and data trends . . . . . . . . . . . . . . . . . . . . . . . . 17 16 Early stopping lower bound and training curves for overfit models . . . . . . . . . . . . . . 23 17 Universal transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 18 Batch size scans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 19 Another look at sample efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 20 Power-law dependence of performance on position in context . . . . . . . . . . . . . . . . . 25 21 Performance at different context positions versus model size . . . . . . . . . . . . . . . . . 25 22 Learning rate schedule scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 23 Comparison of Power-Law and Logarithmic Fits . . . . . . . . . . . . . . . . . . . . . . . 26 24 Generalization versus depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 27
Page 28
<a id="S0377"></a> Source: p.28 S0377
List of Tables 1 Parameter and compute counts for Transformer . . . . . . . . . . . . . . . . . . . . . . . . 7 2 Fits to L(N, D) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Fits to L(N, S) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4 Key trend equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5 Key parameters to trend fits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 6 Trends for compute-efficient training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 References [ACDE12] Eduardo G Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti.
<a id="S0378"></a> Source: p.28 S0378
On the origin of longrange correlations in texts.
<a id="S0379"></a> Source: p.28 S0379
Proceedings of the National Academy of Sciences, 109(29):11582– 11587, 2012. 25 [AS17] Madhu S.
<a id="S0380"></a> Source: p.28 S0380
High-dimensional dynamics of generalization error in neural networks. arXiv, 2017, 1710.03667. 11, 18, 22 [BB01] Michele Banko and Eric Brill.
<a id="S0381"></a> Source: p.28 S0381
Scaling to very very large corpora for natural language disambiguation.
<a id="S0382"></a> Source: p.28 S0382
In Proceedings of the 39th annual meeting on association for computational linguistics, pages 26–33.
<a id="S0383"></a> Source: p.28 S0383
Association for Computational Linguistics, 2001. 18 [BHMM18] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.
<a id="S0384"></a> Source: p.28 S0384
Reconciling modern machine learning and the bias-variance trade-off. arXiv, 2018, 1812.11118. 18 [Bia12] GÊrard Biau.
<a id="S0385"></a> Source: p.28 S0385
Journal of Machine Learning Research, 13(Apr):1063–1095, 2012. 18 [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
<a id="S0386"></a> Source: p.28 S0386
Generating long sequences with sparse transformers.
<a id="S0387"></a> Source: p.28 S0387
URL http://arxiv.org/ abs/1904.10509. 19 [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
<a id="S0388"></a> Source: p.28 S0388
Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805. 2 [DGV+18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.
<a id="S0389"></a> Source: p.28 S0389
URL http://arxiv.org/ abs/1807.03819. 6, 9, 23, 24 [EP94] Werner Ebeling and Thorsten Pöschel.
<a id="S0390"></a> Source: p.28 S0390
Entropy and long-range correlations in literary english.
<a id="S0391"></a> Source: p.28 S0391
EPL (Europhysics Letters), 26(4):241, 1994. 25 [Fou] The Common Crawl Foundation.
<a id="S0392"></a> Source: p.28 S0392
URL http://commoncrawl.org. 7 [GARD18] Guy Gur-Ari, Daniel A.
<a id="S0393"></a> Source: p.28 S0393
Gradient descent happens in a tiny subspace. 2018, arXiv:1812.04754. 18 [GJS+19] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart.
<a id="S0394"></a> Source: p.28 S0394
Scaling description of generalization with number of parameters in deep learning. arXiv, 2019, 1901.01608. 18 [GKX19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.
<a id="S0395"></a> Source: p.28 S0395
An investigation into neural net optimization via hessian eigenvalue density.
<a id="S0396"></a> Source: p.28 S0396
URL http://arxiv.org/abs/1901.10159. 18 [Goo01] Joshua Goodman. A bit of progress in language modeling.
<a id="S0397"></a> Source: p.28 S0397
URL http://arxiv.org/abs/cs.CL/0108005. 18 [GRK17] Scott Gray, Alec Radford, and Diederik P Kingma.
<a id="S0398"></a> Source: p.28 S0398
Gpu kernels for block-sparse weights. openai.com, 2017. 19 [HAD19] Joel Hestness, Newsha Ardalani, and Gregory Diamos.
<a id="S0399"></a> Source: p.28 S0399
Beyond human-level accuracy: Computational challenges in deep learning.
<a id="S0400"></a> Source: p.28 S0400
In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming, PPoPP ’19, pages 1–14, New York, NY, USA, 2019.
Page 29
<a id="S0401"></a> Source: p.29 S0401
[HCC+18] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V.
<a id="S0402"></a> Source: p.29 S0402
Gpipe: Efficient training of giant neural networks using pipeline parallelism.
<a id="S0403"></a> Source: p.29 S0403
URL http://arxiv.org/abs/1811.06965. 19 [HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.
<a id="S0404"></a> Source: p.29 S0404
Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
<a id="S0405"></a> Source: p.29 S0405
Deep learning scaling is predictable, empirically, 2017, 1712.00409. 18 [JGH18] Arthur Jacot, Franck Gabriel, and Clément Hongler.
<a id="S0406"></a> Source: p.29 S0406
Neural tangent kernel: Convergence and generalization in neural networks.
<a id="S0407"></a> Source: p.29 S0407
In Advances in neural information processing systems, pages 8571–8580, 2018. 18 [KB14] Diederik P.
<a id="S0408"></a> Source: p.29 S0408
Adam: A method for stochastic optimization, 2014, 1412.6980. 7 [Kom19] Aran Komatsuzaki.
<a id="S0409"></a> Source: p.29 S0409
One epoch is all you need, 2019, arXiv:1906.06669. 18 [KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E.
<a id="S0410"></a> Source: p.29 S0410
Imagenet classification with deep convolutional neural networks.
<a id="S0411"></a> Source: p.29 S0411
In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA, 2012.
<a id="S0412"></a> Source: p.29 S0412
URL http://dl.acm.org/citation.cfm?id=2999134.2999257. 19 [LCG+19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.
<a id="S0413"></a> Source: p.29 S0413
Albert: A lite bert for self-supervised learning of language representations, 2019, 1909.11942. 9 [LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
<a id="S0414"></a> Source: p.29 S0414
Roberta: A robustly optimized BERT pretraining approach.
<a id="S0415"></a> Source: p.29 S0415
URL http://arxiv.org/abs/ 1907.11692. 2 [LSP+18] Peter J.
<a id="S0416"></a> Source: p.29 S0416
Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.
<a id="S0417"></a> Source: p.29 S0417
Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198.
<a id="S0418"></a> Source: p.29 S0418
URL http://arxiv.org/abs/1801.10198. 2, 6 [LT16] Henry W Lin and Max Tegmark.
<a id="S0419"></a> Source: p.29 S0419
Criticality in formal languages and statistical physics. arXiv preprint arXiv:1606.06737, 2016. 25 [LXS+19] Jaehoon Lee, Lechao Xiao, Samuel S.
<a id="S0420"></a> Source: p.29 S0420
Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington.
<a id="S0421"></a> Source: p.29 S0421
Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720. 18 [MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team.
<a id="S0422"></a> Source: p.29 S0422
An empirical model of large-batch training, 2018, arXiv:1812.06162. 3, 5, 6, 12, 13, 21 [Pap18] Vardan Papyan.
<a id="S0423"></a> Source: p.29 S0423
The full spectrum of deep net hessians at scale: Dynamics with sample size.
<a id="S0424"></a> Source: p.29 S0424
URL http://arxiv.org/abs/1811.07062. 18 [RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
<a id="S0425"></a> Source: p.29 S0425
Improving language understanding by generative pre-training.
<a id="S0426"></a> Source: p.29 S0426
URL https://s3-us-west-2. amazonaws. com/openaiassets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. 2, 6 [RRBS19a] Jonathan S.
<a id="S0427"></a> Source: p.29 S0427
Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, 1909.12673. 18 [RRBS19b] Jonathan S.
<a id="S0428"></a> Source: p.29 S0428
Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, arXiv:1909.12673. 18 [RSR+19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J.
<a id="S0429"></a> Source: p.29 S0429
Exploring the limits of transfer learning with a unified text-to-text transformer, 2019, arXiv:1910.10683. 2 [RWC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
<a id="S0430"></a> Source: p.29 S0430
Language models are unsupervised multitask learners. openai.com, 2019. 2, 5, 6, 7, 8 [SCP+18] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman.
<a id="S0431"></a> Source: p.29 S0431
Mesh-tensorflow: Deep learning for supercomputers, 2018, 1811.02084. 19 [SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch.
<a id="S0432"></a> Source: p.29 S0432
Neural machine translation of rare words with subword units.
Page 30
<a id="S0433"></a> Source: p.30 S0433
Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E.
<a id="S0434"></a> Source: p.30 S0434
Measuring the effects of data parallelism on neural network training, 2018, arXiv:1811.03600. 12 [SS18] Noam Shazeer and Mitchell Stern.
<a id="S0435"></a> Source: p.30 S0435
Adafactor: Adaptive learning rates with sublinear memory cost.
<a id="S0436"></a> Source: p.30 S0436
URL http://arxiv.org/abs/1804.04235. 7 [THK18] Stefan Thurner, Rudolf Hanel, and Peter Klimek.
<a id="S0437"></a> Source: p.30 S0437
Introduction to the theory of complex systems.
<a id="S0438"></a> Source: p.30 S0438
Oxford University Press, 2018. 18 [TL19] Mingxing Tan and Quoc V.
<a id="S0439"></a> Source: p.30 S0439
Efficientnet: Rethinking model scaling for convolutional neural networks.
<a id="S0440"></a> Source: p.30 S0440
URL http://arxiv.org/abs/1905. 11946. 18 [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.
<a id="S0441"></a> Source: p.30 S0441
Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008.
<a id="S0442"></a> Source: p.30 S0442
URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. 2, 6 [VWB16] Andreas Veit, Michael Wilber, and Serge Belongie.
<a id="S0443"></a> Source: p.30 S0443
Residual networks behave like ensembles of relatively shallow networks, 2016, arXiv:1605.06431. 8, 18 [Was06] Larry Wasserman.
<a id="S0444"></a> Source: p.30 S0444
Springer Science & Business Media, 2006. 18 [WPN+19] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
<a id="S0445"></a> Source: p.30 S0445
Superglue: A stickier benchmark for general-purpose language understanding systems, 2019, 1905.00537. 2 [WRH17] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.
<a id="S0446"></a> Source: p.30 S0446
Growing a brain: Fine-tuning by increasing model capacity. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi:10.1109/cvpr.2017.323. 19 [WYL19] Wei Wen, Feng Yan, and Hai Li.
<a id="S0447"></a> Source: p.30 S0447
Autogrow: Automatic layer growing in deep convolutional networks, 2019, 1906.02909. 19 [YDY+19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V.
<a id="S0448"></a> Source: p.30 S0448
Xlnet: Generalized autoregressive pretraining for language understanding, 2019, arXiv:1906.08237. 2 [ZK16] Sergey Zagoruyko and Nikos Komodakis.
<a id="S0449"></a> Source: p.30 S0449
Procedings of the British Machine Vision Conference 2016, 2016. doi:10.5244/c.30.87. 18 [ZKZ+15] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.
<a id="S0450"></a> Source: p.30 S0450
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi:10.1109/iccv.2015.11. 7 [ZLN+19] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E.
<a id="S0451"></a> Source: p.30 S0451
Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model.
<a id="S0452"></a> Source: p.30 S0452
URL http://arxiv.org/abs/1907.04164. 12, 18 30