Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B. - 中英文对照
translated: 2026-07-16
title: "Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B." aliases: - "Scaling Law" - "arXiv:2001.08361" source: "https://arxiv.org/abs/2001.08361" arxiv: "2001.08361" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning
Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B. - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Scaling Laws for Neural Language Models Jared Kaplan ∗ Sam McCandlish∗ Johns Hopkins University, OpenAI OpenAI jaredk@jhu.edu sam@openai.com Tom Henighan Tom B.
中文: 校对:Soup
<a id="S0002"></a> Source: p.1 S0002
Original: Brown Benjamin Chess Rewon Child OpenAI OpenAI OpenAI OpenAI henighan@openai.com tom@openai.com bchess@openai.com rewon@openai.com Scott Gray Alec Radford Jeffrey Wu Dario Amodei OpenAI OpenAI OpenAI OpenAI scott@openai.com alec@openai.com jeffwu@openai.com damodei@openai.com Abstract We study empirical scaling laws for language model performance on the cross-entropy loss.
中文: Brown Benjamin Chess Rewon Child OpenAI OpenAI OpenAI OpenAI henighan@openai.com tom@openai.com bches@openai.com rewon@openai.com Scott Gray Alec Radford Jeffrey Wu Dario Amodei OpenAI OpenAI scott@openai.com Jenwu@openai.com Damodei@openai.com 摘要 我们研究关于语言模型性能的经验性衡量定律 对于交叉性能损失。
<a id="S0003"></a> Source: p.1 S0003
Original: The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude.
中文: 损失尺度作为动力法,具有模型大小,数据集大小,以及用于训练的计算量等,有些趋势跨越了7个多级.
<a id="S0004"></a> Source: p.1 S0004
Original: Other architectural details such as network width or depth have minimal effects within a wide range.
中文: 网络宽度或深度等其它建筑细节在广域范围内影响最小.
<a id="S0005"></a> Source: p.1 S0005
Original: Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size.
中文: 简单的等式规范了过度适应模型/数据集大小的依赖性,以及训练速度对模型大小的依赖性.
<a id="S0006"></a> Source: p.1 S0006
Original: These relationships allow us to determine the optimal allocation of a fixed compute budget.
中文: 这些关系使我们能够确定固定计算预算的最佳分配。
<a id="S0007"></a> Source: p.1 S0007
Original: Larger models are significantly more sampleefficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence. ∗Equal contribution.
中文: 较大模型的采样效率要高得多,因此,最优计算效率高的培训包括就数量相对较少的数据对非常大的模型进行培训,并在汇合前显著停止. * 平等捐款。
<a id="S0008"></a> Source: p.1 S0008
Original: Contributions: Jared Kaplan and Sam McCandlish led the research.
中文: 贡献:Jared Kaplan和Sam McCandlish领导了研究。
<a id="S0009"></a> Source: p.1 S0009
Original: Tom Henighan contributed the LSTM experiments.
中文: 汤姆·赫尼汉贡献了LSTM实验.
<a id="S0010"></a> Source: p.1 S0010
Original: Tom Brown, Rewon Child, and Scott Gray, and Alec Radford developed the optimized Transformer implementation.
中文: Tom Brown, Rewon Child,和Scott Gray,以及Alec Radford开发了"最优化的变形器"执行.
<a id="S0011"></a> Source: p.1 S0011
Original: Jeff Wu, Benjamin Chess, and Alec Radford developed the text datasets.
中文: Jeff Wu, Benjamin Chess,和Alec Radford开发了文本数据集.
<a id="S0012"></a> Source: p.1 S0012
Original: Dario Amodei provided guidance throughout the project. 0202 naJ 32 ]GL.sc[ 1v16380.1002:viXra
中文: 达里奥·阿莫代在整个项目中提供了指导。 0202 naJ 32] GL.sc [1v16380.1002:viXra] (英语).
<a id="S0013"></a> Source: p.2 S0013
Original: Contents 1 Introduction 2 2 Background and Methods 6 3 Empirical Results and Basic Power Laws 7 4 Charting the Infinite Data Limit and Overfitting 10 5 Scaling Laws with Model Size and Training Time 12 6 Optimal Allocation of the Compute Budget 14 7 Related Work 18 8 Discussion 18 Appendices 20 A Summary of Power Laws 20 B Empirical Model of Compute-Efficient Frontier 20 C Caveats 22 D Supplemental Figures 23 1 Introduction Language provides a natural domain for the study of artificial intelligence, as the vast majority of reasoning tasks can be efficiently expressed and evaluated in language, and the world’s text provides a wealth of data for unsupervised learning via generative modeling.
中文: 说明 1 导言 2 背景和方法 6 经验结果和基本权力法 7 4 绘制无限数据限制和与10 5 扩展法相适应的模型大小和培训时间 12 6 优化分配计算预算 14 7 相关工作 18 8 讨论 18 附录 20 权力法摘要 20 计算经验模型 20 D Caveats 22 D 补充图 23 1 导言 语言为研究人工智能提供了一个自然领域,因为绝大多数推理任务可以用语言有效表达和评价,世界文本为通过基因模型进行无监督学习提供了丰富的数据。
<a id="S0014"></a> Source: p.2 S0014
Original: Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY+19, LOG+19, RSR+19] approaching human-level performance on many specific tasks [WPN+19], including the composition of coherent multiparagraph prompted text samples [RWC+19].
中文: 深度学习在语言建模方面最近取得了快速进展,最先进的艺术模型[RNSS18,DCLT18,YDY+19,LOG+19,RSR+19]在许多具体任务上接近人类层面的性能[WPN+19],包括连贯多段的构成催生出文字样本[RWC+19].
<a id="S0015"></a> Source: p.2 S0015
Original: One might expect language modeling performance to depend on model architecture, the size of neural models, the computing power used to train them, and the data available for this training process.
中文: 人们可能期望语言建模的性能取决于模型架构,神经模型的大小,用于训练这些模型的计算力,以及这一训练过程可用的数据.
<a id="S0016"></a> Source: p.2 S0016
Original: In this work we will empirically investigate the dependence of language modeling loss on all of these factors, focusing on the Transformer architecture [VSP+17, LSP+18].
中文: 在这项工作中,我们将从经验上调查语言模型丢失对所有这些因素的依赖性,重点是变形器架构[VSP+17,LSP+18].
<a id="S0017"></a> Source: p.2 S0017
Original: The high ceiling and low floor for performance on language tasks allows us to study trends over more than seven orders of magnitude in scale.
中文: 语文工作执行的上限高而下限低,使我们能够研究规模超过七个级的趋势。
<a id="S0018"></a> Source: p.2 S0018
Original: Throughout we will observe precise power-law scalings for performance as a function of training time, context length, dataset size, model size, and compute budget. 1.1 Summary Our key findings for Transformer language models are are as follows: 2Here we display predicted compute when using a sufficiently small batch size.
中文: 在整个过程中,我们将观察到精确的功率法缩放,视培训时间、背景长度、数据集大小、模型大小和预算计算情况而定。 1.1 总结 我们对变形器语言模型的关键发现如下: 2 在这里,我们使用足够小的批量大小来显示预测计算.
<a id="S0019"></a> Source: p.2 S0019
Original: See Figure 13 for comparison to the purely empirical data. 2
中文: 与纯实证数据的比较见图13。 2个
<a id="S0020"></a> Source: p.3 S0020
Original: Compute Dataset Size Parameters PF-days, non-embedding tokens non-embedding ssoL tseT Figure 1 Language modeling performance improves smoothly as we increase the model size, datasetset size, and amount of compute2 used for training.
中文: 计算 Dataset 大小参数 PF-days,非嵌入符号 ssoL tseT 图1 语言模型的性能随着我们增加模型大小,数据集大小,以及用于训练的计算量而得到平稳改善.
<a id="S0021"></a> Source: p.3 S0021
Original: For optimal performance all three factors must be scaled up in tandem.
中文: 为了取得最佳业绩,所有三个因素必须同步扩大。
<a id="S0022"></a> Source: p.3 S0022
Original: Empirical performance has a power-law relationship with each individual factor when not bottlenecked by the other two.
中文: 经验性表现在不受其他两个因素的束缚时,与每个个体因素都有权力法关系.
<a id="S0023"></a> Source: p.3 S0023
Original: Performance depends strongly on scale, weakly on model shape: Model performance depends most strongly on scale, which consists of three factors: the number of model parameters N (excluding embeddings), the size of the dataset D, and the amount of compute C used for training.
中文: 性能主要依赖于尺度,弱于模型外形:模型性能主要依赖于尺度,由三个因素组成:模型参数N的数量(不包括嵌入),数据集D的大小,以及用于训练的计算C的数量.
<a id="S0024"></a> Source: p.3 S0024
Original: Within reasonable limits, performance depends very weakly on other architectural hyperparameters such as depth vs. width. (Section 3) Smooth power laws: Performance has a power-law relationship with each of the three scale factors N, D, C when not bottlenecked by the other two, with trends spanning more than six orders of magnitude (see Figure 1).
中文: 在合理的限度内,性能非常弱地依赖于其他建筑超参数,如深度与宽度. (第3节)平稳权力法:在不为其他两个因素所束缚的情况下,性能与三个尺度系数N、D、C的每个因素都有权力法关系,其趋势超过六个数量级(见图一)。
<a id="S0025"></a> Source: p.3 S0025
Original: We observe no signs of deviation from these trends on the upper end, though performance must flatten out eventually before reaching zero loss. (Section 3) Universality of overfitting: Performance improves predictably as long as we scale up N and D in tandem, but enters a regime of diminishing returns if either N or D is held fixed while the other increases.
中文: 我们没有看到在上端偏离这些趋势的迹象,不过,在达到零损失之前,业绩必须最终趋于平稳。 (第3节) 过于适应的普遍性:只要我们同时扩大N和D,那么性能可以预测得到改进,但如果N或D被固定,而其他的则增加,则进入一种收益减少的制度。
<a id="S0026"></a> Source: p.3 S0026
Original: The performance penalty depends predictably on the ratio N 0.74/D, meaning that every time we increase the model size 8x, we only need to increase the data by roughly 5x to avoid a penalty. (Section 4) Universality of training: Training curves follow predictable power-laws whose parameters are roughly independent of the model size.
中文: 性能处罚可以预测取决于N 0.74/D的比例,这意味着每当我们增加模型大小为8x时,我们只需要增加大约5x的数据来避免处罚. (第4节)培训的普遍性:培训曲线遵循可预测的权力法,其参数大致独立于模型大小。
<a id="S0027"></a> Source: p.3 S0027
Original: By extrapolating the early part of a training curve, we can roughly predict the loss that would be achieved if we trained for much longer. (Section 5) Transfer improves with test performance: When we evaluate models on text with a different distribution than they were trained on, the results are strongly correlated to those on the training validation set with a roughly constant offset in the loss – in other words, transfer to a different distribution incurs a constant penalty but otherwise improves roughly in line with performance on the training set. (Section 3.2.2) Sample efficiency: Large models are more sample-efficient than small models, reaching the same level of performance with fewer optimization steps (Figure 2) and using fewer data points (Figure 4).
中文: 通过推断训练曲线的早期部分,我们可以大致预测如果我们训练更长时间将会实现的损失. (第5节) 当我们用不同分发方式评价文本的模型时,结果与培训验证套件上的结果密切相关,而且损失被大致固定地抵消了,换句话说,转移到不同分发方式会不断受到处罚,但除此之外,结果与培训套件的表现大致相符。 (第3.2.2节) 样本效率:大型模型比小型模型更具样本效率,以较少的优化步骤(图2)达到相同的性能水平并使用较少的数据点(图4).
<a id="S0028"></a> Source: p.3 S0028
Original: Convergence is inefficient: When working within a fixed compute budget C but without any other restrictions on the model size N or available data D, we attain optimal performance by training very large models and stopping significantly short of convergence (see Figure 3).
中文: 趋同效率低:当在固定计算预算C范围内工作,但对模型大小N或可用数据D没有任何其他限制时,我们通过训练非常大的模型并大大低于趋同(见图3)。
<a id="S0029"></a> Source: p.3 S0029
Original: Maximally compute-efficient training would therefore be far more sample efficient than one might expect based on training small models to convergence, with data requirements growing very slowly as D ∼ C0.27 with training compute. (Section 6) Optimal batch size: The ideal batch size for training these models is roughly a power of the loss only, and continues to be determinable by measuring the gradient noise scale [MKAT18]; it is roughly 1-2 million tokens at convergence for the largest models we can train. (Section 5.1) Taken together, these results show that language modeling performance improves smoothly and predictably as we appropriately scale up model size, data, and compute.
中文: 因此,最高效的计算培训将比根据培训小模式进行整合而预期的样本效率高得多,随着D-C0.27与培训计算,数据要求增长非常缓慢。 (第6节) 最佳批量尺寸: 培训这些模型的理想批量尺寸大致只是损失的动力,通过测量梯度噪声尺度[MKAT18]继续具有确定性;对于我们能够培训的最大模型来说,它大约是120万个符号。 (第5.1节) 综合这些结果表明,随着我们适当扩大模型大小、数据和计算,语言模型的性能可以顺利和预测地改进。
<a id="S0030"></a> Source: p.3 S0030
Original: We expect that larger language models will perform better and be more sample efficient than current models. 3
中文: 我们期望,更大的语言模型将比目前的模型发挥更好,更有效率的样本. 3个
<a id="S0031"></a> Source: p.4 S0031
Original: Larger models require fewer samples The optimal model size grows smoothly to reach the same performance with the loss target and compute budget Line color indicates Test Loss 10 10 number of parameters 103 106 109 8 8 103 Params 6 6 Compute-efficient 109 Params training stops far short of convergence 4 4 107 109 1011 10-9 10-6 10-3 100 Tokens Processed Compute (PF-days) Figure 2 We show a series of language model training runs, with models ranging in size from 103 to 109 parameters (excluding embeddings). M inc in re im as u e m s s n e e r g ia li l g s ib te ly ps <1 1 0 0 x 0 S x e B ri a a t l c S h t e S p iz s e D gr a o t w a r r e e q la u t i i r v e e m ly e s n l t o s w ly >1,000,000x Model Size O in p cr t e im as a e l s m v o e d r e y l q s u iz ic e k ly Figure 3 As more compute becomes available, we can choose how much to allocate towards training larger models, using larger batches, and training for more steps.
中文: 较大的模型需要较少的样本 最佳模型大小平稳地增长,达到与损失目标相同的性能,计算预算线颜色表示测试损失 10 个参数 103 106 109 8 103 Parms 6 6 计算效率 109 Params 训练相去甚远 4 107 109 1011 10-9 10-6 10-3 100 Tokens 加工计算(PF-days) 图2 我们显示一系列语言模型训练,模型大小从103到109个参数(不包括嵌入)不等. m inc in in re as u e m s n e i i g i li l g s ib te ly ps < 1 0 x 0 s x e B ri a a t 1 s s s s s s s d gr a e s p s s s e d gr a r r e q la i i i i i i i i i i i i i i i i i i i i 为 p cr t i i s i i i i i i i i i i i i i i i i i 为 i i i i i i i i 为 i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i i
<a id="S0032"></a> Source: p.4 S0032
Original: We illustrate this for a billion-fold increase in compute.
中文: 我们用计算数的十亿倍来说明这一点。
<a id="S0033"></a> Source: p.4 S0033
Original: For optimally compute-efficient training, most of the increase should go towards increased model size. A relatively small increase in data is needed to avoid reuse.
中文: 为优化计算效率培训,增加的多数应增加模式规模。 为了避免再利用,需要增加较少的数据。
<a id="S0034"></a> Source: p.4 S0034
Original: Of the increase in data, most can be used to increase parallelism through larger batch sizes, with only a very small increase in serial training time required. 1.2 Summary of Scaling Laws The test loss of a Transformer trained to autoregressively model language can be predicted using a power-law when performance is limited by only either the number of non-embedding parameters N , the dataset size D, or the optimally allocated compute budget C (see Figure 1): min 1.
中文: 在增加的数据中,大多数可以通过更大规模的分批量来增加并行性,只需要极小地增加序列训练时间. 1.2 扩展法摘要 当性能被限制在非嵌入参数N的数量,数据集大小D,或优化分配的计算预算C(见图1):分秒1. 时,可以使用电能法来预测训练自动回流模型语言的变形器的测试损失.
<a id="S0035"></a> Source: p.4 S0035
Original: For models with a limited number of parameters, trained to convergence on sufficiently large datasets: L(N ) = (N /N )αN ; α ∼ 0.076, N ∼ 8.8 × 1013 (non-embedding parameters) (1.1) c N c 2.
中文: 对于参数数量有限的模型,在足够大的数据组上进行趋同培训:L(N)=(N/N)αN;α 0.076,N 8.8 × 1013(非嵌入参数)(1.1) c N c c 2.
<a id="S0036"></a> Source: p.4 S0036
Original: For large models trained with a limited dataset with early stopping: L(D) = (D /D)αD ; α ∼ 0.095, D ∼ 5.4 × 1013 (tokens) (1.2) c D c 3.
中文: 对于受过有限数据集培训并提前停止的大型模型:L(D) = (D/D)αD; α 0.095; D 5.4 × 1013 (托) (1.2) c D c 3.
<a id="S0037"></a> Source: p.4 S0037
Original: When training with a limited amount of compute, a sufficiently large dataset, an optimally-sized model, and a sufficiently small batch size (making optimal3 use of compute): L(C ) = (cid:0) Cmin/C (cid:1)αm C in ; αmin ∼ 0.050, Cmin ∼ 3.1 × 108 (PF-days) (1.3) min c min C c 3We also observe an empirical power-law trend with the training compute C (Figure 1) while training at fixed batch size, but it is the trend with C that should be used to make predictions.
中文: 在使用有限的计算量、足够大的数据组、最佳尺寸模型和足够小的分批量尺寸(优化使用计算量):L(C)=(Cid:0)Cmin/C(Cid:1)αC in;αmin 0.050,Cmin 3.1 × 108(PF-日)(1.3分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C分C
<a id="S0038"></a> Source: p.4 S0038
Original: They are related by equation (5.5). min 4
中文: 它们以等式(5.5)相通,分4分。
<a id="S0039"></a> Source: p.5 S0039
Original: 4.5 4.0 3.5 3.0 2.5 107 108 109 1010 Tokens in Dataset ssoL Loss vs Model and Dataset Size 4.4 4.0 Params 708M 3.6 302M 85M 3.2 3M 25M 393.2K 2.8 2.4 104 105 Estimated Smin ssoL Loss vs Model Size and Training Steps 108 107 106 )debme-non( sretemaraP Figure 4 Left: The early-stopped test loss L(N, D) varies predictably with the dataset size D and model size N according to Equation (1.5).
中文: 4.5 4.0 3.5 3.0 2.5 107 108 109 1010 Dataset ssoL Loss vs Model and Dataset Size 4.4 4.0 Params 708M 3.6 302M 85M 3.2 3M 25M 393.2K 2.8 2.4 104 105 估计 Smin ssoL L损失 vs Model SsoL Steps 108 107 106 106 (sretemaraP 图4 左:早期停止测试损失 L(N, D) 与数据集大小 D 和模型大小 N 根据 Equation (1.5) 预测不同.
<a id="S0040"></a> Source: p.5 S0040
Original: Right: After an initial transient period, learning curves for all model sizes N can be fit with Equation (1.6), which is parameterized in terms of S , the number of steps when min training at large batch size (details in Section 5.1).
中文: 右:在初始瞬间后,所有型号的N的学习曲线可以与方程式相适应(1.6),方程式参数为: S.,大批量大小的min训练时的步数(详见第5.1节).
<a id="S0041"></a> Source: p.5 S0041
Original: These relations hold across eight orders of magnitude in C , six orders of magnitude in N , and over two min orders of magnitude in D.
中文: 这些关系涉及C的8个数量级,N的6个数量级,D的2分多分数量级。
<a id="S0042"></a> Source: p.5 S0042
Original: They depend very weakly on model shape and other Transformer hyperparameters (depth, width, number of self-attention heads), with specific numerical values associated with the Webtext2 training set [RWC+19].
中文: 它们非常薄弱地依赖于模型形状和其他变形器超参数(深度,宽度,自留心头数),具体数值与Webtext2训练集[RWC+19]相接.
<a id="S0043"></a> Source: p.5 S0043
Original: The power laws α , α , αmin specify the degree of performance improvement N D C expected as we scale up N , D, or C ; for example, doubling the number of parameters yields a loss that min is smaller by a factor 2−αN = 0.95.
中文: 电力法α、α、αmin具体规定了随着我们扩大N、D或C,预期的性能改善程度;例如,将参数数翻一番,损失的分数会小于系数2-%N=0.95。
<a id="S0044"></a> Source: p.5 S0044
Original: The precise numerical values of N c , C c min, and D c depend on the vocabulary size and tokenization and hence do not have a fundamental meaning.
中文: Nc、Cc min和Dc的精确数值取决于词汇大小和符号化,因此没有根本意义。
<a id="S0045"></a> Source: p.5 S0045
Original: The critical batch size, which determines the speed/efficiency tradeoff for data parallelism ([MKAT18]), also roughly obeys a power law in L: B B (L) = ∗ , B ∼ 2 · 108 tokens, α ∼ 0.21 (1.4) crit L1/αB ∗ B Equation (1.1) and (1.2) together suggest that as we increase the model size, we should increase the dataset size sublinearly according to D ∝ N α α N D ∼ N 0.74.
中文: 关键批量大小决定了数据并行性的速度/效率取舍([MKAT18]),在L:B (L)= ,B − 2 − 108个令牌,α 0.21 (1.4) crit L1/αB B 等方程式(1.1)和1.2 中也大致遵守了权力定律,合起来表明,随着我们增加模型大小,我们应该按照D → N α N D → 0.74 增加数据集分线尺寸。
<a id="S0046"></a> Source: p.5 S0046
Original: In fact, we find that there is a single equation combining (1.1) and (1.2) that governs the simultaneous dependence on N and D and governs the degree of overfitting: (cid:34)(cid:18) (cid:19) αN (cid:35)αD L(N, D) = N c αD + D c (1.5) N D with fits pictured on the left in figure 4.
中文: 事实上,我们发现有一个将1.1和1.2结合起来的单一方程来规范同时对N和D的依赖,并规范了过度适应的程度:(cid:34 (cid:18)(cid:19)αN (cid:35)αD L(N,D) = N c αD + D C (1.5) N D 与图4左侧的相适应相适应的程度.
<a id="S0047"></a> Source: p.5 S0047
Original: We conjecture that this functional form may also parameterize the trained log-likelihood for other generative modeling tasks.
中文: 我们猜想,这种功能形式也可能将训练过的对木相似度参数化,用于其他基因模型制作任务.
<a id="S0048"></a> Source: p.5 S0048
Original: When training a given model for a finite number of parameter update steps S in the infinite data limit, after an initial transient period, the learning curves can be accurately fit by (see the right of figure 4) (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L(N, S) = c + c (1.6) N S (S) min where S ≈ 2.1 × 103 and α ≈ 0.76, and S (S) is the minimum possible number of optimization steps c S min (parameter updates) estimated using Equation (5.4).
中文: 当在无限数据限制中为有限数量的参数更新步骤S培训特定模型时,在初始瞬间后,学习曲线可以(见图4的权利)(cid:18)N(cid:19)αN(Cid:18)S(cid:19)αS L(N,S)=c+c(1.6)N S(S)分,其中S 2.1 × 103和α ^ 0.76,而S(S)是使用 Equation(5.4)估计的最小可能的优化步骤C S min(参数更新).
<a id="S0049"></a> Source: p.5 S0049
Original: When training within a fixed compute budget C, but with no other constraints, Equation (1.6) leads to the prediction that the optimal model size N , optimal batch size B, optimal number of steps S, and dataset size D should grow as N ∝ Cαm C in/αN , B ∝ Cαm C in/αB , S ∝ Cαm C in/αS , D = B · S (1.7) with αmin = 1/ (1/α + 1/α + 1/α ) (1.8) C S B N which closely matches the empirically optimal results N ∝ C0.73, B ∝ C0.24, and S ∝ C0.03.
中文: 如果在固定计算预算C范围内进行训练,但没有其他限制,则方程式(1.6)导致预测最佳模型尺寸N、最佳批量尺寸B、最佳步骤S和数据集尺寸D应随着N-Cαm C in/αN、B-Cαm C in/αB、S-Cαm C in/αS、D-B-S(1.7)与αmin=1 (1/α+1/α+1/α) (1.8) CS BN与经验最佳结果N-C0.73、B-C0.24和S-C0.03相近而增长。
<a id="S0050"></a> Source: p.5 S0050
Original: As the min min min computational budget C increases, it should be spent primarily on larger models, without dramatic increases in training time or dataset size (see Figure 3).
中文: 随着分秒计算预算C的增加,它应主要用于更大的模型,而不大幅度增加培训时间或数据集的规模(见图3)。
<a id="S0051"></a> Source: p.5 S0051
Original: This also implies that as models grow larger, they become increasingly sample efficient.
中文: 这还意味着,随着模型规模的扩大,模型的样本效率日益提高。
<a id="S0052"></a> Source: p.5 S0052
Original: In practice, researchers typically train smaller models for longer than would 5
中文: 在实践中,研究人员通常对较小的模型进行比5年更长的训练。
<a id="S0053"></a> Source: p.6 S0053
Original: be maximally compute-efficient because of hardware constraints.
中文: 由于硬件的限制,应尽量提高计算效率。
<a id="S0054"></a> Source: p.6 S0054
Original: Optimal performance depends on total compute as a power law (see Equation (1.3)).
中文: 最佳性能取决于作为权力法的总计算(见Equation (1.3).
<a id="S0055"></a> Source: p.6 S0055
Original: We provide some basic theoretical motivation for Equation (1.5), an analysis of learning curve fits and their implications for training time, and a breakdown of our results per token.
中文: 我们为方程式学提供了一些基本的理论动机(1.5),对学习曲线及其对培训时间的影响进行了分析,并且逐一细分了我们的成果。
<a id="S0056"></a> Source: p.6 S0056
Original: We also make some brief comparisons to LSTMs and recurrent Transformers [DGV+18]. 1.3 Notation We use the following notation: • L – the cross entropy loss in nats.
中文: 我们还与LSTMs和反复出现的变换器[DGV+18]进行一些简短的比较. 1.3 标注 我们使用下列注解: ^ L – 在nats中交叉的 en损.
<a id="S0057"></a> Source: p.6 S0057
Original: Typically it will be averaged over the tokens in a context, but in some cases we report the loss for specific tokens within the context. • N – the number of model parameters, excluding all vocabulary and positional embeddings • C ≈ 6N BS – an estimate of the total non-embedding training compute, where B is the batch size, and S is the number of training steps (ie parameter updates).
中文: 通常在上下文中它会平均于指使物,但在某些情况下我们报告在上下文中具体指使物的损失。 ^ N – 模型参数的数量,不包括所有词汇和位置嵌入 ^ C → 6N BS – 估计非嵌入式训练总计算,其中B为批量大小,而S为训练步骤的数量(ie参数更新).
<a id="S0058"></a> Source: p.6 S0058
Original: We quote numerical values in PF-days, where one PF-day = 1015 × 24 × 3600 = 8.64 × 1019 floating point operations. • D – the dataset size in tokens • B – the critical batch size [MKAT18], defined and discussed in Section 5.1.
中文: 我们引用PF-day的数值,其中一个PF-day=1015×24×3600=8.64×1019浮动点操作. • D -- -- B -- -- 第5.1节所界定和讨论的关键批量大小[MKAT18]。
<a id="S0059"></a> Source: p.6 S0059
Original: Training at the crit critical batch size provides a roughly optimal compromise between time and compute efficiency. • C – an estimate of the minimum amount of non-embedding compute to reach a given value of min the loss.
中文: 关键批量规模的培训在时间和计算效率之间提供了大致最佳的妥协。 • C -- -- 估计非嵌入计算达到损失分数的某一数值的最低数额。
<a id="S0060"></a> Source: p.6 S0060
Original: This is the training compute that would be used if the model were trained at a batch size much less than the critical batch size. • S – an estimate of the minimal number of training steps needed to reach a given value of the loss. min This is also the number of training steps that would be used if the model were trained at a batch size much greater than the critical batch size. • α X – power-law exponents for the scaling of the loss as L(X) ∝ 1/XαX where X can be any of N, D, C, S, B, Cmin. 2 Background and Methods We train language models on WebText2, an extended version of the WebText [RWC+19] dataset, tokenized using byte-pair encoding [SHB15] with a vocabulary size n = 50257.
中文: 如果该型号的分批培训比关键分批培训大得多,则使用这种计算法。 • S -- -- 估计达到一定损失价值所需的最低限度培训步骤。 分钟 这也是如果该型号的分批培训比分批培训大得多将使用的培训步骤的数量。 • X – 将损失缩放为 L(X) 1/XαX 的动力法说明,其中 X 可以是 N, D, C, S, B, Cmin. 2 背景和方法 我们在 WebText2 上培训语言模型,这是 WebText [RWC+19] 数据集的扩展版本,以字节-pair编码[SHB15]作为标志,词汇大小为n=50257.
<a id="S0061"></a> Source: p.6 S0061
Original: We optimize the autoregresvocab sive log-likelihood (i.e. cross-entropy loss) averaged over a 1024-token context, which is also our principal performance metric.
中文: 我们优化了自动雷格斯沃卡布活性木质相似性(即交叉内质损失),平均在1024个托肯上下文上,这也是我们的主要性能衡量标准.
<a id="S0062"></a> Source: p.6 S0062
Original: We record the loss on the WebText2 test distribution and on a selection of other text distributions.
中文: 我们在WebText2测试分发和其他文本分发的选择上记录损失.
<a id="S0063"></a> Source: p.6 S0063
Original: We primarily train decoder-only [LSP+18, RNSS18] Transformer [VSP+17] models, though we also train LSTM models and Universal Transformers [DGV+18] for comparison. 2.1 Parameter and Compute Scaling of Transformers We parameterize the Transformer architecture using hyperparameters n (number of layers), d (dilayer model mension of the residual stream), d (dimension of the intermediate feed-forward layer), d (dimension of ff attn the attention output), and n (number of attention heads per layer).
中文: 我们主要只培训解码器[LSP+18,RNSS18]变形器[VSP+17]型号,虽然我们也培训了LSTM型号和通用变形器[DGV+18]作比较. 2.1 变形器参数和计算尺寸 我们使用超参数n(层数),d(残流的分层模型外延),d(中间向反馈层的分层),d(注意输出的ff分层)和n(每层注意头数)参数化变形器架构.
<a id="S0064"></a> Source: p.6 S0064
Original: We include n tokens in the input heads ctx context, with n = 1024 except where otherwise noted. ctx We use N to denote the model size, which we define as the number of non-embedding parameters N ≈ 2d n (2d + d ) model layer attn ff = 12n d2 with the standard d = d /4 = d (2.1) layer model attn ff model where we have excluded biases and other sub-leading terms.
中文: 在输入头 ctx 上下文中包含 n 符号,其中 n = 1024,除非另有说明。 页:1 我们使用 N 表示模型大小,我们定义为非嵌入参数 N → 2d n (2d + d) 模型层 attn ff = 12n d2 有标准 d = d/4 = d (2.1) 模型层 attn ff 模型,我们排除了偏差和其他子领导术语.
<a id="S0065"></a> Source: p.6 S0065
Original: Our models also have n d parameters vocab model in an embedding matrix, and use n d parameters for positional embeddings, but we do not include ctx model these when discussing the ‘model size’ N ; we will see that this produces significantly cleaner scaling laws.
中文: 我们的模型在嵌入式矩阵中也有nd参数 vocab模型,并且使用nd参数进行位置嵌入,但我们在讨论"模型大小"N时不包括ctx模型;我们会看到这会产生显著更清洁的缩放定律.
<a id="S0066"></a> Source: p.6 S0066
Original: Evaluating a forward pass of the Transformer involves roughly C ≈ 2N + 2n n d (2.2) forward layer ctx model add-multiply operations, where the factor of two comes from the multiply-accumulate operation used in matrix multiplication. A more detailed per-operation parameter and compute count is included in Table 1. 6
中文: 评估变形器的前传大约涉及C → 2N + 2n n d (2.2) 前传层 ctx 模型加乘操作,其中2的系数出自矩阵乘法中使用的相乘-相乘操作. 表1.6载有更详细的每个操作参数和计算数。
<a id="S0067"></a> Source: p.7 S0067
Original: Operation Parameters FLOPs per Token Embed (n + n ) d 4d vocab ctx model model Attention: QKV n d 3d 2n d 3d layer model attn layer model attn Attention: Mask — 2n n d layer ctx attn Attention: Project n d d 2n d d layer attn model layer attn embd Feedforward n 2d d 2n 2d d layer model ff layer model ff De-embed — 2d n model vocab Total (Non-Embedding) N = 2d n (2d + d ) C = 2N + 2n n d model layer attn ff forward layer ctx attn Table 1 Parameter counts and compute (forward pass) estimates for a Transformer model.
中文: 注意: Token Embed (n + n) d 4d vocab ctx模型 注意: QKV n d 3d 2n d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3d 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3tn 3ttn 注意: mask – 2n n d 2n 3t 3tttx 3ttn 3ttn 3tn 3tn 3tn 3t. 注意: project n d 2n 2n d d 4tn 3t 4tn 3t 3t 4t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t-t
<a id="S0068"></a> Source: p.7 S0068
Original: Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted.
中文: 略去非线性、偏差和层层正常化等次要术语。
<a id="S0069"></a> Source: p.7 S0069
Original: For contexts and models with d > n /12, the context-dependent computational cost per token is a model ctx relatively small fraction of the total compute.
中文: 对于有d > n/12的上下文和模型,根据上下文计算的每令牌成本是ctx模型在总计算中的相对小分.
<a id="S0070"></a> Source: p.7 S0070
Original: Since we primarily study models where d (cid:29) n /12, model ctx we do not include context-dependent terms in our training compute estimate.
中文: 由于我们主要研究d(cid:29 n /12)的模型,模型ctx,我们并不在我们的训练计算估计中包括依赖上下文的术语.
<a id="S0071"></a> Source: p.7 S0071
Original: Accounting for the backwards pass (approximately twice the compute as the forwards pass), we then define the estimated non-embedding compute as C ≈ 6N floating point operators per training token. 2.2 Training Procedures Unless otherwise noted, we train models with the Adam optimizer [KB14] for a fixed 2.5 × 105 steps with a batch size of 512 sequences of 1024 tokens.
中文: 计算后传(大约为前传的两倍计算),然后将估计的非嵌入式计算法定义为C QQ 6N浮点运算符每个训练符. 2.2 训练程序 除非另有说明,否则我们用亚当优化器[KB14]为固定的2.5×105步来训练型号,分批大小为512个序列为1024个令牌.
<a id="S0072"></a> Source: p.7 S0072
Original: Due to memory constraints, our largest models (more than 1B parameters) were trained with Adafactor [SS18].
中文: 由于内存限制,我们最大的模型(超过1B参数)接受了Aductor[SS18]的培训.
<a id="S0073"></a> Source: p.7 S0073
Original: We experimented with a variety of learning rates and schedules, as discussed in Appendix D.6.
中文: 如附录D.6所讨论,我们试验了各种学习率和学习时间表。
<a id="S0074"></a> Source: p.7 S0074
Original: We found that results at convergence were largely independent of learning rate schedule.
中文: 我们发现,趋同的结果基本上独立于学习率时间表。
<a id="S0075"></a> Source: p.7 S0075
Original: Unless otherwise noted, all training runs included in our data used a learning rate schedule with a 3000 step linear warmup followed by a cosine decay to zero. 2.3 Datasets We train our models on an extended version of the WebText dataset described in [RWC+19].
中文: 除非另有说明,我们数据中包含的所有培训都采用了学习速度表,每3000步线性热能,然后将余弦衰变为零。 2.3 数据集 我们在[RWC+19]所描述的WebText数据集的扩展版本上培训我们的模型.
<a id="S0076"></a> Source: p.7 S0076
Original: The original WebText dataset was a web scrape of outbound links from Reddit through December 2017 which received at least 3 karma.
中文: 最初的"WebText"数据集是从Reddit到2017年12月的出行链接的网络刮片,至少收到3个因果.
<a id="S0077"></a> Source: p.7 S0077
Original: In the second version, WebText2, we added outbound Reddit links from the period of January to October 2018, also with a minimum of 3 karma.
中文: 在第二版"WebText2"中,从2018年1月到10月,我们增加了出道的Reddit链接,同样最少3个因果.
<a id="S0078"></a> Source: p.7 S0078
Original: The karma threshold served as a heuristic for whether people found the link interesting or useful.
中文: 因果阈值对于人们是否认为联系有趣或有用,都起到了推力作用.
<a id="S0079"></a> Source: p.7 S0079
Original: The text of the new links was extracted with the Newspaper3k python library.
中文: 新链接的文本被取自"Newspaper3k python"库.
<a id="S0080"></a> Source: p.7 S0080
Original: In total, the dataset consists of 20.3M documents containing 96 GB of text and 1.62 × 1010 words (as defined by wc).
中文: 该数据集总共由包含96GB文本的20.3M文档和1.62×10个单词(由wc定义)组成.
<a id="S0081"></a> Source: p.7 S0081
Original: We then apply the reversible tokenizer described in [RWC+19], which yields 2.29 × 1010 tokens.
中文: 然后我们使用[RWC+19]中描述的可逆活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活性活
<a id="S0082"></a> Source: p.7 S0082
Original: We reserve 6.6 × 108 of these tokens for use as a test set, and we also test on similarlyprepared samples of Books Corpus [ZKZ+15], Common Crawl [Fou], English Wikipedia, and a collection of publicly-available Internet Books. 3 Empirical Results and Basic Power Laws To characterize language model scaling we train a wide variety of models, varying a number of factors including: • Model size (ranging in size from 768 to 1.5 billion non-embedding parameters) • Dataset size (ranging from 22 million to 23 billion tokens) • Shape (including depth, width, attention heads, and feed-forward dimension) • Context length (1024 for most runs, though we also experiment with shorter contexts) • Batch size (219 for most runs, but we also vary it to measure the critical batch size) 7
中文: 我们保留了6.6×108个这些令牌作为测试套件,我们还测试了类似准备的"图书公司"[ZKZ+15],"通俗爬行"[Fou],"英语维基百科","可公开查阅的互联网图书集"等样本. 3 经验成果和基本权力法 • 模型大小(768至15亿个非嵌入参数不等) • 数据集大小(从2200万至230亿个令牌不等) 形状(包括深度、宽度、注意力头和向导维度) • 背景长度(1024个用于大多数跑道,虽然我们也试验了较短的跑道) • 批量大小(219个用于大多数跑道,但我们也改变它以衡量批量的关键大小) 7
<a id="S0083"></a> Source: p.8 S0083
Original: 10% 8% 6% 4% 2% 0% Feed-Fo 5 r 0 w M a r P d a R ra a m ti e o te ( r d s ff / dmodel) Aspect Ratio (dmodel / nlayer) Attention He 2 a 5 d M D P im ara e m ns e i t o e n rs (dmodel / nhead) esaercnI ssoL A wide range of architectures achieve similar performance 22% additional compute compensates for 1% loss increase Figure 5 Performance depends very mildly on model shape when the total number of non-embedding parameters N is held fixed.
中文: 10% 8% 6% 2% 饲料-Fo 5 r 0 w M a r P d a R ra a m ti e te (r s ff / dmodel) 外观比 (模型/ nlayer) 注意 He 2 a 5 d M im ara e ns i t o e rs (模型/ nhead) esaercnI ssoL 多种建筑实现类似性能 22%的额外计算补偿1%的损失增加 5 性能在持有非嵌入参数 N 的总数时非常温和地取决于模型形状.
<a id="S0084"></a> Source: p.8 S0084
Original: The loss varies only a few percent over a wide range of shapes.
中文: 损失在各种形状上只有几分不同。
<a id="S0085"></a> Source: p.8 S0085
Original: Small differences in parameter counts are compensated for by using the fit to L(N ) as a baseline.
中文: 参数计数的微小差异通过将适配到L(N)作为基线得到补偿.
<a id="S0086"></a> Source: p.8 S0086
Original: Aspect ratio in particular can vary by a factor of 40 while only slightly impacting performance; an (n , d ) = (6, 4288) reaches a layer model loss within 3% of the (48, 1600) model used in [RWC+19]. 7 6 5 4 3 2 106 107 108 109 Parameters (with embedding) ssoL tseT 7 6 5 0 Layer 4 1 Layer 2 Layers 3 Layers 3 6 Layers > 6 Layers 2 103 104 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 1 Layer 2 Layers 3 Layers 6 Layers > 6 Layers Figure 6 Left: When we include embedding parameters, performance appears to depend strongly on the number of layers in addition to the number of parameters.
中文: 光谱比可特别以40为系数而变化,而只略微影响性能; (n,d) = (6,4288) 在[RWC+19]中使用的(48,1600)模型的3%范围内达到地层模型损失. 7 6 5 4 3 3 106 107 109参数(有嵌入式) ssoL tseT 7 6 5 5 第4层 1 第2层 3 第3层 6层 > 6层 103 104 105 107 108 109 参数 (非嵌入式) sso L TseT 1 第2层 3 3 6层 > 6 层 图6 左侧:当我们列入嵌入式参数时,性能似乎除了参数的数量之外,还在很大程度上取决于地层的数量.
<a id="S0087"></a> Source: p.8 S0087
Original: Right: When we exclude embedding parameters, the performance of models with different depths converge to a single trend.
中文: 对:当我们排除嵌入参数时,不同深度的模型的性能会趋同到一个单一的趋势.
<a id="S0088"></a> Source: p.8 S0088
Original: Only models with fewer than 2 layers or with extreme depth-to-width ratios deviate significantly from the trend.
中文: 只有分层不足2个或相距极深的模型与趋势相去甚远.
<a id="S0089"></a> Source: p.8 S0089
Original: In this section we will display data along with empirically-motivated fits, deferring theoretical analysis to later sections. 3.1 Approximate Transformer Shape and Hyperparameter Independence Transformer performance depends very weakly on the shape parameters n , n , and d when we hold layer heads ff the total non-embedding parameter count N fixed.
中文: 在本节中,我们将显示数据以及以经验为动力的适合情况,将理论分析推迟到后几节。 3.1 近似变形器外形和超参数独立变形器的性能非常弱地取决于形状参数n,n,和d,当我们持有图层头来对应全部非嵌入参数计数N固定时.
<a id="S0090"></a> Source: p.8 S0090
Original: To establish these results we trained models with fixed size while varying a single hyperparameter.
中文: 为了确定这些结果,我们培训了固定尺寸的模型,同时改变一个超参数。
<a id="S0091"></a> Source: p.8 S0091
Original: When varying n , heads layer we simultaneously varied d while keeping N ≈ 12n d2 fixed.
中文: 当改变n时,我们同时改变d,同时保持N Q 12n d2的固定.
<a id="S0092"></a> Source: p.8 S0092
Original: Similarly, to vary d at fixed model layer model ff model size we also simultaneously varied the d parameter, as required by the parameter counts in Table model 1.
中文: 同样,为了在固定模型层模型 ff 模型大小时改变d,我们还同时按照表模型1参数计数的要求,改变d参数.
<a id="S0093"></a> Source: p.8 S0093
Original: Independence of n would follow if deeper Transformers effectively behave as ensembles of shallower layers models, as has been suggested for ResNets [VWB16].
中文: 如ResNets[VWB16]建议的那样,如果更深的变形器能有效地作为更浅层模型的集合体发挥作用,n的独立就会随之而来。
<a id="S0094"></a> Source: p.8 S0094
Original: The results are shown in Figure 5. 3.2 Performance with Non-Embedding Parameter Count N In Figure 6 we display the performance of a wide variety of models, ranging from small models with shape (n , d ) = (2, 128) through billion-parameter models, ranging in shape from (6, 4288) through layer model (207, 768).
中文: 结果见图5。 3.2 与非嵌入式参数计数 N 的性能 在图6中,我们显示各种模型的性能,从形状(n,d)=(2,128)的小模型到形状从6,428到地层模型(207,768)等十亿参数模型.
<a id="S0095"></a> Source: p.8 S0095
Original: Here we have trained to near convergence on the full WebText2 dataset and observe no overfitting (except possibly for the very largest models).
中文: 在这里,我们训练了在完整的WebText2数据集上接近同位素,并且观测到没有过于匹配(除了可能是最大的模型).
<a id="S0096"></a> Source: p.8 S0096
Original: As shown in Figure 1, we find a steady trend with non-embedding parameter count N , which can be fit to the first term of Equation (1.5), so that (cid:18) N (cid:19)αN L(N ) ≈ c (3.1) N 8
中文: 如图1所显示,我们发现一个稳定的趋势,非嵌入参数计数N ,该数值可与方程式(1.5)的第一个术语相适应,因此(cid:18)N(cid:19)αN L(N) QQ c(3.1)N 8
<a id="S0097"></a> Source: p.9 S0097
Original: Transformers asymptotically outperform LSTMs LSTM plateaus after <100 tokens due to improved use of long contexts Transformer improves through the whole context Test Loss 5.4 Per-token Test Loss 6 4.8 4.2 LSTMs 4 Parameters: 3.6 400K 1 Layer 5 400K 2 Layers 2M 3.0 Transformers 4 Layers 3M 3 200M 2.4 300M 2 105 106 107 108 109 101 102 103 Parameters (non-embedding) Token Index in Context Figure 7 To observe these trends it is crucial to study performance as a function of N ; if we instead use the total parameter count (including the embedding parameters) the trend is somewhat obscured (see Figure 6).
中文: 由于改进了对长环境的使用,LSTMs LSTM 高地在小于100个后,通过整个上下文改进了变形器。
<a id="S0098"></a> Source: p.9 S0098
Original: This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG+19].
中文: 这表明嵌入式矩阵可以变小而不影响性能,正如最近的工作[LCG+19]所见.
<a id="S0099"></a> Source: p.9 S0099
Original: Although these models have been trained on the WebText2 dataset, their test loss on a variety of other datasets is also a power-law in N with nearly identical power, as shown in Figure 8. 3.2.1 Comparing to LSTMs and Universal Transformers In Figure 7 we compare LSTM and Transformer performance as a function of non-embedding parameter count N .
中文: 虽然这些模型在WebText2数据集上接受了培训,但是它们在各种其他数据集上的测试丢失也是N的功率几乎相同的电法,如图8所示. 3.2.1 与 LSTMs 和通用变形器的比较 在图7中,我们比较了LSTM和变形器的性能,作为非嵌入参数计数N的函数.
<a id="S0100"></a> Source: p.9 S0100
Original: The LSTMs were trained with the same dataset and context length.
中文: LSTMs接受了相同的数据集和上下文长度的培训.
<a id="S0101"></a> Source: p.9 S0101
Original: We see from these figures that the LSTMs perform as well as Transformers for tokens appearing early in the context, but cannot match the Transformer performance for later tokens.
中文: 我们从这些数字中看到,LSTMs 以及变形器在上下文中早期出现的指使,但不能与后期指使的变形器相匹配.
<a id="S0102"></a> Source: p.9 S0102
Original: We present power-law relationships between performance and context position Appendix D.5, where increasingly large powers for larger models suggest improved ability to quickly recognize patterns.
中文: 我们提出业绩与背景地位之间的权力法关系附录D.5,其中对较大模型的越来越大的权力表明,迅速识别模式的能力得到了提高。
<a id="S0103"></a> Source: p.9 S0103
Original: We also compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17 in the appendix.
中文: 我们还将标准变形器的性能与附录中图17中反复出现的变形器[DGV+18]进行比较.
<a id="S0104"></a> Source: p.9 S0104
Original: These models re-use parameters, and so perform slightly better as a function of N , at the cost of additional compute per-parameter. 3.2.2 Generalization Among Data Distributions We have also tested our models on a set of additional text data distributions.
中文: 这些模型再用参数,因此作为N的函数性能稍好一些,成本为每参数的额外计算. 3.2.2 数据分配的一般化 我们还测试了一套额外的文本数据分布模型。
<a id="S0105"></a> Source: p.9 S0105
Original: The test loss on these datasets as a function of model size is shown in Figure 8; in all cases the models were trained only on the WebText2 dataset.
中文: 这些数据集作为模型大小函数的测试损失如图8所示;在所有情况下,模型都是在WebText2数据集上训练的.
<a id="S0106"></a> Source: p.9 S0106
Original: We see that the loss on these other data distributions improves smoothly with model size, in direct parallel with the improvement on WebText2.
中文: 我们看到这些其它数据分布的损失随着模型大小的改善而平稳地改善,与WebText2的改进直接平行.
<a id="S0107"></a> Source: p.9 S0107
Original: We find that generalization depends almost exclusively on the in-distribution validation loss, and does not depend on the duration of training or proximity to convergence.
中文: 我们认为,一般化几乎完全取决于分配中的验证损失,并不取决于培训的时间长短或接近趋同的程度。
<a id="S0108"></a> Source: p.9 S0108
Original: We also observe no dependence on model depth (see Appendix D.8). 3.3 Performance with Dataset Size and Compute We display empirical trends for the test loss as a function of dataset size D (in tokens) and training compute C in Figure 1.
中文: 我们还观察到没有依赖模型深度(见附录D.8)。 3.3 数据集大小和计算性能 我们在图1中显示测试损失的经验趋势,作为数据集大小D(以符号表示)和训练计算C的函数。
<a id="S0109"></a> Source: p.9 S0109
Original: For the trend with D we trained a model with (n , n ) = (36, 1280) on fixed subsets of the WebText2 layer embd dataset.
中文: 关于D的趋势,我们用(n,n)=(36,1280)的模型在WebText2层嵌入数据集的固定子集上进行了培训。
<a id="S0110"></a> Source: p.9 S0110
Original: We stopped training once the test loss ceased to decrease.
中文: 一旦测试损失不再减少,我们就停止了培训。
<a id="S0111"></a> Source: p.9 S0111
Original: We see that the resulting test losses can be fit with simple power-law (cid:18) D (cid:19)αD L(D) ≈ c (3.2) D in the dataset size.
中文: 我们看到,由此造成的测试损失可以在数据集大小中与简单的功率法相匹配(cid:18) D(cid:19)αD L(D) QQ C(3.2) D.
<a id="S0112"></a> Source: p.9 S0112
Original: The total amount of non-embedding compute used during training can be estimated as C = 6N BS, where B is the batch size, S is the number of parameter updates, and the factor of 6 accounts for the forward and backward passes.
中文: 训练期间所使用非嵌入式计算的总数量可估计为C=6N BS,其中B为批量大小,S为参数更新数,而6因子为前向和后向通过数.
<a id="S0113"></a> Source: p.9 S0113
Original: Thus for a given value of C we can scan over all models with various N to find the model 9
中文: 因此,对于给定的 C 值,我们可以扫描所有模型 与不同的N 找到模型 9
<a id="S0114"></a> Source: p.10 S0114
Original: 7 6 5 4 3 104 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 5.0 WebText2 (Test) Internet Books 4.5 Books Wikipedia 4.0 Common Crawl 3.5 3.0 2.5 5.0 4.5 4.0 3.5 3.0 2.5 Test Loss on Training Distribution noitubirtsiD rehtO no ssoL Books during training Wikipedia during training Books at convergence Wikipedia at convergence Figure 8 Left: Generalization performance to other data distributions improves smoothly with model size, with only a small and very slowly growing offset from the WebText2 training distribution.
中文: 7 6 5 4 3 104 105 106 107 108 109 参数(非嵌入) ssoL tseT 5.0 WebText2(Test) 互联网图书馆的存檔,存档日期2014-10-0. 维基文库中相关的原始文献: 通俗爬行 3.5 3.0 2.5 5.0 4.0 3.5 3.0 2.5 在培训维基百科期间,在培训维基百科期间,在培训过程中,在培训过程中,在培训过程中,在培训期间,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训过程中,在培训时,在培训过程中,在培训过程中,在培训时,在培训时,在培训时,在培训时,在培训时,在培训时,在培训时,在培训中,在培训时,在培训中,在培训书中,在培训时,在培训时,在培训时,在培训时,在测试中,在培训中,在培训时,在测试中,在测试中,在培训中,在培训中,在培训时
<a id="S0115"></a> Source: p.10 S0115
Original: Right: Generalization performance depends only on training distribution performance, and not on the phase of training.
中文: 右:通化性能只取决于培训分配性能,而不取决于培训阶段.
<a id="S0116"></a> Source: p.10 S0116
Original: We compare generalization of converged models (points) to that of a single large model (dashed curves) as it trains. with the best performance on step S = C .
中文: 我们比较了趋同模型(点)的概括性,与它训练时的单一大模型(凹陷曲线)的概括性. 成绩最佳的S = C。
<a id="S0117"></a> Source: p.10 S0117
Original: Note that in these results the batch size B remains fixed for 6BS all models, which means that these empirical results are not truly optimal.
中文: 请注意,在这些结果中,批量尺寸B仍然固定在6BS的所有模型中,这意味着这些经验性结果并非真正最佳.
<a id="S0118"></a> Source: p.10 S0118
Original: We will account for this in later sections using an adjusted C to produce cleaner trends. min The result appears as the heavy black line on the left-hand plot in Figure 1.
中文: 我们将在后面的章节中说明这一点,使用经过调整的C来产生更清洁的趋势。 分钟 结果以图1中左手地块上重黑线来显示.
<a id="S0119"></a> Source: p.10 S0119
Original: It can be fit with (cid:18) C (cid:19)αC L(C) ≈ c (3.3) C The figure also includes images of individual learning curves to clarify when individual models are optimal.
中文: 它可以和(cid:18)C(cid:19)αC L(C)≈c(3.3)C相适应. 该图还包含个人学习曲线的图像,以澄清单个模型何时是最佳.
<a id="S0120"></a> Source: p.10 S0120
Original: We will study the optimal allocation of compute more closely later on.
中文: 我们稍后将更仔细地研究计算数的最佳分配。
<a id="S0121"></a> Source: p.10 S0121
Original: The data strongly suggests that sample efficiency improves with model size, and we also illustrate this directly in Figure 19 in the appendix. 4 Charting the Infinite Data Limit and Overfitting In Section 3 we found a number of basic scaling laws for language modeling performance.
中文: 数据强烈地表明,样本效率随着模型大小的提高而提高,我们在附录图19中也直接说明了这一点. 4 绘制无限数据限制和过度匹配图 在第三节中,我们发现了一些关于语言建模性能的基本缩放定律.
<a id="S0122"></a> Source: p.10 S0122
Original: Here we will study the performance of a model of size N trained on a dataset with D tokens while varying N and D simultaneously.
中文: 在此,我们将研究在使用D符号的数据集上培训的大小为N的模型的性能,同时进行不同的N和D.
<a id="S0123"></a> Source: p.10 S0123
Original: We will empirically demonstrate that the optimally trained test loss accords with the scaling law of Equation (1.5).
中文: 我们将以经验证明,经过最佳培训的测试损失符合方程式定律(1.5)。
<a id="S0124"></a> Source: p.10 S0124
Original: This provides guidance on how much data we would need to train models of increasing size while keeping overfitting under control. 4.1 Proposed L(N, D) Equation We have chosen the parameterization (1.5) (repeated here for convenience): (cid:34)(cid:18) (cid:19) αN (cid:35)αD L(N, D) = N c αD + D c (4.1) N D using three principles: 1.
中文: 这提供了指导,说明我们需要多少数据来训练规模越来越大的模型,同时保持过度调整的控制。 4.1 拟议L(N,D)方程式 我们选择了参数化(1.5)(为方便起见在此重复): (cid:34 (cid:18)(cid:19)αN (cid:35)αD L (N, D) = N c αD + D c (4.1) N D 使用了三种原理: 1.
<a id="S0125"></a> Source: p.10 S0125
Original: Changes in vocabulary size or tokenization are expected to rescale the loss by an overall factor.
中文: 词汇大小或符号化的变化预计会按照总体因素调整损失规模.
<a id="S0126"></a> Source: p.10 S0126
Original: The parameterization of L(N, D) (and all models of the loss) must naturally allow for such a rescaling. 2.
中文: L(N, D)的参数化(以及损失的所有模型)必须自然地允许这种回缩. 2. 联合国
<a id="S0127"></a> Source: p.10 S0127
Original: Fixing D and sending N → ∞, the overall loss should approach L(D).
中文: 修复D并发送N + + ,总体损失应接近L(D).
<a id="S0128"></a> Source: p.10 S0128
Original: Conversely, fixing N and sending D → ∞ the loss must approach L(N ). 3.
中文: 相反,确定N和发送D + 损失必须接近L(N) 3.
<a id="S0129"></a> Source: p.10 S0129
Original: L(N, D) should be analytic at D = ∞, so that it has a series expansion in 1/D with integer powers.
中文: L(N,D)在 D = Q 时应该具有分析作用,因此在 1/D 有整数功率的序列扩展.
<a id="S0130"></a> Source: p.10 S0130
Original: Theoretical support for this principle is significantly weaker than for the first two.
中文: 对这一原则的理论支持大大弱于前两项.
<a id="S0131"></a> Source: p.10 S0131
Original: Our choice of L(N, D) satisfies the first requirement because we can rescale N , D with changes in the c c vocabulary.
中文: 我们选择 L(N, D) 满足了第一个要求,因为我们可以随着 c c 词汇的改变而重新调整 N, D 。
<a id="S0132"></a> Source: p.10 S0132
Original: This also implies that the values of N , D have no fundamental meaning. c c 10
中文: 这也意味着N,D的价值观没有根本意义. 联合国
<a id="S0133"></a> Source: p.11 S0133
Original: 4.5 4.0 3.5 3.0 2.5 106 107 108 109 Params (non-embed) ssoL tseT Data Size Bottleneck 0.5 Data Size 0.4 21M 43M 86M 0.3 172M 344M 0.2 688M 1.4B 22.0B 0.1 0.0 10 4 10 3 10 2 10 1 N N/ D/D 1 ) =D(L/L Overfitting Data Size 21M 43M 86M 172M 344M 688M 1.4B 22.0B Figure 9 The early-stopped test loss L(N, D) depends predictably on the dataset size D and model size N according to Equation (1.5).
中文: 4.5 4.0 3.0 2.5 107 108 109 参数(非嵌入式) ssoL tseT 数据大小 Bottleneck 0.5 数据大小 0.4 21M 43M 86M 0.3 172M 344M 0.2 688M 1.4B 22.0B 0.1 0.010 4 10 3 10 1 N/D/D 1 =D(L/L 数据大小 21M 43M 86M 172M 344M 688M 1.4B 22.0B 图9 早期停止的测试损失L(N,D)根据方程式(1.5)预测取决于数据集大小D和模型大小N.
<a id="S0134"></a> Source: p.11 S0134
Original: Left: For large D, performance is a straight power law in N .
中文: 左:对于大D来说,性能是N中的直权法.
<a id="S0135"></a> Source: p.11 S0135
Original: For a smaller fixed D, performance stops improving as N increases and the model begins to overfit. (The reverse is also true, αN see Figure 4.) Right: The extent of overfitting depends predominantly on the ratio N αD /D, as predicted in equation (4.3).
中文: 对于一个更小的固定D,性能会随着N的增加而停止改善,而模型开始超合. (反之亦然,αN见图4) 右:过度适应的程度主要取决于等式(4.3)中预测的N αD/D的比例.
<a id="S0136"></a> Source: p.11 S0136
Original: Since we stop training early when the test loss ceases to improve and optimize all models in the same way, we expect that larger models should always perform better than smaller models.
中文: 由于在测试损失停止以同样的方式改进和优化所有模型时,我们很早就停止了训练,我们期望更大的模型总是比更小的模型表现更好.
<a id="S0137"></a> Source: p.11 S0137
Original: But with fixed finite D, we also do not expect any model to be capable of approaching the best possible loss (ie the entropy of text).
中文: 但是有了固定的有限度的D,我们也不期望任何模型能够接近尽可能最好的损失(即文字的 en).
<a id="S0138"></a> Source: p.11 S0138
Original: Similarly, a model with fixed size will be capacity-limited.
中文: 同样,固定大小的模型也将是能力有限的。
<a id="S0139"></a> Source: p.11 S0139
Original: These considerations motivate our second principle.
中文: 这些考虑推动了我们的第二项原则。
<a id="S0140"></a> Source: p.11 S0140
Original: Note that knowledge of L(N ) at infinite D and L(D) at infinite N fully determines all the parameters in L(N, D).
中文: 请注意,在无限 D 和 在无限 N 中,对 L(N) 的了解完全决定了 L(N, D) 中的所有参数.
<a id="S0141"></a> Source: p.11 S0141
Original: The third principle is more speculative.
中文: 第三项原则是更投机。
<a id="S0142"></a> Source: p.11 S0142
Original: There is a simple and general reason one might expect overfitting to scale ∝ 1/D at very large D.
中文: 有个简单而普遍的理由,人们可能期望在非常大的D上超标出QQ1/D.
<a id="S0143"></a> Source: p.11 S0143
Original: Overfitting should be related to the variance or the signal-to-noise ratio of the dataset [AS17], and this scales as 1/D.
中文: 过于适应应当与数据集[AS17]的相差或信号与噪声之比有关,而这个比分为1/D.
<a id="S0144"></a> Source: p.11 S0144
Original: This expectation should hold for any smooth loss function, since we expect to be able to expand the loss about the D → ∞ limit.
中文: 这种期望应维持任何平稳的损失功能,因为我们期望能够扩大D-QQ限制的损失。
<a id="S0145"></a> Source: p.11 S0145
Original: However, this argument assumes that 1/D corrections dominate over other sources of variance, such as the finite batch size and other limits on the efficacy of optimization.
中文: 然而,这一论点假设1/D更正支配了其他差异源,例如有限的批量大小和对优化效果的其他限制。
<a id="S0146"></a> Source: p.11 S0146
Original: Without empirical confirmation, we would not be very confident of its applicability.
中文: 没有经验的证实,我们将不十分相信其适用性。
<a id="S0147"></a> Source: p.11 S0147
Original: Our third principle explains the asymmetry between the roles of N and D in Equation (1.5).
中文: 我们的第三项原则解释了方程式中N和D角色的不对称性(1.5).
<a id="S0148"></a> Source: p.11 S0148
Original: Very similar symmetric expressions4 are possible, but they would not have a 1/D expansion with integer powers, and would require the introduction of an additional parameter.
中文: 非常相近的对称表达式4是可能的,但它们不会有一个有整数功率的1/D扩展,需要引入额外的参数.
<a id="S0149"></a> Source: p.11 S0149
Original: In any case, we will see that our equation for L(N, D) fits the data well, which is the most important justification for our L(N, D) ansatz. 4.2 Results We regularize all our models with 10% dropout, and by tracking test loss and stopping once it is no longer decreasing.
中文: 无论如何,我们将看到,我们的L(N,D)等式与数据相符,这是我们L(N,D)Assatz的最重要理由。 4.2 结果 我们所有模式的规范化 10%的辍学, 通过追踪 测试损失,停止 一旦它不再减少。
<a id="S0150"></a> Source: p.11 S0150
Original: The results are displayed in Figure 9, including a fit to the four parameters α , α , N , D in N D c c Equation (1.5): Parameter α α N D N D c c Value 0.076 0.103 6.4 × 1013 1.8 × 1013 Table 2 Fits to L(N, D) We obtain an excellent fit, with the exception of the runs where the dataset has been reduced by a factor of 1024, to about 2 × 107 tokens.
中文: 图9显示了结果,包括符合四个参数α、α、N、D在N D c 等值(1.5):参数α、α、N D、C 等值 0.076 0.103 6.4 × 1013 1.8 × 1013 表2 适合L(N、D)的值 我们得到一个极佳的适值,但数据组被缩小为1024个系数的跑道除外,大约2×107个令牌。
<a id="S0151"></a> Source: p.11 S0151
Original: With such a small dataset, an epoch consists of only 40 parameter updates.
中文: 由于数据集如此小,一个纪元只包含40个参数更新.
<a id="S0152"></a> Source: p.11 S0152
Original: Perhaps such a tiny dataset represents a different regime for language modeling, as overfitting happens very early in training (see Figure 16).
中文: 也许如此微小的数据集代表了不同的语言建模制度,因为训练初期就出现了过于适应的情况(见图16)。
<a id="S0153"></a> Source: p.11 S0153
Original: Also note that the parameters differ very slightly from those obtained in Section 3, as here we are fitting the full L(N, D) rather than just L(N, ∞) or L(∞, D).
中文: 也注意到这些参数与第3节获得的参数差别很小,因为我们在此安装的是完整的L(N,D)而不只是L(N,Q)或L(Q,D)。
<a id="S0154"></a> Source: p.11 S0154
Original: To chart the borderlands of the infinite data limit, we can directly study the extent of overfitting.
中文: 为了绘制无限数据极限的边界地,我们可以直接研究过度适应的程度.
<a id="S0155"></a> Source: p.11 S0155
Original: For all but the largest models, we see no sign of overfitting when training with the full 22B token WebText2 dataset, so we can take it as representative of D = ∞.
中文: 除了最大的模型外,我们没有看到训练时使用完整的22B令牌WebText2数据集时过于匹配的迹象,因此我们可以把它当作D=QQ的代表.
<a id="S0156"></a> Source: p.11 S0156
Original: Thus we can compare finite D to the infinite data limit by 4For example, one might have used L(N, D) = (cid:2)(cid:0) Nc (cid:1)αN + (cid:0) Dc (cid:1)αD (cid:3)β , but this does not have a 1/D expansion. N D 11
中文: 因此,我们可以将有限度的D和无限数据限制相提并论为4。 例如,人们可能使用L(N,D)=(cid:2(cid:0)Nc(cid:1)αN+(cid:0)Dc(cid:1)αD(cid:3)β,但这没有1D扩展. 编号 11
<a id="S0157"></a> Source: p.12 S0157
Original: 106 105 104 103 101 6 × 100 4 × 100 3 × 100 WebText2 Train Loss )snekoT( eziS hctaB lacitirC Critical Batch Size vs.
中文: 106 105 104 103 101 6 × 100 4 × 100 3 × 100 WebText2 列车损失) snekoT (eziS hctaB lacitirC Critic Batch Size vs.
<a id="S0158"></a> Source: p.12 S0158
Original: Performance Empirical Bcrit, N = 3M Empirical Bcrit, N = 85M Bcrit = 2.1 × 108 tokens L 4.8 Noise Scale Measurement Figure 10 The critical batch size B follows a power law in the loss as performance increase, and does crit not depend directly on the model size.
中文: 性能 实证 Bcrit, N = 3M 实证 Bcrit, N = 85M Bcrit = 2.1 × 108个令牌 L 4.8 噪声分级测量 图10 批量批量大小B在性能增加后在损失中遵循了动力定律,并不会直接依赖于模型大小.
<a id="S0159"></a> Source: p.12 S0159
Original: We find that the critical batch size approximately doubles for every 13% decrease in loss. B is measured empirically from the data shown in Figure 18, but it is also roughly crit predicted by the gradient noise scale, as in [MKAT18]. defining L(N, D) δL(N, D) ≡ − 1 (4.2) L(N, ∞) and studying it as a function of N, D.
中文: 我们发现临界批量大小大约是每减少13%的损失的两倍。 B是从图18显示的数据中经验性地衡量的,但也大致被梯度噪声尺度所预测出,如[MKAT18]. 定义 L(N,D) − L(N,D) − − 1 (4.2) L(N,Q) ,并作为 N, D 的函数对其进行研究.
<a id="S0160"></a> Source: p.12 S0160
Original: In fact, we see empirically that δL depends only a specific combination of N and D, as shown in Figure 16.
中文: 事实上,我们实证地看到,如图16所示,“L”只取决于N和D的具体组合。
<a id="S0161"></a> Source: p.12 S0161
Original: This follows from the scaling law of Equation (1.5), which implies (cid:32) (cid:18) (cid:19) αN (cid:33)αD δL ≈ 1 + N αD D c − 1 (4.3) N D c Note that at large D this formula also has a series expansion in powers of 1/D.
中文: 其取自"方程式"(1.5)的缩放定律(cid:32)(cid:18)(cid:19)αN(cid:33)αD →L + 1 + N αD D c → 1 (4.3) N D c 注意,总的来说,这一公式还具有一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一一
<a id="S0162"></a> Source: p.12 S0162
Original: We estimate that the variation in the loss with different random seeds is roughly 0.02, which means that to avoid overfitting when training to within that threshold of convergence we require D (cid:38) (5 × 103) N 0.74 (4.4) With this relation, models smaller than 109 parameters can be trained with minimal overfitting on the 22B token WebText2 dataset, but our largest models will encounter some mild overfitting.
中文: 我们估计,使用不同随机种子的损失变化约为0.02个,这意味着为了避免在训练达到交汇阈值时过度调整,我们需要D(编:38)(5×103)N 0.74 (4.4) 在这种关系下,小于109个参数的模型可以在22B令牌WebText2数据集上进行最小的过度匹配来训练,但我们最大的模型会遇到一些轻微的过度匹配.
<a id="S0163"></a> Source: p.12 S0163
Original: More generally, this relation shows that dataset size may grow sub-linearly in model size while avoiding overfitting.
中文: 更一般地说,这种关系表明,数据集的大小可能在模型大小中子线性地增长,同时避免了过于相合.
<a id="S0164"></a> Source: p.12 S0164
Original: Note however that this does not typically represent maximally compute-efficient training.
中文: 但请注意,这通常并不代表计算效率最高的培训。
<a id="S0165"></a> Source: p.12 S0165
Original: We should also emphasize that we have not optimized regularization (eg the dropout probability) while varying dataset and model size. 5 Scaling Laws with Model Size and Training Time In this section we will demonstrate that a simple scaling law provides a good description for the loss as a function of model size N and training time.
中文: 我们还应强调,我们没有优化正规化(例如辍学概率),而不同的数据集和模型大小。 5 以示范规模和培训时间扩展法律 在本节中,我们将表明,一个简单的缩放法对损失作了很好的描述,视模型大小N和训练时间而定。
<a id="S0166"></a> Source: p.12 S0166
Original: First we will explain how to use the results of [MKAT18] to define a universal training step S , which accounts for the fact that most of our models have not been min trained at an optimal batch size.
中文: 首先,我们将解释如何使用[MKAT18]的结果来定义一个通用的训练步骤S,这说明我们大多数的型号还没有按照最佳的分批量大小进行分级训练。
<a id="S0167"></a> Source: p.12 S0167
Original: Then we will demonstrate that we can fit the model size and training time dependence of the loss using Equation (1.6).
中文: 然后,我们将证明,我们可以利用方程式(1.6)来适应损失的模型大小和训练时间依赖。
<a id="S0168"></a> Source: p.12 S0168
Original: Later we will use these results to predict the optimal allocation of training compute between model size and training time, and then confirm that prediction. 5.1 Adjustment for Training at B (L) crit A simple empirical theory for the batch size dependence of training was developed in [MKAT18] (see also [SLA+18, ZLN+19]).
中文: 后来,我们将利用这些结果来预测在模型大小和培训时间之间计算出的培训的最佳分配,然后确认这一预测。 5.1 [MKAT18] (另见[SLA+18, ZLN+19])为培训的批量规模依赖性制定了一种简单的经验理论。
<a id="S0169"></a> Source: p.12 S0169
Original: It was argued that there is a critical batch size B for training; for B up to B crit crit the batch size can be increased with very minimal degradation in compute-efficiency, whereas for B > B crit increases in B result in diminishing returns.
中文: 有人争辩说,培训有一个关键的批量尺寸B;对于B级至B级的Crit Crit,批量尺寸可以增加,计算效率降低得极少,而对于B级 > B级的Crit,则导致收益减少。
<a id="S0170"></a> Source: p.12 S0170
Original: It was also argued that the gradient noise scale provides a simple 12
中文: 还有人认为,梯度噪声尺度提供了一个简单的12。
<a id="S0171"></a> Source: p.13 S0171
Original: prediction for B , and that neither depends directly on model size except through the value of the loss that crit has been attained.
中文: B的预测,除了通过已实现的克立特损失的价值外,两者都不直接取决于模型大小。
<a id="S0172"></a> Source: p.13 S0172
Original: These results can be used to predict how training time and compute will vary with the batch size.
中文: 这些结果可用于预测培训时间和计算会如何随分批量大小而变化.
<a id="S0173"></a> Source: p.13 S0173
Original: To utilize both training time and compute as effectively as possible, it is best to train with a batch size B ≈ B .
中文: 为了尽可能有效地利用培训时间和计算,最好用批量的B-QB进行培训。
<a id="S0174"></a> Source: p.13 S0174
Original: Training at B (cid:29) B minimizes the number of training steps, while B (cid:28) B minimizes crit crit crit the use of compute.
中文: B级(cid:29)B级培训能将培训步骤的数量最小化,而B级(cid:28)B级培训能将Crit Crit Crit Crit使用计算.
<a id="S0175"></a> Source: p.13 S0175
Original: More specifically, it was demonstrated that for a wide variety of neural network tasks, the number of training steps S and the number of data examples processed E = BS satisfy the simple relation (cid:18) (cid:19) (cid:18) (cid:19) S E − 1 − 1 = 1 (5.1) S E min min when training to any fixed value of the loss L.
中文: 更具体地说,事实证明,对于各种神经网络任务,训练步骤S的数量和处理过的数据实例E = BS的数量满足了简单的关系(cid:18)(cid:19)(cid:18)(cid:19)S E − 1− 1 = 1 (5.1)S mine在训练达到损失的任何固定值时L.
<a id="S0176"></a> Source: p.13 S0176
Original: Here S is the minimum number of steps necessary to reach min L, while E is the minimum number of data examples that must be processed. min We demonstrate the relation (5.1) for Transformers in Figure 18 in the appendix.
中文: 这里S是达到min L所需的最低步骤数,而E是必须处理的数据示例的最小数目. 分钟 我们在附录图18中展示了变形器(5.1)的关系。
<a id="S0177"></a> Source: p.13 S0177
Original: This relation defines the critical batch size E B (L) ≡ min (5.2) crit S min which is a function of the target value of the loss.
中文: 这种关系定义了关键批量大小 E B (L) QQ min (5.2) crit S min,这是损失目标值的一个函数 。
<a id="S0178"></a> Source: p.13 S0178
Original: Training at the critical batch size makes a roughly optimal time/compute tradeoff, requiring 2S training steps and processing E = 2E data examples. min min In Figure 10 we have plotted the critical batch size and gradient noise scale5 as a function of training loss for two different models.
中文: 关键批量规模的培训可以作出大致最佳的时间/计算权衡,需要2S培训步骤并处理E=2E数据实例。 分钟 在图10中,我们绘制了关键的批量尺寸和梯度噪声比例表5,作为两种不同模型培训损失的一个函数。
<a id="S0179"></a> Source: p.13 S0179
Original: We see that B (L) is independent of model size, and only depends on the loss L.
中文: 我们看到B(L)独立于模型大小,只取决于损失L.
<a id="S0180"></a> Source: p.13 S0180
Original: So crit the predictions of [MKAT18] continue to hold for Transformer language models.
中文: 因此Crit对[MKAT18]的预测继续为变形语模型所持有.
<a id="S0181"></a> Source: p.13 S0181
Original: The critical batch size can be fit with a power-law in the loss B B (L) ≈ ∗ (5.3) crit L1/αB where B ≈ 2 × 108 and α ≈ 0.21. ∗ B We have chosen this parameterization for B (L) because as the loss approaches its minimum value L , crit min the gradient noise scale is expected to diverge, and we expect B to track this noise scale.
中文: 关键批量尺寸可与B(L) → (5.3) Crit L1/αB损失中的动力法相适应,其中B → 2 × 108 和 α 0.21。 联合国 我们选择了B(L)的参数化,因为随着损失接近最低值L,梯度噪声尺度预计会相去甚远,我们期望B跟踪这种噪声尺度。
<a id="S0182"></a> Source: p.13 S0182
Original: We do not crit know L , as we see no sign that our models are approaching it, but L > 0 since the entropy of natural min min language is non-zero.
中文: 我们不知道L,因为我们没有看到我们的模型接近它的迹象,但L > 0,因为自然分钟语言的正弦是非零的。
<a id="S0183"></a> Source: p.13 S0183
Original: Since apparently L is much smaller than the values of L we have achieved, we used min a parameterization where B diverges as L → 0. crit We will use B (L) to estimate the relation between the number of training steps S while training at batch crit size B = 219 tokens and the number of training steps while training at B (cid:29) B .
中文: 由于L显然比我们实现的L值要小得多,我们用一个小参数化,在B位相差为L~0. 我们用B(L)来估计S级培训与B级培训(Cid:29 B)级培训之间的关联。
<a id="S0184"></a> Source: p.13 S0184
Original: This is simply crit S S (S) ≡ (minimum steps, at B (cid:29) B ) (5.4) min 1 + B (L)/B crit crit for any given target value L for the loss.
中文: 这是简单的Crit S(S)-(最低步骤,B(Cid:29) B(5.4) min 1 + B(L)/B Crit crit,用于任何特定目标值L的损失。
<a id="S0185"></a> Source: p.13 S0185
Original: This also defines a critical value of the compute needed to train to L with a model of size N if we were to train at B (cid:28) B (L).
中文: 这还定义了如果我们在B(cid:28 B (L))进行训练,需要用一个尺寸为N的模型来进行训练到L的计算的关键值.
<a id="S0186"></a> Source: p.13 S0186
Original: This is crit C C (C) ≡ (minimum compute, at B (cid:28) B ) (5.5) min 1 + B/B (L) crit crit where C = 6N BS estimates the (non-embedding) compute used at batch size B. 5.2 Results for L(N, S ) and Performance with Model Size and Compute min Now we will use S defined in Equation (5.4) to obtain a simple and universal fit for the dependence of the min loss on model size and training time in the infinite data limit.
中文: 这是Crit C(C)-(最小计算,在B (cid:28) B (5.5) min 1 + B/B(L) crit crit Crit,其中C = 6N BS 估计批量大小 B 5.2 的(非嵌入式)计算结果。 L(N, S) 和性能与模型大小和计算分数的计算结果
<a id="S0187"></a> Source: p.13 S0187
Original: We will fit the stable, Adam-optimized training runs using Equation (1.6), repeated here for convenience: (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L(N, S ) = c + c (5.6) min N S min for the loss.
中文: 我们将使用 Equation (1.6) 进行稳定、优化的训练, 为方便起见在此重复进行:(cid:18) N(cid:19)αN(cid:18) S(cid:19)αS L(N,S) = c + c(5.6) min N S min for the loss.
<a id="S0188"></a> Source: p.13 S0188
Original: We include all training steps after the warmup period of the learning rate schedule, and find a fit to the data with the parameters: 5Although the critical batch size roughly matches the gradient noise scale, we are using a direct measurements of B from Figures 18 and 10 for all our later analyses. crit 13
中文: 我们包括了所有训练步骤,在学习速度表的暖和期之后,并找到与参数相匹配的数据:5 虽然临界批量尺寸大致与梯度噪声尺度相匹配,但我们在以后的所有分析中都使用图18和图10中的B直接测量。 第13号
<a id="S0189"></a> Source: p.14 S0189
Original: 8 7 6 5 4 3 2 104 106 108 Parameters (non-embedding) ssoL tseT Performance vs Compute Budget 100 10 1 10 2 10 3 10 4 10 5 ssyad-FP 5.4 4.8 4.2 3.6 3.0 2.4 106 107 108 109 Parameters (non-embedding) ssoL tseT Performance vs Steps 105 104 spetS Figure 11 When we hold either total compute or number of training steps fixed, performance follows L(N, S) from Equation (5.6).
中文: 8 6 5 4 3 2 104 106 108 参数(非嵌入式) ssoL tseT 性能与计算预算 100 10 1 10 2 10 3 10 4 10 5 ssyad-FP 5.4 4.8 4.2 3.6 2.0 2.4 106 107 108 个参数(非嵌入式) ssoL tseT 性能与步骤 105 104 spets 图11 当我们掌握总计算数或固定的训练步骤数时,业绩将跟随L(N,S)从方程式(5.6)。
<a id="S0190"></a> Source: p.14 S0190
Original: Each value of compute budget has an associated optimal model size that maximizes performance.
中文: 计算预算的每个价值都有一个相关的最佳模型规模,可以最大限度地提高绩效。
<a id="S0191"></a> Source: p.14 S0191
Original: Mediocre fits at small S are unsurprising, as the power-law equation for the learning curves breaks down very early in training.
中文: 平庸地适合小 S 并不出人意料,因为学习曲线的动力-法等式在训练初期就崩溃了.
<a id="S0192"></a> Source: p.14 S0192
Original: Parameter α α N S N S c c Value 0.077 0.76 6.5 × 1013 2.1 × 103 Table 3 Fits to L(N, S) With these parameters, we obtain the learning curve fits in Figure 4.
中文: 参数α N S N S c 数值 0.077 0.76 6.5 × 1013 2.1 × 103 表3 适合L(N,S)的参数 有了这些参数,我们得到的学习曲线与图4相匹配.
<a id="S0193"></a> Source: p.14 S0193
Original: Though the fits are imperfect, we believe they are quite compelling given the simplicity of Equation (5.6).
中文: 尽管不符合标准,但我们认为,鉴于方程式的简单性(5.6)。
<a id="S0194"></a> Source: p.14 S0194
Original: The data and fits can be visualized in a different and more interesting way, as shown in Figure 11.
中文: 如图11所示,可以以不同和更有趣的方式对数据进行可视化。
<a id="S0195"></a> Source: p.14 S0195
Original: There we study the test loss as a function of model size while fixing either the total non-embedding compute C used in training, or the number of steps S.
中文: 在那里,我们研究测试损失作为模型大小的一种函数,同时确定训练中使用的总非嵌入计算C或步骤S的数量。
<a id="S0196"></a> Source: p.14 S0196
Original: For the fits we use Equation (5.5) and (5.4) along with the parameters above and Equation (5.6).
中文: 为了符合要求,我们使用方程式(5.5和5.4)以及上面的参数和方程式(5.6)。
<a id="S0197"></a> Source: p.14 S0197
Original: The power-law dependence of the loss on S reflects the interplay of optimizer dynamics and the loss min landscape.
中文: 损失对S的功率法依赖,反映了优化动力学与损失分钟地貌的相互作用.
<a id="S0198"></a> Source: p.14 S0198
Original: Since the fits are best late in training, when the loss may be approximately quadratic, the powerlaw should provide information about the spectrum of the Hessian of the loss.
中文: 由于适合性在训练后期最好,当损失可能大约为四分位数时,动力法应提供损失的黑森人谱的信息.
<a id="S0199"></a> Source: p.14 S0199
Original: Its universality suggests that the Hessian eigenvalue density is roughly independent of model size. 5.3 Lower Bound on Early Stopping Step The results for L(N, S ) can be used to derive a lower-bound (and rough estimate) of the step at which min early stopping should occur when training is data limited.
中文: 它的普遍性表明,黑森等值密度大致独立于模型大小. 5.3 及早停止步骤的下接线 L(N,S)的结果可以用来得出较低的(和粗略的估计)在培训数据有限时,在分秒早停时的分数。
<a id="S0200"></a> Source: p.14 S0200
Original: It is motivated by the idea that finite and infinite D learning curves for a given model will be very similar until we reach S ≈ S .
中文: 它的动机是,一个特定模型的有限度和无限的D学习曲线将非常相似,直到我们到达S-Q-S.
<a id="S0201"></a> Source: p.14 S0201
Original: Thus overfitting should min stop be proportional to the correction from simply ending training at S .
中文: 因此,过度适应应该与仅仅结束在S的训练的纠正相称。
<a id="S0202"></a> Source: p.14 S0202
Original: This will underestimate S , because stop stop in reality the test loss will decrease more slowly when we have a finite D, and therefore we will require more training steps to reach the optimal test loss at finite D.
中文: 这将低估S,因为停止在现实中停止,当我们有D限制时,测试损失会更缓慢地减少,因此我们需要更多的培训步骤,以便在D限制时达到最佳测试损失。
<a id="S0203"></a> Source: p.14 S0203
Original: This line of reasoning leads to the inequality S S (N, D) (cid:38) c (5.7) stop [L(N, D) − L(N, ∞)]1/αS where L(N, ∞) is the converged loss, evaluated with infinite available data.
中文: 这一推理线导致了S(N,D)(编:38)c(5.7)站[L(N,D)-L(N,X)/αS]的不平等,其中L(N,X)是集中损失,以无限可得数据进行评估。
<a id="S0204"></a> Source: p.14 S0204
Original: This inequality and its comparison to the empirical data is displayed in Figure 16 in the appendix.
中文: 附录图16显示了这种不平等及其与实证数据的比较。
<a id="S0205"></a> Source: p.14 S0205
Original: In that figure, the values of S stop and L(N, D) are empirical (though S is adjusted to mimic training at B (cid:29) B ), while L(N, ∞) is stop crit computed from the fit to L(N, D) evaluated at D = ∞. 6 Optimal Allocation of the Compute Budget We displayed the empirical trend of performance as a function of the computation used during training in the top-right of Figure 1.
中文: 在该图中,Sstop和L(N,D)的数值是实证的(虽然S在B(Cid:29-B)中被调整为模仿训练),而L(N,Q)则停止从D============================================================================================================================================================================================================
<a id="S0206"></a> Source: p.14 S0206
Original: However, this result involved training at a fixed batch size B, whereas we know 14
中文: 然而,这一结果涉及固定批号B的培训,而我们知道14个
<a id="S0207"></a> Source: p.15 S0207
Original: Smaller models require more steps to train, while larger models require fewer Models between 0.6x and 2.2x the optimal size can be trained with a 20% larger compute budget Our framework does not capture early training dynamics Figure 12 Left: Given a fixed compute budget, a particular model size is optimal, though somewhat larger or smaller models can be trained with minimal additional compute.
中文: 更小的模型需要更多的步骤来训练,而更大的模型则需要更少的0.6x到2.2x之间的模型 最佳尺寸可以用20%的计算预算来训练. Our 框架不能捕捉早期的训练动态 图12 左:考虑到固定的计算预算,特定的模型大小是最佳的,尽管有些大的或较小的模型可以用最小的附加计算来训练.
<a id="S0208"></a> Source: p.15 S0208
Original: Right: Models larger than the computeefficient size require fewer steps to train, allowing for potentially faster training if sufficient additional parallelism is possible.
中文: 对:比计算大小更大的模型需要较少的阶梯来训练,如果可能有足够的额外平行性,则允许潜在的更快训练.
<a id="S0209"></a> Source: p.15 S0209
Original: Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve, after initial transient effects. 7 6 5 4 3 2 10 8 10 6 10 4 10 2 100 Compute (PF-days), non-embedding ssoL tseT L = (C min/2.3 108) 0.050 L = (C/2.0 107) 0.057 Figure 13 When adjusting performance to simulate training far below the critical batch size, we find a somewhat altered power law for L(C ) when compared with the fully empirical results.
中文: 请注意,这种等式不应被信任于非常大的模型,因为它只有在学习曲线的动力-法区,在初始瞬态效应后才有效. 7 6 5 4 3 2 10 8 10 6 10 10 10 2 100 计算(PF-days),非嵌入式solo tseT L = (C min/2.3 108) 0.050 L = (C/2.0 107) 0.057 图13 在调整性能以模拟远低于批量规模的训练时,我们发现L(C)的功率定律与完全实证的结果相比有些改变.
<a id="S0210"></a> Source: p.15 S0210
Original: The conspicuous min lump at 10−5 PF-days marks the transition from 1-layer to 2-layer networks; we exclude 1-layer networks in the power-law fits.
中文: 10-5个PF日的分明时间是从1层网络向2层网络的过渡;我们在电力法中排除了1层网络。
<a id="S0211"></a> Source: p.15 S0211
Original: It is the L(C ) trend that we expect to provide a reliable extrapolation for larger min compute. that in fact we could train more efficiently6 by training at the batch size B discussed in Section 5.1. crit Large and small values of the loss could have been achieved with fewer samples or fewer steps, respectively, and correcting for this inefficiency by standardizing to the critical batch size results in cleaner and more predictable trends.
中文: 正是L(C)趋势,我们期望为更大的分数计算提供可靠的推算。 事实上,我们可以通过5.1节所讨论批量规模B的培训,更有效地培训6。 通过分别减少样品或减少步骤,可以实现损失的 " 临界 " 和 " 小 " 值,并通过使批量规模标准化,实现更清洁和更可预测的趋势来纠正这种低效率。
<a id="S0212"></a> Source: p.15 S0212
Original: In this section we will adjust for this oversight.
中文: 在本节中,我们将调整这一监督。
<a id="S0213"></a> Source: p.15 S0213
Original: More importantly, we will use the results of Section 5 to determine the optimal allocation of compute between model size N and the quantity of data processed during training, namely 2B S .
中文: 更重要的是,我们将利用第5节的结果,确定模型N与培训期间处理的数据数量,即2B S之间计算的最佳分配。
<a id="S0214"></a> Source: p.15 S0214
Original: We will determine this allocation both empirically and theoretically, by crit min using the equation for L(N, S ), and we will demonstrate that these methods agree. min 6.1 Optimal Performance and Allocations Let us first study the loss as a function of the optimally allocated compute from Equation (5.5).
中文: 我们将利用L(N,S)的等式,在经验上和理论上确定这一分配,我们将证明这些方法是一致的。 min 6.1 最佳性能和分配让我们首先研究损失作为从方程式计算的最佳分配函数(5.5)。
<a id="S0215"></a> Source: p.15 S0215
Original: The result is plotted in Figure 13, along with a power-law fit.
中文: 其结果如图13所示,并附有适合的权力法。
<a id="S0216"></a> Source: p.15 S0216
Original: We see that as compared to the compute plot of Figure 1, the new fit with C is somewhat improved. min Given L(C ), it is natural to ask for the optimal model size N (C ) that provides the minimal loss with a min min given quantity of training compute.
中文: 我们看到,与图1的计算图相比,与C相适应的新版图有所改进。 min Given L(C),自然要求最佳型号为N(C),该型号为最小损失提供分毫的训练计算。
<a id="S0217"></a> Source: p.15 S0217
Original: The optimal model size is shown in Figure 14.
中文: 最佳模型大小见图14。
<a id="S0218"></a> Source: p.15 S0218
Original: We observe that N (C ) min 6One might ask why we did not simply train at B in the first place.
中文: 我们注意到,N(C)mine 6 One可能会问,为什么我们一开始不是简单地在B训练。
<a id="S0219"></a> Source: p.15 S0219
Original: The reason is that it depends not only on the crit model but also on the target value of the loss we wish to achieve, and so is a moving target. 15
中文: 原因是它不仅取决于克立特模型,而且取决于我们希望实现的损失的目标值,一个可移动的目标也一样. 15个
<a id="S0220"></a> Source: p.16 S0220
Original: 107 105 103 10 7 10 5 10 3 10 1 Compute (PF-days), non-embedding )gniddebme-non( sretemaraP N = (1.3 109) C0.73 min N = (1.6 109) C0.88 15000 10000 5000 0 10 7 10 5 10 3 10 1 Compute (PF-days), excluding embeddings spetS S (adjusted) min S min = (5.4 103) C m 0. i 0 n 3 S (fixed-batch) Figure 14 Left: Each value of the compute budget C has an associated optimal model size N .
中文: 107 105 103 10 7 10 10 10 10 10 1 计算 (PF-days),非嵌入式) gniddebme-non(sretemaraP N = (1.3 109) C0.73 min N = (1.6 109) C0.88 15000 10000 5000 0 10 10 10 10 10 10 1 计算 (PF-days),不包括嵌入式 spets S (调整后) Min S = (5.4 103) C m. 0 i 0 n 3 S (固定-batch) 图14 左:计算 C的每个值都有相关的最佳模型尺寸 N.
<a id="S0221"></a> Source: p.16 S0221
Original: Optimal min model size grows very rapidly with C , increasing by 5x for each 10x increase in compute.
中文: 最佳分数模型大小与 C 的生长非常快,计算每增加10个分数就会增加5x.
<a id="S0222"></a> Source: p.16 S0222
Original: The number min of data examples processed makes up the remainder of the increase, growing relatively modestly by only 2x.
中文: 所处理的数据实例数量占所增加的其余部分,仅略为增加了2x。
<a id="S0223"></a> Source: p.16 S0223
Original: Right: The batch-adjusted number of optimization steps also grows very slowly, if at all, meaning that most of the growth in data examples processed can be used for increased batch sizes. can be fit very well with a power-law N (C ) ∝ (C )0.73. (6.1) min min In Figure 12, we show the effect of training models of sub-optimal sizes (see Appendix B.4).
中文: 对: 批量调整后优化步骤的数量也增长非常缓慢,如果有的话,这意味着所处理的数据示例的大部分增长可以被用在增加批量尺寸上. 电力法N(C)-(C)-(C)-(0.73)-(6.1)分(6.1)分(分)/分(分)/分(分)/分(分)/分(分)/分(分)/分(分)/分(分)/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/分/
<a id="S0224"></a> Source: p.16 S0224
Original: By definition C ≡ 6N B S, and so we can use N (C ) to extract further results.
中文: 顾名思义,C-Q-6N B S,因此我们可以用N(C)来取出进一步的结果.
<a id="S0225"></a> Source: p.16 S0225
Original: In particular, since min crit min prior fits show B ∝ L−4.8 and L ∝ C−0.05, we can conclude that B ∝ C0.24.
中文: 特别是,由于Mind Crit min 事先适合B + L 4.8 和 L + C 0.05,我们可以得出B + C 0.24的结论。
<a id="S0226"></a> Source: p.16 S0226
Original: This leads us to conclude min crit min that the optimal number of steps will only grow very slowly with compute, as S ∝ (C )0.03, (6.2) min min matching the empirical results in Figure 14.
中文: 这使得我们得出一个结论,即最佳步骤的数量将随着计算而缓慢地增长,如S-Q(C)0.03,(6.2)分与图14中的经验结果相匹配。
<a id="S0227"></a> Source: p.16 S0227
Original: In fact the measured exponent is sufficiently small that our results may even be consistent with an exponent of zero.
中文: 事实上,所测量的指数足够小,我们的结果甚至可能与0的指数一致。
<a id="S0228"></a> Source: p.16 S0228
Original: Thus we conclude that as we scale up language modeling with an optimal allocation of computation, we should predominantly increase the model size N , while simultaneously scaling up the batch size via B ∝ B with negligible increase in the number of serial steps.
中文: 因此,我们的结论是,随着我们通过优化的计算分配来扩大语言模型,我们应当主要增加模型的大小N,同时通过B-Q-B扩大批量大小,序列步骤的数目可以忽略不计地增加。
<a id="S0229"></a> Source: p.16 S0229
Original: Since compute-efficient training uses relatively crit few optimization steps, additional work on speeding up early training dynamics may be warranted. 6.2 Predictions from L(N, S ) min The results for L(C ) and the allocations can be predicted from the L(N, S ) equation obtained in min min Section 5.
中文: 由于计算效率高的培训使用较少的优化步骤,可能需要就加快早期培训动态开展更多工作。 6.2 L(N,S)分的预测 L(C)和分配结果可以从分分钟第5节获得的L(N,S)等式中预测出。
<a id="S0230"></a> Source: p.16 S0230
Original: Given our equation for L(N, S min ), we can substitute S min = C 6N m B in and then find the minimum of the loss as a function of N , while fixing the training compute.
中文: 鉴于我们的L(N, S min)等式,我们可以将S min = C 6N m B 置入并找到最小损失值作为N的函数,同时确定训练计算.
<a id="S0231"></a> Source: p.16 S0231
Original: We carry out this procedure in detail in Appendix B, where we also provide some additional predictions.
中文: 我们在附录B中详细介绍了这一程序,并提供了一些额外的预测。
<a id="S0232"></a> Source: p.16 S0232
Original: For the loss as a function of training compute, we predict that (cid:18) Cmin (cid:19)αm C in L(C ) = c (6.3) min C min where 1 αmin ≡ ≈ 0.054 (6.4) C 1/α + 1/α + 1/α S B N in excellent agreement with the exponent of Figure 13.
中文: 对于作为训练计算的一项函数的损失,我们预测(cid:18)Cmin(cid:19)αm C in L(C) = c (6.3) min C分,其中1αmin → 0.054 (6.4) C 1/α + 1/α + 1/α S B N与图13的注释者非常一致。
<a id="S0233"></a> Source: p.16 S0233
Original: We also predict that N (C min ) ∝ (C min )αm C in/αN ≈ (C min )0.71 (6.5) which also matches the scaling of Figure 14 to within a few percent.
中文: 我们还预测,N(C分)-(C分)-(C分)-(αN分)-(C分)-(C分)-(6.5分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(0.71分(6.5分),(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分(C分)-(C分)-(C分)-(C分)-(C分)-(C分(C分))-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分)-(C分
<a id="S0234"></a> Source: p.16 S0234
Original: Our scaling laws provide a predictive framework for the performance of language modeling. 16
中文: 我们的缩放定律为语言建模的性能提供了一个预测性框架. 16个
<a id="S0235"></a> Source: p.17 S0235
Original: The intersection point is sensitive to the precise power-law parameters Figure 15 Far beyond the model sizes we study empirically, we find a contradiction between our equations for L(C ) and L(D) due to the slow growth of data needed for compute-efficient training.
中文: 相交点对精确的功率-法参数很敏感 图15 远超出我们经验研究的模型大小,我们发现由于计算效率训练所需的数据增长缓慢,我们的L(C)和L(D)等方程之间存在矛盾.
<a id="S0236"></a> Source: p.17 S0236
Original: The intersection min marks the point before which we expect our predictions to break down.
中文: 交接点是我们希望我们的预测破灭的点。
<a id="S0237"></a> Source: p.17 S0237
Original: The location of this point is highly sensitive to the precise exponents from our power-law fits. 6.3 Contradictions and a Conjecture We observe no signs of deviation from straight power-law trends at large values of compute, data, or model size.
中文: 这一点的位置对于我们权力法的精确解释非常敏感。 6.3 矛盾和猜想 在计算、数据或模型大小等大值上,我们没有观察到偏离直向电法趋势的迹象。
<a id="S0238"></a> Source: p.17 S0238
Original: Our trends must eventually level off, though, since natural language has non-zero entropy.
中文: 但是,我们的趋势必须最终平息下来, 因为自然语言有非零的 en。
<a id="S0239"></a> Source: p.17 S0239
Original: Indeed, the trends for compute-efficient training described in this section already contain an apparent contradiction.
中文: 实际上,本节所述的计算效率培训的趋势已经存在明显的矛盾。
<a id="S0240"></a> Source: p.17 S0240
Original: At scales several orders of magnitude above those documented here, the performance predicted by the L(C ) scaling law decreases below what should be possible given the slow growth in training data with min compute.
中文: 在比本文所记录的几等规模上,L(C)缩放法预测的成绩会下降,低于可能的程度,因为培训数据增长缓慢,计算不准确。
<a id="S0241"></a> Source: p.17 S0241
Original: This implies that our scaling laws must break down before this point, but we conjecture that the intersection point has a deeper meaning: it provides an estimate of the point at which Transformer language models reach maximal performance.
中文: 这意味着我们的缩放定律必须在此点之前打破,但我们猜想交叉点有更深的含义:它提供了变形语模型达到最大性能的点的估计.
<a id="S0242"></a> Source: p.17 S0242
Original: Since the amount of data used by compute-efficient training grows slowly with the compute budget, the performance predicted by L(C ) eventually hits a lower bound set by the L(D) power law (see Figure 15). min Let us work this out in more detail.
中文: 由于计算高效培训使用的数据量随计算预算而缓慢增长,由L(C)预测的性能最终会击中由L(D)权力法设定的下限(见图15). 分钟 让我们更详细地解决这个问题。
<a id="S0243"></a> Source: p.17 S0243
Original: To keep overfitting under control, the results of Section 4 imply that we should scale the dataset size as D ∝ N 0.74 ∝ C0.54 (6.6) min where we have used the compute-efficient N (C ) from Figure 14. min Let us compare this to the data requirements of compute-efficient training.
中文: 为了保持过度控制,第4节的结果意味着,我们应该将数据集的大小缩放为D-X-N 0.74-X-C0.54(6.6)分,因为我们使用了图14中的计算效率(C)分,让我们将此与计算效率培训的数据要求进行比较。
<a id="S0244"></a> Source: p.17 S0244
Original: If we train at the critical batch size (i.e. C = 2C ) and never re-use data during training, we find that data usage grows with compute as min D(C ) = 2C min ≈ (cid:0) 4 × 1010 tokens (cid:1) (C /PF-Day)0.26 (6.7) min 6N (C ) min min This is the maximum rate at which the dataset size can productively grow with compute, since it means that we are only training for a single epoch.
中文: 如果我们按照关键批量大小(即C=2C)进行训练并在训练期间不再重复使用数据,我们发现数据使用量随着计算而增长,如min D(C)=2C min ≈ (cid:0) 4×10个令牌 (cid:1) (C/PF-Day)0.26 (6.7) min 6N (C) min min 这是数据集大小能够通过计算产生生产力增长的最大速度,因为它意味着我们只是为单一时代而训练.
<a id="S0245"></a> Source: p.17 S0245
Original: But it grows the dataset much more slowly than in Equation (6.6).
中文: 但是,它比Equation(6.6)的数据集增长得慢得多.
<a id="S0246"></a> Source: p.17 S0246
Original: It appears to imply that compute-efficient training will eventually run into a problem with overfitting, even if the training process never re-uses any data!
中文: 这似乎意味着计算效率的训练最终会遇到过于适应的问题,即使训练过程从未重复使用任何数据!
<a id="S0247"></a> Source: p.17 S0247
Original: According to Figure 1, we expect that when we are bottlenecked by the dataset size (ie by overfitting), the loss should scale as L(D) ∝ D−0.095.
中文: 根据图1,我们预计,当我们被数据集的大小所束缚(即过度调整)时,损失将达到L(D)--D-0.095。
<a id="S0248"></a> Source: p.17 S0248
Original: This implies that the loss would scale with compute as L(D(C )) ∝ min C−0.03 once we are data-limited.
中文: 这意味着,一旦我们数据有限,损失将以L(D(C))- min C-0.03计算。
<a id="S0249"></a> Source: p.17 S0249
Original: Once again, we have a contradiction, as this will eventually intersect with min our prediction for L(C ) from Figure 13, where we found a scaling L(C ) ∝ C−0.050. min min min The intersection point of L(D(C )) and L(C ) occurs at min min C∗ ∼ 104 PF-Days N ∗ ∼ 1012 parameters, D∗ ∼ 1012 tokens, L∗ ∼ 1.7 nats/token (6.8) though the numerical values are highly uncertain, varying by an order or magnitude in either direction depending on the precise values of the exponents from the power-law fits.
中文: 我们再次出现矛盾,因为这最终会与图13中的L(C)预测相接,我们在那里发现了一个缩放L(C)-C-0.050。 分钟 L(D(C))和L(C)的相交点出现在分钟C 104 PF-Days N 1012参数 D 1012令牌 L 1.7 nats/token (6.8),尽管数值很不确定,但根据权力法的对应方的精确值在任一方向的顺序或数量上有所不同。
<a id="S0250"></a> Source: p.17 S0250
Original: The most obvious interpretation is that our scaling laws break down at or before we reach this point, which is still many orders of magnitude away in both compute and model size. 17
中文: 最明显的解释是,我们的缩放法则在我们达到这个点之前或在此点之前就崩溃了,在计算大小和模型大小上,这仍然是许多数量级相去甚远的. 页:1
<a id="S0251"></a> Source: p.18 S0251
Original: One might also conjecture that this intersection point has a deeper meaning.
中文: 人们还可能猜想,这个交叉点有更深的含义.
<a id="S0252"></a> Source: p.18 S0252
Original: If we cannot increase the model size beyond N ∗ without qualitatively different data requirements, perhaps this means that once we reach C∗ and N ∗, we have extracted all of the reliable information available in natural language data.
中文: 如果我们不能将模型的大小扩大至N 以外,而数据要求则质量不同,也许这意味着,一旦我们达到C 和N * ,我们就提取出自然语言数据中现有的所有可靠信息。
<a id="S0253"></a> Source: p.18 S0253
Original: In this min interpretation, L∗ would provide a rough estimate for the entropy-per-token7 of natural language.
中文: 在此分解中,L*将粗略估计自然语言的 en-per-per-per7.
<a id="S0254"></a> Source: p.18 S0254
Original: In this scenario, we would expect the loss trend to level off at or before L∗.
中文: 在此情况下,我们预计损失趋势会稳定在L*或之前。
<a id="S0255"></a> Source: p.18 S0255
Original: We can guess at the functional form of L(C ) as it levels off by considering a version of our training min dataset with added noise.
中文: 我们可以猜测L(C)的功能形式, 它通过考虑一个版本的训练分数数据集, 加上噪音。
<a id="S0256"></a> Source: p.18 S0256
Original: For example, we could append a random string of tokens to each context shown to the model to artificially boost the loss by a constant additive factor.
中文: 例如,我们可以在模型上显示的每个上下文上附加一串随机的指使符来人为地使损失增加一个恒定的添加因子.
<a id="S0257"></a> Source: p.18 S0257
Original: Then, the distance from the noise floor L − L would be a more meaningful performance metric, with even a small decrease in this distance noise potentially representing a significant boost in qualitative performance.
中文: 然后,与噪音地层L-L的距离将是一个更有意义的性能衡量标准,这种距离噪音甚至会小幅地减少,从而可能显著地提升质量性能。
<a id="S0258"></a> Source: p.18 S0258
Original: Since the artificial noise would affect all of our trends equally, the critical point of 6.8 would not change (aside from the absolute value of L∗), and may be meaningful even if it occurs after the leveling off. 7 Related Work Power laws can arise from a wide variety of sources [THK18].
中文: 由于人工噪音会同样地影响我们的所有趋势,6.8的临界点不会改变(除了L*的绝对值外),即使它发生在平定后也可能有意义. 7 相关工作权力法可以从多种来源产生[THK18].
<a id="S0259"></a> Source: p.18 S0259
Original: Power-law scalings with model and dataset size in density estimation [Was06] and in random forest models [Bia12] may be connected with our results.
中文: 在密度估计[Was06]和随机森林模型[Bia12]中采用模型和数据集大小的电法缩放可能与我们的结果有关。
<a id="S0260"></a> Source: p.18 S0260
Original: These models suggest that power-law exponents may have a very rough interpretation as the inverse of the number of relevant features in the data.
中文: 这些模型表明,权力法论者可能有一个非常粗略的解释,作为数据中相关特征数量的倒数.
<a id="S0261"></a> Source: p.18 S0261
Original: Some early [BB01, Goo01] work found power-law scalings between performance and dataset size.
中文: 有的早期[BB01,Goo01]作品在性能和数据集大小之间发现了动力法缩放.
<a id="S0262"></a> Source: p.18 S0262
Original: More recent work [HNA+17, HAD19] also investigated scaling between model size and data size; their work is perhaps the closest to ours in the literature8.
中文: 最近的工作[HNA+17,HAD19]也调查了模型大小和数据大小之间的比例;他们的工作或许是文献8中最接近我们的工作.
<a id="S0263"></a> Source: p.18 S0263
Original: Note, however, that [HNA+17] found super-linear scaling of dataset size with model size, whereas we find a sub-linear scaling.
中文: 然而请注意,[HNA+17]发现了带有模型大小的数据集大小的超线性缩放,而我们却发现了子线性缩放.
<a id="S0264"></a> Source: p.18 S0264
Original: There are some parallels between our findings on optimal allocation of compute and [Kom19], including power-law learning curves.
中文: 我们关于计算的最佳分配的结论与[Kom19],包括动力法学习曲线,有一些相似之处。
<a id="S0265"></a> Source: p.18 S0265
Original: EfficientNets [TL19] also appear to obey an approximate power-law relation between accuracy and model size.
中文: 效率网[TL19]似乎也符合精确度和模型大小之间的近似权力法关系。
<a id="S0266"></a> Source: p.18 S0266
Original: Very recent work [RRBS19b] studies scaling with both dataset size and model size for a variety of datasets, and fits an ansatz similar to ours.
中文: 最近的工作[RRBS19b]研究了各种数据集的数据集大小和模型大小的缩放,并搭配了与我们相似的Ansatz.
<a id="S0267"></a> Source: p.18 S0267
Original: EfficientNet [TL19] advocates scaling depth and width exponentially (with different coefficients) for optimal performance of image models, resulting in a power-law scaling of width as a function of depth.
中文: 高效Net[TL19]主张以指数(有不同的系数)来缩放深度和宽度,以优化图像模型的性能,从而形成以宽度为函数的功率法缩放.
<a id="S0268"></a> Source: p.18 S0268
Original: We find that for language models this power should be roughly one when scaling up (as width/depth should remain fixed).
中文: 我们认为,对于语言模型来说,这种力量在扩大时应该大约是一个(因为宽度/深度应该保持不变)。
<a id="S0269"></a> Source: p.18 S0269
Original: But more importantly, we find that the precise architectural hyperparameters are unimportant compared to the overall scale of the language model.
中文: 但更重要的是,我们发现精确的建筑超参数与语言模型的整体尺度相比并不重要.
<a id="S0270"></a> Source: p.18 S0270
Original: In [VWB16] it was argued that deep models can function as ensembles of shallower models, which could potentially explain this finding.
中文: 在[VWB16]中,有人认为深层模型可以作为更浅层模型的集合体来发挥作用,这有可能解释这个发现.
<a id="S0271"></a> Source: p.18 S0271
Original: Earlier work [ZK16] has compared width and depth, and found that wide ResNets can outperform deep ResNets on image classification.
中文: 更早的工作[ZK16]比较了宽度和深度,发现宽度的ResNet可以在图像分类上超越深度的ResNet.
<a id="S0272"></a> Source: p.18 S0272
Original: Some studies fix computation per data example, which tends to scale in proportion to the number of model parameters, whereas we investigate scaling with both model size and the quantity of training computation.
中文: 一些研究固定每个数据示例的计算,这往往与模型参数的数量成正比,而我们则用模型大小和培训计算的数量来调查规模。
<a id="S0273"></a> Source: p.18 S0273
Original: Various works [AS17, BHMM18] have investigated generalization in highly overparameterized models, finding a “jamming transition” [GJS+19] when the model size reaches the dataset size (this may require training many orders of magnitude beyond typical practice, and in particular does not use early stopping).
中文: 各种作品[AS17,BHMM18]在高度高参数化模型中调查了通缩性,在模型大小达到数据集大小时发现了"jamming transform"GJS+19.
<a id="S0274"></a> Source: p.18 S0274
Original: We do not observe such a transition, and find that the necessary training data scales sublinearly in the model size.
中文: 我们没有看到这样的过渡,发现必要的培训数据在模型大小中线性地缩小。
<a id="S0275"></a> Source: p.18 S0275
Original: Expansions in the model size, particularly at large width [JGH18, LXS+19], may provide a useful framework for thinking about some of our scaling relations.
中文: 模型大小的扩展,特别是宽度较大的[JGH18,LXS+19],可以为思考我们的一些缩放关系提供有用的框架.
<a id="S0276"></a> Source: p.18 S0276
Original: Our results on optimization, such as the shape of learning curves, can likely be explained using a noisy quadratic model, which can provide quite accurate predictions [ZLN+19] in realistic settings.
中文: 我们优化的结果,例如学习曲线的形状,很可能可以用一个吵闹的四极模型来解释,这个模型可以在现实环境中提供相当准确的预测[ZLN+19].
<a id="S0277"></a> Source: p.18 S0277
Original: Making this connection quantitative will require a characterization of the Hessian spectrum [Pap18, GKX19, GARD18]. 8 Discussion We have observed consistent scalings of language model log-likelihood loss with non-embedding parameter count N , dataset size D, and optimized training computation C , as encapsulated in Equations (1.5) and min (1.6).
中文: 使这种接通量化将需要对黑森光谱进行定性[Pap18,GKX19,GARD18]. 8 讨论情况 我们观测到在非嵌入参数计数N、数据集大小D以及优化培训计算C方面对语言模型的日志相似性损失进行一致的缩放,这些数值被封入方程式(1.5)和分数(1.6)。
<a id="S0278"></a> Source: p.18 S0278
Original: Conversely, we find very weak dependence on many architectural and optimization hyperparameters.
中文: 相反,我们发现对许多建筑和优化超参数的依赖非常薄弱。
<a id="S0279"></a> Source: p.18 S0279
Original: Since scalings with N, D, C are power-laws, there are diminishing returns with increasing scale. min 7Defining words using the wc utility, the WebText2 dataset has 1.4 tokens per word and 4.3 characters per token. 8After this work was completed, [RRBS19a] also appeared, which makes similar predictions for the dependence of loss on both model and dataset size. 18
中文: 由于与N,D,C的缩放是权力法,因此回报率随着规模的扩大而不断下降. min 7 使用 wc 工具定义单词,WebText2 数据集每个单词有1.4个令牌,每个令牌有4.3个字符. 8 这项工作完成后,[RRBS19a]也出现了,这对损失依赖模型和数据集大小作了类似的预测. 第 18 条
<a id="S0280"></a> Source: p.19 S0280
Original: We were able to precisely model the dependence of the loss on N and D, and alternatively on N and S, when these parameters are varied simultaneously.
中文: 我们能够精确地模拟损失对N和D的依赖,或者说N和S的依赖,因为这些参数同时不同。
<a id="S0281"></a> Source: p.19 S0281
Original: We used these relations to derive the compute scaling, magnitude of overfitting, early stopping step, and data requirements when training large language models.
中文: 我们利用这些关系来得出计算尺度, 超合度, 早期停止步骤, 以及在训练大型语言模型时的数据要求。
<a id="S0282"></a> Source: p.19 S0282
Original: So our scaling relations go beyond mere observation to provide a predictive framework.
中文: 因此,我们的规模关系超越了单纯的观察,提供了一个预测性框架。
<a id="S0283"></a> Source: p.19 S0283
Original: One might interpret these relations as analogues of the ideal gas law, which relates the macroscopic properties of a gas in a universal way, independent of most of the details of its microscopic consituents.
中文: 人们可能将这些关系解释为理想气体法的类似物,它以普遍的方式将气体的宏观特性联系起来,独立于其微观形态的大部分细节。
<a id="S0284"></a> Source: p.19 S0284
Original: It is natural to conjecture that the scaling relations will apply to other generative modeling tasks with a maximum likelihood loss, and perhaps in other settings as well.
中文: 自然可以推测,缩放关系将适用于其他基因模型制作任务,其可能性损失最大,也许在其他情况下也是如此。
<a id="S0285"></a> Source: p.19 S0285
Original: To this purpose, it will be interesting to test these relations on other domains, such as images, audio, and video models, and perhaps also for random network distillation.
中文: 为此,在图像、音频和视频模型等其他领域测试这些关系,或许还可以进行随机网络蒸馏,将很有趣。
<a id="S0286"></a> Source: p.19 S0286
Original: At this point we do not know which of our results depend on the structure of natural language data, and which are universal.
中文: 目前,我们不知道哪些结果取决于自然语言数据的结构,哪些是普遍的。
<a id="S0287"></a> Source: p.19 S0287
Original: It would also be exciting to find a theoretical framework from which the scaling relations can be derived: a ‘statistical mechanics’ underlying the ‘thermodynamics’ we have observed.
中文: 找到一个理论框架,从中得出缩放关系也令人振奋:我们观察到的“热力学”背后的“统计力学”。
<a id="S0288"></a> Source: p.19 S0288
Original: Such a theory might make it possible to derive other more precise predictions, and provide a systematic understanding of the limitations of the scaling laws.
中文: 这种理论可能使人们有可能得出其他更准确的预测,并系统了解缩放法的局限性。
<a id="S0289"></a> Source: p.19 S0289
Original: In the domain of natural language, it will be important to investigate whether continued improvement on the loss translates into improvement on relevant language tasks.
中文: 在自然语言领域,必须调查持续改善损失是否转化为相关语言任务的改进。
<a id="S0290"></a> Source: p.19 S0290
Original: Smooth quantitative change can mask major qualitative improvements: “more is different”.
中文: 数量上的平稳变化可以掩盖重大的质量改进:“更多的是不同的”。
<a id="S0291"></a> Source: p.19 S0291
Original: For example, the smooth aggregate growth of the economy provides no indication of the specific technological developments that underwrite it.
中文: 例如,经济的平稳总体增长并不能说明支撑经济的具体技术发展。
<a id="S0292"></a> Source: p.19 S0292
Original: Similarly, the smooth improvements in language model loss may hide seemingly qualitative changes in capability.
中文: 同样,语言模型丧失的平稳改进可能掩盖能力似乎在质量上的变化。
<a id="S0293"></a> Source: p.19 S0293
Original: Our results strongly suggest that larger models will continue to perform better, and will also be much more sample efficient than has been previously appreciated.
中文: 我们的结果表明,更大的模型将继续发挥更好的作用,而且比以前得到赞赏的样本效率要高得多。
<a id="S0294"></a> Source: p.19 S0294
Original: Big models may be more important than big data.
中文: 大模型可能比大数据更重要.
<a id="S0295"></a> Source: p.19 S0295
Original: In this context, further investigation into model parallelism is warranted.
中文: 在这方面,需要进一步调查模式平行主义。
<a id="S0296"></a> Source: p.19 S0296
Original: Deep models can be trained using pipelining [HCC+18], which splits parameters depth-wise between devices, but eventually requires increased batch sizes as more devices are used.
中文: 深层模型可以使用管道衬线[HCC+18]进行训练,在设备之间将参数深度相分,但随着更多设备的使用,最终需要增加批量尺寸.
<a id="S0297"></a> Source: p.19 S0297
Original: Wide networks on the other hand are more amenable to parallelization [SCP+18], since large layers can be split between multiple workers with less serial dependency.
中文: 另一方面,宽网比较容易被平行化[SCP+18],因为大层可以被分拆成多个工人,而序列依赖性较低.
<a id="S0298"></a> Source: p.19 S0298
Original: Sparsity [CGRS19, GRK17] or branching (e.g. [KSH12]) may allow for even faster training of large networks through increased model parallelism.
中文: Sparsity [CGRS19,GRK17]或分支化(如[KSH12])可能允许通过增加模型平行性对大型网络进行更快的培训.
<a id="S0299"></a> Source: p.19 S0299
Original: And using methods like [WRH17, WYL19], which grow networks as they train, it might be possible to remain on the compute-efficient frontier for an entire training run.
中文: 使用诸如[WRH17,WYL19]的方法,这些方法在训练时会发展网络,因此有可能在整个训练活动中保持在计算效率高的前沿。
<a id="S0300"></a> Source: p.19 S0300
Original: Acknowledgements We would like to thank Shan Carter, Paul Christiano, Jack Clark, Ajeya Cotra, Ethan Dyer, Jason Eisner, Danny Hernandez, Jacob Hilton, Brice Menard, Chris Olah, and Ilya Sutskever for discussions and for feedback on drafts of this work. 19
中文: 鸣谢 我们要感谢掸·卡特、保罗·克里斯蒂亚诺、杰克·克拉克、阿杰亚·科特拉、伊森·戴尔、杰森·艾斯纳、丹尼·赫南德斯、雅各布·希尔顿、布里斯·梅纳德、克莉丝·奥拉和伊利亚·苏茨克韦尔就这项工作的草案进行讨论和反馈。 第 19 条
<a id="S0301"></a> Source: p.20 S0301
Original: Appendices A Summary of Power Laws For easier reference, we provide a summary below of the key trends described throughout the paper.
中文: 附录 权力法摘要 为便于参考,下文概述整个文件所述的主要趋势。
<a id="S0302"></a> Source: p.20 S0302
Original: Parameters Data Compute Batch Size Equation N ∞ ∞ Fixed L (N ) = (N /N )αN c ∞ D Early Stop Fixed L (D) = (D /D)αD c Optimal ∞ C Fixed L (C) = (C /C)αC (naive) c N D C B (cid:28) B L (C ) = (cid:0) Cmin/C (cid:1)αm C in opt opt min crit min c min N D Early Stop Fixed L (N, D) = (cid:20) (cid:0) Nc (cid:1) α α N D + Dc (cid:21)αD N D N ∞ S steps B L (N, S) = (cid:0) Nc (cid:1)αN + (cid:16) Sc (cid:17)αS N Smin(S,B) Table 4 The empirical fitted values for these trends are: Power Law Scale (tokenization-dependent) α = 0.076 N = 8.8 × 1013 params (non-embed) N c α = 0.095 D = 5.4 × 1013 tokens D c α = 0.057 C = 1.6 × 107 PF-days C c αmin = 0.050 Cmin = 3.1 × 108 PF-days C c α = 0.21 B = 2.1 × 108 tokens B ∗ α = 0.76 S = 2.1 × 103 steps S c Table 5 The optimal parameters for compute efficient training are given by: Compute-Efficient Value Power Law Scale N = N · CpN p = 0.73 N = 1.3 · 109 params opt e min N e B (cid:28) B crit = L1 B /α ∗ B = B e C m pB in p B = 0.24 B e = 2.0 · 106 tokens S = S · CpS (lower bound) p = 0.03 S = 5.4 · 103 steps min e min S e D = D · CpD (1 epoch) p = 0.27 D = 2 · 1010 tokens opt e min D e Table 6 B Empirical Model of Compute-Efficient Frontier Throughout this appendix all values of C, S, and α are adjusted for training at the critical batch size B . C crit We have left off the ‘adj’ label to avoid cluttering the notation. B.1 Defining Equations The power-law fit to the learning curves implies a simple prescription for compute-efficient training.
中文: 参数数据计算 Batch 大小方程 N Q 固定 L (N) = (N/N) αN c = (D/D) 早期停止 固定 L (D) = (D/D) 优化 C 固定 L (C) = (C/C) αC (naive) c N D B (cid:28) B (Cid:0) Cmin/C (Cid:1) C 选择选用 min crit min N 早期停止 固定 L (N,D) = (cid:20) (cid:0) α N D + Dc (cid:1) α D + Dc (Cid:21) N Cc (c:1) αN + (Cid:16) Sc (c:17) N Smin (S:B) 表4 这些趋势的实证配给值是: 功率平面(托能化-依赖)α = 0.076 N = 8.8 × 1013个参数(非嵌入) N c α = 0.095 D = 5.4 × 1013个令牌 D c α = 0.057 C = 1.6 × 107 PF-day C c αmin = 0.050 Cmin = 3.1 × 108 PF-day C C α = 0.21 B = 2.1 B × 108个令牌 α = 0.76 S = 2.1 × 103 步骤 S c 表5 计算高效培训的最佳参数由以下因素给出: 计算-有效价值 力法 分级 N = N = CpN p = 0.73 N = 1.3 = 109 p. ams 选择 e min N e B (cid:28) B crit = L1 B / α B = B e C pB 在p B = 0.24 B e = 2.0 = 106个令牌 S = S = CpS (下限) p = 0.03 S = 5.4 = 103 步 e min S e D = D = 0.27 D = 2 = 10个令牌 选择 e min D e 表 6 B 模拟边界模型 在本附录中,对C、S和α的所有值进行调整,以便按关键批号B进行培训。 我们不再贴上“adj”的标签, B.1 界定方程式 适合学习曲线的动力法意味着计算效率培训的简单处方.
<a id="S0303"></a> Source: p.20 S0303
Original: In this appendix, we will derive the optimal performance, model size, and number of training steps as a function of 20
中文: 在本附录中,我们将得出最佳业绩、模式规模和培训步骤的数目,以发挥20个功能。
<a id="S0304"></a> Source: p.21 S0304
Original: We start with the Equation (1.6), repeated here for convenience: (cid:18) N (cid:19)αN (cid:18) S (cid:19)αS L (N, S) = c + c . (B.1) N S Here, S represents the number of parameter updates when training at the critical batch size [MKAT18], which was defined in Equation (5.2)9: B B (L) = ∗ . (B.2) L1/αB We would like to determine optimal training parameters for a fixed compute budget, so we replace S = C/ (6N B (L)), where C is the number of FLOPs used in the training run: (cid:18) N (cid:19)αN (cid:18) N (cid:19)αS L (N, C) = c + 6B S . (B.3) N ∗ c L1/αB C (cid:12) Now, we set ∂ N L(cid:12) C = 0 to find the condition for optimality: ∂L (cid:12) 0 = (cid:12) ∂N C = − α N N (cid:18) N N c (cid:19)αN + α N S (cid:18) 6B ∗ S c L1/ N αB C (cid:19)αS (cid:18) 1 − 5 N L (cid:26)∂ ∂ N L (cid:26)(cid:12) (cid:12) (cid:26) C (cid:19) α (cid:18) N (cid:19)αN (cid:18) N (cid:19)αS =⇒ N c = 6B S (B.4) α S N ∗ c L1/αB C Equation (B.3) and (B.4) together determine the compute-efficient frontier. B.2 Efficient Training Now we assemble the implications of (B.3) and (B.4).
中文: 我们从公式(1.6)开始,为方便起见在此重复:(cid:18)N(cid:19)αN(cid:18)S(cid:19)αS L(N,S) = c + (B.1) N S Here, S 代表关键批量尺寸[MKAT18]培训时参数更新的数量,定义在公式(5.2)9:B(L) = .(B.2) L1/αB. 我们想确定固定计算预算的最佳训练参数,所以我们取代S=C/(6N B (L)),其中C是训练运行中使用的FLOP的数量:(Cid:18)N(Cid:19)αN(Cid:18)N(Cid:19)αS L(N,C)=c+6B S. (B.3) N C L1/αB C (cid:12) 现在,我们设置了 → N L(cid:12) C = 0 以寻找优化条件: → L (cid:12) 0 = (cid:12) → N C (cid:18) → N (c (cid:19) + α N (cid:18) 6 B → S C (Cid:19) αB C (cid:18) αS (cid:18) 1 → N L (Cid:18) → 5 N L (cid:26) → (cid:12) (cid:26) C (cid:19) αN (cid:18) → (cid:18) N (cid:19) αS (Cid:19) → N c = 6B (B.4) αS * CαB (cid:19) αB 1 (B.3) 和 (B.4)共同确定可计算高效边界. B.2 高效培训,我们现在收集(B.3)和(B.4)的影响。
<a id="S0305"></a> Source: p.21 S0305
Original: First, note that inserting (B.4) into (B.3) yields (cid:18) (cid:19) α L (N (C) , C) = 1 + N L (N , ∞) , (B.5) eff α eff S which implies that for compute-efficient training, we should train to a fixed percentage αN ≈ 10% above αS the converged loss.
中文: 首先,注意将(B.4)插入(B.3)产量(Cid:18)(Cid:19)αL(N(C),C)=1+NL(N,Q),(B.5)eff αeff S,这意味着对于计算效率的训练,我们应该在αS上方培训一个固定百分比的αN + 10%的集合损失。
<a id="S0306"></a> Source: p.21 S0306
Original: Next, let’s determine how the optimal loss depends on the compute budget.
中文: 接下来,让我们确定最佳损失如何取决于计算预算。
<a id="S0307"></a> Source: p.21 S0307
Original: Eliminating N yields a power-law dependence of performance on compute: (cid:18) C (cid:19)αC L (C) = c (B.6) C where we defined α = 1/ (1/α + 1/α + 1/α ) ≈ 0.052 (B.7) C S B N (cid:18) α (cid:19)1/αS+1/αN (cid:18) α (cid:19)1/αS C = 6N B S 1 + N S . (B.8) c c ∗ c α α S N Similarly, we can eliminate L to find N (C): N (C) (cid:18) C (cid:19)αC/αN (cid:18) α (cid:19)1/αN = 1 + N (B.9) N C α c c S and C (cid:18) α (cid:19)−1/αN (cid:18) C (cid:19)αC/αS S (C) = c 1 + N (B.10) 6N B α C c ∗ S c 9There is a slight ambiguity here: we can imagine training either at a constant batch size B (L ), or we could target instead train at a variable batch size B˜ (L), where B˜ is the instantaneous critical batch size (as opposed to B, which is the averaged version).
中文: 消除N产生对功率的依赖: C (cid:18) C (cid:19)αC L (C) = c (B.6) C 我们定义的α = 1 (1/α + α + 1/α) → 0.052 (B.7) C B N (Cid:18) α (cid:19) 1 (Cid:19) 1 (Cid:18) αS = 6N B S 1 + N S (B.8) c * αS (Cid:19)相类似,我们可以消除 L αS (Cid:18) C (Cid:18) C (Cid:19) αC/αN (Cid:18) α (Cid:18) αN (Cid:19) αN = 1 (Cid:19) N和 C (Cid:18) → C (Cid:18) B (Cid: αN: αS: αS- αS) 不变的B / C- 等于 C+10) 的B 的
<a id="S0308"></a> Source: p.21 S0308
Original: These two prescriptions result in the same number of steps, so we can ignore this subtlety (see [MKAT18]). 21
中文: 这两个处方产生相同的步骤数量,因此我们可以忽略这种微妙(参见[MKAT18]). 21国
<a id="S0309"></a> Source: p.22 S0309
Original: B.3 Comparison to Inefficient Typically, researchers train models until they appear to be close to convergence.
中文: B.3 与低效率的比较一般情况下,研究人员对模型进行训练,直到它们似乎接近趋同为止。
<a id="S0310"></a> Source: p.22 S0310
Original: In this section, we compare the efficient training procedure described above to this more typical setup.
中文: 在本节中,我们将上述高效培训程序与这一更典型的设置进行比较。
<a id="S0311"></a> Source: p.22 S0311
Original: We define a the convergence factor f as the percent deviation from the converged loss: L (N, C) = (1 + f ) L (N, ∞) . (B.11) For compute-efficient training we have f = α /α ≈ 10% from the previous section, but researchers N S typically use a much smaller value.
中文: 我们将趋同系数f定义为与趋同损失的百分比偏差:L (N, C) = (1 + f) L (N, + ). (B.11). 对于计算效率的训练,我们从上一节有 f = α / α → 10%,但研究人员N S 通常使用更小的值.
<a id="S0312"></a> Source: p.22 S0312
Original: Here, we choose f (cid:48) = 2% as an estimate.
中文: 在这里,我们选择f(cid:48)=2%作为估计值.
<a id="S0313"></a> Source: p.22 S0313
Original: For a fixed value of the loss, we predict: N (cid:18) 1 + f (cid:19)1/αN f = ≈ 2.7 (B.12) N 1 + f (cid:48) f(cid:48) S (cid:32) 1 + 1 (cid:33)1/αS f = f ≈ 0.13 (B.13) S 1 + 1 f(cid:48) f(cid:48) C N S f = f f ≈ 0.35 (B.14) C N S f(cid:48) f(cid:48) f(cid:48) So that compute-efficient training uses 7.7x fewer parameter updates, 2.7x more parameters, and 65% less compute to reach the same loss. B.4 Suboptimal Model Sizes We can solve A.1 to find an expression for the amount of compute needed to reach a given value of the loss L with a model of size N : (cid:18) N (cid:19) (cid:18) (cid:18) N (cid:19)αN (cid:19)−1/αS C (N, L) = 6B S L − c . (B.15) ∗ c L1/αB N Using A.6 and A.9, we can eliminate L in favor of N (L), the model size which reaches L most efficiently. eff From there, we find an expression for the excess compute needed as a consequence of using a suboptimal model size: C (N, N ) N (cid:20) α (cid:18) (cid:18) N (cid:19)αN (cid:19)(cid:21)−1/αS eff = 1 + S 1 − eff . (B.16) C (N , N ) N α N eff eff eff N The result is shown in Figure X.
中文: 对于损失的固定值,我们预测: N (Cid:18) 1 + f (Cid:191/αN f = → 2.7 (B.12) N 1 + f (Cid:48) f (Cid:32) 1 + 1 (Cid:331/αS f = → 0.13 (B.13) S 1 + 1 f (Cid:48) f (Cid:48) f (Cid:48) f (Cid:48) f (Cid:48) f (cid:48) f (cid:48) F. 因此,计算效率训练使用7.7x更少的参数更新,2.7x多的参数,65%的计算达到相同的损失. B.4 低于最佳模型大小 我们可以解决A.1找到一个表达式,说明达到损失L的给定值所需的计算量,其型号为: N (Cid:18) N (Cid:19)(Cid:18)(Cid:18) N (Cid:19)αN (Cid:19)-1/αS C (N,L) = 6B S L - c (B.15) * CL1/αB N 使用A.6和A.9,我们可以将L去掉,以N (L)为主,这个型号最有效率地达到L. eff 从那里,我们发现一种表示因使用次优化模型大小而需要的过量计算: C (N, N) N (Cid:20) α (Cid:18) (Cid:19) α N (Cid:19) (Cid:21)−1/αS eff = 1 + S 1 → eff. C (B.16) (N, N) α Neff eff → eff N. 结果见图十。
<a id="S0314"></a> Source: p.22 S0314
Original: Models between 0.6x and 2.2x the optimal size can be used with only a 20% increase in compute budget.
中文: 最佳尺寸介于0.6x至2.2x之间的模型,在计算预算只增加20%的情况下才能被使用.
<a id="S0315"></a> Source: p.22 S0315
Original: Using a smaller model is useful when accounting for the cost inference. A larger model can be trained the the same level of performance in fewer steps, allowing for more parallelism and faster training if sufficient harware is available (see Figure Y): S (N, N ) (cid:20) α (cid:18) (cid:18) N (cid:19)αN (cid:19)(cid:21)−1/αS eff = 1 + S 1 − eff . (B.17) S (N , N ) α N eff eff N A 2.2x larger model requires 45% fewer steps at a cost of 20% more training compute.
中文: 在计算成本推断时,使用较小的模型是有用的。 一个较大的模型可以在更少的阶梯中接受同样水平的性能训练,如果有足够的机匣(见图Y): S (N, N (cid:20) α (cid:18) (cid:19) αN (cid:19) (cid:21)−1/αS eff = 1 + S 1 - eff. (B.17) S (N, N) α Neff eff N A 2.2x 较大模型需要减少45%的阶梯,需要增加20%的训练计算.
<a id="S0316"></a> Source: p.22 S0316
Original: Note that this equation should not be trusted for very large models, as it is only valid in the power-law region of the learning curve after initial transient effects. C Caveats In this section we list some potential caveats to our analysis. • At present we do not have a solid theoretical understanding for any of our proposed scaling laws.
中文: 注意这个等式不应被信任于非常大的模型,因为它在初始瞬态效应后,只在学习曲线的动力法区有效. 穴居动物 在本节中,我们列出我们的分析中的一些可能的告诫。 二. 支助 目前,我们还没有对我们提出的任何规模法有坚实的理论理解。
<a id="S0317"></a> Source: p.22 S0317
Original: The scaling relations with model size and compute are especially mysterious.
中文: 与模型大小和计算方式的缩放关系特别神秘.
<a id="S0318"></a> Source: p.22 S0318
Original: It may be possible to understand scaling at very large D holding model size fixed [AS17], and also the shape of learning curves late in training, by modeling the loss with a noisy quadratic.
中文: 可以通过用吵闹的四极体来模拟损失,从而了解在非常大范围内的D持有模型大小固定[AS17],以及训练后期学习曲线的形状.
<a id="S0319"></a> Source: p.22 S0319
Original: But the scaling with D at very large model size still remains mysterious.
中文: 但与D的缩放在非常大的模型尺寸上仍然神秘.
<a id="S0320"></a> Source: p.22 S0320
Original: Without a theory or a systematic understanding of the corrections to our scaling laws, it’s difficult to determine in what circumstances they can be trusted. 22
中文: 很难确定在何种情况下, 22个
<a id="S0321"></a> Source: p.23 S0321
Original: 105 104 103 103 104 105 Sc × [L(N, D) L(N, )] 1/ S S pots 6 Early Stopping Step 5 Data Size 4 21M 43M 86M 172M 3 344M 688M 1.4B 2 103 104 105 Step ssoL Test Loss 1010 Train Loss 109 108 )snekoT( eziS tesataD Figure 16 Left: We characterize the step on which early stopping occurs, as a function of the extent of overfitting.
中文: 105 104 103 104 105 Sc × [L(N, D) L(N,)] 1 S 锅 第6个早期停止步骤 第5个数据大小 4 21M 43M 86M 172M 3 344M 688M 1.4B 2 103 104 起步ssoL试验损失 1010 列车损失 109 108 snekoT(eziS tesataD 图16 左侧:我们把发生早期停止的步骤定性为过量适应程度的一种功能.
<a id="S0322"></a> Source: p.23 S0322
Original: The red line indicates a lower bound for early stopping that is derived in Section 5.3.
中文: 红线表示出第5.3节中衍生出早期停止的下限.
<a id="S0323"></a> Source: p.23 S0323
Original: Right: We display train and test loss for a series of 300M parameter models trained on different sized dataset subsamples.
中文: 对:我们显示列车和测试损失 一系列300M参数模型 训练不同的大小数据集子样本。
<a id="S0324"></a> Source: p.23 S0324
Original: The test loss typically follows that of a run done with unrestricted data until diverging.
中文: 测试损失通常与无限制数据进行的运行有关,直至出现差异。
<a id="S0325"></a> Source: p.23 S0325
Original: Note that the degree of overfitting (as compared to the infinite data limit) is significantly overestimated by L − L test train (denoted by a black bar for each run). • We are not especially confident in the prediction of B (L) for values of the loss far outside the crit range we have explored.
中文: 请注意,L-L试验列车(每跑一次用一黑条表示)对配分过大的程度(与无限数据限制相比)的估算明显过高. 二. 支助 我们并不特别相信B(L)的预测值远远超出我们探索的临界值范围。
<a id="S0326"></a> Source: p.23 S0326
Original: Changes in B could have a significant impact on trade-offs between crit data parallelism and the number of serial training steps required, which would have a major impact on training time. • We did not thoroughly investigate the small data regime, and our fits for L(N, D) were poor for the smallest values of D (where an epoch corresponded to only 40 steps).
中文: B部分的变动可能会对零星数据的并行性和所需系列培训步骤的数目之间的取舍产生重大影响,这将对培训时间产生重大影响。 我们没有彻底调查小数据制度,而我们对于L(N,D)的合适程度对于D的最小值(一个时代只相当于40个步骤)来说是很差的.
<a id="S0327"></a> Source: p.23 S0327
Original: Furthermore, we did not experiment with regularization and data augmentation.
中文: 此外,我们没有试验正规化和数据扩充。
<a id="S0328"></a> Source: p.23 S0328
Original: Improvements in these could alter our results, quantitatively or qualitatively. • We used the estimated training compute C ≈ 6N BS, which did not include contributions proportional to n (see Section 2.1).
中文: 这些方面的改进可能从数量或质量上改变我们的成果。 我们使用估计培训计算公式C 6N BS,其中不包括与n成比例的缴款(见第2.1节)。
<a id="S0329"></a> Source: p.23 S0329
Original: So our scalings with compute may be confounded in practice in the ctx regime of very large n , specifically where n (cid:38) 12d . ctx ctx model • We tuned learning rates, and we experimented with learning rate schedules.
中文: 因此,我们的计算缩放在非常大的n的ctx制度中在实践中可能令人困惑,具体来说就是n(cid:38) 12d.ctx ctx 型号的? 我们调整了学习率, 我们试验了学习率时间表。
<a id="S0330"></a> Source: p.23 S0330
Original: But we may have neglected to tune some hyperparameter (e.g. intialization scale or momentum) that have an important effect on scaling. • The optimal choice of learning rate is sensitive to the target loss.
中文: 但是,我们可能忽略了调制一些对缩放有重要影响的超参数(如入声尺度或动能等). . 学习率的最佳选择对目标损失很敏感.
<a id="S0331"></a> Source: p.23 S0331
Original: When training close to convergence, it may be necessary to use a smaller learning rate to avoid divergences.
中文: 当培训接近趋同时,可能有必要使用较小的学习率来避免分歧.
<a id="S0332"></a> Source: p.23 S0332
Original: But when conducting a short training run (eg due to compute limitations), it may be possible to use a larger learning rate.
中文: 但是,在进行短期训练(例如由于计算限制)时,可能使用较大的学习率.
<a id="S0333"></a> Source: p.23 S0333
Original: We did not experiment with higher learning rates for training runs that did not proceed to convergence. D Supplemental Figures D.1 Early Stopping and Test vs Train In section 5.3 we described the result shown in Figure 16, which provides a prediction for a lower bound on the early stopping step.
中文: 我们没有试行没有趋同的高等教育率。 D 补充数字 D.1 提前停止和测试对列车 在第5.3节中,我们描述了图16所示的结果,图16预测了早期停止步骤的下限。
<a id="S0334"></a> Source: p.23 S0334
Original: We also show the train and test loss for a given model size when training on different sized datasets. D.2 Universal Transformers We compare the performance of standard Transformers to recurrent Transformers [DGV+18] in Figure 17.
中文: 我们还在培训不同大小的数据集时显示列车和测试特定模型大小的损失。 D.2 通用变形器 我们在图17中将标准变换器的性能与经常性变换器[DGV+18]进行比较.
<a id="S0335"></a> Source: p.23 S0335
Original: These models re-use parameters, and so perform slightly better as a function of N , but slightly worse as a function of compute C.
中文: 这些模型再用参数,因此作为N的函数性能稍好,而作为计算C的函数性能稍差.
<a id="S0336"></a> Source: p.23 S0336
Original: We include several different different possibilities for parameter re-use. D.3 Batch Size We measure the critical batch size using the data displayed in figure 18.
中文: 我们列入了若干不同的参数再使用的可能性。 D.3 批量大小 我们利用图18中显示的数据测量批量临界大小。
<a id="S0337"></a> Source: p.23 S0337
Original: This made it possible to estimate B (L) in figure 10. crit 23
中文: 这使得有可能在图10中估计B(L). 小学 23
<a id="S0338"></a> Source: p.24 S0338
Original: 4.5 4.0 3.5 3.0 2.5 105 106 107 108 109 Parameters, including reuse (non-embedding) ssoL tseT 4.5 4.0 3.5 2x Reuse 3.0 4x Reuse 8x Reuse Non-recurrent Models 2.5 105 106 107 108 109 Parameters (non-embedding) ssoL tseT 2x Reuse 4x Reuse 8x Reuse Non-recurrent Models Figure 17 We compare recurrent Transformers [DGV+18], which re-use parameters, to standard Transformers.
中文: 4.5 4.0 3.5 3.0 2.5 105 106 107 108 109 参数包括再用(非嵌入)ssoL tseT 4.5 4.0 3.5 2x再用 3.0 4x再用 8x再用非经常型号 2.5 105 106 107 108 109 ssoL tseT 2x再用 4x再用 8x再用 非经常型号 图17 我们比较了重用参数的经常性变压器[DGV+18]与标准变压器.
<a id="S0339"></a> Source: p.24 S0339
Original: Recurrent Transformers perform slightly better when comparing models with equal parameter count, but slightly worse when accounting for reuse and comparing per FLOP. 1011 1010 109 108 107 106 102 103 104 105 Step dessecorP snekoT Batch Size Scan - 3M Params 10 8 6 4 ssoL tseT 1010 108 106 101 102 103 104 105 Step dessecorP snekoT Batch Size Scan - 85M Params 10 8 6 4 ssoL tseT Figure 18 These figures demonstrate fits to Equation (5.1) for a large number of values of the loss L, and for two different Transformer model sizes.
中文: 经常变换器在将模型与等参数计数进行比较时表现略好,但在计算再利用和比较每个FLOP时表现略好. 1011 1010 109 108 107 106 102 103 104 105 Step dessecorP snekoT Batch Sscan - 3M Params 10 8 6 4 ssoL tseT 1010 108 106 101 102 104 105 Step dessecorP snekoT Batch Sscan - 85M Params 10 8 6 4 ssoL tseT 图18 这些数字表明,损失L的大量值和两个不同的变形器模型大小都符合方程式(5.1)。
<a id="S0340"></a> Source: p.24 S0340
Original: These fits were used to measure B (L) for Figure 10. crit D.4 Sample Efficiency vs Model Size It is easy to see from figure 2 that larger models train faster, and are therefore more sample efficient.
中文: 图10. Crit D.4 样本效率与模型大小 从图2中可以很容易地看到,更大的模型的运行速度更快,因此更具有样本效率.
<a id="S0341"></a> Source: p.24 S0341
Original: We provide another way of looking at this phenomenon in figure 19, which shows when different models reach various fixed values of the loss. 105 104 103 106 107 108 Parameters (non-embedding) ) S( spetS muminiM nim 5.5 5.0 4.5 4.0 3.5 3.0 2.5 ssoL 1011 1010 109 108 106 107 108 Parameters (non-embedding) ) E( selpmaxE muminiM nim 5.5 5.0 4.5 4.0 3.5 3.0 2.5 ssoL Figure 19 The number of minimum serial steps needed to reach any fixed value of the test loss decreases precipitously with model size.
中文: 我们在图19中提供了另一种看待这一现象的方法,表明不同的模型何时达到损失的各种固定值. 105 104 103 106 107 108 参数(非嵌入) S( speetS muminiM nim 5.5 5.5 4.0 3.5 2.5 ssoL 1011 10 108 108 106 107 108 参数(非嵌入)) E( selpmaxE muminiM nim 5.5 5.5 4.5 4.0 3.5 3.0 2.5 ssoL 图19 达到试验损失的任何固定值所需的最低序列步骤的数目随模型大小而急剧减少。
<a id="S0342"></a> Source: p.24 S0342
Original: Sample efficiency (show here for training far below the critical batch size) improves greatly as well, improving by a factor of almost 100 when comparing the smallest possible model to a very large one. 24
中文: 样本效率(在这里显示的培训远远低于批量规模)也大大提高,在将最小的可能模式与非常大的模式进行比较时,提高了近100倍。 24个
<a id="S0343"></a> Source: p.25 S0343
Original: 8 7 6 5 4 3 100 101 102 103 Token Index ssoL tseT nekoT-reP 4.0+3.2 T 0.47 3.4+4.0 T 0.56 2 2 . . 9 7 + + 4 4 . . 5 9 T T 0 0 . . 5 6 6 0 108 2.4+5.1 T 0.61 2.3+5.4 T 0.62 107 106 sretemaraP ledoM 10 8 6 4 2 101 103 105 Step ssoL tseT Per-token Loss (774M Params) 103 102 101 100 xednI nekoT Figure 20 This figure provides information about the performance per token as a function of model size and training time.
中文: 8, 7, 5, 3, 10, 100 102 103 Token Index ssoL tseT nekoT-reP 4.0+3.2 T 0.47 3.4+ 4.0 T 0.56 2 2 . 9 7 + + 4 . 5 9 T 0 0. 6 6 0 108 2.4+ 5.1 T 0.61 2.3+5.4 T 0.62 107 sretemaraP leadoM 10 8 6 4 101 105 Step ssoL tseT per-token Loss (774M Params) 103 102 101 100 xednI nekoT 图20 这一数字提供了每个标志的性能信息,作为模型大小和培训时间的函数.
<a id="S0344"></a> Source: p.25 S0344
Original: Left: Loss per token as a function of its position T in the 1024-token context.
中文: 左:每个符号丢失作为其在1024-token上下文中T位置的函数.
<a id="S0345"></a> Source: p.25 S0345
Original: Loss scales predictably as a power-law in T .
中文: 损失可以预测为 T 中的权力法。
<a id="S0346"></a> Source: p.25 S0346
Original: Right: Test loss per token as a function of training step. 7.5 6.0 4.5 3.0 104 105 106 107 108 109 Parameters (excl. embedding) ssoL tseT Token 1/1024 Token 2/1024 Token 4/1024 Token 8/1024 Token 16/1024 Token 64/1024 Token 256/1024 Token 1024/1024 Token 1/8 Token 2/8 Token 4/8 Token 8/8 Figure 21 In addition to the averaged loss, individual tokens within the 1024-token context also improve smoothly as model size increases.
中文: 右:作为训练步骤的一项功能,测试每个标志的损失。 7.5 6.0 4.5 3.0 104 105 106 107 108 109 参数(不包括嵌入) ssoL tseT Token 1/1024 Token 2/1024 Token 4/1024 Token 8/1024 Token 16/1024 Token 64/1024 Token 256/1024 Token 1/8 Token 2/8 Token 4/8 图21 除了平均损失外,在1024-token上下文中,随着模型尺寸的增大,单个代币也得到平稳改善.
<a id="S0347"></a> Source: p.25 S0347
Original: Training runs with shorter context n = 8 (dashed lines) perform better ctx on early tokens, since they can allocate all of their capacity to them. D.5 Context Dependence The trends for loss as a function of model size are displayed for different tokens in the context in Figure 21.
中文: 上下文n = 8 (干线)的训练在早期标志上表现较好的ctx,因为他们可以将全部能力分配给他们. D.5 环境依赖 损失趋势作为模型大小的一种函数,在图21的上下文中为不同的符号显示。
<a id="S0348"></a> Source: p.25 S0348
Original: We see that models trained on n = 1024 show steady improvement with model size on all but the first ctx token.
中文: 我们看到,在n=1024上训练的模型显示稳步改善,除了第一个ctx符号外,所有模型大小都显示稳步改善.
<a id="S0349"></a> Source: p.25 S0349
Original: Fixing model size, it appears that the loss scales as a power-law as a function of position T in the context, see Figure 20.
中文: 修正模型大小,损失尺度似乎作为上下文中T位置的动力法函数,见图20。
<a id="S0350"></a> Source: p.25 S0350
Original: This may be a consequence of underlying power-law correlations in language [EP94, ACDE12, LT16], or a more general feature of the model architecture and optimization.
中文: 这可能是语言[EP94, ACDE12, LT16]中基础性的权力法相关性的结果,或者是模型架构和优化的一个更普遍的特征.
<a id="S0351"></a> Source: p.25 S0351
Original: It provides some suggestion for the potential benefits (or lack thereof) from training on larger contexts.
中文: 它就较大背景的培训的潜在好处(或缺乏好处)提出了一些建议。
<a id="S0352"></a> Source: p.25 S0352
Original: Not only do larger models converge to better performance at T = 1024, but they also improve more quickly at early tokens, suggesting that larger models are more efficient at detecting patterns with less contextual information.
中文: 更大型的模型不仅在T=1024时会汇合到更好的性能上,而且在早期的标语上也会更快地得到改进,这表明更大型的模型更能用更少的上下文信息来检测出模式.
<a id="S0353"></a> Source: p.25 S0353
Original: In the right-hand plot we show how per-token performance varies for a fixed model as a function of the training step.
中文: 在右手图中,我们显示一个固定模型作为训练步骤的一种功能,每个脚趾的性能如何不同。
<a id="S0354"></a> Source: p.25 S0354
Original: The model begins by learning short-range information, and only learns longer-range correlations later in training.
中文: 该模型从学习短程信息入手,后期在训练中仅能学习更远的关联.
<a id="S0355"></a> Source: p.25 S0355
Original: We have also included models trained with a tiny context n = 8 in order to compare with our longer ctx context models.
中文: 我们还包括了经过微小上下文n=8培训的模型,以便同我们较长的ctx上下文模型进行比较。
<a id="S0356"></a> Source: p.25 S0356
Original: Even modestly sized models trained on n = 8 can dominate our largest n = 1024 ctx ctx models on very early tokens.
中文: 即使是在n=8上受过训练的平分大小的模型,也能在非常早期的符号上主导我们最大的n=1024克特克克特克特克特克特模型.
<a id="S0357"></a> Source: p.25 S0357
Original: This also suggests that further improvements should be possible with much larger models trained on large contexts. D.6 Learning Rate Schedules and Error Analysis We experimented with a variety of learning rates and schedules. A host of schedules and resulting test performances for a small language model are plotted in Figure 22.
中文: 这还表明,如果对大范围模式进行培训,就有可能进一步改进。 D.6 学习率表和错误分析 我们试验了各种学习率和时间表。 图22绘制了小语言模型的一系列时间表和由此产生的测试性能。
<a id="S0358"></a> Source: p.25 S0358
Original: We conclude that the choice of learning rate schedule is mostly irrelevant, as long as the total summed learning rate is sufficiently large, and the schedule includes a warmup period and a final decay to near-vanishing learning rate.
中文: 我们的结论是,对学习率时间表的选择大多无关紧要,只要总的汇总学习率足够大,时间表包括一个暖和期和最后衰落到接近蒸发的学习率.
<a id="S0359"></a> Source: p.26 S0359
Original: 0.0010 0.0008 0.0006 0.0004 0.0002 0.0000 0 50000 100000 150000 200000 250000 Step etaR gninraeL 3.90 3.85 3.80 3.75 3.70 3.65 50 100 150 200 250 LR Summed Over Steps ssoL Figure 22 We test a variety of learning rate schedules including cosine decay, linear decay, as well as other faster/slower decays schedules on a 3 million parameter model, shown on the left.
中文: 0.0010 0.00008 0.00006 0.00004 0.00002 0.00000 500000 1000000 1500000 20000000 2500000 Step etaR gninraeL 3.90 3.85 3.80 3.75 3.70 3.65 50 100 150 200 250 LR 图22 我们测试各种学习速率表,包括余弦衰减,线性衰减,以及左侧显示的300万个参数模型上其他更快/更慢的衰减速率表.
<a id="S0360"></a> Source: p.26 S0360
Original: For these experiments we do not decay to zero, since we find that this tends to give a fixed improvement close to the end of training.
中文: 对于这些实验,我们并不衰减到零,因为我们发现这往往会给予接近训练结束的固定改进.
<a id="S0361"></a> Source: p.26 S0361
Original: We find that, as long as the learning rate is not too small and does not decay too quickly, performance does not depend strongly on learning rate.
中文: 我们发现,只要学习率不太小,也不要太快地衰败,表现就不会强烈地依赖于学习率.
<a id="S0362"></a> Source: p.26 S0362
Original: Run-to-run variation is at the level of 0.05 in the loss, so averaging multiple runs is necessary to validate performance changes smaller than this level. 6 5 4 3 2 104 105 106 107 108 109 Parameters (non-embedding) )ecnegrevnoc ta( ssoL tseT L = (N/8.8 1013) 0.076 L = 0.25log(N/7.1 1012) Figure 23 The trend for performance as a function of parameter count, L(N ), is fit better by a power law than by other functions such as a logarithm at a qualitative level. schedules appear to be statistical noise, and provide a rough gauge for the scale of variation between different training runs.
中文: 运行到运行的变异在损失中为0.05级,因此平均多跑对验证低于这一级的性能变化是必要的. 6 5 4 3 2 104 105 106 107 108 109 参数(非嵌入) ecnegrevnoc ta(ssoL tseT L = (N/8 1013) 0.076 L = 6.25log(N/7.1 1012) 图23 作为参数计数(L(N))函数的性能趋势比质量级对数等其他函数更适合权力法. 时间表似乎是统计噪音,为不同培训活动之间的差别提供了粗略的衡量尺度。
<a id="S0363"></a> Source: p.26 S0363
Original: Experiments on larger models suggest that the variation in the final test loss between different random seeds is roughly constant in magnitude for different model sizes.
中文: 对更大型模型的实验表明,不同随机种子之间最终测试损失的变异,对于不同的模型大小来说,大致是不变的.
<a id="S0364"></a> Source: p.26 S0364
Original: We found that larger models require a smaller learning rate to prevent divergence, while smaller models can tolerate a larger learning rate.
中文: 我们发现,较大的模型需要更小的学习率来防止差异,而较小的模型可以容忍更大的学习率.
<a id="S0365"></a> Source: p.26 S0365
Original: To implement this, the following rule of thumb was used for most runs: LR(N ) ≈ 0.003239 + −0.0001395 log(N ) (D.1) We expect that this formula could be improved.
中文: 为了执行这一点,在大多数运行中使用了以下拇指规则: LR(N) + 0.003239 + - 0.0001395 log(N)(D.1) 我们期望这个公式可以改进.
<a id="S0366"></a> Source: p.26 S0366
Original: There may be a dependence on network width, likely set by the initialization scale.
中文: 可能会依赖网络宽度,可能由初始化尺度所设定.
<a id="S0367"></a> Source: p.26 S0367
Original: The formula also breaks down for N > 1010 parameters.
中文: 该公式还分解了N > 1010参数.
<a id="S0368"></a> Source: p.26 S0368
Original: Nevertheless, we found that it works sufficiently well for the models we considered. D.7 Fit Details and Power Law Quality We experimented with a number of functional forms for the fits to L(N ), L(C), and L(D); the power-law fits were qualitatively much more accurate than other functions such as logarithms (see Figure 23).
中文: 然而,我们发现,这对我们所考虑的模式来说是相当有效的。 D.7 适应性细节和权力法质量 我们试验了一些适合L(N)、L(C)和L(D)的功能形式;与对数等其他功能相比,电能法在质量上更为准确(见图23)。
<a id="S0369"></a> Source: p.26 S0369
Original: For L(C), we do not include small models with only 1 layer in the fit, as the transition from 1 to 2 layers causes a noticable lump in the data.
中文: 对于L(C),我们不包括只有一层相合的小模型,因为从一层到二层的过渡导致数据中出现可注意的整块.
<a id="S0370"></a> Source: p.26 S0370
Original: For L(N ) we also do not include very small models with only 1 layer in the fit, and we exclude the largest models that have not trained fully to convergence.
中文: 对于L(N),我们也不包括只有一层相合的非常小的模型,我们排除了没有经过充分训练来趋同的最大模型.
<a id="S0371"></a> Source: p.26 S0371
Original: Fit parameters change marginally if we do include them, and the trend extrapolates well in both directions regardless. D.8 Generalization and Architecture In figure 24 we show that generalization to other data distributions does not depend on network depth when we hold the total parameter count fixed.
中文: 如果我们确实列入适切参数,那么适切参数就略有变化,而且无论如何,趋势在两个方向上都推断良好。 D.8 一般化和建筑 在图24中,我们显示,当我们保留总参数数时,对其他数据分布的概括并不取决于网络深度。
<a id="S0372"></a> Source: p.26 S0372
Original: It seems to depend only on the performance on the training distribution. 26
中文: 这似乎只取决于培训的分布情况。 第26条
<a id="S0373"></a> Source: p.27 S0373
Original: 2.8 2.7 2.6 2.5 2.4 2.3 101 102 Depth ssoL tseT Wikipedia Books Internet Books Common Crawl WebText2 (Train) WebText2 (Test) Figure 24 We show evaluations on a series of datasets for models with approximately 1.5 Billion parameters.
中文: 2.8 2.7 2.5 2.4 2.3 101 102 Depth ssoL tseT Wikipedia Books Internet Books Clawl WebText2(Train) WebText2(Test) 图24 我们展示了对一系列具有约1.5亿个参数的模型数据集的评价.
<a id="S0374"></a> Source: p.27 S0374
Original: We observe no effect of depth on generalization; generalization performance depends primarily on training distribution performance.
中文: 我们没有看到深度对一般化的影响;一般化的表现主要取决于培训分配的表现。
<a id="S0375"></a> Source: p.27 S0375
Original: The 12-layer model overfit the Internet Books dataset and we show the early-stopped performance; we have not seen this surprising result in other experiments.
中文: 12层的模型过度匹配了互联网图书数据集,我们展示了早期停止的性能;我们没有看到这种令人惊讶的结果在其他实验中.
<a id="S0376"></a> Source: p.27 S0376
Original: List of Figures 1 Summary of simple power laws. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Illustration of sample efficiency and compute efficiency. . . . . . . . . . . . . . . . . . . . . 4 3 How to scale up model size, batch size, and serial steps . . . . . . . . . . . . . . . . . . . . 4 4 Performance when varying model and data size, or model and training steps, simultaneously 5 5 Weak dependence of performance on hyperparameter tuning . . . . . . . . . . . . . . . . . 8 6 Comparison of performance trend when including or excluding embeddings . . . . . . . . . 8 7 LSTM and Transformer performance comparison . . . . . . . . . . . . . . . . . . . . . . . 9 8 Generalization to other test datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 9 Universality of overfitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 10 Critical batch size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 11 Performance versus compute budget or number of parameter updates . . . . . . . . . . . . . 14 12 Training on suboptimal models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 13 Comparison between empirical and adjusted compute trends . . . . . . . . . . . . . . . . . 15 14 Optimal model size and serial number of steps versus compute budget . . . . . . . . . . . . 16 15 Contradiction between compute and data trends . . . . . . . . . . . . . . . . . . . . . . . . 17 16 Early stopping lower bound and training curves for overfit models . . . . . . . . . . . . . . 23 17 Universal transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 18 Batch size scans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 19 Another look at sample efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 20 Power-law dependence of performance on position in context . . . . . . . . . . . . . . . . . 25 21 Performance at different context positions versus model size . . . . . . . . . . . . . . . . . 25 22 Learning rate schedule scan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 23 Comparison of Power-Law and Logarithmic Fits . . . . . . . . . . . . . . . . . . . . . . . 26 24 Generalization versus depth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 27
中文: 图1 简单功率定律简表 4 3 如何扩大模型大小,批量大小和相继步骤 3 4 性能 当模型和数据大小不同时,或模型和训练步骤,同时5 5 依赖超等分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 4 分解性能 分解性能 4 分解性能 4 分解性能 4 分解性能 分解性能 分解性能 4 分解性能 4 分解性能 8 分解性能 8 分解性能 分解性能 分化 分解性能 8 分解性能 1110 关键批量尺寸 11 性能对计算预算或参数更新数 B 14 12 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期训练 分期 . . . . . . . . . .
<a id="S0377"></a> Source: p.28 S0377
Original: List of Tables 1 Parameter and compute counts for Transformer . . . . . . . . . . . . . . . . . . . . . . . . 7 2 Fits to L(N, D) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Fits to L(N, S) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4 Key trend equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 5 Key parameters to trend fits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 6 Trends for compute-efficient training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 References [ACDE12] Eduardo G Altmann, Giampaolo Cristadoro, and Mirko Degli Esposti.
中文: 表1 参数和计算法计算出变相器. .
<a id="S0378"></a> Source: p.28 S0378
Original: On the origin of longrange correlations in texts.
中文: 关于文本中远程关联的起源.
<a id="S0379"></a> Source: p.28 S0379
Original: Proceedings of the National Academy of Sciences, 109(29):11582– 11587, 2012. 25 [AS17] Madhu S.
中文: 国家科学院学报, 109(29): 11582–11587, 2012. 25 [AS17] Madhu S.
<a id="S0380"></a> Source: p.28 S0380
Original: High-dimensional dynamics of generalization error in neural networks. arXiv, 2017, 1710.03667. 11, 18, 22 [BB01] Michele Banko and Eric Brill.
中文: 神经网络中通化误差的高维动态. arXiv, 2017, 171.03667. 11, 18, 22 [BB01] 米凯莱·班克和埃里克·布里尔.
<a id="S0381"></a> Source: p.28 S0381
Original: Scaling to very very large corpora for natural language disambiguation.
中文: 放大到非常大的蝎子 自然语言的分化。
<a id="S0382"></a> Source: p.28 S0382
Original: In Proceedings of the 39th annual meeting on association for computational linguistics, pages 26–33.
中文: 《计算语言学协会第三十九届年会议事录》,第26-33页。
<a id="S0383"></a> Source: p.28 S0383
Original: Association for Computational Linguistics, 2001. 18 [BHMM18] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.
中文: 计算语言学协会,2001. 18 [BHMM18]. 米哈伊尔·贝尔金,丹尼尔·许,西也安·马和苏米克·曼达尔.
<a id="S0384"></a> Source: p.28 S0384
Original: Reconciling modern machine learning and the bias-variance trade-off. arXiv, 2018, 1812.11118. 18 [Bia12] GÊrard Biau.
中文: 调和现代机器学习与偏差-变相取舍. arXiv, 2018, 1812.11118. 18 [Bia12] GÊrard Biau.
<a id="S0385"></a> Source: p.28 S0385
Original: Journal of Machine Learning Research, 13(Apr):1063–1095, 2012. 18 [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
中文: "机器学习研究杂志",13 (Apr: 1063–1095) 2012. 18 [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
<a id="S0386"></a> Source: p.28 S0386
Original: Generating long sequences with sparse transformers.
中文: 用稀疏的变压器生成长序.
<a id="S0387"></a> Source: p.28 S0387
Original: URL http://arxiv.org/ abs/1904.10509. 19 [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
中文: URL http://arxiv.org/ abs/1904.10509. 19 [DCLT18]. 雅各布·德夫林,明-魏昌,克通·李克通,克丽斯蒂娜·图塔诺娃.
<a id="S0388"></a> Source: p.28 S0388
Original: Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805. 2 [DGV+18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.
中文: 伯特:深向双向变速器进行语言理解预训, 2018, arXiv: 181.04805. 2 [DGV+18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.
<a id="S0389"></a> Source: p.28 S0389
Original: URL http://arxiv.org/ abs/1807.03819. 6, 9, 23, 24 [EP94] Werner Ebeling and Thorsten Pöschel.
中文: URL http://arxiv.org/abs/1807.03819. 6,9,23,24 [EP94] 维尔纳·埃贝林和索尔斯滕·波舍尔.
<a id="S0390"></a> Source: p.28 S0390
Original: Entropy and long-range correlations in literary english.
中文: 文学英语中的Entropy和远程相关.
<a id="S0391"></a> Source: p.28 S0391
Original: EPL (Europhysics Letters), 26(4):241, 1994. 25 [Fou] The Common Crawl Foundation.
中文: ESL (Europhysics Letters), 26(4):241, 1994. 25 [福] 常见爬行基金会.
<a id="S0392"></a> Source: p.28 S0392
Original: URL http://commoncrawl.org. 7 [GARD18] Guy Gur-Ari, Daniel A.
中文: URL http://commoncrawl.org. 7 [GARD18] 盖·古尔-阿里,丹尼尔·A.
<a id="S0393"></a> Source: p.28 S0393
Original: Gradient descent happens in a tiny subspace. 2018, arXiv:1812.04754. 18 [GJS+19] Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d’Ascoli, Giulio Biroli, Clément Hongler, and Matthieu Wyart.
中文: 渐入低地发生在一个很小的子空间. 2018, arXiv: 1812.04754. 18 [GJS+19] 马里奥·盖格, 阿瑟·贾克特, 斯特凡诺·斯皮格勒, 弗兰克·加布里埃尔, 勒文特·萨贡, 斯特凡·德·阿斯科利, 朱利奥·比罗利, Clément Hongler, 和 马特修·维雅特.
<a id="S0394"></a> Source: p.28 S0394
Original: Scaling description of generalization with number of parameters in deep learning. arXiv, 2019, 1901.01608. 18 [GKX19] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.
中文: 以深度学习中的参数数来放大对通论的描述. arXiv, 2019, 1901.01608. 18 [GKX19] Behrooz Ghorbani, Shankar Krishnan, 和相晓.
<a id="S0395"></a> Source: p.28 S0395
Original: An investigation into neural net optimization via hessian eigenvalue density.
中文: 调查神经网通过黑森等元值密度优化.
<a id="S0396"></a> Source: p.28 S0396
Original: URL http://arxiv.org/abs/1901.10159. 18 [Goo01] Joshua Goodman. A bit of progress in language modeling.
中文: URL http://arxiv.org/abs/1901.10159. 18 [Goo01] 约书亚·古德曼. 语言建模方面稍有进展.
<a id="S0397"></a> Source: p.28 S0397
Original: URL http://arxiv.org/abs/cs.CL/0108005. 18 [GRK17] Scott Gray, Alec Radford, and Diederik P Kingma.
中文: URL http://arxiv.org/abs/cs.CL/010800. 18 [GRK17] Scott Gray, Alec Radford, and Diederik P Kingma.
<a id="S0398"></a> Source: p.28 S0398
Original: Gpu kernels for block-sparse weights. openai.com, 2017. 19 [HAD19] Joel Hestness, Newsha Ardalani, and Gregory Diamos.
中文: Gpu内核用于块-sparse重量. openai.com, 2017. 19 [HAD19] 乔尔·赫斯特内斯, Newsha Ardalani, and Gregory Diamos.
<a id="S0399"></a> Source: p.28 S0399
Original: Beyond human-level accuracy: Computational challenges in deep learning.
中文: 超越人类层面的准确性:在深度学习中的计算挑战.
<a id="S0400"></a> Source: p.28 S0400
Original: In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming, PPoPP ’19, pages 1–14, New York, NY, USA, 2019.
中文: 《平行方案拟订原则和实践第24次专题讨论会记录》,POPP ' 19, 第1-14页,纽约,美国纽约,2019年。
<a id="S0401"></a> Source: p.29 S0401
Original: [HCC+18] Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V.
中文: [HCC+18] 黄延平,永隆成,陈德豪,李孝宗,吉泉·恩吉安,高克五.
<a id="S0402"></a> Source: p.29 S0402
Original: Gpipe: Efficient training of giant neural networks using pipeline parallelism.
中文: Gpipe:利用管道平行性高效地训练巨型神经网络.
<a id="S0403"></a> Source: p.29 S0403
Original: URL http://arxiv.org/abs/1811.06965. 19 [HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.
中文: URL: http://arxiv.org/abs/1811.06965. 19 [HNA+17]. 乔尔·赫斯特内斯,沙兰·纳朗,纽沙·阿尔达拉尼,格雷戈里·迪亚莫斯,希宇俊,哈桑·基安内贾德,Md.
<a id="S0404"></a> Source: p.29 S0404
Original: Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
中文: 穆斯多法·阿里·帕特瓦里,杨洋和周艳琪.
<a id="S0405"></a> Source: p.29 S0405
Original: Deep learning scaling is predictable, empirically, 2017, 1712.00409. 18 [JGH18] Arthur Jacot, Franck Gabriel, and Clément Hongler.
中文: 深层学习缩放是可以预测的,实证的,2017年,1712.00409. 18 [JGH18] Arthur Jacot, Frank Gabriel, and Clément Hongler.
<a id="S0406"></a> Source: p.29 S0406
Original: Neural tangent kernel: Convergence and generalization in neural networks.
中文: 神经正核:神经网络中的同源和通化.
<a id="S0407"></a> Source: p.29 S0407
Original: In Advances in neural information processing systems, pages 8571–8580, 2018. 18 [KB14] Diederik P.
中文: In Advances in 神经信息处理系统, 第8571–8580页, 2018. 18 [KB14] Diederik P.
<a id="S0408"></a> Source: p.29 S0408
Original: Adam: A method for stochastic optimization, 2014, 1412.6980. 7 [Kom19] Aran Komatsuzaki.
中文: Adam: stochastic 优化的一种方法, 2014, 1412.6980. 7 [Kom19] Aran Komatsuzaki.
<a id="S0409"></a> Source: p.29 S0409
Original: One epoch is all you need, 2019, arXiv:1906.06669. 18 [KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E.
中文: 一个时代就是你所需要的, 2019, arXiv:1906.06669. 18 [KSH12] Alex Krizhevsky, Ilya Sutskever, 和 Geoffrey E.
<a id="S0410"></a> Source: p.29 S0410
Original: Imagenet classification with deep convolutional neural networks.
中文: 影像网分类与深演神经网络.
<a id="S0411"></a> Source: p.29 S0411
Original: In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA, 2012.
中文: 在第25届神经信息处理系统国际会议记录 -- -- 第1卷,NIPS ' 12, 第1097-1105页,美国,2012年。
<a id="S0412"></a> Source: p.29 S0412
Original: URL http://dl.acm.org/citation.cfm?id=2999134.2999257. 19 [LCG+19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.
中文: URL http://dl.acm.org/引用.cfm?id=2999134.2999257. 19 [LCG+19] 曾中兰,明达陈,塞巴斯蒂安·古德曼,凯文·金佩尔,皮尤什·夏尔马和拉都·索里克特.
<a id="S0413"></a> Source: p.29 S0413
Original: Albert: A lite bert for self-supervised learning of language representations, 2019, 1909.11942. 9 [LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
中文: 艾伯特:自我监督学习语言表现的Lite bert, 2019, 1909.11942. 9 [LOG+19] 刘燕汉, 弥勒·奥特, 纳南·高雅尔, 静菲·杜, 万达·乔希, 丹奇·陈, 奥梅尔·利维, 迈克·刘易斯, 卢克·泽特勒莫耶, 和维塞林·斯托亚诺夫.
<a id="S0414"></a> Source: p.29 S0414
Original: Roberta: A robustly optimized BERT pretraining approach.
中文: 罗伯塔:强力优化BERT预训方式.
<a id="S0415"></a> Source: p.29 S0415
Original: URL http://arxiv.org/abs/ 1907.11692. 2 [LSP+18] Peter J.
中文: URL http://arxiv.org/abs/1907.11692. 2 [LSP+18] 彼得·J.
<a id="S0416"></a> Source: p.29 S0416
Original: Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.
中文: 刘,穆罕默德·萨利赫,艾蒂安·波特,本·古德里奇,瑞安·塞帕西,卢卡斯克·凯泽尔,和诺姆·沙泽尔.
<a id="S0417"></a> Source: p.29 S0417
Original: Generating wikipedia by summarizing long sequences. arXiv:1801.10198 [cs], 2018, 1801.10198.
中文: 通过总结长序来生成维基百科. arXiv:1801.10198 [cs], 2018,1801.10198.
<a id="S0418"></a> Source: p.29 S0418
Original: URL http://arxiv.org/abs/1801.10198. 2, 6 [LT16] Henry W Lin and Max Tegmark.
中文: URL http://arxiv.org/abs/1801.10198. 2,6 [LT16] 亨利·W·林和马克斯·特格马克.
<a id="S0419"></a> Source: p.29 S0419
Original: Criticality in formal languages and statistical physics. arXiv preprint arXiv:1606.06737, 2016. 25 [LXS+19] Jaehoon Lee, Lechao Xiao, Samuel S.
中文: 正式语言和统计物理的临界性. arXiv preprint arXiv:1606.06737, 2016. 25 [LXS+19] 李建勋, 李绍晓, Samuel S.
<a id="S0420"></a> Source: p.29 S0420
Original: Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington.
中文: 肖恩霍尔茨,亚萨曼·巴赫里,罗曼·诺瓦克,贾夏·索尔-迪克斯坦,和杰弗里·平宁顿.
<a id="S0421"></a> Source: p.29 S0421
Original: Wide neural networks of any depth evolve as linear models under gradient descent, 2019, arXiv:1902.06720. 18 [MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team.
中文: 任何深度的广神经网络会演化为梯度下降下的线性模型, 2019, arXiv:1902.06720. 18 [MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, 和 OpenAI Dota Team.
<a id="S0422"></a> Source: p.29 S0422
Original: An empirical model of large-batch training, 2018, arXiv:1812.06162. 3, 5, 6, 12, 13, 21 [Pap18] Vardan Papyan.
中文: 2018年大批量训练经验模型,arXiv:1812.06162. 3,5,6,12,13,21[Pap18] Vardan Papyan.
<a id="S0423"></a> Source: p.29 S0423
Original: The full spectrum of deep net hessians at scale: Dynamics with sample size.
中文: 规模深网黑森全谱:有样本大小的动态.
<a id="S0424"></a> Source: p.29 S0424
Original: URL http://arxiv.org/abs/1811.07062. 18 [RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
中文: URL http://arxiv.org/abs/1811.07062. 18 [RNSS18] 阿列克·拉德福德,克思克·纳拉西姆汉,蒂姆·萨利曼斯和伊利亚·苏特斯克韦.
<a id="S0425"></a> Source: p.29 S0425
Original: Improving language understanding by generative pre-training.
中文: 通过基因预训提高语言理解.
<a id="S0426"></a> Source: p.29 S0426
Original: URL https://s3-us-west-2. amazonaws. com/openaiassets/research-covers/languageunsupervised/language understanding paper. pdf, 2018. 2, 6 [RRBS19a] Jonathan S.
中文: URL https://s3-us-west-2. amazonaws. com/openaiassets/research-covers/language unsupervised/ 语言理解论文. pdf, 2018. 2, 6 [RRBS19a] Jonathan S.
<a id="S0427"></a> Source: p.29 S0427
Original: Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, 1909.12673. 18 [RRBS19b] Jonathan S.
中文: 罗森费尔德,埃米尔·罗森费尔德,约纳坦·贝林科夫,和尼尔·沙维特. 2019,1909.12673. 18 [RRBS19b] Jonathan S.
<a id="S0428"></a> Source: p.29 S0428
Original: Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019, arXiv:1909.12673. 18 [RSR+19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J.
中文: 罗森费尔德,埃米尔·罗森费尔德,约纳坦·贝林科夫,和尼尔·沙维特. 2019年跨平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平
<a id="S0429"></a> Source: p.29 S0429
Original: Exploring the limits of transfer learning with a unified text-to-text transformer, 2019, arXiv:1910.10683. 2 [RWC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
中文: 探索采用统一文字变换器进行转录学习的极限, 2019, arXiv:1910.10683. 2 [RWC+19] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
<a id="S0430"></a> Source: p.29 S0430
Original: Language models are unsupervised multitask learners. openai.com, 2019. 2, 5, 6, 7, 8 [SCP+18] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman.
中文: 语言模型是无监督的多任务学习者. openai.com, 2019. 2, 5, 6, 7, 8 [SCP+18] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, Hyuk Jong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, 和布莱克·赫特曼.
<a id="S0431"></a> Source: p.29 S0431
Original: Mesh-tensorflow: Deep learning for supercomputers, 2018, 1811.02084. 19 [SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch.
中文: Mesh-tensorflow: 为超级计算机深层学习, 2018, 1811.02084. 19 [SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch.
<a id="S0432"></a> Source: p.29 S0432
Original: Neural machine translation of rare words with subword units.
中文: 神经机能翻译有子词单位的稀有词.
<a id="S0433"></a> Source: p.30 S0433
Original: Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E.
中文: Shalue, Jaehoon Lee, 乔·安多尼, Jascha Sohl-Dickstein, Roy Frostig, 和乔治·E.
<a id="S0434"></a> Source: p.30 S0434
Original: Measuring the effects of data parallelism on neural network training, 2018, arXiv:1811.03600. 12 [SS18] Noam Shazeer and Mitchell Stern.
中文: 测量数据平行主义对神经网络训练的影响, 2018, arXiv:1811.03600. 12 [SS18] Noam Shazeer and Mitchell Stern.
<a id="S0435"></a> Source: p.30 S0435
Original: Adafactor: Adaptive learning rates with sublinear memory cost.
中文: 助推:适应性学习率与子线内存成本.
<a id="S0436"></a> Source: p.30 S0436
Original: URL http://arxiv.org/abs/1804.04235. 7 [THK18] Stefan Thurner, Rudolf Hanel, and Peter Klimek.
中文: URL http://arxiv.org/abs/1804.04235. 7 [THK18]斯特凡·瑟纳,鲁道夫·汉内尔,和彼得·克利梅克.
<a id="S0437"></a> Source: p.30 S0437
Original: Introduction to the theory of complex systems.
中文: 复杂系统理论入门.
<a id="S0438"></a> Source: p.30 S0438
Original: Oxford University Press, 2018. 18 [TL19] Mingxing Tan and Quoc V.
中文: 牛津大学出版社, 2018. 18 [TL19] (英语). Mingxing Tan and Quoc V.
<a id="S0439"></a> Source: p.30 S0439
Original: Efficientnet: Rethinking model scaling for convolutional neural networks.
中文: 高效网:再思考进化神经网络模型缩放.
<a id="S0440"></a> Source: p.30 S0440
Original: URL http://arxiv.org/abs/1905. 11946. 18 [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.
中文: URL http://arxiv.org/abs/1905. 11946. 18 [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser,和 Illia Polosukhin.
<a id="S0441"></a> Source: p.30 S0441
Original: Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998–6008.
中文: Garnett,编辑,神经信息处理系统的进步 30, 第5998-6008页.
<a id="S0442"></a> Source: p.30 S0442
Original: URL http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf. 2, 6 [VWB16] Andreas Veit, Michael Wilber, and Serge Belongie.
中文: URL http://papers.nips.cc/paper/7181-attention-is-all-you- need.pdf. 2, 6 [VWB16] 安德烈亚斯·韦特,迈克尔·威尔伯和塞尔日·鲍利.
<a id="S0443"></a> Source: p.30 S0443
Original: Residual networks behave like ensembles of relatively shallow networks, 2016, arXiv:1605.06431. 8, 18 [Was06] Larry Wasserman.
中文: 残余网络的行为类似相对浅层网络的综艺, 2016, arXiv:1605.06431. 8, 18 [Was06] Larry Wasserman.
<a id="S0444"></a> Source: p.30 S0444
Original: Springer Science & Business Media, 2006. 18 [WPN+19] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
中文: Springer Science & Business Media, 2006. 18 [WPN+19] (英语). Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
<a id="S0445"></a> Source: p.30 S0445
Original: Superglue: A stickier benchmark for general-purpose language understanding systems, 2019, 1905.00537. 2 [WRH17] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.
中文: Superglue:通用语言理解系统的一个粘接基准,2019年,1905.00537. 2 [WRH17] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.
<a id="S0446"></a> Source: p.30 S0446
Original: Growing a brain: Fine-tuning by increasing model capacity. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi:10.1109/cvpr.2017.323. 19 [WYL19] Wei Wen, Feng Yan, and Hai Li.
中文: 培养人才:通过提高模型容量来精细调整. 2017 IEEE Conference on Computer Vision and Pattle Conference (CVPR), Jul 2017. doi:10.1109/cvpr. 2017.323. 19 [WYL19] 魏文,冯燕,和海丽.
<a id="S0447"></a> Source: p.30 S0447
Original: Autogrow: Automatic layer growing in deep convolutional networks, 2019, 1906.02909. 19 [YDY+19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V.
中文: 自动化:深层革命网络中生长的自动地层, 2019, 1906.02909. 19 [YDY+19] 杨志林, Ziham Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V.
<a id="S0448"></a> Source: p.30 S0448
Original: Xlnet: Generalized autoregressive pretraining for language understanding, 2019, arXiv:1906.08237. 2 [ZK16] Sergey Zagoruyko and Nikos Komodakis.
中文: Xlnet:通用自旋预训以通晓语言,2019年,arXiv:1906.08237. 2 [ZK16]谢尔盖·扎戈鲁伊科和尼科斯·克莫多基斯.
<a id="S0449"></a> Source: p.30 S0449
Original: Procedings of the British Machine Vision Conference 2016, 2016. doi:10.5244/c.30.87. 18 [ZKZ+15] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.
中文: 2016. doi:10.5244/c 30.87. 18 [ZKZ+15]. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, 和 Sanja Fidler.
<a id="S0450"></a> Source: p.30 S0450
Original: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi:10.1109/iccv.2015.11. 7 [ZLN+19] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E.
中文: 校正书籍和电影:通过观看电影和阅读书籍,走向故事般的视觉解释. 2015. IEEE International Conference on Computer Vision (ICCV), Dec 2015. doi:10.1109/iccv.2015.11. 7 [ZLN+19] 郭东章,Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E.
<a id="S0451"></a> Source: p.30 S0451
Original: Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model.
中文: 哪种算法选择对哪个批量大小很重要? 一个吵闹的四面体模型的洞察力
<a id="S0452"></a> Source: p.30 S0452
Original: URL http://arxiv.org/abs/1907.04164. 12, 18 30
中文: URL http://arxiv.org/abs/1907.04164. 12,18 30 (英语).