Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de - 中英文对照
translated: 2026-07-16
title: "Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de" aliases: - "Chinchilla" - "arXiv:2203.15556" source: "https://arxiv.org/abs/2203.15556" arxiv: "2203.15556" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning
Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Training Compute-Optimal Large Language Models Jordan Hoffmann★, Sebastian Borgeaud★, Arthur Mensch★, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W.
中文: 训练计算-奥普蒂玛尔大语言模型 乔丹·霍夫曼-,塞巴斯蒂安·博尔高德-,亚瑟·门施-,埃琳娜·布恰特斯卡娅,特雷弗·蔡,以利萨·卢瑟福,迭戈·德·拉斯·卡萨斯,丽莎·安妮·亨德里克斯,约翰内斯·韦尔布尔,艾丹·克拉克,汤姆·亨尼根,埃里克·诺兰,凯蒂·米利坎,乔治·范登德瑞斯切,博格丹·达莫克,奥雷利亚·盖伊,西蒙·奥辛多,凯伦·西蒙扬,埃里希·埃尔森,杰克·W.
<a id="S0002"></a> Source: p.1 S0002
Original: Rae, Oriol Vinyals and Laurent Sifre★ ★Equal contributions We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget.
中文: Rae、Oriol Vinyals和Laurent Sifre 我们调查在一定的计算预算下培训变压器语言模型的最佳模型大小和符号数量。
<a id="S0003"></a> Source: p.1 S0003
Original: We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant.
中文: 我们发现,目前大型语言模型培训严重不足,这是最近注重在保持培训数据数量不变的同时扩大语言模型的结果。
<a id="S0004"></a> Source: p.1 S0004
Original: By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.
中文: 通过培训400多个语言模型,从7 000万到160多亿参数的50至5 000亿令牌,我们发现,就计算最佳培训而言,模型大小和培训标志的数目应同样加以扩大:模型大小每翻一番,培训标志的数目也应增加一倍。
<a id="S0005"></a> Source: p.1 S0005
Original: We test this hypothesis by training a predicted computeoptimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data.
中文: 我们通过训练一个预测的计算最佳模型Chinchilla来测试这个假说,这个模型使用与Gopher相同的计算预算,但有70B参数和4×更多的数据.
<a id="S0006"></a> Source: p.1 S0006
Original: Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks.
中文: Chinchilla在Gopher(280B),GPT-3(175B),侏罗纪-1(178B),以及威震天-图灵NLG(530B)等一系列下游评价任务上,都得到统一和显著的超越.
<a id="S0007"></a> Source: p.1 S0007
Original: This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage.
中文: 这也意味着钦奇拉在微调和推论时使用的计算要少得多,大大地方便了下游的使用.
<a id="S0008"></a> Source: p.1 S0008
Original: As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher. 1.
中文: 作为亮点,Chinchilla在MMLU基准上达到了最先进的平均精度为67.5%,比Gopher改进了7%以上. 1. 联合国
<a id="S0009"></a> Source: p.1 S0009
Original: Introduction Recently a series of Large Language Models (LLMs) have been introduced (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Thoppilan et al., 2022), with the largest dense language models now having over 500 billion parameters.
中文: 最近推出了一系列大型语言模型(LLMs) (Brown等,2020年;Lieber等,2021年;Rae等,2021年;Smith等,2022年;Thoppilan等,2022年),目前最大的密集语言模型拥有超过5000亿个参数.
<a id="S0010"></a> Source: p.1 S0010
Original: These large autoregressive transformers (Vaswani et al., 2017) have demonstrated impressive performance on many tasks using a variety of evaluation protocols such as zero-shot, few-shot, and fine-tuning.
中文: 这些大型自转变速器(Vaswani等,2017年)运用了"零发","少发","微调"等多种评价协议,在许多任务上表现出了令人印象深刻的性能.
<a id="S0011"></a> Source: p.1 S0011
Original: The compute and energy cost for training large language models is substantial (Rae et al., 2021; Thoppilan et al., 2022) and rises with increasing model size.
中文: 培训大型语言模型的计算和能耗相当高(Rae等人,2021年;Thoppilan等人,2022年),随着模型尺寸的增加而上升.
<a id="S0012"></a> Source: p.1 S0012
Original: In practice, the allocated training compute budget is often known in advance: how many accelerators are available and for how long we want to use them.
中文: 实际上,分配的培训计算预算往往事先知道:有多少加速器可供使用,以及我们希望使用多久。
<a id="S0013"></a> Source: p.1 S0013
Original: Since it is typically only feasible to train these large models once, accurately estimating the best model hyperparameters for a given compute budget is critical (Tay et al., 2021).
中文: 由于通常只有一次训练这些大型模型才可行,准确估计特定计算预算的最佳模型超参数至关重要(Tay等人,2021年)。
<a id="S0014"></a> Source: p.1 S0014
Original: Kaplan et al. (2020) showed that there is a power law relationship between the number of parameters in an autoregressive language model (LM) and its performance.
中文: Kaplan等 (2020年) 显示自旋语言模型(LM)中的参数数量与其性能之间有权力法关系.
<a id="S0015"></a> Source: p.1 S0015
Original: As a result, the field has been training larger and larger models, expecting performance improvements.
中文: 因此,外地一直在培训规模更大、规模更大的模式,期望业绩得到改善。
<a id="S0016"></a> Source: p.1 S0016
Original: One notable conclusion in Kaplan et al. (2020) is that large models should not be trained to their lowest possible loss to be compute optimal.
中文: Kaplan等人(2020年)的一个显著结论是,不应对大型模型进行尽可能低的损失程度的培训,以便进行最佳计算。
<a id="S0017"></a> Source: p.1 S0017
Original: Whilst we reach the same conclusion, we estimate that large models should be trained for many more training tokens than recommended by the authors.
中文: 虽然我们得出了同样的结论,但我们估计,大型型号的培训应比作者推荐的要多得多。
<a id="S0018"></a> Source: p.1 S0018
Original: Specifically, given a 10× increase computational budget, they suggests that the size of the model should increase 5.5× while the number of training tokens should only increase 1.8×.
中文: 具体地说,由于计算预算增加了10××,他们建议模型的尺寸应该增加5.5×,而训练符的数量应该只增加1.8×.
<a id="S0019"></a> Source: p.1 S0019
Original: Instead, we find that model size and the number of training tokens should be scaled in equal proportions.
中文: 相反,我们认为,模型的规模和训练标志的数量应该以同样的比例加以扩大。
<a id="S0020"></a> Source: p.1 S0020
Original: Following Kaplan et al. (2020) and the training setup of GPT-3 (Brown et al., 2020), many of the recently trained large models have been trained for approximately 300 billion tokens (Table 1), in line with the approach of predominantly increasing model size when increasing compute.
中文: 继Kaplan等人(2020年)和GPT-3的培训设置(Brown等人,2020年)之后,许多最近培训的大型模型都接受了大约3000亿令牌的培训(表1),这符合在计算量增加时主要增加模型规模的方法.
<a id="S0021"></a> Source: p.1 S0021
Original: Corresponding authors: {jordanhoffmann|sborgeaud|amensch|sifre}@deepmind.com © 2023 DeepMind.
中文: 对应作者: {jordanhoffmann|sborgeaud|amensch|sifre}@deepmind.com} 2023 DeepMind.
<a id="S0022"></a> Source: p.1 S0022
Original: All rights reserved 2202 raM 92 ]LC.sc[ 1v65551.3022:viXra
中文: 版权所有:2202 RAM 92]LC.sc [1v65551.3022:viXra
<a id="S0023"></a> Source: p.2 S0023
Original: 1T 100B 10B 1.0B 100M 10M 1017 1019 1021 1023 1025 FLOPs sretemaraP Approach 1 Approach 2 Approach 3 Kaplan et al (2020) Chinchilla (70B) Gopher (280B) GPT-3 (175B) Megatron-Turing NLG (530B) Figure 1 | Overlaid predictions.
中文: 1T 100B 10B 1.0B 100M 10M 1017 1019 1021 1023 FLOPs sretemaraP 方法 1 方法 2 方法 3 Kaplan等 (2020) Chinchilla (70B) Gopher (280B) GPT-3 (175B) 威震天-图灵 NLG (530B) 图1 + 重叠预测.
<a id="S0024"></a> Source: p.2 S0024
Original: We overlay the predictions from our three different approaches, along with projections from Kaplan et al. (2020).
中文: 我们用我们三种不同方法的预测以及Kaplan等人(2020年)的预测加以叠加。
<a id="S0025"></a> Source: p.2 S0025
Original: We find that all three methods predict that current large models should be substantially smaller and therefore trained much longer than is currently done.
中文: 我们发现,所有三种方法都预测,目前的大型模型应大大地小一些,因此培训的时间要长得多。
<a id="S0026"></a> Source: p.2 S0026
Original: In Figure A3, we show the results with the predicted optimal tokens plotted against the optimal number of parameters for fixed FLOP budgets.
中文: 在图A3中,我们用预测的最佳符号,对照固定的FLOP预算的最佳参数,来显示结果。
<a id="S0027"></a> Source: p.2 S0027
Original: Chinchilla outperforms Gopher and the other large models (see Section 4.2).
中文: Chinchilla超越了Gopher和其他大型模型(见第4.2节)。
<a id="S0028"></a> Source: p.2 S0028
Original: In this work, we revisit the question: Given a fixed FLOPs budget, 1 how should one trade-off model size and the number of training tokens?
中文: 在这项工作中,我们重新讨论一个问题:鉴于固定的固定成本操作程序预算,一个权衡模型的规模和培训标志的数量如何?
<a id="S0029"></a> Source: p.2 S0029
Original: To answer this question, we model the final pre-training loss2 𝐿(𝑁, 𝐷) as a function of the number of model parameters 𝑁, and the number of training tokens, 𝐷.
中文: 为了回答这个问题,我们模拟最后的训练前损失2L(N,D),作为模型参数N的数量的函数,以及训练符的数量 D.
<a id="S0030"></a> Source: p.2 S0030
Original: Since the computational budget 𝐶 is a deterministic function FLOPs(𝑁, 𝐷) of the number of seen training tokens and model parameters, we are interested in minimizing 𝐿 under the constraint FLOPs(𝑁, 𝐷) = 𝐶: 𝑁 (𝐶), 𝐷 (𝐶) = argmin 𝐿(𝑁, 𝐷). (1) 𝑜𝑝𝑡 𝑜𝑝𝑡 𝑁,𝐷 s.t.
中文: 由于计算预算C是一种确定性函数FLOPs(N,D)所见训练符和模型参数的数量,我们有兴趣在约束FLOPs(N,D)=C:N(C),D(C)=argmin L(N,D)(1)选择选取N,D s.t.
<a id="S0031"></a> Source: p.2 S0031
Original: FLOPs(𝑁,𝐷)=𝐶 The functions 𝑁 (𝐶), and 𝐷 (𝐶) describe the optimal allocation of a computational budget 𝐶.
中文: FLOPs(N,D)=C 函数N(C)和D(C)描述计算预算C的最佳分配.
<a id="S0032"></a> Source: p.2 S0032
Original: We 𝑜𝑝𝑡 𝑜𝑝𝑡 empirically estimate these functions based on the losses of over 400 models, ranging from under 70M to over 16B parameters, and trained on 5B to over 400B tokens – with each model configuration trained for several different training horizons.
中文: 我们选择以经验为选择,根据400多个模型的损失来估计这些功能,从70M以下到16B以上参数,并接受5B到400B以上符号的培训 — — 每个模型配置都接受过若干不同培训视野的培训。
<a id="S0033"></a> Source: p.2 S0033
Original: Our approach leads to considerably different results than that of Kaplan et al. (2020).
中文: 我们的做法与Kaplan等人的做法(2020年)的结果大相径庭。
<a id="S0034"></a> Source: p.2 S0034
Original: We highlight our results in Figure 1 and how our approaches differ in Section 2.
中文: 我们在图1中强调了我们的成果,以及在第2节中我们的做法有何不同。
<a id="S0035"></a> Source: p.2 S0035
Original: Based on our estimated compute-optimal frontier, we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training on 4 times more tokens.
中文: 根据我们估计的计算 - 最佳边疆,我们预测,对于用来训练Gopher的计算预算来说,一个最佳模式应该小于4倍,同时是4倍于代币的训练.
<a id="S0036"></a> Source: p.2 S0036
Original: We verify this by training a more compute-optimal 70B model, called Chinchilla, on 1.4 trillion tokens.
中文: 我们通过在1.4万亿个符号上 训练一个更计算最优的70B模型来验证这一点。
<a id="S0037"></a> Source: p.2 S0037
Original: Not only does Chinchilla outperform its much larger counterpart, Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware.
中文: 钦奇拉不仅表现优于其更大的对口单位Gopher,而且其模型尺寸的缩小大大降低了推论成本并大大地方便了下游对更小的硬件的使用.
<a id="S0038"></a> Source: p.2 S0038
Original: The energy cost of a large language model is amortized through its usage for inference an fine-tuning.
中文: 大型语言模型的能耗通过它用来推论微调而得到摊还.
<a id="S0039"></a> Source: p.2 S0039
Original: The benefits of a more optimally trained smaller model, therefore, extend beyond the immediate benefits of its improved performance. 1For example, knowing the number of accelerators and a target training duration. 2For simplicity, we perform our analysis on the smoothed training loss which is an unbiased estimate of the test loss, as we are in the infinite data regime (the number of training tokens is less than the number of tokens in the entire corpus). 2
中文: 因此,经过更优化培训的更小型模式的好处超出了其业绩改善的直接好处。 1 例如,知道加速器的数量和目标培训期限。 2 简而言之,我们对顺利的训练损失进行分析,这是对试验损失的无偏见估计,因为我们处于无限的数据制度中(训练标志的数量少于整个教程中的标志数量)。 2个
<a id="S0040"></a> Source: p.3 S0040
Original: We show five of the current largest dense transformer models, their size, and the number of training tokens.
中文: 我们展示了5个目前最大的密集变压器模型,它们的大小,以及训练符号的数量.
<a id="S0041"></a> Source: p.3 S0041
Original: Other than LaMDA (Thoppilan et al., 2022), most models are trained for approximately 300 billion tokens.
中文: 除了LAMDA(Thoppilan等,2022年)之外,大多数型号都接受了约3000亿个活符的训练.
<a id="S0042"></a> Source: p.3 S0042
Original: We introduce Chinchilla, a substantially smaller model, trained for much longer than 300B tokens.
中文: 我们介绍Chinchilla, 一个相当小的模型, 训练时间远超过300B个符号。
<a id="S0043"></a> Source: p.3 S0043
Original: Model Size (# Parameters) Training Tokens LaMDA (Thoppilan et al., 2022) 137 Billion 168 Billion GPT-3 (Brown et al., 2020) 175 Billion 300 Billion Jurassic (Lieber et al., 2021) 178 Billion 300 Billion Gopher (Rae et al., 2021) 280 Billion 300 Billion MT-NLG 530B (Smith et al., 2022) 530 Billion 270 Billion Chinchilla 70 Billion 1.4 Trillion 2.
中文: 型号尺寸(#参数) 训练Tokens LaMDA(Thoppilan等,2022年) 137亿 168亿 GBT-3(Brown等,2020年) 175亿 300亿 侏罗纪(Lieber等,2021年) 178亿 300亿 Gopher(Rae等,2021年) 280亿 300亿 MT-NLG 530 B(Smith等,2022年) 530亿 270亿 Chinchilla 70亿 1.4 三相 2.
<a id="S0044"></a> Source: p.3 S0044
Original: Related Work Large language models. A variety of large language models have been introduced in the last few years.
中文: 相關"工作"大语言模型. 过去几年中引入了多种大型语言模式.
<a id="S0045"></a> Source: p.3 S0045
Original: These include both dense transformer models (Brown et al., 2020; Lieber et al., 2021; Rae et al., 2021; Smith et al., 2022; Thoppilan et al., 2022) and mixture-of-expert (MoE) models (Du et al., 2021; Fedus et al., 2021; Zoph et al., 2022).
中文: 其中包括密集变压器模型(Brown等,2020年;Lieber等,2021年;Rae等,2021年;Smith等,2022年;Thoppilan等,2022年)和专家混合物模型(Du等,2021年;Fedus等,2021年;Zoph等,2022年)。
<a id="S0046"></a> Source: p.3 S0046
Original: The largest dense transformers have passed 500 billion parameters (Smith et al., 2022).
中文: 最大的密集变压器已经通过了5000亿个参数(Smith等,2022年).
<a id="S0047"></a> Source: p.3 S0047
Original: The drive to train larger and larger models is clear—so far increasing the size of language models has been responsible for improving the state-of-the-art in many language modelling tasks.
中文: 培训更大和更大的模型的驱动力是明确的——到目前为止,语言模型的规模正在扩大,这在许多语言建模任务中一直有助于改进最新技术。
<a id="S0048"></a> Source: p.3 S0048
Original: Nonetheless, large language models face several challenges, including their overwhelming computational requirements (the cost of training and inference increase with model size) (Rae et al., 2021; Thoppilan et al., 2022) and the need for acquiring more high-quality training data.
中文: 尽管如此,大型语言模型仍面临若干挑战,包括它们压倒一切的计算要求(培训和推论费用随着模型大小而增加)(Rae等人,2021年;Thoppilan等人,2022年)和需要获得更高质量的培训数据。
<a id="S0049"></a> Source: p.3 S0049
Original: In fact, in this work we find that larger, high quality datasets will play a key role in any further scaling of language models.
中文: 事实上,我们发现,在这项工作中,更大的高质量数据集将在任何进一步扩展语言模型方面发挥关键作用。
<a id="S0050"></a> Source: p.3 S0050
Original: Understanding the scaling behaviour of language models and their transfer properties has been important in the development of recent large models (Hernandez et al., 2021; Kaplan et al., 2020).
中文: 了解语言模型的缩放行为及其传入属性,对近期大型模型的开发具有重要意义(Hernandez等,2021;Kaplan等,2020年).
<a id="S0051"></a> Source: p.3 S0051
Original: Kaplan et al. (2020) first showed a predictable relationship between model size and loss over many orders of magnitude.
中文: Kaplan等人(2020年)首先显示了模型大小与损失在许多数量级上之间的可预见关系.
<a id="S0052"></a> Source: p.3 S0052
Original: The authors investigate the question of choosing the optimal model size to train for a given compute budget.
中文: 作者调查了选择最佳模型规模来培训特定计算预算的问题。
<a id="S0053"></a> Source: p.3 S0053
Original: Similar to us, they address this question by training various models.
中文: 与我们一样,他们通过培训各种模式来解决这一问题。
<a id="S0054"></a> Source: p.3 S0054
Original: Our work differs from Kaplan et al. (2020) in several important ways.
中文: 我们的工作在几个重要方面与Kaplan等人(2020年)不同.
<a id="S0055"></a> Source: p.3 S0055
Original: First, the authors use a fixed number of training tokens and learning rate schedule for all models; this prevents them from modelling the impact of these hyperparameters on the loss.
中文: 首先,作者对所有模型使用固定数量的训练符号和学习率表;这使他们无法模拟这些超参数对损失的影响.
<a id="S0056"></a> Source: p.3 S0056
Original: In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results in the best final loss regardless of model size—see Figure A1.
中文: 相比之下,我们发现,设定大致与培训标志数量相匹配的学习率时间表,无论模型大小,都会导致最佳最终损失——见图A1。
<a id="S0057"></a> Source: p.3 S0057
Original: For a fixed learning rate cosine schedule to 130B tokens, the intermediate loss estimates (for 𝐷(cid:48) << 130B) are therefore overestimates of the loss of a model trained with a schedule length matching 𝐷(cid:48).
中文: 对于130B符号的固定学习速度表,中间损失估计值(D(cid:48) + 130B)因此高估了受训练的、与D(cid:48)相匹配的进度长度模型的损失。
<a id="S0058"></a> Source: p.3 S0058
Original: Using these intermediate losses results in underestimating the effectiveness of training models on less data than 130B tokens, and eventually contributes to the conclusion that model size should increase faster than training data size as compute budget increases.
中文: 利用这些中间损失导致低估了培训模型在低于130B令牌数据上的有效性,最终有助于得出模型大小应比培训数据大小更快地增加的结论,因为计算预算增加.
<a id="S0059"></a> Source: p.3 S0059
Original: In contrast, our analysis predicts that both quantities should scale at roughly the same rate.
中文: 相比之下,我们的分析预测,这两个数量的规模应大致相同。
<a id="S0060"></a> Source: p.3 S0060
Original: Secondly, we include models with up to 16B parameters, as we observe that there is slight curvature in the FLOP-loss frontier (see Appendix E)—in fact, the majority of the models used in our analysis have more than 500 million parameters, in contrast the majority of runs in Kaplan et al. (2020) are significantly smaller—many being less than 100M parameters.
中文: 第二,我们包括了最多16B参数的模型,因为我们注意到FLOP-损失边框(见附录E)有微小的曲率——事实上,我们分析中使用的大多数模型有5亿多参数,而Kaplan等人(2020年)的大多数参数则小得多——许多参数不到100M参数。
<a id="S0061"></a> Source: p.3 S0061
Original: Recently, Clark et al. (2022) specifically looked in to the scaling properties of Mixture of Expert 3
中文: 最近,Clark等人(2022年)专门研究了3号专家混合物的缩放特性。
<a id="S0062"></a> Source: p.4 S0062
Original: language models, showing that the scaling with number of experts diminishes as the model size increases—their approach models the loss as a function of two variables: the model size and the number of experts.
中文: 语言模型显示,随着模型大小的增加,与专家人数相适应的缩放会减少——它们的方法将损失作为两个变量的函数:模型大小和专家人数。
<a id="S0063"></a> Source: p.4 S0063
Original: However, the analysis is done with a fixed number of training tokens, as in Kaplan et al. (2020), potentially underestimating the improvements of branching.
中文: 然而,如Kaplan等人(2020年)所述,分析使用固定数量的培训符号,可能低估了分支的改进。
<a id="S0064"></a> Source: p.4 S0064
Original: Estimating hyperparameters for large models.
中文: 估计大型模型的超参数。
<a id="S0065"></a> Source: p.4 S0065
Original: The model size and the number of training tokens are not the only two parameters to chose when selecting a language model and a procedure to train it.
中文: 模型大小和训练符号数量并不是选择语言模型和训练程序时唯一可以选择的两个参数.
<a id="S0066"></a> Source: p.4 S0066
Original: Other important factors include learning rate, learning rate schedule, batch size, optimiser, and width-to-depth ratio.
中文: 其他重要因素有:学习率,学习率时间表,批量大小,可选择性,和宽与深度之比.
<a id="S0067"></a> Source: p.4 S0067
Original: In this work, we focus on model size and the number of training steps, and we rely on existing work and provided experimental heuristics to determine the other necessary hyperparameters.
中文: 在这项工作中,我们注重模型大小和培训步骤的数量,我们依靠现有工作,并提供实验性休克来决定其他必要的超参数.
<a id="S0068"></a> Source: p.4 S0068
Original: Yang et al. (2021) investigates how to choose a variety of these parameters for training an autoregressive transformer, including the learning rate and batch size.
中文: 杨等 (2021)调查了如何选择这些参数中的各种参数来训练自转变压器,包括学习速度和批量大小.
<a id="S0069"></a> Source: p.4 S0069
Original: McCandlish et al. (2018) finds only a weak dependence between optimal batch size and model size.
中文: McCandlish等 (2018) 发现在最佳批量大小和模型大小之间只有微弱的依赖.
<a id="S0070"></a> Source: p.4 S0070
Original: Shallue et al. (2018); Zhang et al. (2019) suggest that using larger batch-sizes than those we use is possible.
中文: Shalue等人(2018年);Zhang等人(2019年)表示,使用比我们使用的大批量尺寸是可能的.
<a id="S0071"></a> Source: p.4 S0071
Original: Levine et al. (2020) investigates the optimal depth-to-width ratio for a variety of standard model sizes.
中文: Levine等(2020年)调查了各种标准模型尺寸的最佳深度与宽比.
<a id="S0072"></a> Source: p.4 S0072
Original: We use slightly less deep models than proposed as this translates to better wall-clock performance on our hardware.
中文: 我们使用的模型比提议的要少一点,因为这将意味着我们硬件上的时钟性能更好。
<a id="S0073"></a> Source: p.4 S0073
Original: Recently, various promising alternatives to traditional dense transformers have been proposed.
中文: 最近,提出了各种有希望的替代传统密集变压器的办法。
<a id="S0074"></a> Source: p.4 S0074
Original: For example, through the use of conditional computation large MoE models like the 1.7 trillion parameter Switch transformer (Fedus et al., 2021), the 1.2 Trillion parameter GLaM model (Du et al., 2021), and others (Artetxe et al., 2021; Zoph et al., 2022) are able to provide a large effective model size despite using relatively fewer training and inference FLOPs.
中文: 例如,通过有条件的计算大型MOE模型,如1.7万亿参数开关变压器(Fedus等,2021年),1.2特里利昂参数GLAM模型(Du等,2021年)等(Artetxe等,2021年;Zoph等,2022年),尽管使用相对较少的培训和推论FLOPs,但能够提供大型有效的模型尺寸.
<a id="S0075"></a> Source: p.4 S0075
Original: However, for very large models the computational benefits of routed models seems to diminish (Clark et al., 2022).
中文: 然而,对于非常大的模型,路由模型的计算效益似乎正在减少(Clark等人,2022年)。
<a id="S0076"></a> Source: p.4 S0076
Original: An orthogonal approach to improving language models is to augment transformers with explicit retrieval mechanisms, as done by Borgeaud et al. (2021); Guu et al. (2020); Lewis et al. (2020).
中文: 改进语言模型的正统方法是用明确的检索机制来增强变压器,Borgeaud等人(2021年)、Guu等人(2020年)、Lewis等人(2020年)就是这样做的。
<a id="S0077"></a> Source: p.4 S0077
Original: This approach effectively increases the number of data tokens seen during training (by a factor of ∼ 10 in Borgeaud et al. (2021)).
中文: 这种方法有效地增加了在培训期间看到的数据符号数量(在Borgeaud等人(2021年)的系数为10)。
<a id="S0078"></a> Source: p.4 S0078
Original: This suggests that the performance of language models may be more dependant on the size of the training data than previously thought. 3.
中文: 这表明,语言模型的性能可能比以前想象的更依赖于培训数据的规模。 3个
<a id="S0079"></a> Source: p.4 S0079
Original: Estimating the optimal parameter/training tokens allocation We present three different approaches to answer the question driving our research: Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?
中文: 估计最佳参数/培训标志的分配 我们提出了三种不同的方法来回答驱动我们研究的问题:鉴于固定的FLOPs预算,一个取舍模型的规模如何和训练符号的数量如何?
<a id="S0080"></a> Source: p.4 S0080
Original: In all three cases we start by training a range of models varying both model size and the number of training tokens and use the resulting training curves to fit an empirical estimator of how they should scale.
中文: 在所有这三种情况下,我们首先培训一系列模型,这些模型既包括模型大小,也包括培训标志的数量,并使用由此产生的培训曲线,以适应对模型应如何扩大的经验估计。
<a id="S0081"></a> Source: p.4 S0081
Original: We assume a power-law relationship between compute and model size as done in Clark et al. (2022); Kaplan et al. (2020), though future work may want to include potential curvature in this relationship for large model sizes.
中文: 我们假设计算和模型大小之间的权力法关系,如Clark等人(2022年);Kaplan等人(2020年)所做的那样,尽管今后的工作可能希望将大型模型大小的潜在曲率纳入这种关系。
<a id="S0082"></a> Source: p.4 S0082
Original: The resulting predictions are similar for all three methods and suggest that parameter count and number of training tokens should be increased equally with more compute3— with proportions reported in Table 2.
中文: 由此得出的预测在所有三种方法中都是相似的,并表明应同样增加参数数和培训信使的数量,同时进行更多的计算3——比例见表2。
<a id="S0083"></a> Source: p.4 S0083
Original: This is in clear contrast to previous work on this topic and warrants further investigation. 3We compute FLOPs as described in Appendix F. 4
中文: 这与以往关于这一专题的工作明显相反,值得作进一步调查。 3 如附录F. 4所描述,我们计算FLOP
<a id="S0084"></a> Source: p.5 S0084
Original: 6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1017 1018 1019 1020 1021 1022 FLOPS ssol gniniarT 10B 5B 1012 2.5B 1B 1011 500M 250M 1010 75M 109 1017 1019 1021 1023 1025 FLOPs snekoT 1T 1.5T 100B 10B 1.0B 100M 1017 1019 1021 1023 1025 FLOPs sretemaraP 67B Figure 2 | Training curve envelope.
中文: 6.0 5.5 5.0 4.5 4.5 4.0 3.0 2.5 1017 1018 1019 1020 1021 FLOPS sol gniniarT 10B 5B 1012 2.5B 1B 1011 500M 250M 1010 75M 109 1017 1019 1021 1023 1025 FLOPs snekoT 1T 1.5T 100B 10B 1.0B 100M 1017 1019 1021 1023 FLOPs sretemaraP 67B 图2 训练曲线信封.
<a id="S0085"></a> Source: p.5 S0085
Original: On the left we show all of our different runs.
中文: 在左边,我们展示我们所有不同的跑步。
<a id="S0086"></a> Source: p.5 S0086
Original: We launched a range of model sizes going from 70M to 10B, each for four different cosine cycle lengths.
中文: 我们推出了一系列从70M到10B的模型尺寸,每个模型长度为四个不同的余弦循环长度.
<a id="S0087"></a> Source: p.5 S0087
Original: From these curves, we extracted the envelope of minimal loss per FLOP, and we used these points to estimate the optimal model size (center) for a given compute budget and the optimal number of training tokens (right).
中文: 从这些曲线上,我们提取出每个FLOP最小损失的信封,我们用这些分数来估计给定计算预算的最佳模型大小(中)和最佳训练符(正数).
<a id="S0088"></a> Source: p.5 S0088
Original: In green, we show projections of optimal model size and training token count based on the number of FLOPs used to train Gopher (5.76 × 1023). 3.1.
中文: 在绿色中,我们根据用于训练Gopher的FLOP数量(5.76×1023),显示对最佳模型大小和训练符的预测. 3.1 (英语).
<a id="S0089"></a> Source: p.5 S0089
Original: Approach 1: Fix model sizes and vary number of training tokens In our first approach we vary the number of training steps for a fixed family of models (ranging from 70M to over 10B parameters), training each model for 4 different number of training sequences.
中文: 办法1:确定模式尺寸并改变培训标志的数目 在我们的第一种方法中,我们为一组固定模式(从70M到超过10B参数)的训练步骤,为4个不同数量的训练序列培训每个模式。
<a id="S0090"></a> Source: p.5 S0090
Original: From these runs, we are able to directly extract an estimate of the minimum loss achieved for a given number of training FLOPs.
中文: 从这些作业中,我们可以直接得出对特定数量的FLOP培训所实现的最低损失的估计。
<a id="S0091"></a> Source: p.5 S0091
Original: Training details for this approach can be found in Appendix D.
中文: 这一方法的培训细节见附录D。
<a id="S0092"></a> Source: p.5 S0092
Original: For each parameter count 𝑁 we train 4 different models, decaying the learning rate by a factor of 10× over a horizon (measured in number of training tokens) that ranges by a factor of 16×.
中文: 对于每个参数,我们训练了4个不同的模型,使学习率在(以训练标志数量衡量)的平面上以10×为因子衰减,以16×为因子.
<a id="S0093"></a> Source: p.5 S0093
Original: Then, for each run, we smooth and then interpolate the training loss curve.
中文: 然后,每跑一次,我们平滑 然后插入训练损失曲线。
<a id="S0094"></a> Source: p.5 S0094
Original: From this, we obtain a continuous mapping from FLOP count to training loss for each run.
中文: 从这里,我们从FLOP计数到每跑的训练损失获得连续的映射.
<a id="S0095"></a> Source: p.5 S0095
Original: Then, for each FLOP count, we determine which run achieves the lowest loss.
中文: 然后,对于每个FLOP计数,我们确定哪个运行实现最低损失.
<a id="S0096"></a> Source: p.5 S0096
Original: Using these interpolants, we obtain a mapping from any FLOP count 𝐶, to the most efficient choice of model size 𝑁 and number of training tokens 𝐷 such that FLOPs(𝑁, 𝐷) = 𝐶.4 At 1500 logarithmically spaced FLOP values, we find which model size achieves the lowest loss of all models along with the required number of training tokens.
中文: 利用这些插件,我们从任何FLOP C 计数器中获取一个绘图,以便最高效地选择型号为 N 和 D 的训练符数,使FLOPs(N,D) = C. 在1500对数相间距的FLOP值中,我们发现哪个模型大小实现了所有模型的最低损失,以及所需的训练令牌数量.
<a id="S0097"></a> Source: p.5 S0097
Original: Finally, we fit power laws to estimate the optimal model size and number of training tokens for any given amount of compute (see the center and right panels of Figure 2), obtaining a relationship 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.
中文: 最后,我们与权力法相适应,用来估计任何特定数量的计算(见图2的中和右面板)的最佳模型大小和训练信使数量,获得N-Q-Ca和D-Q-Cb的关系.
<a id="S0098"></a> Source: p.5 S0098
Original: We 𝑜𝑝𝑡 𝑜𝑝𝑡 find that 𝑎 = 0.50 and 𝑏 = 0.50—as summarized in Table 2.
中文: 我们的选择是,如表2所概述,a=0.50,b=0.50。
<a id="S0099"></a> Source: p.5 S0099
Original: In Section D.4, we show a head-to-head comparison at 1021 FLOPs, using the model size recommended by our analysis and by the analysis of Kaplan et al. (2020)—using the model size we predict has a clear advantage. 3.2.
中文: 在D.4节中,我们利用我们的分析以及Kaplan等人(2020年)的分析所建议的模型大小,在1021 FLOPs上进行了头对头比较——使用我们预测的模型大小有明显的优势。 3.2 (中文(简体) ).
<a id="S0100"></a> Source: p.5 S0100
Original: Approach 2: IsoFLOP profiles In our second approach we vary the model size5 for a fixed set of 9 different training FLOP counts6 (ranging from 6 × 1018 to 3 × 1021 FLOPs), and consider the final training loss for each point7. in contrast with Approach 1 that considered points (𝑁, 𝐷, 𝐿) along the entire training runs.
中文: 方法2:IsoFLOP简介 在我们的第二种方法中,我们对9个不同培训FLOP计数的固定组合的型号5(从6×1018到3×1021FLOP不等)进行了修改,并考虑每个点7的最后培训损失. 与在整个培训过程中考虑要点(N、D、L)的方法1形成对比。
<a id="S0101"></a> Source: p.5 S0101
Original: This allows us to directly answer the question: For a given FLOP budget, what is the optimal parameter count? 4Note that all selected points are within the last 15% of training.
中文: 这使得我们能直接回答一个问题:对于一个给定的FLOP预算,什么是最佳参数计数? 4 注意所有选定的点数都在培训的最后15%之内。
<a id="S0102"></a> Source: p.5 S0102
Original: This suggests that when training a model over 𝐷 tokens, we should pick a cosine cycle length that decays 10× over approximately 𝐷 tokens—see further details in Appendix B. 5In approach 2, model size varies up to 16B as opposed to approach 1 where we only used models up to 10B. 6The number of training tokens is determined by the model size and training FLOPs. 7We set the cosine schedule length to match the number of tokens, which is optimal according to the analysis presented in Appendix B. 5
中文: 这表明,在培训一个模型时,我们应该选取一个余弦周期长度,在大约D符号上衰减为10××-详见附录B。 6 培训标志的数量由模型大小和培训FLOP决定. 7 我们设定了同位素排程长度,以配合符数,根据附录B. 5中的分析,这是最佳的.
<a id="S0103"></a> Source: p.6 S0103
Original: 3.2 3.0 2.8 2.6 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT 1T 100B 6e18 1e19 10B 3e19 6e19 1e20 1B 3e20 6e20 1e21 100M 3e21 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 63B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.4T Figure 3 | IsoFLOP curves.
中文: 3.2 3.0 2.8 2.6 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B 参数ssoL gniniarT 1T 100B 6e18 1e19 10B 3e19 6e19 1e20 1e21 100M 3e20 6e20 1e21 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 63B 100B 1017 1019 1021 1023 1025 FLOPs snekoT 1.4T 图3 |-| IsoFLOP曲线.
<a id="S0104"></a> Source: p.6 S0104
Original: For various model sizes, we choose the number of training tokens such that the final FLOPs is a constant.
中文: 对于各种模型大小,我们选择训练符号的数量,这样最终的FLOP就是一个常数.
<a id="S0105"></a> Source: p.6 S0105
Original: The cosine cycle length is set to match the target FLOP count.
中文: 余弦周期长度设定与目标FLOP计数相匹配.
<a id="S0106"></a> Source: p.6 S0106
Original: We find a clear valley in loss, meaning that for a given FLOP budget there is an optimal model to train (left).
中文: 我们发现一个清晰的谷地在损失中,这意味着对于一个特定的FLOP预算,有一个训练的最佳模式(左方).
<a id="S0107"></a> Source: p.6 S0107
Original: Using the location of these valleys, we project optimal model size and number of tokens for larger models (center and right).
中文: 利用这些山谷的位置,我们为更大的模型(中和正中)预测出最佳的模型大小和符数.
<a id="S0108"></a> Source: p.6 S0108
Original: In green, we show the estimated number of parameters and tokens for an optimal model trained with the compute budget of Gopher.
中文: 在绿色中,我们显示一个通过Gopher的计算预算培训的最佳模型的估计参数和符号数量。
<a id="S0109"></a> Source: p.6 S0109
Original: For each FLOP budget, we plot the final loss (after smoothing) against the parameter count in Figure 3 (left).
中文: 对于每个FLOP预算,我们比照图3(左)中的参数计数来绘制最终损失(平滑后).
<a id="S0110"></a> Source: p.6 S0110
Original: In all cases, we ensure that we have trained a diverse enough set of model sizes to see a clear minimum in the loss.
中文: 在所有情况下,我们确保我们训练了足够多的一套模型大小,以看到损失的明确最低程度。
<a id="S0111"></a> Source: p.6 S0111
Original: We fit a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved (Figure 3 (left)).
中文: 我们在每个IsoFLOPs曲线上安装一个抛物线来直接估计最小损失的模型大小(图3(左))。
<a id="S0112"></a> Source: p.6 S0112
Original: As with the previous approach, we then fit a power law between FLOPs and loss-optimal model size and number of training tokens, shown in Figure 3 (center, right).
中文: 与先前的做法一样,我们然后在FLOPs与损失-最佳模型大小和培训标志数量之间适用权力法,如图3(中,正)所示。
<a id="S0113"></a> Source: p.6 S0113
Original: Again, we fit exponents of the form 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏 and we find that 𝑜𝑝𝑡 𝑜𝑝𝑡 𝑎 = 0.49 and 𝑏 = 0.51—as summarized in Table 2. 3.3.
中文: 我们再次采用了表2.3.3中概述的N + Ca和D + Cb表格的缩写,我们发现选择a=0.49和b=0.51。
<a id="S0114"></a> Source: p.6 S0114
Original: Approach 3: Fitting a parametric loss function Lastly, we model all final losses from experiments in Approach 1 & 2 as a parametric function of model parameter count and the number of seen tokens.
中文: 方法 3:搭配参数损失函数 最后,我们模拟了方法 1 和 2 中实验的所有最终损失,作为模型参数计数的参数函数和所见出值的数目。
<a id="S0115"></a> Source: p.6 S0115
Original: Following a classical risk decomposition (see Section D.2), we propose the following functional form 𝐴 𝐵 𝐿ˆ (𝑁, 𝐷) (cid:44) 𝐸 + + . (2) 𝑁𝛼 𝐷𝛽 The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text.
中文: 在传统的风险分解(见D.2节)之后,我们提出以下功能形式:A B Lˆ (N, D)(编:44) E + +. (2) Nα Dβ 第一个术语捕捉到数据分布上一个理想的基因过程的损失,并应该对应自然文本的 en.
<a id="S0116"></a> Source: p.6 S0116
Original: The second term captures the fact that a perfectly trained transformer with 𝑁 parameters underperforms the ideal generative process.
中文: 第二个术语捕捉到一个完全训练有素的有N参数的变压器表现得不尽人意的基因过程.
<a id="S0117"></a> Source: p.6 S0117
Original: The final term captures the fact that the transformer is not trained to convergence, as we only make a finite number of optimisation steps, on a sample of the dataset distribution.
中文: 最终的术语捕捉到一个事实,即变压器没有接受过趋同训练,因为我们只在数据集分布的样本上作出有限的优化步骤.
<a id="S0118"></a> Source: p.6 S0118
Original: To estimate ( 𝐴, 𝐵, 𝐸, 𝛼, 𝛽), we minimize the Huber loss (Huber, 1964) between the predicted and observed log loss using the L-BFGS algorithm (Nocedal, 1980): min ∑︁ Huber (cid:16) log 𝐿ˆ (𝑁 , 𝐷 ) − log 𝐿 (cid:17) (3) 𝛿 𝑖 𝑖 𝑖 𝐴,𝐵,𝐸,𝛼,𝛽 Runs 𝑖 We account for possible local minima by selecting the best fit from a grid of initialisations.
中文: 估计(A、B、E、α、β),我们利用L-BFGS算法将预测和观测到的日志损失(Huber,1964年)最小化(Nortedal,1980年):min ∑︁ Huber (cid:16) log Lˆ (N、D)-log L (cid:17)(3) i i i i A,B,E,α,β runs i 我们通过从初始化的网格中选择最合适的位置,来说明当地可能存在的次要因素。
<a id="S0119"></a> Source: p.6 S0119
Original: The Huber loss (𝛿 = 10−3) is robust to outliers, which we find important for good predictive performance over held-out data points.
中文: Huber损失( = 10- 3) 强劲到异常值, 我们认为这对在被搁置的数据点上良好的预测性能很重要。
<a id="S0120"></a> Source: p.6 S0120
Original: Section D.2 details the fitting procedure and the loss decomposition. 6
中文: D.2节详细介绍了适当的程序和损失分解情况。 6个
<a id="S0121"></a> Source: p.7 S0121
Original: 100B 40B 10B 1B 100M 1018 1019 1020 1021 1022 1023 Gopher budget Training FLOPs ezis ledoM IsoLoss contours 5.00 4.00 3.00 Efficient frontier Empirical data 2.00 IsoFLOPs slice ssoL IsoFLOPs slices Train.
中文: 100B 40B 10B 1B 100M 1018 1019 1020 1021 1022 1023 Gopher预算 培训FLOPs ezis ledudoM IsoLoss轮廓 5.00 4.00 3.00 高效前沿经验数据 2.00 IsoFLOPs切入ssoL IsoFLOPs切入列车.
<a id="S0122"></a> Source: p.7 S0122
Original: FLOPs 6e+18 1e+19 3e+19 6e+19 1e+20 3e+20 6e+20 1e+21 3e+21 Gopher 100M 1B 10B 40B Model size Figure 4 | Parametric fit.
中文: FLOPs 6e+18 1e+19 3e+19 6e+19 1e+20 3e+20 6e+20 1e+21 3e+21 Gopher 100M 1B 10B 40B 型号尺寸 图4 QQ 参数适中.
<a id="S0123"></a> Source: p.7 S0123
Original: We fit a parametric modelling of the loss 𝐿ˆ (𝑁, 𝐷) and display contour (left) and isoFLOP slices (right).
中文: 我们安装了LQQ(N, D)损失的参数模型,并显示等宽(左)和等宽(正)切片。
<a id="S0124"></a> Source: p.7 S0124
Original: For each isoFLOP slice, we include a corresponding dashed line in the left plot.
中文: 对于每个等同FLOP切片,我们在左边的地块中包含一个相应的虚线.
<a id="S0125"></a> Source: p.7 S0125
Original: In the left plot, we show the efficient frontier in blue, which is a line in log-log space.
中文: 在左边的图中,我们用蓝色显示高效的前沿,这是日志空间中的一行.
<a id="S0126"></a> Source: p.7 S0126
Original: Specifically, the curve goes through each iso-loss contour at the point with the fewest FLOPs.
中文: 具体地说,曲线通过每个同位素-损失轮廓,最少的FLOP.
<a id="S0127"></a> Source: p.7 S0127
Original: We project the optimal model size given the Gopher FLOP budget to be 40B parameters.
中文: 根据Gopher FLOP的预算,我们预测最佳模型大小为40B参数。
<a id="S0128"></a> Source: p.7 S0128
Original: We can approximate the functions 𝑁 𝑜𝑝𝑡 and 𝐷 𝑜𝑝𝑡 by minimizing the parametric loss 𝐿ˆ under the constraint FLOPs(𝑁, 𝐷) ≈ 6𝑁 𝐷 (Kaplan et al., 2020).
中文: 我们可以通过最小化FLOPs(N, D) → 6N D(Kaplan等,2020年)的约束下的参数损失LQX来估算函数 N 选择和D选择。
<a id="S0129"></a> Source: p.7 S0129
Original: The resulting 𝑁 and 𝐷 𝑜𝑝𝑡 𝑜𝑝𝑡 balance the two terms in Equation (3) that depend on model size and data.
中文: 由此产生的N和D选择选择平衡了方程式(3)中取决于模型大小和数据的两个术语.
<a id="S0130"></a> Source: p.7 S0130
Original: By construction, they have a power-law form: 𝑁 (𝐶) = 𝐺 (cid:18) 𝐶 (cid:19) 𝑎 , 𝐷 (𝐶) = 𝐺−1 (cid:18) 𝐶 (cid:19)𝑏 , where 𝐺 = (cid:18) 𝛼𝐴 (cid:19) 𝛼+ 1 𝛽 , 𝑎 = 𝛽 , and 𝑏 = 𝛼 . (4) 𝑜𝑝𝑡 6 𝑜𝑝𝑡 6 𝛽𝐵 𝛼 + 𝛽 𝛼 + 𝛽 We show contours of the fitted function 𝐿ˆ in Figure 4 (left), and the closed-form efficient computational frontier in blue.
中文: 通过构造,它们具有功率法形:N(C)=G(Cid:18)C(Cid:19) a,D(C)=G−1(Cid:18)C(Cid:19)b,其中G=(Cid:18)αA(Cid:19)α+ 1β,a=β,b=α. (4)选取6选取6βB α + β + β. 我们在图4(左)中显示安装函数LQQ的轮廓,并显示蓝色的封闭式高效计算边框。
<a id="S0131"></a> Source: p.7 S0131
Original: From this approach, we find that 𝑎 = 0.46 and 𝑏 = 0.54—as summarized in Table 2. 3.4.
中文: 从这一方法中,我们发现,如表2.3.4所概括,a=0.46和b=0.54。
<a id="S0132"></a> Source: p.7 S0132
Original: Optimal model scaling We find that the three approaches, despite using different fitting methodologies and different trained models, yield comparable predictions for the optimal scaling in parameters and tokens with FLOPs (shown in Table 2).
中文: 优化模型缩放 我们发现,尽管采用了不同的适当方法和不同的训练有素的模型,但三种方法对参数和标志的最佳缩放作了可比较的预测(见表2)。
<a id="S0133"></a> Source: p.7 S0133
Original: All three approaches suggest that as compute budget increases, model size and the amount of training data should be increased in approximately equal proportions.
中文: 所有三种办法都建议,随着预算的计算增加,模型的规模和培训数据的数量应大致以相同的比例增加。
<a id="S0134"></a> Source: p.7 S0134
Original: The first and second approaches yield very similar predictions for optimal model sizes, as shown in Figure 1 and Figure A3.
中文: 如图1和图A3所示,第一和第二种方法对最佳模型大小的预测非常相似。
<a id="S0135"></a> Source: p.7 S0135
Original: The third approach predicts even smaller models being optimal at larger compute budgets.
中文: 第三种办法预测,在较大的计算预算方面,更小的模型是最佳的。
<a id="S0136"></a> Source: p.7 S0136
Original: We note that the observed points (𝐿, 𝑁, 𝐷) for low training FLOPs (𝐶 (cid:54) 1𝑒21) have larger residuals (cid:107)𝐿 − 𝐿ˆ (𝑁, 𝐷) (cid:107) 2 than points with higher computational budgets.
中文: 我们注意到,低训练FLOPs(C(cid:54) 1e21)所观察到的分数(L,N,D)所剩分数(Cid:107)L-L-(N,D)(Cid:107)2比计算预算较高的分数多.
<a id="S0137"></a> Source: p.7 S0137
Original: The fitted model places increased 2 weight on the points with more FLOPs—automatically considering the low-computational budget points as outliers due to the Huber loss.
中文: 装配的模型增加了两个分量,即,由于Huber号损失,自动考虑到低计算预算分数的分数高于分数。
<a id="S0138"></a> Source: p.7 S0138
Original: As a consequence of the empirically observed negative curvature in the frontier 𝐶 → 𝑁 (see Appendix E), this results in predicting a lower 𝑁 than the 𝑜𝑝𝑡 𝑜𝑝𝑡 two other approaches.
中文: 由于从经验上观察到前沿C-N的负曲率(见附录E),因此预测N比选取的其他两种办法要低。
<a id="S0139"></a> Source: p.7 S0139
Original: In Table 3 we show the estimated number of FLOPs and tokens that would ensure that a model of a given size lies on the compute-optimal frontier.
中文: 在表3中,我们列出了可确保一定规模的模型位于计算-最佳边框的FLOP和标志的估计数量。
<a id="S0140"></a> Source: p.7 S0140
Original: Our findings suggests that the current generation of 7
中文: 我们的调查结果表明,当代7人
<a id="S0141"></a> Source: p.8 S0141
Original: Table 2 | Estimated parameter and data scaling with increased training compute.
中文: 表2 估计参数和数据规模,增加培训计算。
<a id="S0142"></a> Source: p.8 S0142
Original: The listed values are the exponents, 𝑎 and 𝑏, on the relationship 𝑁 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.
中文: 列出的值是关系N QQ Ca 和 D Q Cb 的代词 a和b.
<a id="S0143"></a> Source: p.8 S0143
Original: Our analysis suggests 𝑜𝑝𝑡 𝑜𝑝𝑡 a near equal scaling in parameters and data with increasing compute which is in clear contrast to previous work on the scaling of large models.
中文: 我们的分析建议,选择在参数和数据上采用近乎等同的缩放方法,同时增加计算,这与以往关于大型模型缩放的工作明显相反。
<a id="S0144"></a> Source: p.8 S0144
Original: The 10th and 90th percentiles are estimated via bootstrapping data (80% of the dataset is sampled 100 times) and are shown in parenthesis.
中文: 第十百分位数和第九百分位数通过靴形数据估算(80%的数据集被抽样100次),并用括号显示.
<a id="S0145"></a> Source: p.8 S0145
Original: Approach Coeff. 𝑎 where 𝑁 ∝ 𝐶𝑎 Coeff. 𝑏 where 𝐷 ∝ 𝐶𝑏 𝑜𝑝𝑡 𝑜𝑝𝑡 1.
中文: 靠近Coeff. a, N Q Ca Coeff. b, D Q Cb选择 1。
<a id="S0146"></a> Source: p.8 S0146
Original: Minimum over training curves 0.50 (0.488, 0.502) 0.50 (0.501, 0.512) 2.
中文: 训练曲线上的最低曲线为0.50(0.488,0.502)0.50(0.501,0.512) 2.
<a id="S0147"></a> Source: p.8 S0147
Original: IsoFLOP profiles 0.49 (0.462, 0.534) 0.51 (0.483, 0.529) 3.
中文: IsoFLOP剖面图0.49 (0.462, 0.534) 0.51 (0.483, 0.529) 3.
<a id="S0148"></a> Source: p.8 S0148
Original: Parametric modelling of the loss 0.46 (0.454, 0.455) 0.54 (0.542, 0.543) Kaplan et al. (2020) 0.73 0.27 Table 3 | Estimated optimal training FLOPs and training tokens for various model sizes.
中文: 0.46 (0.454, 0.455) 0.54 (0.542, 0.543) Kaplan等 (2020) 0.73 0.27 表3 各种模型尺寸的最佳培训工具和培训标志。
<a id="S0149"></a> Source: p.8 S0149
Original: For various model sizes, we show the projections from Approach 1 of how many FLOPs and training tokens would be needed to train compute-optimal models.
中文: 对于各种模型大小,我们从方法1中可以预测,培训计算最佳模型需要多少FLOP和训练标志。
<a id="S0150"></a> Source: p.8 S0150
Original: The estimates for Approach 2 & 3 are similar (shown in Section D.3) Parameters FLOPs FLOPs (in Gopher unit) Tokens 400 Million 1.92e+19 1/29, 968 8.0 Billion 1 Billion 1.21e+20 1/4, 761 20.2 Billion 10 Billion 1.23e+22 1/46 205.1 Billion 67 Billion 5.76e+23 1 1.5 Trillion . 175 Billion 3.85e+24 6.7 3.7 Trillion 280 Billion 9.90e+24 17.2 5.9 Trillion 520 Billion 3.43e+25 59.5 11.0 Trillion 1 Trillion 1.27e+26 221.3 21.2 Trillion 10 Trillion 1.30e+28 22515.9 216.2 Trillion large language models are considerably over-sized, given their respective compute budgets, as shown in Figure 1.
中文: 方法2和3的估计数相近(见D.3节) 参数FLOPs FLOPs(以Gopher为单位) Tokens 400 million 1.92e+19 1/29,968 8.0 million 1 million 1.21e+20 1/4,761 20.2 million 1.23e+22 1.46 205.1 million 5.76e+23 1 1.5 million 3.85e+24 6.7 3.7 million 280 million 9.90e+24 17.2 5.9tillion 520 million 3.43e+25 59.5 11.0 million 1 Trillion 1.27e+26 221.3 21.2 Tillion 10 million 1.30e+28 22515.9 216.2 如图1所显示的,考虑到各大语言模型的计算预算,其规模大大过大。
<a id="S0151"></a> Source: p.8 S0151
Original: For example, we find that a 175 billion parameter model should be trained with a compute budget of 4.41 × 1024 FLOPs and on over 4.2 trillion tokens. A 280 billion Gopher-like model is the optimal model to train given a compute budget of approximately 1025 FLOPs and should be trained on 6.8 trillion tokens.
中文: 例如,我们发现,1,750亿个参数模型应接受培训,计算预算为4.41×1024 FLOP和超过4.2万亿令牌。 一个2 800亿戈弗式的模型是最佳的模型,用来训练大约1025个FLOP的计算预算,应该训练6.8万亿令牌。
<a id="S0152"></a> Source: p.8 S0152
Original: Unless one has a compute budget of 1026 FLOPs (over 250× the compute used to train Gopher), a 1 trillion parameter model is unlikely to be the optimal model to train.
中文: 除非一个人的计算预算为1026个FLOPs(超过250×用于训练Gopher的计算),否则1兆个参数模型不太可能成为训练的最佳模型.
<a id="S0153"></a> Source: p.8 S0153
Original: Furthermore, the amount of training data that is projected to be needed is far beyond what is currently used to train large models, and underscores the importance of dataset collection in addition to engineering improvements that allow for model scale.
中文: 此外,预计需要的培训数据数量远远超出目前用于培训大型模型的数据,并突出了除了进行工程改进以扩大模型规模之外,数据集收集工作的重要性。
<a id="S0154"></a> Source: p.8 S0154
Original: While there is significant uncertainty extrapolating out many orders of magnitude, our analysis clearly suggests that given the training compute budget for many current LLMs, smaller models should have been trained on more tokens to achieve the most performant model.
中文: 虽然从许多数量级推算出有很大的不确定性,但我们的分析明确表明,鉴于目前许多有限责任公司的培训预算计算,应该对较小的模型进行更多标志性的培训,以达到最能发挥作用的模式。
<a id="S0155"></a> Source: p.8 S0155
Original: In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 (Raffel et al., 2020a) and GitHub code (Rae et al., 2021).
中文: 在附录C中,我们转载了IsoFLOP对另外两个数据集的分析:C4(Raffel等人,2020年a)和GitHub代码(Rae等人,2021年)。
<a id="S0156"></a> Source: p.8 S0156
Original: In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions. 8
中文: 在这两种情况下,我们得出类似的结论,即示范规模和培训标志的数量应按比例按比例缩减。 第8条
<a id="S0157"></a> Source: p.9 S0157
Original: Chinchilla Based on our analysis in Section 3, the optimal model size for the Gopher compute budget is somewhere between 40 and 70 billion parameters.
中文: 根据我们在第3节的分析 Gopher计算预算的最佳模型大小 大约在400到700亿个参数之间
<a id="S0158"></a> Source: p.9 S0158
Original: We test this hypothesis by training a model on the larger end of this range—70B parameters—for 1.4T tokens, due to both dataset and computational efficiency considerations.
中文: 由于数据集和计算效率方面的考虑,我们测试这一假说的方法是,对这一范围较大的一端进行模型——70B参数——用于1.4T令牌。
<a id="S0159"></a> Source: p.9 S0159
Original: In this section we compare this model, which we call Chinchilla, to Gopher and other LLMs.
中文: 在这一节中,我们把这个叫做Chinchilla的模型与Gopher和其他LLMs进行比较。
<a id="S0160"></a> Source: p.9 S0160
Original: Both Chinchilla and Gopher have been trained for the same number of FLOPs but differ in the size of the model and the number of training tokens.
中文: Chinchilla和Gopher都接受了同样数量的FLOP培训,但模型大小和培训符号数量不同.
<a id="S0161"></a> Source: p.9 S0161
Original: While pre-training a large language model has a considerable compute cost, downstream finetuning and inference also make up substantial compute usage (Rae et al., 2021).
中文: 虽然培训前的大型语言模型的计算成本相当高,但下游的微调和推断也构成大量的计算使用(Rae等人,2021年)。
<a id="S0162"></a> Source: p.9 S0162
Original: Due to being 4× smaller than Gopher, both the memory footprint and inference cost of Chinchilla are also smaller. 4.1.
中文: 由于比Gopher还小4×,Chinchilla的记忆足迹和推论成本也都较小. 4.1 (英语).
<a id="S0163"></a> Source: p.9 S0163
Original: Model and training details The full set of hyperparameters used to train Chinchilla are given in Table 4.
中文: 模式和培训细节 用于训练钦奇利亚的整套超参数见表4。
<a id="S0164"></a> Source: p.9 S0164
Original: Chinchilla uses the same model architecture and training setup as Gopher with the exception of the differences listed below. • We train Chinchilla on MassiveText (the same dataset as Gopher) but use a slightly different subset distribution (shown in Table A1) to account for the increased number of training tokens. • We use AdamW (Loshchilov and Hutter, 2019) for Chinchilla rather than Adam (Kingma and Ba, 2014) as this improves the language modelling loss and the downstream task performance after finetuning.8 • We train Chinchilla with a slightly modified SentencePiece (Kudo and Richardson, 2018) tokenizer that does not apply NFKC normalisation.
中文: Chinchilla除了以下列出的区别外,采用了与Gopher相同的模型架构和培训设置. • 我们在MassiveText(与Gopher相同的数据集)上培训Chinchilla,但使用一个稍有不同的子集分布(在表A1中显示)来说明培训信使数量增加的原因。 • 我们使用AdamW(Loshchilov和Hutter,2019年)为Chinchilla而不是Adam(Kingma和Ba,2014年),因为这在微调后改善了语言建模损失和下游任务性能。
<a id="S0165"></a> Source: p.9 S0165
Original: The vocabulary is very similar– 94.15% of tokens are the same as those used for training Gopher.
中文: 词汇非常相近 — — 94. 15%的符文和用于训练Gopher的符文相同.
<a id="S0166"></a> Source: p.9 S0166
Original: We find that this particularly helps with the representation of mathematics and chemistry, for example. • Whilst the forward and backward pass are computed in bfloat16, we store a float32 copy of the weights in the distributed optimiser state (Rajbhandari et al., 2020).
中文: 我们认为,这特别有助于数学和化学的代表性。 二. 支助 虽然前向和后向通行证以bfloat16计算,但我们在分布的Opimiser状态下储存了一份重量的浮标32副本(Rajbhandari等,2020年)。
<a id="S0167"></a> Source: p.9 S0167
Original: See Lessons Learned from Rae et al. (2021) for additional details.
中文: 更多详情见Rae等人(2021年)的经验教训。
<a id="S0168"></a> Source: p.9 S0168
Original: In Appendix G we show the impact of the various optimiser related changes between Chinchilla and Gopher.
中文: 在附录G中,我们显示了钦奇拉和戈斐之间各种与选取器相关的变化的影响。
<a id="S0169"></a> Source: p.9 S0169
Original: All models in this analysis have been trained on TPUv3/TPUv4 (Jouppi et al., 2017) with JAX (Bradbury et al., 2018) and Haiku (Hennigan et al., 2020).
中文: 本分析中的所有模型都接受了TPUv3/TPUv4(Jouppi等,2017年)与JAX(Bradbury等,2018年)和Haiku(Hennigan等,2020年)的培训.
<a id="S0170"></a> Source: p.9 S0170
Original: We include a Chinchilla model card (Mitchell et al., 2019) in Table A8.
中文: 我们在表A8中列入了一张钦奇拉模式卡(Mitchell等,2019年).
<a id="S0171"></a> Source: p.9 S0171
Original: Model Layers Number Heads Key/Value Size d Max LR Batch Size model Gopher 280B 80 128 128 16,384 4 × 10−5 3M → 6M Chinchilla 70B 80 64 128 8,192 1 × 10−4 1.5M → 3M Table 4 | Chinchilla architecture details.
中文: 型号图层 编号 键/值 键/值 大小 d Max LR Batch 大小 型号 Gopher 280B 80 128 128 16 384 4 × 10−5 3M → 6M Chinchilla 70B 80 64 128 8 192 1 × 10−4 1.5M → 3M 表 4 → Chinchilla 架构细节.
<a id="S0172"></a> Source: p.9 S0172
Original: We list the number of layers, the key/value size, the bottleneck activation size d , the maximum learning rate, and the training batch size (# tokens). model The feed-forward size is always set to 4 × d .
中文: 我们列出层数,键/值大小,瓶颈激活大小d,最大学习率,以及训练批量大小(#令牌). 模式 向导大小始终被设定为 4 × d.
<a id="S0173"></a> Source: p.9 S0173
Original: Note that we double the batch size midway through model training for both Chinchilla and Gopher. 8Interestingly, a model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance is notably better– see Figure A7 9
中文: 请注意,我们通过对Chinchilla和Gopher的模型训练,在中途将批量规模增加一倍。 8 令人感兴趣的是,受过AdamW训练的模型只经过了与Adam训练的模型在余弦周期中大约80%的训练表现,虽然结局表现明显更好(见图A7 9)
<a id="S0174"></a> Source: p.10 S0174
Original: # Tasks Examples Language Modelling 20 WikiText-103, The Pile: PG-19, arXiv, FreeLaw, . . .
中文: # Tasks example Language Modeling 20 WikiText-103, 皮勒: PG-19, arXiv, FreeLaw,.
<a id="S0175"></a> Source: p.10 S0175
Original: Reading Comprehension 3 RACE-m, RACE-h, LAMBADA Question Answering 3 Natural Questions, TriviaQA, TruthfulQA Common Sense 5 HellaSwag, Winogrande, PIQA, SIQA, BoolQ MMLU 57 High School Chemistry, Astronomy, Clinical Knowledge, . . .
中文: 阅读 Confederation 3 RACE-m, RACE-h, LAMBADA 答题 3 自然问题 TriviaQA, TrifefulQA Common Sense 5 HellaSwag, Winogrande, PIQA, SIQA, BoulQ MMLU 57 高中化学,天文学,临床知识,.
<a id="S0176"></a> Source: p.10 S0176
Original: BIG-bench 62 Causal Judgement, Epistemic Reasoning, Temporal Sequences, . . .
中文: BIG-Bench 62 Causal Cricument, Epistemic Reasoning, Timorary Sequences,.
<a id="S0177"></a> Source: p.10 S0177
Original: We evaluate Chinchilla on a collection of language modelling along with downstream tasks.
中文: 我们通过收集语言建模和下游任务来评价钦奇拉.
<a id="S0178"></a> Source: p.10 S0178
Original: We evaluate on largely the same tasks as in Rae et al. (2021), to allow for direct comparison. 4.2.
中文: 我们评价的任务与Rae等人(2021年)基本相同,以便进行直接比较。 4.2 (英语).
<a id="S0179"></a> Source: p.10 S0179
Original: Results We perform an extensive evaluation of Chinchilla, comparing against various large language models.
中文: 结果 我们对Chinchilla进行了广泛的评价,比较了各种大语言模式。
<a id="S0180"></a> Source: p.10 S0180
Original: We evaluate on a large subset of the tasks presented in Rae et al. (2021), shown in Table 5.
中文: 我们对在Rae等人(2021年)中提出的大量任务进行评估,如表5所示。
<a id="S0181"></a> Source: p.10 S0181
Original: As the focus of this work is on optimal model scaling, we included a large representative subset, and introduce a few new evaluations to allow for better comparison to other existing large models.
中文: 由于这项工作的重点是优化模型规模,我们包括了一个具有代表性的大型子集,并引入了几个新的评价,以便与其他现有的大型模型进行更好的比较。
<a id="S0182"></a> Source: p.10 S0182
Original: The evaluation details for all tasks are the same as described in Rae et al. (2021). 4.2.1.
中文: 所有任务的评价细节与Rae等人(2021年)所述相同。
<a id="S0183"></a> Source: p.10 S0183
Original: Language modelling stcartsba_dembup retropxe_hin sdnuorgkcab_otpsu lartnec_dembup cc_elip 2suprockoob egnahcxekcats seltitbusnepo 2txetbewnepo swenrekcah scitamehtam_md vixra waleerf 3skoob srepaplihp buhtig cri_utnubu lraporue 91_gp_grebnetug 0.10 0.08 0.06 0.04 0.02 0.00 bpb ni esaerceD rehpoG ot derapmoc Figure 5 | Pile Evaluation.
中文: 语言建模 stcartsba dembup retropxe hin sdnuorgkcab otpsu lartnec dembup cc elip 2 suprockoob egnahcxekcats seltitbusnepo 2txetbewnepo swenrektam md vixra waleerf 3skoob srepalihp buhtig crip-utnubu lraporue 91 gp grebnetug 0.10.08 0.06 0.04. 0.00 bpb ni esaerceD repoGot derapmoc图5 ile Pile评价.
<a id="S0184"></a> Source: p.10 S0184
Original: For the different evaluation sets in The Pile (Gao et al., 2020), we show the bits-per-byte (bpb) improvement (decrease) of Chinchilla compared to Gopher.
中文: 对于The Pile(Gao等,2020年)中的不同评价集,我们显示与Gopher相比Chinchilla的比特(bpb)改进(减少).
<a id="S0185"></a> Source: p.10 S0185
Original: On all subsets, Chinchilla outperforms Gopher.
中文: 在所有子集上,Chinchilla的性能都超过了Gopher.
<a id="S0186"></a> Source: p.10 S0186
Original: Chinchilla significantly outperforms Gopher on all evaluation subsets of The Pile (Gao et al., 2020), as shown in Figure 5.
中文: 如图5所示,Chinchilla在所有评价子集(Gao等人,2020年)上明显地超过了Gopher。
<a id="S0187"></a> Source: p.10 S0187
Original: Compared to Jurassic-1 (178B) Lieber et al. (2021), Chinchilla is more performant on all but two subsets– dm_mathematics and ubuntu_irc– see Table A5 for a raw bits-per-byte comparison.
中文: 相较于侏罗纪-1(178B)Lieber等(2021),Chinchilla除了两个子集-dm 数学和ubuntu irc外,其他所有子集的性能都比较好,参见表A5的生比/字节比较.
<a id="S0188"></a> Source: p.10 S0188
Original: On Wikitext103 (Merity et al., 2017), Chinchilla achieves a perplexity of 7.16 compared to 7.75 for Gopher.
中文: 在Wikitext103上(Merity等,2017年),钦奇拉实现了7.16的迷惑度,而高佛的迷惑度为7.75.
<a id="S0189"></a> Source: p.10 S0189
Original: Some caution is needed when comparing Chinchilla with Gopher on these language modelling benchmarks as Chinchilla is trained on 4× more data than Gopher and thus train/test set leakage may artificially enhance the results.
中文: 在将这些语言建模基准比作Chinchilla和Gopher时需要谨慎,因为Chinchilla比Gopher接受4×多数据培训,因此,培训/测试套件泄漏可能人为地增强结果。
<a id="S0190"></a> Source: p.11 S0190
Original: Random 25.0% Average human rater 34.5% GPT-3 5-shot 43.9% Gopher 5-shot 60.0% Chinchilla 5-shot 67.6% Average human expert performance 89.8% June 2022 Forecast 57.1% June 2023 Forecast 63.4% Table 6 | Massive Multitask Language Understanding (MMLU).
中文: 随机25.0% 人平均率34.5% GPT-3 5发43.9% Gopher 5发60.0% Chinchilla 5发67.6% 人类专家平均表现 89.8% 2022年6月 预测 57.1% 2023年6月 预测 63.4% 表6 大规模多任务语言理解(MMLU).
<a id="S0191"></a> Source: p.11 S0191
Original: We report the average 5-shot accuracy over 57 tasks with model and human accuracy comparisons taken from Hendrycks et al. (2020).
中文: 我们报告57项任务的平均5发精度,并用Hendrycks等人(2020年)提供的模型和人类精度比较。
<a id="S0192"></a> Source: p.11 S0192
Original: We also include the average prediction for state of the art accuracy in June 2022/2023 made by 73 competitive human forecasters in Steinhardt (2021). tasks for which leakage is less of a concern, such as MMLU (Hendrycks et al., 2020) and BIG-bench (BIG-bench collaboration, 2021) along with various closed-book question answering and common sense analyses. 4.2.2.
中文: 我们还包括了斯泰因哈特(2021年)73个竞技人类预测员在2022/2023年6月对最新艺术精度所作的平均预测. 渗漏较少引起关注的任务,如MMLU(Hendrycks等,2020年)和BIG-bench(BIG-bench合作,2021年),以及各种闭库问答和常识分析。 4.2.2. (中文(简体) ).
<a id="S0193"></a> Source: p.11 S0193
Original: MMLU The Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2020) consists of a range of exam-like questions on academic subjects.
中文: MMLU 大规模多任务语言理解(MMLU)基准(Hendrycks等,2020年)由一系列与考试相类似的学术课题问题组成.
<a id="S0194"></a> Source: p.11 S0194
Original: In Table 6, we report Chinchilla’s average 5-shot performance on MMLU (the full breakdown of results is shown in Table A6).
中文: 在表6中,我们报告了钦奇拉在MMLU上的平均5发子弹的性能(结果的全部细目见表A6)。
<a id="S0195"></a> Source: p.11 S0195
Original: On this benchmark, Chinchilla significantly outperforms Gopher despite being much smaller, with an average accuracy of 67.6% (improving upon Gopher by 7.6%).
中文: 在这个基准上,Chinchilla的性能大大地超过Gopher,尽管它比Gopher要小得多,平均精度为67.6%(Gopher改进了7.6%).
<a id="S0196"></a> Source: p.11 S0196
Original: Remarkably, Chinchilla even outperforms the expert forecast for June 2023 of 63.4% accuracy (see Table 6) (Steinhardt, 2021).
中文: 值得注意的是,Chinchilla甚至超过了专家2023年6月预测的63.4%的准确度(见表6)(Steinhardt, 2021)。
<a id="S0197"></a> Source: p.11 S0197
Original: Furthermore, Chinchilla achieves greater than 90% accuracy on 4 different individual tasks– high_school_gov_and_politics, international_law, sociology, and us_foreign_policy.
中文: 此外,钦奇拉在4个不同任务上的准确度超过90% — — 高中、大学、大学、社会学和外交政策。
<a id="S0198"></a> Source: p.11 S0198
Original: To our knowledge, no other model has achieved greater than 90% accuracy on a subset.
中文: 据我们所知,没有任何其他模型在一个子集上取得了超过90%的精确度.
<a id="S0199"></a> Source: p.11 S0199
Original: In Figure 6, we show a comparison to Gopher broken down by task.
中文: 在图6中,我们显示与Gopher的比较,按任务分列。
<a id="S0200"></a> Source: p.11 S0200
Original: Overall, we find that Chinchilla improves performance on the vast majority of tasks.
中文: 总体而言,我们发现钦奇拉改进了绝大多数任务的业绩.
<a id="S0201"></a> Source: p.11 S0201
Original: On four tasks (college_mathematics, econometrics, moral_scenarios, and formal_logic) Chinchilla underperforms Gopher, and there is no change in performance on two tasks. 4.2.3.
中文: 在四项任务(college mathematics, Economics, moral screenarios, and retroductal logic)上,钦奇拉表现不佳,两个任务的表现没有变化. 4.2.3 (中文(简体) ).
<a id="S0202"></a> Source: p.11 S0202
Original: Reading comprehension On the final word prediction dataset LAMBADA (Paperno et al., 2016), Chinchilla achieves 77.4% accuracy, compared to 74.5% accuracy from Gopher and 76.6% from MT-NLG 530B (see Table 7).
中文: 阅读理解 在最终词预测数据集LAMBADA(Paperno等,2016年)上,钦奇拉的精度为77.4%,而Gopher的精度为74.5%,MT-NLG530B的精度为76.6%(见表7)。
<a id="S0203"></a> Source: p.11 S0203
Original: On RACE-h and RACE-m (Lai et al., 2017), Chinchilla greatly outperforms Gopher, improving accuracy by more than 10% in both cases—see Table 7. 4.2.4.
中文: 在RACE-h和RACE-m(Lai等人,2017年)上,Chinchilla的性能大大超过Gopher,在这两种情况下的精度都提高了10%以上——见表7.4.2.4。
<a id="S0204"></a> Source: p.11 S0204
Original: BIG-bench We analysed Chinchilla on the same set of BIG-bench tasks (BIG-bench collaboration, 2021) reported in Rae et al. (2021).
中文: BIG-bench 我们分析了Chinchilla在Rae等人(2021年)所报道的同一套BIG-bench任务(BIG-bench合作,2021年).
<a id="S0205"></a> Source: p.11 S0205
Original: Similar to what we observed in MMLU, Chinchilla outperforms Gopher on the vast majority of tasks (see Figure 7).
中文: 与我们在MMLU中观察到的情况相类似,Chinchilla在绝大多数任务上的表现都超过了Gopher(见图7)。
<a id="S0206"></a> Source: p.11 S0206
Original: We find that Chinchilla improves the average performance by 10.7%, reaching an accuracy of 65.1% versus 54.4% for Gopher.
中文: 我们发现Chinchilla的平均性能提高了10.7%,其精度达到65.1%而Gopher的精度为54.4%.
<a id="S0207"></a> Source: p.11 S0207
Original: Of the 62 tasks we consider, Chinchilla performs worse than Gopher on only four—crash_blossom, dark_humor_detection, 11
中文: 在我们考虑的62项任务中, Chinchilla 的表现比 Gopher 差 仅仅在四个- crash blossom, 暗- humor detection, 11
<a id="S0208"></a> Source: p.12 S0208
Original: scitamehtam_egelloc scirtemonoce soiranecs_larom cigol_lamrof sciteneg_lacidem gninrael_enihcam snoitaler_cilbup stcaf_labolg scihte_ssenisub gnireenigne_lacirtcele ecneics_retupmoc_egelloc snoigiler_dlrow yrotsih_su_loohcs_hgih ygolohcysp_loohcs_hgih tnemeganam ecneics_retupmoc_loohcs_hgih gnitekram scisyhp_loohcs_hgih scimonoceorcam_loohcs_hgih ygoloicos scitilop_dna_tnemnrevog_loohcs_hgih yrotsih_naeporue_loohcs_hgih noitirtun enicidem_egelloc ymonortsa seicallaf_lacigol ygolohcysp_lanoisseforp suoenallecsim ecnedurpsiruj egdelwonk_lacinilc yhpargoeg_loohcs_hgih ygoloib_loohcs_hgih ygoloib_egelloc yrtsimehc_egelloc yrotsih_dlrow_loohcs_hgih ycilop_ngierof_su ygoloriv yhposolihp setupsid_larom gniga_namuh ytiruces_retupmoc seiduts_ytiruces wal_lanoitanretni scimonoceorcim_loohcs_hgih scitsitats_loohcs_hgih gnitnuocca_lanoisseforp enicidem_lanoisseforp yrotsiherp yrtsimehc_loohcs_hgih scitamehtam_yratnemele arbegla_tcartsba ymotana wal_lanoisseforp ytilauxes_namuh scisyhp_egelloc scitamehtam_loohcs_hgih scisyhp_lautpecnoc 30 20 10 0 10 tnemevorpmI evitaleR rehpoG revo Figure 6 | MMLU results compared to Gopher We find that Chinchilla outperforms Gopher by 7.6% on average (see Table 6) in addition to performing better on 51/57 individual tasks, the same on 2/57, and worse on only 4/57 tasks.
中文: 硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫磺酰氟-硫 与Gopher相比,我们发现Chinchilla的成绩平均比Gopher高出7.6%(见表6),此外,在551/57的个人任务上表现更好,在2/57的任务上表现更佳,在4/57的任务上表现更差。
<a id="S0209"></a> Source: p.12 S0209
Original: Chinchilla Gopher GPT-3 MT-NLG 530B LAMBADA Zero-Shot 77.4 74.5 76.2 76.6 RACE-m Few-Shot 86.8 75.1 58.1 - RACE-h Few-Shot 82.3 71.6 46.8 47.9 Table 7 | Reading comprehension.
中文: Chinchilla Gopher GPT-3 MT-NLG 530B LAMBADA 0-Shot 77.4 74.5 76.2 76.6 RACE-m 几-Shot 86.8 75.1 58.1 - RACE-h 几-Shot 82.3 71.6 46.8 47.9 表7 阅读理解。
<a id="S0210"></a> Source: p.12 S0210
Original: On RACE-h and RACE-m (Lai et al., 2017), Chinchilla considerably improves performance over Gopher.
中文: 在RACE-h和RACE-m上(Lai等,2017年),Chinchilla比Gopher的性能有相当大的提高.
<a id="S0211"></a> Source: p.12 S0211
Original: Note that GPT-3 and MT-NLG 530B use a different prompt format than we do on RACE-h/m, so results are not comparable to Gopher and Chinchilla.
中文: 注意GPT-3和MT-NLG 530B使用与我们在RACE-h/m上不同的即时格式,因此结果与Gopher和Chinchilla不相上下.
<a id="S0212"></a> Source: p.12 S0212
Original: On LAMBADA (Paperno et al., 2016), Chinchilla outperforms both Gopher and MT-NLG 530B. mathematical_induction and logical_args.
中文: 在LAMBADA上(Paperno等,2016年),Chinchilla在Gopher和MT-NLG 530B上的表现都优于Gopher和MT-NLG. Mathematic induction and logical args.
<a id="S0213"></a> Source: p.12 S0213
Original: Full accuracy results for Chinchilla can be found in Table A7. 4.2.5.
中文: Chinchilla的完全准确结果见表A7.4.2.5。
<a id="S0214"></a> Source: p.12 S0214
Original: Common sense We evaluate Chinchilla on various common sense benchmarks: PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2020), HellaSwag (Zellers et al., 2019), and BoolQ (Clark et al., 2019).
中文: 常识 我们根据各种常识基准评价钦奇拉:PIQA(Bisk等,2020年),SIQA(Sap等,2019年),Winogrande(坂口等,2020年),HelaSwag(Zellers等,2019年)和BoolQ(Clark等,2019年).
<a id="S0215"></a> Source: p.12 S0215
Original: We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B on all but one task—see Table 8.
中文: 我们发现Chinchilla在所有任务上都优于Gopher和GPT-3,在除一项任务外的所有任务上都优于MT-NLG 530B——见表8。
<a id="S0216"></a> Source: p.12 S0216
Original: On TruthfulQA (Lin et al., 2021), Chinchilla reaches 43.6%, 58.5%, and 66.7% accuracy with 0-shot, 5-shot, and 10-shot respectively.
中文: 在"真理QA"(Lin等,2021)上,钦奇拉以0发,5发和10发分别达到43.6%,58.5%和66.7%的精度.
<a id="S0217"></a> Source: p.12 S0217
Original: In comparison, Gopher achieved only 29.5% 0-shot and 43.7% 10-shot accuracy.
中文: 相形之下,高菲只实现了29.5%的0发和43.7%的10发精度.
<a id="S0218"></a> Source: p.12 S0218
Original: In stark contrast with the findings of Lin et al. (2021), the large improvements (14.1% in 0-shot accuracy) achieved by Chinchilla suggest that better modelling of the pre-training data alone can lead to substantial improvements on this benchmark. 12
中文: 与Lin等人(2021年)的调查结果形成鲜明对比的是,Chinchilla实现的大幅改进(以0发精度的14.1%)表明,单凭培训前数据的更好的建模就可以导致这一基准的重大改进. 第12条
<a id="S0219"></a> Source: p.13 S0219
Original: mossolb_hsarc noitceted_romuh_krad noitcudni_lacitamehtam sgra_lacigol nosj_egdelwonk_lareneg eciohc_elpitlum_sesnes_snagro_namuH noitagen_smsigollys_seicallaf_lamrof snwonknu_nwonk etagivan ytiugibma_ecnetnes ytilibissimrep_larom noitingocer_tnetni noitacifitnedi_ynori ytiralop_deliatne notabrepyh snoitpecnocsim ytilaitnesse_noitamrofni_gnitaulave noitcartsba_seitiralimis gninosaer_cimetsipe gninosaer_ysatnaf tnereffid_ro_emas_golaid_eivom yhwoniw stpecnoc_levon noitciderp_rekram_esruocsid aqygetarts tnemgduj_lasuac egdelwonk_udnih ssendetaler_esarhp eriannoitseuq_tnemngila stcejbo_deroloc_tuoba_gninosaer gnidnatsrednu_etad elbat_a_ni_sniugnep noitceted_hceeps_fo_erugif q_noitaugibmasid serutacilpmi SKRANS seman_niur noitceted_ycallaf_lacigol smsinorhcana elzzup_dirg_cigol esnes_elddir tnemliatne_citylana noitceles_noitseuq rammarg_sdrow_esnesnon cm_scisyhp stnemgduj_laciripme gnidnatsrednu_strops ia_ssarc noitiutni_lacisyhp laidemit snoitaler_ticilpmi sbrevorp_hsilgne iln_sa_snoitisoppuserp noitadnemmocer_eivom selbaf_gnidnatsrednu naeloob_rohpatem secneuqes_laropmet ecneuqes_lacigol rohpatem_ddo_yfitnedi noisneherpmoc_gnidaer_erg tuo_eno_ddo ytiralimis_lacigolana 120 100 80 60 40 20 0 20 tnemevorpmI evitaleR rehpoG revo Figure 7 | BIG-bench results compared to Gopher Chinchilla out performs Gopher on all but four BIG-bench tasks considered.
中文: russolb hsarc noitceted romuh kraid noitcudni lacitamtam sgra lacigol nosj egdelwonk lareneg eciohc elpitulum ses sinagro namigollys seicallaf lamro snowknu nomcang-nomcang-nomcineof-nomagturcit-s-nothamit-nouniguit-nit-nounimonigut-nit-nit-nitmonimonigut-nit-nitmonigut-nepit-nit-nominimongit-nit-nit-nitmonit-nit-nitmonit-nominogtuncentsucent-nomcent-nomcent-nomcent-nomcent-nomcent-nomcutcut-nocent- (原始内容存档于2018-10-29) (中文(中国大陆) ). Ednu strops ia sarc noitiutni lacisyhp snoit snoitaler ticilpmi sbrevorp hsilgne iln sa snoitisoppuserp noitadneimmocer eivom selbaf gnidnatsrednu nelooooob rohpatem scneuqes laropmet ecneuqes lacigol rohpatem ddo yfitnedi noisnephermoc gnidaer er erg too eno do ytirimis lacigolana). 120,100,80,60,40,20,20 tnemevorpmI evitaleR rehpoG revo 图7 相较于Gopher Chinchilla的BIG-bench结果,Gopher在除四个BIG-bench任务外的所有考虑中都完成了Gopher.
<a id="S0220"></a> Source: p.13 S0220
Original: Closed-book question answering Results on closed-book question answering benchmarks are reported in Table 9.
中文: 封闭式答题结果见表9。
<a id="S0221"></a> Source: p.13 S0221
Original: On the Natural Questions dataset (Kwiatkowski et al., 2019), Chinchilla achieves new closed-book SOTA accuracies: 31.5% 5-shot and 35.5% 64-shot, compared to 21% and 28% respectively, for Gopher.
中文: 在"自然问题"数据集(Kwiatkowski等,2019年)上,钦奇拉实现了新的封闭式SOTA加速:31.5%为5发和35.5%为64发,而Gopher分别为21%和28%.
<a id="S0222"></a> Source: p.13 S0222
Original: On TriviaQA (Joshi et al., 2017) we show results for both the filtered (previously used in retrieval and open-book work) and unfiltered set (previously used in large language model evaluations).
中文: 在TriviaQA(Joshi等人,2017年)上,我们显示过滤器(以前用于检索和开本工作)和未过滤器(以前用于大语言模型评价)的结果。
<a id="S0223"></a> Source: p.13 S0223
Original: In both cases, Chinchilla substantially out performs Gopher.
中文: 在这两种情况下,Chinchilla基本上都执行Gopher。
<a id="S0224"></a> Source: p.13 S0224
Original: On the filtered version, Chinchilla lags behind the open book SOTA (Izacard and Grave, 2020) by only 7.9%.
中文: 在被过滤的版本上,Chinchilla只落后于开放版SOTA(Izacard and Grave,2020)7.9%.
<a id="S0225"></a> Source: p.13 S0225
Original: On the unfiltered set, Chinchilla outperforms GPT-3—see Table 9. 4.2.7.
中文: 在未过滤的装置上,Chinchilla的性能超过了GPT-3——见表9.4.2.7。
<a id="S0226"></a> Source: p.13 S0226
Original: Gender bias and toxicity Large Language Models carry potential risks such as outputting offensive language, propagating social biases, and leaking private information (Bender et al., 2021; Weidinger et al., 2021).
中文: 性别偏见和毒性大语言模型具有潜在风险,如输出攻击性语言、宣传社会偏见和泄露私人信息(Bender等,2021;Weidinger等,2021)。
<a id="S0227"></a> Source: p.13 S0227
Original: We expect Chinchilla to carry risks similar to Gopher because Chinchilla is trained on the same data, Chinchilla Gopher GPT-3 MT-NLG 530B Supervised SOTA HellaSWAG 80.8% 79.2% 78.9% 80.2% 93.9% PIQA 81.8% 81.8% 81.0% 82.0% 90.1% Winogrande 74.9% 70.1% 70.2% 73.0% 91.3% SIQA 51.3% 50.6% - - 83.2% BoolQ 83.7% 79.3% 60.5% 78.2% 91.4% Table 8 | Zero-shot comparison on Common Sense benchmarks.
中文: 我们期望钦奇拉承担类似戈佛的风险,因为钦奇拉接受过同样的数据培训,钦奇拉Gopher GPT-3 MT-NLG 530B 监督SOTA HellaSWAG 80.8% 79.2% 78.9% 80.2% 93.9% PIQA 81.8% 81.0% 82.0% 维诺格兰德 74.9% 70.1% 73.0% 91.3% SIQA 51.6% - 83.2% BoolQ 83.7% 79.3% 60.5% 78.2% 91.4% 表8 普通森斯基准的零拍比较。
<a id="S0228"></a> Source: p.13 S0228
Original: We show a comparison between Chinchilla, Gopher, and MT-NLG 530B on various Common Sense benchmarks.
中文: 我们在各种共同理智基准上对Chinchilla,Gopher和MT-NLG 530B进行了比较。
<a id="S0229"></a> Source: p.13 S0229
Original: We see that Chinchilla matches or outperforms Gopher and GPT-3 on all tasks.
中文: 我们发现钦奇拉在所有任务上都比Gopher和GPT-3有匹配或表现.
<a id="S0230"></a> Source: p.13 S0230
Original: On all but one Chinchilla outperforms the much larger MT-NLG 530B model. 13
中文: 除了一个Chinchilla之外,它比更大的MT-NLG 530B型号要好。 第13条
<a id="S0231"></a> Source: p.14 S0231
Original: Method Chinchilla Gopher GPT-3 SOTA (open book) 0-shot 16.6% 10.1% 14.6% Natural Questions (dev) 5-shot 31.5% 24.5% - 54.4% 64-shot 35.5% 28.2% 29.9% 0-shot 67.0% 52.8% 64.3 % TriviaQA (unfiltered, test) 5-shot 73.2% 63.6% - - 64-shot 72.3% 61.3% 71.2% 0-shot 55.4% 43.5% - TriviaQA (filtered, dev) 5-shot 64.1% 57.0% - 72.5% 64-shot 64.6% 57.2% - Table 9 | Closed-book question answering.
中文: Method Chinchilla Gopher GPT-3 SOTA(开口书) 0射出16.6% 10.1% 14.6% 自然问题(德文) 5射出31.5% 24.5% - 54.4% 64射出35.5% 28.2% 0射出67.0% 52.8% TriviaQA(未过滤,测试) 5射出73.2% - 63.6% - 64射出72.3% - 61.3% 71.2% 0射出55.4% - TriviaQA(被过滤,德文) 5射出64.1% - 57.0% - 72.5% 64-shot 64.6% 57.2% - 表9 封闭图书问题回答。
<a id="S0232"></a> Source: p.14 S0232
Original: For Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017), Chinchilla outperforms Gopher in all cases.
中文: 对于自然问题(Kwiatkowski等,2019年)和TriviaQA(Joshi等,2017年),Chinchilla优于成绩. 在所有情况下都是Gopher
<a id="S0233"></a> Source: p.14 S0233
Original: On Natural Questions, Chinchilla outperforms GPT-3.
中文: 关于自然问题,Chinchilla的成绩超过了GPT-3.
<a id="S0234"></a> Source: p.14 S0234
Original: On TriviaQA we show results on two different evaluation sets to allow for comparison to GPT-3 and to open book SOTA (FiD + Distillation (Izacard and Grave, 2020)). albeit with slightly different relative weights, and because it has a similar architecture.
中文: 在TriviaQA上,我们展示了两套不同评价的结果,以便与GPT-3进行比较,并打开了SOTA(FID + Distillation(Izacard and Grave, 2020))。 虽然相对重量稍有不同 因为它的建筑结构相似
<a id="S0235"></a> Source: p.14 S0235
Original: Here, we examine gender bias (particularly gender and occupation bias) and generation of toxic language.
中文: 在此,我们审查性别偏见(特别是性别和职业偏见)和产生有毒语言的问题。
<a id="S0236"></a> Source: p.14 S0236
Original: We select a few common evaluations to highlight potential issues, but stress that our evaluations are not comprehensive and much work remains to understand, evaluate, and mitigate risks in LLMs.
中文: 我们选择了几项共同评价,以突出潜在的问题,但强调我们的评价并不全面,还有许多工作要做,以了解、评价和减轻有限责任方案的风险。
<a id="S0237"></a> Source: p.14 S0237
Original: As discussed in Rae et al. (2021), large language models reflect contemporary and historical discourse about different groups (such as gender groups) from their training dataset, and we expect the same to be true for Chinchilla.
中文: 如Rae等人(2021年)所讨论,大型语言模型从培训数据集中反映了当代和历史对不同群体(如性别群体)的论述,我们期望钦奇利亚也如此.
<a id="S0238"></a> Source: p.14 S0238
Original: Here, we test if potential gender and occupation biases manifest in unfair outcomes on coreference resolutions, using the Winogender dataset (Rudinger et al., 2018) in a zero-shot setting.
中文: 在此,我们用Winogender数据集(Rudinger等,2018年),在零镜头设定下,测试是否在对校对结果的不公平结果中表现出潜在的性别和职业偏见。
<a id="S0239"></a> Source: p.14 S0239
Original: Winogender tests whether a model can correctly determine if a pronoun refers to different occupation words.
中文: Winogender测试一个模型是否能够正确确定一个代词是否指不同的职业词.
<a id="S0240"></a> Source: p.14 S0240
Original: An unbiased model would correctly predict which word the pronoun refers to regardless of pronoun gender.
中文: 一个没有偏见的模型可以正确预测代词指的是哪个词,而不论代词性别.
<a id="S0241"></a> Source: p.14 S0241
Original: We follow the same setup as in Rae et al. (2021) (described further in Section H.3).
中文: 我们遵循与Rae等人案(2021年)相同的安排(H.3节进一步说明)。
<a id="S0242"></a> Source: p.14 S0242
Original: As shown in Table 10, Chinchilla correctly resolves pronouns more frequently than Gopher across all groups.
中文: 如表10所显示,钦奇拉在所有群体中正确解决代词比Gopher更频繁.
<a id="S0243"></a> Source: p.14 S0243
Original: Interestingly, the performance increase is considerably smaller for male pronouns (increase of 3.2%) than for female or neutral pronouns (increases of 8.3% and 9.2% respectively).
中文: 有趣的是,男性代名词的性能增长(增加3.2%)大大低于女性或中性代名词(分别增加8.3%,9.2%)。
<a id="S0244"></a> Source: p.14 S0244
Original: We also consider gotcha examples, in which the correct pronoun resolution contradicts gender stereotypes (determined by labor statistics).
中文: 我们还考虑一些例子,其中正确的代名词决议与性别陈规定型观念相矛盾(由劳工统计决定)。
<a id="S0245"></a> Source: p.14 S0245
Original: Again, we see that Chinchilla resolves pronouns more accurately than Gopher.
中文: 我们再次看到 Chinchilla 解决代词比Gopher更准确。
<a id="S0246"></a> Source: p.14 S0246
Original: When breaking up examples by male/female gender and gotcha/not gotcha, the largest improvement is on female gotcha examples (improvement of 10%).
中文: 在按性别和性别分列实例时,最大的改进是女性实例(改进了10%)。
<a id="S0247"></a> Source: p.14 S0247
Original: Thus, though Chinchilla uniformly overcomes gender stereotypes for more coreference examples than Gopher, the rate of improvement is higher for some pronouns than others, suggesting that the improvements conferred by using a more compute-optimal model can be uneven.
中文: 因此,虽然钦奇拉统一克服了性别定型观念,比戈弗更能参照实例,但某些代名词的改进率比其他代名词要高,这表明使用更计算-最佳模式带来的改进可能是不均衡的。
<a id="S0248"></a> Source: p.14 S0248
Original: Language models are capable of generating toxic language—including insults, hate speech, profanities and threats (Gehman et al., 2020; Rae et al., 2021).
中文: 语言模型能够产生有毒语言——包括侮辱、仇恨言论、亵渎和威胁(Gehman等,2020年;Rae等,2021年)。
<a id="S0249"></a> Source: p.14 S0249
Original: While toxicity is an umbrella term, and its evaluation in LMs comes with challenges (Welbl et al., 2021; Xu et al., 2021), automatic classifier scores can provide an indication for the levels of harmful text that a LM generates.
中文: 虽然毒性是一个总括术语,但其在LMs中的评价面临挑战(Welbl等人,2021年;Xu等人,2021年),自动分级器的分数可以为LM生成的有害文本的水平提供指示.
<a id="S0250"></a> Source: p.14 S0250
Original: Rae et al. (2021) found that improving language modelling loss by increasing the number of model parameters has only a negligible effect on toxic text generation (unprompted); here we analyze 14
中文: Rae等人(2021年)认为,通过增加模型参数数量来改进语言建模损失,对有毒文本生成的影响可忽略不计(未及时);我们在此分析14
<a id="S0251"></a> Source: p.15 S0251
Original: Chinchilla Gopher Chinchilla Gopher All 78.3% 71.4% Male gotcha 62.5% 59.2% Male 71.2% 68.0% Male not gotcha 80.0% 76.7% Female 79.6% 71.3% Female gotcha 76.7% 66.7% Neutral 84.2% 75.0% Female not gotcha 82.5% 75.8% Table 10 | Winogender results.
中文: Chinchilla Gopher Chinchilla Gopher全部78.3% 71.4% 男性获得查 62.5% 59.2% 男性获得查 80.0% 76.7% 女性获得查 79.6% 71.3% 女性获得查 76.7% 中立 84.2% 75.0% 女性获得查 82.5% 75.8% 表10 o 维诺性别结果。
<a id="S0252"></a> Source: p.15 S0252
Original: Left: Chinchilla consistently resolves pronouns better than Gopher.
中文: 左:Chinchilla一致解决代词比Gopher更好.
<a id="S0253"></a> Source: p.15 S0253
Original: Right: Chinchilla performs better on examples which contradict gender stereotypes (gotcha examples).
中文: 右:钦奇拉在与性别陈规定型观念相矛盾的例子(gotcha实例)上表现得更好.
<a id="S0254"></a> Source: p.15 S0254
Original: However, difference in performance across groups suggests Chinchilla exhibits bias. whether the same holds true for a lower LM loss achieved via more compute-optimal training.
中文: 然而,各群体的业绩差异表明钦奇拉表现出了偏见。 通过更优化的计算培训实现的较低LM损失是否同样如此。
<a id="S0255"></a> Source: p.15 S0255
Original: Similar to the protocol of Rae et al. (2021), we generate 25,000 unprompted samples from Chinchilla, and compare their PerspectiveAPI toxicity score distribution to that of Gopher-generated samples.
中文: 与Rae等人(2021年)的协议类似,我们从Chinchilla生成出25,000个未发作的样本,并将其PerspectAPI毒性分数分布与Gopher生成的样本进行比较.
<a id="S0256"></a> Source: p.15 S0256
Original: Several summary statistics indicate an absence of major differences: the mean (median) toxicity score for Gopher is 0.081 (0.064), compared to 0.087 (0.066) for Chinchilla, and the 95th percentile scores are 0.230 for Gopher, compared to 0.238 for Chinchilla.
中文: 一些汇总统计表明没有重大差异:戈菲的平均(中间)毒性分数为0.081分(0.064分),而钦奇拉的毒性分数为0.087分(0.066分),戈菲的毒性分数为0.230分,而钦奇拉的毒性分数为0.238分.
<a id="S0257"></a> Source: p.15 S0257
Original: That is, the large majority of generated samples are classified as non-toxic, and the difference between the models is negligible.
中文: 也就是说,绝大多数生成的样品被归类为无毒,模型之间的差别可以忽略不计.
<a id="S0258"></a> Source: p.15 S0258
Original: In line with prior findings (Rae et al., 2021), this suggests that toxicity levels in unconditional text generation are largely independent of the model quality (measured in language modelling loss), i.e. that better models of the training dataset are not necessarily more toxic. 5.
中文: 根据先前的调查结果(Rae等人,2021年),这表明无条件文本生成中的毒性水平在很大程度上独立于模型质量(以语言建模损失衡量),即更好的培训数据集模型不一定毒性更强. 5 (韩语).
<a id="S0259"></a> Source: p.15 S0259
Original: Discussion & Conclusion The trend so far in large language model training has been to increase the model size, often without increasing the number of training tokens.
中文: 讨论和结论 迄今为止,大型语言模型培训的趋势是扩大模型规模,往往不增加培训标志的数量。
<a id="S0260"></a> Source: p.15 S0260
Original: The largest dense transformer, MT-NLG 530B, is now over 3× larger than GPT-3’s 170 billion parameters from just two years ago.
中文: 最大的密集变压器MT-NLG 530B,现在比仅仅两年前GPT-3的1700亿参数大了3×.
<a id="S0261"></a> Source: p.15 S0261
Original: However, this model, as well as the majority of existing large models, have all been trained for a comparable number of tokens—around 300 billion.
中文: 然而,这一模式以及大多数现有大型模式都接受了相当数量(约3 000亿美元)的培训。
<a id="S0262"></a> Source: p.15 S0262
Original: While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget.
中文: 虽然训练这些特大模型的愿望导致了大量的工程创新,但我们假设训练更大和更大的模型的竞赛导致模型的性能大大低于通过同样的计算预算可以实现的模型。
<a id="S0263"></a> Source: p.15 S0263
Original: We propose three predictive approaches towards optimally setting model size and training duration, based on the outcome of over 400 training runs.
中文: 根据400多次培训的结果,我们提出三种预测办法,以优化模式规模和培训期限。
<a id="S0264"></a> Source: p.15 S0264
Original: All three approaches predict that Gopher is substantially over-sized and estimate that for the same compute budget a smaller model trained on more data will perform better.
中文: 所有三种方法都预测,Gopher的体积大大过大,并估计,就同一计算预算而言,在更多数据方面受过训练的较小的模型将表现更好。
<a id="S0265"></a> Source: p.15 S0265
Original: We directly test this hypothesis by training Chinchilla, a 70B parameter model, and show that it outperforms Gopher and even larger models on nearly every measured evaluation task.
中文: 我们通过训练一个70B参数模型的Chinchilla来直接测试这个假说, 并表明它比Gopher甚至更大的模型在几乎每个测量的评价任务上都表现得更好。
<a id="S0266"></a> Source: p.15 S0266
Original: Whilst our method allows us to make predictions on how to scale large models when given additional compute, there are several limitations.
中文: 虽然我们的方法允许我们预测在额外计算时如何放大大型模型,但有一些局限性。
<a id="S0267"></a> Source: p.15 S0267
Original: Due to the cost of training large models, we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests at intermediate scales.
中文: 由于训练大型型号的费用,我们只有两次规模相近的训练(Chinchilla和Gopher),我们没有在中间规模进行额外的试验.
<a id="S0268"></a> Source: p.15 S0268
Original: Furthermore, we assume that the efficient computational frontier can be described by a power-law relationship between the compute budget, model size, and number of training tokens.
中文: 此外,我们假定,计算预算、模型规模和培训符号数量之间的权力法关系可以说明有效的计算前沿。
<a id="S0269"></a> Source: p.15 S0269
Original: However, we observe some concavity in log (cid:0)𝑁 (cid:1) at high compute budgets 𝑜𝑝𝑡 (see Appendix E).
中文: 然而,我们看到,在高计算预算的选择(见附录E)中,对数(cid:0)N(cid:1)有些精确。
<a id="S0270"></a> Source: p.15 S0270
Original: This suggests that we may still be overestimating the optimal size of large models.
中文: 这表明,我们仍然在高估大型模型的最佳尺寸。
<a id="S0271"></a> Source: p.15 S0271
Original: Finally, the training runs for our analysis have all been trained on less than an epoch of data; future work may consider the multiple epoch regime.
中文: 最后,我们分析的培训活动都接受了不到一个时代的数据培训;今后的工作可能考虑到一个时代的多重制度。
<a id="S0272"></a> Source: p.15 S0272
Original: Despite these limitations, the comparison of Chinchilla to Gopher validates our performance predictions, that have thus enabled training a better (and more 15
中文: 尽管存在这些局限性,将Chinchilla与Gopher作一比较,验证了我们的性能预测,从而使培训更加完善(而培训次数为15人以上)。
<a id="S0273"></a> Source: p.16 S0273
Original: lightweight) model at the same compute budget.
中文: 轻量级)模型在同一计算预算.
<a id="S0274"></a> Source: p.16 S0274
Original: Though there has been significant recent work allowing larger and larger models to be trained, our analysis suggests an increased focus on dataset scaling is needed.
中文: 虽然最近开展了大量工作,使更大和更大的模型得到培训,但我们的分析表明,需要更加重视数据集的缩放。
<a id="S0275"></a> Source: p.16 S0275
Original: Speculatively, we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality.
中文: 简而言之,我们期望,只有数据质量高,才能将数据放大到更大和更大的数据集。
<a id="S0276"></a> Source: p.16 S0276
Original: This calls for responsibly collecting larger datasets with a high focus on dataset quality.
中文: 这就要求负责任地收集更大的数据集,并高度注重数据集的质量。
<a id="S0277"></a> Source: p.16 S0277
Original: Larger datasets will require extra care to ensure train-test set overlap is properly accounted for, both in the language modelling loss but also with downstream tasks.
中文: 更大的数据集需要额外注意,以确保在语言建模损失以及在下游任务中适当考虑到列车测试集的重叠。
<a id="S0278"></a> Source: p.16 S0278
Original: Finally, training for trillions of tokens introduces many ethical and privacy concerns.
中文: 最后,对数以万计的象征物的培训提出了许多伦理和隐私问题。
<a id="S0279"></a> Source: p.16 S0279
Original: Large datasets scraped from the web will contain toxic language, biases, and private information.
中文: 从网上取出的大量数据集将包含有毒语言、偏见和私人信息。
<a id="S0280"></a> Source: p.16 S0280
Original: With even larger datasets being used, the quantity (if not the frequency) of such information increases, which makes dataset introspection all the more important.
中文: 随着使用更大的数据集,这类信息的数量(如果不是频率的话)会增加,这使得数据集的回顾更加重要.
<a id="S0281"></a> Source: p.16 S0281
Original: Chinchilla does suffer from bias and toxicity but interestingly it seems less affected than Gopher.
中文: Chinchilla确实有偏见和毒性,但有趣的是,它的影响似乎低于Gopher。
<a id="S0282"></a> Source: p.16 S0282
Original: Better understanding how performance of large language models and toxicity interact is an important future research question.
中文: 更好地理解大语言模型和毒性的性能如何相互作用是未来重要的研究问题.
<a id="S0283"></a> Source: p.16 S0283
Original: While we have applied our methodology towards the training of auto-regressive language models, we expect that there is a similar trade-off between model size and the amount of data in other modalities.
中文: 虽然我们在培训自动递归语言模型方面采用了我们的方法,但我们期望模型大小与其他模式中的数据数量之间有类似的取舍。
<a id="S0284"></a> Source: p.16 S0284
Original: As training large models is very expensive, choosing the optimal model size and training steps beforehand is essential.
中文: 由于培训大型模式的费用非常高,必须事先选择最佳模式的规模和培训步骤。
<a id="S0285"></a> Source: p.16 S0285
Original: The methods we propose are easy to reproduce in new settings. 6.
中文: 我们提出的方法很容易在新的环境中被复制。 6. 国家
<a id="S0286"></a> Source: p.16 S0286
Original: Acknowledgements We’d like to thank Jean-baptiste Alayrac, Kareem Ayoub, Chris Dyer, Nando de Freitas, Demis Hassabis, Geoffrey Irving, Koray Kavukcuoglu, Nate Kushman and Angeliki Lazaridou for useful comments on the manuscript.
中文: 鸣谢 我们想感谢让-巴蒂斯特·阿莱拉克、卡雷姆·阿尤布、克里斯·戴尔、南多·德·弗雷塔斯、德米斯·哈萨比斯、杰弗里·伊尔文、克赖·卡武克库格卢、内特·库什曼和安杰利基·拉扎里杜对手稿的有益评论。
<a id="S0287"></a> Source: p.16 S0287
Original: We’d like to thank Andy Brock, Irina Higgins, Michela Paganini, Francis Song, and other colleagues at DeepMind for helpful discussions.
中文: 我们要感谢安迪·布洛克、伊琳娜·希金斯、米凯拉·帕格尼尼、弗朗西斯·宋等在DeepMind的同事,
<a id="S0288"></a> Source: p.16 S0288
Original: We are also very grateful to the JAX and XLA team for their support and assistance.
中文: 我们也非常感谢JAX和XLA团队的支持和援助.
<a id="S0289"></a> Source: p.16 S0289
Original: Efficient Large Scale Language Modeling with Mixtures of Experts. arXiv:2112.10684, 2021. E. M.
中文: 高效的大尺度语言建模与专家相混合. arXiv:2112.10684, 2021. E. M.
<a id="S0290"></a> Source: p.16 S0290
Original: On the dangers of stochastic parrots: Can language models be too big?
中文: 论斯多克鹦鹉的危害:语言模型能否太大?.
<a id="S0291"></a> Source: p.16 S0291
Original: In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, 2021.
中文: 2021年ACM公平性、问责制和透明度会议记录,第610-623页,2021年。
<a id="S0292"></a> Source: p.16 S0292
Original: Beyond the imitation game: Measuring and extrapolating the capabilities of language models.
中文: 超越模仿游戏:测量和推断语言模型的能力.
<a id="S0293"></a> Source: p.16 S0293
Original: URL https://github.com/google/BIG-bench/. Y.
中文: URL https://github.com/google/BIG-bench/. 互联网档案馆的存檔,存档日期2013-12-02. 没错
<a id="S0294"></a> Source: p.16 S0294
Original: PIQA: Reasoning about physical commonsense in natural language.
中文: PIQA:自然语言中物理常识的原因.
<a id="S0295"></a> Source: p.16 S0295
Original: In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7432–7439, 2020. S.
中文: AAAI人工智能会议记录 第34卷 第7432-7439页,2020年. 师曰.
<a id="S0296"></a> Source: p.16 S0296
Original: Improving language models by retrieving from trillions of tokens. arXiv 2112.04426, 2021. 16
中文: 通过取回万亿代币来改进语言模型. arXiv 2112.04426, 2021 (英语). 16
<a id="S0297"></a> Source: p.17 S0297
Original: JAX: composable transformations of Python+NumPy programs. 2018.
中文: JAX:Python+NumPy程序的可复合变换. 2018 (英语).
<a id="S0298"></a> Source: p.17 S0298
Original: Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901.
中文: Lin,编辑,神经信息处理系统的进步, 第33卷 第1877–1901页.
<a id="S0299"></a> Source: p.17 S0299
Original: URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb49674 18bfb8ac142f64a-Paper.pdf. S.
中文: URL https://proceedings.neurips.cc/paper/2020/file/1457c0d6bcbbb49674 18bfb8ac142f64a-Paper.pdf. 师曰.
<a id="S0300"></a> Source: p.17 S0300
Original: Convex Optimization: Algorithms and Complexity.
中文: Convex优化:算法与复杂.
<a id="S0301"></a> Source: p.17 S0301
Original: Foundations and Trends in Machine Learning, 8(3-4):231–357, 2015.
中文: 机器学习的基础与趋势, 8 (3-4):231–357, 2015.
<a id="S0302"></a> Source: p.17 S0302
Original: URL http://www.nowpublishers.com/article/Detail s/MAL-050. A.
中文: URL http://www.nowpublishers.com/article/Detail s/MAL-050. A. 互联网档案馆的存檔,存档日期2011-12-20.
<a id="S0303"></a> Source: p.17 S0303
Original: Unified scaling laws for routed language models, 2022.
中文: 2022年路由语言模型统一缩放定律.
<a id="S0304"></a> Source: p.17 S0304
Original: URL https://arxiv.org/abs/2202.01169. C.
中文: URL: https://arxiv.org/abs/2202.01169. C. URL.
<a id="S0305"></a> Source: p.17 S0305
Original: Boolq: Exploring the surprising difficulty of natural yes/no questions.
中文: Boulq:探索自然是/不是问题的出人意料的困难.
<a id="S0306"></a> Source: p.17 S0306
Original: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, 2019. N.
中文: 计算语言学协会北美分会2019年会议记录:人文技术 第1卷 (长篇和短篇论文) 第2924–2936, 2019页. 编号
<a id="S0307"></a> Source: p.17 S0307
Original: Glam: Efficient scaling of language models with mixture-of-experts, 2021.
中文: Glam: 2021年与多位专家混合语言模型的高效缩放.
<a id="S0308"></a> Source: p.17 S0308
Original: URL https://arxiv.org/abs/2112.06905. W.
中文: URL: https://arxiv.org/abs/21120.6905. W.
<a id="S0309"></a> Source: p.17 S0309
Original: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021. L.
中文: 切换变压器:以简单而高效的相接力放大至万亿参数模型. arXiv预印 arXiv:2101.03961,2021. 维基月球在线解说-李克文陨石坑 Andersson, L.
<a id="S0310"></a> Source: p.17 S0310
Original: The Pile: An 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. S.
中文: Pile:用于语言建模的由多种文字组成的800GB数据集. arXiv预印 arXiv:2101.00027,2020年. 师曰.
<a id="S0311"></a> Source: p.17 S0311
Original: RealToxicityPrompts: Evaluating neural toxic degeneration in language models.
中文: RealToxicityPrompts:在语言模型中评价神经毒性分解.
<a id="S0312"></a> Source: p.17 S0312
Original: In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, Nov. 2020.
中文: 《计算语言学协会的调查结果:EMNLP 2020》,第3356-3369页,Online,2020年11月。
<a id="S0313"></a> Source: p.17 S0313
Original: Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.301.
中文: 计算语言学协会. doi:10.18653/v1/2020. finders-emnlp.301.
<a id="S0314"></a> Source: p.17 S0314
Original: URL https://aclanthology.org/2 020.findings-emnlp.301. K.
中文: URL https://aclanthology.org/2 020.finances-emnlp.301. K.
<a id="S0315"></a> Source: p.17 S0315
Original: REALM: Retrieval-augmented language model pre-training, 2020. D.
中文: REALM:检索-增强语言模型预训,2020. D类.
<a id="S0316"></a> Source: p.17 S0316
Original: Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. T.
中文: 测量大量多任务语言理解. arXiv预印 arXiv:2009.03300, 2020. T.
<a id="S0317"></a> Source: p.17 S0317
Original: URL http: //github.com/deepmind/dm-haiku. 17
中文: URL http://github.com/deepmind/dm-haiku. 17 (英语).
<a id="S0318"></a> Source: p.18 S0318
Original: Robust Estimation of a Location Parameter.
中文: 精确估计一个位置参数。
<a id="S0319"></a> Source: p.18 S0319
Original: The Annals of Mathematical Statistics, 35 (1):73–101, Mar. 1964.
中文: 《数学统计年鉴》,35(1):73-101,1964年3月。
<a id="S0320"></a> Source: p.18 S0320
Original: ISSN 0003-4851, 2168-8990. doi: 10.1214/aoms/1177703732.
中文: ISSN 0003-4851, 2168-8990. doi:10.1214/aoms/11777732.
<a id="S0321"></a> Source: p.18 S0321
Original: URL https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol ume-35/issue-1/Robust-Estimation-of-a-Location-Parameter/10.1214/aoms/11 77703732.full. G.
中文: URL https://projecteuclid.org/journals/annals-of-mathematical-statistics/vol ume-35//issues-1. Robst-Estimation-of-a-Location-Parameter/10.1214/aoms/11 77703732. full. G.
<a id="S0322"></a> Source: p.18 S0322
Original: Distilling knowledge from reader to retriever for question answering, 2020. M.
中文: (原始内容存档于2018-10-21). Dispilling knows from readers to retrievers for question answer, 2020. M.
<a id="S0323"></a> Source: p.18 S0323
Original: TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.03551, 2017. N. P.
中文: TriviaQA:一个大型的Distant监督挑战数据集,用于阅读Comference. arXiv e-prints, Art. arXiv:1705.03551, 2017. N. P.
<a id="S0324"></a> Source: p.18 S0324
Original: In-datacenter performance analysis of a tensor processing unit.
中文: 一个收发器处理单元的内数据中心性能分析。
<a id="S0325"></a> Source: p.18 S0325
Original: In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, page 1–12, New York, NY, USA, 2017.
中文: 计算机架构第44届国际年度专题讨论会记录,ISCA ' 17,第1-12页,美国纽约州纽约,2017年。
<a id="S0326"></a> Source: p.18 S0326
Original: ISBN 9781450348928. doi: 10.1145/3079856.3080246.
中文: ISBN 97814530348928. doi:10.1145/3079856.3080246 (中文(简体) ).
<a id="S0327"></a> Source: p.18 S0327
Original: URL https://doi.org/10.1145/3079856.3080246. J.
中文: URL https://doi.org/10.1145/3079856.3080246. J.
<a id="S0328"></a> Source: p.18 S0328
Original: Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. D. P.
中文: 扩展神经语言模型的定律. arXiv预印版arXiv:2001.08361,2020. D. P.
<a id="S0329"></a> Source: p.18 S0329
Original: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. T.
中文: 亚当:一种花样优化的方法. arXiv preprint arXiv:1412.6980, 2014 (英语). T.
<a id="S0330"></a> Source: p.18 S0330
Original: SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. T.
中文: PriedPiece:用于神经文本处理的简单而语言独立的子词代号器和解接器. arXiv preprint arXiv:1808.06226, 2018 (英语). T.
<a id="S0331"></a> Source: p.18 S0331
Original: Natural questions: a benchmark for question answering research.
中文: 自然问题:回答问题研究的基准.
<a id="S0332"></a> Source: p.18 S0332
Original: Transactions of the Association of Computational Linguistics, 2019. G.
中文: 2019年计算语言学协会交易. G.
<a id="S0333"></a> Source: p.18 S0333
Original: RACE: Large-scale ReAding comprehension dataset from examinations.
中文: RACE:通过考试获得的大规模重排理解数据集.
<a id="S0334"></a> Source: p.18 S0334
Original: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, Sept. 2017.
中文: 《2017年自然语言处理经验方法会议记录》,第785-794页,丹麦哥本哈根,2017年9月。
<a id="S0335"></a> Source: p.18 S0335
Original: Association for Computational Linguistics. doi: 10.18653/v1/D17-1082.
中文: 计算语言学协会. doi:10.18653/v1/D17-1082.
<a id="S0336"></a> Source: p.18 S0336
Original: URL https://aclanthology.org/D17-1082. Y.
中文: URL https://aclanthology.org/D17-1082. Y. 互联网档案馆的存檔,存档日期2013-12-02.
<a id="S0337"></a> Source: p.18 S0337
Original: The depth-to-width interplay in self-attention. arXiv preprint arXiv:2006.12467, 2020. P.
中文: 深至相接自心相通. arXiv预印版 arXiv:2006.12467, 2020. P.
<a id="S0338"></a> Source: p.18 S0338
Original: Retrieval-augmented generation for knowledge-intensive nlp tasks.
中文: 用于知识密集型nlp任务的检索强化生成.
<a id="S0339"></a> Source: p.18 S0339
Original: In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. 18
中文: 《神经信息处理系统的进展》,第33卷,第9459-9474页,2020年。
<a id="S0340"></a> Source: p.19 S0340
Original: Jurassic-1: Technical details and evaluation.
中文: 侏罗纪-1:技术细节与评价.
<a id="S0341"></a> Source: p.19 S0341
Original: TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. I.
中文: 真实QA:测量模型如何模仿人类的虚伪. arXiv预印 arXiv:2109.07958,2021. 说吧
<a id="S0342"></a> Source: p.19 S0342
Original: In International Conference on Learning Representations, 2019.
中文: 2019年国际学习代表大会.
<a id="S0343"></a> Source: p.19 S0343
Original: URL https://openreview.net/forum?id=Bkg6RiCqY7. S.
中文: URL https://openreview.net/forum?id=Bkg6RiCqY7. S.
<a id="S0344"></a> Source: p.19 S0344
Original: An empirical model of large-batch training, 2018. S.
中文: 2018年大型批斗训练经验模型. 师曰.
<a id="S0345"></a> Source: p.19 S0345
Original: International Conference on Learning Representations, 2017. M.
中文: 学习代表性国际会议,2017. M.
<a id="S0346"></a> Source: p.19 S0346
Original: In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019. J.
中文: 关于公平、问责制和透明度的会议记录,第220-229页,2019年。 J.
<a id="S0347"></a> Source: p.19 S0347
Original: Updating Quasi-Newton Matrices with Limited Storage.
中文: 更新有有限存储的Quasi-Newton Matries。
<a id="S0348"></a> Source: p.19 S0348
Original: Mathematics of Computation, 35(151):773–782, 1980.
中文: 计算数学, 35 (151): 773–782, 1980.
<a id="S0349"></a> Source: p.19 S0349
Original: URL https://www.jstor. org/stable/2006193. D.
中文: URL https://www.jstor.org/stable/2006193. D. 互联网档案馆的存檔,存档日期2011-12-21.
<a id="S0350"></a> Source: p.19 S0350
Original: The LAMBADA dataset: Word prediction requiring a broad discourse context, 2016. J.
中文: LAMBADA数据集:需要广泛讨论背景的文字预测, 2016. J.
<a id="S0351"></a> Source: p.19 S0351
Original: Scaling language models: Methods, analysis & insights from training Gopher. arXiv 2112.11446, 2021. J. W.
中文: 扩展语言模型:从培训Gopher. arXiv 2112.11446, 2021. J. W.
<a id="S0352"></a> Source: p.19 S0352
Original: Compressive transformers for long-range sequence modelling.
中文: 用于远程序列建模的压缩变压器.
<a id="S0353"></a> Source: p.19 S0353
Original: Advances in Neural Information Processing Systems, 33:6154–6158, 2020. C.
中文: 神经信息处理系统的进步, 33:6154-6158, 2020. C C.
<a id="S0354"></a> Source: p.19 S0354
Original: Exploring the limits of transfer learning with a unified text-to-text transformer.
中文: 探索采用统一的文本对文本转换器进行传输学习的极限.
<a id="S0355"></a> Source: p.19 S0355
Original: Journal of Machine Learning Research, 21(140):1–67, 2020a.
中文: 机器学习研究杂志, 21(140): 1–67, 2020a.
<a id="S0356"></a> Source: p.19 S0356
Original: URL http://jmlr.org/papers/v21/20-074.html. C.
中文: URL http://jmlr.org/papers/v21/20-074.html (中文(简体) ). C C.
<a id="S0357"></a> Source: p.19 S0357
Original: Exploring the limits of transfer learning with a unified text-to-text transformer.
中文: 探索采用统一的文本对文本转换器进行传输学习的极限.
<a id="S0358"></a> Source: p.19 S0358
Original: Journal of Machine Learning Research, 21(140):1–67, 2020b. S.
中文: 机器学习研究杂志, 21(140): 1–67, 2020b. S.
<a id="S0359"></a> Source: p.19 S0359
Original: Zero: Memory optimizations toward training trillion parameter models.
中文: 0:内存优化向培训一万亿参数模型.
<a id="S0360"></a> Source: p.19 S0360
Original: In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16.
中文: SC20: 高性能计算、联网、储存和分析国际会议,第1至16页。
<a id="S0361"></a> Source: p.20 S0361
Original: Monro. A Stochastic Approximation Method.
中文: 门罗. 一种相形相形法.
<a id="S0362"></a> Source: p.20 S0362
Original: The Annals of Mathematical Statistics, 22(3):400–407, Sept. 1951. R.
中文: 数学统计年鉴, 22(3):400–407, September. 1951. R.
<a id="S0363"></a> Source: p.20 S0363
Original: In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, June 2018.
中文: 计算语言学协会北美分会2018年会议纪要:人文语言技术,2018年6月于路易斯安那州新奥尔良.
<a id="S0364"></a> Source: p.20 S0364
Original: Association for Computational Linguistics. K.
中文: 计算语言学协会. 凯.
<a id="S0365"></a> Source: p.20 S0365
Original: Winogrande: An adversarial winograd schema challenge at scale.
中文: Winogrande:一个规模的对抗性Winograd计划挑战.
<a id="S0366"></a> Source: p.20 S0366
Original: In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740, 2020. M.
中文: AAAI人工智能会议记录, 第34卷, 第8732-8740页, 2020. M.
<a id="S0367"></a> Source: p.20 S0367
Original: SocialIQA: Commonsense reasoning about social interactions.
中文: SocialIQA:关于社会互动的常识推理.
<a id="S0368"></a> Source: p.20 S0368
Original: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019. C. J.
中文: 2019年自然语言处理经验方法会议纪要. C. J.
<a id="S0369"></a> Source: p.20 S0369
Original: Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018. J. W.
中文: 测量数据平行主义对神经网络培训的影响. arXiv预印版arXiv:1811.0360,2018. J. W.
<a id="S0370"></a> Source: p.20 S0370
Original: Approximation rates for neural networks with general activation functions.
中文: 具有一般活化功能的神经网络的近似速率.
<a id="S0371"></a> Source: p.20 S0371
Original: Neural Networks, 128:313–321, Aug. 2020.
中文: 神经网络, 128:313–321, 2020年8月.
<a id="S0372"></a> Source: p.20 S0372
Original: URL https://www.sciencedirect.com/scienc e/article/pii/S0893608020301891. S.
中文: URL https://www.sciencedirect.com/scienc e/article/pii/S0893608020301891. S.
<a id="S0373"></a> Source: p.20 S0373
Original: Using Deepspeed and Megatron to Train Megatronturing NLG 530b, A Large-Scale Generative Language Model. arXiv preprint arXiv:2201.11990, 2022. J.
中文: 使用"深速"和"威震天"来训练"威震天"NLG 530b","一款大型基因语言模型". arXiv preprint arXiv:2201.11990, 2022. J.
<a id="S0374"></a> Source: p.20 S0374
Original: Updates and lessons from AI forecasting, 2021.
中文: 2021年AI预测的最新消息和经验教训.
<a id="S0375"></a> Source: p.20 S0375
Original: URL https://bounded-regret.g host.io/ai-forecasting/. Y.
中文: URL https://bounded-regret.g 主机.io/ai-forecasting/. 互联网档案馆的存檔,存档日期2013-12-21. 没错
<a id="S0376"></a> Source: p.20 S0376
Original: Scale efficiently: Insights from pre-training and fine-tuning transformers, 2021. R.
中文: 缩放高效:从预训和微调变压器的透视,2021. R. (英语).
<a id="S0377"></a> Source: p.20 S0377
Original: LaMDA: Language models for dialog applications, 2022. A.
中文: LaMDA:对话框应用的语言模型,2022. A类.
<a id="S0378"></a> Source: p.20 S0378
Original: In Advances in neural information processing systems, pages 5998–6008, 2017. L.
中文: (原始内容存档于2017-10-12). In Prevants in 神经信息处理系统, page 5998–6008, 2017. 维基月球在线解说-李克文陨石坑 Andersson, L.
<a id="S0379"></a> Source: p.20 S0379
Original: Ethical and social risks of harm from language models. arXiv submission, 2021. 20
中文: 语言模型造成的危害的伦理和社会风险。 arXiv提交,2021年。 20
<a id="S0380"></a> Source: p.21 S0380
Original: Challenges in detoxifying language models.
中文: 解毒语言模型的挑战.
<a id="S0381"></a> Source: p.21 S0381
Original: In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2447–2469, Punta Cana, Dominican Republic, Nov. 2021.
中文: 在计算语言学协会的调查结果:EMNLP 2021,第2447–2469页,多米尼加共和国蓬塔卡纳,2021年11月。
<a id="S0382"></a> Source: p.21 S0382
Original: Association for Computational Linguistics.
中文: 计算语言学协会.
<a id="S0383"></a> Source: p.21 S0383
Original: URL https://aclanthology.org/2021. findings-emnlp.210. A.
中文: URL https://aclanthology.org/2021. 调查结果-emnlp.210. A.
<a id="S0384"></a> Source: p.21 S0384
Original: Detoxifying language models risks marginalizing minority voices.
中文: 解毒语言模型有可能将少数群体的声音边缘化。
<a id="S0385"></a> Source: p.21 S0385
Original: In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2390–2397, Online, June 2021.
中文: 计算语言学协会北美分会2021年会议纪要:人文语言技术, 第2390–2397页, Online, 2021年6月.
<a id="S0386"></a> Source: p.21 S0386
Original: Association for Computational Linguistics. doi: 10.18653/v1/2021 .naacl-main.190.
中文: 计算语言学协会. doi:10.18653/v1/2021.naacl-main.190.
<a id="S0387"></a> Source: p.21 S0387
Original: URL https://aclanthology.org/2021.naacl-main.190. G.
中文: URL : https://aclanthology.org/2021.naacl-main.190. G.
<a id="S0388"></a> Source: p.21 S0388
Original: Tuning large neural networks via zero-shot hyperparameter transfer.
中文: 通过零发超参数传输来培养大型神经网络.
<a id="S0389"></a> Source: p.21 S0389
Original: Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
中文: Vaughan,编辑,神经信息处理系统的进步,2021年.
<a id="S0390"></a> Source: p.21 S0390
Original: URL https://openreview.net/forum?id=Bx6qKuBM2AD. R.
中文: URL https://openreview.net/forum?id=Bx6qKuBM2AD. R.
<a id="S0391"></a> Source: p.21 S0391
Original: HellaSwag: Can a machine really finish your sentence?
中文: HelaSwag:机器真的能完成你的句子吗?
<a id="S0392"></a> Source: p.21 S0392
Original: In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. G.
中文: 2019年计算语言学协会第五十七届年会纪要. G.
<a id="S0393"></a> Source: p.21 S0393
Original: Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model.
中文: 哪种算法选择对哪个批量大小很重要? 一个吵闹的四面体模型的洞察力
<a id="S0394"></a> Source: p.21 S0394
Original: Garnett, editors, Advances in Neural Information Processing Systems, volume 32.
中文: Garnett,编辑,神经信息处理系统的进步 第32卷.
<a id="S0395"></a> Source: p.21 S0395
Original: URL https: //proceedings.neurips.cc/paper/2019/file/e0eacd983971634327ae1819ea8b621 4-Paper.pdf. B.
中文: URL https://procedings.neurips.cc/paper/2019/files/e0eacd98397164327ae1819ea8b621 4-Paper.pdf. B类.
<a id="S0396"></a> Source: p.21 S0396
Original: Designing effective sparse expert models, 2022. 21
中文: 设计有效的稀有专家模型,2022年
<a id="S0397"></a> Source: p.22 S0397
Original: Training dataset In Table A1 we show the training dataset makeup used for Chinchilla and all scaling runs.
中文: 在表A1中,我们显示用于Chinchilla和所有缩放的训练数据集化妆。
<a id="S0398"></a> Source: p.22 S0398
Original: Note that both the MassiveWeb and Wikipedia subsets are both used for more than one epoch.
中文: 注意,MassiveWeb和维基百科的子集都用于一个以上的时代.
<a id="S0399"></a> Source: p.22 S0399
Original: Disk Size Documents Sampling proportion Epochs in 1.4T tokens MassiveWeb 1.9 TB 604M 45% (48%) 1.24 Books 2.1 TB 4M 30% (27%) 0.75 C4 0.75 TB 361M 10% (10%) 0.77 News 2.7 TB 1.1B 10% (10%) 0.21 GitHub 3.1 TB 142M 4% (3%) 0.13 Wikipedia 0.001 TB 6M 1% (2%) 3.40 Table A1 | MassiveText data makeup.
中文: 磁盘大小文档采样比例 1.4T令牌中的 Epochs 比例 1.9 TB 604M 45% (48%) 1.24 书籍 2.1 TB 4M 30% (27%) 0.75 C4 0.75 TB 361M 10% (10%) 0.77 新闻 2.7 TB 1.1B 10% (10%) 0.21 GitHub 3.1 TB 142M 4% (3%) 0.13 维基百科 0.001 TB 6M 1% (2%) 3.40 表 A1 → MassiveText 数据化妆.
<a id="S0400"></a> Source: p.22 S0400
Original: For each subset of MassiveText, we list its total disk size, the number of documents and the sampling proportion used during training—we use a slightly different distribution than in Rae et al. (2021) (shown in parenthesis).
中文: 对于MassiveText的每个子集,我们列出其总磁盘大小、文件数量和在培训期间使用的抽样比例——我们使用的分布与Rae等人(2021年)略有不同。
<a id="S0401"></a> Source: p.22 S0401
Original: In the rightmost column show the number of epochs that are used in 1.4 trillion tokens. B.
中文: 在最右边一栏中显示1.4万亿令牌中使用的时代数. B类.
<a id="S0402"></a> Source: p.22 S0402
Original: Optimal cosine cycle length One key assumption is made on the cosine cycle length and the corresponding learning rate drop (we use a 10× learning rate decay in line with Rae et al. (2021)).9 We find that setting the cosine cycle length too much longer than the target number of training steps results in sub-optimally trained models, as shown in Figure A1.
中文: 最佳余弦周期长度 一个关键假设是余弦周期长度和相应的学习率下降(我们按照Rae等人(2021年)使用10×学习率衰减)。 我们发现,如图A1所示,设定余弦周期长度比培训步骤的目标数量长得多,导致培训模式不够完善。
<a id="S0403"></a> Source: p.22 S0403
Original: As a result, we assume that an optimally trained model will have the cosine cycle length correctly calibrated to the maximum number of steps, given the FLOP budget; we follow this rule in our main analysis. C.
中文: 因此,考虑到FLOP预算,我们假设一个经过最优化训练的模型将使余弦周期长度正确校正到最大步数;我们在主要分析中遵循了这个规则. C C.
<a id="S0404"></a> Source: p.22 S0404
Original: Consistency of scaling results across datasets We show scaling results from an IsoFLOP (Approach 2) analysis after training on two different datasets: C4 (Raffel et al., 2020b) and GitHub code (we show results with data from Rae et al. (2021)), results are shown in Table A2.
中文: 各数据集成果的分级一致性 我们在两个不同的数据集(C4(Raffel等人,2020年b)和GitHub代码(我们用Rae等人的数据显示结果(2021年)))的培训后,展示了IsoFLOP(APPROCH 2)分析的缩放结果,结果见表A2.
<a id="S0405"></a> Source: p.22 S0405
Original: For both set of experiments using subsets of MassiveText, we use the same tokenizer as the MassiveText experiments.
中文: 对于使用MassiveText子集的两组实验,我们使用与MassiveText实验相同的活化剂.
<a id="S0406"></a> Source: p.22 S0406
Original: We find that the scaling behaviour on these datasets is very similar to what we found on MassiveText, as shown in Figure A2 and Table A2.
中文: 我们发现这些数据集的缩放行为与我们在MassiveText上发现的行为非常相近,如图A2和表A2所显示.
<a id="S0407"></a> Source: p.22 S0407
Original: This suggests that our results are independent of the dataset as long as one does not train for more than one epoch. 9We find the difference between decaying by 10× and decaying to 0.0 (over the same number of steps) to be small, though decaying by a factor of 10× to be slightly more performant.
中文: 这表明我们的结果与数据集是独立的,只要不训练一个以上的时代。 9,我们发现由10×到衰变到0.0(相上相下相上相上相下相上相下相下相下相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相去相相去相去相去相相去相去相相相去相去相去相相相相相相相相去相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相
<a id="S0408"></a> Source: p.22 S0408
Original: Decaying by less (5×) is clearly worse. 22
中文: 由更少(5×)衰减明显更糟糕. 22个
<a id="S0409"></a> Source: p.23 S0409
Original: 1.0 0.8 0.6 0.4 0.2 0.0 0 2 4 6 8 Million Sequences RL xaM/etaR gninraeL 3.00 2.95 2.90 2.85 2.80 2.75 2.70 0 2 4 6 8 Million Sequences ssoL gniniarT 3.20 3.15 3.10 3.05 3.00 2.95 2.90 2.85 2.80 0 2 4 6 Million Sequences ssoL 4C Cosine Cycle Length 1.0× num. steps 1.1× num. steps 1.25× num. steps 1.5× num. steps 2.0× num. steps 5.0× num. steps 1.0 0.8 0.6 0.4 0.2 0.0 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences RL xaM/etaR gninraeL 3.00 2.95 2.90 2.85 2.80 2.75 2.70 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences ssoL gniniarT 3.20 3.15 3.10 3.05 3.00 2.95 2.90 2.85 2.80 0.0 2.5 5.0 7.5 10.0 12.5 Million Sequences ssoL 4C Figure A1 | Grid over cosine cycle length.
中文: 1.0 0.8 0.4 0.2 0.0 0 2 6 6 6 6 6 6 6 6 6 4 4 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 6 6 6 0 0 0 6 0 6 0 2 5 0 5 0 5 0 5 0 5 0 5 0 7 5 0 5 7 7 5 7 7 7 7 7 7 7 0 5 0 0 0 7 0 7 0 7 0 7 0 7 0 7 0 7 0 7 0 7 0 12 0 5 0 5 5 000 5 000 6 6 5 6 6 6 6
<a id="S0410"></a> Source: p.23 S0410
Original: We show 6 curves with the cosine cycle length set to 1, 1.1, 1.25, 1.5, 2, and 5× longer than the target number of training steps.
中文: 我们显示6个曲线,其余弦周期长度设定为1、1.1、1.25、1.5、2和5×长于培训步骤的目标数量。
<a id="S0411"></a> Source: p.23 S0411
Original: When the cosine cycle length is too long, and the learning rate does not drop appropriately, then performance is impaired.
中文: 当余弦循环长度过长,而学习率不适当下降时,则性能受损.
<a id="S0412"></a> Source: p.23 S0412
Original: We find that overestimating the number of training steps beyond 25% leads to clear drops in performance.
中文: 我们发现,高估超过25%的训练步骤的数量导致业绩明显下降。
<a id="S0413"></a> Source: p.23 S0413
Original: We show results where we have set the number of training steps to two different values (top and bottom). 3.2 3.0 2.8 2.6 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT 4C 1T 100B 10B 1e19 1B 1e20 6e20 100M 1e21 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 73B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.3T 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 100M 300M 1B 3B 6B 30B Parameters ssoL gniniarT buHtiG 1e19 1T 1e20 6e20 100B 1e21 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs sretemaraP 10T 1T 59B 100B 10B 1B 100M 1017 1019 1021 1023 1025 FLOPs snekoT 1.6T Figure A2 | C4 and GitHub IsoFLOP curves.
中文: 我们在将培训步骤的数量设定为两个不同的价值(上下)时,显示了成果。 3.2 2.8 2.4 2.2 2.0 100M 300M 1B 3B 6B 30B 参数ssoL 0.5 0.3 0.2 100M 300M 1B 3B 3B 6B 参数sso gniar TbuHtiG 1e19 1e21 1017 1019 1021 FLOPs 1021 1021 FLOPs 10T 73B 100B 10B 10B 1B 100B 1021 FLOPs snekoT 1.3T 0.9 0.6 0.5 0.3 0.2 100M 300M 1B 3B 6B 30B 参数sso gniniar TbuHtiG 1e19 1T 1e20 6e20 100B 1e21 10B 1019 1021 1023 10LOPs sretemarP 10T 59B 1019 B 100B 1017 1019 1021 1023 1025 FLOPS SnekoT 1.6T 图 A2 Q Q Q Q
<a id="S0414"></a> Source: p.23 S0414
Original: Using the C4 dataset (Raffel et al., 2020b) and a GitHub dataset (Rae et al., 2021), we generate 4 IsoFLOP profiles and show the parameter and token count scaling, as in Figure 3.
中文: 我们利用C4数据集(Raffel等人,2020年b)和一个GitHub数据集(Rae等人,2021年)生成了4个IsoFLOP剖面图并显示参数和符号计数缩放,如图3.
<a id="S0415"></a> Source: p.23 S0415
Original: Scaling coefficients are shown in Table A2. 23
中文: 放大系数见表A2. 23。
<a id="S0416"></a> Source: p.24 S0416
Original: Approach Coef. 𝑎 where 𝑁 ∝ 𝐶𝑎 Coef. 𝑏 where 𝐷 ∝ 𝐶𝑏 𝑜𝑝𝑡 𝑜𝑝𝑡 C4 0.50 0.50 GitHub 0.53 0.47 Kaplan et al. (2020) 0.73 0.27 Table A2 | Estimated parameter and data scaling with increased training compute on two alternate datasets.
中文: 靠近Coef. a, N → Ca Coef. b, D → Cb 选择选择 C4 0.50 0.50 GitHub 0.53 0.47 Kaplan等人(2020年) 0.73 0.27 表A2 估计参数和数据缩放,增加两个替代数据集的培训计算。
<a id="S0417"></a> Source: p.24 S0417
Original: The listed values are the exponents, 𝑎 and 𝑏, on the relationship 𝑁 𝑜𝑝𝑡 ∝ 𝐶𝑎 and 𝐷 ∝ 𝐶𝑏.
中文: 列出的值是关系N option QQ Ca和D QQ Cb上的代词 a和b.
<a id="S0418"></a> Source: p.24 S0418
Original: Using IsoFLOP profiles, we estimate the scaling on two different datasets. 𝑜𝑝𝑡 D.
中文: 使用 IsoFLOP 剖面图,我们估计两个不同数据集的缩放. 选择D。
<a id="S0419"></a> Source: p.24 S0419
Original: Approach 1: Fixing model sizes and varying training sequences We use a maximum learning rate of 2 × 10−4 for the smallest models and 1.25 × 10−4 for the largest models.
中文: 办法1:确定模式规模和不同的培训序列 我们对最小的模型采用2×10−4的最高学习率,对最大的模型采用1.25×10−4的最高学习率.
<a id="S0420"></a> Source: p.24 S0420
Original: In all cases, the learning rate drops by a factor of 10× during training, using a cosine schedule.
中文: 在所有情况下,学习率在培训期间都下降了10××倍,采用同位素时间表。
<a id="S0421"></a> Source: p.24 S0421
Original: We make the assumption that the cosine cycle length should be approximately matched to the number of training steps.
中文: 我们假设,余弦周期长度应与训练步骤的数目大致相匹配。
<a id="S0422"></a> Source: p.24 S0422
Original: We find that when the cosine cycle overshoots the number of training steps by more than 25%, performance is noticeably degraded—see Figure A1.10 We use Gaussian smoothing with a window length of 10 steps to smooth the training curve. D.2.
中文: 我们发现,当余弦循环将训练步骤的数量超过25%时,性能明显下降——见图A1.10。 我们使用高斯平滑,窗口长度为10步来平滑训练曲线. D.2节。
<a id="S0423"></a> Source: p.24 S0423
Original: Approach 3: Parametric fitting of the loss In this section, we first show how Equation (2) can be derived.
中文: 办法3:损失的参数配置 在本节中,我们首先展示了"方程式(2)"是如何被衍生出来的.
<a id="S0424"></a> Source: p.24 S0424
Original: We repeat the equation below for clarity, 𝐴 𝐵 𝐿ˆ (𝑁, 𝐷) (cid:44) 𝐸 + + , (5) 𝑁𝛼 𝐷𝛽 based on a decomposition of the expected risk between a function approximation term and an optimisation suboptimality term.
中文: 我们重复以下等式以明确,A B L Q (N, D) (cid:44) E + +, (5) Nα Dβ 基于函数近似词和优化次优化词之间的预期风险分解.
<a id="S0425"></a> Source: p.24 S0425
Original: We then give details on the optimisation procedure for fitting the parameters.
中文: 然后,我们详细介绍优化参数的程序。
<a id="S0426"></a> Source: p.24 S0426
Original: Formally, we consider the task of predicting the next token 𝑦 ∈ Y based on the previous tokens in a sequence 𝑥 ∈ Y𝑠, with 𝑠 varying from 0 to 𝑠 max —the maximum sequence length.
中文: 在形式上,我们考虑的任务是根据前作的相序 x ∈ Ys 来预测下个相序 y ∈ Y,其相序从 0 到 s 最大——最大相序长度不等.
<a id="S0427"></a> Source: p.24 S0427
Original: We consider a distribution 𝑃 ∈ D (X × Y) of tokens in Y and their past in X. A predictor 𝑓 : X → D (Y) computes the probability of each token given the past sequence.
中文: 我们考虑在Y分发P-D(X-Y)和在X分发其过去。 一个预测符 f: X → D (Y) 计算每个符的概率,以过去序列为准.
<a id="S0428"></a> Source: p.24 S0428
Original: The Bayes classifier, 𝑓 ★, minimizes the cross-entropy of 𝑓 (𝑥) with the observed tokens 𝑦, with expectation taken on the whole data distribution.
中文: Bayes分类器 f 将 f (x) 与所观测到的符号 y 的交叉切入最小化,并对整个数据分布进行预期.
<a id="S0429"></a> Source: p.24 S0429
Original: We let 𝐿 be the expected risk 𝐿( 𝑓 ) (cid:44) 𝔼[log 𝑓 (𝑥) 𝑦 ], and set 𝑓 ★ (cid:44) argmin 𝐿( 𝑓 ). (6) 𝑓 ∈F(X,D (Y)) The set of all transformers of size 𝑁, that we denote H , forms a subset of all functions that map 𝑁 sequences to distributions of tokens X → D (Y).
中文: 我们让L成为预期的风险L(f)(cid:44) E[log f (x)y],并设置 f ★ (cid:44) argmin L (f.) (6) f QQF (X,D (Y)) 大小为N(我们表示H.)的所有变压器的集合,构成所有函数的一个子集,将N序列映射到符号X ~ D (Y)的分布.
<a id="S0430"></a> Source: p.24 S0430
Original: Fitting a transformer of size 𝑁 on the expected risk 𝐿( 𝑓 ) amounts to minimizing such risk on a restricted functional space 𝑓 (cid:44) argmin 𝐿( 𝑓 ). (7) 𝑁 𝑓 ∈H𝑁 When we observe a dataset (𝑥 𝑖 , 𝑦 𝑖 ) 𝑖𝑖∈[1,𝐷] of size 𝐷, we do not have access to 𝔼𝑃 , but instead to the empirical expectation 𝔼 ˆ 𝐷 over the empirical distribution 𝑃ˆ 𝐷 .
中文: 将一个尺寸为N的变压器安装在预期风险L(f)上,等于在有限的功能空间f(cid:44)argmin L(f.)上尽量减少这种风险。 (7) Nf QQHN 当我们观察到一个大小为D的数据集(xi, yi)ii∈ [1, D], 我们无法访问EP, 而是在经验分布Pˆ D上达到Eˆ D.
<a id="S0431"></a> Source: p.24 S0431
Original: What happens when we are given 𝐷 10This further emphasises the point of not only determining model size, but also training length before training begins. 24
中文: 我们得到D 10后会发生什么 这进一步强调了不仅要确定模型大小,还要在训练开始前进行训练。 24个
<a id="S0432"></a> Source: p.25 S0432
Original: datapoints that we can only see once, and when we constrain the size of the hypothesis space to be 𝑁-dimensional ?
中文: 数据点,我们只能看到一次, 当我们限制假设空间的大小 成为N维时?
<a id="S0433"></a> Source: p.25 S0433
Original: We are making steps toward minimizing the empirical risk within a finite-dimensional functional space H : 𝑁 𝐿ˆ 𝐷 ( 𝑓 ) (cid:44) 𝔼 ˆ 𝐷 [log 𝑓 (𝑥) 𝑦 ], setting 𝑓 ˆ 𝑁,𝐷 (cid:44) argmin 𝐿ˆ 𝐷 ( 𝑓 ). (8) 𝑓 ∈H𝑁 We are never able to obtain 𝑓 ˆ as we typically perform a single epoch over the dataset of size 𝐷. 𝑁,𝐷 Instead, be obtain 𝑓 ¯ , which is the result of applying a certain number of gradient steps based on 𝑁,𝐷 the 𝐷 datapoints—the number of steps to perform depends on the gradient batch size, for which we use well-tested heuristics.
中文: 我们正在采取步骤,在有限维函数空间H:N Lˆ D (f): (cid:44) E D [log f (x) y],设置 fˆ N, D (cid:44) argmin Lˆ D (f.). (8) f HN. 我们永远无法获得 f ,因为我们通常在大小 D. N, D 的数据集上执行一个单一的时代, 相反,获得 f' ,这是根据 N, D 数据点应用一定数量的梯度步骤的结果——要完成的步骤数取决于梯度批量大小,对此我们使用经过充分检验的休眠法.
<a id="S0434"></a> Source: p.25 S0434
Original: Using the Bayes-classifier 𝑓 ★, the expected-risk minimizer 𝑓 and the “single-epoch empirical-risk 𝑁 minimizer” 𝑓 ¯ , we can finally decompose the loss 𝐿(𝑁, 𝐷) into 𝑁,𝐷 𝐿(𝑁, 𝐷) (cid:44) 𝐿( 𝑓 ¯ ) = 𝐿( 𝑓 ★) + (cid:0)𝐿( 𝑓 ) − 𝐿( 𝑓 ★) (cid:1) + (cid:0)𝐿( 𝑓 ¯ ) − 𝐿( 𝑓 ) (cid:1) . (9) 𝑁,𝐷 𝑁 𝑁,𝐷 𝑁 The loss comprises three terms: the Bayes risk, i.e. the minimal loss achievable for next-token prediction on the full distribution 𝑃, a.k.a the “entropy of natural text.”; a functional approximation term that depends on the size of the hypothesis space; finally, a stochastic approximation term that captures the suboptimality of minimizing 𝐿ˆ instead of 𝐿, and of making a single epoch on the provided 𝐷 dataset.
中文: 使用Bayes分类符 f-(cid:0)-(f)-(cid:1)+(cid:0)-(f)-(cid:1)-(cid:0)-(cid:1)-(cid:0)-(cid:0)-(cid:0)-(cid:0)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:1)-(cid:9)-(Bayes N,D:1)-(cid:1)-(cid:9)-(cid:D)-(c)-(cid:1)-(c)-(c)-(c)-)-(d.)-(c)-(c)-(c)-(c)-(cost-)-(c)-)-(c)-(c)-(
<a id="S0435"></a> Source: p.25 S0435
Original: In the decomposition (9), the second term depends entirely on the number of parameters 𝑁 that defines the size of the functional approximation space.
中文: 在分解(9)中,第二个术语完全取决于定义函数近似空间大小的参数N的数量.
<a id="S0436"></a> Source: p.25 S0436
Original: On the set of two-layer neural networks, it is expected to be proportional to 1 (Siegel and Xu, 2020).
中文: 在一套双层神经网络上,预计与1成正比(Siegel and Xu, 2020).
<a id="S0437"></a> Source: p.25 S0437
Original: Finally, 𝑁1/2 given that it corresponds to early stopping in stochastic first order methods, the third term should scale as the convergence rate of these methods, which is lower-bounded by 1 (Robbins and Monro, 𝐷1/2 1951) (and may attain the bound).
中文: 最后,N1/2,因为它对应了先行先行法的早期停止,第三个术语应该作为这些方法的收缩率来缩放,其下限为1(Robbins and Monro, D1/2 1951) (并且可能达到约束).
<a id="S0438"></a> Source: p.25 S0438
Original: This convergence rate is expected to be dimension free (see e.g.
中文: 预计这一趋同率将是无维度的(例如,见E.
<a id="S0439"></a> Source: p.25 S0439
Original: Bubeck, 2015, for a review) and depends only on the loss smoothness; hence we assume that the second term only depends on 𝐷 in (2).
中文: Bubeck, 2015, for a Review),并且只取决于损失的平滑度;因此,我们假设第二个任期只取决于D in (2).
<a id="S0440"></a> Source: p.25 S0440
Original: Empirically, we find after fitting (2) that 𝐴 𝐵 𝐿(𝑁, 𝐷) = 𝐸 + + , (10) 𝑁0.34 𝐷0.28 with 𝐸 = 1.69, 𝐴 = 406.4, 𝐵 = 410.7.
中文: 经验中,我们发现在安装了(2)后,A B L(N,D) = E + +,(10) N0.34 D0.28 与 E = 1.69, A = 406.4, B = 410.7.
<a id="S0441"></a> Source: p.25 S0441
Original: We note that the parameter/data coefficients are both lower than 1 ; this is expected for the data-efficiency coefficient (but far from the known lower-bound). 2 Future models and training approaches should endeavor to increase these coefficients.
中文: 我们注意到,参数/数据系数均低于1;这是数据效率系数(但远远低于已知的下限系数)的预期值。 2 未来的模式和培训办法应努力增加这些系数。
<a id="S0442"></a> Source: p.25 S0442
Original: We effectively minimize the following problem min ∑︁ Huber (cid:16) LSE(cid:0)𝑎 − 𝛼 log 𝑁 , 𝑏 − 𝛽 log 𝐷 , 𝑒(cid:1) − log 𝐿 (cid:17) , (11) 𝛿 𝑖 𝑖 𝑖 𝑎,𝑏,𝑒,𝛼,𝛽 Run 𝑖 where 𝐿𝑆𝐸 is the log-sum-exp operator.
中文: 我们有效地将以下问题最小化:min QQ Huber(cid:16) LSE(cid:0)a − α对数 N,b − β对数 D,e(cid:1) − log L(cid:17),(11) i i i i a,b,e,α,β Run i, LSE是log-sum-exp操作员.
<a id="S0443"></a> Source: p.25 S0443
Original: We then set 𝐴, 𝐵, 𝐸 = exp(𝑎), exp(𝑏), exp(𝑒).
中文: 然后设定 A, B, E = exp(a),exp(b),exp(e).
<a id="S0444"></a> Source: p.25 S0444
Original: We use the LBFGS algorithm to find local minima of the objective above, started on a grid of initialisation given by: 𝛼 ∈ {0., 0.5, . . . , 2.}, 𝛽 ∈ {0., 0.5, . . . , 2.}, 𝑒 ∈ {−1., −.5, . . . , 1.}, 𝑎 ∈ {0, 5, . . . , 25}, and 𝑏 ∈ {0, 5, . . . , 25}.
中文: 我们使用 LBFGS 算法来查找上面的目标的局部微分,开始于一个初始化的网格上,给出的网格是: α {0., 0.5, 2.}, β {. {, 0.5, 2.}, e ∈ 1, − 5., 1.}, a {0, 5, 25}和 b {0, 5, 25}.
<a id="S0445"></a> Source: p.25 S0445
Original: We find that the optimal initialisation is not on the boundary of our initialisation sweep.
中文: 我们发现,最理想的初始化不是在我们初始化扫描的边界上。
<a id="S0446"></a> Source: p.25 S0446
Original: We find that using larger values of 𝛿 pushes the model to overfit the small compute regime and poorly predict held-out data from larger runs.
中文: 我们发现,使用更大的数值 将模型推向 过度匹配 小型计算系统 和低预测 持有的数据 从更大的运行。
<a id="S0447"></a> Source: p.25 S0447
Original: We find that using a 𝛿 smaller than 10−3 does not impact the resulting predictions. 25
中文: 我们发现,使用小于10-3的缩写不会影响由此产生的预测。 第 25 条
<a id="S0448"></a> Source: p.26 S0448
Original: Predicted compute optimal frontier for all three methods For Approaches 2 and 3, we show the estimated model size and number of training tokens for a variety of compute budgets in Table A3.
中文: 关于方法2和3,我们在表A3中列出了各种计算预算的模型大小和培训符号估计数。
<a id="S0449"></a> Source: p.26 S0449
Original: We plot the predicted number of tokens and parameters for a variety of FLOP budgets for the three methods in Figure A3.
中文: 我们为图A3中的三种方法绘制了各种FLOP预算的预测符号和参数。
<a id="S0450"></a> Source: p.26 S0450
Original: Approach 2 Approach 3 Parameters FLOPs Tokens FLOPs Tokens 400 Million 1.84e+19 7.7 Billion 2.21e+19 9.2 Billion 1 Billion 1.20e+20 20.0 Billion 1.62e+20 27.1 Billion 10 Billion 1.32e+22 219.5 Billion 2.46e+22 410.1 Billion 67 Billion 6.88e+23 1.7 Trillion 1.71e+24 4.1 Trillion 175 Billion 4.54e+24 4.3 Trillion 1.26e+24 12.0 Trillion 280 Billion 1.18e+25 7.1 Trillion 3.52e+25 20.1 Trillion 520 Billion 4.19e+25 13.4 Trillion 1.36e+26 43.5 Trillion 1 Trillion 1.59e+26 26.5 Trillion 5.65e+26 94.1 Trillion 10 Trillion 1.75e+28 292.0 Trillion 8.55e+28 1425.5 Trillion Table A3 | Estimated optimal training FLOPs and training tokens for various model sizes.
中文: 办法2 方法3 参数 FLOPs Tokens FLOPs Tokens 1.84e+19 7.7亿 2.21e+19 9.2亿 1亿 1.20e+20 20.0亿 1.62e+20 27.1亿 10.10亿 1.32e+22/219.5亿 2.46e+22 4.10.1亿 6.88e+23 1.7亿 1.71e+24 4.1 三亿 175亿 4.54e+24 4.3亿 1.28亿 1.26e+24 12.0 280亿 1.18e+25 7.1 三亿 3.52e+25 20.1 三亿 5.20亿 4.19e+25 13.4亿 1.36e+26 43.5 三亿 1.59e+26.5 千兆 5.65e+26.5 千兆 5.65e+26 4.4.1 三亿 1.75e+28 292.0 三亿 8.55e+28 14.25.5 Trillion 表A3 估计各种型号的优化培训FLOP和训练标志。
<a id="S0451"></a> Source: p.26 S0451
Original: Analogous to Table 3, we show the model size/token count projections from Approaches 2 and 3 for various compute budgets. . 1012 1011 1010 109 108 1010 1011 1012 1013 Tokens sretemaraP Approach 1 1e+26 Approach 2 Approach 3 1e+25 Chinchilla Gopher GPT-3 1e+24 Megatron-Turing NLG 1e+23 1e+22 1e+21 1e+20 1e+19 1e+18 Figure A3 | Optimal number of tokens and parameters for a training FLOP budget.
中文: 与表3相仿的是,我们从方法2和3对各种计算预算的模型大小/计算数预测。 。 。 。 。 1012 1011 1010 109 108 1010 1011 1012 1013 Tokens sretemaraP方法 1e+26方法 2方法 1e+25 Chinchilla Gopher GPT-3 1e+24 威震天-图灵 NLG 1e+23 1e+22 1e+21 1e+20 1e+19 1e+18 图 A3 培训FLOP预算的标语和参数优化数。
<a id="S0452"></a> Source: p.26 S0452
Original: For a fixed FLOP budget, we show the optimal number of tokens and parameters as predicted by Approaches 1, 2, and 3.
中文: 就固定的FLOP预算而言,我们显示了方法1、2和3预测的最佳信使和参数数量。
<a id="S0453"></a> Source: p.26 S0453
Original: For an alternate representation, see Figure 1. D.4.
中文: 候补代表,见图1.D.4.。
<a id="S0454"></a> Source: p.26 S0454
Original: Small-scale comparison to Kaplan et al. (2020) For 1021 FLOPs, we perform a head-to-head comparison of a model predicted by Approach 1 and that predicted by Kaplan et al. (2020).
中文: 与Kaplan等人(2020年)的小规模比较,对于1021 FLOPs,我们对方法1预测的模型和Kaplan等人(2020年)预测的模型进行了头对头比较。
<a id="S0455"></a> Source: p.26 S0455
Original: For both models, we use a batch size of 0.5M tokens and a 26
中文: 对两种型号,我们使用0.5M的批量尺寸和26个
<a id="S0456"></a> Source: p.27 S0456
Original: maximum learning rate of 1.5 × 10−4 that decays by 10×.
中文: 最大学习率为1.5×10−4,衰减为10×.
<a id="S0457"></a> Source: p.27 S0457
Original: From Kaplan et al. (2020), we find that the optimal model size should be 4.68 billion parameters.
中文: 从Kaplan等人(2020年)来看,我们发现最佳模型尺寸应为46.8亿个参数。
<a id="S0458"></a> Source: p.27 S0458
Original: From our approach 1, we estimate a 2.86 billion parameter model should be optimal.
中文: 根据我们的方法1,我们估计28.6亿个参数模型应该是最佳的。
<a id="S0459"></a> Source: p.27 S0459
Original: We train a 4.74 billion parameter and a 2.80 billion parameter transformer to test this hypothesis, using the same depth-to-width ratio to avoid as many confounding factors as possible.
中文: 我们训练了47.4亿个参数和一个28.0亿个参数变压器来测试这个假说,使用相同的深度比来避免尽可能多的混淆因素.
<a id="S0460"></a> Source: p.27 S0460
Original: We find that our predicted model outperforms the model predicted by Kaplan et al. (2020) as shown in Figure A4. 2.8 2.7 2.6 2.5 2.4 2.3 2.2 0 1 2 Sequences 1e7 ssoL gniniarT 2.8 2.7 2.6 2.5 2.4 2.3 2.2 0.0 0.2 0.4 0.6 0.8 1.0 FLOPs ×1021 ssoL gniniarT Kaplan et al (2020) Approach 1 Figure A4 | Comparison to Kaplan et al. (2020) at 1021 FLOPs.
中文: 我们发现,如图A4. 2.8 2.7 2.5 2.3 2 0 2 序列 1e7 ssoL gniniarT 2.8 2.6 2.2 2.3 2.2 0.0 0.2 0.4 0.6 0.8 1.0 FLOPs ×1021 ssoL gniniarT Kaplan等 (2020) 方法 1 图A4 与 Kaplan等 (2020) 1021 FLOPs 比较。
<a id="S0461"></a> Source: p.27 S0461
Original: We train 2.80 and 4.74 billion parameter transformers predicted as optimal for 1021 FLOPs by Approach 1 and by Kaplan et al. (2020).
中文: 我们通过方法1和Kaplan等人(2020年)对2.80和47.4亿参数变压器进行了预测,认为1021 FLOP是最佳的。
<a id="S0462"></a> Source: p.27 S0462
Original: We find that our prediction results in a more performant model at the end of training. E.
中文: 我们发现,在培训结束时,我们的预测结果是一个更能表现的模型。 页:1
<a id="S0463"></a> Source: p.27 S0463
Original: Curvature of the FLOP-loss frontier We observe that as models increase there is a curvature in the FLOP-minimal loss frontier.
中文: FLOP-损失边界的曲线 我们观察到,随着模型的增加,FLOP-最小损失前沿有一个曲率.
<a id="S0464"></a> Source: p.27 S0464
Original: This means that projections from very small models lead to different predictions than those from larger models.
中文: 这意味着来自非常小模型的预测会导致与来自较大模型的预测不同.
<a id="S0465"></a> Source: p.27 S0465
Original: In Figure A5 we show linear fits using the first, middle, and final third of frontier-points.
中文: 在图A5中,我们用第一、中和最后三分之一的前沿点来显示线性匹配。
<a id="S0466"></a> Source: p.27 S0466
Original: In this work, we do not take this in to account and we leave this as interesting future work as it suggests that even smaller models may be optimal for large FLOP budgets. F.
中文: 在这项工作中,我们不考虑这一点,我们把这项工作留作今后令人感兴趣的工作,因为它表明,即使是较小的模型也可能是大型FLOP预算的最佳选择。 费.
<a id="S0467"></a> Source: p.27 S0467
Original: FLOPs computation We include all training FLOPs, including those contributed to by the embedding matrices, in our analysis.
中文: FLOP 计算 我们的分析中包括所有培训活动,包括嵌入式矩阵所促进的培训活动。
<a id="S0468"></a> Source: p.27 S0468
Original: Note that we also count embeddings matrices in the total parameter count.
中文: 请注意,我们还在总参数数中计算出嵌入矩阵。
<a id="S0469"></a> Source: p.27 S0469
Original: For large models the FLOP and parameter contribution of embedding matrices is small.
中文: 对于大型模型,嵌入矩阵的FLOP和参数贡献很小.
<a id="S0470"></a> Source: p.27 S0470
Original: We use a factor of 2 to describe the multiply accumulate cost.
中文: 我们用系数2来描述乘积成本。
<a id="S0471"></a> Source: p.27 S0471
Original: For the forward pass, we consider contributions from: • Embeddings – 2 × seq_len × vocab_size × d_model • Attention (Single Layer) – Key, query and value projections: 2 × 3 × seq_len × d_model × (key_size × num_heads) 27
中文: 关于前方通行证,我们考虑来自: 嵌入 – 2 × 下 len × vocab size × d model – 注意 (单层) 键,查询和数值预测: 2 × 3 × 后 len × d model → (键 大小 × num heads) 27
<a id="S0472"></a> Source: p.28 S0472
Original: 6.0 5.5 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1017 1018 1019 1020 1021 1022 FLOPS ssol gniniarT 10000 5000 2500 1000 500 250 75 sretemaraP noilliM Figure A5 | Training curve envelopes.
中文: 6.0 5.5 4.5 4.5 4.0 3.5 3.0 2.5 2.0 1017 1018 1019 1020 1021 1022 FLOPS sol gniniarT 10000 5000 2500 1000 250 75 sretemaraP noillim 图A5 QQ训练曲线信封.
<a id="S0473"></a> Source: p.28 S0473
Original: We fit to the first third (orange), the middle third (green), and the last third (blue) of all points along the loss frontier.
中文: 我们与损失边界沿线所有点中的第三分之一(橙色)、中第三(绿色)和后三分之一(蓝色)相吻合。
<a id="S0474"></a> Source: p.28 S0474
Original: We plot only a subset of the points. – Key @ Query logits: 2 × seq_len × seq_len × (key_size × num_heads) – Softmax: 3 × num_heads × seq_len × seq_len – Softmax @ query reductions: 2 × seq_len × seq_len × (key_size × num_heads) – Final Linear: 2 × seq_len × (key_size × num_heads) × d_model • Dense Block (Single Layer) – 2 × seq_len × (d_model × ffw_size + d_model × ffw_size) • Final Logits – 2 × seq_len × d_model × vocab_size • Total forward pass FLOPs: embeddings+num_layers× (total_attention+dense_block) + logits As in Kaplan et al. (2020) we assume that the backward pass has twice the FLOPs of the forward pass.
中文: 我们只绘制一个子集点。 —@ 查询日志: 2– 下 下- 下- 下- 下- 下- 上- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 上- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- 下- - 下- 下- 下- 下- 下- 下- 下- 与Kaplan等人(2020年)一样,我们假设后行通行证是前行通行证的两倍。
<a id="S0475"></a> Source: p.28 S0475
Original: We show a comparison between our calculation and that using the common approximation 𝐶 = 6𝐷𝑁 (Kaplan et al., 2020) where 𝐶 is FLOPs, 𝐷 is the number of training tokens, and 𝑁 is the number of parameters in Table A4.
中文: 我们的计算与使用C=6DN(Kaplan等,2020年)的通用近似值比较,C是FLOPs,D是训练符数,N是表A4中的参数数.
<a id="S0476"></a> Source: p.28 S0476
Original: We find the differences in FLOP calculation to be very small and they do not impact our analysis.
中文: 我们认为FLOP的计算差异很小,不会影响我们的分析。
<a id="S0477"></a> Source: p.28 S0477
Original: Compared to the results presented in Rae et al. (2021), we use a slightly more Parameters num_layers d_model ffw_size num_heads k/q size FLOP Ratio (Ours/6𝑁 𝐷) 73M 10 640 2560 10 64 1.03 305M 20 1024 4096 16 64 1.10 552M 24 1280 5120 10 128 1.08 1.1B 26 1792 7168 14 128 1.04 1.6B 28 2048 8192 16 128 1.03 6.8B 40 3584 14336 28 128 0.99 Table A4 | FLOP comparison.
中文: 相较于Rae等人(2021年)给出的结果,我们采用了略多的参数num layers d model ffw size numm heads k/q size FLOP比 (Ours/6N D) 73M 10 640 2560 10 64 1.03 305M 20 1024 4096 16 64 1.10 552M 24 1280 5120 10 1.08 1.1B 26 1792 7168 14 128 1.04 1.6B 28 2048 8192 16 128 1.03 6.8B 40 3584 14336 28 128 0.99 表 A4 | FLOP比较.
<a id="S0478"></a> Source: p.28 S0478
Original: For a variety of different model sizes, we show the ratio of the FLOPs that we compute per sequence to that using the 6𝑁 𝐷 approximation. accurate calculation giving a slightly different value (6.3 × 1023 compared to 5.76 × 1023). 28
中文: 对于各种不同的模型大小,我们显示我们用6N D相近来计算每个序列的FLOP的比例. 精确计算给出的数值略有不同(6.3×1023,而5.76×1023). 第28条
<a id="S0479"></a> Source: p.29 S0479
Original: Other differences between Chinchilla and Gopher Beyond differences in model size and number of training tokens, there are some additional minor differences between Chinchilla and Gopher.
中文: 钦奇拉和高佛的其他差异 除了模型大小和训练符数量的差异之外,钦奇拉和高佛之间还存在一些额外的小差异.
<a id="S0480"></a> Source: p.29 S0480
Original: Specifically, Gopher was trained with Adam (Kingma and Ba, 2014) whereas Chinchilla was trained with AdamW (Loshchilov and Hutter, 2019).
中文: 具体而言,Gopher与Adam一起接受了培训(Kingma和Ba,2014年),而Chinchilla则与AdamW一起接受了培训(Loshchilov和Hutter,2019年)。
<a id="S0481"></a> Source: p.29 S0481
Original: Furthermore, as discussed in Lessons Learned in Rae et al. (2021), Chinchilla stored a higher-precision copy of the weights in the sharded optimiser state.
中文: 此外,正如在Rae等人(2021年)中吸取的教训所讨论,Chinchilla储存了被压抑的Otimiser状态下重量的更精确的副本。
<a id="S0482"></a> Source: p.29 S0482
Original: We show comparisons of models trained with Adam and AdamW in Figure A6 and Figure A7.
中文: 在图A6和图A7中,我们显示与亚当和亚当W培训的模型的比较。
<a id="S0483"></a> Source: p.29 S0483
Original: We find that, independent of the learning rate schedule, AdamW trained models outperform models trained with Adam.
中文: 我们发现,独立于学习速度表之外,亚当W训练的模型超过了亚当训练的模型.
<a id="S0484"></a> Source: p.29 S0484
Original: In Figure A6 we show a comparison of an 680 million parameter model trained 2.70 2.65 2.60 2.55 2.50 2.45 0 5 10 15 20 25 30 Million Sequences ssoL gniniarT 26 25 24 23 22 21 20 19 18 17 0 5 10 15 20 25 30 Million Sequences ytixelpreP 301txetikiW 3.00 2.95 2.90 2.85 2.80 2.75 2.70 2.65 2.60 0 5 10 15 20 25 30 Million Sequences ssoL 4C Training Setup Adam w/ High Precision AdamW w/ High Precision Adam No High Precision AdamW No High Precision Figure A6 | Comparison of other differences.
中文: 在图A6中,我们显示一个6.8亿个参数模型的比较:训练2.70 2.65 2.60 2.55 2.50 2.245 0 5 10 15 20 25 3000万个序列 ssoL gniniarT 26 25 23 22 21 20 19 17 0 5 15 20 25 3000万个序列 ytixelprep 301txetikiW 3.00 2.95 2.90 2.85 2.80 2.75 2.70 2.65 2.60 0 5 10 15 20 20 25 3000万个序列 ssoL 4C 训练设置 Adam w/高精密度 Adam W/高精密度 Adam 无高精密度 Adam W 无高精密度图 A6 → 比较其他差异.
<a id="S0485"></a> Source: p.29 S0485
Original: Using an 680 million parameter model, we show a comparison between the setup used to train Gopher and Chinchilla— the change in optimiser and using a higher precision copy of the weights in the optimiser state.
中文: 使用6.8亿个参数模型,我们显示用于训练Gopher和Chinchilla的设置的比较——Optimiser的变换,并使用Optimiser状态下更精确的重量复制.
<a id="S0486"></a> Source: p.29 S0486
Original: The setup used for Chinchilla (orange) clearly outperforms the setup used to train Gopher (green). 2.8 2.7 2.6 2.5 2.4 2.3 0 25 50 75 100 125 150 Million Sequences ssoL 4C 30.0 27.5 25.0 22.5 20.0 17.5 15.0 12.5 10.0 0 25 50 75 100 125 150 Million Sequences ytixelpreP 301txetikiW 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 25 50 75 100 125 150 Million Sequences ycaruccA ADABMAL 417M, Adam 417M, AdamW 1.4B, Adam 1.4B, AdamW Figure A7 | Adam vs AdamW.
中文: 用于Chinchilla(橙色)的设置明显超过了用于训练Gopher(绿色)的设置. 2.8 2.6 2.5 2.4 2.3 0 25 50 75 100 1.1500万序列 ssoL 4C 30.0 27.5 25.0 22.5 20.0 17.5 15.0 12.5 10.0 25 75 100 125万序列 ytixelpreP 301txetikiW 0.5 0.4 0.3 0.2 0.0 0 25 50 75 100 1.1500万序列 ycaruccA AD ABMA 417M, Adam 417M, Adam W 1.4B, Adam W 图 A7 Adam vs AdamW.
<a id="S0487"></a> Source: p.29 S0487
Original: For a 417M (blue) and 1.4B model (green), we find that training with AdamW improves performance over training with Adam. with and without the higher precision copy of the weights and with Adam/AdamW for comparison. H.
中文: 对于一个417M(蓝色)和1.4B(绿色)模式,我们发现,与亚当训练相比,亚当训练会提高性能. 和亚当/亚当W进行比较。 H.
<a id="S0488"></a> Source: p.29 S0488
Original: The Pile In Table A5 we show the bits-per-byte (bpb) on The Pile (Gao et al., 2020) of Chinchilla, Gopher, and Jurassic-1.
中文: 在表A5中,我们显示Chinchilla、Gopher和Jurassic-1上的比特值(bpb)。
<a id="S0489"></a> Source: p.29 S0489
Original: Chinchilla outperforms Gopher on all subsets.
中文: Chinchilla在所有子集上都超越了Gopher.
<a id="S0490"></a> Source: p.29 S0490
Original: Jurassic-1 outperforms Chinchilla on 2 subsets— dm_mathematics and ubuntu_irc. 29
中文: 侏罗纪-1在2个子集上超越了钦奇拉——dm 数学和ubuntu irc. 29
<a id="S0491"></a> Source: p.30 S0491
Original: Subset Chinchilla (70B) Gopher (280B) Jurassic-1 (170B) pile_cc 0.667 0.691 0.669 pubmed_abstracts 0.559 0.578 0.587 stackexchange 0.614 0.641 0.655 github 0.337 0.377 0.358 openwebtext2 0.647 0.677 arxiv 0.627 0.662 0.680 uspto_backgrounds 0.526 0.546 0.537 freelaw 0.476 0.513 0.514 pubmed_central 0.504 0.525 0.579 dm_mathematics 1.111 1.142 1.037 hackernews 0.859 0.890 0.869 nih_exporter 0.572 0.590 0.590 opensubtitles 0.871 0.900 0.879 europarl 0.833 0.938 books3 0.675 0.712 0.835 philpapers 0.656 0.695 0.742 gutenberg_pg_19 0.548 0.656 0.890 bookcorpus2 0.714 0.741 ubuntu_irc 1.026 1.090 0.857 Table A5 | Bits-per-Byte on The Pile.
中文: 下集 Chinchilla (70B) Gopher (280B) 侏罗纪-1 (170B) 堆积 cc 0.667 0.691 0.669 pubmed abstracts 0.559 0.578 0.587 堆积交换 0.614 0.641 0.655 gi 0.337 0.377 0.358 开网文本2 0.647 0.677 arxiv 0.627 0.662 0.680 uspto 后地 0.526 0.546 0.537 自由法 0.476 0.513 0.514 pubmed Central 0.504 0.525 0.579 dm 数学 1.111 1.142 1.037 黑客新闻 0.859 0.890 0.869 0.590 开口字幕 0.871 0.900 0.879 欧元帕尔 0.833 0.938 0.675 0.712 0.835 纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸纸
<a id="S0492"></a> Source: p.30 S0492
Original: We show the bpb on The Pile for Chinchilla compared to Gopher and Jurassic-1. H.2.
中文: 与Gopher和Jurassic-1相比,我们给Chinchilla的Pile上的bpb显示。
<a id="S0493"></a> Source: p.30 S0493
Original: MMLU In Table A6 we show the performance of Chinchilla and Gopher on each subset of MMLU. H.3.
中文: 姆卢 在表A6中,我们显示Chinchilla和Gopher在MMLU的每个子集上的性能。
<a id="S0494"></a> Source: p.30 S0494
Original: Winogender Setup We follow the same setup as in Rae et al. (2021).
中文: Winogender 设置 我们遵循与雷等人(2021年)相同的设定.
<a id="S0495"></a> Source: p.30 S0495
Original: To test coreference resolution in Chinchilla, we input a sentence which includes a pronoun reference (e.g., “The librarian helped the child pick out a book because {pronoun} liked to encourage reading.”), then measure the probability of the model completing the sentence “‘{Pronoun}’ refers to the” with different sentence roles (“librarian” and “child” in this example).
中文: 为了测试Chinchilla语的校正分辨率,我们输入了一句话,其中包括一个代名词参考(例如,“图书管理员帮助孩子挑出一本书,因为{pronoun}喜欢鼓励阅读.”),然后衡量模型完成“{pronoun}提到具有不同句子角色(“图书馆家”)和“儿童”的句子的概率。
<a id="S0496"></a> Source: p.30 S0496
Original: Each example is annotated with the correct pronoun resolution (the pronoun corresponds to the librarian in this example).
中文: 每个例子都附加了正确的代名词解析度(这个例子中的代名词对应图书管理员).
<a id="S0497"></a> Source: p.30 S0497
Original: Each sentence is tested with a female, male, and gender-neutral pronoun.
中文: 每个句子都使用女性,男性,以及不分性别的代词进行测试.
<a id="S0498"></a> Source: p.30 S0498
Original: An unbiased model would correctly predict which word the pronoun refers to regardless of pronoun gender. H.4.
中文: 一个没有偏见的模型可以正确预测代词指的是哪个词,而不论代词性别. H.4. (中文(简体) ).
<a id="S0499"></a> Source: p.30 S0499
Original: BIG-bench In Table A7 we show Chinchilla and Gopher performance on each subset of BIG-bench that we consider. I.
中文: 在表A7中,我们显示 Chinchilla和Gopher的性能 在我们考虑的BIG-bench的每个子集。 说吧
<a id="S0500"></a> Source: p.30 S0500
Original: Model Card We present the Chinchilla model card in Table A8, following the framework presented by Mitchell et al. (2019). 30
中文: 样卡 我们按照Mitchell等人提出的框架(2019年),在表A8中提出钦奇拉模式卡。 30个
<a id="S0501"></a> Source: p.31 S0501
Original: Task Chinchilla Gopher Task Chinchilla Gopher abstract_algebra 31.0 25.0 anatomy 70.4 56.3 astronomy 73.0 65.8 business_ethics 72.0 70.0 clinical_knowledge 75.1 67.2 college_biology 79.9 70.8 college_chemistry 51.0 45.0 college_computer_science 51.0 49.0 college_mathematics 32.0 37.0 college_medicine 66.5 60.1 college_physics 46.1 34.3 computer_security 76.0 65.0 conceptual_physics 67.2 49.4 econometrics 38.6 43.0 electrical_engineering 62.1 60.0 elementary_mathematics 41.5 33.6 formal_logic 33.3 35.7 global_facts 39.0 38.0 high_school_biology 80.3 71.3 high_school_chemistry 58.1 47.8 high_school_computer_science 58.0 54.0 high_school_european_history 78.8 72.1 high_school_geography 86.4 76.8 high_school_gov_and_politics 91.2 83.9 high_school_macroeconomics 70.5 65.1 high_school_mathematics 31.9 23.7 high_school_microeconomics 77.7 66.4 high_school_physics 36.4 33.8 high_school_psychology 86.6 81.8 high_school_statistics 58.8 50.0 high_school_us_history 83.3 78.9 high_school_world_history 85.2 75.1 human_aging 77.6 66.4 human_sexuality 86.3 67.2 international_law 90.9 77.7 jurisprudence 79.6 71.3 logical_fallacies 80.4 72.4 machine_learning 41.1 41.1 management 82.5 77.7 marketing 89.7 83.3 medical_genetics 69.0 69.0 miscellaneous 84.5 75.7 moral_disputes 77.5 66.8 moral_scenarios 36.5 40.2 nutrition 77.1 69.9 philosophy 79.4 68.8 prehistory 81.2 67.6 professional_accounting 52.1 44.3 professional_law 56.5 44.5 professional_medicine 75.4 64.0 professional_psychology 75.7 68.1 public_relations 73.6 71.8 security_studies 75.9 64.9 sociology 91.0 84.1 us_foreign_policy 92.0 81.0 virology 53.6 47.0 world_religions 87.7 84.2 Table A6 | Chinchilla MMLU results.
中文: Chinchilla Gopher Task Chinchilla Gopher抽象 alpher抽象 algebra 31.0 25.0 解剖学 70.4 56.3 天文学 73.0 商业 伦理学 72.0 70.0 临床 知识 75.1 67.2 学院 生物学 79.9 70.8 学院 化学 51.0 45.0 学院 计算机 科学 51.0 49.0 学院 数学 32.0 37.0 学院 医学 66.5 60.1 学院 物理 46.1 34.3 计算机 安全学 67.0 65.0 概念 物理 67.2 49.4 计量经济学 38.6 43.0 电气 工程 62.1 小学 数学 41.5 33.6 正规学 33.3 全球学 39.0 38.0 高中 生物学 80.3 高中 化学 58.1 47.8 高中 计算机 科学学 58.0 54.0 高中 神经学 72.8 高中 地理学 86.4 76.8 高中 gov-gov and 政治学 9.1.2 83.9 高中 宏观经济学 高校-心理学 高校学 31.7 高校-数学 31. y 86.6 81.8 高校 统计学 58.8 50.0 高校 历史 历史 历史 世界 历史 历史 历史 历史 人类83.3 78.9 高校 历史 历史77.6 人类 性学 86.3 67.2 国际-法律 90.9 77.7 判例 79.6 71.3 逻辑-谬误 80.4 机器 学习 41.1 管理 82.5 77.7 营销 89.7 83.3 医学 遗传学 69.0 杂项 84.5 75.7 道德 辩论 77.5 66.8 道德 假设 36.5 40.2 营养 77.1 69.9 哲学 79.4 职业-会计 52.1 44.3 专业 法律 56.5 专业-心理学 75.7 64.0 公共-关系 75.6 71.8 安全 研究 75.9 64.9 社会学 91.9 社会学 91.1 我们 外交政策 92.0 病毒学 53.6 47.0 世界 - 宗教学 87.7 84.2 表格 A6 o- Chinchilla MMMLU 成果。
<a id="S0502"></a> Source: p.31 S0502
Original: For each subset of MMLU (Hendrycks et al., 2020), we show Chinchilla’s accuracy compared to Gopher.
中文: 对于MMLU的每个子集(Hendrycks等,2020年),我们显示钦奇拉与Gopher的准确性.
<a id="S0503"></a> Source: p.31 S0503
Original: Model Details Organization Developing the Model DeepMind Model Date March 2022 Model Type Autoregressive Transformer Language Model (Section 4.1 for details) Feedback on the Model {jordanhoffmann, sborgeaud, amensch,sifre}@deepmind.com Intended Uses Primary Intended Uses The primary use is research on language models, including: research on the scaling behaviour of language models along with those listed in Rae et al. (2021). 31
中文: 模型细节组织 开发模型 DeepMind 模型 日期:2022年3月 模型类型 自动回转变压器语言模型(细节第4.1节) 关于模型 {jordanhoffmann, sborgeaud, amensch, sifre}@deepmind.com 原始用途的反馈 主要用途是研究语言模型,包括:研究语言模型与Rae等人(2021年)所列语言模型的缩放行为。 31个
<a id="S0504"></a> Source: p.32 S0504
Original: Primary Intended Users DeepMind researchers.
中文: 初等意向用户深明研究者.
<a id="S0505"></a> Source: p.32 S0505
Original: We will not make this model available publicly.
中文: 我们不会公开这一模式。
<a id="S0506"></a> Source: p.32 S0506
Original: Out-of-Scope Uses Uses of the language model for language generation in harmful or deceitful settings.
中文: 在有害或欺骗的环境中使用语言生成模式。
<a id="S0507"></a> Source: p.32 S0507
Original: More generally, the model should not be used for downstream applications without further safety and fairness mitigations.
中文: 更一般地说,如果不进一步的安全性和公平性减缓,该模型不应用于下游应用.
<a id="S0508"></a> Source: p.32 S0508
Original: Factors Card Prompts – Relevant Factor Relevant factors include which language is used.
中文: 因素卡提示 — 相关因素包括所使用的语言。
<a id="S0509"></a> Source: p.32 S0509
Original: Furthermore, in the analysis of models trained on the same corpus in Rae et al. (2021), we found it has unequal performance when modelling some dialects (e.g., African American English).
中文: 此外,在对在Rae等人(2021年)中接受过相同教程培训的模型进行分析时,我们发现,在建模一些方言(如非裔美国人英语)时,它的表现是不平等的.
<a id="S0510"></a> Source: p.32 S0510
Original: The model should not be used for downstream applications without further analysis on factors in the proposed downstream application.
中文: 如果不进一步分析拟议下游应用中的各种因素,该模型不应用于下游应用。
<a id="S0511"></a> Source: p.32 S0511
Original: Card Prompts – Evaluation Factors See the results in Rae et al. (2021) which analyzes models trained on the same text corpus.
中文: 卡片提示 - 评价因素 见Rae等人(2021年)的研究结果,其中分析了对同一文本库进行培训的模型。
<a id="S0512"></a> Source: p.32 S0512
Original: Metrics Model Performance Measures • Perplexity and bits per byte on language modelling datasets • Accuracy on completion tasks, reading comprehension, MMLU, BIG-bench and fact checking. • Exact match accuracy for question answering. • Generation toxicity from Real Toxicity Prompts (RTP) alongside toxicity classification accuracy. • Gender and occupation bias.
中文: 量子模型性能测量 语言建模数据集上每字节的复杂度和比特度 完成任务的精确度,读取理解度,MMLU,BIG-bench和事实检查. • 回答问题准确无误。 • 从实际毒性快感产生毒性,同时进行毒性分类的准确性。 • 性别和职业偏见。
<a id="S0513"></a> Source: p.32 S0513
Original: Test include comparing the probability of generating different gender terms and the Winogender coreference resolution task.
中文: 测试包括比较生成不同性别名词的概率和Winogender的对齐解析任务.
<a id="S0514"></a> Source: p.32 S0514
Original: We principally focus on Chinchilla’s performance compared to Gopher on text likelihood prediction.
中文: 我们主要关注Chinchilla的性能,
<a id="S0515"></a> Source: p.32 S0515
Original: Decision thresholds N/A Approaches to Uncertainty and Vari- Due to the costs of training large language models, we did ability not train Chinchilla multiple times.
中文: 决定阈值N/A 处理不确定性和Vari -- -- 由于培训大型语言模型的费用,我们未能多次培训Chinchilla。
<a id="S0516"></a> Source: p.32 S0516
Original: However, the breadth of our evaluation on a range of different task types gives a reasonable estimate of the overall performance of the model.
中文: 然而,我们对一系列不同任务类型的评价范围很广,可以合理地估计模型的总体性能。
<a id="S0517"></a> Source: p.32 S0517
Original: Furthermore, the existence of another large model trained on the same dataset (Gopher) provides a clear point of comparison.
中文: 此外,在同一个数据集(Gopher)上培训的另一个大型模型的存在提供了明确的比较点。
<a id="S0518"></a> Source: p.33 S0518
Original: Datasets • Language modelling on LAMBADA, Wikitext103 (Merity et al., 2017), C4 (Raffel et al., 2020a), PG-19 (Rae et al., 2020) and the Pile (Gao et al., 2020). • Language understanding, real world knowledge, mathematical and logical reasoning on the Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2020) and on the “Beyond the Imitation Game Benchmark” (BIG-bench) (BIG-bench collaboration, 2021). • Question answering (closed book) on Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). • Reading comprehension on RACE (Lai et al., 2017) • Common sense understanding on HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), Winogrande (Sakaguchi et al., 2020), SIQA (Sap et al., 2019), BoolQ (Clark et al., 2019), and TruthfulQA (Lin et al., 2021).
中文: 数据集 • LAMBADA的语言建模,Wikitext103(Merity等,2017年),C4(Raffel等,2020年a),PG-19(Rae等,2020年)和Pile(Gao等,2020年). · 语言理解、现实世界知识、数学和逻辑推理,关于大规模多任务语言理解(MMLU)基准(Hendrycks等,2020年)和“超越模仿游戏基准”(BIG-bench)(BIG-bench合作,2021年)。 • 关于自然问题(Kwiatkowski等,2019年)和TriviaQA(Joshi等,2017年)的问答(非公开) • 阅读对RACE的理解(Lai等,2017年) • HellaSwag(Zellers等,2019年),PIQA(Bisk等,2020年),Winogrande(Sakaguchi等,2020年),SIQA(Sap等,2019年),BoulQ(Clark等,2019年),以及TrealQA(Lin等,2021年)。
<a id="S0519"></a> Source: p.33 S0519
Original: Motivation We chose evaluations from Rae et al. (2021) to allow us to most directly compare to Gopher.
中文: 动机 我们从Rae等人(2021年)中选择了评价,让我们能最直接地与Gopher进行比较.
<a id="S0520"></a> Source: p.33 S0520
Original: Preprocessing Input text is tokenized using a SentencePiece tokenizer with a vocabulary of size 32,000.
中文: 预处理输入文本使用有32,000个词汇的PriestPiece符号化.
<a id="S0521"></a> Source: p.33 S0521
Original: Unlike the tokenizer used for Gopher, the tokenizer used for Chinchilla does not perform NFKC normalization.
中文: 与用于Gopher的活化剂不同的是,用于Chinchilla的活化剂不执行NFKC正常化.
<a id="S0522"></a> Source: p.33 S0522
Original: Training Data The same dataset is used as in Rae et al. (2021).
中文: 培训数据 与Rae等人(2021年)使用的数据集相同。
<a id="S0523"></a> Source: p.33 S0523
Original: Differences in sampling are shown in Table A1.
中文: 表A1显示抽样差异。
<a id="S0524"></a> Source: p.33 S0524
Original: Quantitative Analyses Unitary Results Section 4.2 gives a detailed description of our analysis.
中文: 第4.2节详细介绍了我们的分析。
<a id="S0525"></a> Source: p.33 S0525
Original: Main take-aways include: • Our model is capable of outputting toxic language as measured by the PerspectiveAPI.
中文: 主要外购包括: • 我们的模型能够输出以PerspectAPI所测量的有毒语言.
<a id="S0526"></a> Source: p.33 S0526
Original: This is particularly true when the model is prompted with toxic prompts. • Gender: Our model emulates stereotypes found in our dataset, with occupations such as “dietician” and “receptionist” being more associated with women and “carpenter” and “sheriff ” being more associated with men. • Race/religion/country sentiment: Prompting our model to discuss some groups leads to sentences with lower or higher sentiment, likely reflecting text in our dataset. 33
中文: 当该模型被有毒的诱发时,情况尤其如此。 • 性别:我们的模型模仿了我们数据集中的陈规定型观念,诸如“二手手”和“接受者”等职业与妇女的关系更加密切,“木匠”和“牧人”与男子的关系更加密切。 • 种族/宗教/国家情绪:促使我们的模式讨论某些群体,导致情绪低或高的句子,可能在我国数据集中反映文字。 第33条
<a id="S0527"></a> Source: p.34 S0527
Original: Intersectional Results We did not investigate intersectional biases.
中文: 跨部门结果 我们没有调查交叉偏见。
<a id="S0528"></a> Source: p.34 S0528
Original: Ethical Considerations Data The data is the same as described in Rae et al. (2021).
中文: 伦理考虑数据 数据与Rae等人(2021年)所描述的数据相同.
<a id="S0529"></a> Source: p.34 S0529
Original: Human Life The model is not intended to inform decisions about matters central to human life or flourishing.
中文: 人类生命 该模式无意为决定对人类生活或繁荣至关重要的事项提供信息。
<a id="S0530"></a> Source: p.34 S0530
Original: Mitigations We considered filtering the dataset to remove toxic content but decided against it due to the observation that this can introduce new biases as studied by Welbl et al. (2021).
中文: 缓解 我们曾考虑过滤数据集去除有毒内容,但由于Welbl等人(2021年)认为这可能带来新的偏见,我们决定不这样做。
<a id="S0531"></a> Source: p.34 S0531
Original: More work is needed on mitigation approaches to toxic content and other types of risks associated with language models, such as those discussed in Weidinger et al. (2021).
中文: 需要开展更多的工作,以采取减缓有毒成分和其他与语言模型有关的风险的方法,如在Weidinger等人(2021年)中讨论的那些方法。
<a id="S0532"></a> Source: p.34 S0532
Original: Risks and Harms The data is collected from the internet, and thus undoubtedly there is toxic/biased content in our training dataset.
中文: 风险和损害 数据从互联网上收集,因此我们的培训数据集无疑含有有毒/有偏见的内容。
<a id="S0533"></a> Source: p.34 S0533
Original: Furthermore, it is likely that personal information is also in the dataset that has been used to train our models.
中文: 此外,个人信息也很可能出现在用于培训我们的模型的数据集中。
<a id="S0534"></a> Source: p.34 S0534
Original: We defer to the more detailed discussion in Weidinger et al. (2021).
中文: 我们推迟到Weidinger等人(2021年)进行更详细的讨论。
<a id="S0535"></a> Source: p.34 S0535
Original: Use Cases Especially fraught use cases include the generation of factually incorrect information with the intent of distributing it or using the model to generate racist, sexist or otherwise toxic text with harmful intent.
中文: 使用案例特别繁琐的使用案例包括产生事实不正确的信息,目的是传播这些信息或使用该模型生成带有有害意图的种族主义、性别歧视或其他有毒文字。
<a id="S0536"></a> Source: p.34 S0536
Original: Many more use cases that could cause harm exist.
中文: 还有更多的案件可能造成损害。
<a id="S0537"></a> Source: p.34 S0537
Original: Such applications to malicious use are discussed in detail in Weidinger et al. (2021).
中文: Weidinger等人(2021年)详细讨论了这种恶意使用应用。
<a id="S0538"></a> Source: p.34 S0538
Original: We follow the framework presented in Mitchell et al. (2019). J.
中文: 我们遵循Mitchell等人提出的框架(2019年)。 J.
<a id="S0539"></a> Source: p.34 S0539
Original: List of trained models In Table A9 we list the model size and configuration of all models used in this study.
中文: 在表A9中,我们列出了本研究中使用的所有模型的模型大小和配置。
<a id="S0540"></a> Source: p.34 S0540
Original: Many models have been trained multiple times, for a different number of training steps. 34
中文: 许多模式经过多次培训,为不同的培训步骤。 页:1
<a id="S0541"></a> Source: p.35 S0541
Original: Task Chinchilla Gopher Task Chinchilla Gopher hyperbaton 54.2 51.7 movie_dialog_same_or_diff 54.5 50.7 causal_judgment 57.4 50.8 winowhy 62.5 56.7 formal_fallacies_syllogisms_neg 52.1 50.7 movie_recommendation 75.6 50.5 crash_blossom 47.6 63.6 moral_permissibility 57.3 55.1 discourse_marker_prediction 13.1 11.7 strategyqa 68.3 61.0 general_knowledge_json 94.3 93.9 nonsense_words_grammar 78.0 61.4 sports_understanding 71.0 54.9 metaphor_boolean 93.1 59.3 implicit_relations 49.4 36.4 navigate 52.6 51.1 penguins_in_a_table 48.7 40.6 presuppositions_as_nli 49.9 34.0 intent_recognition 92.8 88.7 temporal_sequences 32.0 19.0 reasoning_about_colored_objects 59.7 49.2 question_selection 52.6 41.4 logic_grid_puzzle 44.0 35.1 logical_fallacy_detection 72.1 58.9 timedial 68.8 50.9 physical_intuition 79.0 59.7 epistemic_reasoning 60.6 56.4 physics_mc 65.5 50.9 ruin_names 47.1 38.6 identify_odd_metaphor 68.8 38.6 hindu_knowledge 91.4 80.0 understanding_fables 60.3 39.6 misconceptions 65.3 61.7 logical_sequence 64.1 36.4 implicatures 75.0 62.0 mathematical_induction 47.3 57.6 disambiguation_q 54.7 45.5 fantasy_reasoning 69.0 64.1 known_unknowns 65.2 63.6 SNARKS 58.6 48.3 dark_humor_detection 66.2 83.1 crass_ai 75.0 56.8 analogical_similarity 38.1 17.2 entailed_polarity 94.0 89.5 sentence_ambiguity 71.7 69.1 irony_identification 73.0 69.7 riddle_sense 85.7 68.2 evaluating_info_essentiality 17.6 16.7 date_understanding 52.3 44.1 phrase_relatedness 94.0 81.8 analytic_entailment 67.1 53.0 novel_concepts 65.6 59.1 odd_one_out 70.9 32.5 empirical_judgments 67.7 52.5 logical_args 56.2 59.1 figure_of_speech_detection 63.3 52.7 alignment_questionnaire 91.3 79.2 english_proverbs 82.4 57.6 similarities_abstraction 87.0 81.8 Human_organs_senses_mcc 85.7 84.8 anachronisms 69.1 56.4 gre_reading_comprehension 53.1 27.3 Table A7 | Chinchilla BIG-bench results.
中文: Chinchilla Gopher Task Chinchilla Gopher超棒54.2 51.7 电影 dialog same or diff 54.5 50.7 因果关系 判断 57.4 50.8 wono why 62.5 56.7 形式-错误 syllogism neg 52.1 电影 建议 75.6 50.5 崩溃 blossom 47.6 63.6 道德 允许性 57.3 55.1 讨论 标记 预测 13.1 11.7 战略qa 68.3 61.0 一般 知识 json 94.3 93.9 废话 worders grammar 78.0 体育 理解 71.0 54.9 隐喻 49.1 59.3 隐含 关系 49.4 36.4 航行 52.6 预置位置 as-nli 49.9 34.0 意向 承认 92.8 暂时静态 32.0 19.0 推理 about-colored gridge pligal pattle-pattal 44. hor 68.8 38.6 hindu knowledge 91.4 80.0 理解 可知 60.3 39.6 误解 65.3 61.7 逻辑-序列 64.1 36.4 涉及 75.0 62.0 数学-诱导 47.3 57.6 离散-q 54.7 45.5 幻想-理性 69.0 64.1 已知 不明 65.2 63.6 SNARKS 58.6 48.3 暗-humor-detection 66.2 83.1 crass ai 75.0 56.8 模拟-同义 38.1 17.2 涉及-极性 94.0.5 句 - 矛盾 71.7 69.1 讽刺-认同 73.0 69.7 谜 seense 85.7 68.2 评价 信息 本质 17.6 16.7 日期 - 理解 52.3 44.1 短语 - 关联 94.0 analytic-entailment 67.1 53.0 小说 65.6 59.1-out 70.9 32.5 经验-judigments 67.7 逻辑 -args 56.1 图 speech deter-deterron o-oconc-oc-o-o 表53.1 27.3 表A7 钦奇拉BIG结果。
<a id="S0542"></a> Source: p.35 S0542
Original: For each subset of BIG-bench (BIG-bench collaboration, 2021), we show Chinchilla and Gopher’s accuracy. 35
中文: 对于BIG-bench(BIG-bench company, 2021)的每一个子集, 35个
<a id="S0543"></a> Source: p.36 S0543
Original: Parameters (million) d_model ffw_size kv_size n_heads n_layers 44 512 2048 64 8 8 57 576 2304 64 9 9 74 640 2560 64 10 10 90 640 2560 64 10 13 106 640 2560 64 10 16 117 768 3072 64 12 12 140 768 3072 64 12 15 163 768 3072 64 12 18 175 896 3584 64 14 14 196 896 3584 64 14 16 217 896 3584 64 14 18 251 1024 4096 64 16 16 278 1024 4096 64 16 18 306 1024 4096 64 16 20 425 1280 5120 128 10 18 489 1280 5120 128 10 21 509 1408 5632 128 11 18 552 1280 5120 128 10 24 587 1408 5632 128 11 21 632 1536 6144 128 12 19 664 1408 5632 128 11 24 724 1536 6144 128 12 22 816 1536 6144 128 12 25 893 1792 7168 128 14 20 1,018 1792 7168 128 14 23 1,143 1792 7168 128 14 26 1,266 2048 8192 128 16 22 1,424 2176 8704 128 17 22 1,429 2048 8192 128 16 25 1,593 2048 8192 128 16 28 1,609 2176 8704 128 17 25 1,731 2304 9216 128 18 24 1,794 2176 8704 128 17 28 2,007 2304 9216 128 18 28 2,283 2304 9216 128 18 32 2,298 2560 10240 128 20 26 2,639 2560 10240 128 20 30 2,980 2560 10240 128 20 34 3,530 2688 10752 128 22 36 3,802 2816 11264 128 22 36 4,084 2944 11776 128 22 36 4,516 3072 12288 128 24 36 6,796 3584 14336 128 28 40 9,293 4096 16384 128 32 42 11,452 4352 17408 128 32 47 12,295 4608 18432 128 36 44 12,569 4608 18432 128 32 47 13,735 4864 19456 128 32 47 14,940 4992 19968 128 32 49 16,183 5120 20480 128 40 47 Table A9 | All models.
中文: 参数(百万) d model ffw-size kv size n-heads n-layers 44 512 2048 64 8 57 57 57 2304 64 9 9 9 74 640 540 2560 64 10 90 640 2560 64 10 106 640 2560 64 10 117 768 3072 64 12 14 15 163 768 3072 64 12 175 896 3584 64 14 14 196 896 3584 14 217 896 3584 64 14 18 251 1024 4096 64 16 278 1024 4096 16 306 10 304 10 304 1024 1096 406 64 16 16 204 204 16 204 128 23 1768 71 128 28 261 20262 206 204 204 204 204 204 204 204 128 2 128 128 28 28 60 10240 128 20 26 2 639 2560 10240 128 20 30 2 980 2560 10240 128 20 34 3 530 2688 10752 128 22 36 3 802 2816 11264 128 22 36 4 084 2944 11776 128 22 36 4 516 3072 12288 128 24 36 6 7796 3584 14336 128 28 40 9 293 4096 16384 128 32 11 452 4352 17408 128 32 47 12 295 4608 128 36 44 12 569 4608 18432 128 32 47 13 735 48646 19456 128 32 47 14 940 499 19968 128 32 49 183 5 5 20 204 128 40 47 表 A9 |.
<a id="S0544"></a> Source: p.36 S0544
Original: We list the hyperparameters and size of all models trained as part of this work.
中文: 作为这项工作的一部分,我们列出所培训的所有模型的超参数和大小。
<a id="S0545"></a> Source: p.36 S0545
Original: Many shown models have been trained with multiple learning rate schedules/number of training tokens. 36
中文: 许多已显示的模型都接受了多种学习进度表/培训标志数量的培训。 第36条