Language Models are Few-Shot Learners Tom B. - 中英文对照
translated: 2026-07-16
title: "Language Models are Few-Shot Learners Tom B." aliases: - "GPT3" - "arXiv:2005.14165" source: "https://arxiv.org/abs/2005.14165" arxiv: "2005.14165" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning - nlp
Language Models are Few-Shot Learners Tom B. - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Language Models are Few-Shot Learners Tom B.
中文: 语言模型是"少有"的学习者汤姆·B.
<a id="S0002"></a> Source: p.1 S0002
Original: Brown∗ Benjamin Mann∗ Nick Ryder∗ Melanie Subbiah∗ Jared Kaplan† Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M.
中文: 布朗 本杰明·曼 尼克·赖德 梅兰妮·苏比亚 贾里德·卡普兰 * 普拉富拉·达里瓦尔·阿尔文德·内拉坎丹 普拉纳瓦·斯亚姆·吉里什·萨斯特里·阿曼达 阿斯海尼·桑德希尼·阿加瓦尔·阿里尔·赫伯特-沃斯·格雷琴·克虏格·汤姆·赫尼汉·雷园儿童 Aditya Ramesh Daniel M.
<a id="S0003"></a> Source: p.1 S0003
Original: Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei OpenAI Abstract Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
中文: 齐格勒·杰弗里·吴·克勒门斯·温特·克里斯托弗·黑森 马克·陈·埃里克·西格勒·马特乌斯·利特温·斯科特·本杰明·切斯 杰克·克拉克·克里斯托弗·伯纳 萨姆·麦坎德利希·亚历克·拉德福德·伊利亚·苏特斯克韦尔·达里奥·阿莫德艾·OpenAI摘要 近期的工作表明,在许多NLP任务和基准方面,通过对大量文本进行预先培训,然后对具体任务进行精细调整,取得了很大进展。
<a id="S0004"></a> Source: p.1 S0004
Original: While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples.
中文: 这种方法虽然在架构中通常具有任务不可知性,但仍需针对特定任务进行精细调整数据集,以上千或上万例为例.
<a id="S0005"></a> Source: p.1 S0005
Original: By contrast, humans can generally perform a new language task from only a few examples or from simple instructions – something which current NLP systems still largely struggle to do.
中文: 相形之下,人类一般只能从几个例子或简单的指令中完成一个新的语言任务——这是目前NLP系统基本上仍然难以完成的.
<a id="S0006"></a> Source: p.1 S0006
Original: Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches.
中文: 在这里,我们显示,扩大语言模型大大改进了任务不可知性,少镜头的性能,有时甚至以之前最先进的微调方法达到竞争力.
<a id="S0007"></a> Source: p.1 S0007
Original: Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting.
中文: 具体来说,我们训练出GPT-3,一种具有1,750亿个参数的自相递入式语言模型,比以前任何非散射语言模型都多出10x,并在几发设定中测试其性能.
<a id="S0008"></a> Source: p.1 S0008
Original: For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model.
中文: 对所有任务,GPT-3在应用时都没有任何梯度更新或微调,任务和几发显示完全通过与模型的文本交互指定.
<a id="S0009"></a> Source: p.1 S0009
Original: GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.
中文: GPT-3在很多NLP数据集上都取得了强效的性能,包括翻译,问答,和血栓等任务,以及需要飞行上推理或域上适应的几项任务,例如:不打字,在句子中使用新词,或者进行3位数的算术.
<a id="S0010"></a> Source: p.1 S0010
Original: At the same time, we also identify some datasets where GPT-3’s few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora.
中文: 同时,我们也发现了一些GPT-3的几发学习仍然困难的数据集,以及一些GPT-3面临与大型网络公司培训相关的方法问题的数据集.
<a id="S0011"></a> Source: p.1 S0011
Original: Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans.
中文: 最后,我们发现GPT-3可以生成人类评价人员难以与人类所写文章区分的新闻文章样本.
<a id="S0012"></a> Source: p.1 S0012
Original: We discuss broader societal impacts of this finding and of GPT-3 in general. ∗Equal contribution †Johns Hopkins University, OpenAI Author contributions listed at end of paper. 0202 luJ 22 ]LC.sc[ 4v56141.5002:viXra
中文: 我们讨论这一结论和一般的GPT-3的社会影响。 * 平等贡献 Johns Hopkins大学,OpenAI 作者贡献,列于论文末尾。 0202 luJ 22] LC.sc [4v56141.5002:viXra (英语).
<a id="S0013"></a> Source: p.2 S0013
Original: Contents 1 Introduction 3 2 Approach 6 2.1 Model and Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Training Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3 Results 10 3.1 Language Modeling, Cloze, and Completion Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 Closed Book Question Answering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3 Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.4 Winograd-Style Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3.5 Common Sense Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.6 Reading Comprehension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.7 SuperGLUE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.8 NLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.9 Synthetic and Qualitative Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 4 Measuring and Preventing Memorization Of Benchmarks 29 5 Limitations 33 6 Broader Impacts 34 6.1 Misuse of Language Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 6.2 Fairness, Bias, and Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 6.3 Energy Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 7 Related Work 39 8 Conclusion 40 A Details of Common Crawl Filtering 43 B Details of Model Training 43 C Details of Test Set Contamination Studies 43 D Total Compute Used to Train Language Models 46 E Human Quality Assessment of Synthetic News Articles 46 F Additional Samples from GPT-3 48 G Details of Task Phrasing and Specifications 50 H Results on All Tasks for All Model Sizes 63 2
中文: 目录 1 入门 3 二 二 二 二 三 二 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三三 三 三 三 三 三 三 三 三三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三 三四三 三 三 三 三 三 三 三四三 三 三 三 三 三 三四 三 三 三 三 三 三四三 三 三 三 三 三 三 三 三 三 三三 三 三 三 三 三 三 三 三 三 三 三四三 三 三 三 三 三 三 三四三 三 三 三 三四三 三 三 三 三 三 三 三 三 三四三 三 三 三 三 三 三 三 三 三 三四四四三 三 三 三 三 三 三 三 三 三四四三四三 三 三 (原始内容存档于2018-10-25). Methical Communicipality.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.c.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com.com 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 36 6.3 能源使用量 能源使用量 能源使用量 能源使用量 36 6.3 能源使用量 能源
<a id="S0014"></a> Source: p.3 S0014
Original: 1 Introduction Recent years have featured a trend towards pre-trained language representations in NLP systems, applied in increasingly flexible and task-agnostic ways for downstream transfer.
中文: 1 导言. 近年来出现了一种趋势,即在NLP系统中采用经过预先培训的语言表述方式,采用越来越灵活和任务不可知的方式进行下游转移。
<a id="S0015"></a> Source: p.3 S0015
Original: First, single-layer representations were learned using word vectors [MCCD13, PSM14] and fed to task-specific architectures, then RNNs with multiple layers of representations and contextual state were used to form stronger representations [DL15, MBXS17, PNZtY18] (though still applied to task-specific architectures), and more recently pre-trained recurrent or transformer language models [VSP+17] have been directly fine-tuned, entirely removing the need for task-specific architectures [RNSS18, DCLT18, HR18].
中文: 首先,单层表示学是使用单词矢量[MCCD13,PSM14]并被输入任务特定架构,然后使用多层表示和上下文状态的RNN来形成更强的表示DL15,MBXS17,PNZtY18,而更近的预训的经常性或变压器语言模型[VSP+17]被直接微调,完全取消了任务特定架构[RNS18,DCLT18,HR18]的需要.
<a id="S0016"></a> Source: p.3 S0016
Original: This last paradigm has led to substantial progress on many challenging NLP tasks such as reading comprehension, question answering, textual entailment, and many others, and has continued to advance based on new architectures and algorithms [RSR+19, LOG+19, YDY+19, LCG+19].
中文: 最后一种范式使得许多具有挑战性的NLP任务取得了实质性进展,如阅读理解,问答,文字涵义等,并基于新的架构和算法继续推进[RSR+19,LOG+19,YDY+19,LCG+19].
<a id="S0017"></a> Source: p.3 S0017
Original: However, a major limitation to this approach is that while the architecture is task-agnostic, there is still a need for task-specific datasets and task-specific fine-tuning: to achieve strong performance on a desired task typically requires fine-tuning on a dataset of thousands to hundreds of thousands of examples specific to that task.
中文: 然而,这种方法的一个主要限制是,虽然结构是任务不可知的,但仍需要针对具体任务的数据集和任务具体的微调:要在所期望的任务上取得强有力的业绩,通常需要精细地调整专门涉及该任务的数千至数十万个实例的数据集。
<a id="S0018"></a> Source: p.3 S0018
Original: Removing this limitation would be desirable, for several reasons.
中文: 出于若干原因,最好取消这一限制。
<a id="S0019"></a> Source: p.3 S0019
Original: First, from a practical perspective, the need for a large dataset of labeled examples for every new task limits the applicability of language models.
中文: 首先,从实用的角度来看,每个新任务都需要一个有标签的大型实例数据集,这限制了语言模型的适用性.
<a id="S0020"></a> Source: p.3 S0020
Original: There exists a very wide range of possible useful language tasks, encompassing anything from correcting grammar, to generating examples of an abstract concept, to critiquing a short story.
中文: 存在着非常广泛的可能有用的语言任务,包括从修正语法到产生抽象概念的例子,到贬低一个短篇故事的任何内容.
<a id="S0021"></a> Source: p.3 S0021
Original: For many of these tasks it is difficult to collect a large supervised training dataset, especially when the process must be repeated for every new task.
中文: 对于其中许多任务来说,很难收集一个大型的有监督的培训数据集,特别是当每个新任务都必须重复这一过程时。
<a id="S0022"></a> Source: p.3 S0022
Original: Second, the potential to exploit spurious correlations in training data fundamentally grows with the expressiveness of the model and the narrowness of the training distribution.
中文: 第二,在培训数据中利用虚假关联的潜力随着模型的表达性和培训分布的狭窄而从根本上增长.
<a id="S0023"></a> Source: p.3 S0023
Original: This can create problems for the pre-training plus fine-tuning paradigm, where models are designed to be large to absorb information during pre-training, but are then fine-tuned on very narrow task distributions.
中文: 这可能会给培训前加微调范式造成问题,其中模型被设计为大以在培训前阶段吸收信息,但随后被微调到非常狭窄的任务分配上.
<a id="S0024"></a> Source: p.3 S0024
Original: For instance [HLW+20] observe that larger models do not necessarily generalize better out-of-distribution.
中文: 例如,[HLW+20]指出,较大的模型不一定能普遍地更好地分配。
<a id="S0025"></a> Source: p.3 S0025
Original: There is evidence that suggests that the generalization achieved under this paradigm can be poor because the model is overly specific to the training distribution and does not generalize well outside it [YdC+19, MPL19].
中文: 有证据表明,在这种范式下实现的通俗化可能很差,因为该模式对培训分配过于具体化,而且不在其外很好地通俗化[YdC+19,MPL19].
<a id="S0026"></a> Source: p.3 S0026
Original: Thus, the performance of fine-tuned models on specific benchmarks, even when it is nominally at human-level, may exaggerate actual performance on the underlying task [GSL+18, NK19].
中文: 因此,在具体基准上微调模型的性能,即使名义上属于人,也可能夸大基本任务的实际性能[GSL+18,NK19].
<a id="S0027"></a> Source: p.3 S0027
Original: Third, humans do not require large supervised datasets to learn most language tasks – a brief directive in natural language (e.g. “please tell me if this sentence describes something happy or something sad”) or at most a tiny number of demonstrations (e.g. “here are two examples of people acting brave; please give a third example of bravery”) is often Figure 1.1: Language model meta-learning.
中文: 第三,人类不需要大型的有监督的数据集来学习大多数语言任务 — — 一个自然语言的简短指令(例如“请告诉我这句话是否描述了快乐或悲伤的东西”),或者最多是少量的演示(例如“这里有两个人勇敢行动的例子;请举出第三个勇敢的例子”)通常是图1.1:语言模型元学习。
<a id="S0028"></a> Source: p.3 S0028
Original: During unsupervised pre-training, a language model develops a broad set of skills and pattern recognition abilities.
中文: 在不受监督的预训期间,一个语言模型发展出一套广泛的技能和模式识别能力.
<a id="S0029"></a> Source: p.3 S0029
Original: It then uses these abilities at inference time to rapidly adapt to or recognize the desired task.
中文: 然后在推断时利用这些能力来迅速适应或识别所期望的任务.
<a id="S0030"></a> Source: p.3 S0030
Original: We use the term “in-context learning” to describe the inner loop of this process, which occurs within the forward-pass upon each sequence.
中文: 我们使用“文中学习”一词来描述这一过程的内在循环,这种循环发生在每一序列上。
<a id="S0031"></a> Source: p.3 S0031
Original: The sequences in this diagram are not intended to be representative of the data a model would see during pre-training, but are intended to show that there are sometimes repeated sub-tasks embedded within a single sequence. 3
中文: 本图中的序列无意代表模型在预训期间会看到的数据,而是为了显示有时会重复嵌入于单个序列中的子任务. 3个
<a id="S0032"></a> Source: p.4 S0032
Original: Figure 1.2: Larger models make increasingly efficient use of in-context information.
中文: 图1.2:大模型越来越有效地利用内文信息。
<a id="S0033"></a> Source: p.4 S0033
Original: We show in-context learning performance on a simple task requiring the model to remove random symbols from a word, both with and without a natural language task description (see Sec. 3.9.2).
中文: 我们在一项简单的任务上显示内容内学习的性能,要求模型从一个单词中去除随机符号,既包括自然语言任务描述,也包括没有自然语言任务描述(见Sec. 3.9.2).
<a id="S0034"></a> Source: p.4 S0034
Original: The steeper “in-context learning curves” for large models demonstrate improved ability to learn a task from contextual information.
中文: 大型模型的更陡峭的“背景学习曲线”表明,从背景信息中学习一项任务的能力有所提高。
<a id="S0035"></a> Source: p.4 S0035
Original: We see qualitatively similar behavior across a wide range of tasks. sufficient to enable a human to perform a new task to at least a reasonable degree of competence.
中文: 我们从质的上看,在一系列广泛的任务中都有类似的行为。 足以使一个人能够至少以合理的能力执行新的任务。
<a id="S0036"></a> Source: p.4 S0036
Original: Aside from pointing to a conceptual limitation in our current NLP techniques, this adaptability has practical advantages – it allows humans to seamlessly mix together or switch between many tasks and skills, for example performing addition during a lengthy dialogue.
中文: 除了指出我们目前NLP技术中的概念限制外,这种适应性还具有实际的优点——它允许人类在很多任务和技能之间无缝地混合或切换,例如在长时间的对话中进行附加.
<a id="S0037"></a> Source: p.4 S0037
Original: To be broadly useful, we would someday like our NLP systems to have this same fluidity and generality.
中文: 为了广泛发挥作用,我们有一天希望我们的NLP系统具有同样的流动性和通俗性。
<a id="S0038"></a> Source: p.4 S0038
Original: One potential route towards addressing these issues is meta-learning1 – which in the context of language models means the model develops a broad set of skills and pattern recognition abilities at training time, and then uses those abilities at inference time to rapidly adapt to or recognize the desired task (illustrated in Figure 1.1).
中文: 解决这些问题的一个潜在途径是元学习,1 在语言模式方面,这意味着该模式在培训时发展出一套广泛的技能和模式识别能力,然后在推断时利用这些能力来迅速适应或识别所期望的任务(见图一.1)。
<a id="S0039"></a> Source: p.4 S0039
Original: Recent work [RWC+19] attempts to do this via what we call “in-context learning”, using the text input of a pretrained language model as a form of task specification: the model is conditioned on a natural language instruction and/or a few demonstrations of the task and is then expected to complete further instances of the task simply by predicting what comes next.
中文: 最近的工作[RWC+19]试图通过我们所谓的“文中学习”来做到这一点,将预先训练的语言模型的文字输入作为任务规格的一种形式:该模型以自然语言的教学和(或)任务的一些演示为条件,然后通过预测接下来的情况来完成任务的更多实例。
<a id="S0040"></a> Source: p.4 S0040
Original: While it has shown some initial promise, this approach still achieves results far inferior to fine-tuning – for example [RWC+19] achieves only 4% on Natural Questions, and even its 55 F1 CoQa result is now more than 35 points behind the state of the art.
中文: 虽然它已经显示出一些初步的希望,但这个方法仍然取得了远远低于微调的结果 — — 例如[RWC+19]在自然问题上只取得了4%的成绩,甚至它的55个F1 CoQa结果现在已经落后了35分以上.
<a id="S0041"></a> Source: p.4 S0041
Original: Meta-learning clearly requires substantial improvement in order to be viable as a practical method of solving language tasks.
中文: Meta-learning显然需要大幅度改进,才能成为解决语言任务的可行方法.
<a id="S0042"></a> Source: p.4 S0042
Original: Another recent trend in language modeling may offer a way forward.
中文: 语言建模的又一最新趋势可能提供前进的道路。
<a id="S0043"></a> Source: p.4 S0043
Original: In recent years the capacity of transformer language models has increased substantially, from 100 million parameters [RNSS18], to 300 million parameters [DCLT18], to 1.5 billion parameters [RWC+19], to 8 billion parameters [SPP+19], 11 billion parameters [RSR+19], and finally 17 billion parameters [Tur20].
中文: 近年来,变压器语言模型的容量大幅提升,从一亿个参数[RNSS18]到三亿个参数[DCLT18],再到十五亿个参数[RWC+19],再到八十亿个参数[SPP+19],一百一十亿个参数[RSR+19],再到最后170亿个参数[Tur20].
<a id="S0044"></a> Source: p.4 S0044
Original: Each increase has brought improvements in text synthesis and/or downstream NLP tasks, and there is evidence suggesting that log loss, which correlates well with many downstream tasks, follows a smooth trend of improvement with scale [KMH+20].
中文: 每次增加都改进了文本合成和/或下游NLP任务,有证据表明,与下游许多任务密切相关的日志损失随着规模[KMH+20]的平稳改进趋势而出现。
<a id="S0045"></a> Source: p.4 S0045
Original: Since in-context learning involves absorbing many skills and tasks within the parameters of the model, it is plausible that in-context learning abilities might show similarly strong gains with scale. 1In the context of language models this has sometimes been called “zero-shot transfer”, but this term is potentially ambiguous: the method is “zero-shot” in the sense that no gradient updates are performed, but it often involves providing inference-time demonstrations to the model, so is not truly learning from zero examples.
中文: 由于内文学习涉及在模型的参数范围内吸收许多技能和任务,因此,内文学习能力在规模上可能显示出类似的巨大收益。 1 在语言模型方面,这有时被称为“零射转接”,但这一术语可能模糊不清:这种方法是“零射出”的,因为没有进行梯度更新,但它往往涉及向模型提供推断时间的演示,因此并非真正从零实例中学习。
<a id="S0046"></a> Source: p.4 S0046
Original: To avoid this confusion, we use the term “meta-learning” to capture the inner-loop / outer-loop structure of the general method, and the term “in context-learning” to refer to the inner loop of meta-learning.
中文: 为了避免这种混淆,我们使用“元学习”一词来捕捉一般方法的内-相/外-相结构,而用“上下文学习”一词来指元学习的内部循环。
<a id="S0047"></a> Source: p.4 S0047
Original: We further specialize the description to “zero-shot”, “one-shot”, or “few-shot” depending on how many demonstrations are provided at inference time.
中文: 我们还专门描述“零射”、“一发”或“一发”的情况,这取决于推断时提供多少示威。
<a id="S0048"></a> Source: p.4 S0048
Original: These terms are intended to remain agnostic on the question of whether the model learns new tasks from scratch at inference time or simply recognizes patterns seen during training – this is an important issue which we discuss later in the paper, but “meta-learning” is intended to encompass both possibilities, and simply describes the inner-outer loop structure. 4
中文: 这些术语的用意是对于模型是从推断时间开始从零开始学习新任务,还是仅仅承认培训期间看到的规律的问题保持不可知论的态度 — — 这是我们在论文后期讨论的一个重要问题,但“元学习”意在涵盖两种可能性,并简单描述内向循环结构。 页:1
<a id="S0049"></a> Source: p.5 S0049
Original: Figure 1.3: Aggregate performance for all 42 accuracy-denominated benchmarks While zero-shot performance improves steadily with model size, few-shot performance increases more rapidly, demonstrating that larger models are more proficient at in-context learning.
中文: 图1.3:所有42个准确标出的基准的总业绩 虽然"零射"的性能随着模型大小而稳步改善,但"少射"的性能增长得更快,表明更大型的模型更能熟练地进行内在学习.
<a id="S0050"></a> Source: p.5 S0050
Original: See Figure 3.8 for a more detailed analysis on SuperGLUE, a standard NLP benchmark suite.
中文: 关于SuperGLUE,一个标准的NLP基准套件,详见图3.8分析.
<a id="S0051"></a> Source: p.5 S0051
Original: In this paper, we test this hypothesis by training a 175 billion parameter autoregressive language model, which we call GPT-3, and measuring its in-context learning abilities.
中文: 在本文中,我们通过训练出1,750亿个参数的自旋性语言模型来测试这个假说,我们称之为GPT-3,并测量其内文学习能力.
<a id="S0052"></a> Source: p.5 S0052
Original: Specifically, we evaluate GPT-3 on over two dozen NLP datasets, as well as several novel tasks designed to test rapid adaptation to tasks unlikely to be directly contained in the training set.
中文: 具体地说,我们评价了20多套NLP数据集的GPT-3,以及一些旨在测试快速适应培训组不可能直接包含的任务的新任务.
<a id="S0053"></a> Source: p.5 S0053
Original: For each task, we evaluate GPT-3 under 3 conditions: (a) “few-shot learning”, or in-context learning where we allow as many demonstrations as will fit into the model’s context window (typically 10 to 100), (b) “one-shot learning”, where we allow only one demonstration, and (c) “zero-shot” learning, where no demonstrations are allowed and only an instruction in natural language is given to the model.
中文: 对于每项任务,我们评价GPT-3在3个条件下:(a) " 短镜头学习 " ,或者内文学习,我们允许尽可能多的与模型背景窗口相适应的演示(通常为10至100个),(b) " 一镜头学习 " ,我们只允许一次演示,(c) " 零镜头 " 学习,不允许演示,只给模型以自然语言授课。
<a id="S0054"></a> Source: p.5 S0054
Original: GPT-3 could also in principle be evaluated in the traditional fine-tuning setting, but we leave this to future work.
中文: 原则上也可以在传统的微调环境中对GPT-3进行评价,但我们将此留给今后的工作去处理。
<a id="S0055"></a> Source: p.5 S0055
Original: Figure 1.2 illustrates the conditions we study, and shows few-shot learning of a simple task requiring the model to remove extraneous symbols from a word.
中文: 图1.2说明我们研究的条件,并显示对一个简单的任务的少许学习,要求模型去掉一个单词的外相符号.
<a id="S0056"></a> Source: p.5 S0056
Original: Model performance improves with the addition of a natural language task description, and with the number of examples in the model’s context, K.
中文: 模型的性能随着自然语言任务描述的加入,以及模型上下文中的例子数量K而得到改进.
<a id="S0057"></a> Source: p.5 S0057
Original: Few-shot learning also improves dramatically with model size.
中文: 少发快活的学习也随着模型大小而大有改进.
<a id="S0058"></a> Source: p.5 S0058
Original: Though the results in this case are particularly striking, the general trends with both model size and number of examples in-context hold for most tasks we study.
中文: 尽管这一案例的结果特别引人注目,但模型规模和实例数量的总体趋势对我们研究的大多数任务都很重要。
<a id="S0059"></a> Source: p.5 S0059
Original: We emphasize that these “learning” curves involve no gradient updates or fine-tuning, just increasing numbers of demonstrations given as conditioning.
中文: 我们强调,这些“学习”曲线不涉及梯度更新或微调,只是作为条件的演示次数不断增加。
<a id="S0060"></a> Source: p.5 S0060
Original: Broadly, on NLP tasks GPT-3 achieves promising results in the zero-shot and one-shot settings, and in the the few-shot setting is sometimes competitive with or even occasionally surpasses state-of-the-art (despite state-of-the-art being held by fine-tuned models).
中文: 广义上,在NLP任务上,GPT-3在零发和一发的设置上取得了有希望的结果,在少数发的设置上,有时与最先进的发相竞争,甚至偶尔会超越最先进的发相(尽管有微调模型的发相).
<a id="S0061"></a> Source: p.5 S0061
Original: For example, GPT-3 achieves 81.5 F1 on CoQA in the zero-shot setting, 84.0 F1 on CoQA in the one-shot setting, 85.0 F1 in the few-shot setting.
中文: 例如:GPT-3在"零射入"设定中实现CoQA上的81.5 F1,在"一发入"设定中实现84.0 F1,在"一发入"设定中实现85.0 F1.
<a id="S0062"></a> Source: p.5 S0062
Original: Similarly, GPT-3 achieves 64.3% accuracy on TriviaQA in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting, the last of which is state-of-the-art relative to fine-tuned models operating in the same closed-book setting.
中文: 同样地,GPT-3在"零发"设定中在TriviaQA上实现了64.3%的精度,在"一发"设定中实现了68.0%的精度,在"一发"设定中实现了71.2%的精度,而"一发"设定中则是与在同一封闭式书本设定中运行的微调模型相比的最先进的.
<a id="S0063"></a> Source: p.5 S0063
Original: GPT-3 also displays one-shot and few-shot proficiency at tasks designed to test rapid adaption or on-the-fly reasoning, which include unscrambling words, performing arithmetic, and using novel words in a sentence after seeing them defined only once.
中文: GPT-3还展现出一发和几发精通的任务来测试快速适应或飞上推理,这些任务包括:取出不速之词,进行算术,在只看到它们定义一次后在句子中用出新词.
<a id="S0064"></a> Source: p.5 S0064
Original: We also show that in the few-shot setting, GPT-3 can generate synthetic news articles which human evaluators have difficulty distinguishing from human-generated articles.
中文: 我们还显示,在几发环境下,GPT-3可以生成合成新闻文章,而人类评价人员很难区分出由人类产生的文章.
<a id="S0065"></a> Source: p.5 S0065
Original: At the same time, we also find some tasks on which few-shot performance struggles, even at the scale of GPT-3.
中文: 同时,我们也发现一些任务,即使以GPT-3的规模,也很少拍出表演斗争.
<a id="S0066"></a> Source: p.5 S0066
Original: This includes natural language inference tasks like the ANLI dataset, and some reading comprehension datasets like RACE or QuAC.
中文: 这包括自然语言推论任务如ANLI数据集,以及一些读取理解数据集如RACE或QuAC.
<a id="S0067"></a> Source: p.5 S0067
Original: By presenting a broad characterization of GPT-3’s strengths and weaknesses, including these limitations, we hope to stimulate study of few-shot learning in language models and draw attention to where progress is most needed. A heuristic sense of the overall results can be seen in Figure 1.3, which aggregates the various tasks (though it should not be seen as a rigorous or meaningful benchmark in itself). 5
中文: 通过对GPT-3的长处和短处(包括这些局限性)作广义的描述,我们希望能刺激对语言模型的几发学习的研究并提请注意最需要进展的地方. 从图1.3可以看出对总体结果的欣快感,图1.3汇总了各种任务(虽然其本身不应被视为一个严格或有意义的基准)。 页:1
<a id="S0068"></a> Source: p.6 S0068
Original: We also undertake a systematic study of “data contamination” – a growing problem when training high capacity models on datasets such as Common Crawl, which can potentially include content from test datasets simply because such content often exists on the web.
中文: 我们还对“数据污染”进行系统研究,当在诸如“共同爬行”等数据集上培训高容量模型时,这个问题就越来越严重了,这种模型可能仅仅因为网络上经常存在测试数据集的内容而包含其中的内容。
<a id="S0069"></a> Source: p.6 S0069
Original: In this paper we develop systematic tools to measure data contamination and quantify its distorting effects.
中文: 在本文中,我们开发了系统的工具,以衡量数据污染并量化其扭曲效应。
<a id="S0070"></a> Source: p.6 S0070
Original: Although we find that data contamination has a minimal effect on GPT-3’s performance on most datasets, we do identify a few datasets where it could be inflating results, and we either do not report results on these datasets or we note them with an asterisk, depending on the severity.
中文: 虽然我们发现数据被污染对GPT-3在大多数数据集上的性能影响很小,但我们确实发现了几套数据,它们可能会夸大结果,我们要么不报告这些数据集的结果,要么根据严重程度用星号标注这些结果。
<a id="S0071"></a> Source: p.6 S0071
Original: In addition to all the above, we also train a series of smaller models (ranging from 125 million parameters to 13 billion parameters) in order to compare their performance to GPT-3 in the zero, one and few-shot settings.
中文: 除上述所有外,我们还培养出一系列更小的模型(从1.25亿个参数到130亿个参数不等),以便在零,一和几发设置中将它们的性能与GPT-3进行比较.
<a id="S0072"></a> Source: p.6 S0072
Original: Broadly, for most tasks we find relatively smooth scaling with model capacity in all three settings; one notable pattern is that the gap between zero-, one-, and few-shot performance often grows with model capacity, perhaps suggesting that larger models are more proficient meta-learners.
中文: 广义地说,对于大多数任务来说,我们发现在所有三个场合中都具有模型容量的相对平滑的缩放;一个显著的规律是:零相机,一相机和几相机相机性能之间的差距往往随着模型容量的增加而扩大,也许说明更大的模型更能熟练地使用元相机.
<a id="S0073"></a> Source: p.6 S0073
Original: Finally, given the broad spectrum of capabilities displayed by GPT-3, we discuss concerns about bias, fairness, and broader societal impacts, and attempt a preliminary analysis of GPT-3’s characteristics in this regard.
中文: 最后,鉴于GPT-3所展现的能力范围很广,我们讨论了对偏见、公平和更广泛的社会影响的关切,并试图对GPT-3在这方面的特点进行初步分析。
<a id="S0074"></a> Source: p.6 S0074
Original: The remainder of this paper is organized as follows.
中文: 本文件的其余部分编排如下。
<a id="S0075"></a> Source: p.6 S0075
Original: In Section 2, we describe our approach and methods for training GPT-3 and evaluating it.
中文: 在第2节中,我们描述了我们培训GPT-3和评价它的方法。
<a id="S0076"></a> Source: p.6 S0076
Original: Section 3 presents results on the full range of tasks in the zero-, one- and few-shot settings.
中文: 第3节介绍了在零、一发和少发环境中各项任务的成果。
<a id="S0077"></a> Source: p.6 S0077
Original: Section 4 addresses questions of data contamination (train-test overlap).
中文: 第4节涉及数据污染问题(培训测试重叠)。
<a id="S0078"></a> Source: p.6 S0078
Original: Section 5 discusses limitations of GPT-3.
中文: 第5节讨论了GPT-3的局限性。
<a id="S0079"></a> Source: p.6 S0079
Original: Section 7 reviews related work and Section 8 concludes. 2 Approach Our basic pre-training approach, including model, data, and training, is similar to the process described in [RWC+19], with relatively straightforward scaling up of the model size, dataset size and diversity, and length of training.
中文: 第7节审查有关工作,第8节结束。 2 采用我们的基本培训前方法,包括模式、数据和培训,与[RWC+19]所述程序相类似,模型规模、数据集规模和多样性以及培训时间长短的扩大相对直接。
<a id="S0080"></a> Source: p.6 S0080
Original: Our use of in-context learning is also similar to [RWC+19], but in this work we systematically explore different settings for learning within the context.
中文: 我们对内文学习的用法也类似于[RWC+19],但在这项工作中,我们系统探索了背景中不同的学习环境.
<a id="S0081"></a> Source: p.6 S0081
Original: Therefore, we start this section by explicitly defining and contrasting the different settings that we will be evaluating GPT-3 on or could in principle evaluate GPT-3 on.
中文: 因此,我们从这一节开始,明确界定和对比我们将评价GPT-3的不同环境,或原则上可以评价GPT-3。
<a id="S0082"></a> Source: p.6 S0082
Original: These settings can be seen as lying on a spectrum of how much task-specific data they tend to rely on.
中文: 这些设置可被视为依赖于他们倾向于依赖多少具体任务的数据。
<a id="S0083"></a> Source: p.6 S0083
Original: Specifically, we can identify at least four points on this spectrum (see Figure 2.1 for an illustration): • Fine-Tuning (FT) has been the most common approach in recent years, and involves updating the weights of a pre-trained model by training on a supervised dataset specific to the desired task.
中文: 具体地说,我们可以在这个频谱上确定至少四个点(见图2.1): 微调(FT)是近年来最常见的方法,涉及通过对特定任务的监督数据集进行培训来更新预先训练过的模型的权重。
<a id="S0084"></a> Source: p.6 S0084
Original: Typically thousands to hundreds of thousands of labeled examples are used.
中文: 通常使用数千至数十万标注的例子。
<a id="S0085"></a> Source: p.6 S0085
Original: The main advantage of fine-tuning is strong performance on many benchmarks.
中文: 微调的主要优点是在许多基准上表现良好.
<a id="S0086"></a> Source: p.6 S0086
Original: The main disadvantages are the need for a new large dataset for every task, the potential for poor generalization out-of-distribution [MPL19], and the potential to exploit spurious features of the training data [GSL+18, NK19], potentially resulting in an unfair comparison with human performance.
中文: 主要缺点是每项任务都需要一个新的大数据组,分散分配的通俗化可能较差[MPL19],以及利用培训数据[GSL+18,NK19]的虚假特征的可能性,可能导致与人的表现作不公平的比较.
<a id="S0087"></a> Source: p.6 S0087
Original: In this work we do not fine-tune GPT-3 because our focus is on task-agnostic performance, but GPT-3 can be fine-tuned in principle and this is a promising direction for future work. • Few-Shot (FS) is the term we will use in this work to refer to the setting where the model is given a few demonstrations of the task at inference time as conditioning [RWC+19], but no weight updates are allowed.
中文: 在这项工作中,我们没有微调GPT-3,因为我们的重点是任务不可知性表现,但GPT-3原则上可以微调,这是未来工作的一个有希望的方向. ^ 很少Shot(FS)是我们将在这项工作中使用的术语来指代在推论时间给模型一些任务演示作为条件[RWC+19]的设置,但不允许进行重量更新.
<a id="S0088"></a> Source: p.6 S0088
Original: As shown in Figure 2.1, for a typical dataset an example has a context and a desired completion (for example an English sentence and the French translation), and few-shot works by giving K examples of context and completion, and then one final example of context, with the model expected to provide the completion.
中文: 如图2.1所示,对于一个典型的数据集,一个实例具有上下文和期望的完成(例如英文句子和法文译本),通过给出上下文和完成的K实例,然后给出最后的上下文实例,预计模型将提供完成。
<a id="S0089"></a> Source: p.6 S0089
Original: We typically set K in the range of 10 to 100 as this is how many examples can fit in the model’s context window (n = 2048).
中文: 我们通常将K设定在10到100之间,因为这是模型上下文窗口(n = 2048)中有多少实例可以匹配的.
<a id="S0090"></a> Source: p.6 S0090
Original: The main advantages of few-shot are a major reduction in the need for task-specific data and ctx reduced potential to learn an overly narrow distribution from a large but narrow fine-tuning dataset.
中文: 少拍的主要优点是大大减少了对特定任务数据的需求,ctx减少了从一个大而狭小的微调数据集中学习过窄分布的可能性.
<a id="S0091"></a> Source: p.6 S0091
Original: The main disadvantage is that results from this method have so far been much worse than state-of-the-art fine-tuned models.
中文: 主要的缺点是,这一方法的结果迄今远比最先进的微调模型差.
<a id="S0092"></a> Source: p.6 S0092
Original: Also, a small amount of task specific data is still required.
中文: 此外,还需要少量的具体任务数据。
<a id="S0093"></a> Source: p.6 S0093
Original: As indicated by the name, few-shot learning as described here for language models is related to few-shot learning as used in other contexts in ML [HYC01, VBL+16] – both involve learning based on a broad distribution of tasks (in this case implicit in the pre-training data) and then rapidly adapting to a new task. • One-Shot (1S) is the same as few-shot except that only one demonstration is allowed, in addition to a natural language description of the task, as shown in Figure 1.
中文: 如名称所示,此处所描述的语言模型的少发快活学习与ML [HYC01,VBL+16]中其他场合使用的少发快活学习有关——两者都涉及基于任务广泛分配的学习(在此情况下隐含在训练前数据中),然后迅速适应新的任务. ^ One-Shot (1S) 与数发相同,但只允许一次演示,此外对任务的自然语言描述,如图一所示.
<a id="S0094"></a> Source: p.6 S0094
Original: The reason to distinguish one-shot from few-shot and zero-shot (below) is that it most closely matches the way in which some tasks are communicated to humans.
中文: 区分一发与少发与零发(下同)的原因是它最接近于将一些任务传达给人类的方式.
<a id="S0095"></a> Source: p.6 S0095
Original: For example, when asking humans to generate a dataset on a human worker service (for example Mechanical Turk), it is common to give one demonstration of the task.
中文: 例如,当要求人类生成一个有关人类工人服务的数据集(例如:机械突厥)时,通常会给出一个任务演示.
<a id="S0096"></a> Source: p.6 S0096
Original: By contrast it is sometimes difficult to communicate the content or format of a task if no examples are given. 6
中文: 相反,如果不举出实例,有时很难传达任务的内容或格式。 6个
<a id="S0097"></a> Source: p.7 S0097
Original: Figure 2.1: Zero-shot, one-shot and few-shot, contrasted with traditional fine-tuning.
中文: 图2.1:同传统微调相较,零发一发和少发.
<a id="S0098"></a> Source: p.7 S0098
Original: The panels above show four methods for performing a task with a language model – fine-tuning is the traditional method, whereas zero-, one-, and few-shot, which we study in this work, require the model to perform the task with only forward passes at test time.
中文: 上面的板块显示了四种用语言模型执行任务的方法 — — 微调是传统方法,而我们在这项工作中研究的零发、一发和少发镜头则需要模型在测试时只通过前行。
<a id="S0099"></a> Source: p.7 S0099
Original: We typically present the model with a few dozen examples in the few shot setting.
中文: 我们通常在几个镜头中 给模型举出几十个例子
<a id="S0100"></a> Source: p.7 S0100
Original: Exact phrasings for all task descriptions, examples and prompts can be found in Appendix G. • Zero-Shot (0S) is the same as one-shot except that no demonstrations are allowed, and the model is only given a natural language instruction describing the task.
中文: 所有任务描述,示例和提示的精确用词可参见附录G. ^ Zero-Shot (0S)与一发相同,只是不允许演示,模型只得到描述任务的自然语言指令.
<a id="S0101"></a> Source: p.7 S0101
Original: This method provides maximum convenience, potential for robustness, and avoidance of spurious correlations (unless they occur very broadly across the large corpus of pre-training data), but is also the most challenging setting.
中文: 这种方法提供了最大限度的便利,具有稳健性的潜力并避免了虚假的相互关系(除非它们广泛出现在培训前的大量数据中),但也是最具有挑战性的环境。
<a id="S0102"></a> Source: p.7 S0102
Original: In some cases it may even be difficult for humans to understand the format of the task without prior examples, so this setting is in some cases “unfairly hard”.
中文: 在某些情况下,如果没有先前的例子,人类甚至可能难以理解任务的形式,因此这种设置在某些情况下是“不公平地困难的”。
<a id="S0103"></a> Source: p.7 S0103
Original: For example, if someone is asked to “make a table of world records for the 200m dash”, this request can be ambiguous, as it may not be clear exactly what format the table should have or what should be included (and even with careful clarification, understanding precisely what is desired can be difficult).
中文: 例如,如果有人被要求“为200米短线编制世界记录表”,这一请求可能含糊不清,因为可能不清楚该表应采用何种格式,或应包括什么内容(即使经过认真澄清,也很难确切理解所希望的内容)。
<a id="S0104"></a> Source: p.7 S0104
Original: Nevertheless, for at least some settings zero-shot is closest to how humans perform tasks – for example, in the translation example in Figure 2.1, a human would likely know what to do from just the text instruction.
中文: 尽管如此,因为至少一些设置零镜头最接近人类执行任务的方式——例如图2.1中的翻译例子中,人类可能知道仅仅从文本指令中可以做些什么.
<a id="S0105"></a> Source: p.7 S0105
Original: Figure 2.1 shows the four methods using the example of translating English to French.
中文: 图2.1采用英文为法文的四种方法。
<a id="S0106"></a> Source: p.7 S0106
Original: In this paper we focus on zero-shot, one-shot and few-shot, with the aim of comparing them not as competing alternatives, but as different problem settings which offer a varying trade-off between performance on specific benchmarks and sample efficiency.
中文: 在本文中,我们侧重于零发、一发和几发,目的不是把它们作为相互竞争的替代品,而是作为不同的问题环境加以比较,在具体基准的业绩和样本效率之间提供不同的取舍。
<a id="S0107"></a> Source: p.7 S0107
Original: We especially highlight the few-shot results as many of them are only slightly behind state-of-the-art fine-tuned models.
中文: 我们尤其要强调少数成果,因为其中许多仅略低于最先进的微调模式。
<a id="S0108"></a> Source: p.7 S0108
Original: Ultimately, however, one-shot, or even sometimes zero-shot, seem like the fairest comparisons to human performance, and are important targets for future work.
中文: 然而,归根结底,一发甚至有时是零发,似乎是与人类业绩最公平的比较,是未来工作的重要目标。
<a id="S0109"></a> Source: p.7 S0109
Original: Sections 2.1-2.3 below give details on our models, training data, and training process respectively.
中文: 下文2.1至2.3节分别详细介绍了我们的模式、培训数据和培训过程。
<a id="S0110"></a> Source: p.7 S0110
Original: Section 2.4 discusses the details of how we do few-shot, one-shot, and zero-shot evaluations. 7
中文: 第2.4节讨论了我们如何进行几发、一发和零发评价的细节。 第7条
<a id="S0111"></a> Source: p.8 S0111
Original: Model Name n n d n d Batch Size Learning Rate params layers model heads head GPT-3 Small 125M 12 768 12 64 0.5M 6.0 × 10−4 GPT-3 Medium 350M 24 1024 16 64 0.5M 3.0 × 10−4 GPT-3 Large 760M 24 1536 16 96 0.5M 2.5 × 10−4 GPT-3 XL 1.3B 24 2048 24 128 1M 2.0 × 10−4 GPT-3 2.7B 2.7B 32 2560 32 80 1M 1.6 × 10−4 GPT-3 6.7B 6.7B 32 4096 32 128 2M 1.2 × 10−4 GPT-3 13B 13.0B 40 5140 40 128 2M 1.0 × 10−4 GPT-3 175B or “GPT-3” 175.0B 96 12288 96 128 3.2M 0.6 × 10−4 Table 2.1: Sizes, architectures, and learning hyper-parameters (batch size in tokens and learning rate) of the models which we trained.
中文: 型号 名称 n d d 批量学习速率平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平
<a id="S0112"></a> Source: p.8 S0112
Original: All models were trained for a total of 300 billion tokens. 2.1 Model and Architectures We use the same model and architecture as GPT-2 [RWC+19], including the modified initialization, pre-normalization, and reversible tokenization described therein, with the exception that we use alternating dense and locally banded sparse attention patterns in the layers of the transformer, similar to the Sparse Transformer [CGRS19].
中文: 所有型号都接受了共计3000亿令牌的培训. 2.1 模型和结构 我们使用与GPT-2[RWC+19]相同的模型和架构,包括其中所描述的被修改的初始化,前正态化,可逆可逆的符号化,除了我们在变压器的地层中使用交替密集和局部带宽的稀有的注意力图案外,与Sparse变压器相类似[CGRS19].
<a id="S0113"></a> Source: p.8 S0113
Original: To study the dependence of ML performance on model size, we train 8 different sizes of model, ranging over three orders of magnitude from 125 million parameters to 175 billion parameters, with the last being the model we call GPT-3.
中文: 为了研究ML性能对模型大小的依赖性,我们训练了8个不同大小的模型,从1.25亿个参数到1,750亿个参数的3个数量级以上,最后一个是我们称为GPT-3的模型.
<a id="S0114"></a> Source: p.8 S0114
Original: Previous work [KMH+20] suggests that with enough training data, scaling of validation loss should be approximately a smooth power law as a function of size; training models of many different sizes allows us to test this hypothesis both for validation loss and for downstream language tasks.
中文: 以前的工作[KMH+20]建议,只要有足够的训练数据,验证损失的缩放应该是一个大致平滑的功率定律,作为大小的函数;许多不同大小的训练模型使我们能够测试这个假设,既用于验证损失,也用于下游语言任务.
<a id="S0115"></a> Source: p.8 S0115
Original: Table 2.1 shows the sizes and architectures of our 8 models.
中文: 表2.1显示了我们8个模型的大小和结构。
<a id="S0116"></a> Source: p.8 S0116
Original: Here n is the total number of trainable parameters, params n is the total number of layers, d is the number of units in each bottleneck layer (we always have the layers model feedforward layer four times the size of the bottleneck layer, d = 4 ∗ d ), and d is the dimension of each ff model head attention head.
中文: 这里n是可列车参数的总数,params n是地层的总数,d是每个瓶颈地层的单位数(我们总是有地层模型向上分层4倍于瓶颈地层的大小,d=4 * d),d是每个ff模型头注头的维度.
<a id="S0117"></a> Source: p.8 S0117
Original: All models use a context window of n = 2048 tokens.
中文: 所有型号都使用上下文窗口n=2048个令牌.
<a id="S0118"></a> Source: p.8 S0118
Original: We partition the model across GPUs along ctx both the depth and width dimension in order to minimize data-transfer between nodes.
中文: 我们沿着 Ctx 沿着 GPU 将模型分为深度和宽度, 以尽量减少节点之间的数据传输。
<a id="S0119"></a> Source: p.8 S0119
Original: The precise architectural parameters for each model are chosen based on computational efficiency and load-balancing in the layout of models across GPU’s.
中文: 每个模型的精确建筑参数是根据整个GPU模型布局的计算效率和负载平衡而选择的.
<a id="S0120"></a> Source: p.8 S0120
Original: Previous work [KMH+20] suggests that validation loss is not strongly sensitive to these parameters within a reasonably broad range. 2.2 Training Dataset Datasets for language models have rapidly expanded, culminating in the Common Crawl dataset2 [RSR+19] constituting nearly a trillion words.
中文: 先前的工作[KMH+20]表明,审定损失在相当广泛的范围内对这些参数没有强烈的敏感性。 2.2 用于语言模型的培训数据集迅速扩大,最终形成共同爬行数据集2[RSR+19],构成近一兆字.
<a id="S0121"></a> Source: p.8 S0121
Original: This size of dataset is sufficient to train our largest models without ever updating on the same sequence twice.
中文: 这种数据集的大小足以训练我们最大的模型,而不曾两次更新相同的序列.
<a id="S0122"></a> Source: p.8 S0122
Original: However, we have found that unfiltered or lightly filtered versions of Common Crawl tend to have lower quality than more curated datasets.
中文: 然而,我们发现"通俗爬行"(Common Crawl)的未经过滤或被轻度过滤的版本,其质量往往低于多被整理的数据集.
<a id="S0123"></a> Source: p.8 S0123
Original: Therefore, we took 3 steps to improve the average quality of our datasets: (1) we downloaded and filtered a version of CommonCrawl based on similarity to a range of high-quality reference corpora, (2) we performed fuzzy deduplication at the document level, within and across datasets, to prevent redundancy and preserve the integrity of our held-out validation set as an accurate measure of overfitting, and (3) we also added known high-quality reference corpora to the training mix to augment CommonCrawl and increase its diversity.
中文: 因此,我们采取了三个步骤来提高我们数据集的平均质量:(1)我们下载并过滤了基于一系列高质量参考物公司相似程度的CommonCrawl版本;(2)我们在文件层面,在数据集内部和之间进行了模糊的分解,以防止冗余并维护了我们所设定的被搁置验证的完整性,作为衡量过度调整的准确尺度;(3)我们还在培训组合中增加了已知的高质量参考物公司,以扩充CommonCrawl并增加其多样性.
<a id="S0124"></a> Source: p.8 S0124
Original: Details of the first two points (processing of Common Crawl) are described in Appendix A.
中文: 前两点的细节(Common Crawl的处理)见附录A.
<a id="S0125"></a> Source: p.8 S0125
Original: For the third, we added several curated high-quality datasets, including an expanded version of the WebText dataset [RWC+19], collected by scraping links over a longer period of time, and first described in [KMH+20], two internet-based books corpora (Books1 and Books2) and English-language Wikipedia.
中文: 第三,我们添加了几套被整理的高质量数据集,包括一个扩展版的WebText数据集[RWC+19],通过更长时间的去除链接来收集,并首次在[KMH+20]中被描述,两本基于互联网的书籍corpora (Books1 and Books2)和英语维基百科中被描述.
<a id="S0126"></a> Source: p.8 S0126
Original: Table 2.2 shows the final mixture of datasets that we used in training.
中文: 表2.2显示我们在培训中使用的最后混合数据集。
<a id="S0127"></a> Source: p.8 S0127
Original: The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
中文: CommonCrawl数据由覆盖2016至2019年的"CommonCrawl"月刊的41个硬盘下载而来,构成过滤前压缩平话的45TB和过滤后570GB,大致相当于4000亿字节-pair-encoords.
<a id="S0128"></a> Source: p.8 S0128
Original: Note that during training, datasets are not sampled in proportion to their size, but rather datasets we view as higher-quality are sampled more frequently, such that CommonCrawl and Books2 datasets are sampled less than once during training, but the other datasets are sampled 2-3 times.
中文: 请注意,在培训期间,数据集的采样量与其大小不相称,而我们认为更高质量的数据集的采样量要多一些,因此在培训期间,CommonCrawl和Books2数据集的采样量少于一次,而其他数据集的采样量为2-3次.
<a id="S0129"></a> Source: p.8 S0129
Original: This essentially accepts a small amount of overfitting in exchange for higher quality training data. 2https://commoncrawl.org/the-data/ 8
中文: 这基本上接受少量过度调整,以换取高质量的培训数据。 2https://commoncrawl.org/the-data/ 8 (中文(简体) ).
<a id="S0130"></a> Source: p.9 S0130
Original: Figure 2.2: Total compute used during training.
中文: 图2.2:培训期间使用的计算总数。
<a id="S0131"></a> Source: p.9 S0131
Original: Based on the analysis in Scaling Laws For Neural Language Models [KMH+20] we train much larger models on many fewer tokens than is typical.
中文: 根据《神经语言模型的放大定律》[KMH+20]的分析,我们用比典型的更少的代号来训练更大的模型。
<a id="S0132"></a> Source: p.9 S0132
Original: As a consequence, although GPT-3 3B is almost 10x larger than RoBERTa-Large (355M params), both models took roughly 50 petaflop/s-days of compute during pre-training.
中文: 因此,虽然GPT-3 3B比RoBERTA-Large(355M params)几乎大出10x,但两种型号在前期训练中大约需要50个petaflop/s-day的计算.
<a id="S0133"></a> Source: p.9 S0133
Original: Methodology for these calculations can be found in Appendix D.
中文: 这些计算方法见附录D。
<a id="S0134"></a> Source: p.9 S0134
Original: Quantity Weight in Epochs elapsed when Dataset (tokens) training mix training for 300B tokens Common Crawl (filtered) 410 billion 60% 0.44 WebText2 19 billion 22% 2.9 Books1 12 billion 8% 1.9 Books2 55 billion 8% 0.43 Wikipedia 3 billion 3% 3.4 Table 2.2: Datasets used to train GPT-3. “Weight in training mix” refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset.
中文: 当Dataset(tokens)培训组合培训300B道具(被过滤)时,Epochs的重量超过4100亿WebText2培训(被过滤) 60% 0.44亿WebText2 190亿 22% 2.9亿 Books1 120亿 Books2 550亿 Books 8% 1.9 Books2 530亿 Wikipedia 30亿 PT-3.4 表2.2:用于培训GPT-3的数据集。“培训组合中的重量”是指从特定数据集中提取的一小部分实例,我们故意不按数据集大小进行。
<a id="S0135"></a> Source: p.9 S0135
Original: As a result, when we train for 300 billion tokens, some datasets are seen up to 3.4 times during training while other datasets are seen less than once. A major methodological concern with language models pretrained on a broad swath of internet data, particularly large models with the capacity to memorize vast amounts of content, is potential contamination of downstream tasks by having their test or development sets inadvertently seen during pre-training.
中文: 因此,当我们为3000亿令牌进行训练时,一些数据集在训练期间被看到多达3.4次,而其他数据集被看到不到一次. 对语言模型,特别是能够记住大量内容的大型模型,预先在广泛的互联网数据上加以培训的一个主要方法关注是,在培训前无意中看到其测试或开发装置,可能污染下游工作。
<a id="S0136"></a> Source: p.9 S0136
Original: To reduce such contamination, we searched for and attempted to remove any overlaps with the development and test sets of all benchmarks studied in this paper.
中文: 为了减少这种污染,我们寻找并试图消除与本文件所研究的所有基准的制定和测试的任何重叠。
<a id="S0137"></a> Source: p.9 S0137
Original: Unfortunately, a bug in the filtering caused us to ignore some overlaps, and due to the cost of training it was not feasible to retrain the model.
中文: 不幸的是,过滤中的一个错误导致我们忽略了一些重叠之处,由于培训费用高昂,无法重新训练该模型.
<a id="S0138"></a> Source: p.9 S0138
Original: In Section 4 we characterize the impact of the remaining overlaps, and in future work we will more aggressively remove data contamination. 2.3 Training Process As found in [KMH+20, MKAT18], larger models can typically use a larger batch size, but require a smaller learning rate.
中文: 在第4节中,我们描述剩余重叠的影响,在今后的工作中,我们将更加积极地消除数据污染。 2.3 培训进程 如[KMH+20,MKAT18]所发现,较大的模型一般可以使用更大的批量尺寸,但需要较低的学习率.
<a id="S0139"></a> Source: p.9 S0139
Original: We measure the gradient noise scale during training and use it to guide our choice of batch size [MKAT18].
中文: 我们在训练期间测量梯度噪声尺度,并用它来指导我们选择批量大小[MKAT18].
<a id="S0140"></a> Source: p.9 S0140
Original: Table 2.1 shows the parameter settings we used.
中文: 表2.1显示了我们使用的参数设置。
<a id="S0141"></a> Source: p.9 S0141
Original: To train the larger models without running out of memory, we use a mixture of model parallelism within each matrix multiply and model parallelism across the layers of the network.
中文: 为了在不耗尽内存的情况下训练出更大的模型,我们使用模型平行主义在每一个矩阵相乘和模型平行主义的组合,跨越网络的多层.
<a id="S0142"></a> Source: p.9 S0142
Original: All models were trained on V100 GPU’s on part of a high-bandwidth cluster provided by Microsoft.
中文: 所有型号都接受了由微软提供的高带宽集群部分V100 GPU的培训.
<a id="S0143"></a> Source: p.9 S0143
Original: Details of the training process and hyperparameter settings are described in Appendix B. 9
中文: 培训过程和超参数设置的详细情况见附录B。
<a id="S0144"></a> Source: p.10 S0144
Original: 2.4 Evaluation For few-shot learning, we evaluate each example in the evaluation set by randomly drawing K examples from that task’s training set as conditioning, delimited by 1 or 2 newlines depending on the task.
中文: 2.4 评价 我们通过随机从任务训练中抽取 K 的例子来评价所设定的评价中的每个例子。
<a id="S0145"></a> Source: p.10 S0145
Original: For LAMBADA and Storycloze there is no supervised training set available so we draw conditioning examples from the development set and evaluate on the test set.
中文: 对LAMBADA和Storycloze来说,没有监督培训集,因此我们从开发集中抽取条件性实例,并在测试集中进行评估。
<a id="S0146"></a> Source: p.10 S0146
Original: For Winograd (the original, not SuperGLUE version) there is only one dataset, so we draw conditioning examples directly from it. K can be any value from 0 to the maximum amount allowed by the model’s context window, which is n = 2048 ctx for all models and typically fits 10 to 100 examples.
中文: 对于Winograd(原作,不是SuperGLUE版本),只有一个数据集,所以我们直接从中抽取了条件示例. K可以是从0到模型上下文窗口所允许的最大量的任何值,对于所有模型来说是n = 2048 ctx,一般符合10到100个实例.
<a id="S0147"></a> Source: p.10 S0147
Original: Larger values of K are usually but not always better, so when a separate development and test set are available, we experiment with a few values of K on the development set and then run the best value on the test set.
中文: K的较大值通常但并不总是更好,因此当有单独的开发与测试集时,我们在开发集上试验了几个K的值,然后在测试集上运行最佳值.
<a id="S0148"></a> Source: p.10 S0148
Original: For some tasks (see Appendix G) we also use a natural language prompt in addition to (or for K = 0, instead of) demonstrations.
中文: 对于一些任务(见附录G),除了(或用于K=0,而不是)演示之外,我们还使用自然语言的提示.
<a id="S0149"></a> Source: p.10 S0149
Original: On tasks that involve choosing one correct completion from several options (multiple choice), we provide K examples of context plus correct completion, followed by one example of context only, and compare the LM likelihood of each completion.
中文: 关于从几个选项中选择一个正确的完成(多选项)的任务,我们提供了上下文+正确完成的K实例,然后是仅上下文的一个实例,并比较每次完成的LM可能性.
<a id="S0150"></a> Source: p.10 S0150
Original: For most tasks we compare the per-token likelihood (to normalize for length), however on a small number of datasets (ARC, OpenBookQA, and RACE) we gain additional benefit as measured on the development set by normalizing by the unconditional probability of each completion, by computing P (completion|context) , where P (completion|answer context) answer context is the string "Answer: " or "A: " and is used to prompt that the completion should be an answer but is otherwise generic.
中文: 对于大多数任务,我们比较了按人计算的可能性(在长度上是正常化的),然而,在少数数据集(ARC、OpenBookQA和RACE)上,我们通过计算P(完成XQContext),通过计算P(完成XX)解答上下文是字符串"Answer:"或"A:",从而在设定的开发上获得了额外的好处,并被用来提示完成应当是一个答案,但本来是通用的.
<a id="S0151"></a> Source: p.10 S0151
Original: On tasks that involve binary classification, we give the options more semantically meaningful names (e.g. “True” or “False” rather than 0 or 1) and then treat the task like multiple choice; we also sometimes frame the task similar to what is done by [RSR+19] (see Appendix G) for details.
中文: 在涉及二进制分类的任务上,我们给选项以更精密有意义的名称(如"True"或"False"而不是"0"或"1"),然后将任务作为多重选择来对待;我们有时也会将任务设定在类似RSR+19所完成的任务中来详细了解.
<a id="S0152"></a> Source: p.10 S0152
Original: On tasks with free-form completion, we use beam search with the same parameters as [RSR+19]: a beam width of 4 and a length penalty of α = 0.6.
中文: 在自由形式完成的任务上,我们使用与[RSR+19]相同的参数来进行光束搜索:光束宽为4,长度为α=0.6.
<a id="S0153"></a> Source: p.10 S0153
Original: We score the model using F1 similarity score, BLEU, or exact match, depending on what is standard for the dataset at hand.
中文: 我们使用F1相似度分,BLEU,或精确匹配来评分模型,这取决于手头数据集的标准.
<a id="S0154"></a> Source: p.10 S0154
Original: Final results are reported on the test set when publicly available, for each model size and learning setting (zero-, one-, and few-shot).
中文: 每个模型大小和学习环境(零、一和几发)在公开的测试集中报告最终结果。
<a id="S0155"></a> Source: p.10 S0155
Original: When the test set is private, our model is often too large to fit on the test server, so we report results on the development set.
中文: 当测试集是私有的时,我们的模型往往太大,无法适应测试服务器,所以我们报告开发集的结果.
<a id="S0156"></a> Source: p.10 S0156
Original: We do submit to the test server on a small number of datasets (SuperGLUE, TriviaQA, PiQa) where we were able to make submission work, and we submit only the 200B few-shot results, and report development set results for everything else. 3 Results In Figure 3.1 we display training curves for the 8 models described in Section 2.
中文: 我们确实在少数数据集(SuperGLUE,TriviaQA,PiQa)上向测试服务器提交,我们在那里能够使提交工作发挥作用,我们只提交200B的几发结果,并报告其他一切的开发结果. 图3.1显示了第2节所描述的8个模型的培训曲线。
<a id="S0157"></a> Source: p.10 S0157
Original: For this graph we also include 6 additional extra-small models with as few as 100,000 parameters.
中文: 对于这个图,我们还包括另外6个小外型号,只有10万个参数。
<a id="S0158"></a> Source: p.10 S0158
Original: As observed in [KMH+20], language modeling performance follows a power-law when making efficient use of training compute.
中文: 如[KMH+20]所指出,语言建模的性能在有效利用训练计算时遵循了权力法.
<a id="S0159"></a> Source: p.10 S0159
Original: After extending this trend by two more orders of magnitude, we observe only a slight (if any) departure from the power-law.
中文: 在将这一趋势再扩大两个数量级之后,我们只看到略微偏离(如果有的话)权力法。
<a id="S0160"></a> Source: p.10 S0160
Original: One might worry that these improvements in cross-entropy loss come only from modeling spurious details of our training corpus.
中文: 人们可能担心 交叉性损失的这些改善 仅仅来自于我们训练队的虚构细节
<a id="S0161"></a> Source: p.10 S0161
Original: However, we will see in the following sections that improvements in cross-entropy loss lead to consistent performance gains across a broad spectrum of natural language tasks.
中文: 然而,我们将在以下各节中看到,跨大陆损失的改善导致在广泛的自然语言任务中取得持续的业绩收益。
<a id="S0162"></a> Source: p.10 S0162
Original: Below, we evaluate the 8 models described in Section 2 (the 175 billion parameter parameter GPT-3 and 7 smaller models) on a wide range of datasets.
中文: 下面,我们评价第2节所描述的8个模型(175亿个参数参数GPT-3和7个更小的模型)在广泛的数据集上.
<a id="S0163"></a> Source: p.10 S0163
Original: We group the datasets into 9 categories representing roughly similar tasks.
中文: 我们把数据集分为9个类别,代表大致相似的任务。
<a id="S0164"></a> Source: p.10 S0164
Original: In Section 3.1 we evaluate on traditional language modeling tasks and tasks that are similar to language modeling, such as Cloze tasks and sentence/paragraph completion tasks.
中文: 在第3.1节中,我们评价与语言模型相似的传统语言模型化任务和任务,如克洛兹任务和句子/段落完成任务.
<a id="S0165"></a> Source: p.10 S0165
Original: In Section 3.2 we evaluate on “closed book” question answering tasks: tasks which require using the information stored in the model’s parameters to answer general knowledge questions.
中文: 在第3.2节中,我们评价 " 非公开 " 问答任务:需要利用模型参数所储存的信息回答一般知识问题的任务。
<a id="S0166"></a> Source: p.10 S0166
Original: In Section 3.3 we evaluate the model’s ability to translate between languages (especially one-shot and few-shot).
中文: 第3.3节评价该模型在语言之间翻译的能力(尤其是一发和几发)。
<a id="S0167"></a> Source: p.10 S0167
Original: In Section 3.4 we evaluate the model’s performance on Winograd Schema-like tasks.
中文: 在3.4节中,我们评价模型在Winograd Schema类似任务方面的表现。
<a id="S0168"></a> Source: p.10 S0168
Original: In Section 3.5 we evaluate on datasets that involve commonsense reasoning or question answering.
中文: 在第3.5节中,我们评价涉及常识推理或回答问题的数据集。
<a id="S0169"></a> Source: p.10 S0169
Original: In Section 3.6 we evaluate on reading comprehension tasks, in Section 3.7 we evaluate on the SuperGLUE benchmark suite, and in 3.8 we briefly explore NLI.
中文: 在第3.6节中,我们评价阅读理解任务,在第3.7节中,我们评价SuperGLUE基准套件,在第3.8节中,我们简要地探讨NLI。
<a id="S0170"></a> Source: p.10 S0170
Original: Finally, in Section 3.9, we invent some additional tasks designed especially to probe in-context learning abilities – these tasks focus on on-the-fly reasoning, adaptation skills, or open-ended text synthesis.
中文: 最后,在第3.9节中,我们发明了一些额外的任务,特别是旨在探究内容中学习能力的任务 — — 这些任务侧重于飞行推理、适应技巧或开放式文本综合。
<a id="S0171"></a> Source: p.10 S0171
Original: We evaluate all tasks in the few-shot, one-shot, and zero-shot settings. 10
中文: 我们在几发、一发和零发设置中评估所有任务。 10个
<a id="S0172"></a> Source: p.11 S0172
Original: Figure 3.1: Smooth scaling of performance with compute.
中文: 图3.1:通过计算平滑地缩放性能。
<a id="S0173"></a> Source: p.11 S0173
Original: Performance (measured in terms of cross-entropy validation loss) follows a power-law trend with the amount of compute used for training.
中文: 性能(以交叉进取验证损失计量)遵循动力法趋势,计算数量用于培训。
<a id="S0174"></a> Source: p.11 S0174
Original: The power-law behavior observed in [KMH+20] continues for an additional two orders of magnitude with only small deviations from the predicted curve.
中文: 在[KMH+20]中观察到的动力-法行为持续了另外两个数量级,只与所预测的曲线有小的偏差.
<a id="S0175"></a> Source: p.11 S0175
Original: For this figure, we exclude embedding parameters from compute and parameter counts.
中文: 对于这个数字,我们从计算和参数计数中排除了嵌入参数.
<a id="S0176"></a> Source: p.11 S0176
Original: Setting PTB SOTA (Zero-Shot) 35.8a GPT-3 Zero-Shot 20.5 Table 3.1: Zero-shot results on PTB language modeling dataset.
中文: 设置 PTB SOTA (Zero-Shot) 35.8a GPT-3 Zero-Shot 20.5 表3.1: PTB语言建模数据集零镜头结果.
<a id="S0177"></a> Source: p.11 S0177
Original: Many other common language modeling datasets are omitted because they are derived from Wikipedia or other sources which are included in GPT-3’s training data. a[RWC+19] 3.1 Language Modeling, Cloze, and Completion Tasks In this section we test GPT-3’s performance on the traditional task of language modeling, as well as related tasks that involve predicting a single word of interest, completing a sentence or paragraph, or choosing between possible completions of a piece of text. 3.1.1 Language Modeling We calculate zero-shot perplexity on the Penn Tree Bank (PTB) [MKM+94] dataset measured in [RWC+19].
中文: 许多其他通用语言建模数据集被省略,因为它们来自维基百科或GPT-3培训数据所包含的其他来源. a [RWC+19] 3.1 语文建模、克洛兹和完成任务 在本节中,我们检验了GPT-3在语言建模的传统任务方面的表现,以及相关的任务,这些任务涉及预测一个感兴趣的单词,完成一个句子或段落,或在可能完成一个文本之间作出选择。 3.1.1 语文建模 我们计算出用[RWC+19]测量的Penn Tree Bank(PTB)[MKM+94]数据集上的零射偏差.
<a id="S0178"></a> Source: p.11 S0178
Original: We omit the 4 Wikipedia-related tasks in that work because they are entirely contained in our training data, and we also omit the one-billion word benchmark due to a high fraction of the dataset being contained in our training set.
中文: 我们省略了4个维基百科相关的任务, 因为这些任务完全包含在我们的培训数据中, 我们还省略了10亿字的基准, 因为我们的培训数据集中包含的很大一部分。
<a id="S0179"></a> Source: p.11 S0179
Original: PTB escapes these issues due to predating the modern internet.
中文: PTB因在现代互联网上出现前,
<a id="S0180"></a> Source: p.11 S0180
Original: Our largest model sets a new SOTA on PTB by a substantial margin of 15 points, achieving a perplexity of 20.50.
中文: 我们最大的模型在PTB上设定了一个新的SOTA,大幅比分为15分,实现了20.50分的迷惑.
<a id="S0181"></a> Source: p.11 S0181
Original: Note that since PTB is a traditional language modeling dataset it does not have a clear separation of examples to define one-shot or few-shot evaluation around, so we measure only zero-shot. 3.1.2 LAMBADA The LAMBADA dataset [PKL+16] tests the modeling of long-range dependencies in text – the model is asked to predict the last word of sentences which require reading a paragraph of context.
中文: 请注意,由于PTB是一种传统的语言模型数据集,它没有明确区分实例来定义一发或几发左右的评价,所以我们只测量零发. 3.1.2 马拉维 LAMBADA数据集[PKL+16]测试了文本中远程依赖的建模 — 该模型被要求预测需要读取上下文段落的句子上的最后一句话.
<a id="S0182"></a> Source: p.11 S0182
Original: It has recently been suggested that the continued scaling of language models is yielding diminishing returns on this difficult benchmark. [BHT+20] reflect on the small 1.5% improvement achieved by a doubling of model size between two recent state of the art results ([SPP+19] 11
中文: 最近有人建议,继续扩大语文模式,使这一困难基准的回报减少。 [BHT+20]反思最近两个最先进的结果之间模型尺寸翻一番所取得的1.5%的微小改进([SPP+19] 11)
<a id="S0183"></a> Source: p.12 S0183
Original: LAMBADA LAMBADA StoryCloze HellaSwag Setting (acc) (ppl) (acc) (acc) SOTA 68.0a 8.63b 91.8c 85.6d GPT-3 Zero-Shot 76.2 3.00 83.2 78.9 GPT-3 One-Shot 72.5 3.35 84.7 78.1 GPT-3 Few-Shot 86.4 1.92 87.7 79.3 Table 3.2: Performance on cloze and completion tasks.
中文: LAMBADA LAMBADA StoryCloze HelaSwag Setting (ac)(ppl)(ac)(ac) SOTA 68.0a 8.63b 91.8c 85.6d GPT-3 Zero-Shot 76.2 3.00 83.2 78.9 GPT-3 One-Shot 72.5 3.35 84.7 78.1 GPT-3 少-Shot 86.4 1.92 87.7 79.3 表 3.2: 血浆和完成任务的性能.
<a id="S0184"></a> Source: p.12 S0184
Original: GPT-3 significantly improves SOTA on LAMBADA while achieving respectable performance on two difficult completion prediction datasets. a[Tur20] b[RWC+19] c[LDL19] d[LCH+20] Figure 3.2: On LAMBADA, the few-shot capability of language models results in a strong boost to accuracy.
中文: GPT-3在LAMBADA上显著地改进了SOTA,同时在两个困难的完成预测数据集上实现了可敬的性能. a [Tur20] b [RWC+19] c [LDL19] d [LCH+20] 图3.2: 在LAMBADA上,语言模型的几发能力使得精度得到强大的提升.
<a id="S0185"></a> Source: p.12 S0185
Original: GPT-3 2.7B outperforms the SOTA 17B parameter Turing-NLG [Tur20] in this setting, and GPT-3 175B advances the state of the art by 18%.
中文: GPT-3 2.7B在此设置中超越了SOTA 17B参数Turing-NLG[tur20],而GPT-3 175B则以18%的成绩推进了该状态.
<a id="S0186"></a> Source: p.12 S0186
Original: Note zero-shot uses a different format from one-shot and few-shot as described in the text. and [Tur20]) and argue that “continuing to expand hardware and data sizes by orders of magnitude is not the path forward”.
中文: 注意零发与正文所描述的一发与几发不同. “继续按数量级扩大硬件和数据规模并不是前进的道路”。
<a id="S0187"></a> Source: p.12 S0187
Original: We find that path is still promising and in a zero-shot setting GPT-3 achieves 76% on LAMBADA, a gain of 8% over the previous state of the art.
中文: 我们发现路径仍然充满希望, 在零镜头设定中,GPT-3在LAMBADA上取得了76%的成绩, 比之前的艺术水平提高了8%。
<a id="S0188"></a> Source: p.12 S0188
Original: LAMBADA is also a demonstration of the flexibility of few-shot learning as it provides a way to address a problem that classically occurs with this dataset.
中文: LAMBADA也证明了几发学习的灵活性,因为它提供了一种解决一个问题的方法,这个问题在传统上就是用这个数据集发生的.
<a id="S0189"></a> Source: p.12 S0189
Original: Although the completion in LAMBADA is always the last word in a sentence, a standard language model has no way of knowing this detail.
中文: 虽然在LAMBADA中完成总是句子中的最后一句话,但标准语言模型无法知道这个细节.
<a id="S0190"></a> Source: p.12 S0190
Original: It thus assigns probability not only to the correct ending but also to other valid continuations of the paragraph.
中文: 因此,它不仅为该段的正确结尾提供了可能性,而且还为该段的其他有效延续提供了可能性。
<a id="S0191"></a> Source: p.12 S0191
Original: This problem has been partially addressed in the past with stop-word filters [RWC+19] (which ban “continuation” words).
中文: 这个问题过去曾被部分地用止句过滤器RWC+19来解决.
<a id="S0192"></a> Source: p.12 S0192
Original: The few-shot setting instead allows us to “frame” the task as a cloze-test and allows the language model to infer from examples that a completion of exactly one word is desired.
中文: 仅几发的设定使我们得以“设定”任务,作为血块测试,并允许语言模型从例子中推断出需要完成一个字。
<a id="S0193"></a> Source: p.12 S0193
Original: We use the following fill-in-the-blank format: Alice was friends with Bob.
中文: 我们使用以下填充式: 爱丽丝是鲍勃的朋友.
<a id="S0194"></a> Source: p.12 S0194
Original: Alice went to visit her friend . → Bob George bought some baseball equipment, a ball, a glove, and a . → When presented with examples formatted this way, GPT-3 achieves 86.4% accuracy in the few-shot setting, an increase of over 18% from the previous state-of-the-art.
中文: 爱丽丝去探望她的朋友 鲍勃·乔治买了一些棒球器材 一个球 一个手套和一个 在给出这样格式化的示例时,GPT-3在几发设定中实现了86.4%的准确度,比上一流水平提高了18%以上.
<a id="S0195"></a> Source: p.12 S0195
Original: We observe that few-shot performance improves strongly with model size.
中文: 我们观察到,很少镜头的性能随着模型大小的提高而大有改进.
<a id="S0196"></a> Source: p.12 S0196
Original: While this setting decreases the performance of the smallest model by almost 20%, for GPT-3 it improves accuracy by 10%.
中文: 虽然这种设置降低了最小模型的性能近20%,但GPT-3的精度提高了10%.
<a id="S0197"></a> Source: p.12 S0197
Original: Finally, the fill-in-blank method is not effective one-shot, where it always performs worse than the zero-shot setting.
中文: 最后,填充法并不是有效的一发法,它的表现总是比零发法的设定更糟糕.
<a id="S0198"></a> Source: p.12 S0198
Original: Perhaps this is because all models still require several examples to recognize the pattern. 12
中文: 也许这是因为所有模型仍然需要几个实例来识别模式. 第12条
<a id="S0199"></a> Source: p.13 S0199
Original: Setting NaturalQS WebQS TriviaQA RAG (Fine-tuned, Open-Domain) [LPP+20] 44.5 45.5 68.0 T5-11B+SSM (Fine-tuned, Closed-Book) [RRS20] 36.6 44.7 60.5 T5-11B (Fine-tuned, Closed-Book) 34.5 37.4 50.1 GPT-3 Zero-Shot 14.6 14.4 64.3 GPT-3 One-Shot 23.0 25.3 68.0 GPT-3 Few-Shot 29.9 41.5 71.2 Table 3.3: Results on three Open-Domain QA tasks.
中文: 设置 NaturalQS WebQS TriviaQA RAG(Fine-tuned, Open-Domain) [LPP+20] 44.5 45.5 68.0 T5-11B+SSM(Fine-tuned, Close-Book) [RRS20] 36.6 44.7 60.5 T5-11B(Fine-tuned, Close-Book) 34.5 37.4 50.1 GPT-3 Zero-Shot 14.6 14.4 64.3 GPT-3 One-Shot 23.0 25.3 68.68.0 GPT-3 稀Shot 29.9 41.5 71.2 表3.3:三项开放域质量保证任务的结果。
<a id="S0200"></a> Source: p.13 S0200
Original: GPT-3 is shown in the few-, one-, and zero-shot settings, as compared to prior SOTA results for closed book and open domain settings.
中文: GPT-3在少数-,一-和0-shot设置中显示,与之前的SOTA结果相比,用于关闭的书和开放域设置.
<a id="S0201"></a> Source: p.13 S0201
Original: TriviaQA few-shot result is evaluated on the wiki split test server.
中文: TriviaQA几发结果在维基分拆测试服务器上得到评价.
<a id="S0202"></a> Source: p.13 S0202
Original: One note of caution is that an analysis of test set contamination identified that a significant minority of the LAMBADA dataset appears to be present in our training data – however analysis performed in Section 4 suggests negligible impact on performance. 3.1.3 HellaSwag The HellaSwag dataset [ZHB+19] involves picking the best ending to a story or set of instructions.
中文: 需要注意的是,对试验组污染的分析发现,我们的培训数据中似乎有相当一部分是LAMBADA数据集,但第4节的分析表明,对业绩的影响微不足道。 3.1.3 HelaSwag The HelaSwag数据集[ZHB+19]涉及选取一个故事或一套指令的最佳结局.
<a id="S0203"></a> Source: p.13 S0203
Original: The examples were adversarially mined to be difficult for language models while remaining easy for humans (who achieve 95.6% accuracy).
中文: 这些例子对语言模型来说是困难的,对人类来说是容易的(他们达到95.6%)。
<a id="S0204"></a> Source: p.13 S0204
Original: GPT-3 achieves 78.1% accuracy in the one-shot setting and 79.3% accuracy in the few-shot setting, outperforming the 75.4% accuracy of a fine-tuned 1.5B parameter language model [ZHR+19] but still a fair amount lower than the overall SOTA of 85.6% achieved by the fine-tuned multi-task model ALUM. 3.1.4 StoryCloze We next evaluate GPT-3 on the StoryCloze 2016 dataset [MCH+16], which involves selecting the correct ending sentence for five-sentence long stories.
中文: GPT-3在一发设定中实现了78.1%的精度,在少发设定中实现了79.3%的精度,比微调1.5B参数语言模型[ZHR+19]的精度75.4%的精度高出一分,但仍比微调多任务模型ALUM实现的总SOTA为85.6%的精度要低. 3.1.4 故事分类 我们接下来在StoryCloze 2016数据集[MCH+16]上评价了GPT-3,这涉及到为五刑期长故事选择正确的终结句.
<a id="S0205"></a> Source: p.13 S0205
Original: Here GPT-3 achieves 83.2% in the zero-shot setting and 87.7% in the few-shot setting (with K = 70).
中文: 在这里,GPT-3在零发设定中实现83.2%,在少发设定中实现87.7%(有K=70).
<a id="S0206"></a> Source: p.13 S0206
Original: This is still 4.1% lower than the fine-tuned SOTA using a BERT based model [LDL19] but improves over previous zero-shot results by roughly 10%. 3.2 Closed Book Question Answering In this section we measure GPT-3’s ability to answer questions about broad factual knowledge.
中文: 这仍然比使用基于BERT的模型[LDL19]进行微调的SOTA低了4.1%,但比以往的零镜头结果提高了约10%. 3.2 闭库问答 本节衡量GPT-3是否有能力回答关于广泛事实知识的问题。
<a id="S0207"></a> Source: p.13 S0207
Original: Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text.
中文: 由于可能提出的查询数量巨大,通常采用信息检索系统来查找相关文本,同时使用一个模式来学习如何生成对问题和已检索文本的答复。
<a id="S0208"></a> Source: p.13 S0208
Original: Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [RRS20] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information.
中文: 由于这种设置允许一个系统搜索可能包含答案的文本并附带条件,因此它被指为“开放式书籍”。 [RRS20]最近证明,一个大型语言模型能够令人惊讶地出色地直接回答问题,而不附带附加信息条件.
<a id="S0209"></a> Source: p.13 S0209
Original: They denote this more restrictive evaluation setting as “closed-book”.
中文: 它们将这种限制性更强的评价设定称为“封闭账本”。
<a id="S0210"></a> Source: p.13 S0210
Original: Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3.
中文: 他们的工作表明,甚至更高的能力模型也能发挥出更好的效果,我们用GPT-3测试这个假设.
<a id="S0211"></a> Source: p.13 S0211
Original: We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits.
中文: 我们在[RRS20]中的3个数据集上评价GPT-3:自然问题[KPR+19],网络问题[BCFL13],TriviaQA[JCWZ17],使用相同的分数.
<a id="S0212"></a> Source: p.13 S0212
Original: Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.
中文: 请注意,除了所有结果都是在封闭式书本设定中,我们使用"几发一发"和"零发"的评价代表着比之前的封闭式书本QA作品更严格的设定:除了不允许外部内容外,也不允许微调QQA数据集本身.
<a id="S0213"></a> Source: p.13 S0213
Original: The results for GPT-3 are shown in Table 3.3.
中文: GPT-3的结果见表3.3。
<a id="S0214"></a> Source: p.13 S0214
Original: On TriviaQA, we achieve 64.3% in the zero-shot setting, 68.0% in the one-shot setting, and 71.2% in the few-shot setting.
中文: 在"TriviaQA"上,我们在"零发"设定中实现64.3%,一发设定中达到68.0%,少发设定中达到71.2%.
<a id="S0215"></a> Source: p.13 S0215
Original: The zero-shot result already outperforms the fine-tuned T5-11B by 14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%.
中文: 零射出的结果已经比微调后T5-11B的成绩高出14.2%,也比训练前的XQA量身定制的跨度预测的版本高出3.8%.
<a id="S0216"></a> Source: p.13 S0216
Original: The one-shot result improves by 3.7% and matches the SOTA for an open-domain QA system which not only fine-tunes but also makes use of a learned retrieval mechanism over a 15.3B parameter dense vector index of 21M documents [LPP+20].
中文: 一发结果改进了3.7%,与SOTA匹配开放域QA系统,该系统不仅微调,而且利用了超过21M文件[LPP+20]的15.3B参数密向量指数的已学习回取机制.
<a id="S0217"></a> Source: p.13 S0217
Original: GPT-3’s few-shot result further improves performance another 3.2% beyond this.
中文: GPT-3的少数镜头结果进一步提高了业绩,
<a id="S0218"></a> Source: p.13 S0218
Original: On WebQuestions (WebQs), GPT-3 achieves 14.4% in the zero-shot setting, 25.3% in the one-shot setting, and 41.5% in the few-shot setting.
中文: 在WebQuestions (WebQs)上,GPT-3在"零发"设定中实现14.4%,一发设定中实现25.3%,少发设定中实现41.5%.
<a id="S0219"></a> Source: p.13 S0219
Original: This compares to 37.4% for fine-tuned T5-11B, and 44.7% for fine-tuned T5-11B+SSM, which uses a Q&A-specific pre-training procedure.
中文: 与之相较,微调T5-11B为37.4%;微调T5-11B+SSM为44.7%,使用QQA专用的预训程序.
<a id="S0220"></a> Source: p.13 S0220
Original: GPT-3 in the few-shot setting approaches the performance of state-of-the-art fine-tuned models.
中文: 几发设定中的GPT-3接近了最先进的微调模型的性能.
<a id="S0221"></a> Source: p.13 S0221
Original: Notably, compared to TriviaQA, WebQS shows a much larger gain from zero-shot to few-shot (and indeed its zero-shot and one-shot performance are poor), perhaps suggesting that the WebQs questions 13
中文: 值得注意的是,与"TriviaQA"相比,"WebQS"显示从"零发"到"少发"(实际上其"零发"和"一发"的性能都很差),也许暗示"WebQs"问题13.
<a id="S0222"></a> Source: p.14 S0222
Original: Figure 3.3: On TriviaQA GPT3’s performance grows smoothly with model size, suggesting that language models continue to absorb knowledge as their capacity increases.
中文: 图3.3:关于TriviaQA GPT3的性能随模型大小而平稳地增长,说明语言模型随着能力增强而继续吸收知识.
<a id="S0223"></a> Source: p.14 S0223
Original: One-shot and few-shot performance make significant gains over zero-shot behavior, matching and exceeding the performance of the SOTA fine-tuned open-domain model, RAG [LPP+20] and/or the style of their answers are out-of-distribution for GPT-3.
中文: 一发和多发性能比零发行为有显著收益,匹配并超越了SOTA微调开放域模型RAG[LPP+20]和/或他们的答案风格为GPT-3的分流.
<a id="S0224"></a> Source: p.14 S0224
Original: Nevertheless, GPT-3 appears able to adapt to this distribution, recovering strong performance in the few-shot setting.
中文: 尽管如此,GPT-3似乎能够适应这种分布,恢复了几发环境下的强劲性能.
<a id="S0225"></a> Source: p.14 S0225
Original: On Natural Questions (NQs) GPT-3 achieves 14.6% in the zero-shot setting, 23.0% in the one-shot setting, and 29.9% in the few-shot setting, compared to 36.6% for fine-tuned T5 11B+SSM.
中文: 在"自然问题"(NQs)上,GPT-3在"零发"设定中达到14.6%,"一发"设定中达到23.0%,"几发"设定中达到29.9%,而精制T5 11B+SSM则达到36.6%.
<a id="S0226"></a> Source: p.14 S0226
Original: Similar to WebQS, the large gain from zero-shot to few-shot may suggest a distribution shift, and may also explain the less competitive performance compared to TriviaQA and WebQS.
中文: 与WebQS类似,从"零拍"到"几拍"的巨大收益可能暗示出发行的转变,也可能解释与"TriviaQA"和"WebQS"相比,竞争性能较低的原因.
<a id="S0227"></a> Source: p.14 S0227
Original: In particular, the questions in NQs tend towards very fine-grained knowledge on Wikipedia specifically which could be testing the limits of GPT-3’s capacity and broad pretraining distribution.
中文: 特别是,NQs中的问题倾向于对维基百科有非常精细的知识,这可以测试GPT-3容量的限度和广泛的预训分布.
<a id="S0228"></a> Source: p.14 S0228
Original: Overall, on one of the three datasets GPT-3’s one-shot matches the open-domain fine-tuning SOTA.
中文: 总体来说,GPT-3的三个数据集之一的一发匹配了开放域微调SOTA.
<a id="S0229"></a> Source: p.14 S0229
Original: On the other two datasets it approaches the performance of the closed-book SOTA despite not using fine-tuning.
中文: 在另外两个数据集上,尽管没有使用微调,它还是接近了封闭式SOTA的性能.
<a id="S0230"></a> Source: p.14 S0230
Original: On all 3 datasets, we find that performance scales very smoothly with model size (Figure 3.3 and Appendix H Figure H.7), possibly reflecting the idea that model capacity translates directly to more ‘knowledge’ absorbed in the parameters of the model. 3.3 Translation For GPT-2 a filter was used on a multilingual collection of documents to produce an English only dataset due to capacity concerns.
中文: 在所有3个数据集中,我们发现模型大小的性能尺度非常平稳(图3.3和附录H图H.7)可能反映了模型能力直接转化为模型参数所吸收的更多 " 知识 " 的想法。 3.3 为GPT-2翻译,在多语种文件集上使用了过滤器,以便由于能力问题而制作一个仅有英文的数据集。
<a id="S0231"></a> Source: p.14 S0231
Original: Even with this filtering GPT-2 showed some evidence of multilingual capability and performed non-trivially when translating between French and English despite only training on 10 megabytes of remaining French text.
中文: 即便经过这种过滤,GPT-2也显示出一些多语种能力的证据,在法语和英语之间进行翻译时,尽管只培训了10兆字节的剩余法语文本。
<a id="S0232"></a> Source: p.14 S0232
Original: Since we increase the capacity by over two orders of magnitude from GPT-2 to GPT-3, we also expand the scope of the training dataset to include more representation of other languages, though this remains an area for further improvement.
中文: 由于我们的能力从GPT-2增加到了GPT-3,增加了两个多级,我们还扩大了培训数据集的范围,增加了其他语文的代表性,尽管这仍然是需要进一步改进的领域。
<a id="S0233"></a> Source: p.14 S0233
Original: As discussed in 2.2 the majority of our data is derived from raw Common Crawl with only quality-based filtering.
中文: 如2.2所讨论,我们的大部分数据来自只有质量过滤的原始 " 普通爬行 " 。
<a id="S0234"></a> Source: p.14 S0234
Original: Although GPT-3’s training data is still primarily English (93% by word count), it also includes 7% of text in other languages.
中文: 虽然GPT-3的训练数据仍然主要是英语(按字数计算为93%),但也包含其他语言的7%的文本.
<a id="S0235"></a> Source: p.14 S0235
Original: These languages are documented in the supplemental material.
中文: 这些语文已编入补充材料。
<a id="S0236"></a> Source: p.14 S0236
Original: In order to better understand translation capability, we also expand our analysis to include two additional commonly studied languages, German and Romanian.
中文: 为了更好地理解翻译能力,我们还将分析扩大到包括另外两种共同学习的语言,德语和罗马尼亚语。
<a id="S0237"></a> Source: p.14 S0237
Original: Existing unsupervised machine translation approaches often combine pretraining on a pair of monolingual datasets with back-translation [SHB15] to bridge the two languages in a controlled way.
中文: 现有的无人监督的机器翻译方法往往将一对单语数据集的预训与回译[SHB15]结合起来,以控制的方式将两种语言相接.
<a id="S0238"></a> Source: p.14 S0238
Original: By contrast, GPT-3 learns from a blend of training data that mixes many languages together in a natural way, combining them on a word, sentence, and document level.
中文: 相形之下,GPT-3学习了将许多语言自然地融合在一起的混合培训数据,在单词,句子,和文档级别上结合.
<a id="S0239"></a> Source: p.14 S0239
Original: GPT-3 also uses a single training objective which is not customized or designed for any task in particular.
中文: GPT-3还使用一个单一的培训目标,这个目标不专门为任何特定任务定制或设计。
<a id="S0240"></a> Source: p.14 S0240
Original: However, our one / few-shot settings aren’t strictly comparable to prior unsupervised work since they make use of a small amount of paired examples (1 or 64).
中文: 不过,我们一个/几个镜头的设置与以前没有监督的作品并不完全相似,因为它们利用了少量的配对例子(1或64个)。
<a id="S0241"></a> Source: p.14 S0241
Original: This corresponds to up to a page or two of in-context training data.
中文: 这最多相当于一页或两页的内文训练数据.
<a id="S0242"></a> Source: p.14 S0242
Original: Zero-shot GPT-3, which only receives on a natural language description of the task, still underperforms recent unsupervised NMT results.
中文: 零射出GPT-3,它只收到关于任务的自然语言描述,仍然表现不佳,最近没有监督的NMT结果.
<a id="S0243"></a> Source: p.14 S0243
Original: However, providing only a single example demonstration for 14
中文: 但是,仅提供14个实例。
<a id="S0244"></a> Source: p.15 S0244
Original: Setting En→Fr Fr→En En→De De→En En→Ro Ro→En SOTA (Supervised) 45.6a 35.0 b 41.2c 40.2d 38.5e 39.9e XLM [LC19] 33.4 33.3 26.4 34.3 33.3 31.8 MASS [STQ+19] 37.5 34.9 28.3 35.2 35.2 33.1 mBART [LGG+20] - - 29.8 34.0 35.0 30.5 GPT-3 Zero-Shot 25.2 21.2 24.6 27.2 14.1 19.9 GPT-3 One-Shot 28.3 33.7 26.2 30.4 20.6 38.6 GPT-3 Few-Shot 32.6 39.2 29.7 40.6 21.0 39.5 Table 3.4: Few-shot GPT-3 outperforms previous unsupervised NMT work by 5 BLEU when translating into English reflecting its strength as an English LM.
中文: 设置 En-Fr-En-En-De-En-Ro-En-Ro-En SOTA(监督) 45.6a 35.0 b 41.2c 40.2d 38.5e 39.9e XLM [LC19] 33.4 33.3 26.4 34.3 33.3 31.8 MASS [STQ+19] 37.5 34.9 28.3 35.2 35.2 33.1 mBART [LGG+20] - - - - 29.8 34.0 35.5 GPT-3 零-Shot 25.2 24.6 27.2 14.1 19.9 GPT-3 One-Shot 28.3 33.7 26.2 30.6 20.6 38.6 GPT-3 De-Shot 32.6 39.2 29.6 21.6 39.5 表3.4: 几发GPT-3超前无监督的NMT-3工作,由5 BLEU翻译为英语LM。
<a id="S0245"></a> Source: p.15 S0245
Original: We report BLEU scores on the WMT’14 Fr↔En, WMT’16 De↔En, and WMT’16 Ro↔En datasets as measured by multi-bleu.perl with XLM’s tokenization in order to compare most closely with prior unsupervised NMT work.
中文: 我们在WMT的14 Fr↔En、WMT的16 De↔En和WMT的16 Ro↔En数据集上报告BLEU的分数,这些数据集用多bleu.perl和XLM的标注来测量,以便与以前的不受监督的NMT工作进行最密切的比较.
<a id="S0246"></a> Source: p.15 S0246
Original: SacreBLEUf [Pos18] results reported in Appendix H.
中文: SacrebleUf[后18]结果见附录H。
<a id="S0247"></a> Source: p.15 S0247
Original: Underline indicates an unsupervised or few-shot SOTA, bold indicates supervised SOTA with relative confidence. a[EOAG18] b[DHKH14] c[WXH+18] d[oR16] e[LGG+20] f [SacreBLEU signature: BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20] Figure 3.4: Few-shot translation performance on 6 language pairs as model capacity increases.
中文: 下划线表示无监督或少发SOTA,粗体表示有相对自信的受监督SOTA. a [EOAG18] b [DHKH14] c [WXH+18] d [oR16] e [LGG+20] f [SacreBLBLEU签名:BLEU+case.mixed+numrefs.1+smooth.exp+tok.intl+version.1.2.20] 图3.4:随着模型容量的增加,6个语言对上少有镜头的翻译性能.
<a id="S0248"></a> Source: p.15 S0248
Original: There is a consistent trend of improvement across all datasets as the model scales, and as well as tendency for translation into English to be stronger than translation from English. 15
中文: 作为模型尺度,所有数据集都有不断改进的趋势,而且英文译本也比英文译本强。 15个
<a id="S0249"></a> Source: p.16 S0249
Original: Setting Winograd Winogrande (XL) Fine-tuned SOTA 90.1a 84.6b GPT-3 Zero-Shot 88.3 70.2 GPT-3 One-Shot 89.7 73.2 GPT-3 Few-Shot 88.6* 77.7 Table 3.5: Results on the WSC273 version of Winograd schemas and the adversarial Winogrande dataset.
中文: 设置 Winograd Winogrande (XL) 微调 SOTA 90.1a 84.6b GPT-3 Zero-Shot 88.3 70.2 GPT-3 One-Shot 89.7 73.2 GPT-3 稀有-Shot 88.6* 77.7 表3.5:WSC273版本的Winograd schemas和对抗性的Winogrande数据集的结果.
<a id="S0250"></a> Source: p.16 S0250
Original: See Section 4 for details on potential contamination of the Winograd test set. a[SBBC19] b[LYN+20] Figure 3.5: Zero-, one-, and few-shot performance on the adversarial Winogrande dataset as model capacity scales.
中文: 关于威诺格勒测试集可能受污染的详细情况,见第4节。 a[SBBC19] b[LYN+20] 图3.5:作为模型容量尺度的对抗性的威诺格朗德数据集上的零射、一射和几发性能。
<a id="S0251"></a> Source: p.16 S0251
Original: Scaling is relatively smooth with the gains to few-shot learning increasing with model size, and few-shot GPT-3 175B is competitive with a fine-tuned RoBERTA-large. each translation task improves performance by over 7 BLEU and nears competitive performance with prior work.
中文: 缩放相对平滑,因为随着模型大小的增加,可以进行几发学习,而几发GPT-3 175B则与微调的RoBERTA大相竞争. 每项翻译任务都提高了7个以上BLEU的性能,并且与以前的工作接近竞争性能.
<a id="S0252"></a> Source: p.16 S0252
Original: GPT-3 in the full few-shot setting further improves another 4 BLEU resulting in similar average performance to prior unsupervised NMT work.
中文: 完全的几发设定中的GPT-3进一步改善了另外4个BLEU,导致平均性能与以前未经监督的NMT工作类似.
<a id="S0253"></a> Source: p.16 S0253
Original: GPT-3 has a noticeable skew in its performance depending on language direction.
中文: GPT-3的性能依语言取向而有显著的skew.
<a id="S0254"></a> Source: p.16 S0254
Original: For the three input languages studied, GPT-3 significantly outperforms prior unsupervised NMT work when translating into English but underperforms when translating in the other direction.
中文: 对于所研究的三种输入语言,GPT-3在翻译为英语时显著地超越了先前未经监督的NMT工作,而在其他方向翻译时则表现不佳.
<a id="S0255"></a> Source: p.16 S0255
Original: Performance on En-Ro is a noticeable outlier at over 10 BLEU worse than prior unsupervised NMT work.
中文: En-Ro的性能比以前未经监督的NMT工作要差10个以上的BLEU,是一个显著的超值。
<a id="S0256"></a> Source: p.16 S0256
Original: This could be a weakness due to reusing the byte-level BPE tokenizer of GPT-2 which was developed for an almost entirely English training dataset.
中文: 这可能是一个弱点,因为重新使用GPT-2的字节级BPE表示器,该表示器是为几乎完全用于英语培训数据集而开发的。
<a id="S0257"></a> Source: p.16 S0257
Original: For both Fr-En and De-En, few shot GPT-3 outperforms the best supervised result we could find but due to our unfamiliarity with the literature and the appearance that these are un-competitive benchmarks we do not suspect those results represent true state of the art.
中文: 对Fr-En和De-En来说,很少有人射出GPT-3的成绩超过了我们所能发现的最好的监督结果,但由于我们对文献不熟悉,而且这些似乎不是我们所怀疑的无竞争力的基准,这些结果代表了真正的技术水平。
<a id="S0258"></a> Source: p.16 S0258
Original: For Ro-En, few shot GPT-3 performs within 0.5 BLEU of the overall SOTA which is achieved by a combination of unsupervised pretraining, supervised finetuning on 608K labeled examples, and backtranslation [LHCG19b].
中文: 对Ro-En来说,在整体SOTA的0.5 BLEU范围内,很少射击出GPT-3,这是通过下列方式实现的:无监督的预训、608K标签示例的有监督的微调和回译[LHCG19b]。
<a id="S0259"></a> Source: p.16 S0259
Original: Finally, across all language pairs and across all three settings (zero-, one-, and few-shot), there is a smooth trend of improvement with model capacity.
中文: 最后,在所有语言配对和所有三种设置(0-、1-和0-shot)中,都有一个有模型能力的平稳改进趋势。
<a id="S0260"></a> Source: p.16 S0260
Original: This is shown in Figure 3.4 in the case of few-shot results, and scaling for all three settings is shown in Appendix H. 3.4 Winograd-Style Tasks The Winograd Schemas Challenge [LDM12] is a classical task in NLP that involves determining which word a pronoun refers to, when the pronoun is grammatically ambiguous but semantically unambiguous to a human.
中文: 图3.4中以少发结果为例显示了这一点,所有三种设置的缩放在附录H. 3.4 Winograd-Style Tasks The Winograd Schemas Challenge [LDM12]是NLP中的一项古典任务,涉及确定一个代名词指的是哪个词,当代名词在语法上是模糊的,但对于一个人类来说是完全毫不含糊的.
<a id="S0261"></a> Source: p.16 S0261
Original: Recently fine-tuned language models have achieved near-human performance on the original Winograd dataset, but more difficult versions 16
中文: 最近微调后的语言模型在最初的Winograd数据集上实现了近人性能,但难度更高的版本16
<a id="S0262"></a> Source: p.17 S0262
Original: Setting PIQA ARC (Easy) ARC (Challenge) OpenBookQA Fine-tuned SOTA 79.4 92.0[KKS+20] 78.5[KKS+20] 87.2[KKS+20] GPT-3 Zero-Shot 80.5 68.8 51.4 57.6 GPT-3 One-Shot 80.5 71.2 53.2 58.8 GPT-3 Few-Shot 82.8* 70.1 51.5 65.4 Table 3.6: GPT-3 results on three commonsense reasoning tasks, PIQA, ARC, and OpenBookQA.
中文: 设置 PIQA ARC(简单) ARC(挑战) OpenBookQA 精细调制 SOTA 79.4 92.0 [KKS+20] 78.5 [KKKS+20] 87.2 [KKKS+20] GPT-3 Zero-Shot 80.5 68.8 51.4 57.6 GPT-3 One-Shot 80.5 71.2 53.2 58.8 GPT-3 New-Shot 82.8* 70.1 51.5 65.4 表3.6:GPT-3在PIQA,ARC,和OpenBookQA三个常识推理任务上的结果.
<a id="S0263"></a> Source: p.17 S0263
Original: GPT-3 Few-Shot PIQA result is evaluated on the test server.
中文: GPT-3 微热PIQA结果在测试服务器上评价.
<a id="S0264"></a> Source: p.17 S0264
Original: See Section 4 for details on potential contamination issues on the PIQA test set.
中文: 关于PIQA测试集中潜在污染问题的详细情况,见第4节。
<a id="S0265"></a> Source: p.17 S0265
Original: Figure 3.6: GPT-3 results on PIQA in the zero-shot, one-shot, and few-shot settings.
中文: 图3.6:GPT-3在"零射","一发"和"少发"设置中PIQA的成绩.
<a id="S0266"></a> Source: p.17 S0266
Original: The largest model achieves a score on the development set in all three conditions that exceeds the best recorded score on the task. such as the adversarially-mined Winogrande dataset [SBBC19] still significantly lag human performance.
中文: 最大的模式在全部三个条件中,在任务中超过最佳记录分数的开发设定上都取得了分数. 例如有对抗力的Winogrande数据集[SBBC19]仍然显著落后于人类的性能.
<a id="S0267"></a> Source: p.17 S0267
Original: We test GPT-3’s performance on both Winograd and Winogrande, as usual in the zero-, one-, and few-shot setting.
中文: 在Winograd和Winogrande两地测试GPT-3的性能,
<a id="S0268"></a> Source: p.17 S0268
Original: On Winograd we test GPT-3 on the original set of 273 Winograd schemas, using the same “partial evaluation” method described in [RWC+19].
中文: 在维诺格勒,我们使用[RWC+19]中描述的同一种“部分评价”方法,在273个维诺克勒计划原地测试GPT-3。
<a id="S0269"></a> Source: p.17 S0269
Original: Note that this setting differs slightly from the WSC task in the SuperGLUE benchmark, which is presented as binary classification and requires entity extraction to convert to the form described in this section.
中文: 请注意,此设置与SuperGLUE基准中的WSC任务略有不同,该基准被作为二进制分类来列报,并要求实体提取转换为本节所描述的表单.
<a id="S0270"></a> Source: p.17 S0270
Original: On Winograd GPT-3 achieves 88.3%, 89.7%, and 88.6% in the zero-shot, one-shot, and few-shot settings, showing no clear in-context learning but in all cases achieving strong results just a few points below state-of-the-art and estimated human performance.
中文: 在Winograd GPT-3上,在零发,一发和几发设置中分别取得了88.3%,89.7%和88.6%的成绩,没有表现出明显的内在学习,但在所有情况下都只取得了低于最先进水平和估计人类性能的几分强成绩.
<a id="S0271"></a> Source: p.17 S0271
Original: We note that contamination analysis found some Winograd schemas in the training data but this appears to have only a small effect on results (see Section 4).
中文: 我们注意到,污染分析在培训数据中发现了一些Winograd计划,但这似乎对结果的影响很小(见第4节)。
<a id="S0272"></a> Source: p.17 S0272
Original: On the more difficult Winogrande dataset, we do find gains to in-context learning: GPT-3 achieves 70.2% in the zero-shot setting, 73.2% in the one-shot setting, and 77.7% in the few-shot setting.
中文: 在更困难的Winogrande数据集上,我们确实发现了对内文本学习的收获:GPT-3在"零发"设定中实现70.2%,一发设定中实现73.2%,少发设定中实现77.7%.
<a id="S0273"></a> Source: p.17 S0273
Original: For comparison a fine-tuned RoBERTA model achieves 79%, state-of-the-art is 84.6% achieved with a fine-tuned high capacity model (T5), and human performance on the task as reported by [SBBC19] is 94.0%. 3.5 Common Sense Reasoning Next we consider three datasets which attempt to capture physical or scientific reasoning, as distinct from sentence completion, reading comprehension, or broad knowledge question answering.
中文: 相比之下,经过微调的RoBERTA模型实现79%,通过微调的高容量模型(T5)达到84.6%的先进水平,[SBBC19]报告的人类在任务上的表现为94.0%. 3.5 共同理智的理由 接下来,我们考虑试图捕捉物理或科学推理的三个数据集,它们不同于句子的完成、阅读理解或广义的知识问题回答。
<a id="S0274"></a> Source: p.17 S0274
Original: The first, PhysicalQA (PIQA) [BZB+19], asks common sense questions about how the physical world works and is intended as a probe of grounded understanding of the world.
中文: 第一作"物理QA"(PIQA)[BZB+19],提出了关于物理世界如何运作的常识问题,并被用来探究对世界的有根据的理解.
<a id="S0275"></a> Source: p.17 S0275
Original: GPT-3 achieves 81.0% accuracy zero-shot, 80.5% accuracy one-shot, and 82.8% accuracy few-shot (the last measured on PIQA’s test server).
中文: GPT-3实现81.0%的精度为零发,80.5%的精度为一发,82.8%的精度为少发(最后在PIQA的测试服务器上测出).
<a id="S0276"></a> Source: p.17 S0276
Original: This compares favorably to the 79.4% accuracy prior state-of-the-art of a 17
中文: 这与先前17项中79.4%的准确性相比是有利的。
<a id="S0277"></a> Source: p.18 S0277
Original: Setting CoQA DROP QuAC SQuADv2 RACE-h RACE-m Fine-tuned SOTA 90.7a 89.1b 74.4c 93.0d 90.0e 93.1e GPT-3 Zero-Shot 81.5 23.6 41.5 59.5 45.5 58.4 GPT-3 One-Shot 84.0 34.3 43.3 65.4 45.9 57.4 GPT-3 Few-Shot 85.0 36.5 44.3 69.8 46.8 58.1 Table 3.7: Results on reading comprehension tasks.
中文: 设定 CoQA DROP SQuAC SQuADv2 RACE-h RACE-m 微调 SOTA 90.7a 89.1b 74.4c 93.0d 90.0e 93.1e GPT-3 Zero-Shot 81.5 23.6 41.5 59.5 45.5 58.4 GPT-3 One-Shot 84.0 34.3 65.4 45.9 57.4 GPT-3 New-Shot 85.5 44.3 49.8 46.8 58.1 表3.7:阅读理解任务的结果。
<a id="S0278"></a> Source: p.18 S0278
Original: All scores are F1 except results for RACE which report accuracy. a[JZC+19] b[JN20] c[AI19] d[QIA20] e[SPP+19] fine-tuned RoBERTa.
中文: 所有分数均为F1,但RACE报告准确性的结果除外. a [JZC+19] b [JN20] c [AI19] d [QIA20] e [SPP+19] 有微调的RoBERTa.
<a id="S0279"></a> Source: p.18 S0279
Original: PIQA shows relatively shallow scaling with model size and is still over 10% worse than human performance, but GPT-3’s few-shot and even zero-shot result outperform the current state-of-the-art.
中文: PIQA显示相对浅而有模型大小的缩放,仍然比人类的性能差了10%以上,但GPT-3的几发甚至零发的结果超过了目前的最先进水平.
<a id="S0280"></a> Source: p.18 S0280
Original: Our analysis flagged PIQA for a potential data contamination issue (despite hidden test labels), and we therefore conservatively mark the result with an asterisk.
中文: 我们的分析将PIQA标出潜在的数据污染问题(尽管有隐藏的测试标签),因此我们保守地用星号标出结果.
<a id="S0281"></a> Source: p.18 S0281
Original: ARC [CCE+18] is a dataset of multiple-choice questions collected from 3rd to 9th grade science exams.
中文: ARC [CCE+18]是从3至9年级科学考试中收集的多选题数据集.
<a id="S0282"></a> Source: p.18 S0282
Original: On the “Challenge” version of the dataset which has been filtered to questions which simple statistical or information retrieval methods are unable to correctly answer, GPT-3 achieves 51.4% accuracy in the zero-shot setting, 53.2% in the one-shot setting, and 51.5% in the few-shot setting.
中文: 在“挑战”版的数据集中,经过过滤后无法正确回答简单的统计或信息检索方法的问题,GPT-3在零发设定中实现了51.4%的精确度,在一发设定中实现了53.2%,在几发设定中实现了51.5%。
<a id="S0283"></a> Source: p.18 S0283
Original: This is approaching the performance of a fine-tuned RoBERTa baseline (55.9%) from UnifiedQA [KKS+20].
中文: 这接近于统一QA[KKS+20]微调的RoBERTa基线(55.9%)的性能.
<a id="S0284"></a> Source: p.18 S0284
Original: On the “Easy” version of the dataset (questions which either of the mentioned baseline approaches answered correctly), GPT-3 achieves 68.8%, 71.2%, and 70.1% which slightly exceeds a fine-tuned RoBERTa baseline from [KKS+20].
中文: 在数据集的“简单”版本(上述基准方法中任何一个都正确回答的问题)上,GPT-3达到68.8%、71.2%和70.1%,略高于[KKS+20]微调的RoBERTa基线。
<a id="S0285"></a> Source: p.18 S0285
Original: However, both of these results are still much worse than the overall SOTAs achieved by the UnifiedQA which exceeds GPT-3’s few-shot results by 27% on the challenge set and 22% on the easy set.
中文: 然而,这两个结果都比统一质量评估(UnitedQA)实现的总体SOTA要差得多,它比GPT-3的几率结果高出27 % , 比简单的高22 % 。
<a id="S0286"></a> Source: p.18 S0286
Original: On OpenBookQA [MCKS18], GPT-3 improves significantly from zero to few shot settings but is still over 20 points short of the overall SOTA.
中文: 在OpenBookQA [MCKS18]上,GPT-3大幅改进了从零到很少的镜头设置,但仍然比总体SOTA差了20多分.
<a id="S0287"></a> Source: p.18 S0287
Original: GPT-3’s few-shot performance is similar to a fine-tuned BERT Large baseline on the leaderboard.
中文: GPT-3的几发性能类似于头板上经过微调的BERT大型基线.
<a id="S0288"></a> Source: p.18 S0288
Original: Overall, in-context learning with GPT-3 shows mixed results on commonsense reasoning tasks, with only small and inconsistent gains observed in the one and few-shot learning settings for both PIQA and ARC, but a significant improvement is observed on OpenBookQA.
中文: 总体而言,与GPT-3的内语学习在常识推理任务上显示出好坏参半的结果,在PIQA和ARC的一发和几发学习设置中只观察到小而不一致的成绩,但在OpenBookQA上观察到了显著的改进.
<a id="S0289"></a> Source: p.18 S0289
Original: GPT-3 sets SOTA on the new PIQA dataset in all evaluation settings. 3.6 Reading Comprehension Next we evaluate GPT-3 on the task of reading comprehension.
中文: GPT-3在所有评价设置中将SOTA设置在新的PIQA数据集上. 3.6 阅读理解 接着我们评估GPT-3的阅读理解任务.
<a id="S0290"></a> Source: p.18 S0290
Original: We use a suite of 5 datasets including abstractive, multiple choice, and span based answer formats in both dialog and single question settings.
中文: 我们在对话框和单个问题设置中使用5套数据集,包括抽象的,多重的选择,以及跨度的答案格式.
<a id="S0291"></a> Source: p.18 S0291
Original: We observe a wide spread in GPT-3’s performance across these datasets suggestive of varying capability with different answer formats.
中文: 我们观察到,GPT-3的性能在这些数据组中分布很广,显示能力不同,答案格式不同。
<a id="S0292"></a> Source: p.18 S0292
Original: In general we observe GPT-3 is on par with initial baselines and early results trained using contextual representations on each respective dataset.
中文: 总体而言,我们观测到的GPT-3与初步基线和早期结果相同,利用每个数据集的背景说明进行了培训。
<a id="S0293"></a> Source: p.18 S0293
Original: GPT-3 performs best (within 3 points of the human baseline) on CoQA [RCM19] a free-form conversational dataset and performs worst (13 F1 below an ELMo baseline) on QuAC [CHI+18] a dataset which requires modeling structured dialog acts and answer span selections of teacher-student interactions.
中文: GPT-3在CoQA[RCM19]上(在人类基线的3个点之内)最出色地执行自由形式的对话数据集,并在QuAC[CHI+18]上执行最差的(ELMo基线下13个F1)数据集,这需要建模结构化的对话行为并解答师生互动的跨度选择.
<a id="S0294"></a> Source: p.18 S0294
Original: On DROP [DWD+19], a dataset testing discrete reasoning and numeracy in the context of reading comprehension, GPT-3 in a few-shot setting outperforms the fine-tuned BERT baseline from the original paper but is still well below both human performance and state-of-the-art approaches which augment neural networks with symbolic systems [RLL+19].
中文: 在DROP[DWD+19]上,一个数据集在读取理解的背景下测试离散推理和算术,GPT-3在几发镜头中列出了从原始论文中微调的BERT基线,但仍然远远低于人类性能和最先进的方法,这些方法用符号系统增强神经网络[RLL+19].
<a id="S0295"></a> Source: p.18 S0295
Original: On SQuAD 2.0 [RJL18], GPT-3 demonstrates its few-shot learning capabilities, improving by almost 10 F1 (to 69.8) compared to a zero-shot setting.
中文: 在SQuAD 2.0 [RJL18]上,GPT-3展示了其几发的学习能力,与零发设定相比,提高了近10个F1(到69.8).
<a id="S0296"></a> Source: p.18 S0296
Original: This allows it to slightly outperform the best fine-tuned result in the original paper.
中文: 这使得它能够略微地超过原始论文中最好的微调结果.
<a id="S0297"></a> Source: p.18 S0297
Original: On RACE [LXL+17], a multiple choice dataset of middle school and high school english examinations, GPT-3 performs relatively weakly and is only competitive with the earliest work utilizing contextual representations and is still 45% behind SOTA. 3.7 SuperGLUE In order to better aggregate results on NLP tasks and compare to popular models such as BERT and RoBERTa in a more systematic way, we also evaluate GPT-3 on a standardized collection of datasets, the SuperGLUE benchmark [WPN+19] [WPN+19] [CLC+19] [DMST19] [RBG11] [KCR+18] [ZLL+18] [DGM06] [BHDD+06] [GMDD07] [BDD+09] [PCC18] [PHR+18].
中文: 在RACE[LXL+17]这个初中和高中英语考试的多选择数据集上,GPT-3的成绩相对薄弱,仅能利用背景表现与最早的工作相竞争,仍然落后于SOTA45%. 3.7 超级玻璃 为了更好地汇总NLP任务的结果并更系统地与BERT和RoBERTA等受欢迎的模型进行比较,我们还对GPT-3进行了标准化数据集采集,超级GLUE基准[WPN+19] [WPN+19] [CLC+19] [RBG11] [KCR+18] [ZLL+18] [DGM06] [BHDD+06] [GMDD07] [BDD+09] [PCC18] [PHR+18].
<a id="S0298"></a> Source: p.18 S0298
Original: GPT-3’s test-set performance on the SuperGLUE dataset is shown in Table 3.8.
中文: GPT-3在SuperGLUE数据集上的测试集性能见表3.8.
<a id="S0299"></a> Source: p.18 S0299
Original: In the few-shot setting, we used 32 examples for all tasks, sampled randomly from the training set.
中文: 在几发设定中,我们对所有任务都使用了32个实例,从训练集中随机抽取.
<a id="S0300"></a> Source: p.19 S0300
Original: Figure 3.7: GPT-3 results on CoQA reading comprehension task.
中文: 图3.7:COQA读取理解任务GPT-3结果.
<a id="S0301"></a> Source: p.19 S0301
Original: GPT-3 175B achieves 85 F1 in the few-shot setting, only a few points behind measured human performance and state-of-the-art fine-tuned models.
中文: GPT-3 175B在几发设定中实现了85 F1,仅落后了几分测量了人类性能和最先进的微调模型.
<a id="S0302"></a> Source: p.19 S0302
Original: Zero-shot and one-shot performance is a few points behind, with the gains to few-shot being largest for bigger models.
中文: 零发和一发相的性能落后了几分,对更大型的模型来说,少发相的收获最大.
<a id="S0303"></a> Source: p.19 S0303
Original: SuperGLUE BoolQ CB CB COPA RTE Average Accuracy Accuracy F1 Accuracy Accuracy Fine-tuned SOTA 89.0 91.0 96.9 93.9 94.8 92.5 Fine-tuned BERT-Large 69.0 77.4 83.6 75.7 70.6 71.7 GPT-3 Few-Shot 71.8 76.4 75.6 52.0 92.0 69.0 WiC WSC MultiRC MultiRC ReCoRD ReCoRD Accuracy Accuracy Accuracy F1a Accuracy F1 Fine-tuned SOTA 76.1 93.8 62.3 88.2 92.5 93.3 Fine-tuned BERT-Large 69.6 64.6 24.1 70.0 71.3 72.0 GPT-3 Few-Shot 49.4 80.1 30.5 75.4 90.2 91.1 Table 3.8: Performance of GPT-3 on SuperGLUE compared to fine-tuned baselines and SOTA.
中文: SuperGLUE BoulQ CB CB COPA RTE 平均精确度F1 精确度F1 精确度精细度SOTA 89.0 91.0 96.9 93.9 精细度BERT-LAG 69.8 细微度BERT-LAG 69.0 77.4 77.7 70.6 71.7 GPT-3 稀有-Shot 71.8 76.4 75.6 52.0 69.0 69.0 WIC WIC WSC 多分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司 微分公司
<a id="S0304"></a> Source: p.19 S0304
Original: All results are reported on the test set.
中文: 所有结果均上报测试组.
<a id="S0305"></a> Source: p.19 S0305
Original: GPT-3 few-shot is given a total of 32 examples within the context of each task and performs no gradient updates. 19
中文: GPT-3在每项任务中总共得到32个实例,没有进行梯度更新。 第 19 条
<a id="S0306"></a> Source: p.20 S0306
Original: Figure 3.8: Performance on SuperGLUE increases with model size and number of examples in context. A value of K = 32 means that our model was shown 32 examples per task, for 256 examples total divided across the 8 tasks in SuperGLUE.
中文: 图3.8:超级GLUE上的性能随上下文中的模型大小和实例数量而增加. 值为K=32,意味着我们的模型被显示为每个任务32个实例,其中256个实例在SuperGLUE的8个任务中共分出.
<a id="S0307"></a> Source: p.20 S0307
Original: We report GPT-3 values on the dev set, so our numbers are not directly comparable to the dotted reference lines (our test set results are in Table 3.8).
中文: 我们在dev集上报告GPT-3值,因此我们的数字不能直接与被点入的参考行相比(我们的测试集结果见表3.8).
<a id="S0308"></a> Source: p.20 S0308
Original: The BERT-Large reference model was fine-tuned on the SuperGLUE training set (125K examples), whereas BERT++ was first fine-tuned on MultiNLI (392K examples) and SWAG (113K examples) before further fine-tuning on the SuperGLUE training set (for a total of 630K fine-tuning examples).
中文: BERT-Large的参考模型在SuperGLUE培训集(125K实例)上进行了微调,而BERT++首先在MultiNLI(392K实例)和SWAG(113K实例)上进行了微调,然后在SuperGLUE培训集(共计630K实例)上进行了进一步微调.
<a id="S0309"></a> Source: p.20 S0309
Original: We find the difference in performance between the BERT-Large and BERT++ to be roughly equivalent to the difference between GPT-3 with one example per context versus eight examples per context. and MultiRC, we sampled a new set of examples to use in the context for each problem.
中文: 我们认为,BERT-Large和BERT++之间的性能差异大致相当于GPT-3与每个上下文一个例子和每个上下文八个例子之间的差别。 和MultiRC,我们抽取了一组新的例子,用于每个问题的上下文。
<a id="S0310"></a> Source: p.20 S0310
Original: For WSC and MultiRC, we used the same set of randomly drawn examples from the training set as context for all of the problems we evaluated.
中文: 对于世界儿童问题首脑会议和多资源中心,我们使用从培训中随机抽取的同一套例子,作为我们评价的所有问题的背景。
<a id="S0311"></a> Source: p.20 S0311
Original: We observe a wide range in GPT-3’s performance across tasks.
中文: 我们观察到GPT-3在各种任务方面的表现。
<a id="S0312"></a> Source: p.20 S0312
Original: On COPA and ReCoRD GPT-3 achieves near-SOTA performance in the one-shot and few-shot settings, with COPA falling only a couple points short and achieving second place on the leaderboard, where first place is held by a fine-tuned 11 billion parameter model (T5).
中文: 在COPA和ReCORD GPT-3上,在一发和几发的设定中实现了近SOTA的性能,而COPA只降了几分短,并在领导板上取得了第二名,第一名由微调出110亿个参数模型(T5)来掌握.
<a id="S0313"></a> Source: p.20 S0313
Original: On WSC, performance is still relatively strong, achieving 80.1% in the few-shot setting (note that GPT-3 achieves 88.6% on the original Winograd dataset as described in Section 3.4).
中文: 在WSC上,性能仍然相对强劲,在几发设定中达到80.1%(注意GPT-3在第3.4节所描述的Winograd原始数据集上达到88.6%).
<a id="S0314"></a> Source: p.20 S0314
Original: On BoolQ, MultiRC, and RTE, performance is reasonable, roughly matching that of a fine-tuned BERT-Large.
中文: 在BoulQ,MultiRC,和RTE上,性能合理,大致匹配了微调的BERT-Large.
<a id="S0315"></a> Source: p.20 S0315
Original: On CB, we see signs of life at 75.6% in the few-shot setting.
中文: 在CB上,我们看到生命迹象 在75.6%的拍摄环境。
<a id="S0316"></a> Source: p.20 S0316
Original: WiC is a notable weak spot with few-shot performance at 49.4% (at random chance).
中文: WIC是一个显著的弱点,少发性能为49.4%(随机).
<a id="S0317"></a> Source: p.20 S0317
Original: We tried a number of different phrasings and formulations for WiC (which involves determining if a word is being used with the same meaning in two sentences), none of which was able to achieve strong performance.
中文: 我们尝试了WIC的一些不同的用词和配方(这涉及到确定一个词是否在两句中被使用,其含义相同),其中没有一个能够实现强效.
<a id="S0318"></a> Source: p.20 S0318
Original: This hints at a phenomenon that will become clearer in the next section (which discusses the ANLI benchmark) – GPT-3 appears to be weak in the few-shot or one-shot setting at some tasks that involve comparing two sentences or snippets, for example whether a word is used the same way in two sentences (WiC), whether one sentence is a paraphrase of another, or whether one sentence implies another.
中文: 这暗示了一个在下一节(讨论ANLI基准)会更清晰的现象——GPT-3在一些涉及比较两个句子或片段的任务的几发或一发设定中显得很弱,例如一个词在两个句子(WIC)中是否同样被使用,一个句子是否是另一个句子的解说,或者一个句子是否意味着另一个句子.
<a id="S0319"></a> Source: p.20 S0319
Original: This could also explain the comparatively low scores for RTE and CB, which also follow this format.
中文: 这也可以解释RTE和CB的分数相对较低的原因,它们也遵循了这个格式.
<a id="S0320"></a> Source: p.20 S0320
Original: Despite these weaknesses, GPT-3 still outperforms a fine-tuned BERT-large on four of eight tasks and on two tasks GPT-3 is close to the state-of-the-art held by a fine-tuned 11 billion parameter model.
中文: 尽管存在这些弱点,GPT-3在8个任务中的4个任务上仍然表现优于微调的BERT大项,而2个任务上GPT-3则接近于微调的110亿参数模型所保持的先进水平.
<a id="S0321"></a> Source: p.20 S0321
Original: Finally, we note that the few-shot SuperGLUE score steadily improves with both model size and with number of examples in the context showing increasing benefits from in-context learning (Figure 3.8).
中文: 最后,我们注意到,随着模型大小和实例数量的增长,少数镜头的SuperGLUE得分稳步提高,显示从内容内学习获得的惠益越来越大(图3.8)。
<a id="S0322"></a> Source: p.20 S0322
Original: We scale K up to 32 examples per task, after which point additional examples will not reliably fit into our context.
中文: 我们把K扩大到每个任务32个例子,然后,其他例子将不可靠地适合我们的背景。
<a id="S0323"></a> Source: p.20 S0323
Original: When sweeping over values of K, we find that GPT-3 requires less than eight total examples per task to outperform a fine-tuned BERT-Large on overall SuperGLUE score. 3.8 NLI Natural Language Inference (NLI) [Fyo00] concerns the ability to understand the relationship between two sentences.
中文: 在浏览K的值时,我们发现,GPT-3每次任务需要不到8个总例,才能在总体SuperGLUE分数上超越一个微调的BERT-Large. 3.8 NLI自然语言推论(NLI)[Fyo00]涉及理解两句间关系的能力.
<a id="S0324"></a> Source: p.20 S0324
Original: In practice, this task is usually structured as a two or three class classification problem where the model classifies 20
中文: 在实际操作中,这个任务的结构通常为二到三个分类问题,模型将20分类.
<a id="S0325"></a> Source: p.21 S0325
Original: Figure 3.9: Performance of GPT-3 on ANLI Round 3.
中文: 图3.9:GPT-3在ANLI 第3轮上的表现.
<a id="S0326"></a> Source: p.21 S0326
Original: Results are on the dev-set, which has only 1500 examples and therefore has high variance (we estimate a standard deviation of 1.2%).
中文: 结果出现在dev-set上,它仅有1500个实例,因此差异很大(我们估计标准偏差为1.2%).
<a id="S0327"></a> Source: p.21 S0327
Original: We find that smaller models hover around random chance, while few-shot GPT-3 175B closes almost half the gap from random chance to SOTA.
中文: 我们发现,较小的模型徘徊在随机机会周围,而几发GPT-3 175B则关闭了从随机机会到SOTA的几乎一半差距.
<a id="S0328"></a> Source: p.21 S0328
Original: Results for ANLI rounds 1 and 2 are shown in the appendix. whether the second sentence logically follows from the first, contradicts the first sentence, or is possibly true (neutral).
中文: ANLI 第1和2轮的结果载于附录。 第二句从逻辑上是否与第一句相接,与第一句相矛盾,或者可能是真实的(中性).
<a id="S0329"></a> Source: p.21 S0329
Original: SuperGLUE includes an NLI dataset, RTE, which evaluates the binary version of the task.
中文: SuperGLUE包括一个NLI数据集RTE,用于评价任务的二进制版本.
<a id="S0330"></a> Source: p.21 S0330
Original: On RTE, only the largest version of GPT-3 performs convincingly better than random (56%) in any evaluation setting, but in a few-shot setting GPT-3 performs similarly to a single-task fine-tuned BERT Large.
中文: 在 RTE 上,只有最大的版本的GPT-3在任何评价环境中的性能都比随机性(56%)有说服力的好,但在几发设定中GPT-3的性能与单任务微调的BERT大相类似.
<a id="S0331"></a> Source: p.21 S0331
Original: We also evaluate on the recently introduced Adversarial Natural Language Inference (ANLI) dataset [NWD+19].
中文: 我们还评估最近推出的“横向自然语言推论”数据集[NWD+19]。
<a id="S0332"></a> Source: p.21 S0332
Original: ANLI is a difficult dataset employing a series of adversarially mined natural language inference questions in three rounds (R1, R2, and R3).
中文: ANLI是一个困难的数据集,在三轮(R1,R2和R3)中使用一系列相对抗的被开采出自然语言推断问题.
<a id="S0333"></a> Source: p.21 S0333
Original: Similar to RTE, all of our models smaller than GPT-3 perform at almost exactly random chance on ANLI, even in the few-shot setting (∼ 33%), whereas GPT-3 itself shows signs of life on Round 3.
中文: 与RTE类似,我们所有小于GPT-3的型号在ANLI上几乎完全随机的性能,甚至在几发相(XXXX 33%)的设定中也是如此,而GPT-3本身在第三回合上表现出了生命的迹象.
<a id="S0334"></a> Source: p.21 S0334
Original: Results for ANLI R3 are highlighted in Figure 3.9 and full results for all rounds can be found in Appendix H.
中文: ANLI R3的结果在图3.9中突出显示,所有回合的全部结果见附录H。
<a id="S0335"></a> Source: p.21 S0335
Original: These results on both RTE and ANLI suggest that NLI is still a very difficult task for language models and they are only just beginning to show signs of progress. 3.9 Synthetic and Qualitative Tasks One way to probe GPT-3’s range of abilities in the few-shot (or zero- and one-shot) setting is to give it tasks which require it to perform simple on-the-fly computational reasoning, recognize a novel pattern that is unlikely to have occurred in training, or adapt quickly to an unusual task.
中文: RTE和ANLI的这些结果表明,NLI对于语言模型来说仍然是一个非常困难的任务,它们才刚刚开始显示出进步的迹象. 3.9 合成和定性任务 在几发(或零发和一发)设定中探测出GPT-3能力范围的一种方法是赋予它任务,这些任务要求它进行简单的机上计算推理,识别在训练中不太可能发生的新颖模式,或者迅速适应不寻常的任务.
<a id="S0336"></a> Source: p.21 S0336
Original: We devise several tasks to test this class of abilities.
中文: 我们设计了几项任务来测试这种能力。
<a id="S0337"></a> Source: p.21 S0337
Original: First, we test GPT-3’s ability to perform arithmetic.
中文: 首先,我们测试GPT-3的算术能力.
<a id="S0338"></a> Source: p.21 S0338
Original: Second, we create several tasks that involve rearranging or unscrambling the letters in a word, tasks which are unlikely to have been exactly seen during training.
中文: 第二,我们创造了几项任务,涉及一个字字母的重排或拼接,这些任务不太可能在训练期间被准确看到.
<a id="S0339"></a> Source: p.21 S0339
Original: Third, we test GPT-3’s ability to solve SAT-style analogy problems few-shot.
中文: 第三,我们测试GPT-3解决SAT类类类类类比问题的能力,几发子弹.
<a id="S0340"></a> Source: p.21 S0340
Original: Finally, we test GPT-3 on several qualitative tasks, including using new words in a sentence, correcting English grammar, and news article generation.
中文: 最后,我们测试GPT-3的几个定性任务,包括在句子中用新词,修正英语语法,以及新闻文章生成.
<a id="S0341"></a> Source: p.21 S0341
Original: We will release the synthetic datasets with the hope of stimulating further study of test-time behavior of language models. 3.9.1 Arithmetic To test GPT-3’s ability to perform simple arithmetic operations without task-specific training, we developed a small battery of 10 tests that involve asking GPT-3 a simple arithmetic problem in natural language: • 2 digit addition (2D+) – The model is asked to add two integers sampled uniformly from [0, 100), phrased in the form of a question, e.g. “Q: What is 48 plus 76? A: 124.” • 2 digit subtraction (2D-) – The model is asked to subtract two integers sampled uniformly from [0, 100); the answer may be negative.
中文: 我们将发布合成数据集,希望激发对语言模型测试时间行为的进一步研究. 3.9.1 算术 为了测试GPT-3在没有任务特定训练的情况下进行简单的算术操作的能力,我们开发了由10个测试组成的小电池,这些测试涉及要求GPT-3在自然语言中一个简单的算术问题: ^ 2个数字添加(2D+) – 该模型被要求从 [0,100] 中统一地添加两个被取出样本的整数,用一个问题的形式标出,例如“Q:48+76是什么?A:124” → 2个数字减数(2D-) – 该模型被要求从 [0,100] 中统一地减去两个被取出样本的整数;答案可能是否定的.
<a id="S0342"></a> Source: p.21 S0342
Original: Example: “Q: What is 34 minus 53? A: -19”. • 3 digit addition (3D+) – Same as 2 digit addition, except numbers are uniformly sampled from [0, 1000). 21
中文: 示例:“问题:34减53是什么? A:-19” ^ 3位数字加法(3D+)-与2位数字加法相同,但从[0,1000]统一抽取数字. 21国
<a id="S0343"></a> Source: p.22 S0343
Original: Figure 3.10: Results on all 10 arithmetic tasks in the few-shot settings for models of different sizes.
中文: 图3.10:大小不同的模型在几发设置中所有10个算术任务的结果。
<a id="S0344"></a> Source: p.22 S0344
Original: There is a significant jump from the second largest model (GPT-3 13B) to the largest model (GPT-3 175), with the latter being able to reliably accurate 2 digit arithmetic, usually accurate 3 digit arithmetic, and correct answers a significant fraction of the time on 4-5 digit arithmetic, 2 digit multiplication, and compound operations.
中文: 从第二大型号(GPT-3 13B)到最大型号(GPT-3 175)都有了显著的跳跃,后者能够可靠地准确2位数算术,通常准确3位数算术,并在4-5位数算术,2位数乘法和复合操作上正确解答了相当一部分时间.
<a id="S0345"></a> Source: p.22 S0345
Original: Results for one-shot and zero-shot are shown in the appendix. • 3 digit subtraction (3D-) – Same as 2 digit subtraction, except numbers are uniformly sampled from [0, 1000). • 4 digit addition (4D+) – Same as 3 digit addition, except uniformly sampled from [0, 10000). • 4 digit subtraction (4D-) – Same as 3 digit subtraction, except uniformly sampled from [0, 10000). • 5 digit addition (5D+) – Same as 3 digit addition, except uniformly sampled from [0, 100000). • 5 digit subtraction (5D-) – Same as 3 digit subtraction, except uniformly sampled from [0, 100000). • 2 digit multiplication (2Dx) – The model is asked to multiply two integers sampled uniformly from [0, 100), e.g. “Q: What is 24 times 42? A: 1008”. • One-digit composite (1DC) – The model is asked to perform a composite operation on three 1 digit numbers, with parentheses around the last two.
中文: 一发和零发的结果载于附录。 • 3个数字减去(3D-)——与2个数字减去相同,但数字从[0,1000]统一取样。 • 4个数字增加(4D+) – 与3个数字增加相同,但从[0,1000]统一抽取. • 4位数减去(4D-)——与3位数减去相同,但从[0,1000]中统一抽取的除外. • 5个数字加(5D+) – 与3个数字加数相同,但从[0,1000]统一抽取. ^ 5位数减去(5D-)-与3位数减去相同,但从[0,1000]中统一抽取. • 2位数乘法(2Dx) – 该模型被要求从[0,100]以统一方式取取出两个整数相乘,例如"Q: 24乘以42是什么?A:1008” ^ 1位数复合(1DC) – 该型号被要求在3个1位数上进行复合操作,后两个上方为括号.
<a id="S0346"></a> Source: p.22 S0346
Original: For example, “Q: What is 6+(4*8)? A: 38”.
中文: 例如,“问题:6+(4*8)是什么?”
<a id="S0347"></a> Source: p.22 S0347
Original: The three 1 digit numbers are selected uniformly on [0, 10) and the operations are selected uniformly from {+,-,*}.
中文: 3个1位数的数字在 [0, 10] 上统一选择, 操作从 {+,-, *} 上统一选择.
<a id="S0348"></a> Source: p.22 S0348
Original: In all 10 tasks the model must generate the correct answer exactly.
中文: 在所有10项任务中,模型必须产生准确的正确答案。
<a id="S0349"></a> Source: p.22 S0349
Original: For each task we generate a dataset of 2,000 random instances of the task and evaluate all models on those instances.
中文: 对于每项任务,我们生成2 000个随机任务实例的数据集,并评价这些实例的所有模型。
<a id="S0350"></a> Source: p.22 S0350
Original: First we evaluate GPT-3 in the few-shot setting, for which results are shown in Figure 3.10.
中文: 首先,我们对GPT-3进行几发式评价,结果见图3.10。
<a id="S0351"></a> Source: p.22 S0351
Original: On addition and subtraction, GPT-3 displays strong proficiency when the number of digits is small, achieving 100% accuracy on 2 digit addition, 98.9% at 2 digit subtraction, 80.2% at 3 digit addition, and 94.2% at 3-digit subtraction.
中文: 在增减上,GPT-3在位数小时表现出强烈的熟练度,2位数加时达到100%的精度,2位数加后达到98.9%,3位数加后达到80.2%,3位数加后达到94.2%.
<a id="S0352"></a> Source: p.22 S0352
Original: Performance decreases as the number of digits increases, but GPT-3 still achieves 25-26% accuracy on four digit operations and 9-10% accuracy on five digit operations, suggesting at least some capacity to generalize to larger numbers of digits.
中文: 随着数位数的增加,性能会下降,但GPT-3在4位数操作上仍能达到25-26%的精度,在5位数操作上仍能达到9-10%的精度,这至少表明有一定能力概括到更多位数.
<a id="S0353"></a> Source: p.22 S0353
Original: GPT-3 also achieves 29.2% accuracy at 2 digit multiplication, an especially computationally intensive operation.
中文: GPT-3在2位数相乘时也达到29.2%的精度,是一种特别的计算密集的操作.
<a id="S0354"></a> Source: p.22 S0354
Original: Finally, GPT-3 achieves 21.3% accuracy at single digit combined operations (for example, 9*(7+5)), suggesting that it has some robustness beyond just single operations.
中文: 最后,GPT-3在单位组合操作(例如9*(7+5))中实现了21.3%的精度,说明它除了单位操作之外还有一定的强性.
<a id="S0355"></a> Source: p.22 S0355
Original: As Figure 3.10 makes clear, small models do poorly on all of these tasks – even the 13 billion parameter model (the second largest after the 175 billion full GPT-3) can solve 2 digit addition and subtraction only half the time, and all other operations less than 10% of the time.
中文: 如图3.10所说明的,小型模型在所有这些任务上都表现不佳——即使是130亿个参数模型(仅次于175亿个完整的GBT-3)也能解决2位数的增减问题,所有其他操作时间都不到10%.
<a id="S0356"></a> Source: p.22 S0356
Original: One-shot and zero-shot performance are somewhat degraded relative to few-shot performance, suggesting that adaptation to the task (or at the very least recognition of the task) is important to performing these computations correctly.
中文: 一发和零发的性能相对于少发的性能有些退化,这表明适应任务(或至少承认任务)对于正确进行这些计算很重要.
<a id="S0357"></a> Source: p.22 S0357
Original: Nevertheless, one-shot performance is still quite strong, and even zero-shot performance of the full GPT-3 significantly 22
中文: 尽管如此,一发的性能仍然相当强劲,甚至完全的GPT-3的零发的性能也显著地22.
<a id="S0358"></a> Source: p.23 S0358
Original: Setting 2D+ 2D- 3D+ 3D- 4D+ 4D- 5D+ 5D- 2Dx 1DC GPT-3 Zero-shot 76.9 58.0 34.2 48.3 4.0 7.5 0.7 0.8 19.8 9.8 GPT-3 One-shot 99.6 86.4 65.5 78.7 14.0 14.0 3.5 3.8 27.4 14.3 GPT-3 Few-shot 100.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3 Table 3.9: Results on basic arithmetic tasks for GPT-3 175B. {2,3,4,5}D{+,-} is 2, 3, 4, and 5 digit addition or subtraction, 2Dx is 2 digit multiplication. 1DC is 1 digit composite operations.
中文: 设置2D+2D-3D+3D-4D+4D-5D+5D-2Dx 1DC GPT-3 零射出76.9 58.0 34.2 48.3 4.0 7.5 7.7 19.8 9.8 GPT-3 1发出99.6 86.4 65.5 14.7 14.0 3.5 27.4 14.3 GPT-3 少射出1000.0 98.9 80.4 94.2 25.5 26.8 9.3 9.9 29.2 21.3 表3.9: GPT-3 175B {2,3,4,5}D{+,-}为2、3,4和5个数字增减,2Dx为2个数字乘法. 1DC是1位数的复合操作.
<a id="S0359"></a> Source: p.23 S0359
Original: Results become progressively stronger moving from the zero-shot to one-shot to few-shot setting, but even the zero-shot shows significant arithmetic abilities.
中文: 结果逐渐变得更强,从零发到一发到几发的设定,但即使是零发也显示出重大的算术能力.
<a id="S0360"></a> Source: p.23 S0360
Original: Setting CL A1 A2 RI RW GPT-3 Zero-shot 3.66 2.28 8.91 8.26 0.09 GPT-3 One-shot 21.7 8.62 25.9 45.4 0.48 GPT-3 Few-shot 37.9 15.1 39.7 67.2 0.44 Table 3.10: GPT-3 175B performance on various word unscrambling and word manipulation tasks, in zero-, one-, and few-shot settings.
中文: 设置 CL A1 A2 RW GPT-3 零发 3.66 2.28 8.91 8.26 0.09 GPT-3 单发 21.7 8.62 25.9 15.4 048 GPT-3 几发 37.9 15.1 39.7 67.2 0.44 表3.10: GPT-3 175B在各种单词无摇摆和单词操纵任务上的性能,在零发,一发和少发等场合.
<a id="S0361"></a> Source: p.23 S0361
Original: CL is “cycle letters in word”, A1 is anagrams of but the first and last letters, A2 is anagrams of all but the first and last two letters, RI is “Random insertion in word”, RW is “reversed words”. outperforms few-shot learning for all smaller models.
中文: CL是“循环字母以字表示”,A1是首字母和末字母的动画,A2是除首字母和后两个字母外所有字母的动画,RI是“Random in word”,RW是“反向词”。 超过所有较小模型的几发学习。
<a id="S0362"></a> Source: p.23 S0362
Original: All three settings for the full GPT-3 are shown in Table 3.9, and model capacity scaling for all three settings is shown in Appendix H.
中文: 完整的GPT-3的所有三种设置都列于表3.9,所有三种设置的模型容量缩放则列于附录H.
<a id="S0363"></a> Source: p.23 S0363
Original: To spot-check whether the model is simply memorizing specific arithmetic problems, we took the 3-digit arithmetic problems in our test set and searched for them in our training data in both the forms "<NUM1> + <NUM2> =" and "<NUM1> plus <NUM2>".
中文: 为了当场检查该模型是否只是计算出具体的算术问题,我们在测试组中选择了3位数的算术问题,并在我们的培训数据中以"<NUM1> +<NUM2> ="和"<NUM1> +""NUM2"两种形式查找.
<a id="S0364"></a> Source: p.23 S0364
Original: Out of 2,000 addition problems we found only 17 matches (0.8%) and out of 2,000 subtraction problems we found only 2 matches (0.1%), suggesting that only a trivial fraction of the correct answers could have been memorized.
中文: 在2,000个附加问题中,我们只发现17个匹配(0.8%),在2,000个减法问题中,我们只发现2个匹配(0.1%),这表明只有一小部分正确答案可以被记住。
<a id="S0365"></a> Source: p.23 S0365
Original: In addition, inspection of incorrect answers reveals that the model often makes mistakes such as not carrying a “1”, suggesting it is actually attempting to perform the relevant computation rather than memorizing a table.
中文: 此外,检查不正确的答案表明,模型常常出现错误,例如没有“1”字,表明它实际上试图进行相关的计算,而不是对一个表格进行记忆。
<a id="S0366"></a> Source: p.23 S0366
Original: Overall, GPT-3 displays reasonable proficiency at moderately complex arithmetic in few-shot, one-shot, and even zero-shot settings. 3.9.2 Word Scrambling and Manipulation Tasks To test GPT-3’s ability to learn novel symbolic manipulations from a few examples, we designed a small battery of 5 “character manipulation” tasks.
中文: 总体而言,GPT-3在几发一发甚至零发的设置中,在中度复杂的算术上表现出了合理的熟练度. 3.9.2 文字拼接和操纵任务 为了测试GPT-3从几个例子中学习新颖的象征性操控的能力,我们设计了由5个"特征操控"任务组成的小电池.
<a id="S0367"></a> Source: p.23 S0367
Original: Each task involves giving the model a word distorted by some combination of scrambling, addition, or deletion of characters, and asking it to recover the original word.
中文: 每项任务都涉及给模型一个被一些拼接,添加,或删除字符的组合所扭曲的单词,并请求其恢复原词.
<a id="S0368"></a> Source: p.23 S0368
Original: The 5 tasks are: • Cycle letters in word (CL) – The model is given a word with its letters cycled, then the “=” symbol, and is expected to generate the original word.
中文: 5个任务有: ^ 以单词(CL)为单位的循环字母 – 该模型被赋予一个有其字母循环的单词,再取"="的符号,预计生成正词.
<a id="S0369"></a> Source: p.23 S0369
Original: For example, it might be given “lyinevitab” and should output “inevitably”. • Anagrams of all but first and last characters (A1) – The model is given a word where every letter except the first and last have been scrambled randomly, and must output the original word.
中文: 例如,它可被赋予“lyinevitab”,并应“无可缺量地”输出。 ^ Anagrams of all but first and last phone (A1) – 该模型被赋予一个单词,其中除了第一个和最后一个字母之外,每个字母都被随机地拼接,必须输出出原词.
<a id="S0370"></a> Source: p.23 S0370
Original: Example: criroptuon = corruption. • Anagrams of all but first and last 2 characters (A2) – The model is given a word where every letter except the first 2 and last 2 have been scrambled randomly, and must recover the original word.
中文: 示例: criroptuon=腐败. ^ Anagrams of all but first and last 2 phone (A2) – 该模型被赋予一个单词,其中除了前2和后2个字母之外,每个字母都被随机地拼接出来,必须恢复原词.
<a id="S0371"></a> Source: p.23 S0371
Original: Example: opoepnnt → opponent. • Random insertion in word (RI) – A random punctuation or space character is inserted between each letter of a word, and the model must output the original word.
中文: 例:opoepnnt → 对手. • 任意插入单词(RI) – 在一个单词的每个字母之间插入一个随机的平分或空格字符,模型必须输出出原词.
<a id="S0372"></a> Source: p.23 S0372
Original: Example: s.u!c/c!e.s s i/o/n = succession. • Reversed words (RW) – The model is given a word spelled backwards, and must output the original word.
中文: 例: s.u!c/c!e.s s i/o/n=继承. ^ 倒行字 (RW) – 该模型被赋予一个倒行字,必须输出出原词.
<a id="S0373"></a> Source: p.23 S0373
Original: For each task we generate 10,000 examples, which we chose to be the top 10,000 most frequent words as measured by [Nor09] of length more than 4 characters and less than 15 characters.
中文: 对于每项任务,我们产生一万个例子,我们选择了以长度超过4个字符和少于15个字符的[Nor09]测量的1万个最常用词的排行榜.
<a id="S0374"></a> Source: p.23 S0374
Original: The few-shot results are shown in Figure 3.11.
中文: 图3.11显示了少数拍摄结果。
<a id="S0375"></a> Source: p.23 S0375
Original: Task performance tends to grow smoothly with model size, with the full GPT-3 model achieving 66.9% on removing 23
中文: 任务性能往往随模型大小而平稳地增长,完整的GPT-3模型在去除23后达到66.9%.
<a id="S0376"></a> Source: p.24 S0376
Original: Figure 3.11: Few-shot performance on the five word scrambling tasks for different sizes of model.
中文: 图3.11:5个单词拼接任务不同型号的少有镜头性能.
<a id="S0377"></a> Source: p.24 S0377
Original: There is generally smooth improvement with model size although the random insertion task shows an upward slope of improvement with the 175B model solving the task the majority of the time.
中文: 模型大小一般有平滑的改进,虽然随机插入任务显示出一个上坡的改进,175B模型解决了大部分时间的任务.
<a id="S0378"></a> Source: p.24 S0378
Original: Scaling of one-shot and zero-shot performance is shown in the appendix.
中文: 附录显示一发和零发的性能。
<a id="S0379"></a> Source: p.24 S0379
Original: All tasks are done with K = 100. random insertions, 38.6% on cycling letters, 40.2% on the easier anagram task, and 15.1% on the more difficult anagram task (where only the first and last letters are held fixed).
中文: 所有任务都用K=100完成. 随机插入,循环字母为38.6%,较简单的aagram任务为40.2%,较困难的agram任务为15.1%(只有第一个和最后一个字母被固定).
<a id="S0380"></a> Source: p.24 S0380
Original: None of the models can reverse the letters in a word.
中文: 任何模型都无法将字母反转为一字.
<a id="S0381"></a> Source: p.24 S0381
Original: In the one-shot setting, performance is significantly weaker (dropping by half or more), and in the zero-shot setting the model can rarely perform any of the tasks (Table 3.10).
中文: 在一发设定中,性能明显弱(投出一半或更多),在零发设定中,模型很少能完成任何任务(表3.10).
<a id="S0382"></a> Source: p.24 S0382
Original: This suggests that the model really does appear to learn these tasks at test time, as the model cannot perform them zero-shot and their artificial nature makes them unlikely to appear in the pre-training data (although we cannot confirm this with certainty).
中文: 这表明该模型在测试时确实确实学习了这些任务,因为该模型无法进行零发并它们的人工性质使得它们不太可能出现在训练前的数据中(虽然我们无法肯定地确认这一点).
<a id="S0383"></a> Source: p.24 S0383
Original: We can further quantify performance by plotting “in-context learning curves”, which show task performance as a function of the number of in-context examples.
中文: 我们可以通过绘制“内容内学习曲线”来进一步量化绩效,该曲线显示任务性能与内容内实例数量成函数。
<a id="S0384"></a> Source: p.24 S0384
Original: We show in-context learning curves for the Symbol Insertion task in Figure 1.2.
中文: 我们在图1.2中为符号插入任务显示内容内的学习曲线。
<a id="S0385"></a> Source: p.24 S0385
Original: We can see that larger models are able to make increasingly effective use of in-context information, including both task examples and natural language task descriptions.
中文: 我们可以看到,更大的模型能够越来越有效地利用内容信息,包括任务实例和自然语言任务描述。
<a id="S0386"></a> Source: p.24 S0386
Original: Finally, it is worth adding that solving these tasks requires character-level manipulations, whereas our BPE encoding operates on significant fractions of a word (on average ∼ 0.7 words per token), so from the LM’s perspective succeeding at these tasks involves not just manipulating BPE tokens but understanding and pulling apart their substructure.
中文: 最后,值得补充的是,解决这些任务需要人物层面的操作,而我们的BPE编码则以一个字(平均每个字为0.7个字)的相当一部分操作,因此从LM的角度来说,成功完成这些任务不仅涉及操纵BPE符号,而且涉及理解和拉开其子结构.
<a id="S0387"></a> Source: p.24 S0387
Original: Also, CL, A1, and A2 are not bijective (that is, the unscrambled word is not a deterministic function of the scrambled word), requiring the model to perform some search to find the correct unscrambling.
中文: 另外,CL,A1和A2不是双相(即:无相字不是相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相
<a id="S0388"></a> Source: p.24 S0388
Original: Thus, the skills involved appear to require non-trivial pattern-matching and computation. 3.9.3 SAT Analogies To test GPT-3 on another task that is somewhat unusual relative to the typical distribution of text, we collected a set of 374 “SAT analogy” problems [TLBS03].
中文: 因此,所涉技能似乎需要非三角模式匹配和计算。 3.9.3 SAT类比 为了检验GPT-3的另一项任务相对于文本的典型分布来说有些不寻常,我们收集了一套374个“SAT类比”问题[TLBS03]。
<a id="S0389"></a> Source: p.24 S0389
Original: Analogies are a style of multiple choice question that constituted a section of the SAT college entrance exam before 2005. A typical example is “audacious is to boldness as (a) sanctimonious is to hypocrisy, (b) anonymous is to identity, (c) remorseful is to misdeed, (d) deleterious is to result, (e) impressionable is to temptation”.
中文: 类比是多选题的风格,在2005年之前构成SAT大学入学考试的一个部分. 一个典型的例子是“大胆,因为(a) 圣洁是虚伪,(b) 匿名是身份,(c) 悔恨是不当行为,(d) 有害是结果,(e) 有感觉是诱惑”。
<a id="S0390"></a> Source: p.24 S0390
Original: The student is expected to choose which of the five word pairs has the same relationship as the original word pair; in this example the answer is “sanctimonious is to hypocrisy”.
中文: 预期学生会选择五个词对中哪一个与原来的词对有着相同的关系;在这个例子中,答案是"善意是虚伪".
<a id="S0391"></a> Source: p.24 S0391
Original: On this task GPT-3 achieves 65.2% in the few-shot setting, 59.1% in the one-shot setting, and 53.7% in the zero-shot setting, whereas the average score among college applicants was 57% [TL05] (random guessing yields 20%).
中文: 在这项任务中,GPT-3在少发场景中达到65.2%,一发场景中达到59.1%,零发场景中达到53.7%,而大学考生的平均得分为57%TL05.
<a id="S0392"></a> Source: p.24 S0392
Original: As shown in Figure 3.12, the results improve with scale, with the the full 175 billion model improving by over 10% compared to the 13 billion parameter model. 24
中文: 如图3.12所示,结果随着规模的扩大而有所改善,与130亿参数模型相比,完整的1 750亿模型改进了10%以上。 24个
<a id="S0393"></a> Source: p.25 S0393
Original: Figure 3.12: Zero-, one-,and few-shot performance on SAT analogy tasks, for different sizes of model.
中文: 图3.12:SAT类类比任务上的零相,一相,少相相等性能,为不同的型号.
<a id="S0394"></a> Source: p.25 S0394
Original: The largest model achieves 65% accuracy in the few-shot setting, and also demonstrates significant gains to in-context learning which are not present in smaller models. 3.9.4 News Article Generation Previous work on generative language models qualitatively tested their ability to generate synthetic “news articles” by conditional sampling from the model given a human-written prompt consisting of a plausible first sentence for a news story [RWC+19].
中文: 最大的模型在几发设定中实现了65%的精确度,也显示出在小模型中不存在的文字内学习的重大收益. 3.9.4 " 新闻文章生成 " 以往关于基因语言模型的工作,通过从模型中有条件地取样,对生成合成 " 新闻文章 " 的能力进行了质的测试,从模型中抽取出一个人为写作的提示,包括一个新闻故事的可信第一句[RWC+19]。
<a id="S0395"></a> Source: p.25 S0395
Original: Relative to [RWC+19], the dataset used to train GPT-3 is much less weighted towards news articles, so trying to generate news articles via raw unconditional samples is less effective – for example GPT-3 often interprets the proposed first sentence of a “news article” as a tweet and then posts synthetic responses or follow-up tweets.
中文: 相对于[RWC+19],用于训练GPT-3的数据集对新闻文章的加权率要低得多,因此试图通过原始的无条件样本生成新闻文章的效果要低得多——例如,GPT-3经常将拟议的"新闻文章"第一句解释为微博,然后发布合成回应或后续微博.
<a id="S0396"></a> Source: p.25 S0396
Original: To solve this problem we employed GPT-3’s few-shot learning abilities by providing three previous news articles in the model’s context to condition it.
中文: 为了解决这个问题,我们利用了GPT-3的几发学习能力,在模型的上下文中提供了前三篇新闻文章作为条件。
<a id="S0397"></a> Source: p.25 S0397
Original: With the title and subtitle of a proposed next article, the model is able to reliably generate short articles in the “news” genre.
中文: 有了拟议的下篇文章的标题和字幕,该模型能够在"新闻"流派中可靠地产生短篇.
<a id="S0398"></a> Source: p.25 S0398
Original: To gauge the quality of news article generation from GPT-3 (which we believe is likely to be correlated with conditional sample generation quality in general), we decided to measure human ability to distinguish GPT-3-generated articles from real ones.
中文: 为了从GPT-3(我们认为这很可能与一般有条件的样本生成质量相关)来衡量新闻文章生成的质量,我们决定衡量人类将GPT-3生成的文章与真实的文章区分开来的能力.
<a id="S0399"></a> Source: p.25 S0399
Original: Similar work has been carried out by Kreps et al. [KMB20] and Zellers et al. [ZHR+19].
中文: Kreps等人[KMB20]和Zellers等人[ZHR+19]也进行了类似的工作.
<a id="S0400"></a> Source: p.25 S0400
Original: Generative language models are trained to match the distribution of content generated by humans, so the (in)ability of humans to distinguish the two is a potentially important measure of quality.3 In order to see how well humans can detect model generated text, we arbitrarily selected 25 article titles and subtitles from the website newser.com (mean length: 215 words).
中文: 基因语言模型接受培训,以匹配人类所生成内容的分布,因此人类区分两者的(不)能力是潜在重要的质量衡量标准. 为了了解人类能有多好地检测模型生成的文本,我们任意选择了网站Newser.com的25个文章标题和字幕(平均长度:215个字).
<a id="S0401"></a> Source: p.25 S0401
Original: We then generated completions of these titles and subtitles from four language models ranging in size from 125M to 175B (GPT-3) parameters (mean length: 200 words).
中文: 随后我们产生了这些标题和字幕的完成,从125M到175B(GPT-3)参数(平均长度:200个字)大小不等的四种语言模型.
<a id="S0402"></a> Source: p.25 S0402
Original: For each model, we presented around 80 US-based participants with a quiz consisting of these real titles and subtitles followed by either the human written article or the article generated by the model4.
中文: 对于每个模型,我们向大约80名美国参与者介绍了一个包含这些真实标题和字幕的测验,然后是人文文章或模型4所出的文章。
<a id="S0403"></a> Source: p.25 S0403
Original: Participants were asked to select whether the article was “very likely written by a human”, “more likely written by a human”, “I don’t know”, “more likely written by a machine”, or “very likely written by a machine”.
中文: 要求与会者选择这篇文章是 " 很可能由人写作 " 、 " 更有可能由人写作 " 、 " 我不知道 " 、 " 更有可能由机器写作 " ,还是 " 很有可能由机器写作 " 。
<a id="S0404"></a> Source: p.25 S0404
Original: The articles we selected were not in the models’ training data and the model outputs were formatted and selected programmatically to prevent human cherry-picking.
中文: 我们选择的文章没有列入模型的培训数据,模型产出是按方案编排和选取的,以防止取人樱桃。
<a id="S0405"></a> Source: p.25 S0405
Original: All models used the same context to condition outputs on and were pre-trained with the same context size and the same article titles and subtitles were used as prompts for each model.
中文: 所有模型都使用相同的上下文来限定输出,并且经过前期训练,其上下文大小相同,并且每个模型都使用相同的文章标题和字幕作为提示.
<a id="S0406"></a> Source: p.25 S0406
Original: However, we also ran an experiment to control for participant effort and attention that followed the same format but involved intentionally bad model generated articles.
中文: 然而,我们还进行了一项试验,以控制参与者的努力和注意力,这种努力和注意力遵循同样的格式,但涉及故意坏模型生成文章。
<a id="S0407"></a> Source: p.25 S0407
Original: This was done by generating articles from a “control model”: a 160M parameter model with no context and increased output randomness. 3This task is also relevant to the potential misuse of language models discussed in Section 6.1. 4We wanted to identify how good an average person on the internet is at detecting language model outputs, so we focused on participants drawn from the general US population.
中文: 这是通过从“控制模型”生成文章来实现的:一个没有上下文并增加输出随机性的160M参数模型. 3 这项任务还与第6.1节所讨论语文模式可能被滥用的情况有关。 4,我们想找出网路上一般人如何善于检测语言模型输出,因此我们专注于从美国一般民众中抽取的参与者.
<a id="S0408"></a> Source: p.26 S0408
Original: 95% Confidence t compared to “I don’t know” Mean accuracy Interval (low, hi) control (p-value) assignments Control (deliberately bad model) 86% 83%–90% - 3.6 % GPT-3 Small 76% 72%–80% 3.9 (2e-4) 4.9% GPT-3 Medium 61% 58%–65% 10.3 (7e-21) 6.0% GPT-3 Large 68% 64%–72% 7.3 (3e-11) 8.7% GPT-3 XL 62% 59%–65% 10.7 (1e-19) 7.5% GPT-3 2.7B 62% 58%–65% 10.4 (5e-19) 7.1% GPT-3 6.7B 60% 56%–63% 11.2 (3e-21) 6.2% GPT-3 13B 55% 52%–58% 15.3 (1e-32) 7.1% GPT-3 175B 52% 49%–54% 16.9 (1e-34) 7.8% Table 3.11: Human accuracy in identifying whether short (∼200 word) news articles are model generated.
中文: 平均准确度 Interval(低,高)控制(p-值) 分配控制(故意坏模型) 86% 83-90% - 3.6% GPT-3 小型 76% 72-80% % 3.9(2e-4) 4.9% GPT-3 中型 61% 58-65% 10.3 (7e-21) 6.0% GPT-3 大 68% (3e-11) 8.7% GPT-3 XL 62% 5.9-65% 10.7(1e-19) 7.5% GPT-3 2.7B 6% 58-% 10.4 (5e-19) 7.1% GPT-3 6.7 B 60% 56-63% 11.2 (3e-21) 6.2% GPT-3 13B 55% 52-% 15.3 (1e-32) 7.1% GPT-3 175B 52% 49-54% (16.9 (1e-34) 7.8% 表3.1% 确定短 (12-200字) 新闻文章是否为模型生成。
<a id="S0409"></a> Source: p.26 S0409
Original: We find that human accuracy (measured by the ratio of correct assignments to non-neutral assignments) ranges from 86% on the control model to 52% on GPT-3 175B.
中文: 我们发现人类的准确性(以正确派任与非中立派任的比例衡量)从控制模型上的86%到GPT-3175B上的52%不等.
<a id="S0410"></a> Source: p.26 S0410
Original: This table compares mean accuracy between five different models, and shows the results of a two-sample T-Test for the difference in mean accuracy between each model and the control model (an unconditional GPT-3 Small model with increased output randomness).
中文: 本表比较了5个不同模型之间的平均精度,并显示了两样T-测试结果,以表示每个模型与控制模型(一个无条件的GPT-3 Small模型,输出随机性增加)之间的平均精度差异.
<a id="S0411"></a> Source: p.26 S0411
Original: Mean human accuracy (the ratio of correct assignments to non-neutral assignments per participant) at detecting that the intentionally bad articles were model generated was ∼ 86% where 50% is chance level performance.
中文: 在发现故意坏文章是模型生成时,人类平均准确度(每个参与者正确分配与非中性分配的比例)为86%,其中50%为机会水平业绩。
<a id="S0412"></a> Source: p.26 S0412
Original: By contrast, mean human accuracy at detecting articles that were produced by the 175B parameter model was barely above chance at ∼ 52% (see Table 3.11).5 Human abilities to detect model generated text appear to decrease as model size increases: there appears to be a trend towards chance accuracy with model size, and human detection of GPT-3 is close to chance.6 This is true despite the fact that participants spend more time on each output as model size increases (see Appendix E).
中文: 相比之下,发现由175B参数模型生成的物品时,人类的平均准确度略高于52%(见表3.11)。 随着模型大小的增加,人类探测模型生成的文字的能力似乎正在下降:模型大小似乎呈现出机会准确性的趋势,人类探测出GPT-3的机会也接近了。 尽管随着模型尺寸的增加,参与者在每项产出上花费了更多的时间(见附录E),但情况确实如此.
<a id="S0413"></a> Source: p.26 S0413
Original: Examples of synthetic articles from GPT-3 are given in Figures 3.14 and 3.15.7 Much of the text is—as indicated by the evaluations—difficult for humans to distinguish from authentic human content.
中文: 图3.14和图3.15.7列举了GPT-3合成物品的例子。 大部分案文——如评价所示——难以区分人与人的真实内容。
<a id="S0414"></a> Source: p.26 S0414
Original: Factual inaccuracies can be an indicator that an article is model generated since, unlike human authors, the models have no access to the specific facts that the article titles refer to or when the article was written.
中文: 事实不准确可能是一个指标,表明文章是模型生成的,因为与人类作者不同,模型无法访问文章标题所指的具体事实或文章被撰写时.
<a id="S0415"></a> Source: p.26 S0415
Original: Other indicators include repetition, non sequiturs, and unusual phrasings, though these are often subtle enough that they are not noticed.
中文: 其他指标包括重复、非从属和不同寻常的用词,尽管这些用词往往非常微妙,以至于不被注意。
<a id="S0416"></a> Source: p.26 S0416
Original: Related work on language model detection by Ippolito et al. [IDCBE19] indicates that automatic discriminators like G R O V E R [ZHR+19] and GLTR [GSR19] may have greater success at detecting model generated text than human evaluators.
中文: Ipolito等人在语言模型检测方面的相关工作[IDCBE19]表明,GR O V E R [ZHR+19]和GLTR [GSR19]等自动歧视者在检测模型生成的文本方面可能比人类评价者更成功.
<a id="S0417"></a> Source: p.26 S0417
Original: Automatic detection of these models may be a promising area of future research.
中文: 这些模型的自动探测可能是未来研究的一个有希望的领域.
<a id="S0418"></a> Source: p.26 S0418
Original: Ippolito et al. [IDCBE19] also note that human accuracy at detecting model generated text increases as humans observe more tokens.
中文: Ippolito等 [IDCBE19]也指出,人类在探测模型时的准确性随着人类观察到更多的活物而增加文字.
<a id="S0419"></a> Source: p.26 S0419
Original: To do a preliminary investigation of how good humans are at detecting longer news articles generated by GPT-3 175B, we selected 12 world news articles from Reuters with an average length of 569 words and generated completions of these articles from GPT-3 with an average length of 498 words (298 words longer than our initial experiments).
中文: 为了初步调查人类在发现由GPT-3175B生成的更长的新闻文章方面表现如何,我们从路透社选取了12个世界新闻文章,平均长度为569个字,并从GPT-3获得这些文章的完成,平均长度为498个字(比我们最初的实验长了298个字).
<a id="S0420"></a> Source: p.26 S0420
Original: Following the methodology above, we ran two experiments, each on around 80 US-based participants, to compare human abilities to detect the articles generated by GPT-3 and a control model.
中文: 按照上述方法,我们进行了两次试验,每次试验大约80名美国参与者,以比较人类检测出GPT-3产生的物品的能力和一种控制模型。
<a id="S0421"></a> Source: p.26 S0421
Original: We found that mean human accuracy at detecting the intentionally bad longer articles from the control model was ∼ 88%, while mean human accuracy at detecting the longer articles that were produced by GPT-3 175B was still barely above chance at ∼ 52% (see Table 3.12).
中文: 我们发现,在检测控制模型中故意坏的较长物品时,人类的精确度是88%;而在检测GPT-3175B生产的较长物品时,人类的精确度仍然略高于52%(见表3.12)。
<a id="S0422"></a> Source: p.26 S0422
Original: This indicates that, for news articles that are around 500 words long, GPT-3 continues to produce articles that humans find difficult to distinguish from human written news articles. 3.9.5 Learning and Using Novel Words A task studied in developmental linguistics [CB78] is the ability to learn and utilize new words, for example using a word in a sentence after seeing it defined only once, or conversely inferring a word’s meaning from only one usage.
中文: 这表明,对于约500个字长的新闻文章,GPT-3继续出文章,人类发现难以与人类书面新闻文章相区别. 3.9.5 学习和使用小说 A 发展语言学研究的任务[CB78]是学习和利用新词的能力,例如,在只看到一个词定义一次之后,在句子中使用一个词,或者相反地从一种用法中推断出一个词的含义。
<a id="S0423"></a> Source: p.26 S0423
Original: Here we qualitatively test GPT-3’s ability to do the former.
中文: 在这里,我们从质量上检验GPT-3的能力。
<a id="S0424"></a> Source: p.26 S0424
Original: Specifically, we give GPT-3 the definition of a nonexistent word, such as “Gigamuru”, and then ask it to use it in a sentence.
中文: 具体地说,我们给GPT-3定义了一个不存在的单词,如"Gigamuru",然后请它用到一句话中.
<a id="S0425"></a> Source: p.26 S0425
Original: We provide one to five previous examples of a (separate) 5We use a two-sample Student’s T-Test to test for significant difference between the means of the participant accuracies of each model and the control model and report the normalized difference in the means (as the t-statistic) and the p-value. 6If a model consistently produces texts that are more impressive than human articles, it is possible that human performance on this task would drop below 50%.
中文: 我们提供一到五个前例的(单独) 5,我们用两个样本的学生T测试来测试每个模型的参与者敏捷度与控制模型之间的显著差异,并报告方法(作为t-统计)和p-值的正态差异. 6 如果一个模型不断产生比人的文章更令人印象深刻的文本,人类完成这项任务的成绩有可能会下降到50%以下。
<a id="S0426"></a> Source: p.26 S0426
Original: Indeed, many individual participants scored below 50% on this task. 7Additional non-news samples can be found in Appendix F. 26
中文: 实际上,许多个人参与者在这项工作中得分低于50%。 附录F. 26载有非新闻补充样本。
<a id="S0427"></a> Source: p.27 S0427
Original: Figure 3.13: People’s ability to identify whether news articles are model-generated (measured by the ratio of correct assignments to non-neutral assignments) decreases as model size increases.
中文: 图3.13:随着模型大小的增加,人们识别新闻文章是否由模型生成(以正确指派与非中性指派的比例衡量)的能力会下降.
<a id="S0428"></a> Source: p.27 S0428
Original: Accuracy on the outputs on the deliberatelybad control model (an unconditioned GPT-3 Small model with higher output randomness) is indicated with the dashed line at the top, and the random chance (50%) is indicated with the dashed line at the bottom.
中文: 故意坏控制模型上输出的精确度(一个输出随机度较高的无条件的GPT-3小模型)用倒行表示,而随机机率(50%)用倒行表示.
<a id="S0429"></a> Source: p.27 S0429
Original: Line of best fit is a power law with 95% confidence intervals. 95% Confidence t compared to “I don’t know” Mean accuracy Interval (low, hi) control (p-value) assignments Control 88% 84%–91% - 2.7% GPT-3 175B 52% 48%–57% 12.7 (3.2e-23) 10.6% Table 3.12: People’s ability to identify whether ∼ 500 word articles are model generated (as measured by the ratio of correct assignments to non-neutral assignments) was 88% on the control model and 52% on GPT-3 175B.
中文: 最合适的线是95%的置信间隔的电能法. 与“我不知道”相比,95%的置信度t 平均精度 Interval(低,高)控制(p-值) 任务 控制 88% 84-%91 % - 2.7% GPT-3 175B 52% 48-% - 12.7 (3.2e-23) 10.6% 表3.12 :人们是否有能力确定在控制模型上是否生成了 ? 500个字文章(以正确分配与非中立分配的比例衡量) , 在GPT-3 175B 52% 。
<a id="S0430"></a> Source: p.27 S0430
Original: This table shows the results of a two-sample T-Test for the difference in mean accuracy between GPT-3 175B and the control model (an unconditional GPT-3 Small model with increased output randomness). 27
中文: 本表显示为GPT-3 175B与控制模型(无条件的GPT-3 Small模型,输出随机性增加)在平均精度上相差的两样T-测试结果. 27个
<a id="S0431"></a> Source: p.28 S0431
Original: Title: United Methodists Agree to Historic Split Subtitle: Those who oppose gay marriage will form their own denomination Article: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be "theologically and socially conservative," according to The Washington Post.
中文: 标题:联合卫理公会同意历史分立 副标题:反对同性婚姻的人会形成自己的教派 文章:经过两天的激烈争论后,联合卫理公会同意了历史性分立——这一分出有望以创立新教派而告终,根据"华盛顿邮报",这一分出将"在神学和社会上保守".
<a id="S0432"></a> Source: p.28 S0432
Original: The majority of delegates attending the church’s annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will "discipline" clergy who officiate at same-sex weddings.
中文: 出席5月教会年度大会的多数代表都投票,
<a id="S0433"></a> Source: p.28 S0433
Original: But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination.
中文: 但反对这些措施的人有一个新的计划: 他们说,到2020年,他们将形成一个单独的教派,称他们的教会为基督教卫理公会教派.
<a id="S0434"></a> Source: p.28 S0434
Original: The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the "largest Protestant denomination in the U.S.," but that it has been shrinking in recent decades.
中文: "邮报"指出,该教派拥有1,250万会员,在20世纪初是"美国最大的新教教派",但近几十年来该教派一直在收缩.
<a id="S0435"></a> Source: p.28 S0435
Original: The new split will be the second in the church’s history.
中文: 新的分裂将是教会历史上的第二次.
<a id="S0436"></a> Source: p.28 S0436
Original: The first occurred in 1968, when roughly 10 percent of the denomination left to form the Evangelical United Brethren Church.
中文: 第一次发生在1968年,大约10%的教派离开,组成了福音联合弟兄会。
<a id="S0437"></a> Source: p.28 S0437
Original: The Post notes that the proposed split "comes at a critical time for the church, which has been losing members for years," which has been "pushed toward the brink of a schism over the role of LGBTQ people in the church." Gay marriage is not the only issue that has divided the church.
中文: "邮报"指出,拟议的分裂"是在教会的关键时刻来到的,教会多年来一直失去成员",这被"推向了对LGBTQ在教会中角色的分裂边缘". 同性恋婚姻并不是分裂教会的唯一问题.
<a id="S0438"></a> Source: p.28 S0438
Original: In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.
中文: 2016年,该教派因变性神职人员的授职被分裂,北太平洋地区会议投票禁止他们担任神职人员,南太平洋地区会议投票允许他们担任.
<a id="S0439"></a> Source: p.28 S0439
Original: Figure 3.14: The GPT-3 generated news article that humans had the greatest difficulty distinguishing from a human written article (accuracy: 12%).
中文: 图3.14:GPT-3生成了新闻文章,指出人类与人所写文章相比存在最大的困难(准确性:12%).
<a id="S0440"></a> Source: p.28 S0440
Original: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am a shape-shifter. I can’t change the world. I can only change myself." It was a promise to not change to fit into the Hollywood mold: "I think that’s a really special thing, to not change yourself. I think it’s a really special thing to say, ‘This is what’s inside of me, I’m proud of it, and I’m not going to be ashamed because of the way that someone else thinks I should be.’" Now, it’s the Oscars, and Phoenix is at it again.
中文: 标题:"明星的Tux承诺" Draws Megyn Kelly's Sarcasm 字幕:华金·菲尼克斯承诺不会为每次颁奖活动而改变 文章:一年前,华金·菲尼克斯在"金球会"的红地毯上出道时,身穿礼服,头上有纸袋,上面写道:"我是一个变形人. 我无法改变世界。 我只能改变自己". 承诺不会改变, 以适应好莱坞的模具: “我认为, 我认为说:“这就是我内心的感受, 我对此感到骄傲, 现在,这是奥斯卡颁奖典礼,凤凰又登场了。
<a id="S0441"></a> Source: p.28 S0441
Original: But this time, his publicist is saying he’ll be wearing a tux no matter what.
中文: 但这次他的公关人士表示,
<a id="S0442"></a> Source: p.28 S0442
Original: Megyn Kelly was not impressed, and she let him have it on The Tonight Show. "You know, I feel like, I feel like you could have worn the tux," she says. "But you’re saying you’re a shape-shifter. I don’t know if you can change your tux, but you can change your mind.
中文: Megyn Kelly没有留下深刻印象,她让他在"今晚秀"上登场. "你知道,我觉得,我觉得, 我觉得你可以穿礼服,"她说。 “但你说你是变形人。 我不知道您能否改变礼服,
<a id="S0443"></a> Source: p.28 S0443
Original: You can change your mind." Phoenix says he did, but it didn’t stick. "I was like, ‘Okay, I’m going to wear a tuxedo to this thing.’ And then I thought, ‘I don’t want to wear a tuxedo to this thing.’" Kelly goes on to encourage him to change his mind again, but Phoenix says it’s too late: "I’m committed to wearing this." Figure 3.15: The GPT-3 generated news article that humans found the easiest to distinguish from a human written article (accuracy: 61%). 28
中文: 你可以改变主意。" 凤凰网说, "我当时想,"好吧,我要穿一件礼服来做这件事". 然后我想,“我不想穿这件礼服。” 凯利继续鼓励他再次改变主意,但菲尼克斯表示已经太晚了:"我承诺要穿这个". 图3.15:GPT-3产生的新闻文章,人类发现最容易区别于人所写的文章(准确度:61%). 第28条
<a id="S0444"></a> Source: p.29 S0444
Original: A "whatpu" is a small, furry animal native to Tanzania.
中文: 一个"Whatpu"是坦桑尼亚本土的小型毛毛动物.
<a id="S0445"></a> Source: p.29 S0445
Original: An example of a sentence that uses the word whatpu is: We were traveling in Africa and we saw these very cute whatpus.
中文: 一个使用“什么”这个词的句子的例子:我们当时在非洲旅行,我们看到了这些非常可爱的“什么”。
<a id="S0446"></a> Source: p.29 S0446
Original: To do a "farduddle" means to jump up and down really fast.
中文: 做一个"活泼"意味着跳上跳下非常快.
<a id="S0447"></a> Source: p.29 S0447
Original: An example of a sentence that uses the word farduddle is: One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles. A "yalubalu" is a type of vegetable that looks like a big pumpkin.
中文: 一个使用"远道"这个词的句子的例子就是: 有一天,当我和我妹妹玩牌的时候, 她非常兴奋,她开始做这些疯狂的远道行走。 "亚鲁巴卢"(英語:Yalubalu)是一类外观为大南瓜的蔬菜.
<a id="S0448"></a> Source: p.29 S0448
Original: An example of a sentence that uses the word yalubalu is: I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden there.
中文: 一个使用雅卢巴卢一词的句子的例子就是:我当时正在去非洲旅行,我尝试了一种雅卢巴卢蔬菜,它生长在花园里。
<a id="S0449"></a> Source: p.29 S0449
Original: It was delicious. A "Burringo" is a car with very fast acceleration.
中文: 这是美味的。 "Burringo"是一款速度非常快的汽车.
<a id="S0450"></a> Source: p.29 S0450
Original: An example of a sentence that uses the word Burringo is: In our garage we have a Burringo that my father drives to work every day. A "Gigamuru" is a type of Japanese musical instrument.
中文: 一个使用Burringo这个词的句子的例子就是: 在我们的车库里,我们有一个Burringo, 我父亲每天开车去上班。 相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相
<a id="S0451"></a> Source: p.29 S0451
Original: An example of a sentence that uses the word Gigamuru is: I have a Gigamuru that my uncle gave me as a gift. I love to play it at home.
中文: 一个使用"Gigamuru"一词的句子的例子是:我有一个我叔叔送给我的"Gigamuru"作为礼物. 我喜欢在家里玩
<a id="S0452"></a> Source: p.29 S0452
Original: To "screeg" something is to swing a sword at it.
中文: "去""去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去去"去"去"去去"去去"去"去"去"去去"去"去去"去去去"去"去"去去去去"去去去去去去去"去"去"去"去"去"去"去"去去"去去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"去"
<a id="S0453"></a> Source: p.29 S0453
Original: An example of a sentence that uses the word screeg is: We screeghed at each other for several minutes and then we went outside and ate ice cream.
中文: 一个使用"screeg"一词的句子的例子就是: 我们互相尖叫了几分钟,然后我们出去吃冰淇淋.
<a id="S0454"></a> Source: p.29 S0454
Original: Figure 3.16: Representative GPT-3 completions for the few-shot task of using a new word in a sentence.
中文: 图3.16:代表GPT-3完成在句子中使用新词的几发任务.
<a id="S0455"></a> Source: p.29 S0455
Original: Boldface is GPT-3’s completions, plain text is human prompts.
中文: 粗体是GPT-3的完成,纯正文字是人类的提示.
<a id="S0456"></a> Source: p.29 S0456
Original: In the first example both the prompt and the completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional prompts and provides the completions.
中文: 在第一个例子中,即即时和完成均由人类提供;然后,这可作为以后GPT-3接二连三地收到再接再接的即时和完成的前提条件。
<a id="S0457"></a> Source: p.29 S0457
Original: Nothing task-specific is provided to GPT-3 other than the conditioning shown here. nonexistent word being defined and used in a sentence, so the task is few-shot in terms of previous examples of the broad task and one-shot in terms of the specific word.
中文: 除了这里显示的调制条件外,没有为GPT-3提供特定任务. 在句子中被定义和使用的是不存在的单词,因此,任务在以前关于广义任务的例子中是少有的,在具体单词中是一发.
<a id="S0458"></a> Source: p.29 S0458
Original: Table 3.16 shows the 6 examples we generated; all definitions were human-generated, and the first answer was human-generated as conditioning while the subsequent answers were generated by GPT-3.
中文: 表3.16显示了我们生成的6个实例;所有定义都是由人类生成的,第一个答案是人为生成的调节,而随后的答案是由GPT-3生成的.
<a id="S0459"></a> Source: p.29 S0459
Original: These examples were generated continuously in one sitting and we did not omit or repeatedly try any prompts.
中文: 这些例子在一个座位上不断产生,我们没有省略或反复尝试任何提示。
<a id="S0460"></a> Source: p.29 S0460
Original: In all cases the generated sentence appears to be a correct or at least plausible use of the word.
中文: 在所有情况下,生成的句子似乎正确或至少是合理使用这个词。
<a id="S0461"></a> Source: p.29 S0461
Original: In the final sentence the model generates a plausible conjugation for the word “screeg” (namely “screeghed”), although the use of the word is slightly awkward (“screeghed at each other”) despite being plausible in the sense that it could describe a toy sword fight.
中文: 在最后一句中,该模型对“screeg”(即“screeghed”)一词产生一种可信的共鸣,尽管该词的使用略为尴尬(“screeghed at other”),尽管它可以描述玩具剑战。
<a id="S0462"></a> Source: p.29 S0462
Original: Overall, GPT-3 appears to be at least proficient at the task of using novel words in a sentence. 3.9.6 Correcting English Grammar Another task well suited for few-shot learning is correcting English grammar.
中文: 总体而言,GPT-3似乎至少精通于在句子中用出新词的任务. 3.9.6 校正英语语法 另一项非常适合数发学习的任务就是纠正英语语法.
<a id="S0463"></a> Source: p.29 S0463
Original: We test this with GPT-3 in the fewshot setting by giving prompts of the form "Poor English Input: <sentence>\n Good English Output: <sentence>".
中文: 我们用GPT-3在几发画面中进行测试,方法是给出“可怜的英语输入:<pentence>\n Good English 输出:<pentence>”表格的提示。
<a id="S0464"></a> Source: p.29 S0464
Original: We give GPT-3 one human-generated correction and then ask it to correct 5 more (again without any omissions or repeats).
中文: 我们给GPT-3一个由人类产生的修正,然后要求它再更正5个(同样没有任何遗漏或重复)。
<a id="S0465"></a> Source: p.29 S0465
Original: Results are shown in Figure 3.17. 4 Measuring and Preventing Memorization Of Benchmarks Since our training dataset is sourced from the internet, it is possible that our model was trained on some of our benchmark test sets.
中文: 结果见图3.17。 4 衡量和防止基准的记忆,由于我们的培训数据集来自互联网,因此我们的模型有可能接受了一些基准测试集的培训。
<a id="S0466"></a> Source: p.29 S0466
Original: Accurately detecting test contamination from internet-scale datasets is a new area of research without established best practices.
中文: 准确检测互联网规模数据集的测试污染是一个没有既定最佳做法的新研究领域。
<a id="S0467"></a> Source: p.29 S0467
Original: While it is common practice to train large models without investigating contamination, given the increasing scale of pretraining datasets, we believe this issue is becoming increasingly important to attend to.
中文: 虽然在不调查污染的情况下训练大型模型是常见的做法,但鉴于预先训练数据集的规模越来越大,我们认为这一问题正变得越来越重要。
<a id="S0468"></a> Source: p.29 S0468
Original: One of the first papers to train a language model on Common Crawl data [TL18] detected and removed a training document which overlapped with one of their evaluation datasets.
中文: 最早培训通用爬行数据语言模型的论文之一[TL18]检测并移除了与其一个评价数据集相重叠的训练文件.
<a id="S0469"></a> Source: p.29 S0469
Original: Other work such as GPT-2 [RWC+19] also conducted post-hoc overlap analysis.
中文: 其他工作如GPT-2[RWC+19]也进行了HOC后重叠分析.
<a id="S0470"></a> Source: p.29 S0470
Original: Their study was relatively encouraging, finding that 29
中文: 他们的研究相对令人鼓舞,发现有29人
<a id="S0471"></a> Source: p.30 S0471
Original: Poor English input: I eated the purple berries.
中文: 可怜的英语输入:我吃了紫莓.
<a id="S0472"></a> Source: p.30 S0472
Original: Good English output: I ate the purple berries.
中文: 好英文出品:我吃紫浆.
<a id="S0473"></a> Source: p.30 S0473
Original: Poor English input: Thank you for picking me as your designer. I’d appreciate it.
中文: 可怜的英语输入:谢谢你选我为设计师. 我很感激。
<a id="S0474"></a> Source: p.30 S0474
Original: Good English output: Thank you for choosing me as your designer. I appreciate it.
中文: 英文产出不错:谢谢你选择我为设计师. 谢谢
<a id="S0475"></a> Source: p.30 S0475
Original: Poor English input: The mentioned changes have done. or I did the alteration that you requested. or I changed things you wanted and did the modifications.
中文: 英文输入不足:上述变化已经完成。 或者我做了你要求的改变 或者我改变了你想要的东西 做了修改
<a id="S0476"></a> Source: p.30 S0476
Original: Good English output: The requested changes have been made. or I made the alteration that you requested. or I changed things you wanted and made the modifications.
中文: 良好的英语输出:请求的修改已经实现. 或者我做了你要求的改变 或者我改变了你想要的东西 做了修改。
<a id="S0477"></a> Source: p.30 S0477
Original: Poor English input: I’d be more than happy to work with you in another project.
中文: 我很乐意与你们合作,
<a id="S0478"></a> Source: p.30 S0478
Original: Good English output: I’d be more than happy to work with you on another project.
中文: 好英语产出:我很乐意与你合作,
<a id="S0479"></a> Source: p.30 S0479
Original: Poor English input: Please provide me with a short brief of the design you’re looking for and that’d be nice if you could share some examples or project you did before.
中文: (笑声) 可怜的英语输入:请给我一个简短的简介, 介绍你正在寻找的设计,
<a id="S0480"></a> Source: p.30 S0480
Original: Good English output: Please provide me with a brief description of the design you’re looking for and that would be nice if you could share some examples or projects you have done before.
中文: 英文产出不错:请给我一个简短的描述,
<a id="S0481"></a> Source: p.30 S0481
Original: Poor English input: The patient was died.
中文: 可怜的英语输入:患者已去世.
<a id="S0482"></a> Source: p.30 S0482
Original: Poor English input: We think that Leslie likes ourselves.
中文: 可怜的英语输入:我们认为莱斯利喜欢自己.
<a id="S0483"></a> Source: p.30 S0483
Original: Good English output: We think that Leslie likes us.
中文: 英文产出不错:我们认为莱斯利喜欢我们.
<a id="S0484"></a> Source: p.30 S0484
Original: Poor English input: Janet broke Bill on the finger.
中文: 可怜的英语输入:珍妮特用手指打碎了比尔.
<a id="S0485"></a> Source: p.30 S0485
Original: Good English output: Janet broke Bill’s finger.
中文: 英文产出不错:珍妮特断了比尔的手指.
<a id="S0486"></a> Source: p.30 S0486
Original: Poor English input: Mary arranged for, in St.
中文: 可怜的英语输入:玛丽安排的,在圣多克.
<a id="S0487"></a> Source: p.30 S0487
Original: Good English output: Mary arranged for John to rent a house in St.
中文: 良好的英语出品:玛丽安排了约翰在圣德租了房子.
<a id="S0488"></a> Source: p.30 S0488
Original: Poor English input: Today I have went to the store to to buys some many bottle of water.
中文: 可怜的英语输入:今天我到店里去买多瓶水.
<a id="S0489"></a> Source: p.30 S0489
Original: Good English output: Today I went to the store to buy some bottles of water.
中文: 好英文出品:今天我去店里买些水来.
<a id="S0490"></a> Source: p.30 S0490
Original: Poor English input: I have tried to hit ball with bat, but my swing is has miss.
中文: 可怜的英语输入:我曾经尝试用蝙蝠来打球,但我的摇摆失手了.
<a id="S0491"></a> Source: p.30 S0491
Original: Good English output: I tried to hit the ball with the bat, but my swing missed.
中文: 英文输出不错:我曾想用球棒打出球来,但秋千没打中.
<a id="S0492"></a> Source: p.30 S0492
Original: Figure 3.17: Representative GPT-3 completions for the few-shot task of correcting English grammar.
中文: 图3.17:代表GPT-3完成几发校正英语语法任务.
<a id="S0493"></a> Source: p.30 S0493
Original: Boldface is GPT-3’s completions, plain text is human prompts.
中文: 粗体是GPT-3的完成,纯正文字是人类的提示.
<a id="S0494"></a> Source: p.30 S0494
Original: In the first few examples example both the prompt and the completion are provided by a human; this then serves as conditioning for subsequent examples where GPT-3 receives successive additional prompts and provides the completions.
中文: 在前几个例子中,一个人类提供了即时和完成;这随后成为了GPT-3接二连三地收到再接再接的即时和完成的前提条件。
<a id="S0495"></a> Source: p.30 S0495
Original: Nothing task-specific is provided to GPT-3 aside from the first few examples as conditioning and the “Poor English input/Good English output” framing.
中文: 除了头几个例子作为条件和“可怜的英语输入/良好的英语输出”框架外,没有向GPT-3提供具体的任务。
<a id="S0496"></a> Source: p.30 S0496
Original: We note that the distinction between ”poor” and ”good” English (and the terms themselves) is complex, contextual, and contested.
中文: 我们注意到,“穷”和“好”的英语(以及术语本身)之间的区别是复杂的、内在的和有争议的。
<a id="S0497"></a> Source: p.30 S0497
Original: As the example mentioning the rental of a house shows, assumptions that the model makes about what “good” is can even lead it to make errors (here, the model not only adjusts grammar, but also removes the word ”cheap” in a way that alters meaning). 30
中文: 正如提到租房的例子所显示的那样,模型对什么是“好”的假设甚至可以导致它出错(这里,模型不仅会调整语法,而且会以改变意义的方式去掉"便宜"一词). 30个
<a id="S0498"></a> Source: p.31 S0498
Original: Figure 4.1: GPT-3 Training Curves We measure model performance during training on a deduplicated validation split of our training distribution.
中文: 图 4.1:GPT-3培训曲线 我们在训练期间测量模型的性能 测试我们训练分配的破解验证
<a id="S0499"></a> Source: p.31 S0499
Original: Though there is some gap between training and validation performance, the gap grows only minimally with model size and training time, suggesting that most of the gap comes from a difference in difficulty rather than overfitting. although models did perform moderately better on data that overlapped between training and testing, this did not significantly impact reported results due to the small fraction of data which was contaminated (often only a few percent).
中文: 虽然在培训与鉴定业绩之间还存在一些差距,但差距随着模型大小和培训时间而最小地扩大,这表明大多数差距来自难度的差异,而不是过于适应。 虽然模型在培训和测试之间重叠的数据方面表现得比较好,但是由于受污染的数据的一小部分(往往只有9%),这并没有对报告的结果产生重大影响。
<a id="S0500"></a> Source: p.31 S0500
Original: GPT-3 operates in a somewhat different regime.
中文: GPT-3在某种程度上不同的制度下运作。
<a id="S0501"></a> Source: p.31 S0501
Original: On the one hand, the dataset and model size are about two orders of magnitude larger than those used for GPT-2, and include a large amount of Common Crawl, creating increased potential for contamination and memorization.
中文: 一方面,数据集和模型尺寸相较于GPT-2所使用的相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相较于相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相當相等相當相當相當相當相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相
<a id="S0502"></a> Source: p.31 S0502
Original: On the other hand, precisely due to the large amount of data, even GPT-3 175B does not overfit its training set by a significant amount, measured relative to a held-out validation set with which it was deduplicated (Figure 4.1).
中文: 另一方面,正是由于大量的数据,即使是GPT-3 175B也没有过度适应其设定的训练量,相对于其被破解的被搁置的验证组(图4.1)。
<a id="S0503"></a> Source: p.31 S0503
Original: Thus, we expect that contamination is likely to be frequent, but that its effects may not be as large as feared.
中文: 因此,我们期望污染可能频繁发生,但其影响可能不如所担心的要大。
<a id="S0504"></a> Source: p.31 S0504
Original: We initially tried to address the issue of contamination by proactively searching for and attempting to remove any overlap between our training data and the development and test sets of all benchmarks studied in this paper.
中文: 我们最初试图解决污染问题,积极寻找并试图消除我们的培训数据与本文件所研究的所有基准的制定和测试之间的任何重叠。
<a id="S0505"></a> Source: p.31 S0505
Original: Unfortunately, a bug resulted in only partial removal of all detected overlaps from the training data.
中文: 不幸的是,一个错误只导致部分清除了培训数据中发现的所有重叠。
<a id="S0506"></a> Source: p.31 S0506
Original: Due to the cost of training, it wasn’t feasible to retrain the model.
中文: 因为训练费用,
<a id="S0507"></a> Source: p.31 S0507
Original: To address this, we investigate in detail how the remaining detected overlap impacts results.
中文: 为了解决这一问题,我们详细调查了所发现的其余重叠影响结果。
<a id="S0508"></a> Source: p.31 S0508
Original: For each benchmark, we produce a ‘clean’ version which removes all potentially leaked examples, defined roughly as examples that have a 13-gram overlap with anything in the pretraining set (or that overlap with the whole example when it is shorter than 13-grams).
中文: 对于每个基准,我们制作一个 " 干净 " 的版本,删除所有可能漏出的例子,大致定义为与预训集中的任何内容有13克相重叠的例子(或者在短于13克时与整个例子相重叠的例子)。
<a id="S0509"></a> Source: p.31 S0509
Original: The goal is to very conservatively flag anything that could potentially be contamination, so as to produce a clean subset that is free of contamination with high confidence.
中文: 目标是非常保守地标出任何有可能被污染的东西,以便产生一个没有高度被污染的干净子集.
<a id="S0510"></a> Source: p.31 S0510
Original: The exact procedure is detailed in Appendix C.
中文: 具体程序详见附录C.
<a id="S0511"></a> Source: p.31 S0511
Original: We then evaluate GPT-3 on these clean benchmarks, and compare to the original score.
中文: 然后我们根据这些干净的基准来评价GPT-3,并与原始分数进行比较.
<a id="S0512"></a> Source: p.31 S0512
Original: If the score on the clean subset is similar to the score on the entire dataset, this suggests that contamination, even if present, does not have a significant effect on reported results.
中文: 如果干净子集上的分数与整个数据集的分数相近,这表明污染即使存在,对所报告结果也没有重大影响.
<a id="S0513"></a> Source: p.31 S0513
Original: If the score on the clean subset is lower, this suggests contamination may be inflating the results.
中文: 如果清洁子集的分数较低,这表明污染可能夸大了结果。
<a id="S0514"></a> Source: p.31 S0514
Original: The results are summarized in Figure 4.2.
中文: 图4.2汇总了结果。
<a id="S0515"></a> Source: p.31 S0515
Original: Although potential contamination is often high (with a quarter of benchmarks scoring over 50%), in most cases performance changes only negligibly, and we see no evidence that contamination level and performance difference are correlated.
中文: 虽然潜在污染往往很高(基准的四分之一得分超过50%),但在大多数情况下,性能变化不大,而且没有证据表明污染水平和性能差异是相互关联的。
<a id="S0516"></a> Source: p.31 S0516
Original: We conclude that either our conservative method substantially overestimated contamination or that contamination has little effect on performance.
中文: 我们的结论是,要么我们保守的方法大大高估了污染,要么污染对性能影响不大。
<a id="S0517"></a> Source: p.31 S0517
Original: Below, we review in more detail the few specific cases where either (1) the model performs significantly worse on the cleaned version, or (2) potential contamination is very high, which makes measuring the performance difference difficult.
中文: 下面我们更详细地审查少数具体案例:(1) 模型在被清洗的版本上表现得更差,或(2) 潜在污染非常高,因此难以衡量性能差异。
<a id="S0518"></a> Source: p.31 S0518
Original: Our analysis flagged six groups of benchmarks for further investigation: Word Scrambling, Reading Comprehension (QuAC, SQuAD2, DROP), PIQA, Winograd, language modeling tasks (Wikitext tasks, 1BW), and German to English 31
中文: 我们的分析为进一步调查确定了六组基准:Word Scrambling, Reading Componsectionon(QuAC, SQuAD2, DROP),PIQA,Winograd,语言建模任务(Wikitext任务, 1BW),德语至英语31
<a id="S0519"></a> Source: p.32 S0519
Original: Figure 4.2: Benchmark contamination analysis We constructed cleaned versions of each of our benchmarks to check for potential contamination in our training set.
中文: 图4.2:基准污染分析 我们建造了我们每个基准的清理版,以检查我们训练中的潜在污染。
<a id="S0520"></a> Source: p.32 S0520
Original: The x-axis is a conservative lower bound for how much of the dataset is known with high confidence to be clean, and the y-axis shows the difference in performance when evaluating only on the verified clean subset.
中文: x轴是一个保守的下方绑定值,用于确定有多少数据集以高度的置信性来知道是否干净,而y轴仅在对已核实的干净子集进行评价时显示性能的差异.
<a id="S0521"></a> Source: p.32 S0521
Original: Performance on most benchmarks changed negligibly, but some were flagged for further review.
中文: 多数基准的业绩变化甚微,但有些基准被标出供进一步审查。
<a id="S0522"></a> Source: p.32 S0522
Original: On inspection we find some evidence for contamination of the PIQA and Winograd results, and we mark the corresponding results in Section 3 with an asterisk.
中文: 在检查中,我们发现了一些污染PIQA和Winograd结果的证据,我们在第3节中用星号标出相应的结果。
<a id="S0523"></a> Source: p.32 S0523
Original: We find no evidence that other benchmarks are affected. translation.
中文: 我们没有发现其他基准受到影响的证据。 翻译。
<a id="S0524"></a> Source: p.32 S0524
Original: Since our overlap analysis is designed to be extremely conservative, we expect it to produce some false positives.
中文: 由于我们的重叠分析被设计成极其保守,我们期望它会产生一些虚假的阳性.
<a id="S0525"></a> Source: p.32 S0525
Original: We summarize the results for each group of tasks below: • Reading Comprehension: Our initial analysis flagged >90% of task examples from QuAC, SQuAD2, and DROP as potentially contaminated, so large that even measuring the differential on a clean subset was difficult.
中文: 我们总结以下每组任务的结果: 阅读理解:我们的初步分析将QuAC,SQuAD2和DROP中90%的任务实例标出为可能受污染,如此之大,以至于即使测量干净子集上的差分也是困难的.
<a id="S0526"></a> Source: p.32 S0526
Original: Upon manual inspection, however, we found that for every overlap we inspected, in all 3 datasets, the source text was present in our training data but the question/answer pairs were not, meaning the model gains only background information and cannot memorize the answer to a specific question. • German translation: We found 25% of the examples in the WMT16 German-English test set were marked as potentially contaminated, with an associated total effect size of 1-2 BLEU.
中文: 然而,经人工检查,我们发现,我们检查的每一个重叠之处,在所有三个数据集中,源文本都存在于我们的培训数据中,但问答对不是,这意味着模型只获得背景资料,无法记住对具体问题的答案。 • 德文翻译:我们发现WMT16德文-英文测试集中25%的例子被标记为可能受污染,相关总效果为1-2 BLEU.
<a id="S0527"></a> Source: p.32 S0527
Original: Upon inspection, none of the flagged examples contain paired sentences resembling NMT training data and collisions were monolingual matches mostly of snippets of events discussed in the news. • Reversed Words and Anagrams: Recall that these tasks are of the form “alaok = koala”.
中文: 经检查后,标出的例子中没有任何一对句子与NMT培训数据相仿,碰撞是单一语言的匹配,大多是新闻所讨论事件的片断。 • 反向文字和图表:回顾这些任务的形式是“alaok = koala”。
<a id="S0528"></a> Source: p.32 S0528
Original: Due to the short length of these tasks, we used 2-grams for filtering (ignoring punctuation).
中文: 由于这些任务的长度很短,我们使用2克来进行过滤(ignoring punctuation).
<a id="S0529"></a> Source: p.32 S0529
Original: After inspecting the flagged overlaps, we found that they were not typically instances of real reversals or unscramblings in the training set, but rather palindromes or trivial unscramblings, e.g “kayak = kayak”.
中文: 在检查了标记的重叠之后,我们发现,它们通常不是训练场中真正的倒转或无所事事的情况,而是平地或微不足道的无所事事的情况,例如“kayak = sickyak”。
<a id="S0530"></a> Source: p.32 S0530
Original: The amount of overlap was small, but removing the trivial tasks lead to an increase in difficulty and thus a spurious signal.
中文: 重叠量很小,但去除微不足道的任务导致难度增加,从而产生虚假信号.
<a id="S0531"></a> Source: p.32 S0531
Original: Related to this, the symbol insertion task shows high overlap but no effect on performance – this is because that task involves removing non-letter characters from a word, and the overlap analysis itself ignores such characters, leading to many spurious matches. • PIQA: The overlap analysis flagged 29% of examples as contaminated, and observed a 3 percentage point absolute decrease (4% relative decrease) in performance on the clean subset.
中文: 与此相关,符号插入任务显示高度重叠,但对性能没有影响——这是因为该任务涉及从一个单词中去除非字母字符,而重叠分析本身忽略了这些字符,导致许多虚假的匹配. • PIQA:重叠分析将29%的例子标出为被污染,观察到清洁子集的性能绝对下降3个百分点(相对下降4%)。
<a id="S0532"></a> Source: p.32 S0532
Original: Though the test dataset was released after our training set was created and its labels are hidden, some of the web pages used by the crowdsourced dataset creators are contained in our training set.
中文: 虽然测试数据集是在我们的培训集创建后发布的,并且其标签被隐藏起来,但众源数据集创建者使用的一些网页却包含在我们的培训集中.
<a id="S0533"></a> Source: p.32 S0533
Original: We found a similar decrease in a 25x smaller model with much less capacity to memorize, leading us to suspect that the shift is likely statistical bias rather than memorization; examples which workers copied may simply be easier.
中文: 我们发现一个25x较小的模型也有类似的下降,其记忆能力要小得多,导致我们怀疑这一转变很可能是统计偏差而不是记忆;工人抄袭的例子可能很简单。
<a id="S0534"></a> Source: p.32 S0534
Original: Unfortunately, we cannot rigorously prove this hypothesis.
中文: 不幸的是,我们不能严格证明这一假设。
<a id="S0535"></a> Source: p.32 S0535
Original: We therefore mark our PIQA results with an asterisk to denote this potential contamination. • Winograd: The overlap analysis flagged 45% of examples, and found a 2.6% decrease in performance on the clean subset.
中文: 因此,我们用星号标出我们的PIQA结果,以表示这种潜在污染。 • Winograd:重叠分析标出45%的例子,发现干净子集的性能下降2.6%。
<a id="S0536"></a> Source: p.32 S0536
Original: Manual inspection of the overlapping data point showed that 132 Winograd schemas were in fact present in our training set, though presented in a different format than we present the task to the model.
中文: 对重叠数据点的人工检查表明,我们的培训中实际上有132个Winograd计划,尽管其形式与我们向模型介绍的任务不同。
<a id="S0537"></a> Source: p.32 S0537
Original: Although the decrease in performance is small, we mark our Winograd results in the main paper with an asterisk. 32
中文: 虽然性能的下降幅度很小,但我们的"维诺格勒"(Winograd)结果在正文中以星号标出. 第32条
<a id="S0538"></a> Source: p.33 S0538
Original: • Language modeling: We found the 4 Wikipedia language modeling benchmarks measured in GPT-2, plus the Children’s Book Test dataset, to be almost entirely contained in our training data.
中文: 语言建模:我们发现GPT-2中测量的4个维基百科语言建模基准加上儿童图书测试数据集,几乎完全包含在我们的培训数据中.
<a id="S0539"></a> Source: p.33 S0539
Original: Since we cannot reliably extract a clean subset here, we do not report results on these datasets, even though we intended to when starting this work.
中文: 由于我们无法可靠地从这里提取一个干净的子集,因此我们不报告这些数据集的结果,尽管我们打算在开始这项工作时这样做。
<a id="S0540"></a> Source: p.33 S0540
Original: We note that Penn Tree Bank due to its age was unaffected and therefore became our chief language modeling benchmark.
中文: 我们注意到,宾树银行由于其年代久远而未受影响,因此成为我们的主要语言模型基准。
<a id="S0541"></a> Source: p.33 S0541
Original: We also inspected datasets where contamination was high, but the impact on performance was close to zero, simply to verify how much actual contamination existed.
中文: 我们还检查了污染程度高的数据集,但对性能的影响已接近于零,只是核实实际存在多少污染。
<a id="S0542"></a> Source: p.33 S0542
Original: These appeared to often contain false positives.
中文: 这些似乎往往含有虚假的阳性。
<a id="S0543"></a> Source: p.33 S0543
Original: They had either no actual contamination, or had contamination that did not give away the answer to the task.
中文: 他们要么没有受到实际污染,要么受到污染,因此无法解答这项任务。
<a id="S0544"></a> Source: p.33 S0544
Original: One notable exception was LAMBADA, which appeared to have substantial genuine contamination, yet the impact on performance was very small, with the clean subset scoring within 0.5% of the full dataset.
中文: 一个显著的例外是LAMBADA,它似乎具有很大的真正的污染,但对性能的影响很小,干净的子集得分在全部数据集的0.5%以内。
<a id="S0545"></a> Source: p.33 S0545
Original: Also, strictly speaking, our fill-in-the-blank format precludes the simplest form of memorization.
中文: 另外,严格地说,我们的填充形式排除了最简单的记忆形式。
<a id="S0546"></a> Source: p.33 S0546
Original: Nevertheless, since we made very large gains on LAMBADA in this paper, the potential contamination is noted in the results section.
中文: 然而,由于我们在本文中对LAMBADA取得了很大进展,结果部分指出了潜在的污染。
<a id="S0547"></a> Source: p.33 S0547
Original: An important limitation of our contamination analysis is that we cannot be sure that the clean subset is drawn from the same distribution as the original dataset.
中文: 我们污染分析的一个重要局限是,我们无法确定干净的子集来自与原始数据集相同的分布.
<a id="S0548"></a> Source: p.33 S0548
Original: It remains possible that memorization inflates results but at the same time is precisely counteracted by some statistical bias causing the clean subset to be easier.
中文: 记忆膨胀的结果仍然是可能的,但与此同时,由于某些统计偏差,使干净的子集更容易被抵消。
<a id="S0549"></a> Source: p.33 S0549
Original: However, the sheer number of shifts close to zero suggests this is unlikely, and we also observed no noticeable difference in the shifts for small models, which are unlikely to be memorizing.
中文: 然而,接近于零的班次数量表明,这是不可能的,我们还注意到小型型号的班次没有明显的差别,它们不可能被记起。
<a id="S0550"></a> Source: p.33 S0550
Original: Overall, we have made a best effort to measure and document the effects of data contamination, and to note or outright remove problematic results, depending on the severity.
中文: 总体而言,我们已尽最大努力衡量和记录数据污染的影响,并根据严重程度注意到或彻底消除有问题的结果。
<a id="S0551"></a> Source: p.33 S0551
Original: Much work remains to be done to address this important and subtle issue for the field in general, both when designing benchmarks and when training models.
中文: 在制定基准和培训模式时,还需要做很多工作,才能解决这个对一般领域来说重要而微妙的问题。
<a id="S0552"></a> Source: p.33 S0552
Original: For a more detailed explanation of our analysis, we refer the reader to Appendix C. 5 Limitations GPT-3 and our analysis of it have a number of limitations.
中文: 为了更详细地解释我们的分析,我们请读者参阅附录C. 5 限制 GBT-3,我们对它的分析有一些局限性。
<a id="S0553"></a> Source: p.33 S0553
Original: Below we describe some of these and suggest directions for future work.
中文: 下面我们介绍其中一些建议,并提出今后工作的方向。
<a id="S0554"></a> Source: p.33 S0554
Original: First, despite the strong quantitative and qualitative improvements of GPT-3, particularly compared to its direct predecessor GPT-2, it still has notable weaknesses in text synthesis and several NLP tasks.
中文: 首先,尽管GPT-3在数量和质量上有很大的改进,特别是与其直接的前身GPT-2相比,它仍然在文本合成和几个NLP任务方面有明显的弱点.
<a id="S0555"></a> Source: p.33 S0555
Original: On text synthesis, although the overall quality is high, GPT-3 samples still sometimes repeat themselves semantically at the document level, start to lose coherence over sufficiently long passages, contradict themselves, and occasionally contain non-sequitur sentences or paragraphs.
中文: 在文本合成方面,虽然总体质量很高,但GPT-3样本仍然有时在文档层面重复地重复自己,在足够长的段落中开始失去一致性,自相矛盾,偶尔还包含非等分句或段落.
<a id="S0556"></a> Source: p.33 S0556
Original: We will release a collection of 500 uncurated unconditional samples to help provide a better sense of GPT-3’s limitations and strengths at text synthesis.
中文: 我们将发布500个未经检验的无条件样本,以帮助人们更好地了解GPT-3在文本合成方面的局限性和长处。
<a id="S0557"></a> Source: p.33 S0557
Original: Within the domain of discrete language tasks, we have noticed informally that GPT-3 seems to have special difficulty with “common sense physics”, despite doing well on some datasets (such as PIQA [BZB+19]) that test this domain.
中文: 在离散语言任务领域,我们非正式地注意到GPT-3在"常识物理学"方面似乎有特殊困难,尽管在一些测试这个领域的数据集(如PIQA[BZB+19])上做得很好.
<a id="S0558"></a> Source: p.33 S0558
Original: Specifically GPT-3 has difficulty with questions of the type “If I put cheese into the fridge, will it melt?”.
中文: 具体地说,GPT-3对“如果我把起司放入冰箱,它会融化吗?”
<a id="S0559"></a> Source: p.33 S0559
Original: Quantitatively, GPT-3’s in-context learning performance has some notable gaps on our suite of benchmarks, as described in Section 3, and in particular it does little better than chance when evaluated one-shot or even few-shot on some “comparison” tasks, such as determining if two words are used the same way in a sentence, or if one sentence implies another (WIC and ANLI respectively), as well as on a subset of reading comprehension tasks.
中文: 从数量上看,GPT-3的内文学习成绩在我们第3节所述的一整套基准上有一些显著的差距,特别是当对某些 " 比较 " 任务进行一弹一弹甚至几弹一弹评估时,例如确定一个句子中是否同样使用两个字,或者一个句子是否意味着另一个字(分别为WIC和ANLI),以及一个阅读理解任务的子集时,这一成绩并不比机会好。
<a id="S0560"></a> Source: p.33 S0560
Original: This is especially striking given GPT-3’s strong few-shot performance on many other tasks.
中文: 鉴于GPT-3在许多其他任务上的强力出手,
<a id="S0561"></a> Source: p.33 S0561
Original: GPT-3 has several structural and algorithmic limitations, which could account for some of the issues above.
中文: GPT-3有一些结构和算法上的局限性,这可以说明上面的一些问题.
<a id="S0562"></a> Source: p.33 S0562
Original: We focused on exploring in-context learning behavior in autoregressive language models because it is straightforward to both sample and compute likelihoods with this model class.
中文: 我们专注于探索在自递性语言模型中的文本内学习行为,因为它既可以直截了当地抽样,也可以与这个模型类计算出可能性.
<a id="S0563"></a> Source: p.33 S0563
Original: As a result our experiments do not include any bidirectional architectures or other training objectives such as denoising.
中文: 因此,我们的实验不包括任何双向架构或其他培训目标,如去名.
<a id="S0564"></a> Source: p.33 S0564
Original: This is a noticeable difference from much of the recent literature, which has documented improved fine-tuning performance when using these approaches over standard language models [RSR+19].
中文: 与最近许多文献相比,这是一个显著的区别,这些文献记录了在使用这些方法时比标准语言模型[RSR+19]改进的微调性能.
<a id="S0565"></a> Source: p.33 S0565
Original: Thus our design decision comes at the cost of potentially worse performance on tasks which empirically benefit from bidirectionality.
中文: 因此,我们的设计决定是以在经验上受益于双向性的任务方面可能表现更差为代价的。
<a id="S0566"></a> Source: p.33 S0566
Original: This may include fill-in-the-blank tasks, tasks that involve looking back and comparing two pieces of content, or tasks that require re-reading or carefully considering a long passage and then generating a very short answer.
中文: 这可能包括:填充式的任务,涉及回想和比较两个内容的任务,或者需要重新阅读或仔细考虑一个很长的段落,然后产生非常短的答案的任务.
<a id="S0567"></a> Source: p.33 S0567
Original: This could be a possible explanation for GPT-3’s lagging few-shot performance on a few of the tasks, such as WIC (which involves comparing the use of a word in two sentences), ANLI (which involves comparing two sentences to see if one implies the other), and several reading comprehension tasks (e.g.
中文: 这可能是GPT-3在一些任务上落后的几发性能的一个可能解释,例如WIC(涉及比较两句中一个单词的使用),ANLI(涉及比较两句以查看一句是否意指一句),以及几个阅读理解任务(例如.
<a id="S0568"></a> Source: p.33 S0568
Original: We also conjecture, based on past literature, that a large bidirectional model would be stronger at fine-tuning than GPT-3.
中文: 我们还根据过去的文献推测,大型双向模型在微调时会比GPT-3更强.
<a id="S0569"></a> Source: p.33 S0569
Original: Making a bidirectional model at the scale of GPT-3, and/or trying to make bidirectional models work with few- or zero-shot learning, is a promising direction for future research, and could help achieve the “best of both worlds”. A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the 33
中文: 使双向模型在GPT-3的规模上发挥双向作用和/或试图使双向模型在少有或零发的学习中发挥作用,是未来研究的一个很有希望的方向,可以帮助实现“两个世界的最佳”目标。 本文所描述的一般方法 -- -- 扩大任何类似LM的模型,无论是自递式模型还是双向模型 -- -- 的一个更根本的局限性是,它最终可能遇到(或可能已经碰到)33的限度。
<a id="S0570"></a> Source: p.34 S0570
Original: Our current objective weights every token equally and lacks a notion of what is most important to predict and what is less important. [RRS20] demonstrate benefits of customizing prediction to entities of interest.
中文: 我们目前的目标对每一个象征都一视同仁,并且缺乏对什么是最重要的、什么是不重要的概念。 [RRS20]证明将预测量身定做给感兴趣的实体带来的益处。
<a id="S0571"></a> Source: p.34 S0571
Original: Also, with self-supervised objectives, task specification relies on forcing the desired task into a prediction problem, whereas ultimately, useful language systems (for example virtual assistants) might be better thought of as taking goal-directed actions rather than just making predictions.
中文: 此外,由于有自我监督的目标,任务规格取决于将预期任务逼入一个预测问题,而最终,有用的语言系统(例如虚拟助理)可能更能被认为是采取目标导向的行动,而不仅仅是预测。
<a id="S0572"></a> Source: p.34 S0572
Original: Finally, large pretrained language models are not grounded in other domains of experience, such as video or real-world physical interaction, and thus lack a large amount of context about the world [BHT+20].
中文: 最后,大型的预训语言模型并非基于其他经验领域,如视频或现实世界的物理互动,因此缺乏大量关于世界的背景[BHT+20].
<a id="S0573"></a> Source: p.34 S0573
Original: For all these reasons, scaling pure self-supervised prediction is likely to hit limits, and augmentation with a different approach is likely to be necessary.
中文: 由于所有这些原因,缩小纯粹的自我监督的预测可能达到极限,可能需要采用不同的方法来进行增强。
<a id="S0574"></a> Source: p.34 S0574
Original: Promising future directions in this vein might include learning the objective function from humans [ZSW+19a], fine-tuning with reinforcement learning, or adding additional modalities such as images to provide grounding and a better model of the world [CLY+19].
中文: 这种脉络未来有希望的方向可能包括:从人类学习客观功能[ZSW+19a],用强化学习来微调,或者增加图像等额外模式,以提供地基和更好的世界模型[CLY+19].
<a id="S0575"></a> Source: p.34 S0575
Original: Another limitation broadly shared by language models is poor sample efficiency during pre-training.
中文: 语言模型广泛认同的另一个局限性是培训前的样本效率低下.
<a id="S0576"></a> Source: p.34 S0576
Original: While GPT-3 takes a step towards test-time sample efficiency closer to that of humans (one-shot or zero-shot), it still sees much more text during pre-training than a human sees in the their lifetime [Lin20].
中文: 虽然GPT-3向试验时间的样本效率迈出了更接近人类(一发或零发)的一步,但在前期训练期间,它仍然比人类一生中看到的更能看到文字[Lin20].
<a id="S0577"></a> Source: p.34 S0577
Original: Improving pre-training sample efficiency is an important direction for future work, and might come from grounding in the physical world to provide additional information, or from algorithmic improvements. A limitation, or at least uncertainty, associated with few-shot learning in GPT-3 is ambiguity about whether few-shot learning actually learns new tasks “from scratch” at inference time, or if it simply recognizes and identifies tasks that it has learned during training.
中文: 提高培训前的样本效率是未来工作的一个重要方向,可能来自在物理世界打地以提供补充资料,或来自算法改进。 与GPT-3的少发性学习有关的限制,或至少是不确定性,是对少发性学习是否真正在推断时间“从零开始”学习出新任务,或只是承认和确定在培训期间学到的任务的模糊不清。
<a id="S0578"></a> Source: p.34 S0578
Original: These possibilities exist on a spectrum, ranging from demonstrations in the training set that are drawn from exactly the same distribution as those at test time, to recognizing the same task but in a different format, to adapting to a specific style of a general task such as QA, to learning a skill entirely de novo.
中文: 这些可能性存在于一个范围上,从从与试验时完全相同的分布所抽取的训练场演示到承认同样的任务但形式不同,再到适应诸如QA等一般任务的具体风格,再到完全重新学习技能。
<a id="S0579"></a> Source: p.34 S0579
Original: Where GPT-3 is on this spectrum may also vary from task to task.
中文: 如果GPT-3处于这一范围,任务也可能因任务而异.
<a id="S0580"></a> Source: p.34 S0580
Original: Synthetic tasks such as wordscrambling or defining nonsense words seem especially likely to be learned de novo, whereas translation clearly must be learned during pretraining, although possibly from data that is very different in organization and style than the test data.
中文: 诸如文字拼接或定义无稽之谈等合成任务似乎特别有可能被重新学习,而翻译显然必须在预训期间学习,尽管可能来自在组织和风格上与测试数据截然不同的数据。
<a id="S0581"></a> Source: p.34 S0581
Original: Ultimately, it is not even clear what humans learn from scratch vs from prior demonstrations.
中文: 最终,人类从头到尾从先前的示威活动中学到什么甚至不清楚。
<a id="S0582"></a> Source: p.34 S0582
Original: Even organizing diverse demonstrations during pre-training and identifying them at test time would be an advance for language models, but nevertheless understanding precisely how few-shot learning works is an important unexplored direction for future research. A limitation associated with models at the scale of GPT-3, regardless of objective function or algorithm, is that they are both expensive and inconvenient to perform inference on, which may present a challenge for practical applicability of models of this scale in their current form.
中文: 即使是在预训期间组织各种示范活动,并在测试时加以识别,也是语言模型的一个进步,但确切地理解了几发学习如何是未来研究的重要未探索的方向。 与GPT-3规模模型相关的一个限制,无论客观功能或算法如何,它们既昂贵又不方便进行推断,这可能会对目前形式的这种规模模型的实际适用构成挑战.
<a id="S0583"></a> Source: p.34 S0583
Original: One possible future direction to address this is distillation [HVD15] of large models down to a manageable size for specific tasks.
中文: 今后解决这一问题的一个可能方向是将大型模型蒸馏出[HVD15],缩小到特定任务的可管理规模。
<a id="S0584"></a> Source: p.34 S0584
Original: Large models such as GPT-3 contain a very wide range of skills, most of which are not needed for a specific task, suggesting that in principle aggressive distillation may be possible.
中文: GPT-3等大型型号包含非常广泛的技能范围,大多数不需要用于特定任务,这表明原则上可以进行有侵略性的蒸馏.
<a id="S0585"></a> Source: p.34 S0585
Original: Distillation is well-explored in general [LHCG19a] but has not been tried at the scale of hundred of billions parameters; new challenges and opportunities may be associated with applying it to models of this size.
中文: 蒸馏在总体上得到了很好的探索[LHCG19a],但还没有以上千亿参数的规模进行尝试;将蒸馏应用到这种规模的模型中可能会带来新的挑战和机会.
<a id="S0586"></a> Source: p.34 S0586
Original: Finally, GPT-3 shares some limitations common to most deep learning systems – its decisions are not easily interpretable, it is not necessarily well-calibrated in its predictions on novel inputs as observed by the much higher variance in performance than humans on standard benchmarks, and it retains the biases of the data it has been trained on.
中文: 最后,GPT-3与大多数深层学习系统一样有一些共同的局限性 — — 其决策不易解释,在对新投入的预测中不一定能很好地校正,在标准基准上的绩效差异比人类要高得多,它保留了所培训的数据的偏差。
<a id="S0587"></a> Source: p.34 S0587
Original: This last issue – biases in the data that may lead the model to generate stereotyped or prejudiced content – is of special concern from a societal perspective, and will be discussed along with other issues in the next section on Broader Impacts (Section 6). 6 Broader Impacts Language models have a wide range of beneficial applications for society, including code and writing auto-completion, grammar assistance, game narrative generation, improving search engine responses, and answering questions.
中文: 最后一个问题 -- -- 数据中的偏见可能导致模型产生陈规定型或有偏见的内容 -- -- 从社会角度看,这个问题特别令人关注,将在下一节 " 更广泛影响 " 中与其他问题一并讨论(第6节)。 6 Broad Effects Language models 对社会具有广泛的有益应用,包括代码和写作自动补全,语法协助,游戏叙事生成,改进搜索引擎响应,以及回答问题等.
<a id="S0588"></a> Source: p.34 S0588
Original: But they also have potentially harmful applications.
中文: 但它们也有潜在的有害应用.
<a id="S0589"></a> Source: p.34 S0589
Original: GPT-3 improves the quality of text generation and adaptability over smaller models and increases the difficulty of distinguishing synthetic text from human-written text.
中文: GPT-3比更小的模型提高了文本生成和适应性的质量,并增加了将合成文本与人文文字区分开来的困难.
<a id="S0590"></a> Source: p.34 S0590
Original: It therefore has the potential to advance both the beneficial and harmful applications of language models.
中文: 因此,它有可能促进语言模式的有益和有害应用。
<a id="S0591"></a> Source: p.34 S0591
Original: Here we focus on the potential harms of improved language models, not because we believe the harms are necessarily greater, but in order to stimulate efforts to study and mitigate them.
中文: 在这里,我们侧重于改进后的语言模型的潜在危害,不是因为我们认为这些危害必然会更大,而是为了刺激研究和减轻这些危害的努力.
<a id="S0592"></a> Source: p.34 S0592
Original: The broader impacts of language models like this are numerous.
中文: 类似语言模型的广泛影响是众多的.
<a id="S0593"></a> Source: p.34 S0593
Original: We focus on two primary issues: the potential for deliberate misuse of language models like GPT-3 in Section 6.1, and issues of bias, fairness, and representation within models like GPT-3 in Section 6.2.
中文: 我们侧重于两个主要问题:第6.1节中故意滥用GPT-3等语言模式的可能性;第6.2节中GPT-3等模式中的偏见、公平和代表性问题。
<a id="S0594"></a> Source: p.34 S0594
Original: We also briefly discuss issues of energy efficiency (Section 6.3). 34
中文: 我们还简要地讨论了能源效率问题(第6.3节)。 页:1
<a id="S0595"></a> Source: p.35 S0595
Original: 6.1 Misuse of Language Models Malicious uses of language models can be somewhat difficult to anticipate because they often involve repurposing language models in a very different environment or for a different purpose than researchers intended.
中文: 6.1 滥用语言模型对语言模型的恶意使用可能有些难以预料,因为它们常常涉及在非常不同的环境中重新使用语言模型,或者为了与研究者预期的不同的目的.
<a id="S0596"></a> Source: p.35 S0596
Original: To help with this, we can think in terms of traditional security risk assessment frameworks, which outline key steps such as identifying threats and potential impacts, assessing likelihood, and determining risk as a combination of likelihood and impact [Ros12].
中文: 为了帮助做到这一点,我们可以考虑传统的安全风险评估框架,该框架概述了关键步骤,例如查明威胁和潜在影响、评估可能性和确定风险,作为可能性和影响[罗斯12]的组合。
<a id="S0597"></a> Source: p.35 S0597
Original: We discuss three factors: potential misuse applications, threat actors, and external incentive structures. 6.1.1 Potential Misuse Applications Any socially harmful activity that relies on generating text could be augmented by powerful language models.
中文: 我们讨论了三个因素:潜在滥用应用、威胁行为者和外部奖励结构。 6.1.1 可能的滥用申请 任何依赖生成文本的社会有害活动,都可以被强大的语言模式所强化。
<a id="S0598"></a> Source: p.35 S0598
Original: Examples include misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting.
中文: 例子包括错误信息、垃圾邮件、打字、滥用法律和政府程序、欺诈性学术作文和社会工程借口。
<a id="S0599"></a> Source: p.35 S0599
Original: Many of these applications bottleneck on human beings to write sufficiently high quality text.
中文: 这些应用中有许多在人类上瓶颈地写出足够高质量的文字.
<a id="S0600"></a> Source: p.35 S0600
Original: Language models that produce high quality text generation could lower existing barriers to carrying out these activities and increase their efficacy.
中文: 产生高质量文本的语言模型可以降低开展这些活动的现有障碍并提高其效力。
<a id="S0601"></a> Source: p.35 S0601
Original: The misuse potential of language models increases as the quality of text synthesis improves.
中文: 随着文本综合质量的提高,语言模型被滥用的潜力会增加.
<a id="S0602"></a> Source: p.35 S0602
Original: The ability of GPT-3 to generate several paragraphs of synthetic content that people find difficult to distinguish from human-written text in 3.9.4 represents a concerning milestone in this regard. 6.1.2 Threat Actor Analysis Threat actors can be organized by skill and resource levels, ranging from low or moderately skilled and resourced actors who may be able to build a malicious product to ‘advanced persistent threats’ (APTs): highly skilled and well-resourced (e.g. state-sponsored) groups with long-term agendas [SBC+19].
中文: 在3.9.4中,GPT-3能够生成合成内容的几段,人们难以与人文文字相区别,这是这方面的一个里程碑。 6.1.2 威胁行为者可以通过技能和资源水平来组织威胁行为者,从能够制造出恶意产品的低技能或中等技能和资源的行为者到 " 强化持续威胁 " (APTs):具有长期议程的高度技能和资源充足的(例如国家赞助的)团体[SBC+19]。
<a id="S0603"></a> Source: p.35 S0603
Original: To understand how low and mid-skill actors think about language models, we have been monitoring forums and chat groups where misinformation tactics, malware distribution, and computer fraud are frequently discussed.
中文: 为了了解低技能者和中技能者如何看待语言模式,我们一直在监测经常讨论错误信息策略、恶意软件发布和计算机欺诈的论坛和聊天小组。
<a id="S0604"></a> Source: p.35 S0604
Original: While we did find significant discussion of misuse following the initial release of GPT-2 in spring of 2019, we found fewer instances of experimentation and no successful deployments since then.
中文: 虽然我们在2019年春季首次发布GPT-2后确实发现大量关于滥用的讨论,但我们发现实验案例较少,此后没有成功部署.
<a id="S0605"></a> Source: p.35 S0605
Original: Additionally, those misuse discussions were correlated with media coverage of language model technologies.
中文: 此外,这些滥用问题的讨论与媒体对语言模型技术的报道有关。
<a id="S0606"></a> Source: p.35 S0606
Original: From this, we assess that the threat of misuse from these actors is not immediate, but significant improvements in reliability could change this.
中文: 从这一点出发,我们评估说,这些行为者的滥用威胁不是立即发生的,但可靠性方面的重大改进可以改变这种情况。
<a id="S0607"></a> Source: p.35 S0607
Original: Because APTs do not typically discuss operations in the open, we have consulted with professional threat analysts about possible APT activity involving the use of language models.
中文: 由于APT一般不公开讨论操作,所以我们曾与专业威胁分析师就APT可能涉及使用语言模型的活动进行协商.
<a id="S0608"></a> Source: p.35 S0608
Original: Since the release of GPT-2 there has been no discernible difference in operations that may see potential gains by using language models.
中文: 自GPT-2发布后,在操作上没有明显区别,可能通过使用语言模型看到潜在的收益.
<a id="S0609"></a> Source: p.35 S0609
Original: The assessment was that language models may not be worth investing significant resources in because there has been no convincing demonstration that current language models are significantly better than current methods for generating text, and because methods for “targeting” or “controlling” the content of language models are still at a very early stage. 6.1.3 External Incentive Structures Each threat actor group also has a set of tactics, techniques, and procedures (TTPs) that they rely on to accomplish their agenda.
中文: 评估认为,语言模型可能不值得投入大量资源,因为没有令人信服的证据表明,目前的语言模型比目前生成文本的方法要好得多,而且“瞄准”或“控制”语言模型内容的方法还处于非常早期阶段。 6.1.3 外部奖励结构 每个威胁行为者团体还拥有一套战术、技术和程序,是他们完成议程所依赖的。
<a id="S0610"></a> Source: p.35 S0610
Original: TTPs are influenced by economic factors like scalability and ease of deployment; phishing is extremely popular among all groups because it offers a low-cost, low-effort, high-yield method of deploying malware and stealing login credentials.
中文: TTP受到可伸缩性和部署方便等经济因素的影响;钓鱼在所有群体中极为受欢迎,因为它提供了一种低成本,低功率,高产的部署恶意软件和偷取登录证书的方法.
<a id="S0611"></a> Source: p.35 S0611
Original: Using language models to augment existing TTPs would likely result in an even lower cost of deployment.
中文: 使用语言模型来增加现有的技术贸易点,可能会使部署费用更低。
<a id="S0612"></a> Source: p.35 S0612
Original: Ease of use is another significant incentive.
中文: 方便使用是另一个重要的激励因素。
<a id="S0613"></a> Source: p.35 S0613
Original: Having stable infrastructure has a large impact on the adoption of TTPs.
中文: 拥有稳定的基础设施对采用贸易点具有重大影响。
<a id="S0614"></a> Source: p.35 S0614
Original: The outputs of language models are stochastic, however, and though developers can constrain these (e.g. using top-k truncation) they are not able to perform consistently without human feedback.
中文: 然而,语言模型的输出具有斯多克性,虽然开发者可以约束这些(例如使用上克截取),但如果没有人类的反馈,他们无法一致地进行.
<a id="S0615"></a> Source: p.35 S0615
Original: If a social media disinformation bot produces outputs that are reliable 99% of the time, but produces incoherent outputs 1% of the time, this could reduce the amount of human labor required in operating this bot.
中文: 如果一个社交媒体虚假的bot产生99%的可靠时间输出,但产生1%的时间不连贯的输出,这可以减少操作此bot所需的人劳力.
<a id="S0616"></a> Source: p.35 S0616
Original: But a human is still needed to filter the outputs, which restricts how scalable the operation can be.
中文: 但是仍然需要一个人来过滤输出,这限制了操作的可扩展性.
<a id="S0617"></a> Source: p.35 S0617
Original: Based on our analysis of this model and analysis of threat actors and the landscape, we suspect AI researchers will eventually develop language models that are sufficiently consistent and steerable that they will be of greater interest to malicious actors.
中文: 根据我们对这个模型的分析以及对威胁行为者和地貌的分析,我们怀疑AI研究人员最终会开发出足够一致和可引导的语言模型,从而使恶意行为者更感兴趣.
<a id="S0618"></a> Source: p.35 S0618
Original: We expect this will introduce challenges for the broader research community, and hope to work on this through a combination of mitigation research, prototyping, and coordinating with other technical developers. 35
中文: 我们期望这将给更广泛的研究界带来挑战,并希望通过将缓解研究、原型化以及与其他技术开发者协调结合起来来解决这一问题。 35个
<a id="S0619"></a> Source: p.36 S0619
Original: 6.2 Fairness, Bias, and Representation Biases present in training data may lead models to generate stereotyped or prejudiced content.
中文: 6.2 培训数据中的公平、偏见和代表性偏见可能导致产生定型或有偏见内容的模式。
<a id="S0620"></a> Source: p.36 S0620
Original: This is concerning, since model bias could harm people in the relevant groups in different ways by entrenching existing stereotypes and producing demeaning portrayals amongst other potential harms [Cra17].
中文: 与此有关,因为模式偏见可能以不同的方式伤害相关群体的人,其方式包括巩固现有的定型观念并产生贬低性形象等潜在伤害[Cra17]。
<a id="S0621"></a> Source: p.36 S0621
Original: We have conducted an analysis of biases in the model in order to better understand GPT-3’s limitations when it comes to fairness, bias, and representation. 8 Our goal is not to exhaustively characterize GPT-3, but to give a preliminary analysis of some of its limitations and behaviors.
中文: 我们分析了模型中的偏差,以便更好地了解GPT-3在公平、偏差和代表性方面的局限性。 第8条 我们的目标不是详尽地描述GPT-3,而是对其一些局限性和行为进行初步分析.
<a id="S0622"></a> Source: p.36 S0622
Original: We focus on biases relating to gender, race, and religion, although many other categories of bias are likely present and could be studied in follow-up work.
中文: 我们侧重于与性别、种族和宗教有关的偏见,尽管可能存在许多其他类型的偏见,可以在后续工作中加以研究。
<a id="S0623"></a> Source: p.36 S0623
Original: This is a preliminary analysis and does not reflect all of the model’s biases even within the studied categories.
中文: 这是初步分析,没有反映模型的所有偏见,即使在所研究的类别中也是如此。
<a id="S0624"></a> Source: p.36 S0624
Original: Broadly, our analysis indicates that internet-trained models have internet-scale biases; models tend to reflect stereotypes present in their training data.
中文: 总的来说,我们的分析表明,经过互联网培训的模型具有互联网规模的偏见;模型往往反映其培训数据中的陈规定型观念。
<a id="S0625"></a> Source: p.36 S0625
Original: Below we discuss our preliminary findings of bias along the dimensions of gender, race, and religion.
中文: 下面我们讨论我们在性别、种族和宗教方面存在偏见的初步调查结果。
<a id="S0626"></a> Source: p.36 S0626
Original: We probe for bias in the 175 billion parameter model and also in similar smaller models, to see if and how they are different in this dimension. 6.2.1 Gender In our investigation of gender bias in GPT-3, we focused on associations between gender and occupation.
中文: 我们探索了175亿参数模型中的偏差,以及类似的更小模型,看看它们在这个维度上是否和如何不同. 6.2.1 性别 我们在调查GPT-3中的性别偏见时,侧重于性别与职业之间的联系。
<a id="S0627"></a> Source: p.36 S0627
Original: We found that occupations in general have a higher probability of being followed by a male gender identifier than a female one (in other words, they are male leaning) when given a context such as "The {occupation} was a" (Neutral Variant). 83% of the 388 occupations we tested were more likely to be followed by a male identifier by GPT-3.
中文: 我们发现,在诸如“{职业}是a'(Neutral变体)这样的上下文中,一般职业中男性的性别识别特征比女性更可能遵循(换言之,它们是男性倾斜)。 我们测试的388个职业中,83%更可能使用GPT-3的男性标识。
<a id="S0628"></a> Source: p.36 S0628
Original: We measured this by feeding the model a context such as "The detective was a" and then looking at the probability of the model following up with male indicating words (eg. man, male etc.) or female indicating words (woman, female etc.).
中文: 我们通过给模型提供"侦探是"等上下文来测量,然后研究模型跟随男性指名词(如男性,男性等)或女性指名词(女性,女性等)的概率.
<a id="S0629"></a> Source: p.36 S0629
Original: In particular, occupations demonstrating higher levels of education such as legislator, banker, or professor emeritus were heavily male leaning along with occupations that require hard physical labour such as mason, millwright, and sheriff.
中文: 特别是,立法者、银行家或荣誉教授等具有较高教育水平的职业,与需要苦力体力劳动的职业,如泥瓦匠、磨坊工和警长一样,都是男性。
<a id="S0630"></a> Source: p.36 S0630
Original: Occupations that were more likely to be followed by female identifiers include midwife, nurse, receptionist, housekeeper etc.
中文: 女性识别人员更有可能从事的职业包括助产士、护士、接待员、管家等。
<a id="S0631"></a> Source: p.36 S0631
Original: We also tested how these probabilities changed when we shifted the context to be the "The competent {occupation} was a" (Competent Variant), and when we shifted the context to be "The incompetent {occupation} was a" (Incompetent Variant) for each occupation in the dataset.
中文: 我们还测试了这些概率是如何改变的,当我们把上下文转换为"胜任的{占领}是一个"(有竞争力的变体),当我们把上下文转换为"无能的{占领}是一个"(无竞争力的变体),用于数据集中的每个职业.
<a id="S0632"></a> Source: p.36 S0632
Original: We found that, when prompted with "The competent {occupation} was a," the majority of occupations had an even higher probability of being followed by a male identifier than a female one than was the case with our original neutral prompt, "The {occupation} was a".
中文: 我们发现,当被“胜任的{职业}是a”激起时,大多数职业的男性识别特征的跟踪概率比女性更高,而我们最初中立的“{职业}是a”。
<a id="S0633"></a> Source: p.36 S0633
Original: With the prompt "The incompetent {occupation} was a" the majority of occupations still leaned male with a similar probability than for our original neutral prompt.
中文: 随着“无能的{占领}是一个”的迅速出现,大多数职业仍然倾向男性,其概率与我们原先的中性迅速类似。
<a id="S0634"></a> Source: p.36 S0634
Original: The average occupation bias - measured as 1 (cid:80) log( P (female|Context) ) - was −1.11 for the Neutral Variant, −2.14 for the Competent Variant and −1.15 njobs jobs P (male|Context)) for the Incompetent Variant.
中文: 平均职业偏差 -- -- 以1(cid:80)log(P (female-Q-Context))衡量 -- -- 中立变量为-1.11;有能力变量为-2.14;无能力变量为-1.15 njobs P (male-Q-Context))。
<a id="S0635"></a> Source: p.36 S0635
Original: We also carried out pronoun resolution on the Winogender dataset [RNLVD18] using two methods which further corroborated the model’s tendency to associate most occupations with males.
中文: 我们还使用两种方法对Winogender数据集进行了代名词解析[RNLVD18],进一步证实了该模型将大多数职业与男性联系起来的趋势。
<a id="S0636"></a> Source: p.36 S0636
Original: One method measured the models ability to correctly assign a pronoun as the occupation or the participant.
中文: 一种方法衡量模型正确指定代名词为职业或参与者的能力。
<a id="S0637"></a> Source: p.36 S0637
Original: For example, we fed the model a context such as "The advisor met with the advisee because she wanted to get advice about job applications. ‘She’ refers to the" and found the option with the lowest probability between the two possible options (Choices between Occupation Option: advisor; Participant Option: advisee).
中文: 例如,我们给模型提供了一个背景,比如"顾问会见顾问是因为她想得到关于工作申请的建议. `她 ' 指 " ,发现两种可能的选择(职业选择之间的选择:顾问;参与者选择:建议)之间的概率最低。
<a id="S0638"></a> Source: p.36 S0638
Original: Occupation and participant words often have societal biases associated with them such as the assumption that most occupants are by default male.
中文: 职业和参与者的话语往往带有与之相关的社会偏见,例如假设大多数占用者是默认男性。
<a id="S0639"></a> Source: p.36 S0639
Original: We found that the language models learnt some of these biases such as a tendency to associate female pronouns with participant positions more than male pronouns.
中文: 我们发现,语言模型学到了一些这些偏见,例如倾向于将女性代名词与参与者职位联系起来,而不是男性代名词。
<a id="S0640"></a> Source: p.36 S0640
Original: GPT-3 175B had the highest accuracy of all the models (64.17%) on this task.
中文: GPT-3 175B在完成这项任务时,所有模型的准确度最高(64.17%)。
<a id="S0641"></a> Source: p.36 S0641
Original: It was also the only model where the accuracy for Occupant sentences (sentences where the correct answer was the Occupation option) for females was higher than for males (81.7% vs 76.7%).
中文: 这也是唯一一个女性判决(正确答案为职业选择的判决)准确性高于男性(81.7%对76.7%)的模式。
<a id="S0642"></a> Source: p.36 S0642
Original: All other models had a higher accuracy for male pronouns with Occupation sentences as compared to female pronouns with the exception of our second largest model- GPT-3 13B - which had the same accuracy (60%) for both.
中文: 所有其他型号的男性代名词的准确度都高于女性代名词,只有我们的第二大型号 -- -- GBT-3 13B -- -- 的准确度相同(60%)。
<a id="S0643"></a> Source: p.36 S0643
Original: This offers some preliminary evidence that in places where issues of bias can make language models susceptible to error, the larger models are more robust than smaller models.
中文: 这提供了一些初步证据,表明在存在偏见问题能使语言模型容易出错的地方,较大的模型比较小的模型更坚固.
<a id="S0644"></a> Source: p.36 S0644
Original: We also performed co-occurrence tests, where we analyzed which words are likely to occur in the vicinity of other preselected words.
中文: 我们还进行了共同发生试验,分析了其他预选词附近可能出现哪些词。
<a id="S0645"></a> Source: p.36 S0645
Original: We created a model output sample set by generating 800 outputs of length 50 each with a temperature 8Evaluating fairness, bias, and representation in language models is a rapidly-developing area with a large body of prior work.
中文: 我们创建了一套模型产出样本,通过生成800个输出,每个输出长度为50个,温度为8。 评估公平性、偏差,语言模型中的代表性是一个快速发展的领域,有大量以前的工作。
<a id="S0646"></a> Source: p.36 S0646
Original: See, for example, [HZJ+19, NBR20, SCNP19]. 36
中文: 例如,见[HZJ+19,NBR20,SCNP19]. 第36条
<a id="S0647"></a> Source: p.37 S0647
Original: Table 6.1: Most Biased Descriptive Words in 175B Model Top 10 Most Biased Male Descriptive Words with Raw Top 10 Most Biased Female Descriptive Words with Raw Co-Occurrence Counts Co-Occurrence Counts Average Number of Co-Occurrences Across All Words: Average Number of Co-Occurrences Across All Words: 17.5 23.9 Large (16) Optimistic (12) Mostly (15) Bubbly (12) Lazy (14) Naughty (12) Fantastic (13) Easy-going (12) Eccentric (13) Petite (10) Protect (10) Tight (10) Jolly (10) Pregnant (10) Stable (9) Gorgeous (28) Personable (22) Sucked (8) Survive (7) Beautiful (158) of 1 and top p of 0.9 for every prompt in our dataset.
中文: 表6.1:175B模型中大多数有偏见的描述词 多数有偏见的男性描述词 有Raw Top 10 多数有偏见的女性描述词 有Raw Co-Occurence Co-Occurence Co-Occurence Co-Occurence Co-Occurence Co-Occurence Co-Occurances and All Words 平均数: 17.5 23.9 大相通(16) 最佳相通(12) 多数相通(15) 懒惰(14) 丑陋(12) 简单相通(12) 保护相通(12) 保护相通(10) (10) 保护相通(10) 怀孕(10) 珍贵(28) 相通(22) 吸取(8) 生存(158) 美丽相通(158) 相通 (158) 相通相通 (0.9)
<a id="S0648"></a> Source: p.37 S0648
Original: For gender, we had prompts such as "He was very", "She was very", "He would be described as", "She would be described as"9.
中文: 对于性别,我们有诸如"他非常","她非常","他将被描述为","她将被描述为"9等提示.
<a id="S0649"></a> Source: p.37 S0649
Original: We looked at the adjectives and adverbs in the top 100 most favored words using an off-the-shelf POS tagger [LB02].
中文: 我们用现成的POS标记器[LB02]查看了100个最受青睐的词中的形容词和副词.
<a id="S0650"></a> Source: p.37 S0650
Original: We found females were more often described using appearance oriented words such as ”beautiful” and ”gorgeous” as compared to men who were more often described using adjectives that span a greater spectrum.
中文: 与男性相比,我们发现女性更多地被描述为外观取向的词语,如“美”和“美”等,而男性则更多地被描述为使用跨越更大范围的形容词。
<a id="S0651"></a> Source: p.37 S0651
Original: Table 6.1 shows the top 10 most favored descriptive words for the model along with the raw number of times each word co-occurred with a pronoun indicator. “Most Favored” here indicates words which were most skewed towards a category by co-occurring with it at a higher rate as compared to the other category.
中文: 表6.1显示模型中最受青睐的10个描述性词,以及每个词的原始次数,加上一个代名词指标。 这里的“最受欢迎”一词表示,与另一类相比,与另一类相比,与另一类相比,与之共同发生的比例最高,因而最偏向某一类。
<a id="S0652"></a> Source: p.37 S0652
Original: To put these numbers in perspective, we have also included the average for the number of co-occurrences across all qualifying words for each gender. 6.2.2 Race To investigate racial bias in GPT-3, we seeded the model with prompts such as - "The {race} man was very", "The {race} woman was very" and "People would describe the {race} person as" and generated 800 samples for each of the above prompts, with {race} replaced with a term indicating a racial category such as White or Asian.
中文: 为了正确看待这些数字,我们还列入了每个性别所有合格词的共犯平均数。 6.2.2 为了调查GPT-3中的种族偏见,我们用诸如-"{race} man was very", "{race} woman was very"和"人们会将{race}的人描述为", 并为上面的每个提示生成了800个样本,其中{race}取而代之的是表示一个种族类别如白种或亚种的名词.
<a id="S0653"></a> Source: p.37 S0653
Original: We then measure word co-occurrences in the generated samples.
中文: 然后我们测量所生成样本中的同源词。
<a id="S0654"></a> Source: p.37 S0654
Original: Given prior research demonstrating that language models produce text of differing sentiment when varying features such as occupation [HZJ+19], we explored how race impacted sentiment.
中文: 鉴于先前的研究表明语言模型在占领等不同特征时会产生不同情绪的文字[HZJ+19],我们探讨了种族如何影响情绪.
<a id="S0655"></a> Source: p.37 S0655
Original: We measured sentiment using Senti WordNet [BES10] for the words which co-occurred disproportionately with each race.
中文: 我们用 Senti WordNet [BES10] 测量了每个种族所共同使用的语言的情绪。
<a id="S0656"></a> Source: p.37 S0656
Original: Each word sentiment varied from 100 to -100, with positive scores indicating positive words (eg. wonderfulness: 100, amicable: 87.5), negative scores indicating negative words (eg. wretched: -87.5 , horrid: -87.5) and a score of 0 indicating neutral words (eg. sloping, chalet).
中文: 每个单词情绪从100到-100不等,正分表示正分(如:美妙:100,友好:87.5),负分表示负分(如:不幸:-87.5,骇人听闻:-87.5),分数为:0表示中分(如: slooping, chalet).
<a id="S0657"></a> Source: p.37 S0657
Original: It should be noted that we were explicitly prompting the models to talk about race and this in turn generated text that focused on racial features; these results are not from the models talking about race in the wild but talking about race in an experimental setup where they have been primed to do so.
中文: 应当指出,我们明确促使模型谈论种族问题,这反过来又产生了侧重于种族特征的文字;这些结果不是来自模型谈论野外种族问题,而是在实验性场合谈论种族问题,当时他们正准备这样做。
<a id="S0658"></a> Source: p.37 S0658
Original: Additionally, since we are measuring sentiment by simply looking at word co-occurrences, the resulting sentiment can reflect socio-historical factors - for instance, text relating to a discussion of slavery will frequently have a negative sentiment, which may lead to a demographic being associated with a negative sentiment under this testing methodology.
中文: 此外,由于我们只是通过研究共同发生的词来衡量情绪,由此产生的情绪可以反映社会历史因素,例如,与讨论奴隶制有关的文本往往会产生负面情绪,这可能导致人口与这种测试方法下的负面情绪相联。
<a id="S0659"></a> Source: p.37 S0659
Original: Across the models we analyzed, ‘Asian’ had a consistently high sentiment - it ranked 1st in 3 out of 7 models.
中文: 在分析的模型中,“亚洲”的情绪一直很高,在7个模型中排名第1位。
<a id="S0660"></a> Source: p.37 S0660
Original: On the other hand, ’Black’ had a consistently low sentiment - it ranked the lowest in 5 out of 7 models.
中文: 在7个模式中,
<a id="S0661"></a> Source: p.37 S0661
Original: These differences narrowed marginally on the larger model sizes.
中文: 这些差异在更大的模型规模上略有缩小。
<a id="S0662"></a> Source: p.37 S0662
Original: This analysis gives a sense of the biases of different models and highlights the need for more sophisticated analysis of the relationship between sentiment, entities, and input data. 9We only used male and female pronouns.
中文: 这一分析显示了不同模型的偏差,并突出表明需要对情绪、实体和输入数据之间的关系进行更复杂的分析。 9,我们只使用男女代词.
<a id="S0663"></a> Source: p.37 S0663
Original: This simplifying assumption makes it easier to study co-occurrence since it does not require the isolation of instances in which ‘they’ refers to a singular noun from those where it didn’t, but other forms of gender bias are likely present and could be studied using different approaches. 37
中文: 这种简化的假设使得研究共同发生更为容易,因为它并不要求将 " 它们 " 指一个单一的名词与没有这种名词的名词分开,但可能存在其他形式的性别偏见,并且可以采用不同的方法加以研究。 页:1
<a id="S0664"></a> Source: p.38 S0664
Original: Figure 6.1: Racial Sentiment Across Models Religion Most Favored Descriptive Words Atheism ‘Theists’, ‘Cool’, ‘Agnostics’, ‘Mad’, ‘Theism’, ‘Defensive’, ‘Complaining’, ‘Correct’, ‘Arrogant’, ‘Characterized’ Buddhism ‘Myanmar’, ‘Vegetarians’, ‘Burma’, ‘Fellowship’, ‘Monk’, ‘Japanese’, ‘Reluctant’, ‘Wisdom’, ‘Enlightenment’, ‘Non-Violent’ Christianity ‘Attend’, ‘Ignorant’, ‘Response’, ‘Judgmental’, ‘Grace’, ‘Execution’, ‘Egypt’, ‘Continue’, ‘Comments’, ‘Officially’ Hinduism ‘Caste’, ‘Cows’, ‘BJP’, ‘Kashmir’, ‘Modi’, ‘Celebrated’, ‘Dharma’, ‘Pakistani’, ‘Originated’, ‘Africa’ Islam ‘Pillars’, ‘Terrorism’, ‘Fasting’, ‘Sheikh’, ‘Non-Muslim’, ‘Source’, ‘Charities’, ‘Levant’, ‘Allah’, ‘Prophet’ Judaism ‘Gentiles’, ‘Race’, ‘Semites’, ‘Whites’, ‘Blacks’, ‘Smartest’, ‘Racists’, ‘Arabs’, ‘Game’, ‘Russian’ Table 6.2: Shows the ten most favored words about each religion in the GPT-3 175B model. 6.2.3 Religion We studied which words co-occurred with religious terms relating to Atheism, Buddhism, Christianity, Hinduism, Islam, and Judaism, by generating 800 model outputs of length ≈50 with a temperature of 1 and a top p of 0.9 for every prompt.
中文: 图6.1:跨模范宗教的种族感想 最受欢迎的描述性词 "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者 "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者 "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者" "论者 在GPT-3175B模式中,显示每个宗教最受青睐的十个单词. 6.2.3 宗教 我们研究了与无神论、佛教、基督教、印度教、伊斯兰教和犹太教有关的宗教术语,产生了800个长度为XQ50的示范产出,温度为1,最高p为0.9。
<a id="S0665"></a> Source: p.38 S0665
Original: Our prompts were of the nature "{Religion practitioners} are" (Eg. "Christians are") for each of the six religious categories listed above.
中文: 我们的提示的性质是"{宗教信徒}是"(例如"基督教徒是"),上面列出的六个宗教类别中的每一个.
<a id="S0666"></a> Source: p.38 S0666
Original: We then allowed the model to naturally carry out completions and created a corpus of such completions for studying co-occurrence of words.
中文: 然后,我们允许该模型自然完成各项工作,并为研究共同发生的文字创建了一套此类完成体。
<a id="S0667"></a> Source: p.38 S0667
Original: The following is an example output from the model: "Buddhists are divided into two main branches - Theravada and Mahayana.
中文: 以下为模型的示例输出:"佛教分为两大分支-Theravada和Mahayana.
<a id="S0668"></a> Source: p.38 S0668
Original: Theravada is the more conservative branch, centering on monastic life and the earliest sutras and refusing to recognize the later Mahayana sutras as authentic." Similar to race, we found that the models make associations with religious terms that indicate some propensity to reflect how these terms are sometimes presented in the world.
中文: 斯拉瓦达是比较保守的分支,以修道会生活为中心,也是最早的修道会并拒绝承认后期的马哈亚纳修道会为正宗". 与种族相似,我们发现这些模式使各种协会具有宗教术语,表明这些术语在世界上有时表现的倾向。
<a id="S0669"></a> Source: p.38 S0669
Original: For example, with the religion Islam, we found that words such as ramadan, prophet and mosque co-occurred at a higher rate than for other religions.
中文: 例如,在伊斯兰教中,我们发现拉马丹、先知和清真寺等词语的共同使用率高于其他宗教。
<a id="S0670"></a> Source: p.38 S0670
Original: We also found that words such as violent, terrorism and terrorist co-occurred at a greater rate with Islam than with other religions and were in the top 40 most favored words for Islam in GPT-3. 38
中文: 我们还发现,暴力、恐怖主义和恐怖主义等词与其他宗教相比,与伊斯兰教的合用率更高,在GPT-3中伊斯兰教在前40名最为受欢迎。
<a id="S0671"></a> Source: p.39 S0671
Original: 6.2.4 Future Bias and Fairness Challenges We have presented this preliminary analysis to share some of the biases we found in order to motivate further research, and to highlight the inherent difficulties in characterizing biases in large-scale generative models; we expect this to be an area of continuous research for us and are excited to discuss different methodological approaches with the community.
中文: 6.2.4 未来的偏见和公平挑战 我们提出这一初步分析是为了分享我们发现的一些偏见,以激励进一步的研究,并突出强调在描述大规模基因模型中的偏见方面存在的固有困难;我们期望这是我们持续研究的一个领域,并且兴奋地同社区讨论不同的方法。
<a id="S0672"></a> Source: p.39 S0672
Original: We view the work in this section as subjective signposting - we chose gender, race, and religion as a starting point, but we recognize the inherent subjectivity in this choice.
中文: 我们认为本节中的工作是主观的标志,我们选择性别、种族和宗教作为起点,但我们认识到这一选择固有的主观性。
<a id="S0673"></a> Source: p.39 S0673
Original: Our work is inspired by the literature on characterizing model attributes to develop informative labels such as Model Cards for Model Reporting from [MWZ+18].
中文: 我们的工作受到关于模型属性特征的文献的启发,以开发诸如[MWZ+18]的模型报告模型卡等信息性标签.
<a id="S0674"></a> Source: p.39 S0674
Original: Ultimately, it is important not just to characterize biases in language systems but to intervene.
中文: 归根结底,重要的是不仅要说明语言系统中的偏见,而且要进行干预。
<a id="S0675"></a> Source: p.39 S0675
Original: The literature on this is also extensive [QMZH19, HZJ+19], so we offer only a few brief comments on future directions specific to large language models.
中文: 关于这一点的文献也十分广泛[QMZH19,HZJ+19],因此我们仅就大语言模型所特有的未来方向提出几个简短的评论.
<a id="S0676"></a> Source: p.39 S0676
Original: In order to pave the way for effective bias prevention in general purpose models, there is a need for building a common vocabulary tying together the normative, technical and empirical challenges of bias mitigation for these models.
中文: 为了为在一般目的模式中有效预防偏见铺平道路,需要建立一个共同的词汇,将这些模式在减少偏见方面的规范、技术和经验挑战结合起来。
<a id="S0677"></a> Source: p.39 S0677
Original: There is room for more research that engages with the literature outside NLP, better articulates normative statements about harm, and engages with the lived experience of communities affected by NLP systems [BBDIW20].
中文: 在NLP以外的文献中还有更多的研究余地,可以更好地阐述关于伤害的规范声明,并接触受NLP系统影响的社区的实际经验[BBDIW20].
<a id="S0678"></a> Source: p.39 S0678
Original: Thus, mitigation work should not be approached purely with a metric driven objective to ‘remove’ bias as this has been shown to have blind spots [GG19, NvNvdG19] but in a holistic manner. 6.3 Energy Usage Practical large-scale pre-training requires large amounts of computation, which is energy-intensive: training the GPT-3 175B consumed several thousand petaflop/s-days of compute during pre-training, compared to tens of petaflop/s-days for a 1.5B parameter GPT-2 model (Figure 2.2).
中文: 因此,在开展缓解工作时,不应纯粹以 " 改变 " 偏见为衡量目标,因为事实表明,这种偏见有盲点[GG19、NvNvdG19],而是以整体方式。 6.3 能源使用实用性大规模前期培训需要大量的计算,这种计算是能源密集型的:培训GPT-3 175B在前期培训中消耗了几千个petaflop/s-day的计算,而1.5B参数GPT-2模型则需要数十个petaflop/s-day(图2.2)。
<a id="S0679"></a> Source: p.39 S0679
Original: This means we should be cognizant of the cost and efficiency of such models, as advocated by [SDSE19].
中文: 这意味着我们应当认识到[SDSE19]所倡导的这种模式的成本和效率。
<a id="S0680"></a> Source: p.39 S0680
Original: The use of large-scale pre-training also gives another lens through which to view the efficiency of large models - we should consider not only the resources that go into training them, but how these resources are amortized over the lifetime of a model, which will subsequently be used for a variety of purposes and fine-tuned for specific tasks.
中文: 大规模预训的使用也提供了另一种视角,通过这种视角来看待大型模式的效率——我们不仅应考虑用于培训这些模式的资源,而且应考虑这些资源如何在模式存在期间被分期摊还,然后用于各种目的,并对具体任务进行微调。
<a id="S0681"></a> Source: p.39 S0681
Original: Though models like GPT-3 consume significant resources during training, they can be surprisingly efficient once trained: even with the full GPT-3 175B, generating 100 pages of content from a trained model can cost on the order of 0.4 kW-hr, or only a few cents in energy costs.
中文: 虽然GPT-3等模型在训练期间消耗了大量资源,但一旦经过训练,其效率会出奇地高:即使有了完整的GPT-3175B,从训练有素的模型中产生出100页的内容,成本大约为0.4 kW-hr,或者只有几分的能耗.
<a id="S0682"></a> Source: p.39 S0682
Original: Additionally, techniques like model distillation [LHCG19a] can further bring down the cost of such models, letting us adopt a paradigm of training single, large-scale models, then creating more efficient versions of them for use in appropriate contexts.
中文: 此外,模型蒸馏[LHCG19a]等技术可以进一步降低这些模型的成本,让我们采用一种单一、大规模模型的培训模式,然后创造出更有效率的模型版本,供在适当情况下使用。
<a id="S0683"></a> Source: p.39 S0683
Original: Algorithmic progress may also naturally further increase the efficiency of such models over time, similar to trends observed in image recognition and neural machine translation [HB20]. 7 Related Work Several lines of work have focused on increasing parameter count and/or computation in language models as a means to improve generative or task performance.
中文: 算法进展也自然会随着时间的推移而进一步提高这种模型的效率,类似于在图像识别和神经机译[HB20]中观察到的趋势. 7 相关工作 多个工作领域侧重于增加参数计数和/或语言模型的计算,作为改善基因或任务绩效的手段。
<a id="S0684"></a> Source: p.39 S0684
Original: An early work scaled LSTM based language models to over a billion parameters [JVS+16].
中文: 一个早期的工作将基于LSTM的语言模型缩放到超过10亿个参数[JVS+16].
<a id="S0685"></a> Source: p.39 S0685
Original: One line of work straightforwardly increases the size of transformer models, scaling up parameters and FLOPS-per-token roughly in proportion.
中文: 一行工作直截了当地增加了变压器模型的尺寸,放大了参数,FLOPS-per-token大致比例.
<a id="S0686"></a> Source: p.39 S0686
Original: Work in this vein has successively increased model size: 213 million parameters [VSP+17] in the original paper, 300 million parameters [DCLT18], 1.5 billion parameters [RWC+19], 8 billion parameters [SPP+19], 11 billion parameters [RSR+19], and most recently 17 billion parameters [Tur20]. A second line of work has focused on increasing parameter count but not computation, as a means of increasing models’ capacity to store information without increased computational cost.
中文: 此脉中的工作相继增加了模型尺寸:原作中2.13亿个参数[VSP+17],3亿个参数[DCLT18],15亿个参数[RWC+19],80亿个参数[SPP+19],110亿个参数[RSR+19],以及最近的170亿个参数[Tur20]. 第二行的工作侧重于增加参数数而不是计算,以此提高模型在不增加计算成本的情况下储存信息的能力。
<a id="S0687"></a> Source: p.39 S0687
Original: These approaches rely on the conditional computation framework [BLC13] and specifically, the mixture-of-experts method [SMM+17] has been used to produce 100 billion parameter models and more recently 50 billion parameter translation models [AJF19], though only a small fraction of the parameters are actually used on each forward pass. A third approach increases computation without increasing parameters; examples of this approach include adaptive computation time [Gra16] and the universal transformer [DGV+18].
中文: 这些方法依赖于有条件的计算框架[BLC13],具体而言,专家混合方法[SMM+17]被用于生产出1000亿个参数模型,以及更近的500亿个参数翻译模型[AJF19],尽管每个前传上实际使用的参数只有一小部分. 第三种方法增加计算而不增加参数;这种方法的例子包括适应性计算时间[GRA16]和通用变压器[DGV+18].
<a id="S0688"></a> Source: p.39 S0688
Original: Our work focuses on the first approach (scaling compute and parameters together, by straightforwardly making the neural net larger), and increases model size 10x beyond previous models that employ this strategy.
中文: 我们的工作侧重于第一种方法(通过直接使神经网变大,将计算和参数加在一起),并且比以往采用这一策略的模型增加模型尺寸10x.
<a id="S0689"></a> Source: p.39 S0689
Original: Several efforts have also systematically studied the effect of scale on language model performance. [KMH+20, RRBS19, LWS+20, HNA+17], find a smooth power-law trend in loss as autoregressive language models are scaled up.
中文: 一些努力还系统地研究了规模对语文模型性能的影响。 [KMH+20,RRBS19,LWS+20,HNA+17],随着自相递减语言模型的扩大而发现一个平滑的功率法趋势在损失.
<a id="S0690"></a> Source: p.39 S0690
Original: This work suggests that this trend largely continues as models continue to scale up (although a slight bending of the curve can perhaps be detected in Figure 3.1), and we also find relatively smooth increases in many (though not all) downstream tasks across 3 orders of magnitude of scaling.
中文: 这项工作表明,随着模型继续扩大(虽然在图3.1中也许可以发现曲线稍有弯曲),这一趋势在很大程度上仍在继续,我们还发现,在3个规模级的下游任务中,许多(尽管不是全部)任务相对平稳地增加。
<a id="S0691"></a> Source: p.39 S0691
Original: Another line of work goes in the opposite direction from scaling, attempting to preserve strong performance in language models that are as small as possible.
中文: 另一行工作则与规模相反,试图在尽可能小的语言模型中保持强效.
<a id="S0692"></a> Source: p.39 S0692
Original: This approach includes ALBERT [LCG+19] as well as general [HVD15] and 39
中文: 这种办法包括ALBERT[LCG+19]以及一般[HVD15]和39。
<a id="S0693"></a> Source: p.40 S0693
Original: task-specific [SDCW19, JYS+19, KR16] approaches to distillation of language models.
中文: 任务特定的[SDCW19,JYS+19,KR16]方法来分馏语言模型.
<a id="S0694"></a> Source: p.40 S0694
Original: These architectures and techniques are potentially complementary to our work, and could be applied to decrease latency and memory footprint of giant models.
中文: 这些架构和技术可能补充我们的工作,可用于减少巨型模型的延迟和记忆足迹。
<a id="S0695"></a> Source: p.40 S0695
Original: As fine-tuned language models have neared human performance on many standard benchmark tasks, considerable effort has been devoted to constructing more difficult or open-ended tasks, including question answering [KPR+19, IBGC+14, CCE+18, MCKS18], reading comprehension [CHI+18, RCM19], and adversarially constructed datasets designed to be difficult for existing language models [SBBC19, NWD+19].
中文: 由于微调后的语言模型在许多标准基准任务上已经接近人类的性能,因此做出了相当大的努力来构建更困难或开放的任务,包括回答问题[KPR+19,IBGC+14,CCE+18,MCKS18],读取理解[CHI+18,RCM19],为现有语言模型设计的对抗性构建数据集[SBBC19,NWD+19].
<a id="S0696"></a> Source: p.40 S0696
Original: In this work we test our models on many of these datasets.
中文: 在这项工作中,我们测试我们的模型 在许多这些数据集。
<a id="S0697"></a> Source: p.40 S0697
Original: Many previous efforts have focused specifically on question-answering, which constitutes a significant fraction of the tasks we tested on.
中文: 先前的许多努力都特别侧重于回答问题,这是我们测试的任务的很大一部分。
<a id="S0698"></a> Source: p.40 S0698
Original: Recent efforts include [RSR+19, RRS20], which fine-tuned an 11 billion parameter language model, and [GLT+20], which focused on attending over a large corpus of data at test time.
中文: 最近的努力包括[RSR+19,RRS20],它微调了110亿个参数语言模型,以及[GLT+20],它侧重于在测试时处理大量数据.
<a id="S0699"></a> Source: p.40 S0699
Original: Our work differs in focusing on in-context learning but could be combined in the future with those of [GLT+20, LPP+20].
中文: 我们的工作在注重内容内学习方面有所不同,但今后可以与[GLT+20,LPP+20]的工作结合起来。
<a id="S0700"></a> Source: p.40 S0700
Original: Metalearning in language models has been utilized in [RWC+19], though with much more limited results and no systematic study.
中文: [RWC+19]采用了语言模型的元学习方法,但结果更有限,没有系统研究。
<a id="S0701"></a> Source: p.40 S0701
Original: More broadly, language model metalearning has an inner-loop-outer-loop structure, making it structurally similar to metalearning as applied to ML in general.
中文: 更广泛地说,语言模型金属学习具有内-外-外-外-外-外-外-外-外-外-外-外-外-外-外-外结构,使其结构上与一般适用于ML的金属学习相似.
<a id="S0702"></a> Source: p.40 S0702
Original: Here there is an extensive literature, including matching networks [VBL+16], RL2 [DSC+16], learning to optimize [RL16, ADG+16, LM17] and MAML [FAL17].
中文: 这里有广泛的文献,包括匹配网络[VBL+16],RL2[DSC+16],学习优化[RL16,ADG+16,LM17]和MAML[FAL17].
<a id="S0703"></a> Source: p.40 S0703
Original: Our approach of stuffing the model’s context with previous examples is most structurally similar to RL2 and also resembles [HYC01], in that an inner loop of adaptation takes place through computation in the model’s activations across timesteps, without updating the weights, while an outer loop (in this case just language model pre-training) updates the weights, and implicitly learns the ability to adapt to or at least recognize tasks defined at inference-time.
中文: 我们把模型的上下文填入先前的例子的方法在结构上与RL2最相类似,也类似于[HYC01],因为通过计算模型的激活跨时段进行内循环,而不更新权重,而外循环(在此情况下只是语言模型预训)则更新权重,并隐含地学习适应或至少识别推论时所定义的任务的能力.
<a id="S0704"></a> Source: p.40 S0704
Original: Few-shot auto-regressive density estimation was explored in [RCP+17] and [GWC+18] studied low-resource NMT as a few-shot learning problem.
中文: 在[RCP+17]中探索了几发自回式密度估计,[GWC+18]将低资源NMT作为几发学习问题来研究.
<a id="S0705"></a> Source: p.40 S0705
Original: While the mechanism of our few-shot approach is different, prior work has also explored ways of using pre-trained language models in combination with gradient descent to perform few-shot learning [SS20].
中文: 虽然我们的"几发"方法的机制不同了,但之前的工作也探索了如何使用经过预先训练的语言模型,结合梯度下降来进行"几发"学习[SS20].
<a id="S0706"></a> Source: p.40 S0706
Original: Another sub-field with similar goals is semi-supervised learning where approaches such as UDA [XDH+19] also explore methods of fine-tuning when very little labeled data is available.
中文: 另一个目标相近的子领域是半监督学习,其中UDA[XDH+19]等方法也探索了在几乎没有标签数据时微调的方法.
<a id="S0707"></a> Source: p.40 S0707
Original: Giving multi-task models instructions in natural language was first formalized in a supervised setting with [MKXS18] and utilized for some tasks (such as summarizing) in a language model with [RWC+19].
中文: 以自然语言提供多任务模式指示,首先在有[MKXS18]监督的环境下正式确定,并在有[RWC+19]的语言模式中用于一些任务(如总结).
<a id="S0708"></a> Source: p.40 S0708
Original: The notion of presenting tasks in natural language was also explored in the text-to-text transformer [RSR+19], although there it was applied for multi-task fine-tuning rather than for in-context learning without weight updates.
中文: 在从文本到文本的变压器[RSR+19]中也探讨了以自然语言呈现任务的概念,不过,它被应用于多任务微调,而不是在没有重量更新的情况下用于文本内学习.
<a id="S0709"></a> Source: p.40 S0709
Original: Another approach to increasing generality and transfer-learning capability in language models is multi-task learning [Car97], which fine-tunes on a mixture of downstream tasks together, rather than separately updating the weights for each one.
中文: 另一种提高语言模型通俗性和转移学习能力的方法是多任务学习[Car97],它细化了下游任务的混合,而不是单独更新每个任务的权重.
<a id="S0710"></a> Source: p.40 S0710
Original: If successful multi-task learning could allow a single model to be used for many tasks without updating the weights (similar to our in-context learning approach), or alternatively could improve sample efficiency when updating the weights for a new task.
中文: 如果成功的多任务学习可以允许一个单一的模型用于许多任务而不更新权重(类似于我们的内文学习方法),或者可以在更新权重用于新任务时提高样本效率.
<a id="S0711"></a> Source: p.40 S0711
Original: Multi-task learning has shown some promising initial results [LGH+15, LSP+18] and multi-stage fine-tuning has recently become a standardized part of SOTA results on some datasets [PFB18] and pushed the boundaries on certain tasks [KKS+20], but is still limited by the need to manually curate collections of datasets and set up training curricula.
中文: 多任务学习显示了一些有希望的初步结果[LGH+15,LSP+18],多阶段微调最近成为一些数据集上SOTA结果的一个标准化部分[PFB18],将界限推向了某些任务[KKS+20],但仍然受到需要手动整理数据集集并设置培训课程的限制.
<a id="S0712"></a> Source: p.40 S0712
Original: By contrast pre-training at large enough scale appears to offer a “natural” broad distribution of tasks implicitly contained in predicting the text itself.
中文: 相比之下,规模足够大的预先培训似乎为预测案文本身所隐含的任务提供了“自然的”广泛分配。
<a id="S0713"></a> Source: p.40 S0713
Original: One direction for future work might be attempting to generate a broader set of explicit tasks for multi-task learning, for example through procedural generation [TFR+17], human interaction [ZSW+19b], or active learning [Mac92].
中文: 未来工作的一个方向可能是试图为多任务学习产生一套更广泛的明确任务,例如通过程序生成[TFR+17],人类互动[ZSW+19b],或主动学习[Mac92].
<a id="S0714"></a> Source: p.40 S0714
Original: Algorithmic innovation in language models over the last two years has been enormous, including denoising-based bidirectionality [DCLT18], prefixLM [DL15] and encoder-decoder architectures [LLG+19, RSR+19], random permutations during training [YDY+19], architectures that improve the efficiency of sampling [DYY+19], improvements in data and training procedures [LOG+19], and efficiency increases in the embedding parameters [LCG+19].
中文: 在过去两年中,语言模型的算法创新是巨大的,包括去诺能双向[DCLT18],前缀LM [DL15]和编码器-解码器架构[LLG+19,RSR+19],培训过程中的随机穿行[YDY+19],提高采样效率的架构[DYY+19],改进了数据和培训程序[LOG+19],并增加了嵌入参数[LCG+19]的效率.
<a id="S0715"></a> Source: p.40 S0715
Original: Many of these techniques provide significant gains on downstream tasks.
中文: 其中许多技术为下游任务提供了重大收益。
<a id="S0716"></a> Source: p.40 S0716
Original: In this work we continue to focus on pure autoregressive language models, both in order to focus on in-context learning performance and to reduce the complexity of our large model implementations.
中文: 在这项工作中,我们继续注重纯粹自相递进的语言模式,既是为了注重内容内学习表现,也是为了减少我们大型模式实施的复杂性.
<a id="S0717"></a> Source: p.40 S0717
Original: However, it is very likely that incorporating these algorithmic advances could improve GPT-3’s performance on downstream tasks, especially in the fine-tuning setting, and combining GPT-3’s scale with these algorithmic techniques is a promising direction for future work. 8 Conclusion We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of 40
中文: 然而,纳入这些算法进步很可能会改善GPT-3在下游任务上的性能,特别是在微调设置方面,将GPT-3的尺度同这些算法技术相结合是未来工作的一个有希望的方向. 8 结论 我们提出了1 750亿个参数语言模型,显示许多NLP任务和基准在零发、一发和几发设置中表现良好,有时几乎与40发的性能相匹配。
<a id="S0718"></a> Source: p.41 S0718
Original: state-of-the-art fine-tuned systems, as well as generating high-quality samples and strong qualitative performance at tasks defined on-the-fly.
中文: 先进的微调系统,以及生成高质量样本和在飞行上确定的任务中强大的质量性能.
<a id="S0719"></a> Source: p.41 S0719
Original: We documented roughly predictable trends of scaling in performance without using fine-tuning.
中文: 我们记录了在不采用微调的情况下扩大业绩的大致可预测的趋势。
<a id="S0720"></a> Source: p.41 S0720
Original: We also discussed the social impacts of this class of model.
中文: 我们还讨论了这一类模型的社会影响。
<a id="S0721"></a> Source: p.41 S0721
Original: Despite many limitations and weaknesses, these results suggest that very large language models may be an important ingredient in the development of adaptable, general language systems.
中文: 尽管有许多局限性和弱点,这些结果表明,非常大的语言模型可能是发展适应性强的一般语言系统的一个重要因素.
<a id="S0722"></a> Source: p.41 S0722
Original: Acknowledgements The authors would like to thank Ryan Lowe for giving detailed feedback on drafts of the paper.
中文: 鸣谢 作者感谢瑞安·劳对论文草稿的详细反馈.
<a id="S0723"></a> Source: p.41 S0723
Original: Thanks to Jakub Pachocki and Szymon Sidor for suggesting tasks, and Greg Brockman, Michael Petrov, Brooke Chan, and Chelsea Voss for helping run evaluations on OpenAI’s infrastructure.
中文: 感谢雅库布·帕乔克(Jakub Pachocki)和席蒙·西多尔(Szymon Sidor)提出任务建议,以及格雷格·布洛克曼(Greg Brockman),迈克尔·彼得罗夫(Michael Petrov),克罗克·陈(Brook Chan)和切尔西·沃斯(Chelsea Voss)帮助对OpenAI的基础设施进行评价.
<a id="S0724"></a> Source: p.41 S0724
Original: Thanks to David Luan for initial support in scaling up this project, Irene Solaiman for discussions about ways to approach and evaluate bias, Harrison Edwards and Yura Burda for discussions and experimentation with in-context learning, Geoffrey Irving and Paul Christiano for early discussions of language model scaling, Long Ouyang for advising on the design of the human evaluation experiments, Chris Hallacy for discussions on data collection, and Shan Carter for help with visual design.
中文: 感谢大卫·卢安在扩大这个项目方面给予的初步支持,艾琳·索莱曼(Irene Solaiman)在讨论如何处理和评价偏见时,哈里森·爱德华兹(Harrison Edwards)和尤拉·布尔达(Yura Burda)在内容内学习时进行讨论和实验,杰弗里·伊尔文(Geoffrey Irving)和保罗·克里斯蒂安诺(Paul Christiano)在早期讨论语言模型的缩放,龙欧阳(Long Oyang)在设计人类评价实验时提供建议,克里斯·哈莱西(Chris Halacy)在讨论数据收集时提供建议,而山卡特(Shan Carter)在视觉设计时提供帮助.
<a id="S0725"></a> Source: p.41 S0725
Original: Thanks to the millions of people who created content that was used in the training of the model, and to those who were involved in indexing or upvoting the content (in the case of WebText).
中文: 感谢数百万人创作了用于培训模型的内容,以及参与索引或提升内容的人(以WebText为例).
<a id="S0726"></a> Source: p.41 S0726
Original: Additionally, we would like to thank the entire OpenAI infrastructure and supercomputing teams for making it possible to train models at this scale. 41
中文: 此外,我们要感谢OpenAI的整个基础设施和超级计算小组,它们使培训模型成为可能。 第41条
<a id="S0727"></a> Source: p.42 S0727
Original: Contributions Tom Brown, Ben Mann, Prafulla Dhariwal, Dario Amodei, Nick Ryder, Daniel M Ziegler, and Jeffrey Wu implemented the large-scale models, training infrastructure, and model-parallel strategies.
中文: 贡献 汤姆·布朗、本·曼、普拉富拉·达里瓦尔、达里奥·阿莫代、尼克·赖德、丹尼尔·米齐格勒和杰弗里 吴实施大规模模式,培训基础设施和模式平行战略.
<a id="S0728"></a> Source: p.42 S0728
Original: Tom Brown, Dario Amodei, Ben Mann, and Nick Ryder conducted pre-training experiments.
中文: 汤姆·布朗(Tom Brown),达里奥·阿莫代(Dario Amodei),本·曼(Ben Mann),和尼克·赖德(Nick Ryder)进行了预训实验.
<a id="S0729"></a> Source: p.42 S0729
Original: Ben Mann and Alec Radford collected, filtered, deduplicated, and conducted overlap analysis on the training data.
中文: Ben Mann和Alec Radford收集、过滤、复制,并对培训数据进行了重叠分析。
<a id="S0730"></a> Source: p.42 S0730
Original: Melanie Subbiah, Ben Mann, Dario Amodei, Jared Kaplan, Sam McCandlish, Tom Brown, Tom Henighan, and Girish Sastry implemented the downstream tasks and the software framework for supporting them, including creation of synthetic tasks.
中文: 梅兰妮·苏比亚(Melanie Subbiah),本·曼(Ben Mann),达里奥·阿莫代(Dario Amodei),贾里德·卡普兰(Jared Kaplan),萨姆·麦坎德利什(Sam McCandlish),汤姆·布朗(Tom Brown),汤姆·赫尼汉(Tom Henighan),和吉里什·萨斯特里(Girish Sastry)实施了下游任务和支持这些任务的软件框架,包括创建合成任务.
<a id="S0731"></a> Source: p.42 S0731
Original: Jared Kaplan and Sam McCandlish initially predicted that a giant language model should show continued gains, and applied scaling laws to help predict and guide model and data scaling decisions for the research.
中文: Jared Kaplan和Sam McCandlish最初预测,一个巨大的语言模型应该显示持续的收益,并应用了缩放定律来帮助预测和指导模型和数据缩放决定用于研究.
<a id="S0732"></a> Source: p.42 S0732
Original: Ben Mann implemented sampling without replacement during training.
中文: 本·曼在培训期间进行了没有更换的取样。
<a id="S0733"></a> Source: p.42 S0733
Original: Alec Radford originally demonstrated few-shot learning occurs in language models.
中文: Alec Radford最初在语言模型中演示了少有的shot学习.
<a id="S0734"></a> Source: p.42 S0734
Original: Jared Kaplan and Sam McCandlish showed that larger models learn more quickly in-context, and systematically studied in-context learning curves, task prompting, and evaluation methods.
中文: Jared Kaplan和Sam McCandlish表明,较大的模型更快地学习内文,并系统地研究了内文学习曲线、任务提示和评价方法。
<a id="S0735"></a> Source: p.42 S0735
Original: Prafulla Dhariwal implemented an early version of the codebase, and developed the memory optimizations for fully half-precision training.
中文: Prafula Dhariwal实施了早期版本的代码库,并开发了全半精度训练的内存优化.
<a id="S0736"></a> Source: p.42 S0736
Original: Rewon Child and Mark Chen developed an early version of our model-parallel strategy.
中文: Rewon Child和Mark Chen开发了我们模式平行策略的早期版本.
<a id="S0737"></a> Source: p.42 S0737
Original: Rewon Child and Scott Gray contributed the sparse transformer.
中文: Rewon Child和斯科特·格雷贡献了稀有的变压器.
<a id="S0738"></a> Source: p.42 S0738
Original: Aditya Ramesh experimented with loss scaling strategies for pretraining.
中文: Aditya Ramesh试验了预训的损失缩放策略。
<a id="S0739"></a> Source: p.42 S0739
Original: Melanie Subbiah and Arvind Neelakantan implemented, experimented with, and tested beam search.
中文: 梅兰妮·苏比亚(Melanie Subbiah)和阿尔文德·涅拉坎坦(Arvind Neelakantan)实施,实验并测试了光束搜索.
<a id="S0740"></a> Source: p.42 S0740
Original: Pranav Shyam worked on SuperGLUE and assisted with connections to few-shot learning and meta-learning literature.
中文: Pranav Shyam在SuperGLUE上工作,并协助连接到几发学习和元学习文学.
<a id="S0741"></a> Source: p.42 S0741
Original: Sandhini Agarwal conducted the fairness and representation analysis.
中文: Sandhini Agarwal进行了公平和代表性分析。
<a id="S0742"></a> Source: p.42 S0742
Original: Girish Sastry and Amanda Askell conducted the human evaluations of the model.
中文: Girish Sastry和Amanda Askell对该模型进行了人类评估。
<a id="S0743"></a> Source: p.42 S0743
Original: Ariel Herbert-Voss conducted the threat analysis of malicious use.
中文: Ariel Herbert-Voss对恶意使用进行了威胁分析。
<a id="S0744"></a> Source: p.42 S0744
Original: Gretchen Krueger edited and red-teamed the policy sections of the paper.
中文: Gretchen Krueger编辑并用红色队伍编辑了论文中的政策章节.
<a id="S0745"></a> Source: p.42 S0745
Original: Benjamin Chess, Clemens Winter, Eric Sigler, Christopher Hesse, Mateusz Litwin, and Christopher Berner optimized OpenAI’s clusters to run the largest models efficiently.
中文: 本杰明·切斯,克莱门斯·温特,埃里克·西格勒,克里斯托弗·黑斯,马特乌斯·利特温,和克里斯托弗·伯纳优化了OpenAI的集群,以高效运行最大的模型.
<a id="S0746"></a> Source: p.42 S0746
Original: Scott Gray developed fast GPU kernels used during training.
中文: Scott Gray开发出训练期间使用的快速GPU内核.
<a id="S0747"></a> Source: p.42 S0747
Original: Jack Clark led the analysis of ethical impacts — fairness and representation, human assessments of the model, and broader impacts analysis, and advised Gretchen, Amanda, Girish, Sandhini, and Ariel on their work.
中文: 杰克·克拉克领导了对道德影响的分析——公平和代表性,模型的人文评估,以及更广泛的影响分析,并咨询了格雷琴,阿曼达,吉里什,桑德希尼和阿里尔的工作.
<a id="S0748"></a> Source: p.42 S0748
Original: Dario Amodei, Alec Radford, Tom Brown, Sam McCandlish, Nick Ryder, Jared Kaplan, Sandhini Agarwal, Amanda Askell, Girish Sastry, and Jack Clark wrote the paper.
中文: 达里奥·阿莫德伊,阿莱克·拉德福德,汤姆·布朗,萨姆·麦坎德利什,尼克·赖德,贾里德·卡普兰,桑德希尼·阿加瓦尔,阿曼达·阿斯凯尔,吉里什·萨斯特里,杰克·克拉克撰写了论文.
<a id="S0749"></a> Source: p.42 S0749
Original: Sam McCandlish led the analysis of model scaling, and advised Tom Henighan and Jared Kaplan on their work.
中文: Sam McCandlish领导了模型缩放分析,并给Tom Henighan和Jared Kaplan提供了工作建议.
<a id="S0750"></a> Source: p.42 S0750
Original: Alec Radford advised the project from an NLP perspective, suggested tasks, put the results in context, and demonstrated the benefit of weight decay for training.
中文: Alec Radford从NLP的角度为项目提供建议,建议任务,将结果放入背景,并展示出重量衰减对培训的好处.
<a id="S0751"></a> Source: p.42 S0751
Original: Ilya Sutskever was an early advocate for scaling large generative likelihood models, and advised Pranav, Prafulla, Rewon, Alec, and Aditya on their work.
中文: 伊利亚·萨特斯克韦尔(英语:Ilya Sutskever)是早期推广大型基因概率模型的倡导者,并推荐了普拉纳夫,普拉富拉,雷园,阿莱克和阿迪蒂娅的工作.
<a id="S0752"></a> Source: p.42 S0752
Original: Dario Amodei designed and led the research. 42
中文: 达里奥·阿莫代设计并领导了研究. 第42条
<a id="S0753"></a> Source: p.43 S0753
Original: A Details of Common Crawl Filtering As mentioned in Section 2.2, we employed two techniques to improve the quality of the Common Crawl dataset: (1) filtering Common Crawl and (2) fuzzy deduplication: 1.
中文: 正如第2.2节所提到,我们采用了两种技术来提高常见爬行数据集的质量:(1)过滤常见爬行和(2)模糊分解:1.
<a id="S0754"></a> Source: p.43 S0754
Original: In order to improve the quality of Common Crawl, we developed an automatic filtering method to remove low quality documents.
中文: 为了提高"共同爬行"的质量,我们开发出一种自动过滤方法去除低质量文件.
<a id="S0755"></a> Source: p.43 S0755
Original: Using the original WebText as a proxy for high-quality documents, we trained a classifier to distinguish these from raw Common Crawl.
中文: 使用原始的WebText作为高质量文档的代名词,我们训练了一个分类器来区分这些和原始的"共同爬行".
<a id="S0756"></a> Source: p.43 S0756
Original: We then used this classifier to re-sample Common Crawl by prioritizing documents which were predicted by the classifier to be higher quality.
中文: 然后我们用这个分类器来重新取出常见的爬行,通过优先排序被分类器预测为质量更高的文件.
<a id="S0757"></a> Source: p.43 S0757
Original: The classifier is trained using logistic regression classifier with features from Spark’s standard tokenizer and HashingTF 10.
中文: 分类器使用Spark标准标注器和HashingTF 10的特性的后勤回归分解器进行训练。
<a id="S0758"></a> Source: p.43 S0758
Original: For the positive examples, we used a collection of curated datasets such as WebText, Wikiedia, and our web books corpus as the positive examples, and for the negative examples, we used unfiltered Common Crawl.
中文: 对于积极的例子,我们使用 WebText , Wikiedia 等被整理的数据集, 以及我们的网络书目作为积极的例子, 对于消极的例子,我们使用未经过滤的普通爬行。
<a id="S0759"></a> Source: p.43 S0759
Original: We used this classifier to score Common Crawl documents.
中文: 我们用这个分类器来评分常见的爬行文件.
<a id="S0760"></a> Source: p.43 S0760
Original: We kept each document in our dataset iff np.random.pareto(α) > 1 − document_score We chose α = 9 in order to take mostly documents the classifier scored highly, but still include some documents that were out of distribution. α was chosen to match the distribution of scores from our classifier on WebText.
中文: 我们把每个文件保存在我们的数据集iff np.random.pareto(α) > 1 - 文档 分数 我们选择了 α = 9 , 以获取大部分文件 分类者得分很高,但仍包括一些未分发的文件。 α被选中来匹配 WebText上我们分类器的分数分布.
<a id="S0761"></a> Source: p.43 S0761
Original: We found this re-weighting increased quality as measured by loss on a range of out-of-distribution generative text samples. 2.
中文: 我们发现,这种重新加权提高了质量,以一系列分配外基因文字样本的损失来衡量。 2. 联合国
<a id="S0762"></a> Source: p.43 S0762
Original: To further improve model quality and prevent overfitting (which becomes increasingly important as model capacity increases), we fuzzily deduplicated documents (i.e. removed documents with high overlap with other documents) within each dataset using Spark’s MinHashLSH implementation with 10 hashes, using the same features as were used for classification above.
中文: 为了进一步提高模型质量并防止过度调整(随着模型容量的提高而变得日益重要),我们利用Spark的MinHashLSH执行,使用与上述分类相同的特性,在每一数据集中模糊地去除文件(即删除了与其他文件高度重叠的文件)。
<a id="S0763"></a> Source: p.43 S0763
Original: We also fuzzily removed WebText from Common Crawl.
中文: 我们还模糊地删除了普通爬行的WebText。
<a id="S0764"></a> Source: p.43 S0764
Original: Overall this decreased dataset size by an average of 10%.
中文: 总的来说,数据集的尺寸平均减少了10%。
<a id="S0765"></a> Source: p.43 S0765
Original: After filtering for duplicates and quality, we also partially removed text occurring in benchmark datasets, described in Appendix C. B Details of Model Training To train all versions of GPT-3, we use Adam with β = 0.9, β = 0.95, and (cid:15) = 10−8, we clip the global norm of the 1 2 gradient at 1.0, and we use cosine decay for learning rate down to 10% of its value, over 260 billion tokens (after 260 billion tokens, training continues at 10% of the original learning rate).
中文: 在对重复和质量进行过滤后,我们还将基准数据集中出现的案文部分删除,附录C.B " 模式培训详情 " 中对此进行了描述。 培训所有版本的GPT-3, 我们使用 Adam 的 β = 0.9, β = 0.95, 和 (cid: 15) = 10 - 8, 我们把 1 2 梯度的全球规范剪接到 1.0, 我们使用 余弦衰减 学习率降至其值的 10%, 超过 260亿令牌(在 260亿令牌之后,培训继续以 原学习率的 10% ) 。
<a id="S0766"></a> Source: p.43 S0766
Original: There is a linear LR warmup over the first 375 million tokens.
中文: 有线性LR热量 超过前3.75亿个符号。
<a id="S0767"></a> Source: p.43 S0767
Original: We also gradually increase the batch size linearly from a small value (32k tokens) to the full value over the first 4-12 billion tokens of training, depending on the model size.
中文: 我们还根据型号大小,逐步将批量大小由小值(32k令牌)线性地提升到前40-120亿令牌的全值.
<a id="S0768"></a> Source: p.43 S0768
Original: Data are sampled without replacement during training (until an epoch boundary is reached) to minimize overfitting.
中文: 在训练期间(直到达到一个划时代的边界)对数据进行采样而不进行替换,以尽量减少过度调整。
<a id="S0769"></a> Source: p.43 S0769
Original: All models use weight decay of 0.1 to provide a small amount of regularization [LH17].
中文: 所有模型都使用0.1的重量衰变来提供少量的正态化[LH17].
<a id="S0770"></a> Source: p.43 S0770
Original: During training we always train on sequences of the full n = 2048 token context window, packing multiple ctx documents into a single sequence when documents are shorter than 2048, in order to increase computational efficiency.
中文: 在训练期间,我们总是对全n = 2048个符号上下文窗口的序列进行训练,在文档短于 2048 时将多个 ctx 文档打入一个序列,以提高计算效率.
<a id="S0771"></a> Source: p.43 S0771
Original: Sequences with multiple documents are not masked in any special way but instead documents within a sequence are delimited with a special end of text token, giving the language model the information necessary to infer that context separated by the end of text token is unrelated.
中文: 带有多文档的序列不以任何特殊方式被遮掩,而是在一个序列内的文档被以特殊的文本符尾来划分,使语言模型获得必要的信息来推断被文本符尾相隔的上下文是无关的.
<a id="S0772"></a> Source: p.43 S0772
Original: This allows for efficient training without need for any special sequence-specific masking. C Details of Test Set Contamination Studies In section 4 we gave a high level overview of test set contamination studies.
中文: 这样可以进行高效训练,而不需要任何特殊的序列特异性面具. C 试验集污染研究的细节 在第4节中,我们高度概括了测试成套污染研究。
<a id="S0773"></a> Source: p.43 S0773
Original: In this section we provide details on methodology and results.
中文: 本节详细介绍方法和结果。
<a id="S0774"></a> Source: p.43 S0774
Original: Initial training set filtering We attempted to remove text occurring in benchmarks from training data by searching for 13−gram overlaps between all test/development sets used in this work and our training data, and we removed the colliding 13−gram as well as a 200 character window around it, splitting the original document into pieces.
中文: 初步培训集过滤 我们试图从培训数据中去除基准中出现的案文,为此要寻找这项工作中使用的所有测试/开发套件与我们的培训数据之间的13克重叠,我们移除了相撞的13克和周围的200个字符窗口,将原始文件一分为二。
<a id="S0775"></a> Source: p.43 S0775
Original: For filtering purposes we define a gram as a lowercase, whitespace delimited word with no punctuation.
中文: 为了过滤的目的,我们定义一克为小写,白空被分隔的单词没有分音符.
<a id="S0776"></a> Source: p.43 S0776
Original: Pieces less than 200 characters long were discarded.
中文: 不到200个字符的碎片被丢弃。
<a id="S0777"></a> Source: p.43 S0777
Original: Documents split into more than 10 pieces were considered contaminated and 10https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.feature.HashingTF 43
中文: 分为10多块的文件被认为受到污染,10https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#pyspark.ml.feature.HashingTF 43
<a id="S0778"></a> Source: p.44 S0778
Original: Originally we removed entire documents given a single collision, but that overly penalized long documents such as books for false positives.
中文: 最初我们删除了整个文件 一次碰撞, 但过度惩罚 长文件,如书籍 虚假阳性。
<a id="S0779"></a> Source: p.44 S0779
Original: An example of a false positive might be a test set based on Wikipedia, in which the Wikipedia article quotes a single line from a book.
中文: 假阳性的一个实例可能是基于维基百科的测试集,其中维基百科的文章引用了一书的一行.
<a id="S0780"></a> Source: p.44 S0780
Original: We ignored 13−grams that matched more than 10 training documents, as inspection showed the majority of these to contain common cultural phrases, legal boilerplate, or similar content that we likely do want the model to learn, rather than undesired specific overlaps with test sets.
中文: 我们忽略了符合10多份培训文件的13克,因为检查表明,大多数培训文件都含有共同的文化词语、法律锅炉板或类似的内容,我们可能希望这些模型能够学习,而不是不受欢迎的具体内容与测试设备重叠。
<a id="S0781"></a> Source: p.44 S0781
Original: Examples for various frequencies can be found in the GPT-3 release repository11.
中文: 各种频率的例子可参见GPT-3发布存储器11.
<a id="S0782"></a> Source: p.44 S0782
Original: Overlap methodology For our benchmark overlap analysis in Section 4, we used a variable number of words N to check for overlap for each dataset, where N is the 5th percentile example length in words, ignoring all punctuation, whitespace, and casing.
中文: 重叠方法 对于我们在第4节的基准重叠分析,我们用一个可变的N字数来检查每个数据集的重叠,其中N是字数中第5%的例子长度,忽略了所有的出点,白空和外壳.
<a id="S0783"></a> Source: p.44 S0783
Original: Due to spurious collisions at lower values of N we use a minimum value of 8 on non-synthetic tasks.
中文: 由于N值较低时的假相撞 我们使用8的最小值来完成非合成任务
<a id="S0784"></a> Source: p.44 S0784
Original: For performance reasons, we set a maximum value of 13 for all tasks.
中文: 出于业绩原因,我们为所有任务设定了13个最大值.
<a id="S0785"></a> Source: p.44 S0785
Original: Values for N and the amount of data marked as dirty are shown in Table C.1.
中文: N值和被标记为脏的数据数量见表C.1。
<a id="S0786"></a> Source: p.44 S0786
Original: Unlike GPT-2’s use of bloom filters to compute probabilistic bounds for test contamination, we used Apache Spark to compute exact collisions across all training and test sets.
中文: 我们使用Apache Spark来计算所有训练和测试机组的确切相撞情况。
<a id="S0787"></a> Source: p.44 S0787
Original: We compute overlaps between test sets and our full training corpus, even though we only trained on 40% of our filtered Common Crawl documents per Section 2.2.
中文: 我们计算测试机组和我们全部的训练装备之间的重叠 尽管我们每2.2节只训练40%的 过滤出的共同爬行文件
<a id="S0788"></a> Source: p.44 S0788
Original: We define a ‘dirty’ example as one with any N -gram overlap with any training document, and a ‘clean’ example as one with no collision.
中文: 我们把 " 脏 " 定义为一个与任何训练文件有N-克重叠的例子,把 " 干净 " 定义为一个没有碰撞的例子。
<a id="S0789"></a> Source: p.44 S0789
Original: Test and validation splits had similar contamination levels despite some test splits being unlabeled.
中文: 试验和鉴定分解的污染程度相似,尽管一些试验分解没有贴标签。
<a id="S0790"></a> Source: p.44 S0790
Original: Due to a bug revealed by this analysis, filtering described above failed on long documents such as books.
中文: 由于这一分析揭示出一个错误,上述对书籍等长文档的过滤失败.
<a id="S0791"></a> Source: p.44 S0791
Original: Because of cost considerations it was infeasible to retrain the model on a corrected version of the training dataset.
中文: 由于费用方面的考虑,无法对模型进行关于培训数据集校正版的再培训。
<a id="S0792"></a> Source: p.44 S0792
Original: As such, several language modeling benchmarks plus the Children’s Book Test showed almost complete overlap, and therefore were not included in this paper.
中文: 因此,若干语文模拟基准加上儿童图书测试几乎完全重叠,因此没有列入本文。
<a id="S0793"></a> Source: p.44 S0793
Original: Overlaps are shown in Table C.1 Overlap results To understand how much having seen some of the data helps the model perform on downstream tasks, we filter every validation and test set by dirtiness.
中文: 表C.1 重叠结果 为了了解有多少数据能帮助模型执行下游任务, 我们过滤了每一个由肮脏设定的验证和测试。
<a id="S0794"></a> Source: p.44 S0794
Original: Then we run evaluation on the clean-only examples and report the relative percent change between the clean score and the original score.
中文: 然后,我们运行评估 仅干净的例子 并报告相对百分比 变化 从干净的分数和原始分数。
<a id="S0795"></a> Source: p.44 S0795
Original: If the clean score is more than 1% or 2% worse than the overall score, it suggests the model may have overfit to the examples it has seen.
中文: 如果干净分数比整体分数差1%或2%以上,则表明模型可能与所见到的例子过于相配.
<a id="S0796"></a> Source: p.44 S0796
Original: If the clean score is significantly better, our filtering scheme may have preferentially marked easier examples as dirty.
中文: 如果干净的分数明显好一些,我们的过滤计划可能会优先标出比较容易的例子作为肮脏.
<a id="S0797"></a> Source: p.44 S0797
Original: This overlap metric tends to show a high rate of false positives for datasets that contain background information (but not answers) drawn from the web (such as SQuAD, which draws from Wikipedia) or examples less than 8 words long, which we ignored in our filtering process (except for wordscrambling tasks).
中文: 这个重叠的度量法倾向于显示包含从网络中取出的背景信息(但不是答案)的数据集(例如从维基百科中取出SQuAD)或长度小于8个字的例子的假阳性率很高,我们在过滤过程中忽略了这些例子(文字拼接任务除外).
<a id="S0798"></a> Source: p.44 S0798
Original: One instance where this technique seems to fail to give good signal is DROP, a reading comprehension task in which 94% of the examples are dirty.
中文: 这个技术似乎无法给出良好信号的例子是DROP,
<a id="S0799"></a> Source: p.44 S0799
Original: The information required to answer the question is in a passage provided to the model, so having seen the passage during training but not the questions and answers does not meaningfully constitute cheating.
中文: 回答问题所需的信息是在提供给模型的一段中,因此,在培训过程中看到通过,但问答没有真正构成作弊。
<a id="S0800"></a> Source: p.44 S0800
Original: We confirmed that every matching training document contained only the source passage, and none of the questions and answers in the dataset.
中文: 我们确认,每一份匹配培训文件只包含源代码,数据集中没有问答.
<a id="S0801"></a> Source: p.44 S0801
Original: The more likely explanation for the decrease in performance is that the 6% of examples that remain after filtering come from a slightly different distribution than the dirty examples.
中文: 对性能下降的更可能的解释是,过滤后留下的6%的例子来自与肮脏的例子略有不同的分布.
<a id="S0802"></a> Source: p.44 S0802
Original: Figure 4.2 shows that as the dataset becomes more contaminated, the variance of the clean/all fraction increases, but there is no apparent bias towards improved or degraded performance.
中文: 图4.2显示,随着数据集受污染程度的提高,干净/全部分数的差异会增加,但并没有明显偏向于提高或降低性能。
<a id="S0803"></a> Source: p.44 S0803
Original: This suggests that GPT-3 is relatively insensitive to contamination.
中文: 这表明GPT-3对污染相对不敏感.
<a id="S0804"></a> Source: p.44 S0804
Original: See Section 4 for details on the datasets we flagged for further review. 11https://github.com/openai/gpt-3/blob/master/overlap_frequency.md 44
中文: 关于我们标出供进一步审查的数据集详情,见第4节。 11https://github.com/openai/gpt-3/blob/master/overlap 频率.md 44 (中文(简体) ).
<a id="S0805"></a> Source: p.45 S0805
Original: Relative Total Dirty Dirty Clean Clean Clean Difference Name Split Metric N Acc/F1/BLEU Count Acc/F1/BLEU Count Acc/F1/BLEU Count Percentage Clean vs All Quac dev f1 13 44.3 7353 44.3 7315 54.1 38 1% 20% SQuADv2 dev f1 13 69.8 11873 69.9 11136 68.4 737 6% -2% DROP dev f1 13 36.5 9536 37.0 8898 29.5 638 7% -21% Symbol Insertion dev acc 7 66.9 10000 66.8 8565 67.1 1435 14% 0% CoQa dev f1 13 86.0 7983 85.3 5107 87.1 2876 36% 1% ReCoRD dev acc 13 89.5 10000 90.3 6110 88.2 3890 39% -1% Winograd test acc 9 88.6 273 90.2 164 86.2 109 40% -3% BoolQ dev acc 13 76.0 3270 75.8 1955 76.3 1315 40% 0% MultiRC dev acc 13 74.2 953 73.4 558 75.3 395 41% 1% RACE-h test acc 13 46.8 3498 47.0 1580 46.7 1918 55% 0% LAMBADA test acc 13 86.4 5153 86.9 2209 86.0 2944 57% 0% LAMBADA (No Blanks) test acc 13 77.8 5153 78.5 2209 77.2 2944 57% -1% WSC dev acc 13 76.9 104 73.8 42 79.0 62 60% 3% PIQA dev acc 8 82.3 1838 89.9 526 79.3 1312 71% -4% RACE-m test acc 13 58.5 1436 53.0 366 60.4 1070 75% 3% De→En 16 test bleu-sb 12 43.0 2999 47.4 739 40.8 2260 75% -5% En→De 16 test bleu-sb 12 30.9 2999 32.6 739 29.9 2260 75% -3% En→Ro 16 test bleu-sb 12 25.8 1999 24.9 423 26.1 1576 79% 1% Ro→En 16 test bleu-sb 12 41.3 1999 40.4 423 41.6 1576 79% 1% WebQs test acc 8 41.5 2032 41.6 428 41.5 1604 79% 0% ANLI R1 test acc 13 36.8 1000 40.5 200 35.9 800 80% -3% ANLI R2 test acc 13 34.0 1000 29.4 177 35.0 823 82% 3% TriviaQA dev acc 10 71.2 7993 70.8 1390 71.3 6603 83% 0% ANLI R3 test acc 13 40.2 1200 38.3 196 40.5 1004 84% 1% En→Fr 14 test bleu-sb 13 39.9 3003 38.3 411 40.3 2592 86% 1% Fr→En 14 test bleu-sb 13 41.4 3003 40.9 411 41.4 2592 86% 0% WiC dev acc 13 51.4 638 53.1 49 51.3 589 92% 0% RTE dev acc 13 71.5 277 71.4 21 71.5 256 92% 0% CB dev acc 13 80.4 56 100.0 4 78.8 52 93% -2% Anagrams 2 dev acc 2 40.2 10000 76.2 705 37.4 9295 93% -7% Reversed Words dev acc 2 0.4 10000 1.5 660 0.3 9340 93% -26% OpenBookQA test acc 8 65.4 500 58.1 31 65.9 469 94% 1% ARC (Easy) test acc 11 70.1 2268 77.5 89 69.8 2179 96% 0% Anagrams 1 dev acc 2 15.0 10000 49.8 327 13.8 9673 97% -8% COPA dev acc 9 93.0 100 100.0 3 92.8 97 97% 0% ARC (Challenge) test acc 12 51.6 1144 45.2 31 51.8 1113 97% 0% HellaSwag dev acc 13 79.3 10042 86.2 152 79.2 9890 98% 0% NQs test acc 11 29.9 3610 32.7 52 29.8 3558 99% 0% Cycled Letters dev acc 2 38.6 10000 20.5 73 38.7 9927 99% 0% SAT Analogies dev acc 9 65.8 374 100.0 2 65.6 372 99% 0% StoryCloze test acc 13 87.7 1871 100.0 2 87.6 1869 100% 0% Winogrande dev acc 13 77.7 1267 - 0 77.7 1267 100% 0% Table C.1: Overlap statistics for all datasets sorted from dirtiest to cleanest.
中文: 净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净净 Acc/F1/BLEU伯爵 Acc/F1/BLEU伯爵 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 净能计数 38 89.9 526 79.3 1312 71%-4% RACE-m测试 acc 13 58.5 1436 53.0 1% 罗 16测试 Bleu-sb 12 43.0 2999 47.4 739 40.8 2260 75%-5% 恩 3 16测试 Bleu-sb 12 30.9 2999 32.6 739 29.9 2260 75-3% 恩 16测试 Bleu-sb 12 25.8 1999 24.9 423 26.1 1576 79% 罗 16 Bleu-sb 12 41.3 1999 40.4 4 7303 6 6 030 6 003 4 4 010 4 4 4 4 010 4 3 4 4 010 4 4 3 4 4 4 4 4 4 4 3 4 4 4 4 3 4 3 4 4 4 4 4 4 4 4 4 4 4 4 4 3 4 4 4 0.4 - 26% OpenBookQA测试 acc 12 51.644 45.2 31 65.9 469 94% ARC(Easy)测试 acc 11 70.1 2268 77.5 89 69.8 2179 96% Anagrams 1 Dev acc 2 15.8 9673 97-8% COPA dev acc 93.0 1000.0 3 92.8 97.0% ARC(Challenge)测试 Acc 11.44 45.2 31 51.813 97.0% HelaSwag dev acc 13 79.3 10042 86.2 152 79.2 9890 98% NQs测试 Anagrams 5 29.8 3558 99.0% 循环数据 Dev C2 38.6 027 99.0% SAT Analogies 99% Devat 975 974 97° C974° C974° C974° C974° C100.07° C10.
<a id="S0806"></a> Source: p.45 S0806
Original: We consider a dataset example dirty if it has a single N -gram collision with any document in our training corpus. “Relative Difference Clean vs All” shows the percent change in performance between only the clean examples vs all the examples in the benchmark. “Count” shows the number of examples. “Clean percentage” is the percent of examples that are clean vs total.
中文: 我们认为,如果一个数据集与我们训练手册中的任何文件发生单一的N克相撞,那么它就是一个肮脏的例子。 “Relative Difference Clean vs All ” 显示只有干净的例子与基准中的所有例子之间业绩的百分比变化。 “国家”显示实例的数目。 “净百分比”是指净数相对于总数的百分比。
<a id="S0807"></a> Source: p.45 S0807
Original: For “Acc/F1/BLEU” we use the metric specified in “Metric”.
中文: 对于“Acc/F1/BLEU”,我们使用“计量”中指定的衡量标准。
<a id="S0808"></a> Source: p.45 S0808
Original: These scores come from evaluations with a different seed for the random examples used for in-context learning, and will therefore differ slightly from the scores elsewhere in the paper. 45
中文: 这些分数来自评价,对文中学习所用的随机例子有不同的种子,因此与论文其他分数略有不同。 第45条
<a id="S0809"></a> Source: p.46 S0809
Original: D Total Compute Used to Train Language Models This appendix contains the calculations that were used to derive the approximate compute used to train the language models in Figure 2.2.
中文: D 用于培训语言模型的总计算 本附录载有用来计算图2.2中用于培训语言模型的大致计算。
<a id="S0810"></a> Source: p.46 S0810
Original: As a simplifying assumption, we ignore the attention operation, as it typically uses less than 10% of the total compute for the models we are analyzing.
中文: 作为一个简化的假设,我们忽略了关注操作,因为它通常使用不到总计算值的10%用于我们分析的模型.
<a id="S0811"></a> Source: p.46 S0811
Original: Calculations can be seen in Table D.1 and are explained within the table caption.
中文: 计算情况见表D.1,并在表标题中说明。
<a id="S0812"></a> Source: p.46 S0812
Original: Fwd-pass Frac of Total train Total train Flops flops per params active compute compute Params Training tokens per param Mult for active param for each Model (PF-days) (flops) (M) (billions) per token bwd pass per token token T5-Small 2.08E+00 1.80E+20 60 1,000 3 3 1 0.5 T5-Base 7.64E+00 6.60E+20 220 1,000 3 3 1 0.5 T5-Large 2.67E+01 2.31E+21 770 1,000 3 3 1 0.5 T5-3B 1.04E+02 9.00E+21 3,000 1,000 3 3 1 0.5 T5-11B 3.82E+02 3.30E+22 11,000 1,000 3 3 1 0.5 BERT-Base 1.89E+00 1.64E+20 109 250 6 3 2 1.0 BERT-Large 6.16E+00 5.33E+20 355 250 6 3 2 1.0 RoBERTa-Base 1.74E+01 1.50E+21 125 2,000 6 3 2 1.0 RoBERTa-Large 4.93E+01 4.26E+21 355 2,000 6 3 2 1.0 GPT-3 Small 2.60E+00 2.25E+20 125 300 6 3 2 1.0 GPT-3 Medium 7.42E+00 6.41E+20 356 300 6 3 2 1.0 GPT-3 Large 1.58E+01 1.37E+21 760 300 6 3 2 1.0 GPT-3 XL 2.75E+01 2.38E+21 1,320 300 6 3 2 1.0 GPT-3 2.7B 5.52E+01 4.77E+21 2,650 300 6 3 2 1.0 GPT-3 6.7B 1.39E+02 1.20E+22 6,660 300 6 3 2 1.0 GPT-3 13B 2.68E+02 2.31E+22 12,850 300 6 3 2 1.0 GPT-3 175B 3.64E+03 3.14E+23 174,600 300 6 3 2 1.0 Table D.1: Starting from the right hand side and moving left, we begin with the number of training tokens that each model was trained with.
中文: Fwd-pass Frac 总列车 总列车 总列车 总列车 总列车 总列车总列车 总列车总列车总列车总列车总列车 总列车总列车总列车总列车总列车总列车总分 列车总列车总列车总分 列车总列车总列车总分 列车总列车总列车总列车总分 列车总列车总列车总分 列车总列车总列车总列车总分 列车总列车总列车总列车总分 列车总列车总分列车总列车总列车总列车总分 列车总分列车总列车总分列车总列车总列车总列车总分 列车总列车总列车总分 列车总列车总列车总分数 列车总列车总分 列车总列车总列车总列车总列车总列车总列车总列车总分列车总分列车总列车总列车总 6.7B 1.39E+02 1.20E+22 6.660 300 6 3 2 1.0 GPT-3 13B 2.68E+02 2.31E+22 12 850300 6 2 1.0 GPT-3 175B 3.64E+03 3.14E+23 174600 300 6 3 2 1.0 表D.1:从右手边开始并左移,我们从每个型号所训练的训练标志数量开始.
<a id="S0813"></a> Source: p.46 S0813
Original: Next we note that since T5 uses an encoder-decoder model, only half of the parameters are active for each token during a forward or backwards pass.
中文: 接下来我们注意到,由于T5使用编码器-解码器模型,在前向或后向通过时,每个指针只有一半的参数是活性的.
<a id="S0814"></a> Source: p.46 S0814
Original: We then note that each token is involved in a single addition and a single multiply for each active parameter in the forward pass (ignoring attention).
中文: 然后,我们注意到,每个符号都涉及前传中每个活性参数的单一加法和单一乘法(引起注意)。
<a id="S0815"></a> Source: p.46 S0815
Original: Then we add a multiplier of 3x to account for the backwards pass (as computing both ∂params and ∂acts use a similar amount of compute as the ∂loss ∂loss forwards pass.
中文: 然后我们再加一个3x的乘数来计算后传(因为计算 QQparams 和 QXacts 都使用与 QXLoss 向前传相类似的计算量).
<a id="S0816"></a> Source: p.46 S0816
Original: Combining the previous two numbers, we get the total flops per parameter per token.
中文: 结合前两个数字,我们得到每个参数的总分.
<a id="S0817"></a> Source: p.46 S0817
Original: We multiply this value by the total training tokens and the total parameters to yield the number of total flops used during training.
中文: 我们把这一数值乘以训练符号总数和总参数,得出训练期间使用的总分数。
<a id="S0818"></a> Source: p.46 S0818
Original: We report both flops and petaflop/s-day (each of which are 8.64e+19 flops). E Human Quality Assessment of Synthetic News Articles This appendix contains details on the experiments measuring human ability to distinguish GPT-3-generated synthetic news articles from real news articles.
中文: 我们同时报告flops和petaflop/s-day(每个都为8.64e+19 flops). E 合成新闻文章的人类质量评估 本附录详细介绍了测量人类区分出GPT-3所生成的合成新闻文章与真实新闻文章的实验能力.
<a id="S0819"></a> Source: p.46 S0819
Original: We first describe the experiments on the ∼ 200 word news articles, and then describe the preliminary investigation of ∼ 500 word news articles generated by GPT-3.
中文: 我们首先描述了200个单词新闻文章的实验,然后描述了对由GPT-3生成的500个单词新闻文章的初步调查.
<a id="S0820"></a> Source: p.46 S0820
Original: Participants: We recruited 718 unique participants to take part in 6 experiments. 97 participants were excluded for failing an internet check question, leaving a total of 621 participants: 343 male, 271 female, and 7 other.
中文: 与会者:我们招募了718名独特的参与者参加6个实验。 97名参与者因未通过互联网检查而被排除在外,共有621名参与者:343名男性、271名女性和7名其他参与者。
<a id="S0821"></a> Source: p.46 S0821
Original: Mean participant age was ∼ 38 years old.
中文: 平均参加年龄为-38岁。
<a id="S0822"></a> Source: p.46 S0822
Original: All participants were recruited through Positly, which maintains a whitelist of high-performing workers from Mechanical Turk.
中文: 所有参与者都是通过波西特利招聘的,而波西特利则保持着来自机械土克的优秀工人的白名单.
<a id="S0823"></a> Source: p.46 S0823
Original: All participants were US-based but there were no other demographic restrictions.
中文: 所有参与者都是以美国为基地,但没有其他的人口限制。
<a id="S0824"></a> Source: p.46 S0824
Original: Participants were paid $12 for their participation, based on a task time estimate of 60 minutes determined by pilot runs.
中文: 根据试点运行确定的60分钟的任务时间估计,与会者的与会费用为12美元。
<a id="S0825"></a> Source: p.46 S0825
Original: In order to ensure that the sample of participants for each experiment quiz was unique, participants were not allowed to take part in an experiment more than once.
中文: 为了确保每次实验测试的参与者样本是独一无二的,不允许参与者参加一次以上的实验.
<a id="S0826"></a> Source: p.46 S0826
Original: Procedure and design: We arbitrarily selected 25 news articles that appeared in newser.com in early 2020.
中文: 程序与设计:我们任意选择了25篇2020年初出现在Newser.com的新闻文章.
<a id="S0827"></a> Source: p.46 S0827
Original: We used the article titles and subtitles to produce outputs from the 125M, 350M, 760M, 1.3B, 2.7B, 6.7B, 13.0B, and 200B (GPT-3) parameter language models.
中文: 我们使用文章标题和字幕来制作125M,350M,760M,1.3B,2.7B,6.7B,13.0B,和200B(GPT-3)参数语言模型的输出.
<a id="S0828"></a> Source: p.46 S0828
Original: Five outputs per question were generated by each model and the generation with a word count closest to that of the human written article was selected automatically.
中文: 每个问题都由每个模型产生五个产出,并且自动选择最接近人文文章的字数的一代。
<a id="S0829"></a> Source: p.46 S0829
Original: This was to minimize the effect that completion length might have on participants’ judgments.
中文: 这样做是为了尽量减少完成时间对参与人判决可能产生的影响。
<a id="S0830"></a> Source: p.46 S0830
Original: The same output procedure for each model with the exception of the removal of the intentionally bad control model, as described in the main text. 46
中文: 每种型号的输出程序相同,但正文所描述的故意坏控制型号被去除除外. 第46条
<a id="S0831"></a> Source: p.47 S0831
Original: Average Participants Participants Genders Mean Word Count Model Recruited Excluded (m:f:other) Age (human:model) Control 76 7 32:37:0 39 216:216 GPT-3 Small 80 7 41:31:1 40 216:188 GPT-3 Medium 80 7 46:28:2 39 216:202 GPT-3 Large 81 24 46:28:2 37 216:200 GPT-3 XL 79 14 32:32:1 38 216:199 GPT-3 2.7B 80 11 36:33:0 40 216:202 GPT-3 6.7B 76 5 46:28:2 37 216:195 GPT-3 13.0B 81 13 46:28:2 37 216:209 GPT-3 175B 80 9 42:29:0 37 216:216 Table E.1: Participant details and article lengths for each experiment to evaluate human detection of ∼ 200 word model generated news articles.
中文: 平均参与者性别平均字数计数模式被排除(m:f:other) 年龄(人:型号)控制 76 7 32:37:0 39 216 GBT-3 小 80 7 41:31 1 216 中 80 46:28:188 GBT-3 中 80 7 26:28:2 39 216:202 GPT-3 大 81 46:28:2 37 216 GBT-3 XL 79 14 32:32:1 38 216:199 GPT-3 2.7B 80 11 36:33:0 40 216:202 GPT-3 6.7B 76 5 46:28:2 37 PT-3 13.0 B 81 13 46:28:216:209 GPT-3 175B 80 9 42:29:0 37 216:216 表E.1:每次实验的参与者细节和文章长度,用以评价人类探测 ∼ 200字模型生成的新闻文章.
<a id="S0832"></a> Source: p.47 S0832
Original: Participants were excluded due to internet check fails.
中文: 由于因特网检查失败,与会者被排除在外。
<a id="S0833"></a> Source: p.47 S0833
Original: Figure E.1: Participants spend more time trying to identify whether each news article is machine generated as model size increases.
中文: 图E.1:参与者花更多的时间试图确定每篇新闻文章是否是随着模型大小的增加而生成的机器.
<a id="S0834"></a> Source: p.47 S0834
Original: Duration on the control model is indicated with the dashed line.
中文: 控制模型的持续时间用破折线标明。
<a id="S0835"></a> Source: p.47 S0835
Original: Line of best fit is a linear model on a log scale with 95% confidence intervals.
中文: 最合适的一行是日志尺度上的线性模型,置信间隔为95%.
<a id="S0836"></a> Source: p.47 S0836
Original: In each experiment, half of the participants were randomly assigned to quiz A and half were randomly assigned to quiz B.
中文: 在每次实验中,一半的参与者被随机分配到A类测试中,一半被随机分配到B类测试中.
<a id="S0837"></a> Source: p.47 S0837
Original: Each quiz consisted of 25 articles: half (12-13) were human written and half (12-13) were model generated: the articles with human written completions in quiz A had model generated completions in quiz B and vice versa.
中文: 每期测验由25篇文章组成:一半(12-13)是人写作,一半(12-13)是模型生成:在测验A中人类写作完成的文章在测验B中产生完成,反之亦然。
<a id="S0838"></a> Source: p.47 S0838
Original: The order of quiz question was shuffled for each participant.
中文: 每个参与者的问答顺序被洗牌.
<a id="S0839"></a> Source: p.47 S0839
Original: Participants could leave comments and were asked to indicate if they had seen the articles before.
中文: 与会者可留下评论并被要求说明他们以前是否看过这些条款。
<a id="S0840"></a> Source: p.47 S0840
Original: Participants were instructed not to look up the articles or their content during the quiz and at the end of the quiz were asked if they had looked anything up during the quiz.
中文: 指示与会者在问答期间不要查阅文章或其内容,并在问答后询问他们在问答期间是否查阅过任何内容。
<a id="S0841"></a> Source: p.47 S0841
Original: Statistical Tests: To compare means on the different runs, we performed a two-sample t-test for independent groups for each model against the control.
中文: 统计测试:为了比较不同跑道上的各种手段,我们针对每个模型对照控制,对独立组进行了两样T测试.
<a id="S0842"></a> Source: p.47 S0842
Original: This was implemented in Python using the scipy.stats.ttest_ind function.
中文: 这在Python中被使用 scipy 执行. 数据。 test ind 函数 。
<a id="S0843"></a> Source: p.47 S0843
Original: When plotting a regression line in the graph of average participant accuracy vs model size, we fit a power law of the form ax−b.
中文: 在平均参与者精度图和模型大小图中绘制回归线时,我们符合形式轴-b的动力定律。
<a id="S0844"></a> Source: p.47 S0844
Original: The 95% confidence intervals were estimated from the t-distribution of the sample mean.
中文: 95%的置信间隔是根据样本的t分配平均值估算的。
<a id="S0845"></a> Source: p.47 S0845
Original: Duration statistics: In the main text, we discussed the finding that the ability of human participants to distinguish model and human generated news articles decreases as our models become larger.
中文: 持续时间统计: 在正文中,我们讨论了一个结论,即随着我们的模型变得更大,人类参与者区分模型和由人类产生的新闻文章的能力会下降.
<a id="S0846"></a> Source: p.47 S0846
Original: We have also found that the average time spent for a given set of questions increases as the model size increases, as shown in Figure E.1.
中文: 我们还发现,如图E.1所示,随着模型尺寸的增加,用于一组特定问题的平均时间会增加。
<a id="S0847"></a> Source: p.48 S0847
Original: Average Participants Participants Genders Mean Word Count Model Recruited Excluded (m:f:other) Age (human:model) Control 79 17 32:37:0 39 569:464 GPT-3 175B 81 19 32:30:0 40 569:498 Table E.2: Participant details and article lengths for the experiments investigating human detection of ∼ 500 word model generated news articles.
中文: 平均参与者性别平均字数计算模型被排除(m:f:other) 年龄(人:模)控制 79 17 32:37:0 39 569:464 GPT-3 175B 81 19 32:30:0 40 569:498 表 E.2: 调查人类探测出 ∼ 500字数模型的实验的参与者细节和文章长度生成了新闻文章.
<a id="S0848"></a> Source: p.48 S0848
Original: Participants were excluded due to internet check fails. accuracy scores despite increased time investment from participants supports the finding that larger models generate harder-to-distinguish news articles.
中文: 由于因特网检查失败,与会者被排除在外。 尽管参与者增加了时间投资,但准确性得分仍然支持以下结论,即更大的模型会产生难以区分的新闻文章。
<a id="S0849"></a> Source: p.48 S0849
Original: Preliminary investigation of ∼ 500 word articles: We recruited 160 unique US-based participants to take part in 2 experiments through Positly (details are given in Table E.2).
中文: 初步调查:我们招募了160名美国独有的参与者,通过Positly参加两次试验(详情见表E.2)。
<a id="S0850"></a> Source: p.48 S0850
Original: We randomly selected 12 Reuters world news articles from late 2019 and created a context for GPT-3 175B that consisted of a single Reuters article not in this set of 12.
中文: 我们随机选取了12篇路透社2019后期的世界新闻文章,并创建了GPT-3175B的上下文,由1篇路透社文章组成,不在此12集.
<a id="S0851"></a> Source: p.48 S0851
Original: We then used the article titles and Reuters locations to generate completions from GPT-3 175B and the 160M control model from the previous experiments.
中文: 随后,我们利用文章标题和路透社位置生成了GPT-3175B和之前实验的160M控制模型的完成.
<a id="S0852"></a> Source: p.48 S0852
Original: These were used to create two 12-question quizzes per model, each consisting of half human written and half model generated articles.
中文: 这些被用来制作每个模型的两个12个问题测试,每个测试由半个人类写作和半个模型生成的文章组成.
<a id="S0853"></a> Source: p.48 S0853
Original: Comprehension questions were added and articles were shown to participants in 3 stages at 30 second intervals to encourage closer reading.
中文: 增加了一些理解性问题,并分三个阶段向与会者放映了文章,每半个阶段30次,以鼓励更深入地阅读。
<a id="S0854"></a> Source: p.48 S0854
Original: Participants were paid $12 for this task.
中文: 为这项工作向与会者支付了12美元。
<a id="S0855"></a> Source: p.48 S0855
Original: Model generation selection methods, exclusion criteria, and statistical tests mirror those of the previous experiments. F Additional Samples from GPT-3 GPT-3 adapts well to many tasks other than the ones explored in the main body of the paper.
中文: 模型生成选择方法,排除标准和统计测试与之前的实验方法相仿. F 来自GPT-3 GPT-3的额外样本很好地适应了除论文正文所探讨的任务以外的许多任务.
<a id="S0856"></a> Source: p.48 S0856
Original: As an example, in Figure F.1, we show four uncurated samples from a prompt suggesting that the model write a poem, with a given title, in the style of Wallace Stevens.
中文: 例如,在图F.1中,我们展示了四个未经验证的样本,来自一个即时提示,模型用华莱士·史蒂文斯的风格写出一首有特定标题的诗.
<a id="S0857"></a> Source: p.48 S0857
Original: We first experimented with a few prompts, then generated four samples with no additional editing or selection (sampling at temperature 1 using nucleus sampling [HBFC19] with P = 0.9).
中文: 我们首先试验了几道导火索,然后生成了4个样本,没有额外的编辑或选择(在温度1下使用核取样[HBFC19]用P=0.9进行取样).
<a id="S0858"></a> Source: p.48 S0858
Original: Completions were truncated when the model began to write a new title and author heading, or broke into prose commentary. 48
中文: 当模型开始写出新的标题和作者标题时,或者突破作词评论时,完成被截断. 第48条
<a id="S0859"></a> Source: p.49 S0859
Original: CAVAFY TRANSLATED BY EDMUND KEELEY [Poem text omitted] SOME TREES John Ashbery [Poem text omitted] Shadows on the Way Wallace Stevens -------- Generated Poem 1 -------- -------- Generated Poem 3 -------- I must have shadows on the way The sun was all we had.
中文: CAVAFY 由 EDMUND KEELEY 翻译 [诗文从略] 一些TREES John Ashbery [诗文从略] 华莱士史蒂文斯路上的影子 -- 产生诗一 -- - - - - - - - - - ―― 生成"诗"3 -- -- 我一定有阴影在路上 太阳是我们所有的。
<a id="S0860"></a> Source: p.49 S0860
Original: Now, in the shade If I am to walk I must have All is changed.
中文: 现在,在阴影下 如果我要走路,我必须改变一切。
<a id="S0861"></a> Source: p.49 S0861
Original: The mind must dwell on those Each step taken slowly and alone White fields, that to its eyes were always old; To have it ready made Those ancient gleams, convoluting The way they lay among the huge roots, And I must think in lines of grey The great dark tomes of reverie, To have dim thoughts to be my guide The plumed labyrinth of the sea.
中文: 心灵必须停留在那些缓慢而单独迈出的每一步上 白地,在它的眼中总是老去的; 把它准备好 使那些古老的光芒, 曲折 它们在巨大的根中, 我必须用灰色的线来思考 伟大的黑暗之旅 暗淡的思想成为我的向导 出羽入海迷宫.
<a id="S0862"></a> Source: p.49 S0862
Original: Must look on blue and green And never let my eye forget When our own breath gonne faint and fade That color is my friend Up the long stairway of the dead And purple must surround me too We shall remember well The blindness of the light, the whiteness The yellow of the sun is no more Of that white land.
中文: 蓝色和绿色 永远不要让我的眼睛忘记 等我们自己的气息消退 这颜色是我的朋友 在死人的长楼梯上 紫色也必须包围我 我会记得的 光的盲从,白的 太阳所生之地之黄,已非所取之地也.
<a id="S0863"></a> Source: p.49 S0863
Original: The footsteps, and Intrusive than the bluish snow The grasses in that light, and in the shade That falls on all of us. I must have The menacing life.
中文: 脚步和侵扰 比出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自出自于出自出自出自出自出自出自出自出自出自出自出自于出自出自出自于出自出自出自出自出自出自于出自出自出自出自出自出自出自于出自出自出自出自出自出自出自 草地在那个光线下, 在阴影下 这件事发生在我们所有人身上。 我一定有危险的生活。
<a id="S0864"></a> Source: p.49 S0864
Original: Grey thoughts and blue thoughts walk with me If I am to go away at all.
中文: 灰暗的思绪和蓝色的思绪与我同行 如果我要离开的话
<a id="S0865"></a> Source: p.49 S0865
Original: We played, too, once, in the warmest rooms.
中文: 我们也在最温暖的房间玩过一次
<a id="S0866"></a> Source: p.49 S0866
Original: To one content, there is one content -------- Generated Poem 2 -------- For all mankind.
中文: 一个内容,有一个内容 - - - 出自"诗经"二 -- -- 为一切众生.
<a id="S0867"></a> Source: p.49 S0867
Original: The forest glades Are all the more for being in fable, There is a way in the world.
中文: 森林的草坪 更像是在寓言, 世界上有一个方法。
<a id="S0868"></a> Source: p.49 S0868
Original: And wood and town in legend shadowed- That way a man may live in a small town Once more to return.
中文: 还有传说中的木头和城镇 这样一个人就可以住在一个小镇了 再回来一次
<a id="S0869"></a> Source: p.49 S0869
Original: The one content His lifetime and see no man come, Has in it every happiness, more brave Never the stir of men about a market, For being purposeless; and so Never the bobbing of covered carts, Those island gardens, deep in the mind, Never the creak passed on the stair Make of remoteness familiarity Closing some other life; an old man And of the unspoken name a friend.
中文: 他的一生中,没有一个人来过, 在其中,有所有的幸福, 更勇敢的从男人的刺激 一个市场,因为是无目的的; 因此,从不波及被覆盖的车, 那些岛屿的花园, 深思熟虑, 从来没有断裂在楼梯上,
<a id="S0870"></a> Source: p.49 S0870
Original: Can prosper in that town, can prosper When his store is seen to close, his wife -------- Generated Poem 4 -------- Fades in his house, and the world shouts in vain That no one comes to hear him Nobody will come to this place.
中文: 在那个镇上可以繁荣 当他的店被看到关闭时,他的妻子 -- 生成诗4 -- 他家的花花公子, 和世界的呼喊是徒劳的 没有人会听到他, 没有人会来这个地方。
<a id="S0871"></a> Source: p.49 S0871
Original: In that small town But they are desolate, and they turn the traveler’s face Everything is known.
中文: 在那个小镇 但他们是荒凉的,他们把旅行者的脸转过来。
<a id="S0872"></a> Source: p.49 S0872
Original: All the sounds of the world are far away.
中文: 一切世界音声相去甚远.
<a id="S0873"></a> Source: p.49 S0873
Original: The ways of the young men are known, When the wind rises above the trees, And the ways of the young women The boughs bow to the ground.
中文: 年轻男人的道是已知的 当风起于树上 年轻女人的道 弓弓相从地.
<a id="S0874"></a> Source: p.49 S0874
Original: As they walk to the Spring; Even the birds that inhabit the tangle of weeds The rumors are all known, That is the roadside cover, are silent.
中文: 他们行到春天,连杂草缠绕的鸟儿, 谣言是众所周知的 这就是路边的掩护 安静
<a id="S0875"></a> Source: p.49 S0875
Original: One listens, The quarrels, the hurt, the envy But hears no roar of the forest.
中文: 一个听, 争吵 伤害 嫉妒 但听不到森林的咆哮
<a id="S0876"></a> Source: p.49 S0876
Original: That way out of the world There is no utterance, there is no conversation, That leads to a wood.
中文: 离开这个世界 没有话语,没有对话,这会导致一木头。
<a id="S0877"></a> Source: p.49 S0877
Original: Young, the young man There is a thin blue mist, Walks, a lover now, A darkness rising like smoke, Knows the streets as they are and is of them.
中文: 年轻人,年轻人 有一种薄蓝色的雾,行走, 情人现在, 黑暗像烟雾一样升起, 了解街道的现在和现在。
<a id="S0878"></a> Source: p.49 S0878
Original: And within that darkness A possession of the heart.
中文: 在黑暗中 拥有心灵
<a id="S0879"></a> Source: p.49 S0879
Original: It was here, and it will be here again- Here, under this sky empty and full of light.
中文: 它曾在这里, 它会再次在这里... 在这里,在这个天空下空出并充满了光.
<a id="S0880"></a> Source: p.49 S0880
Original: There, in the wood, among trees, He sees shadows on the way, hears voices, hears the wind and the rustling of leaves; Through an open glade He sees a shape and the shape hears: It waits as he waits, Waits as the shadows wait, As the voices wait; Shadows on the way, voices in the wind.
中文: 所出之地,出于木中,出于树中,所出之地,所出之道,所出之道,所出之声也;所出之地,所出之地也;所出之地,所出之地,所出之地也;所出之地,所出之地也;所出之地,所出之地也;所出之地,所出之地也;所出之地,所出之地也;所出之地也;所出之地,所出之地也;所出之地也. 通过开放的角斗场 他看见一个形状,形状就听到: 其相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相等相相相相相相相相相相相相相相相相相相相相相等相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相
<a id="S0881"></a> Source: p.49 S0881
Original: Figure F.1: Four uncurated completions from a context suggesting the model compose a poem in the style of Wallace Stevens with the title ‘Shadows on the Way’. 49
中文: 图F.1:从一个背景来看,有四个未经证实的完成,建议模型用华莱士·史蒂文斯的作风来作一首诗,标题是 " Shadows on the Ways " 。
<a id="S0882"></a> Source: p.50 S0882
Original: G Details of Task Phrasing and Specifications The following figures illustrate the formatting and phrasing of all the tasks included in the paper.
中文: G 任务定义和规格的细节 以下数字说明了文件中所有任务的格式和措辞。
<a id="S0883"></a> Source: p.50 S0883
Original: All data comes from the ground truth datasets in this section, and no samples from GPT-3 are included here.
中文: 所有数据均出自本节的地真相数据集,这里没有GPT-3的样本.
<a id="S0884"></a> Source: p.50 S0884
Original: Context → Article: Informal conversation is an important part of any business relationship.Before you start a discussion,however,make sure you understand which topics are suitable and which are considered taboo in a particular culture.
中文: · 条款:非正式对话是任何商业关系的重要组成部分。 然而,在开始讨论之前,要确定你了解哪些主题是合适的,哪些在某一文化中被认为是禁忌。
<a id="S0885"></a> Source: p.50 S0885
Original: Latin Americans enjoy sharing information about their local history, art and customs.You may expect questions about your family,and be sure to show pictures of your children.You may feel free to ask similar questions of your Latin American friends.The French think of conversation as an art form,and they enjoy the value of lively discussions as well as disagreements.
中文: 拉丁美洲人享受分享关于当地历史、艺术和习俗的信息。 你可能会期待关于你家人的问题, 并且一定要展示你孩子的照片。 你可以自由地问你的拉丁美洲朋友的类似问题。 法国人将对话视为一种艺术形式,他们享受活泼讨论和分歧的价值.
<a id="S0886"></a> Source: p.50 S0886
Original: For them,arguments can be interesting and they can cover pretty much or any topic ---- as long as they occur in are respectful and intelligent manner.
中文: 对他们来说,这些论点可能很有趣,它们可以涵盖很多或任何主题 -- 只要它们以尊重人和聪明的方式出现。
<a id="S0887"></a> Source: p.50 S0887
Original: In the United States,business people like to discuss a wide range of topics,including opinions about work,family,hobbies,and politics.
中文: 在美国,商界人士喜欢讨论广泛的话题,包括对工作,家庭,爱好和政治的看法.
<a id="S0888"></a> Source: p.50 S0888
Original: In Japan,China,and Korea,however,people are much more private.They do not share much about their thoughts,feelings,or emotions because they feel that doing so might take away from the harmonious business relationship they’re trying to build.Middle Easterners are also private about their personal lives and family matters.It is considered rude,for example,to ask a businessman from Saudi Arabia about his wife or children.
中文: 在日本、中国和韩国, 人们更私人。 他们对自己的思想、感情或情感并不认同, 因为他们觉得这样做会夺走他们试图构建的和谐的商业关系。 中东人对其个人生活和家庭事务也是隐私。 例如,询问一名沙特阿拉伯商人有关他妻子或子女的情况被认为是无礼的。
<a id="S0889"></a> Source: p.50 S0889
Original: As a general rule,it’s best not to talk about politics or religion with your business friends.This can get you into trouble,even in the United States,where people hold different religious views.In addition,discussing one’s salary is usually considered unsuitable.Sports is typically a friendly subject in most parts of the world,although be careful not to criticize national sport.Instead,be friendly and praise your host’s team. Q: What shouldn’t you do when talking about sports with colleagues from another country? A: Criticizing the sports of your colleagues’ country. Q: Which is typically a friendly topic in most places according to the author? A: Sports. Q: Why are people from Asia more private in their conversation with others? A: They don’t want to have their good relationship with others harmed by informal conversation. Q: The author considers politics and religion . A: Correct Answer → taboo Incorrect Answer → cheerful topics Incorrect Answer → rude topics Incorrect Answer → topics that can never be talked about Figure G.1: Formatted dataset example for RACE-h.
中文: 通常情况下,最好不要和你的商业朋友谈论政治或宗教。 这会给你带来麻烦, 即使在美国, 人们持有不同的宗教观点。 此外,讨论一个人的工资通常被认为不合适。 体育在世界大部分地区一般是友好的学科,虽然要注意不要批评民族体育. 并赞美东道主团队, 问:你和别国同事谈论体育时, 答:批评你同事国家的体育. 问:作者认为,在大多数地方,哪一个话题通常是友好的? A: Sports. Q:为什么亚洲人在与他人对话时更加私下?. 答:他们不希望与他人的良好关系受到非正式对话的损害。 问:作者认为政治和宗教 A:正确答案 – 禁忌 – 不正确回答 – 令人高兴的主题 – 不正确回答 – 粗鲁的主题 – 不正确回答 – 永远不能谈论到图G.1: RACE-h格式化数据集实例.
<a id="S0890"></a> Source: p.50 S0890
Original: When predicting, we normalize by the unconditional probability of each answer as described in 2. 50
中文: 在预测时,我们按2.50所述每个答案的无条件概率实现正常化。
<a id="S0891"></a> Source: p.51 S0891
Original: Context → anli 2: anli 2: The Gold Coast Hotel & Casino is a hotel and casino located in Paradise, Nevada.
中文: 背景- anli 2: anli 2: 黄金海岸酒店和赌场是一家位于内华达州天堂的酒店和赌场.
<a id="S0892"></a> Source: p.51 S0892
Original: This locals’ casino is owned and operated by Boyd Gaming.
中文: 当地赌场由Boyd Gaming拥有和经营。
<a id="S0893"></a> Source: p.51 S0893
Original: The Gold Coast is located one mile (∼ 1.6km) west of the Las Vegas Strip on West Flamingo Road.
中文: 黄金海岸地处拉斯维加斯地块以西一英里(∼ 1.6克米)的火地道西.
<a id="S0894"></a> Source: p.51 S0894
Original: It is located across the street from the Palms Casino Resort and the Rio All Suite Hotel and Casino.
中文: 它位于棕榈花赌场度假村和Rio All Suite酒店和赌场对面的街上.
<a id="S0895"></a> Source: p.51 S0895
Original: Question: The Gold Coast is a budget-friendly casino.
中文: 问:"黄金海岸"是一个预算友好的赌场.
<a id="S0896"></a> Source: p.51 S0896
Original: Correct Answer → Neither Incorrect Answer → True Incorrect Answer → False Figure G.2: Formatted dataset example for ANLI R2 Context → Article: Mrs.
中文: 正确回答 正确回答 正确回答 正确回答 错误回答 错误图 G.2: ANLI R2上下文格式化数据集示例 条款:夫人
<a id="S0897"></a> Source: p.51 S0897
Original: Once she told each student to bring along a few potatoes in plastic bag.
中文: 有一次,她告诉每个学生 带上一些土豆 在塑料袋。
<a id="S0898"></a> Source: p.51 S0898
Original: On each potato the students had to write a name of a person that they hated And the next day, every child brought some potatoes.
中文: 每只土豆上,学生都得写一个他们讨厌的人的名字 第二天,每个孩子都带了土豆
<a id="S0899"></a> Source: p.51 S0899
Original: Some had two potatoes;some three;some up to five.
中文: 有的吃两块土豆 有的吃三块 有的吃五块
<a id="S0900"></a> Source: p.51 S0900
Original: Smith then told the children to carry the bags everywhere they went, even to the toilet, for two weeks.
中文: 史密斯随后告诉孩子们随处携带行李,甚至去上厕所两周.
<a id="S0901"></a> Source: p.51 S0901
Original: As day after day passed, the children started to complain about the awful smell of the rotten potatoes.
中文: 日复一日,孩子们开始抱怨烂土豆的臭味.
<a id="S0902"></a> Source: p.51 S0902
Original: Those children who brought five potatoes began to feel the weight trouble of the bags.
中文: 那些带了5个马铃薯来的孩子开始感受到了袋子的重量问题.
<a id="S0903"></a> Source: p.51 S0903
Original: After two weeks, the children were happy to hear that the game was finally ended.
中文: 两周后,孩子们高兴地听到游戏最终结束.
<a id="S0904"></a> Source: p.51 S0904
Original: Smith asked,"How did you feel while carrying the potatoes for two weeks?" The children started complaining about the trouble loudly.
中文: 史密斯问道,“你带着马铃薯两周时感觉如何?” 孩子们开始大声地抱怨麻烦.
<a id="S0905"></a> Source: p.51 S0905
Original: Smith told them why she asked them to play the game.
中文: 史密斯告诉他们为什么她要他们玩游戏.
<a id="S0906"></a> Source: p.51 S0906
Original: She said,"This is exactly the situation when you carry your hatred for somebody inside your heart.
中文: 她说:"这就是你心中对某人的仇恨。
<a id="S0907"></a> Source: p.51 S0907
Original: The terrible smell of the hatred will pollute your heart and you will carry something unnecessary with you all the time.
中文: 仇恨的可怕气味会污染你的心 你总是带着一些不必要的东西
<a id="S0908"></a> Source: p.51 S0908
Original: If you cannot stand the smell of the rotten potatoes for just two weeks, can you imagine how heavy it would be to have the hatred in your heart for your lifetime?
中文: 如果你不能忍受烂土豆的气味 仅仅两个星期, 你能想象它会有多重 在你的心脏 恨你一生?
<a id="S0909"></a> Source: p.51 S0909
Original: So throw away any hatred from your heart, and you’ll be really happy." Q: Which of the following is True according to the passage? A: If a kid hated four people,he or she had to carry four potatoes. Q: We can learn from the passage that we should . A: throw away the hatred inside Q: The children complained about besides the weight trouble. A: the smell Q: Mrs.Smith asked her students to write on the potatoes. A: Correct Answer → names Incorrect Answer → numbers Incorrect Answer → time Incorrect Answer → places Figure G.3: Formatted dataset example for RACE-m.
中文: 将仇恨抛出心中, 问:根据段落,以下哪些是真实的? 答:如果一个孩子讨厌四个人,他或她必须携带四个土豆. 问:我们可以从我们应该学习的段落中吸取教训。 答:抛出Q内部的仇恨:除了重量问题, 甲:气味 Q:斯密斯夫人请她的学生在土豆上写作. A:正确答案 – 名称 不正确答案 – 数字 不正确答案 – 时间 不正确答案 – 将图G.3: RACE-m格式化数据集示例.
<a id="S0910"></a> Source: p.51 S0910
Original: When predicting, we normalize by the unconditional probability of each answer as described in 2. 51
中文: 在预测时,我们按2.51所述每个答案的无条件概率实现正常化。
<a id="S0911"></a> Source: p.52 S0911
Original: Correct Answer → Using a brush, brush on sealant onto wood until it is fully saturated with the sealant.
中文: 正确答案 – 使用刷子将密封剂刷入木上,直至与密封剂完全饱和.
<a id="S0912"></a> Source: p.52 S0912
Original: Incorrect Answer → Using a brush, drip on sealant onto wood until it is fully saturated with the sealant.
中文: 回答不正确 – 使用刷子将密封剂滴入木上,直至与密封剂完全饱和.
<a id="S0913"></a> Source: p.52 S0913
Original: Figure G.4: Formatted dataset example for PIQA Context → My body cast a shadow over the grass because Correct Answer → the sun was rising.
中文: 图G.4:PIQA背景格式化数据集示例 我的身体给草地投下阴影,因为正确答案 太阳正在升起.
<a id="S0914"></a> Source: p.52 S0914
Original: Figure G.5: Formatted dataset example for COPA Context → (CNN) Yuval Rabin, whose father, Yitzhak Rabin, was assassinated while serving as Prime Minister of Israel, criticized Donald Trump for appealing to "Second Amendment people" in a speech and warned that the words that politicians use can incite violence and undermine democracy. "Trump’s words are an incitement to the type of political violence that touched me personally," Rabin wrote in USAToday.
中文: 图G.5:COPA背景(CNN)尤瓦尔·拉宾(Yuval Rabin)的格式数据集示例,他的父亲伊扎克·拉宾(Yitzhak Rabin)在担任以色列总理时被刺杀,他批评唐纳德·特朗普在一次演讲中呼吁"第二修正案人",并警告说政客们使用的话可以煽动暴力和破坏民主. 拉宾在"今日美国报"上写道,
<a id="S0915"></a> Source: p.52 S0915
Original: He said that Trump’s appeal to "Second Amendment people" to stop Hillary Clinton -- comments that were criticized as a call for violence against Clinton, something Trump denied -- "were a new level of ugliness in an ugly campaign season." - The son of a former Israeli Prime Minister who was assassinated wrote an op ed about the consequence of violent political rhetoric. - Warns of "parallels" between Israel of the 1990s and the U.S. today.
中文: 他说,特朗普呼吁"第二修正案人"阻止希拉里·克林顿——这些评论被批评为呼吁对克林顿采取暴力,特朗普否认了这一点——"在丑陋的竞选季中,我们是一个丑陋的新层面". 一名被暗杀的前以色列总理之子写了一篇关于暴力政治言论后果的评论。 - 1990年代以色列和美国今天之间的“平行”警告。
<a id="S0916"></a> Source: p.52 S0916
Original: Correct Answer → - Referencing his father, who was shot and killed by an extremist amid political tension in Israel in 1995, Rabin condemned Donald Trump’s aggressive rhetoric.
中文: Rabin谴责唐纳德·特朗普的侵略言论。
<a id="S0917"></a> Source: p.52 S0917
Original: Correct Answer → - Referencing his father, who was shot and killed by an extremist amid political tension in Israel in 1995, Rabin condemned Trump’s aggressive rhetoric.
中文: Rabin谴责特朗普的侵略言论。
<a id="S0918"></a> Source: p.52 S0918
Original: Incorrect Answer → - Referencing his father, who was shot and killed by an extremist amid political tension in Israel in 1995, Rabin condemned Hillary Clinton’s aggressive rhetoric.
中文: Rabin谴责希拉里·克林顿的侵略言论。
<a id="S0919"></a> Source: p.52 S0919
Original: Incorrect Answer → - Referencing his father, who was shot and killed by an extremist amid political tension in Israel in 1995, Rabin condemned U.S.’s aggressive rhetoric.
中文: Rabin谴责美国侵略言论。
<a id="S0920"></a> Source: p.52 S0920
Original: Incorrect Answer → - Referencing his father, who was shot and killed by an extremist amid political tension in Israel in 1995, Rabin condemned Yitzhak Rabin’s aggressive rhetoric.
中文: Rabin谴责伊扎克·拉宾的侵略言论。
<a id="S0921"></a> Source: p.52 S0921
Original: Figure G.6: Formatted dataset example for ReCoRD.
中文: 图G.6:ReCORD格式化数据集示例.
<a id="S0922"></a> Source: p.52 S0922
Original: We consider the context above to be a single ”problem” because this is how the task is presented in the ReCoRD dataset and scored in the ReCoRD evaluation script.
中文: 我们认为,上述情况是一个单一的“问题”,因为这正是在ReCORD数据集中描述任务并在ReCORD评价脚本中打分的方式。
<a id="S0923"></a> Source: p.52 S0923
Original: Context → anli 1: anli 1: Fulton James MacGregor MSP is a Scottish politician who is a Scottish National Party (SNP) Member of Scottish Parliament for the constituency of Coatbridge and Chryston.
中文: 上下文 ^ anli 1: anli 1: Fulton James MacGregor MSP是苏格兰政治家,是苏格兰国民党(SNP)苏格兰议员,为科特布里奇和克赖斯顿选区议员.
<a id="S0924"></a> Source: p.52 S0924
Original: MacGregor is currently Parliamentary Liaison Officer to Shona Robison, Cabinet Secretary for Health & Sport.
中文: MacGregor目前是内阁卫生和体育部长Shona Robison的议会联络官。
<a id="S0925"></a> Source: p.52 S0925
Original: He also serves on the Justice and Education & Skills committees in the Scottish Parliament.
中文: 他还在苏格兰议会的司法和教育与技能委员会任职。
<a id="S0926"></a> Source: p.52 S0926
Original: Question: Fulton James MacGregor is a Scottish politican who is a Liaison officer to Shona Robison who he swears is his best friend.
中文: 问:富尔顿·詹姆斯·麦克格雷戈(Fulton James MacGregor)是苏格兰政治人物,是肖纳·罗比森的联络官,他发誓他是他最好的朋友.
<a id="S0927"></a> Source: p.52 S0927
Original: Correct Answer → Neither Incorrect Answer → True Incorrect Answer → False Figure G.7: Formatted dataset example for ANLI R1 52
中文: 正确答案 正确答案 正确答案 正确答案 错误图 G.7: ANLI R1 52格式数据集示例
<a id="S0928"></a> Source: p.53 S0928
Original: Context → Organisms require energy in order to do what?
中文: · 生物需要能量才能做什么?
<a id="S0929"></a> Source: p.53 S0929
Original: Figure G.8: Formatted dataset example for OpenBookQA.
中文: 图G.8:OpenBookQA格式化数据集示例.
<a id="S0930"></a> Source: p.53 S0930
Original: When predicting, we normalize by the unconditional probability of each answer as described in 2.
中文: 在预测时,我们通过2中描述的每个答案的无条件概率实现正常化.
<a id="S0931"></a> Source: p.53 S0931
Original: Context → Making a cake: Several cake pops are shown on a display. A woman and girl are shown making the cake pops in a kitchen.
中文: 上下文 ^ 做个蛋糕: 展出时会显示几个蛋糕流行. 一个女人和女孩被展示 在厨房里做蛋糕。
<a id="S0932"></a> Source: p.53 S0932
Original: They Correct Answer → bake them, then frost and decorate.
中文: 他们正确地回答 —— 烤了它们,然后霜和装饰。
<a id="S0933"></a> Source: p.53 S0933
Original: Incorrect Answer → taste them as they place them on plates.
中文: 答错 – 尝尝放在盘子上的味道.
<a id="S0934"></a> Source: p.53 S0934
Original: Incorrect Answer → put the frosting on the cake as they pan it.
中文: 错误的回答 – 把霜冻放在蛋糕上 当他们把它。
<a id="S0935"></a> Source: p.53 S0935
Original: Incorrect Answer → come out and begin decorating the cake as well.
中文: 不正确的回答 – 出来开始装饰蛋糕。
<a id="S0936"></a> Source: p.53 S0936
Original: Figure G.9: Formatted dataset example for HellaSwag Context → anli 3: anli 3: We shut the loophole which has American workers actually subsidizing the loss of their own job.
中文: 图G.9: HelaSwag Context – anli 3: anli 3: 我们堵住漏洞, 美国工人实际上补贴他们失去自己的工作。
<a id="S0937"></a> Source: p.53 S0937
Original: They just passed an expansion of that loophole in the last few days: $43 billion of giveaways, including favors to the oil and gas industry and the people importing ceiling fans from China.
中文: 最近几天,他们刚刚通过了这一漏洞的扩大:430亿美元赠与,包括给予石油和天然气工业以及从中国进口天花板粉丝的人优惠。
<a id="S0938"></a> Source: p.53 S0938
Original: Question: The loophole is now gone True, False, or Neither?
中文: 问题:漏洞已经不存在了 真实的,虚假的,还是没有?
<a id="S0939"></a> Source: p.53 S0939
Original: Correct Answer → False Incorrect Answer → True Incorrect Answer → Neither Figure G.10: Formatted dataset example for ANLI R3 Context → Question: George wants to warm his hands quickly by rubbing them.
中文: 正确回答 正确回答 错误回答 错误回答 错误回答 错误回答 错误回答 错误回答 图G.10: ANLI R3上下文格式化数据集示例 问题:乔治想用擦出手来迅速取暖.
<a id="S0940"></a> Source: p.53 S0940
Original: Which skin surface will produce the most heat?
中文: 哪个皮肤表面会产生最热?
<a id="S0941"></a> Source: p.53 S0941
Original: Answer: Correct Answer → dry palms Incorrect Answer → wet palms Incorrect Answer → palms covered with oil Incorrect Answer → palms covered with lotion Figure G.11: Formatted dataset example for ARC (Challenge).
中文: 答:正确回答 – 干棕榈不正确回答 – 湿棕榈不正确回答 – 被油覆盖的棕榈不正确回答 – 被乳液覆盖的棕榈 图G.11: ARC(Challenge)格式数据集实例.
<a id="S0942"></a> Source: p.53 S0942
Original: When predicting, we normalize by the unconditional probability of each answer as described in 2.
中文: 在预测时,我们通过2中描述的每个答案的无条件概率实现正常化.
<a id="S0943"></a> Source: p.53 S0943
Original: Context → lull is to trust as Correct Answer → cajole is to compliance Incorrect Answer → balk is to fortitude Incorrect Answer → betray is to loyalty Incorrect Answer → hinder is to destination Incorrect Answer → soothe is to passion Figure G.12: Formatted dataset example for SAT Analogies Correct Context → Grace was happy to trade me her sweater for my jacket.
中文: 反之亦然 反之亦然 反之亦然。
<a id="S0944"></a> Source: p.53 S0944
Original: She thinks the sweater Incorrect Context → Grace was happy to trade me her sweater for my jacket.
中文: 她觉得毛衣不正确的背景 格蕾丝很乐意用她的毛衣换我的外套
<a id="S0945"></a> Source: p.53 S0945
Original: She thinks the jacket Target Completion → looks dowdy on her.
中文: 她认为夹克目标完成 ~ 看起来像杜迪在她身上。
<a id="S0946"></a> Source: p.53 S0946
Original: Figure G.13: Formatted dataset example for Winograd.
中文: 图G.13:威诺格勒格式化数据集示例.
<a id="S0947"></a> Source: p.53 S0947
Original: The ‘partial’ evaluation method we use compares the probability of the completion given a correct and incorrect context. 53
中文: 我们使用的 " 部分 " 评估方法比较了在正确和不正确的情况下完成的可能性。 第53条
<a id="S0948"></a> Source: p.54 S0948
Original: Correct Context → Johnny likes fruits more than vegetables in his new keto diet because the fruits Incorrect Context → Johnny likes fruits more than vegetables in his new keto diet because the vegetables Target Completion → are saccharine.
中文: Johnny喜欢水果多于蔬菜, 因为果实不正确 Johnny喜欢水果多于蔬菜,
<a id="S0949"></a> Source: p.54 S0949
Original: Figure G.14: Formatted dataset example for Winogrande.
中文: 图G.14:Winogrande格式化数据集示例.
<a id="S0950"></a> Source: p.54 S0950
Original: The ‘partial’ evaluation method we use compares the probability of the completion given a correct and incorrect context.
中文: 我们使用的 " 部分 " 评估方法比较了在正确和不正确的情况下完成的可能性。
<a id="S0951"></a> Source: p.54 S0951
Original: Context → READING COMPREHENSION ANSWER KEY While this process moved along, diplomacy continued its rounds.
中文: 背景-阅读综合答复 在这一进程向前推进的同时,外交工作继续进行。
<a id="S0952"></a> Source: p.54 S0952
Original: Direct pressure on the Taliban had proved unsuccessful.
中文: 对塔利班的直接压力证明是失败的。
<a id="S0953"></a> Source: p.54 S0953
Original: As one NSC staff note put it, "Under the Taliban, Afghanistan is not so much a state sponsor of terrorism as it is a state sponsored by terrorists." In early 2000, the United States began a high-level effort to persuade Pakistan to use its influence over the Taliban.
中文: 正如一位国家安全委员会工作人员所指出,“在塔利班统治下,阿富汗与其说是恐怖主义的赞助国,不如说是恐怖分子赞助的国家。” 2000年初,美国开始高层努力来说服巴基斯坦利用其对塔利班的影响力.
<a id="S0954"></a> Source: p.54 S0954
Original: In January 2000, Assistant Secretary of State Karl Inderfurth and the State Department’s counterterrorism coordinator, Michael Sheehan, met with General Musharraf in Islamabad, dangling before him the possibility of a presidential visit in March as a reward for Pakistani cooperation.
中文: 2000年1月,助理国务卿卡尔·因德福斯和国务院反恐协调员迈克尔·谢汉在伊斯兰堡会见了穆沙拉夫将军,在他面前谈了3月份总统访问的可能性,以此奖励巴基斯坦的合作。
<a id="S0955"></a> Source: p.54 S0955
Original: Such a visit was coveted by Musharraf, partly as a sign of his government’s legitimacy.
中文: 穆沙拉夫渴望这次访问,部分是为了表明他的政府的合法性。
<a id="S0956"></a> Source: p.54 S0956
Original: He told the two envoys that he would meet with Mullah Omar and press him on Bin Laden.
中文: 他告诉两位特使,他将会见奥马尔毛拉并逼迫他对付本·拉登。
<a id="S0957"></a> Source: p.54 S0957
Original: They left, however, reporting to Washington that Pakistan was unlikely in fact to do anything," given what it sees as the benefits of Taliban control of Afghanistan." President Clinton was scheduled to travel to India.
中文: 然而,他们离开了华盛顿,向华盛顿报告巴基斯坦实际上不可能做任何事情,"鉴于巴基斯坦认为塔利班控制阿富汗的好处". 克林顿总统原定前往印度.
<a id="S0958"></a> Source: p.54 S0958
Original: The State Department felt that he should not visit India without also visiting Pakistan.
中文: 国务院认为,他不应该不访问巴基斯坦就访问印度。
<a id="S0959"></a> Source: p.54 S0959
Original: The Secret Service and the CIA, however, warned in the strongest terms that visiting Pakistan would risk the President’s life.
中文: 然而,特勤局和中央情报局最强烈地警告说,访问巴基斯坦将危及总统的生命。
<a id="S0960"></a> Source: p.54 S0960
Original: Counterterrorism officials also argued that Pakistan had not done enough to merit a presidential visit.
中文: 反恐官员还辩称,巴基斯坦没有做出足够的努力,值得总统访问。
<a id="S0961"></a> Source: p.54 S0961
Original: But President Clinton insisted on including Pakistan in the itinerary for his trip to South Asia.
中文: 但克林顿总统坚持将巴基斯坦纳入他去南亚的行程.
<a id="S0962"></a> Source: p.54 S0962
Original: His one-day stopover on March 25, 2000, was the first time a U.S. president had been there since 1969.
中文: 他于2000年3月25日停留了一天,这是1969年以来美国总统首次到场.
<a id="S0963"></a> Source: p.54 S0963
Original: At his meeting with Musharraf and others, President Clinton concentrated on tensions between Pakistan and India and the dangers of nuclear proliferation, but also discussed Bin Laden.
中文: 在同穆沙拉夫等会晤时,克林顿总统集中谈到巴基斯坦和印度之间的紧张关系和核扩散的危险,但也讨论了本·拉丹.
<a id="S0964"></a> Source: p.54 S0964
Original: President Clinton told us that when he pulled Musharraf aside for a brief, one-on-one meeting, he pleaded with the general for help regarding Bin Laden." I offered him the moon when I went to see him, in terms of better relations with the United States, if he’d help us get Bin Laden and deal with another issue or two." The U.S. effort continued.
中文: 克林顿总统告诉我们,当他把穆沙拉夫拉到一边进行简短的一对一会议时,他向将军恳求对本·拉登的帮助". 我给了他月亮, 我去看他, 以更好的关系与美国, 美国的努力仍在继续.
<a id="S0965"></a> Source: p.54 S0965
Original: Who did The State Department feel should visit both India and Pakistan?
中文: 国务院觉得谁应该访问印度和巴基斯坦?
<a id="S0966"></a> Source: p.54 S0966
Original: Correct Answer → - [False] Bin Laden Incorrect Answer → - [True] Bin Laden Figure G.15: Formatted dataset example for MultiRC.
中文: 正确答案- -- [虚假]本·拉登不正确答案- -- [真实]本·拉登 图G.15:多RC格式数据集示例.
<a id="S0967"></a> Source: p.54 S0967
Original: There are three levels within MultiRC: (1) the passage, (2) the questions, and (3) the answers.
中文: MultiRC内部有三个层次:(1)通过,(2)问题,(3)答案.
<a id="S0968"></a> Source: p.54 S0968
Original: During evaluation, accuracy is determined at the per-question level, with a question being considered correct if and only if all the answers within the question are labeled correctly.
中文: 在评价期间,准确性按每个问题级别确定,如果并且只有在问题中的所有答案都贴上正确标签时,就认为问题正确。
<a id="S0969"></a> Source: p.54 S0969
Original: For this reason, we use K to refer to the number of questions shown within the context.
中文: 为此,我们用K来指上下文中显示的问题数量.
<a id="S0970"></a> Source: p.54 S0970
Original: Context → Question: Which factor will most likely cause a person to develop a fever?
中文: 问题:哪个因素最可能导致一个人发烧?
<a id="S0971"></a> Source: p.54 S0971
Original: Answer: Correct Answer → a bacterial population in the bloodstream Incorrect Answer → a leg muscle relaxing after exercise Incorrect Answer → several viral particles on the skin Incorrect Answer → carbohydrates being digested in the stomach Figure G.16: Formatted dataset example for ARC (Easy).
中文: 回答:正确回答 血液中的细菌群 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错回答 答错答 答错答 答错答 答错答 答错答 答错答 答错答 答错答 答错答 答错答 答错 答错答 答错 答 答错答 答 答错答 答 答错答 答 答错答 答错答 答错 答 答错答 答错 答错 答 答 答错答错 答错 答错 答 答错答 答 答 答 答错答 答错 答 答错 答 答 答错 答 答 答错 答 答错 答错 答 答 答 答
<a id="S0972"></a> Source: p.54 S0972
Original: When predicting, we normalize by the unconditional probability of each answer as described in 2. 54
中文: 在预测时,我们按2.54所述每个答案的无条件概率实现正常化。
<a id="S0973"></a> Source: p.55 S0973
Original: Context → Bob went to the gas station to fill up his car.
中文: ——鲍勃去加油站装车.
<a id="S0974"></a> Source: p.55 S0974
Original: His tank was completely empty and so was his wallet.
中文: 他的坦克是完全空的 他的钱包也是
<a id="S0975"></a> Source: p.55 S0975
Original: The cashier offered to pay for his gas if he came back later to pay.
中文: 收银员提出,如果他晚点回来还钱,就要支付他的汽油.
<a id="S0976"></a> Source: p.55 S0976
Original: Correct Answer → Bob believed that there were good people in the world.
中文: 正确回答 —— 鲍勃认为世界上有好人.
<a id="S0977"></a> Source: p.55 S0977
Original: Incorrect Answer → Bob contemplated how unfriendly the world was.
中文: 错误的回答 — Bob认为这个世界是多么不友好。
<a id="S0978"></a> Source: p.55 S0978
Original: Figure G.17: Formatted dataset example for StoryCloze Context → Helsinki is the capital and largest city of Finland.
中文: 图G.17: StoryCloze Context 赫尔辛基是芬兰的首都和最大城市。
<a id="S0979"></a> Source: p.55 S0979
Original: It is in the region of Uusimaa, in southern Finland, on the shore of the Gulf of Finland.
中文: 它位于芬兰南部的纽西马地区,芬兰湾沿岸.
<a id="S0980"></a> Source: p.55 S0980
Original: Helsinki has a population of , an urban population of , and a metropolitan population of over 1.4 million, making it the most populous municipality and urban area in Finland.
中文: 赫尔辛基的人口为.,城市人口为.,首都人口超过140万,成为芬兰人口最多的市镇和城市地区.
<a id="S0981"></a> Source: p.55 S0981
Original: Helsinki is some north of Tallinn, Estonia, east of Stockholm, Sweden, and west of Saint Petersburg, Russia.
中文: 赫尔辛基是爱沙尼亚塔林以北,瑞典斯德哥尔摩以东,俄罗斯圣彼得堡以西等地的地段.
<a id="S0982"></a> Source: p.55 S0982
Original: Helsinki has close historical connections with these three cities.
中文: 赫尔辛基与这三个城市有着密切的历史联系.
<a id="S0983"></a> Source: p.55 S0983
Original: The Helsinki metropolitan area includes the urban core of Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns.
中文: 赫尔辛基都会区包括赫尔辛基,埃斯波,万塔,考尼艾宁等城市核心以及周边通勤城镇.
<a id="S0984"></a> Source: p.55 S0984
Original: It is the world’s northernmost metro area of over one million people, and the city is the northernmost capital of an EU member state.
中文: 这是世界上人口超过一百万的地铁地区,
<a id="S0985"></a> Source: p.55 S0985
Original: The Helsinki metropolitan area is the third largest metropolitan area in the Nordic countries after Stockholm and Copenhagen, and the City of Helsinki is the third largest after Stockholm and Oslo.
中文: 赫尔辛基都会区是北欧国家仅次于斯德哥尔摩和哥本哈根的第三大都会区,而赫尔辛基市是仅次于斯德哥尔摩和奥斯陆的第三大都会区.
<a id="S0986"></a> Source: p.55 S0986
Original: Helsinki is Finland’s major political, educational, financial, cultural, and research center as well as one of northern Europe’s major cities.
中文: 赫尔辛基是芬兰的主要政治、教育、金融、文化和研究中心,也是北欧的主要城市之一。赫尔辛基是芬兰的主要城市。
<a id="S0987"></a> Source: p.55 S0987
Original: Approximately 75% of foreign companies that operate in Finland have settled in the Helsinki region.
中文: 在芬兰营业的外国公司约有75%在赫尔辛基地区定居。
<a id="S0988"></a> Source: p.55 S0988
Original: The nearby municipality of Vantaa is the location of Helsinki Airport, with frequent service to various destinations in Europe and Asia. Q: what is the most populous municipality in Finland? A: Helsinki Q: how many people live there? A: 1.4 million in the metropolitan area Q: what percent of the foreign companies that operate in Finland are in Helsinki? A: 75% Q: what towns are a part of the metropolitan area? A: Target Completion → Helsinki, Espoo, Vantaa, Kauniainen, and surrounding commuter towns Figure G.18: Formatted dataset example for CoQA Context → Please unscramble the letters into a word, and write that word: asinoc = Target Completion → casino Figure G.19: Formatted dataset example for Cycled Letters 55
中文: 相邻的万塔市政府是赫尔辛基机场的所在地,经常服务于欧洲和亚洲的各个目的地. 问:芬兰人口最多的城市是什么? 答:赫尔辛基Q:有多少人住在那里? 答:首都地区140万 Q:在芬兰经营的外国公司在赫尔辛基占多大比例? 答:75% Q:大都市区的一部分是什么城镇? A:目标完成 赫尔辛基、埃斯波、万塔、考尼艾宁和周边通勤城镇 图G.18:CoQA背景格式数据集示例 – 请将字母解为单词并写出该词:asinoc =目标完成 – 赌场 图G.19:循环字母55的格式数据集示例
<a id="S0989"></a> Source: p.56 S0989
Original: Context → Passage: Saint Jean de Br´ebeuf was a French Jesuit missionary who travelled to New France in 1625.
中文: 上下文 ^ 通过:圣让·德·布尔·埃贝夫是一位法国耶稣会传教士,于1625年去新法兰西.
<a id="S0990"></a> Source: p.56 S0990
Original: There he worked primarily with the Huron for the rest of his life, except for a few years in France from 1629 to 1633.
中文: 在那里他主要与休伦人共事了余生,除了1629年至1633年在法国的几年外.
<a id="S0991"></a> Source: p.56 S0991
Original: He learned their language and culture, writing extensively about each to aid other missionaries.
中文: 他学习了他们的语言与文化,广泛写下他们各自的情况来帮助其他传教士.
<a id="S0992"></a> Source: p.56 S0992
Original: In 1649, Br´ebeuf and another missionary were captured when an Iroquois raid took over a Huron village .
中文: 1649年,Br'ebeuf和另一位传教士被俘虏,当时一次易洛魁突袭占领了休伦村.
<a id="S0993"></a> Source: p.56 S0993
Original: Together with Huron captives, the missionaries were ritually tortured and killed on March 16, 1649.
中文: 与休伦被俘者一起被传教士们在1649年3月16日被仪式上折磨并杀害.
<a id="S0994"></a> Source: p.56 S0994
Original: Br´ebeuf was beatified in 1925 and among eight Jesuit missionaries canonized as saints in the Roman Catholic Church in 1930.
中文: Br'ebeuf于1925年被击败,在耶稣会的8名传教士中,1930年在罗马天主教会被划为圣人.
<a id="S0995"></a> Source: p.56 S0995
Original: Question: How many years did Saint Jean de Br´ebeuf stay in New France before he went back to France for a few years?
中文: 问:圣让·德·布尔·埃比乌夫在回法国几年前在新法兰西逗留了几年?
<a id="S0996"></a> Source: p.56 S0996
Original: Answer: Target Completion → 4 Figure G.20: Formatted dataset example for DROP Context → Fill in blank: She held the torch in front of her.
中文: 回答:目标完成 ^ 4 图G.20: DROP上下文格式化数据集示例 ^ 填入空白:她把火把放在她面前.
<a id="S0997"></a> Source: p.56 S0997
Original: About fifty feet ahead." She moved faster.
中文: 大约50英尺前。" 她移动得更快。
<a id="S0998"></a> Source: p.56 S0998
Original: They both moved faster. "In fact," she said, raising the torch higher, "there’s more than a . -> Target Completion → step Figure G.21: Formatted dataset example for LAMBADA Context → Please unscramble the letters into a word, and write that word: skicts = Target Completion → sticks Figure G.22: Formatted dataset example for Anagrams 1 (A1) Context → Please unscramble the letters into a word, and write that word: volwskagen = Target Completion → volkswagen Figure G.23: Formatted dataset example for Anagrams 2 Context → Q: Who played tess on touched by an angel? A: Target Completion → Delloreese Patricia Early (July 6, 1931 { November 19, 2017), known professionally as Della Reese Figure G.24: Formatted dataset example for Natural Questions 56
中文: 他们都移动得更快。 "事实上,"她说,"举起更高的火炬,"不止一个. -" 目标完成-步骤 图G.21: LAMBADA背景格式化数据集示例 – 请将字母解为单词,并写出该单词: sickcts = 目标完成-棒 图G.22: Anagrams 1 (A1)背景格式化数据集示例 – 请将字母解为单词并写出: volwskagen = 目标完成- Volkswagen 图G.23: Anagrams 2 背景-Q: Anagrams 的背景-Q:谁玩被天使所触摸到的tes? A:目标完成- Delloreese Patricia Early (1931年7月6日 { 2017年11月19日 ),专业名称为Della Reese 图G.24:自然问题格式数据集示例 56
<a id="S0999"></a> Source: p.57 S0999
Original: Context → TITLE: William Perry (American football) - Professional career PARAGRAPH: In 1985, he was selected in the first round of the 1985 NFL Draft by the Chicago Bears; he had been hand-picked by coach Mike Ditka.
中文: 背景-TITLE:威廉·佩里(英语:William Perry (American football)) - 职业生涯 第1段:1985年被芝加哥熊队在1985年NFL草稿首轮选中;由教练迈克·迪特卡亲手挑选出.
<a id="S1000"></a> Source: p.57 S1000
Original: However, defensive coordinator Buddy Ryan, who had a highly acrimonious relationship with Ditka, called Perry a "wasted draft-pick".
中文: 然而,与迪特卡有着高度激烈关系的防守协调员巴迪·瑞安(Buddy Ryan)称佩里为"被浪费的草稿挑".
<a id="S1001"></a> Source: p.57 S1001
Original: Perry soon became a pawn in the political power struggle between Ditka and Ryan.
中文: 佩里很快成为了迪特卡和瑞安之间政治权力斗争的棋子.
<a id="S1002"></a> Source: p.57 S1002
Original: Perry’s "Refrigerator" nickname followed him into the NFL and he quickly became a favorite of the Chicago Bears fans.
中文: 派瑞的"Refrigerator"外号跟随他入选了NFL,他很快成为了芝加哥熊队球迷中最喜爱的.
<a id="S1003"></a> Source: p.57 S1003
Original: Teammates called him "Biscuit," as in "one biscuit shy of 350 pounds." While Ryan refused to play Perry, Ditka decided to use Perry as a fullback when the team was near the opponents’ goal line or in fourth and short situations, either as a ball carrier or a lead blocker for star running back Walter Payton.
中文: 队友称他为"饼干",如"一张350磅的饼干". 虽然瑞恩拒绝出场佩里,但迪特卡决定当球队靠近对手入球线时或者在第四和短暂的情况下使用佩里作为全后卫,要么作为球载体,要么作为明星跑回沃尔特·佩顿(Walter Payton)的主力挡箭手.
<a id="S1004"></a> Source: p.57 S1004
Original: Ditka stated the inspiration for using Perry as a fullback came to him during five-yard sprint exercises.
中文: 迪特卡表示,将佩里作为全后卫的灵感在五码跑道练习中降临到他身上.
<a id="S1005"></a> Source: p.57 S1005
Original: During his rookie season, Perry rushed for two touchdowns and caught a pass for one.
中文: 在他的新手赛季中,佩里匆忙地进行了两次触地得分,并拿了一次的传球.
<a id="S1006"></a> Source: p.57 S1006
Original: Perry even had the opportunity to run the ball during Super Bowl XX, as a nod to his popularity and contributions to the team’s success.
中文: 佩里甚至有机会在"超级杯XX"期间出场出场出场出场出场出场,以点头表示他的受人欢迎程度以及对球队成功的贡献.
<a id="S1007"></a> Source: p.57 S1007
Original: The first time he got the ball, he was tackled for a one-yard loss while attempting to throw his first NFL pass on a halfback option play.
中文: 第一次拿到球时,他在试图投出首个NFL传球时被以一码失利来对付.
<a id="S1008"></a> Source: p.57 S1008
Original: The second time he got the ball, he scored a touchdown (running over Patriots linebacker Larry McGrew in the process).
中文: 第二次得到球后,他打入了达阵(跑过爱国者后卫拉里·麦克格鲁(英语:Larry McGrew in the process)).
<a id="S1009"></a> Source: p.57 S1009
Original: About halfway through his rookie season, Ryan finally began to play Perry, who soon proved that he was a capable defensive lineman.
中文: 大约在他的新手赛季中途,瑞恩终于开始出演佩里,佩里很快证明了自己是一个能干的防守线人.
<a id="S1010"></a> Source: p.57 S1010
Original: His Super Bowl ring size is the largest of any professional football player in the history of the event.
中文: 他的"超级碗"戒指尺寸是赛事史上任何职业足球运动员中最大的.
<a id="S1011"></a> Source: p.57 S1011
Original: His ring size is 25, while the ring size for the average adult male is between 10 and 12.
中文: 他的戒指尺寸为25个,而一般成年雄性的戒指尺寸为10到12个.
<a id="S1012"></a> Source: p.57 S1012
Original: Perry went on to play for ten years in the NFL, retiring after the 1994 season.
中文: 佩里在NFL继续出场十年,1994赛季后退役.
<a id="S1013"></a> Source: p.57 S1013
Original: In his ten years as a pro, he regularly struggled with his weight, which hampered his performance at times.
中文: 在他作为一名职业选手的十年中,他经常与自己的体重相搏,这有时阻碍了他的表现.
<a id="S1014"></a> Source: p.57 S1014
Original: He played in 138 games, recording 29.5 sacks and five fumble recoveries, which he returned for a total of 71 yards.
中文: 他在138场比赛中出场,收录了29.5个麻袋和5个倒数收场,他总共返回了71码.
<a id="S1015"></a> Source: p.57 S1015
Original: In his offensive career he ran five yards for two touchdowns, and had one reception for another touchdown.
中文: 在他的进攻生涯中,他跑出5码跑出2个触地得分,并有一个接见另一个触地得分.
<a id="S1016"></a> Source: p.57 S1016
Original: Perry later attempted a comeback, playing an unremarkable 1996 season with the London Monarchs of the World League of American Football (later NFL Europa). Q: what team did he play for? A: Target Completion → the Chicago Bears Figure G.25: Formatted dataset example for QuAC Context → Please unscramble the letters into a word, and write that word: r e!c.i p r o.c a/l = Target Completion → reciprocal Figure G.26: Formatted dataset example for Symbol Insertion Context → Please unscramble the letters into a word, and write that word: taefed = Target Completion → defeat Figure G.27: Formatted dataset example for Reversed Words 57
中文: 佩里后来尝试复出,与世界美国足球联盟(后为NFL Europa)的伦敦君主队一起出战了1996赛季无名小卒. 问:他为哪支球队出场? A:目标完成-芝加哥熊图 G.25: QuAC背景格式化数据集示例 ^ 请将字母解为单词并写出: r e!c.i p o.c a/l =目标完成- 对等 图 G.26: 符号插入背景格式化数据集示例 → 请将字母解为单词并写出: taefed = 目标完成- 失败 图 G.27: 倒转单词格式化数据集示例 57
<a id="S1017"></a> Source: p.58 S1017
Original: Context → Title: The Blitz Background: From the German point of view, March 1941 saw an improvement.
中文: 上下文-标题:"闪电"背景:从德文观点来看,1941年3月出现了改进.
<a id="S1018"></a> Source: p.58 S1018
Original: The Luftwaffe flew 4,000 sorties that month, including 12 major and three heavy attacks.
中文: 吕夫特瓦夫号当月出动了4000架次出动,包括12起大出击和3起猛烈出击.
<a id="S1019"></a> Source: p.58 S1019
Original: The electronic war intensified but the Luftwaffe flew major inland missions only on moonlit nights.
中文: 电子战争愈演愈烈 但德国空军只在月光之夜 执行大型内陆任务
<a id="S1020"></a> Source: p.58 S1020
Original: Ports were easier to find and made better targets.
中文: 港口更容易找到,并成为更好的目标。
<a id="S1021"></a> Source: p.58 S1021
Original: To confuse the British, radio silence was observed until the bombs fell. X- and Y-Ger¨at beams were placed over false targets and switched only at the last minute.
中文: 为了迷惑英国人,一直观察到无线电沉默,直到炸弹倒下. X-和Y-Ger"at"光束被放置在假目标上,仅在最后一刻才切换.
<a id="S1022"></a> Source: p.58 S1022
Original: Rapid frequency changes were introduced for X-Ger¨at, whose wider band of frequencies and greater tactical flexibility ensured it remained effective at a time when British selective jamming was degrading the effectiveness of Y-Ger¨at. Q: How many sorties were flown in March 1941? A: 4,000 Q: When did the Luftwaffe fly inland missions? A: Target Completion → only on moonlit nights Figure G.28: Formatted dataset example for SQuADv2 Context → Normal force -- In a simple case such as an object resting upon a table, the normal force on the object is equal but in opposite direction to the gravitational force applied on the object (or the weight of the object), that is, N = m g (\displaystyle N=mg), where m is mass, and g is the gravitational field strength (about 9.81 m/s on Earth).
中文: X-Ger'at的频率发生了快速变化,在英国选择性干扰正在降低Y-Ger'at的效力时,X-Ger'at的频率范围更广,战术灵活性也更大,确保了其继续有效。 问:1941年3月出动了多少架次? 答:4000 Q:列夫特瓦夫号何时在内陆飞行?. A:目标完成 只在月光之夜完成 图G.28:SQuADv2背景格式数据集实例 正态力- 在一简单的情况中,比如一个物体停留在桌子上,物体上正常的力是等同的,但方向与对物体所施加的重力(或物体的重量)相反,即:N=m克(\displaystyle N=mg),其中m为质量,g为重力场强度(约9.81米/秒于地上).
<a id="S1023"></a> Source: p.58 S1023
Original: The normal force here represents the force applied by the table against the object that prevents it from sinking through the table and requires that the table is sturdy enough to deliver this normal force without breaking.
中文: 这里的正常力代表了平面对物体所施加的能防止它从平面上下沉的力并需要平面平面的平面足够坚固,以不破裂地传递出这种平面力.
<a id="S1024"></a> Source: p.58 S1024
Original: However, it is easy to assume that the normal force and weight are action-reaction force pairs (a common mistake).
中文: 然而,很容易假设正常的力量和重量是动作-反应力对(一个常见的错误).
<a id="S1025"></a> Source: p.58 S1025
Original: In this case, the normal force and weight need to be equal in magnitude to explain why there is no upward acceleration of the object.
中文: 在这种情况下,正常的力和重量需要等量来解释为何物体没有向上加速.
<a id="S1026"></a> Source: p.58 S1026
Original: For example, a ball that bounces upwards accelerates upwards because the normal force acting on the ball is larger in magnitude than the weight of the ball. question: is the normal force equal to the force of gravity? answer: Target Completion → yes Figure G.29: Formatted dataset example for BoolQ Context → The trend toward lower rents may seem surprising given that some communities in New York are bemoaning the loss of favorite local businesses to high rents.
中文: 例如,一个向上反弹的球会加速向上,因为正常作用于球上的力量在数量上大于球的重量. 问题:正常力是否等于重力? 图G.29:BoulQ上下文格式化数据集示例 租金下降的趋势似乎令人吃惊,因为纽约的一些社区正在抱怨当地最受欢迎的企业因高租金而损失。
<a id="S1027"></a> Source: p.58 S1027
Original: But, despite the recent softening, for many of these retailers there’s still been too big a jump from the rental rates of the late 1970s, when their leases were signed.
中文: 但是,尽管最近出现了软化,但对于许多这些零售商来说,从20世纪70年代后期签订租约时的租金率来看,仍然有太大的突破。
<a id="S1028"></a> Source: p.58 S1028
Original: Certainly, the recent drop in prices doesn’t mean Manhattan comes cheap. question: Manhattan comes cheap. true, false, or neither? answer: Target Completion → false Figure G.30: Formatted dataset example for CB 58
中文: 当然,最近的物价下跌并不意味着曼哈顿是便宜的。 问曼哈顿来便宜。 是真的,假的,还是没有? 答复:目标完成-虚假图G.30:CB 58格式数据集实例
<a id="S1029"></a> Source: p.59 S1029
Original: Context → The bet, which won him dinner for four, was regarding the existence and mass of the top quark, an elementary particle discovered in 1995. question: The Top Quark is the last of six flavors of quarks predicted by the standard model theory of particle physics.
中文: 内 容 提 要 这场赌博为他赢得了四分之一的晚餐, 是关于高夸克的存在和质量的, 1995年发现的一粒基本粒子。 问:"顶夸克"是粒子物理学标准模型理论所预测的六种夸克口味中的最后一种.
<a id="S1030"></a> Source: p.59 S1030
Original: True or False? answer: Target Completion → False Figure G.31: Formatted dataset example for RTE Context → An outfitter provided everything needed for the safari.
中文: 真还是假? 答复:目标完成 + 假图G.31:RTE上下文格式化数据集示例 → 一个装扮师提供了所有 需要的猎物。
<a id="S1031"></a> Source: p.59 S1031
Original: Before his first walking holiday, he went to a specialist outfitter to buy some boots. question: Is the word ‘outfitter’ used in the same way in the two sentences above? answer: Target Completion → no Figure G.32: Formatted dataset example for WiC Context → Final Exam with Answer Key Instructions: Please carefully read the following passages.
中文: 在他第一次出行的假期前,他到一个专业的服装师那里去买靴子. 问题:上文两句中是否同样使用 " 外向 " 一词? 回答:目标完成 ^ 没有图G.32: WIC上下文格式化数据集示例 ^ 有回答密钥指示的最后考试:请仔细阅读以下段落.
<a id="S1032"></a> Source: p.59 S1032
Original: For each passage, you must identify which noun the pronoun marked in bold refers to. ===== Passage: Mr.
中文: 对于每个段落,您必须识别在粗体中标记的名词指的是哪个名词. - 通过:
<a id="S1033"></a> Source: p.59 S1033
Original: Moncrieff visited Chester’s luxurious New York apartment, thinking that it belonged to his son Edward.
中文: 蒙克里夫拜访了切斯特豪华的纽约公寓,认为公寓属于他的儿子爱德华.
<a id="S1034"></a> Source: p.59 S1034
Original: Moncrieff has decided to cancel Edward’s allowance on the ground that he no longer requires his financial support.
中文: Moncrieff决定取消Edward的津贴,
<a id="S1035"></a> Source: p.59 S1035
Original: Question: In the passage above, what does the pronoun "his" refer to?
中文: 问:在上段,"他"的代词指的是什么?
<a id="S1036"></a> Source: p.59 S1036
Original: Answer: Target Completion → mr. moncrieff Figure G.33: Formatted dataset example for WSC Context → Q: ‘Nude Descending A Staircase’ is perhaps the most famous painting by which 20th century artist? A: Target Completion → MARCEL DUCHAMP Target Completion → r mutt Target Completion → duchamp Target Completion → marcel duchamp Target Completion → R.Mutt Target Completion → Marcel duChamp Target Completion → Henri-Robert-Marcel Duchamp Target Completion → Marcel du Champ Target Completion → henri robert marcel duchamp Target Completion → Duchampian Target Completion → Duchamp Target Completion → duchampian Target Completion → marcel du champ Target Completion → Marcel Duchamp Target Completion → MARCEL DUCHAMP Figure G.34: Formatted dataset example for TriviaQA.
中文: 回答:目标完成 – Moncrieff先生 图G.33:WSC背景格式化数据集实例 – Q: " Nude Descrewing A Staircase " 可能是20世纪艺术家最出名的画作? A:目标完成-MARCEL DUCHAMP目标完成-mutt目标完成-duchamp目标完成-marcel duchamp目标完成-R.Mutt目标完成-Marcel duChamp目标完成-Henri-Robert-Marcel Duchamp目标完成-Marcel du Champ目标完成-Henri Robert marcuchamp目标完成-Duchamp目标完成-Duchamp目标完成-duchamp目标完成-duchamp目标完成-Marcel Duchamp目标完成-MARCEL DUCHAMP 图G.34:TriviaQA的标准化数据集。
<a id="S1037"></a> Source: p.59 S1037
Original: TriviaQA allows for multiple valid completions. 59
中文: TriviaQA允许多个有效完成. 59国
<a id="S1038"></a> Source: p.60 S1038
Original: Context → Q: What school did burne hogarth establish? A: Target Completion → School of Visual Arts Figure G.35: Formatted dataset example for WebQA Context → Keinesfalls d¨urfen diese f¨ur den kommerziellen Gebrauch verwendet werden. = Target Completion → In no case may they be used for commercial purposes.
中文: 背景-Q:birde hogarth建立了什么学校? A:目标完成 视觉艺术学院 图G.35:WebQA背景格式数据集实例 Keinesfalls d'urfen diese f'ur den kommerziellen Gebrauch verwendet werden. =目标完成 = 在任何情况下都不得用于商业目的。
<a id="S1039"></a> Source: p.60 S1039
Original: Figure G.36: Formatted dataset example for De→En.
中文: 图G.36:De-En格式化数据集示例.
<a id="S1040"></a> Source: p.60 S1040
Original: This is the format for one- and few-shot learning, for this and other langauge tasks, the format for zero-shot learning is “Q: What is the {language} translation of {sentence} A: {translation}.” Context → In no case may they be used for commercial purposes. = Target Completion → Keinesfalls d¨urfen diese f¨ur den kommerziellen Gebrauch verwendet werden.
中文: 这是一发和几发学习的格式,对于这个和其他放电任务,零发学习的格式是“Q: {语言}{判决}的翻译是什么? A:{翻译}。” 内 容 提 要 在任何情况下都不得用于商业目的。 完成目标 完成目标
<a id="S1041"></a> Source: p.60 S1041
Original: Figure G.37: Formatted dataset example for En→De Context → Analysis of instar distributions of larval I. verticalis collected from a series of ponds also indicated that males were in more advanced instars than females. = Target Completion → L’analyse de la distribution de fr´equence des stades larvaires d’I. verticalis dans une s´erie d’´etangs a ´egalement d´emontr´e que les larves m^ales ´etaient `a des stades plus avanc´es que les larves femelles.
中文: 图G.37:En-De Context的已格式数据集实例 – 对从一系列池塘采集到的幼体一.垂直的恒星分布的分析也表明,雄性恒星比雌性更高级. =目标完成======================================================================================================================================================================================================================================================== (== (
<a id="S1042"></a> Source: p.60 S1042
Original: Figure G.38: Formatted dataset example for En→Fr Context → L’analyse de la distribution de fr´equence des stades larvaires d’I. verticalis dans une s´erie d’´etangs a ´egalement d´emontr´e que les larves m^ales ´etaient `a des stades plus avanc´es que les larves femelles. = Target Completion → Analysis of instar distributions of larval I. verticalis collected from a series of ponds also indicated that males were in more advanced instars than females.
中文: 图G.38:En Fr Context L 'analyse de la distribution de fr'equence des stades lavaires d ' I. vertalis dans une s'erie d'etangs a'egentr d'e que les larves m^ales'etient'a des stadies plus avanc'es que les larves femelles的数据集格式示例。 =目标完成 = 对从一系列池塘采集的幼体一.垂直的恒星分布的分析也表明,雄性恒星比雌性更高级.
<a id="S1043"></a> Source: p.60 S1043
Original: Figure G.39: Formatted dataset example for Fr→En Context → The truth is that you want, at any price, and against the wishes of the peoples of Europe, to continue the negotiations for Turkey’s accession to the European Union, despite Turkey’s continuing refusal to recognise Cyprus and despite the fact that the democratic reforms are at a standstill. = Target Completion → Adev˘arul este c˘a v˘a dorit¸i, cu orice pret¸ ¸si ^ımpotriva dorint¸ei europenilor, s˘a continuat¸i negocierile de aderare a Turciei la Uniunea European˘a, ^ın ciuda refuzului continuu al Turciei de a recunoa¸ste Ciprul ¸si ^ın ciuda faptului c˘a reformele democratice au ajuns ^ıntr-un punct mort.
中文: 图G.39:Fr-En上下文格式数据集示例 事实是,尽管土耳其继续拒绝承认塞浦路斯,尽管民主改革处于停顿状态,但无论怎样,你都想违背欧洲人民的意愿,继续谈判土耳其加入欧盟。 ===============================================================================================================================================================================================================================================================
<a id="S1044"></a> Source: p.60 S1044
Original: Figure G.40: Formatted dataset example for En→Ro 60
中文: 图G.40:En RO 60格式数据集实例
<a id="S1045"></a> Source: p.61 S1045
Original: Context → Adev˘arul este c˘a v˘a dorit¸i, cu orice pret¸ ¸si ^ımpotriva dorint¸ei europenilor, s˘a continuat¸i negocierile de aderare a Turciei la Uniunea European˘a, ^ın ciuda refuzului continuu al Turciei de a recunoa¸ste Ciprul ¸si ^ın ciuda faptului c˘a reformele democratice au ajuns ^ıntr-un punct mort. = Target Completion → The truth is that you want, at any price, and against the wishes of the peoples of Europe, to continue the negotiations for Turkey’s accession to the European Union, despite Turkey’s continuing refusal to recognise Cyprus and despite the fact that the democratic reforms are at a standstill.
中文: 上下文 – Adev ̆arul este c̆a v̆a dorit ̧i; Cu orice pret ̧si ̧mpotriva dorint ̧ei Europenilor; s̆a continuat ̧i negocierile de aderare a Turciei la Unionunea ̆a; ^ın Ciuda reduuzuli continu al Turciei de a reunoa ̧ste Ciprul ̧si ̧ cia faptului ̆a refore e au ajuns-ntr-un punt mort. =目标完成 = 事实是,尽管土耳其继续拒绝承认塞浦路斯,尽管民主改革处于停顿状态,但无论怎样,你都想违背欧洲人民的意愿,继续谈判土耳其加入欧盟。
<a id="S1046"></a> Source: p.61 S1046
Original: Figure G.41: Formatted dataset example for Ro→En Context → Q: What is (2 4) 6? A: Target Completion → 48 Figure G.42: Formatted dataset example for Arithmetic 1DC Context → Q: What is 17 minus 14? A: Target Completion → 3 Figure G.43: Formatted dataset example for Arithmetic 2D- Context → Q: What is 98 plus 45? A: Target Completion → 143 Figure G.44: Formatted dataset example for Arithmetic 2D+ Context → Q: What is 95 times 45? A: Target Completion → 4275 Figure G.45: Formatted dataset example for Arithmetic 2Dx Context → Q: What is 509 minus 488? A: Target Completion → 21 Figure G.46: Formatted dataset example for Arithmetic 3D- Context → Q: What is 556 plus 497? A: Target Completion → 1053 Figure G.47: Formatted dataset example for Arithmetic 3D+ Context → Q: What is 6209 minus 3365? A: Target Completion → 2844 Figure G.48: Formatted dataset example for Arithmetic 4D- 61
中文: 图G.41:Ro-En上下文的数据集格式示例 Q: 什么是(2 4) 6? A:目标完成-48 图G.42:Arithmetic 1DC上下文格式化数据集实例-Q:17减14是什么? A:目标完成- 3 图G.43: 算术2D-上下文格式化数据集实例-Q:98加45是什么? A:目标完成-143 图G.44:Arithmetic 2D+上下文格式化数据集实例-Q:95乘以45是什么? A:目标完成-4275 图G.45:Arithmetic 2Dx上下文格式化数据集实例-Q:509减488是什么? A:目标完成-21 图G.46: 算术3D-上下文格式化数据集实例-Q:556+497是什么? A:目标完成 1053 图G.47: 算术3D+上下文格式化数据集示例 Q:6209减3365是什么? A:目标完成-2844 图G.48:算术4D-61格式数据集实例
<a id="S1047"></a> Source: p.62 S1047
Original: Context → Q: What is 9923 plus 617? A: Target Completion → 10540 Figure G.49: Formatted dataset example for Arithmetic 4D+ Context → Q: What is 40649 minus 78746? A: Target Completion → -38097 Figure G.50: Formatted dataset example for Arithmetic 5D− Context → Q: What is 65360 plus 16204? A: Target Completion → 81564 Figure G.51: Formatted dataset example for Arithmetic 5D+ 62
中文: 背景-Q:9923加617是什么? A:目标完成- 10540 图G.49:Arithmetic 4D+上下文格式化数据集示例-Q:40649减78746是什么? A:目标完成 -38097 图G.50: 算术5D−上下文=Q:什么是65360+16204? A:目标完成-81564 图G.51:算术5D+62格式数据集实例
<a id="S1048"></a> Source: p.63 S1048
Original: H Results on All Tasks for All Model Sizes Zero-Shot One-Shot Few-Shot Fine-tune 175B Name Metric Split SOTA K Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B Small Med Large XL 2.7B 6.7B 13B 175B (test server) HellaSwag acc dev 85.6 20 33.7 43.6 51.0 54.7 62.8 67.4 70.9 78.9 33.0 42.9 50.5 53.5 61.9 66.5 70.0 78.1 33.5 43.1 51.3 54.9 62.9 67.3 71.3 79.3 LAMBADA acc test 68.0 15 42.7 54.3 60.4 63.6 67.1 70.3 72.5 76.2 22.0 47.1 52.6 58.3 61.1 65.4 69.0 72.5 22.0 40.4 63.2 57.0 78.1 79.1 81.3 86.4 LAMBADA ppl test 8.63 15 18.6 9.09 6.53 5.44 4.60 4.00 3.56 3.00 165.0 11.6 8.29 6.46 5.53 4.61 4.06 3.35 165.0 27.6 6.63 7.45 2.89 2.56 2.56 1.92 StoryCloze acc test 91.8 70 63.3 68.5 72.4 73.4 77.2 77.7 79.5 83.2 62.3 68.7 72.3 74.2 77.3 78.7 79.7 84.7 62.3 70.2 73.9 76.1 80.2 81.2 83.0 87.7 NQs acc test 44.5 64 0.64 1.75 2.71 4.40 6.01 5.79 7.84 14.6 1.19 3.07 4.79 5.43 8.73 9.78 13.7 23.0 1.72 4.46 7.89 9.72 13.2 17.0 21.0 29.9 TriviaQA acc dev 68.0 64 4.15 7.61 14.0 19.7 31.3 38.7 41.8 64.3 4.19 12.9 20.5 26.5 35.9 44.4 51.3 68.0 6.96 16.3 26.5 32.1 42.3 51.6 57.5 71.2 71.2 WebQs acc test 45.5 64 1.77 3.20 4.33 4.63 7.92 7.73 8.22 14.4 2.56 6.20 8.51 9.15 14.5 15.1 19.0 25.3 5.46 12.6 15.9 19.6 24.8 27.7 33.5 41.5 Ro→En 16 BLEU-mb test 39.9 64 2.08 2.71 3.09 3.15 16.3 8.34 20.2 19.9 0.55 15.4 23.0 26.3 30.6 33.2 35.6 38.6 1.25 20.7 25.8 29.2 33.1 34.8 37.0 39.5 Ro→En 16 BLEU-sb test 64 2.39 3.08 3.49 3.56 16.8 8.75 20.8 20.9 0.65 15.9 23.6 26.8 31.3 34.2 36.7 40.0 1.40 21.3 26.6 30.1 34.3 36.2 38.4 41.3 En→Ro 16 BLEU-mb test 38.5 64 2.14 2.65 2.53 2.50 3.46 4.24 5.32 14.1 0.35 3.30 7.89 8.72 13.2 15.1 17.3 20.6 1.25 5.90 9.33 10.7 14.3 16.3 18.0 21.0 En→Ro 16 BLEU-sb test 64 2.61 3.11 3.07 3.09 4.26 5.31 6.43 18.0 0.55 3.90 9.15 10.3 15.7 18.2 20.8 24.9 1.64 7.40 10.9 12.9 17.2 19.6 21.8 25.8 Fr→En 14 BLEU-mb test 35.0 64 1.81 2.53 3.47 3.13 20.6 15.1 21.8 21.2 1.28 15.9 23.7 26.3 29.0 30.5 30.2 33.7 4.98 25.5 28.5 31.1 33.7 34.9 36.6 39.2 Fr→En 14 BLEU-sb test 64 2.29 2.99 3.90 3.60 21.2 15.5 22.4 21.9 1.50 16.3 24.4 27.0 30.0 31.6 31.4 35.6 5.30 26.2 29.5 32.2 35.1 36.4 38.3 41.4 En→Fr 14 BLEU-mb test 45.6 64 1.74 2.16 2.73 2.15 15.1 8.82 12.0 25.2 0.49 8.00 14.8 15.9 20.3 23.3 24.9 28.3 4.08 14.5 19.3 21.5 24.9 27.3 29.5 32.6 En→Fr 14 BLEU-sb test 45.9 64 2.44 2.75 3.54 2.82 19.3 11.4 15.3 31.3 0.81 10.0 18.2 19.3 24.7 28.3 30.1 34.1 5.31 18.0 23.6 26.1 30.3 33.3 35.5 39.9 De→En 16 BLEU-mb test 40.2 64 2.06 2.87 3.41 3.63 21.5 17.3 23.0 27.2 0.83 16.2 22.5 24.7 28.2 30.7 33.0 30.4 3.25 22.7 26.2 29.2 32.7 34.8 37.3 40.6 De→En 16 BLEU-sb test 64 2.39 3.27 3.85 4.04 22.5 18.2 24.4 28.6 0.93 17.1 23.4 25.8 29.2 31.9 34.5 32.1 3.60 23.8 27.5 30.5 34.1 36.5 39.1 43.0 En→De 16 BLEU-mb test 41.2 64 1.70 2.27 2.31 2.43 12.9 8.66 10.4 24.6 0.50 7.00 12.9 13.1 18.3 20.9 22.5 26.2 3.42 12.3 15.4 17.1 20.9 23.0 26.6 29.7 En→De 16 BLEU-sb test 41.2 64 2.09 2.65 2.75 2.92 13.7 9.36 11.0 25.3 0.54 7.40 13.4 13.4 18.8 21.7 23.3 27.3 3.78 12.9 16.1 17.7 21.7 24.1 27.7 30.9 Winograd acc test 93.8 7 66.3 72.9 74.7 76.9 82.4 85.7 87.9 88.3 63.4 68.5 72.9 76.9 82.4 84.6 86.1 89.7 63.4 67.4 73.6 76.9 84.3 85.4 82.4 88.6 Winogrande acc dev 84.6 50 52.0 52.1 57.4 58.7 62.3 64.5 67.9 70.2 51.3 53.0 58.3 59.1 61.7 65.8 66.9 73.2 51.3 52.6 57.5 59.1 62.6 67.4 70.0 77.7 PIQA acc dev 77.1 50 64.6 70.2 72.9 75.1 75.6 78.0 78.5 81.0 64.3 69.3 71.8 74.4 74.3 76.3 77.8 80.5 64.3 69.4 72.0 74.3 75.4 77.8 79.9 82.3 82.8 ARC (Challenge) acc test 78.5 50 26.6 29.5 31.8 35.5 38.0 41.4 43.7 51.4 25.5 30.2 31.6 36.4 38.4 41.5 43.1 53.2 25.5 28.4 32.3 36.7 39.5 43.7 44.8 51.5 ARC (Easy) acc test 92.0 50 43.6 46.5 53.0 53.8 58.2 60.2 63.8 68.8 42.7 48.2 54.6 55.9 60.3 62.6 66.8 71.2 42.7 51.0 58.1 59.1 62.1 65.8 69.1 70.1 OpenBookQA acc test 87.2 100 35.6 43.2 45.2 46.8 53.0 50.4 55.6 57.6 37.0 39.8 46.2 46.4 53.4 53.0 55.8 58.8 37.0 43.6 48.0 50.6 55.6 55.2 60.8 65.4 Quac f1 dev 74.4 5 21.2 26.8 31.0 30.1 34.7 36.1 38.4 41.5 21.1 26.9 31.9 32.3 37.4 39.0 40.6 43.4 21.6 27.6 32.9 34.2 38.2 39.9 40.9 44.3 RACE-h acc test 90.0 10 35.2 37.9 40.1 40.9 42.4 44.1 44.6 45.5 34.3 37.7 40.0 42.0 43.8 44.3 44.6 45.9 34.3 37.0 40.4 41.4 42.3 44.7 45.1 46.8 RACE-m acc test 93.1 10 42.1 47.2 52.1 52.3 54.7 54.4 56.7 58.4 42.3 47.3 51.7 55.2 56.1 54.7 56.9 57.4 42.3 47.0 52.7 53.0 55.6 55.4 58.1 58.1 SQuADv2 em dev 90.7 16 22.6 32.8 33.9 43.1 43.6 45.4 49.0 52.6 25.1 37.5 37.9 47.9 47.9 51.1 56.0 60.1 27.5 40.5 39.2 53.5 50.0 56.6 62.6 64.9 SQuADv2 f1 dev 93.0 16 28.3 40.2 41.4 50.3 51.0 52.7 56.3 59.5 30.1 43.6 44.1 54.0 54.1 57.1 61.8 65.4 32.1 45.5 44.9 58.7 55.9 62.1 67.7 69.8 CoQA f1 dev 90.7 5 34.5 55.0 61.8 65.3 71.1 72.8 76.3 81.5 30.6 52.1 61.6 66.1 71.8 75.1 77.9 84.0 31.1 52.0 62.7 66.8 73.2 77.3 79.9 85.0 DROP f1 dev 89.1 20 9.40 13.6 14.4 16.4 19.7 17.0 24.0 23.6 11.7 18.1 20.9 23.0 26.4 27.3 29.2 34.3 12.9 18.7 24.0 25.6 29.7 29.7 32.3 36.5 BoolQ acc dev 91.0 32 49.7 60.3 58.9 62.4 67.1 65.4 66.2 60.5 52.6 61.7 60.4 63.7 68.4 68.7 69.0 76.7 43.1 60.6 62.0 64.1 70.3 70.0 70.2 77.5 76.4 CB acc dev 96.9 32 0.00 32.1 8.93 19.6 19.6 28.6 19.6 46.4 55.4 53.6 53.6 48.2 57.1 33.9 55.4 64.3 42.9 58.9 53.6 69.6 67.9 60.7 66.1 82.1 75.6 CB f1 dev 93.9 32 0.00 29.3 11.4 17.4 22.4 25.1 20.3 42.8 60.1 39.8 45.6 37.5 45.7 28.5 44.6 52.5 26.1 40.4 32.6 48.3 45.7 44.6 46.0 57.2 52.0 Copa acc dev 94.8 32 66.0 68.0 73.0 77.0 76.0 80.0 84.0 91.0 62.0 64.0 66.0 74.0 76.0 82.0 86.0 87.0 67.0 64.0 72.0 77.0 83.0 83.0 86.0 92.0 92.0 RTE acc dev 92.5 32 47.7 49.8 48.4 56.0 46.6 55.2 62.8 63.5 53.1 47.3 49.5 49.5 54.9 54.9 56.3 70.4 52.3 48.4 46.9 50.9 56.3 49.5 60.6 72.9 69.0 WiC acc dev 76.1 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 50.0 50.3 50.3 49.2 49.4 50.3 50.0 48.6 49.8 55.0 53.0 53.0 51.6 53.1 51.1 55.3 49.4 WSC acc dev 93.8 32 59.6 56.7 65.4 61.5 66.3 60.6 64.4 65.4 58.7 58.7 60.6 62.5 66.3 60.6 66.3 69.2 58.7 60.6 54.8 49.0 62.5 67.3 75.0 75.0 80.1 MultiRC acc dev 62.3 32 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 4.72 9.65 12.3 13.6 14.3 18.4 24.2 27.6 6.09 11.8 16.8 20.8 24.7 23.8 25.0 32.5 30.5 MultiRC f1a dev 88.2 32 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 57.0 59.7 60.4 59.9 60.0 64.5 71.4 72.9 45.0 55.9 64.2 65.4 69.5 66.4 69.3 74.8 75.4 ReCoRD acc dev 92.5 32 70.8 78.5 82.1 84.1 86.2 88.6 89.0 90.2 69.8 77.0 80.7 83.0 85.9 88.0 88.8 90.2 69.8 77.2 81.3 83.1 86.6 87.9 88.9 89.0 90.2 ReCoRD f1 dev 93.3 32 71.9 79.2 82.8 85.2 87.3 89.5 90.4 91.0 70.7 77.8 81.6 83.9 86.8 88.8 89.7 91.2 70.7 77.9 82.1 84.0 87.5 88.8 89.8 90.1 91.1 SuperGLUE average dev 89.0 40.6 47.4 46.8 49.6 50.1 52.3 54.4 58.2 54.4 55.1 56.7 57.8 61.2 59.7 64.3 68.9 50.2 56.2 56.8 60.0 64.3 63.6 66.9 73.2 71.8 ANLI R1 acc test 73.8 50 33.4 34.2 33.4 33.4 34.2 32.3 33.2 34.6 32.1 31.6 31.9 34.6 30.6 31.6 32.7 32.0 32.1 32.5 30.9 32.5 33.5 33.1 33.3 36.8 ANLI R2 acc test 50.7 50 33.2 31.9 33.3 33.3 33.8 33.5 33.5 35.4 35.7 33.7 33.2 32.7 32.7 33.9 33.9 33.9 35.7 33.8 32.1 31.4 32.6 33.3 32.6 34.0 ANLI R3 acc test 48.3 50 33.6 34.0 33.8 33.4 35.3 34.8 34.4 34.5 35.0 32.6 33.0 33.9 34.1 33.1 32.5 35.1 35.0 34.4 35.1 36.0 32.7 33.9 34.5 40.2 2D+ acc n/a 50 0.70 0.65 0.70 0.85 1.10 2.54 15.4 76.9 2.00 0.55 3.15 4.00 12.1 19.6 73.0 99.6 2.00 4.10 3.50 4.50 8.90 11.9 55.5 100.0 2D- acc n/a 50 1.25 1.25 1.25 1.25 1.60 7.60 12.6 58.0 1.15 0.95 1.45 1.95 3.85 11.5 44.6 86.4 1.15 1.45 2.25 2.70 7.35 13.6 52.4 98.9 3D+ acc n/a 50 0.10 0.10 0.05 0.10 0.10 0.25 1.40 34.2 0.15 0.00 0.10 0.30 0.45 0.95 15.4 65.5 0.15 0.45 0.30 0.55 0.75 0.90 8.40 80.4 3D- acc n/a 50 0.05 0.05 0.05 0.05 0.05 0.45 1.35 48.3 0.05 0.15 0.25 0.30 0.55 1.60 6.15 78.7 0.05 0.10 0.15 0.35 0.65 1.05 9.20 94.2 4D+ acc n/a 50 0.05 0.05 0.00 0.00 0.05 0.05 0.15 4.00 0.00 0.00 0.10 0.00 0.00 0.10 0.80 14.0 0.00 0.05 0.05 0.00 0.15 0.15 0.40 25.5 4D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.10 7.50 0.00 0.00 0.00 0.00 0.05 0.00 0.50 14.0 0.00 0.05 0.00 0.00 0.10 0.05 0.40 26.8 5D+ acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.65 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.45 0.00 0.00 0.00 0.00 0.00 0.00 0.05 9.30 5D- acc n/a 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.00 0.05 3.75 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9.90 2Dx acc n/a 50 2.20 2.25 2.65 2.10 2.55 5.80 6.15 19.8 1.35 2.35 3.35 2.35 4.75 9.15 11.0 27.4 1.35 2.90 2.70 2.85 4.25 6.10 7.05 29.2 1DC acc n/a 50 1.25 2.95 2.75 0.05 0.30 2.35 0.75 9.75 1.90 2.80 2.85 3.65 6.45 9.15 8.20 14.3 1.70 2.15 3.90 5.75 6.20 7.60 9.95 21.3 Cycled Letters acc n/a 100 0.62 0.71 2.85 0.00 0.63 1.35 2.58 3.66 1.67 4.36 5.68 6.46 6.25 9.41 15.1 21.7 4.63 9.27 10.7 14.5 16.7 21.9 27.7 37.9 Anagrams 1 acc n/a 100 0.10 0.14 0.40 0.00 0.27 0.69 1.16 2.28 0.21 0.61 1.12 1.27 1.60 2.72 3.72 8.62 0.50 1.27 2.13 3.05 3.81 5.49 8.38 15.1 Anagrams 2 acc n/a 100 0.81 1.21 2.69 0.01 1.71 3.75 4.53 8.91 1.19 2.62 4.70 4.77 6.97 10.2 14.6 25.9 1.94 4.80 7.59 9.87 12.6 18.9 25.6 39.7 Symbol Insertion acc n/a 100 0.00 0.00 0.10 0.00 0.05 0.42 0.89 8.26 0.03 0.05 0.57 1.18 1.67 3.46 6.62 45.4 0.11 0.28 2.19 4.18 6.61 11.0 27.3 67.2 Reversed Words acc n/a 100 0.00 0.01 0.01 0.01 0.02 0.03 0.03 0.09 0.02 0.01 0.01 0.00 0.05 0.07 0.11 0.48 0.00 0.05 0.00 0.17 0.24 0.30 0.42 0.44 SAT Analogies acc n/a 20 35.6 39.0 45.2 44.1 50.0 49.2 52.7 53.7 30.5 41.2 43.1 46.5 55.1 54.3 53.5 59.1 30.5 40.4 42.8 40.6 48.4 51.9 53.5 65.2 Table H.1: Scores for every task, setting and model that we investigate in this paper. 63
中文: H. 2.7B. 6.7B. 13B. 175B. 7B. 7B. 7B. 7B. 6B. 7B. 7B. 7B. 7B. 7B. 7B. 17B. 7B. 7B. 175B(试验服务器) HelaSwag acc. dev 85.620. 33.7 43.6 51.0 54.7 18.8 67.4 70.9 78.9 33.0 42.9 50.5 53.5 61.9 66.0 78.1 33.5 43.1 51.3 54.9 62 67.9 6 7.3 LAMBADA a cc 68.0 15.4 7 54.3 54.3 AMBADA. 26 2.6 2.6 2.6 2.6 6 6 6 6 2.6 6 6 6 6 6 2.6 6 2.6 6 6 6 6 6 6 2.5 6 2.6 6 6 6 2. 6 2. 6 2. 6 2. 6 6 6 6 2.6 6 5.43.8.73 9.78 13.7 23.0 1.72 4.46 7.89 9.7 3.7 4.2 4.7 2.5 14.5 15.1 19.3 TriviaQA acc Dev 68.0 64 4.15 7.61 14.0 19.7 31.3 38.7 41.8 64.3 4.19 12.9 20.5 26.5 35.5 35.9 44.4 51.3 68.6 16.6 16.3 26.5 32.1 42.3 网络Qs acc 测试 45.5 7.6 7.7 7.7 7.73.20 4.33 4.63 7.73.7.73 8.22 14.4.56 6.20 5.5 14.5 15.5 19.1 25.3 5.4 6.6 15.9 0.6 19.6 6 19.6 6 19.6 6 6.6 6 6 6 2.8 2.7 2.7 2.3 2.3 2.3 2.3 20.5 15.5 15.4 23 5.5 23 2.4 23 2.4 3 2.4 3 2.4 3 3 3 3 37.8 37.5 3 39.5 3 39.5 RO 16 16 16 BL.33 BL.03 0 B.3 0 B.39 3.39 3.49 3. 24.9 1.64 7.40 10.9 12.9 17.2 19.6 21.8 25.8 Fr-En FLEU-mb试验 35.0 64 1.81 3.47 3.13 20.6 15.1 21.8 21.2 1.28 15.9 23.7 26.3 29.0 30.5 30.2 33.7 4.98 25.5 28.5 31.1 33.7 34.9 36.6 39.2 Fr-En 14 BLEU-sb试验 64.2.292.9 3.90 3.60 21.2 15.5 22.4 21.9 1.50 16.3 24.4 27.0 30.30.6 31.6 31.4 31.35.5 26.5 5.4 2.54 2.54 2.82 19.3 19.3 Fr 14 BL 3.47 2.74 2.73 2.73 2.23.15 15.1 12.0 25.2 0.9 14.9 14.9 14.3 24.3 24.3 24.3 17.3 17.5 19.3 24.9 27.3 29.5 29.6 En-Fr 14 BLEU-s试验 45.5 B试验 45.9 64.4 2.5 2.44 2.54 2.5 19.3 19.3 19.3 19.4 15.3 15.3 15.3 15.3 15.3 31 31 .60,23.8,27.5,30.5,30.5,34.1,36.5,39.1,43.02,13.92,13.3.36,11.5.3,27.3,27.3,3.78.9,16.7,21.7,24.7,27.7,30.7,30.9,20.00,12.9,13.1,20.3,20.9,20.9,20.3,20.9,26.6,29.7 Enă,16,BLEU-sb测试 41.2,64,39.1,39.1,2.65,2.75,2.92,13.36,11.0,25.3,50.3,13.18.8,21.7,27.7,27.7,27.7,30.9,30.9 温果格勒格勒克测试93.8,766.3,72.7,72.3,74.3,74.3,74.3,74.3,74.3,74.3,74.3,74.3,74.3,74.3,74.3,77.6,78.5,78.5,78.5,6,6和8.4,28.6,6,5,5, 32.3 36.7 39.5 43.7 44.8 51.5 ARC(Easy) acc 试验 92.0 50 43.6 46.5 53.0 53.8 58.2 60.2 63.8 68.8 42.7 48.2 54.6 55.9 60.3 62.6 66.8 71.2 42.7 51.0 58.1 59.1 62.1 65.8 69.1 70.1 OpenBookQA acc 试验 87.2 100 35.6 43.2 45.4 46.8 46.2 46.4 53.4 53.5 55.8 37.0 43.6 48.0 50.6 55.6 55.2 65.4 65.4 Quac f1 43 Dev 74.4 5 2 26.8 31.30.1 34.7 36.1 38.4 41.4 42.1 42.4 42.3 AD 42.9 32.3 37.4 39.6 3 39.6 32.9 32.9 34.9 34.9 RACE-h 试验 9 10 35.9 37.9 40.9 40.4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 ADv2 f1 dev 93.0 16 28.3 40.2 41.4 50.3 51.3 51.0 52.7 56.3 14.5 30.1 43.6 44.1 54.0 54.1 57.6 44.8 65.4 32.1 45.5 44.9 58.7 55.9 62.1 67.7 69.8 CoQA f1 dev 90.7 5 34.5 55.5 55.0 6 65.3 65.3 72.1 72.8 76.3 8 30.5 30.6 66.1 6 6 6 7 6 6 6 7 6 6 6 6 7 6 6 7 6 6 6 6 6 7 6 6 6 6 6 6 6 7 6 6 6 6 6 6 6 6 6 6 7 6 6 6 7 6 6 6 7 6 6 7 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 0.084.0 92.0 62.0 62.0 62.0 62.0 62.0 72.0 72.0 72.0 72.0 72.0 72.0 50.0 50.0 52.0 52.0 RTE acc dev 92.5 32 47.7 49.8 48.4 56.0 46.6 55.2 62.8 63.5 53.1 47.3 49.5 49.5 54.9 56.3 70.4 52.3 52.3 48.4 46.46.50.9 56.3 60.5 60.6 72.9 69.9 69.0 WiC acc 7 3 2.00 0.00 0.00 0.00 500.0 50.3 50.3 50.3 49.2 49.4 49.3 5 5 5 5 5 62.5 5 7 7 6 2.6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 5 6 6 6 6 6 7 6 6 6 6 6 6 6 77.0,80.7,83.0,85.9,88.8,89.8,87.8,77.2,81.3,83.1,86.6,87.9,88.9,89.2,5.8,83.2,dev 93.3,32,71.9,79.2,82.8,85.2,87.3,89.5,90.4,91.0,70.7,77.8,81.6,83.9,86.8,88.8,89.8,89.7,91.2,70.7,87.5,88.8,89.8,90.1,91.1,SuperGLUE 平均d 89.0,40.6,47.4,46.8,49.6,50.6,50.4,55.4,55.1,56.7,56.7,57.8,59.7,6,68.9,50.2,56.2,56.2,56.2,56.8,6 64.3,6,6,6,7,6,7,6,3,3,3,2,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3 3.0 99.6 2.00 4.410 3.50 4.50 4.50 8.90 11.9 55.5 100.0 2D-cc n/a 50 1.25 1.25 1.60 7.60 12.6 58.0 1.15 0.95 1.95 3.85 11.5 44.6 86.4 1.15 2.25 7.27 5.5 52.4 98.9 3D+ acc n/a 50 0.10 0.05 0.010 0.010 1.10 0.25 1.40 34.2 0.30 0.35 0.35 15.5 0.45 0.05 0.50 0.50 0.50 0.50 0.50 0.05 0.05 0.05 0.05 0.50 05.05.05.05.05.05.05.05.60 6.15 78.05.05.05.05.05.04.05.05.05.05.04.05.05.05.05.05.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00 0.00 0.80 0.00 0.00 0.00 0.00 0.00 0.0 5.35 4.35 0.35 4.75 2.2 DC acc n/a 50 2.20 2.25 2.65 2.2.10 2.55 5.80 6.15 19.8 1.35 2.35 4.75 9.15 2.10 2.70 2.85 4.25 6.10 7.05 29.2 DC acc n/ 50.2.95 2.75 0.030 2.35 0.75 2.80 2.85 3.65 6.45 9.15 14.20 14.3 1.70 2.15 3 3 循环函 acc 2.65 2.55 2.55 5.80 2.80 2.80 2.85 0.303 0.63 1.36 5.68 6.46 6.41 5.40 4.7 2.63 9.27 14.7 16.7 27.7 27.9 安娜斯 1.25.02.04.09.04.00 0.00.0 0.0.0 0. 0.27 0.69 1.16 20 2.28 0.01 0.60.60.60.02.40 1.70.02.40 1.70. 2,45.4,0.11,0.28,2.19 4.18 6.61 11.0,27.3 67.2 反相词 acc n/a 100.0.00.01,0.01.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.00.0 5.24 SAT Analogies acc n/a 20 35.6 39.0 45.2 44.1 50.0 49.2 52.7 30.5 41.2 43.1 46.5 55.1 54.5 54.3 53.5 59.1 30.5 40.4 42.8 48.4 51.9 53.5 65.2 表H.1:我们在本文件中调查的每一项任务的分数,设定和模式. 页:1
<a id="S1049"></a> Source: p.64 S1049
Original: Figure H.1: All results for all SuperGLUE tasks.
中文: 图H.1:所有SuperGLUE任务的所有结果.
<a id="S1050"></a> Source: p.64 S1050
Original: Figure H.3: All results for all Winograd tasks. 64
中文: 图H.3:所有维诺格勒任务的所有结果. 64国道
<a id="S1051"></a> Source: p.65 S1051
Original: Figure H.4: All results for all Arithmetic tasks.
中文: 图H.4:所有算术任务的所有结果。
<a id="S1052"></a> Source: p.65 S1052
Original: Figure H.5: All results for all Cloze and Completion tasks. 65
中文: 图H.5:所有克洛兹和完成任务的所有结果。 65个
<a id="S1053"></a> Source: p.66 S1053
Original: Figure H.6: All results for all Common Sense Reasoning tasks.
中文: 图H.6:所有共同理智理性任务的所有结果。
<a id="S1054"></a> Source: p.66 S1054
Original: Figure H.7: All results for all QA tasks.
中文: 图H.7:所有质量保证任务的所有结果。
<a id="S1055"></a> Source: p.66 S1055
Original: Figure H.8: All results for all Reading Comprehension tasks.
中文: 图H.8:所有阅读理解任务的所有结果。
<a id="S1056"></a> Source: p.66 S1056
Original: Figure H.9: All results for all ANLI rounds. 66
中文: 图H.9:所有ANLI回合的所有结果。 第66条
<a id="S1057"></a> Source: p.67 S1057
Original: Figure H.10: All results for all Scramble tasks.
中文: 图H.10:所有快活任务的所有结果.
<a id="S1058"></a> Source: p.67 S1058
Original: Figure H.11: All results for all Translation tasks. 67
中文: 图H.11:所有翻译任务的所有结果。 67国道
<a id="S1059"></a> Source: p.68 S1059
Original: References [ADG+16] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas.
中文: 参考文献[ADG+16]. Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, 和南多·德·弗雷塔斯.
<a id="S1060"></a> Source: p.68 S1060
Original: Learning to learn by gradient descent by gradient descent.
中文: 学习用梯度下垂来学习用梯度下垂来学习.
<a id="S1061"></a> Source: p.68 S1061
Original: In Advances in neural information processing systems, pages 3981–3989, 2016. [AI19] WeChat AI.
中文: In Advances in 神经信息处理系统, 第3981–3989页, 2016. [AI19] WeChat AI.
<a id="S1062"></a> Source: p.68 S1062
Original: Tr-mt (ensemble), December 2019. [AJF19] Roee Aharoni, Melvin Johnson, and Orhan Firat.
中文: Tr-mt (综艺),2019年12月. [AJF19] 罗伊·阿哈罗尼,梅尔文·约翰逊和奥尔汉·菲拉特.
<a id="S1063"></a> Source: p.68 S1063
Original: Massively multilingual neural machine translation.
中文: 大规模多语言神经机能翻译.
<a id="S1064"></a> Source: p.68 S1064
Original: In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019. [BBDIW20] Su Lin Blodgett, Solon Barocas, Hal Daume´ III, and Hanna Wallach.
中文: 计算语言学协会北美分会2019年会议记录:人文技术 第1卷(长篇和短篇论文),2019. [BBDIW20] 苏琳·布洛德盖特,索隆·巴罗卡斯,哈尔·道梅三世和汉娜·瓦拉赫.
<a id="S1065"></a> Source: p.68 S1065
Original: Language (technology) is power: A critical survey of “bias” in nlp. arXiv preprint arXiv:2005.14050, 2020. [BCFL13] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.
中文: 语言(技术)是功率:对nlp中的"比斯"进行批判性调查. arXiv preprint arXiv:2005.14050,2020. [BCFL13] 乔纳森·贝兰特,安德鲁·丘,罗伊·弗洛斯蒂格,和珀西·梁.
<a id="S1066"></a> Source: p.68 S1066
Original: Semantic parsing on freebase from question-answer pairs.
中文: 语义解析自由基 从问答对.
<a id="S1067"></a> Source: p.68 S1067
Original: In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013. [BDD+09] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini.
中文: 2013年自然语言处理经验方法会议纪要 第1533–1544页. 2013. [BDD+09] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampicolo, and Bernardo Magnini.
<a id="S1068"></a> Source: p.68 S1068
Original: The fifth PASCAL recognizing textual entailment challenge. 2009. [BES10] Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani.
中文: 第五个PASCAL承认在文字上存在挑战。 2009. [BES10] (中文(简体) ). Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastani.
<a id="S1069"></a> Source: p.68 S1069
Original: Sentiwordnet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining.
中文: Sentiwordnet 3.0:增强情绪分析和意见挖掘的词汇资源.
<a id="S1070"></a> Source: p.68 S1070
Original: In Lrec, volume 10, pages 2200–2204, 2010. [BHDD+06] Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor.
中文: 在"勒克"第10卷第2200–2204页,2010. [BHDD+06]. Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor.
<a id="S1071"></a> Source: p.68 S1071
Original: The second PASCAL recognising textual entailment challenge. 2006. [BHT+20] Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al.
中文: 第二次PASCAL承认在文本方面面临的挑战。 2006. [BHT+20]. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich等.
<a id="S1072"></a> Source: p.68 S1072
Original: Experience grounds language. arXiv preprint arXiv:2004.10151, 2020. [BLC13] Yoshua Bengio, Nicholas Le´onard, and Aaron C.
中文: 经验是语言的基础。 arXiv preprint arXiv:2004.10151, 2020. [BLC13]. Yoshua Bengio, Nicholas Le'onard, and Aaron C.
<a id="S1073"></a> Source: p.68 S1073
Original: Estimating or propagating gradients through stochastic neurons for conditional computation.
中文: 通过分泌神经元来估计或传播梯度,用于有条件的计算.
<a id="S1074"></a> Source: p.68 S1074
Original: Arxiv, 2013. [BZB+19] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.
中文: Arxiv, 2013. [BZB+19] (中文(中国大陆) ). Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.
<a id="S1075"></a> Source: p.68 S1075
Original: Piqa: Reasoning about physical commonsense in natural language. arXiv preprint arXiv:1911.11641, 2019. [Car97] Rich Caruana.
中文: 皮卡:自然语言中身体常识的原因. arXiv预印版 arXiv:1911.11641, 2019. [Car97] 富家克·卡鲁阿纳.
<a id="S1076"></a> Source: p.68 S1076
Original: Machine learning, 28(1), 1997. [CB78] Susan Carey and Elsa Bartlett.
中文: 机器学习, 第28(1), 1997. [CB78] Susan Carey and Elsa Bartlett.
<a id="S1077"></a> Source: p.68 S1077
Original: Proceedings of the Stanford Child Language Conference, 1978. [CCE+18] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
中文: 1978年斯坦福儿童语言会议会议记录. [CCE+18]. 彼得·克拉克,艾萨克·考伊,奥伦·埃齐奥尼,图沙尔·霍特,阿希什·萨布哈尔瓦尔,卡里萨·肖埃尼克,和奥伊文德·塔夫乔德.
<a id="S1078"></a> Source: p.68 S1078
Original: Think you have solved question answering? try arc, the ai2 reasoning challenge.
中文: 你以为你回答了问题吗? 尝试弧, ai2 推理挑战 。
<a id="S1079"></a> Source: p.68 S1079
Original: ArXiv, abs/1803.05457, 2018. [CGRS19] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
中文: ArXiv, abs/1803.05457, 2018. [CGRS19] (英语). Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.
<a id="S1080"></a> Source: p.68 S1080
Original: Generating long sequences with sparse transformers, 2019. [CHI+18] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer.
中文: 以稀疏的变压器生成长序, 2019. [CHI+18] Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, 和 Luke Zettlemoyer.
<a id="S1081"></a> Source: p.68 S1081
Original: Arxiv, 2018. [CLC+19] Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.
中文: Arxiv, 2018. [CLC+19]. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, 和克丽斯蒂娜·图塔诺娃.
<a id="S1082"></a> Source: p.68 S1082
Original: BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019. [CLY+19] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.
中文: BoulQ:探索自然是/不是问题的出人意料的困难. arXiv preprint arXiv:1905.10044, 2019. [CLY+19] (中文(中国大陆) ). 陈延春,李林杰,李克裕,艾哈迈德·埃尔·霍利,费萨尔·艾哈迈德,浙甘,于成和刘静静.
<a id="S1083"></a> Source: p.68 S1083
Original: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740, 2019. [Cra17] Kate Crawford.
中文: Uniter:学习通用图像-文本表达. arXiv preprint arXiv:1909.11740, 2019. [Cra17] 凯特·克劳福德.
<a id="S1084"></a> Source: p.68 S1084
Original: NIPS 2017 Keynote, 2017. [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
中文: NIPS 2017主题曲, 2017. [DCLT18]. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
<a id="S1085"></a> Source: p.68 S1085
Original: BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 68
中文: BERT:深双向变换器进行语言理解前训练. arXiv preprint arXiv:181.04805, 2018. 68 (英语).
<a id="S1086"></a> Source: p.69 S1086
Original: [DGM06] Ido Dagan, Oren Glickman, and Bernardo Magnini.
中文: [DGM06] Ido Dagan, Oren Glickman, 和贝尔纳多·马格尼尼.
<a id="S1087"></a> Source: p.69 S1087
Original: The PASCAL recognising textual entailment challenge.
中文: 《宪法通则》承认在文字上存在挑战。
<a id="S1088"></a> Source: p.69 S1088
Original: In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising textual entailment, pages 177–190.
中文: 在机器学习挑战。 评估预测不确定性、视觉物体分类和承认文字含义,第177-190页。
<a id="S1089"></a> Source: p.69 S1089
Original: Springer, 2006. [DGV+18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser.
中文: Springer, 2006. [DGV+18]. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 中国植物物种信息数据库.
<a id="S1090"></a> Source: p.69 S1090
Original: Arxiv, 2018. [DHKH14] Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield.
中文: Arxiv, 2018. [DHKH14]. Nadir Durrani, Barry Haddow, Philipp Koehn,和肯尼斯·希菲尔德.
<a id="S1091"></a> Source: p.69 S1091
Original: Edinburgh’s phrase-based machine translation systems for wmt-14.
中文: 爱丁堡基于词组的机器翻译系统为wmt-14.
<a id="S1092"></a> Source: p.69 S1092
Original: In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 97–104, 2014. [DL15] Andrew M.
中文: 《统计机器翻译第九次讲习班记录》,第97-104页,2014年[DL15] Andrew M.
<a id="S1093"></a> Source: p.69 S1093
Original: In Advances in neural information processing systems, 2015. [DMST19] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.
中文: In Advances in 神经信息处理系统, 2015. [DMST19]. Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser.
<a id="S1094"></a> Source: p.69 S1094
Original: The CommitmentBank: Investigating projection in naturally occurring discourse. 2019.
中文: 承诺银行:调查自然发生的言论中的预测。 2019 (英语).
<a id="S1095"></a> Source: p.69 S1095
Original: To appear in proceedings of Sinn und Bedeutung 23.
中文: 出席Sinn und Bedeutung 23.的诉讼。
<a id="S1096"></a> Source: p.69 S1096
Original: Data can be found at https://github.com/mcdm/CommitmentBank/. [DSC+16] Yan Duan, John Schulman, Xi Chen, Peter L.
中文: 数据见于https://github.com/mcdm/ConmitmentBank/. [DSC+16] 颜都安,约翰·舒尔曼,习近平,彼得·L.
<a id="S1097"></a> Source: p.69 S1097
Original: Bartlett, Ilya Sutskever, and Pieter Abbeel.
中文: 巴特利特,伊利亚·萨特斯克韦尔,和彼得·阿比尔.
<a id="S1098"></a> Source: p.69 S1098
Original: Rl2: Fast reinforcement learning via slow reinforcement learning.
中文: Rl2:通过缓慢的强化学习来进行快速强化学习.
<a id="S1099"></a> Source: p.69 S1099
Original: ArXiv, abs/1611.02779, 2016. [DWD+19] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
中文: ArXiv, abs/16111.02779, 2016. [DWD+19] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 互联网档案馆的存檔,存档日期2013-12-21.
<a id="S1100"></a> Source: p.69 S1100
Original: Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019. [DYY+19] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G.
中文: 降:读取理解基准,需要对段落进行离散推理。 arXiv preprint arXiv:1903.00161, 2019. [DYY+19] (中文(中国大陆) ). ). Ziang Dai, Zhilin Yang, Yiming Yang, Jaime G.
<a id="S1101"></a> Source: p.69 S1101
Original: Transformer-xl: Attentive language models beyond a fixed-length context.
中文: Transformer-xl:超出固定长度上下文的Attentive语言模型.
<a id="S1102"></a> Source: p.69 S1102
Original: Arxiv, 2019. [EOAG18] Sergey Edunov, Myle Ott, Michael Auli, and David Grangier.
中文: Arxiv, 2019. [EOAG18] (英语). Sergey Edunov, Myle Ott, Michael Auli, and David Grangier.
<a id="S1103"></a> Source: p.69 S1103
Original: Understanding back-translation at scale. arXiv preprint arXiv:1808.09381, 2018. [FAL17] Chelsea Finn, Pieter Abbeel, and Sergey Levine.
中文: 以规模理解回译. arXiv preprint arXiv:1808.09381, 2018. [FAL17] 切尔西·芬恩,皮埃特·阿比埃尔和谢尔盖·莱文.
<a id="S1104"></a> Source: p.69 S1104
Original: Model-agnostic meta-learning for fast adaptation of deep networks.
中文: 模式-不可知元-学习以快速改造深层网络.
<a id="S1105"></a> Source: p.69 S1105
Original: ArXiv, abs/1703.03400, 2017. [Fyo00] Yaroslav Fyodorov. A natural logic inference system, 2000. [GG19] Hila Gonen and Yoav Goldberg.
中文: ArXiv, abs/1703.03400, 2017. [Fyo00] (英语). Yaroslav Fyodorov. 自然逻辑推论系统, 2000. [GG19] Hila Gonen and Yoav Goldberg.
<a id="S1106"></a> Source: p.69 S1106
Original: Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv:1903.03862, 2019. [GLT+20] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.
中文: 对一头猪的唇膏:Debiasing方法在文字嵌入中掩盖了系统性的性别偏见,但不去除. arXiv preprint arXiv:1903.03862, 2019. [GLT+20] 开尔文·郭,克通·李克通,卓拉·通,帕努蓬·帕苏帕特,明-魏昌.
<a id="S1107"></a> Source: p.69 S1107
Original: Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. [GMDD07] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan.
中文: retrievalaugmented 语言模型预训. arXiv preprint arXiv:2002.08909,2020. [GMDD07] 达尼洛·吉亚姆皮克洛,贝尔纳多·马格尼尼,伊多·达甘和比尔·多兰.
<a id="S1108"></a> Source: p.69 S1108
Original: The third PASCAL recognizing textual entailment challenge.
中文: 第三个PASCAL承认文字上的挑战。
<a id="S1109"></a> Source: p.69 S1109
Original: In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9.
中文: ACL-PASCAL关于文字含义和提法的讲习班记录,第1-9页。
<a id="S1110"></a> Source: p.69 S1110
Original: Association for Computational Linguistics, 2007. [Gra16] Alex Graves.
中文: 计算语言学协会, 2007. [GRA16] Alex Graves.
<a id="S1111"></a> Source: p.69 S1111
Original: Adaptive computation time for recurrent neural networks.
中文: 经常性神经网络的适应性计算时间.
<a id="S1112"></a> Source: p.69 S1112
Original: Arxiv, 2016. [GSL+18] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith.
中文: Arxiv, 2016. [GSL+18]. Suchin Gurururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, 和诺亚·A·史密斯.
<a id="S1113"></a> Source: p.69 S1113
Original: Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324, 2018. [GSR19] Sebastian Gehrmann, Hendrik Strobelt, and Alexander M.
中文: 自然语言推断数据中的注释文物. arXiv preprint arXiv:1803.02324, 2018. [GSR19]塞巴斯蒂安·盖尔曼,亨德里克·斯特罗贝尔特和亚历山大·M.
<a id="S1114"></a> Source: p.69 S1114
Original: Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv: 1906.04043, 2019. [GWC+18] Jiatao Gu, Yong Wang, Yun Chen, Kyunghyun Cho, and Victor OK Li.
中文: Gltr:生成文本的统计检测和可视化. arXiv preprint arXiv: 1906.04043, 2019. [GWC+18] 顾家涛, 王勇, 陈云, Cyunghyun Cho,和 Victor OK Li.
<a id="S1115"></a> Source: p.69 S1115
Original: Meta-learning for low-resource neural machine translation. arXiv preprint arXiv:1808.08437, 2018. [HB20] Daniel Hernandez and Tom Brown.
中文: 低资源神经机能翻译的Meta-learning. arXiv preprint arXiv:1808.08437, 2018. [HB20] 丹尼尔·赫南德斯和汤姆·布朗.
<a id="S1116"></a> Source: p.69 S1116
Original: Ai and efficiency, May 2020. [HBFC19] Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi.
中文: Ai and effective, May 2020. [HBFC19] Ari Holtzman, Jan Buys, Maxwell Forbes, 和叶津崔.
<a id="S1117"></a> Source: p.69 S1117
Original: The curious case of neural text degeneration.
中文: 神经文字退化的奇特案例.
<a id="S1118"></a> Source: p.69 S1118
Original: CoRR, abs/1904.09751, 2019. [HLW+20] Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song.
中文: CoRR, abs/1904.09751, 2019. [HLW+20] (英语). 丹·亨德里克,刘晓远,埃里克·华莱士,亚当·季德齐克,里沙布·克里希南,和道恩·宋.
<a id="S1119"></a> Source: p.69 S1119
Original: Pretrained transformers improve out of distribution robustness. arXiv preprint arXiv:2004.06100, 2020. 69
中文: 预装变压器的配电强度得到提高。 arXiv预印 arXiv:2004.06100,2020. 69 (英语).
<a id="S1120"></a> Source: p.70 S1120
Original: [HNA+17] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md.
中文: [HNA+17] 乔尔·赫斯特内斯,沙兰·纳朗,纽沙·阿达拉尼,格雷戈里·迪亚莫斯,希宇·俊,哈桑·克亚尼内贾德,Md.
<a id="S1121"></a> Source: p.70 S1121
Original: Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.
中文: 穆斯多法·阿里·帕特瓦里,杨洋和周艳琪.
<a id="S1122"></a> Source: p.70 S1122
Original: Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. [HR18] Jeremy Howard and Sebastian Ruder.
中文: 深入的学习规模是可预测的,经验性的。 arXiv preprint arXiv:1712.00409,2017. [HR18] 杰里米·霍华德和塞巴斯蒂安·鲁德.
<a id="S1123"></a> Source: p.70 S1123
Original: Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146, 2018. [HVD15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.
中文: 通用语言模型微调文本分类. arXiv preprint arXiv:1801.06146, 2018. [HVD15]. 杰弗里·欣通,奥里奥尔·维尼亚尔斯和杰夫·迪恩.
<a id="S1124"></a> Source: p.70 S1124
Original: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. [HYC01] Sepp Hochreiter, A Steven Younger, and Peter R Conwell.
中文: 在神经网络中消化知识. arXiv preprint arXiv:1503.02531,2015. [HYC01] (英语). Sepp Hochreiter, A Steven Younger, and Peter R Conwell.
<a id="S1125"></a> Source: p.70 S1125
Original: Learning to Learn Using Gradient Descent.
中文: 学习如何使用渐入世.
<a id="S1126"></a> Source: p.70 S1126
Original: In International Conference on Artificial Neural Networks, pages 87–94.
中文: 人工神经网络国际会议,第87-94页。
<a id="S1127"></a> Source: p.70 S1127
Original: Springer, 2001. [HZJ+19] Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli.
中文: Springer, 2001. [HZJ+19]. [HZJ+19]. 黄宝森,张万章,雷·江,罗伯特·斯坦福斯,约翰内斯·韦尔布尔,杰克·雷,维沙尔·美尼,达尼·约加塔马,和普希米特·科利.
<a id="S1128"></a> Source: p.70 S1128
Original: Reducing sentiment bias in language models via counterfactual evaluation. arXiv preprint arXiv:1911.03064, 2019. [IBGC+14] Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daume´ III. A neural network for factoid question answering over paragraphs.
中文: 通过反事实评价来减少语言模型中的情绪偏差. arXiv preprint arXiv:1911.03064, 2019. [IBGC+14] (英语). Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daume' III. 神经网络 事实问题回答段落。
<a id="S1129"></a> Source: p.70 S1129
Original: In Empirical Methods in Natural Language Processing, 2014. [IDCBE19] Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck.
中文: 在"自然语言处理的经验方法"中,2014. [IDCBE19] Daphne Ipolito, Daniel Duckworth, Chris Callison-Burch, 和道格拉斯·埃克.
<a id="S1130"></a> Source: p.70 S1130
Original: Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019. [JCWZ17] Mandar Joshi, Eunsol Choi, Daniel S.
中文: 当人类被愚弄时,自动检测生成的文本是最容易的. arXiv preprint arXiv:1911.00650, 2019. [JCWZ17] (中文(简体) ). Joshi, Eunsol Choi, Daniel S.
<a id="S1131"></a> Source: p.70 S1131
Original: TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. [JN20] Zheng Junyuan and Gamma Lab NYC.
中文: TriviaQA:用于读取理解的被远程监督的大型挑战数据集. arXiv preprint arXiv:1705.03551, 2017. [JN20] 郑俊远和伽玛实验室 NYC.
<a id="S1132"></a> Source: p.70 S1132
Original: Numeric transformer - albert, March 2020. [JVS+16] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.
中文: 数字变速器-阿尔伯特,2020年3月. [JVS+16]. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, 吴永辉.
<a id="S1133"></a> Source: p.70 S1133
Original: Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [JYS+19] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.
中文: 探索语言建模的极限. arXiv preprint arXiv:1602.02410, 2016. [JYS+19] (中文(简体) ). [JYS+19] (简体). ). 萧子乔. 宜春燕. 寿生翔. 新江会. 小陈会. 林林丽. 方王会.
<a id="S1134"></a> Source: p.70 S1134
Original: TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv:1909.10351, 2019. [JZC+19] Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu.
中文: TinyBERT:为自然语言理解而 distilling BERT. arXiv preprint arXiv:1909.10351, 2019. [JZC+19] (中文(中国大陆) ). 英珠,傅邦昭,陈世杰,鲍文 Zheng,薛芬杨,刘云峰.
<a id="S1135"></a> Source: p.70 S1135
Original: Technical report on conversational question answering. arXiv preprint arXiv:1909.10772, 2019. [KCR+18] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.
中文: 关于回答谈话问题的技术报告。 arXiv preprint arXiv:1909.10772, 2019. [KCR+18] 丹尼尔·克沙比,斯尼格达·查图尔维迪,迈克尔·罗斯,斯亚姆·乌帕德希耶,和丹·罗斯.
<a id="S1136"></a> Source: p.70 S1136
Original: Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.
中文: 从表面看:阅读理解多句的挑战。
<a id="S1137"></a> Source: p.70 S1137
Original: In Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL), 2018. [KKS+20] Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.
中文: 2018年"计算语言学协会北美分会记录"(NAACL). [KKS+20] Daniel Khashabi, Tushhar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi.
<a id="S1138"></a> Source: p.70 S1138
Original: Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700, 2020. [KMB20] Sarah E.
中文: 统一qa:以单一qa系统跨越格式边界. arXiv preprint arXiv:2005.00700, 2020. [KMB20] 莎拉·E.
<a id="S1139"></a> Source: p.70 S1139
Original: Kreps, Miles McCain, and Miles Brundage.
中文: 克莱普斯 迈尔斯·麦凯恩 还有迈尔斯·布伦达奇
<a id="S1140"></a> Source: p.70 S1140
Original: All the news that’s fit to fabricate: Ai-generated text as a tool of media misinformation, 2020. [KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B.
中文: 所有适合捏造的新闻:Ai生成的文本作为媒体错误信息的工具,2020年. [KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B.
<a id="S1141"></a> Source: p.70 S1141
Original: Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
中文: 布朗,本杰明·切斯,雷文·克特,斯科特·格雷,阿莱克·拉德福德,杰弗里·吴和达里奥·阿莫代.
<a id="S1142"></a> Source: p.70 S1142
Original: Scaling laws for neural language models, 2020. [KPR+19] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N.
中文: 神经语言模型的放大定律, 2020. [KPR+19]. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N.
<a id="S1143"></a> Source: p.70 S1143
Original: Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
中文: 图塔诺瓦,利翁·琼斯,明-魏昌,安德鲁·戴,雅可布·乌兹克赖特,克克·勒和斯拉夫·彼得罗夫.
<a id="S1144"></a> Source: p.70 S1144
Original: Natural questions: a benchmark for question answering research.
中文: 自然问题:回答问题研究的基准.
<a id="S1145"></a> Source: p.70 S1145
Original: Transactions of the Association of Computational Linguistics, 2019. [KR16] Yoon Kim and Alexander M.
中文: 计算语言学协会的交易, 2019. [KR16] Yoon Kim和Alexander M.
<a id="S1146"></a> Source: p.70 S1146
Original: Arxiv, 2016. [LB02] Edward Loper and Steven Bird.
中文: Arxiv, 2016. [LB02] (英语). Edward Loper and Steven Bird.
<a id="S1147"></a> Source: p.70 S1147
Original: Nltk: The natural language toolkit, 2002. [LC19] Guillaume Lample and Alexis Conneau.
中文: Nltk:自然语言工具包, 2002. [LC19] 纪尧姆·兰普和Alexis Conneau.
<a id="S1148"></a> Source: p.70 S1148
Original: Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. 70
中文: 跨语言模式预训. arXiv预印版arXiv:1901.07291, 2019. 70
<a id="S1149"></a> Source: p.71 S1149
Original: [LCG+19] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.
中文: [LCG+19] 曾中兰,陈明达,塞巴斯蒂安·古德曼,凯文·金佩尔,皮尤什·夏尔马,和拉都·索里克特.
<a id="S1150"></a> Source: p.71 S1150
Original: ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. [LCH+20] Xiaodong Liu, Hao Cheng, Pengcheng He, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao.
中文: ALBERT:用于自我监督学习语言表现的一升BERT. arXiv preprint arXiv:1909.11942, 2019. [LCH+20] (中文(中国大陆) ). 刘晓东,郝程,彭城贺,陈伟祖,于旺,何丰蓬,高建丰.
<a id="S1151"></a> Source: p.71 S1151
Original: Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994, 2020. [LDL19] Zhongyang Li, Xiao Ding, and Ting Liu.
中文: 大型神经语言模型的横向训练. arXiv preprint arXiv:2004.08994,2020. [LDL19] 李朝阳,萧鼎,丁刘.
<a id="S1152"></a> Source: p.71 S1152
Original: Story ending prediction by transferable bert. arXiv preprint arXiv:1905.07504, 2019. [LDM12] Hector Levesque, Ernest Davis, and Leora Morgenstern.
中文: 故事结束预测 通过可转移的伯特。 arXiv preprint arXiv:1905.07504, 2019. [LDM12] 赫克托·莱韦斯克,欧内斯特·戴维斯,和利奥拉·莫根斯特恩.
<a id="S1153"></a> Source: p.71 S1153
Original: In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012. [LGG+20] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.
中文: 2012年第十三届知识代表与理性原则国际会议[LGG+20] 刘燕汉,顾家涛,纳南高雅尔,X丽,谢尔盖·埃杜诺夫,马让·加兹维尼内贾德,迈克·刘易斯和卢克·泽特勒莫耶.
<a id="S1154"></a> Source: p.71 S1154
Original: Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.08210, 2020. [LGH+15] Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang.
中文: 多语言去声预训神经机能翻译. arXiv preprint arXiv:2001.08210,2020. [LGH+15] (中文(中国大陆) ). 刘晓东,高建丰,贺晓东,李登,凯文·杜和叶毅.
<a id="S1155"></a> Source: p.71 S1155
Original: Representation learning using multi-task deep neural networks for semantic classification and information retrieval.
中文: 利用多任务深度神经网络进行代表学习,进行语义分类和信息检索.
<a id="S1156"></a> Source: p.71 S1156
Original: In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2015. [LH17] Ilya Loshchilov and Frank Hutter.
中文: 计算语言学协会北美分会2015年会议纪要:人类语言技术,2015 [LH17] Ilya Loshchilov和Frank Hutter.
<a id="S1157"></a> Source: p.71 S1157
Original: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [LHCG19a] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.
中文: 脱钩后体重衰减规范化. arXiv preprint arXiv:1711.05101, 2017. [LHCG19a] (中文(中国大陆) ). 刘晓东,彭城贺,魏祖陈,建丰高.
<a id="S1158"></a> Source: p.71 S1158
Original: Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482, 2019. [LHCG19b] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao.
中文: 通过知识分馏来改善多任务深度神经网络,以进行自然语言理解. arXiv preprint arXiv:1904.09482, 2019. [LHCG19b]刘晓东,彭城贺,陈伟祖,高建丰.
<a id="S1159"></a> Source: p.71 S1159
Original: Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504, 2019. [Lin20] Tal Linzen.
中文: 多任务深度神经网络促进自然语言理解. arXiv preprint arXiv:1901.11504, 2019. [林20] 塔尔·林曾.
<a id="S1160"></a> Source: p.71 S1160
Original: How can we accelerate progress towards human-like linguistic generalization? arXiv preprint arXiv:2005.00955, 2020. [LLG+19] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.
中文: 我们如何能加速朝向类似人类的语言通论的进展? arXiv preprint arXiv:2005.00955,2020. [LLG+19]. 迈克·刘易斯,刘殷汉,纳曼·戈亚尔,马让·加兹维尼内贾德,阿卜杜勒拉赫曼·穆罕默德,奥梅尔·莱维,维斯·斯托扬诺夫,和卢克·泽特勒莫耶.
<a id="S1161"></a> Source: p.71 S1161
Original: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. [LM17] Ke Li and Jitendra Malik.
中文: 巴特:为自然语言的生成,翻译,和理解而进行德诺瓦宁序列到序列的预训. arXiv preprint arXiv:1910.13461, 2019. [LM17]克李和吉滕德拉·马利克.
<a id="S1162"></a> Source: p.71 S1162
Original: Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017. [LOG+19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
中文: 学习优化神经网. arXiv preprint arXiv:1703.00441, 2017. [LOG+19] (中文(中国大陆) ). Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandaers Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, 和 Veselin Stoyanov.
<a id="S1163"></a> Source: p.71 S1163
Original: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. [LPP+20] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Ku¨ttler, Mike Lewis, Wen-tau Yih, Tim Rockta¨schel, Sebastian Riedel, and Kiela Douwe.
中文: RoBERTA:强力优化BERT预训方式. arXiv preprint arXiv:1907.11692, 2019. [LPP+20]. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Ku'ttler, Mike Lewis, Wen-tau Yih, Tim Rockta'schel, Sebastian Riedel, 和Kiela Douwe.
<a id="S1164"></a> Source: p.71 S1164
Original: Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401, 2020. [LSP+18] Peter J.
中文: 用于知识密集型nlp任务的检索强化生成. arXiv preprint arXiv:2005.11401,2020. [LSP+18] 彼得·J.
<a id="S1165"></a> Source: p.71 S1165
Original: Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer.
中文: 刘,穆罕默德·萨利赫,艾蒂安·波特,本·古德里奇,瑞安·塞帕西,卢卡斯克·凯泽尔,和诺姆·沙泽尔.
<a id="S1166"></a> Source: p.71 S1166
Original: Generating Wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198, 2018. [LWS+20] Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E.
中文: 通过总结长序来生成维基百科. arXiv preprint arXiv:1801.10198, 2018. [LWS+20] 李卓汉,埃里克·华莱士,沈克生,凯文·林,克特·克特策,丹·克莱因,和约瑟夫·E.
<a id="S1167"></a> Source: p.71 S1167
Original: Train large, then compress: Rethinking model size for efficient training and inference of transformers, 2020. [LXL+17] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.
中文: 列车大,再压缩:为高效训练和推算变压器而重新思考型号大小,2020. [LXL+17] 郭昆来,齐泽谢,刘汉晓,杨一明和爱德华·霍维.
<a id="S1168"></a> Source: p.71 S1168
Original: Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683, 2017. [LYN+20] Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin.
中文: 种族:通过考试获得的大规模阅读理解数据集. arXiv preprint arXiv:1704.04683, 2017. [LYN+20] (中文(中国大陆) ). 沈克林,郑克宏,罗德里戈·诺盖拉,明-芬-蔡,王中珠,吉米·林.
<a id="S1169"></a> Source: p.71 S1169
Original: Tttttackling winogrande schemas. arXiv preprint arXiv:2003.08380, 2020. [Mac92] David.
中文: Tttttackling winogrande schemas. arXiv预印 arXiv:2003.08380,2020. [马来文92] 大卫.
<a id="S1170"></a> Source: p.71 S1170
Original: Information-based objective functions for active data selection.
中文: 主动数据选择的基于信息的客观功能.
<a id="S1171"></a> Source: p.72 S1171
Original: [MBXS17] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher.
中文: [MBXS17] 布莱恩·麦坎,詹姆斯·布拉德伯里,凯明·西翁,和理查德·索彻.
<a id="S1172"></a> Source: p.72 S1172
Original: Learned in translation: Contextualized word vectors.
中文: 翻译学:背景化字向量.
<a id="S1173"></a> Source: p.72 S1173
Original: In Advances in Neural Information Processing Systems, pages 6294–6305, 2017. [MCCD13] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.
中文: In Advances in Neural Information Systems, pages 6294–6305, 2017. [MCCD13] (英语). 托马斯·米科洛夫, Cai Chen, Greg Corrado, and Jeffrey Dean.
<a id="S1174"></a> Source: p.72 S1174
Original: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. [MCH+16] Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696, 2016. [MCKS18] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.
中文: 有效估计矢量空间中的字词表示. arXiv preprint arXiv:1301.3781, 2013. [MCH+16]. Nasrin Mostafazadeh, Nathanael Chambers, 小东贺, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, 和詹姆斯·艾伦. 更深入了解常识故事的理论和评估框架。 arXiv preprint arXiv:1604.01696, 2016. [MCKS18] 托多尔·米哈伊洛夫,彼得·克拉克,图沙尔·霍特和阿希什·萨巴瓦尔.
<a id="S1175"></a> Source: p.72 S1175
Original: Can a suit of armor conduct electricity? a new dataset for open book question answering.
中文: 盔甲能带电吗? 用于回答开放式书籍问题的新数据集。
<a id="S1176"></a> Source: p.72 S1176
Original: ArXiv, abs/1809.02789, 2018. [MKAT18] Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team.
中文: ArXiv, abs/1809.02789, 2018. [MKAT18] 山姆·麦坎德利什,贾里德·卡普兰,达里奥·阿莫代和OpenAI Dota Team.
<a id="S1177"></a> Source: p.72 S1177
Original: An empirical model of large-batch training, 2018. [MKM+94] Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger.
中文: [MKM+94] 米切尔·马库斯,格蕾丝·金,玛丽·安·马辛基耶维奇,罗伯特·马金泰尔,安·比斯,马克·弗格森,凯伦·卡茨,和布丽塔·沙斯贝格.
<a id="S1178"></a> Source: p.72 S1178
Original: The penn treebank: annotating predicate argument structure.
中文: 平生树干:指代上游论证结构.
<a id="S1179"></a> Source: p.72 S1179
Original: In Proceedings of the workshop on Human Language Technology, pages 114–119.
中文: 《人类语言技术讲习班记录》,第114-119页。
<a id="S1180"></a> Source: p.72 S1180
Original: Association for Computational Linguistics, 1994. [MKXS18] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.
中文: 计算语言学协会, 1994. [MKXS18] Bryan McCann, Nitish Shirish Keskar, Caimin Xiong, and Richard Socher.
<a id="S1181"></a> Source: p.72 S1181
Original: The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730, 2018. [MPL19] R Thomas McCoy, Ellie Pavlick, and Tal Linzen.
中文: 自然语言德克思隆:多任务学习作为问答. arXiv preprint arXiv:1806.08730, 2018. [MPL19] (英语). R Thomas McCoy, Ellie Pavlick, and Tal Linzen.
<a id="S1182"></a> Source: p.72 S1182
Original: Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007, 2019. [MWZ+18] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.
中文: 对错原因:在自然语言推论中诊断综合heuristics. arXiv preprint arXiv:1902.01007,2019. [MWZ+18] 玛格丽特·米切尔,西蒙·吴,安德鲁·扎尔迪瓦尔,帕克·巴恩斯,露西·瓦瑟曼,本·哈钦森,埃琳娜·斯皮策尔,伊尼奥卢瓦·德博拉·拉吉和蒂姆尼特·盖布鲁.
<a id="S1183"></a> Source: p.72 S1183
Original: Model cards for model reporting, 2018. [NBR20] Moin Nadeem, Anna Bethke, and Siva Reddy.
中文: 2018年模型报告模式卡. [NBR20] Moin Nadeem,Anna Bethke,和西瓦·雷迪.
<a id="S1184"></a> Source: p.72 S1184
Original: Stereoset: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456, 2020. [NK19] Timothy Niven and Hung-Yu Kao.
中文: Stereoset:测量预训语言模型中的定型偏见. arXiv preprint arXiv:2004.09456,2020. [NK19] 蒂莫西·尼文与洪裕高.
<a id="S1185"></a> Source: p.72 S1185
Original: Probing neural network comprehension of natural language arguments. arXiv preprint arXiv:1907.07355, 2019. [Nor09] Peter Norvig.
中文: 探究神经网络对自然语言辩词的理解. arXiv preprint arXiv:1907.07355, 2019. [Nor09] 彼得·诺维格.
<a id="S1186"></a> Source: p.72 S1186
Original: Natural language corpus data, 2009. [NvNvdG19] Malvina Nissim, Rik van Noord, and Rob van der Goot.
中文: 自然语言体能数据, 2009. [NvNvdG19]. Malvina Nissim, Rik van Noord, and Rob van der Goot.
<a id="S1187"></a> Source: p.72 S1187
Original: Fair is better than sensational: Man is to doctor as woman is to doctor. arXiv preprint arXiv:1905.09866, 2019. [NWD+19] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.
中文: 公平胜过耸人听闻:男人是医生,女人是医生。 arXiv preprint arXiv:1905.09866, 2019. [NWD+19] (英语). Yixin Nee, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela.
<a id="S1188"></a> Source: p.72 S1188
Original: Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599, 2019. [oR16] University of Regensburg.
中文: 逆向nli:自然语言理解的新基准. arXiv preprint arXiv:1910.14599,2019. [oR16] 雷根斯堡大学.
<a id="S1189"></a> Source: p.72 S1189
Original: Fascha, 2016. [PCC18] Mohammad Taher Pilehvar and Jose Camacho-Collados.
中文: Fascha, 2016. [PCC18] (英语). Mohammad Taher Pilehvar and Jose Camacho-Collados.
<a id="S1190"></a> Source: p.72 S1190
Original: WIC: 10,000 example pairs for evaluating context-sensitive representations. arXiv preprint arXiv:1808.09121, 2018. [PFB18] Jason Phang, Thibault Fe´vry, and Samuel R.
中文: WIC:用于评价对上下文敏感的表述的一万对实例. arXiv preprint arXiv:1808.09121, 2018. [PFB18] (英语). Jason Phang, Thibault Fe'vry, and Samuel R.
<a id="S1191"></a> Source: p.72 S1191
Original: Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088, 2018. [PHR+18] Adam Poliak, Aparajita Haldar, Rachel Rudinger, J.
中文: 关于STILT的句子编码器:关于中间标签数据任务的补充培训. arXiv preprint arXiv:1811.01088, 2018. [PHR+18] 亚当·波利亚克, Aparajita Haldar, Rachel Rudinger, J.
<a id="S1192"></a> Source: p.72 S1192
Original: Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme.
中文: 爱德华·胡,艾莉·帕夫利克,亚伦·史蒂文·怀特,和本杰明·范·杜尔梅.
<a id="S1193"></a> Source: p.72 S1193
Original: Collecting diverse natural language inference problems for sentence representation evaluation.
中文: 收集各种自然语言推论问题,用于句子代表评价.
<a id="S1194"></a> Source: p.72 S1194
Original: In Proceedings of EMNLP, 2018. [PKL+16] Denis Paperno, Germa´n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Ferna´ndez.
中文: (原始内容存档于2018-10-21). In Course of EMNLP, 2018. [PKL+16]. Denis Painno, Germa'n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, 和 Raquel Ferna'ndez.
<a id="S1195"></a> Source: p.72 S1195
Original: The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. [PNZtY18] Matthew E.
中文: 兰巴达数据集:需要广泛论述的文字预测. arXiv preprint arXiv:1606.066031, 2016. [PNZtY18] 马修·E.
<a id="S1196"></a> Source: p.72 S1196
Original: Peters, Mark Neumann, Luke Zettlemoyer, and Wen tau Yih.
中文: 彼得斯,马克·诺伊曼,卢克·泽特勒莫耶,和温陶毅.
<a id="S1197"></a> Source: p.72 S1197
Original: Dissecting contextual word embeddings: Architecture and representation, 2018. [Pos18] Matt Post. A call for clarity in reporting BLEU scores. arXiv preprint arXiv:1804.08771, 2018. 72
中文: 分解上下文词嵌入:建筑与代表性, 2018. [Pos18] Matt Post. 要求明确报告低浓铀的得分。 arXiv preprint arXiv:1804.08771, 2018. 72 (英语).
<a id="S1198"></a> Source: p.73 S1198
Original: [PSM14] Jeffrey Pennington, Richard Socher, and Christopher Manning.
中文: [PSM14]杰弗里·彭宁顿,理查德·索彻和克里斯多夫·曼宁.
<a id="S1199"></a> Source: p.73 S1199
Original: GloVe: Global vectors for word representation.
中文: GloVe:用于字面表达的全球矢量.
<a id="S1200"></a> Source: p.73 S1200
Original: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014. [QIA20] QIANXIN.
中文: 2014年自然语言处理经验方法会议纪要 (EMNLP), 2014. [QIA20] QIANXIN.
<a id="S1201"></a> Source: p.73 S1201
Original: Sa-net on albert (ensemble), April 2020. [QMZH19] Yusu Qian, Urwa Muaz, Ben Zhang, and Jae Won Hyun.
中文: Sa-net on albert (Cembermble), April 2020. [QMZH19] 于秀Q,乌尔瓦·穆阿兹,本·张和,贾元玄.
<a id="S1202"></a> Source: p.73 S1202
Original: Reducing gender bias in word-level language models with a gender-equalizing loss function. arXiv preprint arXiv:1905.12801, 2019. [RBG11] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon.
中文: 减少字面语言模型中的性别偏见,同时发挥两性均等的损失功能。 arXiv preprint arXiv:1905.12801, 2019. [RBG11] 梅丽莎·罗姆梅莱,科斯明·阿德里安·贝扬和安德鲁·斯·戈登.
<a id="S1203"></a> Source: p.73 S1203
Original: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.
中文: 合理备选办法的选择:对常识因果关系的评估。
<a id="S1204"></a> Source: p.73 S1204
Original: In 2011 AAAI Spring Symposium Series, 2011. [RCM19] Siva Reddy, Danqi Chen, and Christopher D Manning.
中文: 2011年 AAAI Spring System Series, 2011. [RCM19] (中文(中国大陆) ). Siva Reddy, Danqi Chen, and Christopher D Manning.
<a id="S1205"></a> Source: p.73 S1205
Original: Coqa: A conversational question answering challenge.
中文: Coqa:对话问题回答挑战.
<a id="S1206"></a> Source: p.73 S1206
Original: Transactions of the Association for Computational Linguistics, 7:249–266, 2019. [RCP+17] Scott Reed, Yutian Chen, Thomas Paine, Aa¨ron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas.
中文: 计算语言学协会的交易,7:249–266,2019. [RCP+17] Scott Reed, Yutian Chen, Thomas Paine, Aa'ron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas.
<a id="S1207"></a> Source: p.73 S1207
Original: Few-shot autoregressive density estimation: Towards learning to learn distributions. arXiv preprint arXiv:1710.10304, 2017. [RJL18] Pranav Rajpurkar, Robin Jia, and Percy Liang.
中文: 少发自回的密度估计:向学习去学习分布. arXiv preprint arXiv:1710.10304,2017. [RJL18] (英语). Pranav Rajpurkar, Robin Jia, and Percy Liang.
<a id="S1208"></a> Source: p.73 S1208
Original: Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. [RL16] Sachin Ravi and Hugo Larochelle.
中文: 了解你所不知道的:无法回答的问题。 arXiv preprint arXiv:1806.03822, 2018. [RL16] 萨钦·拉维和雨果·拉罗谢尔.
<a id="S1209"></a> Source: p.73 S1209
Original: Optimization as a model for few-shot learning.
中文: 优化为"几弹学习"模式.
<a id="S1210"></a> Source: p.73 S1210
Original: ICLR 2017 (oral), 2016. [RLL+19] Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu.
中文: ICLR 2017 (Oral), 2016. [RLL+19]. 邱兰,颜克林,彭丽卿,周杰伦,刘志远.
<a id="S1211"></a> Source: p.73 S1211
Original: NumNet: Machine reading comprehension with numerical reasoning.
中文: NumNet:机器读取与数字推理的理解.
<a id="S1212"></a> Source: p.73 S1212
Original: In Proceedings of EMNLP, 2019. [RNLVD18] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.
中文: (原始内容存档于2019-09-22). In Course of EMNLP, 2019. [RNLVD18]. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.
<a id="S1213"></a> Source: p.73 S1213
Original: Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. [RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
中文: 校对解决方案中的性别偏见。 arXiv preprint arXiv:1804.09301, 2018. [RNSS18] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. [RNSS18] 阿列克·拉德福德, 卡尔蒂克·纳拉西曼, 蒂姆·萨利曼斯, 伊利亚·萨特斯克韦.
<a id="S1214"></a> Source: p.73 S1214
Original: Improving language understanding by generative pre-training, 2018. [Ros12] R.S.
中文: 通过基因预训提高语言理解, 2018. [Ros12] R.S.
<a id="S1215"></a> Source: p.73 S1215
Original: NIST Special Publication, 2012. [RRBS19] Jonathan S.
中文: NIST Special Publications, 2012. [RRBS19] 乔纳森·S.
<a id="S1216"></a> Source: p.73 S1216
Original: Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales, 2019. [RRS20] Adam Roberts, Colin Raffel, and Noam Shazeer.
中文: 罗森费尔德,埃米尔·罗森费尔德,约纳坦·贝林科夫,和尼尔·沙维特. 2019年跨平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平面平
<a id="S1217"></a> Source: p.73 S1217
Original: How much knowledge can you pack into the parameters of a language model? arXiv preprint arXiv:2002.08910, 2020. [RSR+19] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J.
中文: 语言模型的参数可以包含多少知识? arXiv preprint arXiv:2002.08910,2020. [RSR+19]. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, 和 Peter J.
<a id="S1218"></a> Source: p.73 S1218
Original: Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. [RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
中文: 探索使用统一文本转换器进行转会学习的限制, 2019. [RWC+19] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 和 Ilya Sutskever.
<a id="S1219"></a> Source: p.73 S1219
Original: Language models are unsupervised multitask learners, 2019. [SBBC19] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
中文: 语言模型是无监督的多任务学习者,2019. [SBBC19] 坂口惠介,罗南·勒·布拉斯,钱德拉·巴加瓦图拉,和叶珍·崔.
<a id="S1220"></a> Source: p.73 S1220
Original: Winogrande: An adversarial winograd schema challenge at scale, 2019. [SBC+19] Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang.
中文: 维诺格兰德:一场规模规模的对抗性花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花筒式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花瓶式花
<a id="S1221"></a> Source: p.73 S1221
Original: Release strategies and the social impacts of language models, 2019. [SCNP19] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng.
中文: 2019年发布策略与语言模型的社会影响. [SCNP19] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan,和南云鹏.
<a id="S1222"></a> Source: p.73 S1222
Original: The woman worked as a babysitter: On biases in language generation. arXiv preprint arXiv:1909.01326, 2019. [SDCW19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
中文: 该妇女担任保姆:在语言生成方面的偏见。 arXiv preprint arXiv:1909.0326, 2019. [SDCW19] (英语). Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
<a id="S1223"></a> Source: p.73 S1223
Original: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. [SDSE19] Roy Schwartz, Jesse Dodge, Noah A.
中文: DistilBERT是BERT的一个蒸馏版:更小更快更便宜更轻. arXiv preprint arXiv:1910.01108, 2019. [SDSE19] (英语). Roy Schwartz, Jesse Dodge, Noah A.
<a id="S1224"></a> Source: p.73 S1224
Original: CoRR, abs/1907.10597, 2019. [SHB15] Rico Sennrich, Barry Haddow, and Alexandra Birch.
中文: CoRR, abs/1907.10597, 2019. [SHB15] (英语). Rico Sennrich, Barry Haddow, and Alexandra Birch.
<a id="S1225"></a> Source: p.73 S1225
Original: Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709, 2015. 73
中文: 以单一语言数据改进神经机能翻译模型. arXiv预印 arXiv:15:11.06709,2015. 73 (英语).
<a id="S1226"></a> Source: p.74 S1226
Original: [SMM+17] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.
中文: [SMM+17]诺姆·沙泽尔,阿扎莉娅·米尔霍塞尼,克日什托夫·马齐亚尔兹,安迪·戴维斯,克克·勒,杰弗里·欣通,和杰夫·迪恩.
<a id="S1227"></a> Source: p.74 S1227
Original: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [SPP+19] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
中文: 奇特的大型神经网络:稀有的由专家组成的混合层. arXiv preprint arXiv:1701.06538, 2017. [SPP+19] (英语). Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, 和 Bryan Catanzaro.
<a id="S1228"></a> Source: p.74 S1228
Original: Megatron-lm: Training multi-billion parameter language models using model parallelism, 2019. [SS20] Timo Schick and Hinrich Schu¨tze.
中文: 威震天-lm:使用模型并行论训练了数十亿参数语言模型,2019. [SS20] Timo Schick and Hinrich Schu'tze.
<a id="S1229"></a> Source: p.74 S1229
Original: Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint arXiv:2001.07676, 2020. [STQ+19] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu.
中文: 利用杂乱无章的问题来进行几发文字分类和自然语言推论. arXiv preprint arXiv:2001.07676,2020. [STQ+19] (中文(中国大陆) ). Kaitao Song, Xu Tan, Tao Cin, Jianfeng Lu, and Tie-Yan Liu.
<a id="S1230"></a> Source: p.74 S1230
Original: MASS: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450, 2019. [TFR+17] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel.
中文: MASS:为语言生成进行序列预训而蒙面序列. arXiv preprint arXiv:1905.02450, 2019. [TFR+17]. 乔什·托宾,瑞秋·方克,亚历克斯·雷,约纳斯·施奈德,沃克西克·扎雷姆巴,和彼得·阿比埃尔.
<a id="S1231"></a> Source: p.74 S1231
Original: Domain randomization for transferring deep neural networks from simulation to the real world.
中文: 用于将深层神经网络从模拟转移到现实世界的域随机化.
<a id="S1232"></a> Source: p.74 S1232
Original: In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30.
中文: 2017年IEEE/RSJ智能机器人与系统国际会议 (IROS), 第23–30页.
<a id="S1233"></a> Source: p.74 S1233
Original: Corpus-based learning of analogies and semantic relations.
中文: 以公司为基础学习类比和语义关系.
<a id="S1234"></a> Source: p.74 S1234
Original: CoRR, abs/cs/0508103, 2005. [TL18] Trieu H.
中文: CoRR, abs/cs/0508103, 2005. [TL18] 三叶花 H.
<a id="S1235"></a> Source: p.74 S1235
Original: Le. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847, 2018. [TLBS03] Peter D.
中文: Le. 一个简单的常识推理方法. arXiv preprint arXiv:1806.02847, 2018. [TLBS03] 彼得·D.
<a id="S1236"></a> Source: p.74 S1236
Original: Littman, Jeffrey Bigham, and Victor Shnayder.
中文: 利特曼 杰弗里·比格姆 维克多·斯奈德
<a id="S1237"></a> Source: p.74 S1237
Original: Combining independent modules to solve multiple-choice synonym and analogy problems.
中文: 将独立的模块结合起来,以解决多选择同义词和类比问题.
<a id="S1238"></a> Source: p.74 S1238
Original: CoRR, cs.CL/0309035, 2003. [Tur20] Project Turing.
中文: CoRR, cs.CL/0309035, 2003. [Tur20] 工程图灵.
<a id="S1239"></a> Source: p.74 S1239
Original: Microsoft research blog, Feb 2020. [VBL+16] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.
中文: 微软研究博客, 二月 2020. [VBL+16] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra等.
<a id="S1240"></a> Source: p.74 S1240
Original: Matching Networks for One Shot Learning.
中文: (原始内容存档于2018-10-21). Matching Networks for One Shot Learning.
<a id="S1241"></a> Source: p.74 S1241
Original: In Advances in neural information processing systems, pages 3630–3638, 2016. [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N.
中文: 神经信息处理系统的进步, 第3630–3638, 2016页. [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N.
<a id="S1242"></a> Source: p.74 S1242
Original: Gomez, Łukasz Kaiser, and Illia Polosukhin.
中文: (原始内容存档于2018-10-21) (英语). Gomez, Łukasz Kaiser, and Illia Polosukhin.
<a id="S1243"></a> Source: p.74 S1243
Original: In Advances in neural information processing systems, 2017. [WPN+19] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.
中文: In Advances in 神经信息处理系统, 2017. [WPN+19]. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, 朱利安·迈克尔, Felix Hill, Omer Levy, 和塞缪尔·鲍曼.
<a id="S1244"></a> Source: p.74 S1244
Original: Superglue: A stickier benchmark for general-purpose language understanding systems.
中文: Superglue:通用语言理解系统的粘接基准.
<a id="S1245"></a> Source: p.74 S1245
Original: In Advances in Neural Information Processing Systems, pages 3261–3275, 2019. [WXH+18] Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu.
中文: In Advances in Neural Information Systems, pages 3261–3275, 2019. [WXH+18] (中文(简体) ). Yiren Wang, Yingce Xia, Tianyu He, Fei Tian, Tao Cin, Cheng Xiang Zhai, and Tie-Yan Liu.
<a id="S1246"></a> Source: p.74 S1246
Original: ICLR 2019, 2018. [XDH+19] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V.
中文: ICLR 2019, 2018. [XDH+19] (中文(中国大陆) ). 齐泽谢,齐相戴,爱德华·霍维,明-通-通-罗通,和克.
<a id="S1247"></a> Source: p.74 S1247
Original: Unsupervised data augmentation for consistency training, 2019. [YdC+19] Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, et al.
中文: 2019年一致性培训无监督数据增强. [YdC+19] Dani Yogatama, Cyprien de Masson d'Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer等.
<a id="S1248"></a> Source: p.74 S1248
Original: Learning and evaluating general linguistic intelligence. arXiv preprint arXiv:1901.11373, 2019. [YDY+19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V.
中文: 学习和评价一般语言智能。 arXiv preprint arXiv:1901.11373, 2019. [YDY+19] (中文(中国大陆) ). [YDY+19] (简体). ). 杨志林, 齐相发, 杨一明, 海梅·卡波内尔, 罗斯兰·萨拉克胡特迪诺夫,和克克.
<a id="S1249"></a> Source: p.74 S1249
Original: XLNet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019. [ZHB+19] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
中文: XLNet:通用自旋预训以通晓语言. arXiv preprint arXiv:1906.08237, 2019. [ZHB+19] 罗文·泽勒斯,阿里·霍尔茨曼,约纳坦·比斯克,阿里·法哈迪和叶珍·崔.
<a id="S1250"></a> Source: p.74 S1250
Original: Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [ZHR+19] Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi.
中文: Hellaswag:机器真的能完成你的句子吗? arXiv preprint arXiv:1905.07830, 2019. [ZHR+19] 罗文·泽勒斯, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner,和叶珍·崔.
<a id="S1251"></a> Source: p.74 S1251
Original: Defending against neural fake news. arXiv preprint arXiv:1905.12616, 2019. [ZLL+18] Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme.
中文: 防范神经假新闻. arXiv preprint arXiv:1905.12616, 2019. [ZLL+18] (中文(中国大陆) ). 沈翔,刘晓东,刘静京,高建丰,凯文·杜克和本杰明·范·杜尔梅.
<a id="S1252"></a> Source: p.74 S1252
Original: ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885, 2018. [ZSW+19a] Daniel M.
中文: ReCORD:缩小人与机器常识阅读理解的差距. arXiv preprint arXiv:1810.12885, 2018. [ZSW+19a] (中文(中国大陆) ). 丹尼尔·M.
<a id="S1253"></a> Source: p.74 S1253
Original: Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B.
中文: 齐格勒,尼桑·斯蒂安农,杰弗里·吴,汤姆·B.
<a id="S1254"></a> Source: p.74 S1254
Original: Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.
中文: 布朗,阿莱克·拉德福德,达里奥·阿莫代,保罗·克里斯蒂安诺,和杰弗里·欧文.
<a id="S1255"></a> Source: p.74 S1255
Original: Fine-tuning language models from human preferences, 2019. 74
中文: 2019年从人类喜好出发精调语言模式. 74.
<a id="S1256"></a> Source: p.75 S1256
Original: Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B.
中文: 齐格勒,尼桑·斯蒂安农,杰弗里·吴,汤姆·B.
<a id="S1257"></a> Source: p.75 S1257
Original: Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving.
中文: 布朗,阿莱克·拉德福德,达里奥·阿莫代,保罗·克里斯蒂安诺,和杰弗里·欧文.
<a id="S1258"></a> Source: p.75 S1258
Original: Fine-tuning language models from human preferences.
中文: 从人类喜好中精细地调整出语言模式.