BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language - 中英文对照
translated: 2026-07-16
title: "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language" aliases: - "BERT" - "arXiv:1810.04805" source: "https://arxiv.org/abs/1810.04805" arxiv: "1810.04805" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning - nlp
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language {jacobdevlin,mingweichang,kentonl,kristout}@google.com Abstract There are two existing strategies for applying pre-trained language representations to down- We introduce a new language representastream tasks: feature-based and fine-tuning.
中文: BERT:深双向变换器预训 语言理解 Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language {jacobdevlin, mingweichang, kentonl, kristout}@-google.com 互联网档案馆的存檔,存档日期2013-12-22. 摘要 现有两项战略是应用经过预先培训的语文表述来降低 -- 我们引入了一种新的语言代表流任务:基于特征和微调.
<a id="S0002"></a> Source: p.1 S0002
Original: The tion model called BERT, which stands for feature-based approach, such as ELMo (Peters Bidirectional Encoder Representations from Transformers.
中文: 称为BERT的TV模型,代表了基于地物的方法,如ELMo(从变形器得到的Peters双向编码器代表).
<a id="S0003"></a> Source: p.1 S0003
Original: Unlike recent language repre- et al., 2018a), uses task-specific architectures that sentation models (Peters et al., 2018a; Rad- include the pre-trained representations as addiford et al., 2018), BERT is designed to pre- tional features.
中文: 与近期语言 repre-et al.,2018a不同,BERT采用了任务特有架构来发送模型(Peters等,2018a;Rad-包括了经过预训的表示作为Addiford等,2018年),BERT的设计是为了预取通篇特征.
<a id="S0004"></a> Source: p.1 S0004
Original: The fine-tuning approach, such as train deep bidirectional representations from the Generative Pre-trained Transformer (OpenAI unlabeled text by jointly conditioning on both GPT) (Radford et al., 2018), introduces minimal left and right context in all layers.
中文: 微调方法,如从Generative Pre-trained Transformer(OpenAI unlabeled text by both GBT (Radford et al., 2018))中训练出深度双向表示法,在所有层面引入了最小左右上下文.
<a id="S0005"></a> Source: p.1 S0005
Original: As a retask-specific parameters, and is trained on the sult, the pre-trained BERT model can be finedownstream tasks by simply fine-tuning all pretuned with just one additional output layer trained parameters.
中文: 作为一种针对再任务的特定参数,并被训练于sult上,被预先训练的BERT模型可以通过简单的微调所有预调,仅用一个额外的输出层被训练出参数即可成为精细下游任务.
<a id="S0006"></a> Source: p.1 S0006
Original: The two approaches share the to create state-of-the-art models for a wide range of tasks, such as question answering and same objective function during pre-training, where language inference, without substantial task- they use unidirectional language models to learn specific architecture modifications. general language representations.
中文: 这两种方法共同为一系列广泛的任务创建最先进的模型,比如在前期训练期间的问答和相同的客观功能,语言推断,没有实质性的任务——它们使用单向语言模型来学习具体的架构修改. 一般语言表述。
<a id="S0007"></a> Source: p.1 S0007
Original: BERT is conceptually simple and empirically We argue that current techniques restrict the powerful.
中文: BERT在概念上简单,经验上 我们认为,目前的技术限制了强者。
<a id="S0008"></a> Source: p.1 S0008
Original: It obtains new state-of-the-art re- power of the pre-trained representations, espesults on eleven natural language processing cially for the fine-tuning approaches.
中文: 它获得了经过预先训练的表达方式,即十一种自然语言处理方式的新的最新再现能力,用于微调方法.
<a id="S0009"></a> Source: p.1 S0009
Original: The matasks, including pushing the GLUE score to jor limitation is that standard language models are 80.5% (7.7% point absolute improvement), unidirectional, and this limits the choice of archi- MultiNLI accuracy to 86.7% (4.6% absolute tectures that can be used during pre-training.
中文: matask,包括将GLUE分数推向jor限制,是标准语言模型为80.5%(7.7%的点绝对改进),单向,这限制了Arti-MultiNLI精度的选择为86.7%(4.6%的绝对构造可以用于预训.
<a id="S0010"></a> Source: p.1 S0010
Original: For improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute im- example, in OpenAI GPT, the authors use a left-toprovement) and SQuAD v2.0 Test F1 to 83.1 right architecture, where every token can only at- (5.1 point absolute improvement). tend to previous tokens in the self-attention layers of the Transformer (Vaswani et al., 2017).
中文: 为改进起见,SQuAD v1.1问题回答测试 F1至93.2(1.5分绝对im-例,在OpenAI GPT中,作者使用左对接)和SQuAD v2.0 测试 F1至83.1 右架构,每个令牌只能到-(5.1分绝对改进). 倾向于"变形金刚"自发地层中的前作符号(Vaswani等,2017).
<a id="S0011"></a> Source: p.1 S0011
Original: Such re- 1 Introduction strictions are sub-optimal for sentence-level tasks, Language model pre-training has been shown to and could be very harmful when applying finebe effective for improving many natural language tuning based approaches to token-level tasks such processing tasks (Dai and Le, 2015; Peters et al., as question answering, where it is crucial to incor- 2018a; Radford et al., 2018; Howard and Ruder, porate context from both directions. 2018).
中文: 此类再版 1 入门约束对于句子级任务来说是次优的入门约束,语言模型预训已经显示出来,在应用微调有效改进许多基于自然语言的调整方法来完成以符号级任务时可能非常有害(Dai和Le,2015;Peters等作为问题回答,对于2018a的入门至关重要;Radford等,2018;Howard和Ruder, 双向上下文。 2018 (英语).
<a id="S0012"></a> Source: p.1 S0012
Original: These include sentence-level tasks such as In this paper, we improve the fine-tuning based natural language inference (Bowman et al., 2015; approaches by proposing BERT: Bidirectional Williams et al., 2018) and paraphrasing (Dolan Encoder Representations from Transformers. and Brockett, 2005), which aim to predict the re- BERT alleviates the previously mentioned unidilationships between sentences by analyzing them rectionality constraint by using a “masked lanholistically, as well as token-level tasks such as guage model” (MLM) pre-training objective, innamed entity recognition and question answering, spired by the Cloze task (Taylor, 1953).
中文: 其中包括句子层面的任务,如在本文中,我们改进了基于微调的自然语言推论(Bowman等,2015年;通过提出BERT: Biofled Williams等,2018年)和参数化(Dolan Encoder Representations from Transformers)的方法. 后者旨在预测再培训BERT, 2005年),它通过使用“假冒的单词以及象征性的任务,如Gugage模型(MLM)培训前目标,在克洛兹任务(Taylor,1953年)的激励下,以命名实体识别和回答问题的方式,分析这些句子的回旋性限制,从而缓解了上述两句之间的分歧。
<a id="S0013"></a> Source: p.1 S0013
Original: The where models are required to produce fine-grained masked language model randomly masks some of output at the token level (Tjong Kim Sang and the tokens from the input, and the objective is to De Meulder, 2003; Rajpurkar et al., 2016). predict the original vocabulary id of the masked 9102 yaM 42 ]LC.sc[ 2v50840.0181:viXra
中文: 需要模型来生产精致的口罩语言模型随机地遮住一些符号级的输出(Tjong Kim Sang和输入的符号,目标是De Meulder, 2003; Rajpurkar等, 2016). 预测被遮掩的9102 yaM 42)的原始词汇id.LC.sc[2v50840.0181:viXra]
<a id="S0014"></a> Source: p.2 S0014
Original: Unlike left-to- These approaches have been generalized to right language model pre-training, the MLM ob- coarser granularities, such as sentence embedjective enables the representation to fuse the left dings (Kiros et al., 2015; Logeswaran and Lee, and the right context, which allows us to pre- 2018) or paragraph embeddings (Le and Mikolov, train a deep bidirectional Transformer.
中文: 不像左转 这些方法被泛指为右语言模型预训,MLM相接相接颗粒法,如句子嵌入能使代表能将左起子(Kiros等,2015;Logeswaran和Lee,以及右上下文能让我们在2018年之前被嵌入)或段落嵌入法(Le和Mikolov,训练深双向变换器.
<a id="S0015"></a> Source: p.2 S0015
Original: To train sentence representations, prior tion to the masked language model, we also use work has used objectives to rank candidate next a “next sentence prediction” task that jointly pre- sentences (Jernite et al., 2017; Logeswaran and trains text-pair representations.
中文: 为了训练句子表达方式,在使用蒙面语言模式之前,我们还利用工作目标,将候选人排入下一个“下个句子预测”任务,即共同进行句子前期陈述(Jernite等,2017年;Logeswaran并培训文本页表达方式)。
<a id="S0016"></a> Source: p.2 S0016
Original: The contributions Lee, 2018), left-to-right generation of next senof our paper are as follows: tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto- • We demonstrate the importance of bidirectional encoder derived objectives (Hill et al., 2016). pre-training for language representations.
中文: 作者 Lee, 2018),左到右一代 我们的论文的下个 senof 如下: Tence 单词代表了上一句(Kiros et al., 2015),或denoising auto - • 我们证明了双向编码器衍生目标的重要性(Hill等人,2016年)。 语文表述培训。
<a id="S0017"></a> Source: p.2 S0017
Original: Un- ELMo and its predecessor (Peters et al., 2017, like Radford et al. (2018), which uses unidirec- 2018a) generalize traditional word embedding retional language models for pre-training, BERT search along a different dimension.
中文: Un-ELMo及其前身 (Peters等, 2017年, 如Radford等 (2018年), 使用unidirec- 2018a) 将传统单词嵌入回旋语言模型概括为预训, BERT 沿着不同的维度搜索.
<a id="S0018"></a> Source: p.2 S0018
Original: They extract uses masked language models to enable precontext-sensitive features from a left-to-right and a trained deep bidirectional representations.
中文: 其取出使用蒙面语言模型,使得从左到右的预结语敏感特征和经过训练的深双向表达功能成为可能.
<a id="S0019"></a> Source: p.2 S0019
Original: The contextual repis also in contrast to Peters et al. (2018a), which resentation of each token is the concatenation of uses a shallow concatenation of independently the left-to-right and right-to-left representations. trained left-to-right and right-to-left LMs.
中文: 与Peters等人(2018年a)相对的是背景描述,后者对每个符号都表示不满,即使用由左到右和右到左分别独立表达的浅接来表示。 训练出左右相接的LM.
<a id="S0020"></a> Source: p.2 S0020
Original: When integrating contextual word embeddings • We show that pre-trained representations reduce with existing task-specific architectures, ELMo the need for many heavily-engineered task- advances the state of the art for several major NLP specific architectures.
中文: 在整合上下文嵌入词时,我们显示,经过预先训练的表示与现有的特定任务架构相比有所减少,ELMo需要许多经过大量设计的任务,为几个主要的NLP特定架构推进了艺术状态.
<a id="S0021"></a> Source: p.2 S0021
Original: BERT is the first fine- benchmarks (Peters et al., 2018a) including questuning based representation model that achieves tion answering (Rajpurkar et al., 2016), sentiment state-of-the-art performance on a large suite analysis (Socher et al., 2013), and named entity of sentence-level and token-level tasks, outper- recognition (Tjong Kim Sang and De Meulder, forming many task-specific architectures. 2003).
中文: BERT是最早的精细基准(Peters等,2018年a),包括实现tion应答(Rajpurkar等,2016年)的以追求为基础的代表模式,大型套接字分析(Socher等,2013年)的情感状态性能,并被命名为句子级别和符号级别任务的实体,超越识别(Tjong Kim Sang和De Meulder,形成许多任务特定架构. 2003年,中国出版。
<a id="S0022"></a> Source: p.2 S0022
Original: Melamud et al. (2016) proposed learning contextual representations through a task to pre- • BERT advances the state of the art for eleven dict a single word from both left and right context NLP tasks.
中文: Melamud等(2016年)提议通过一项任务来学习背景表述——BERT从左上下文和右上下文任务中推进11个单词的艺术状态。
<a id="S0023"></a> Source: p.2 S0023
Original: The code and pre-trained modusing LSTMs.
中文: 代码和预训操作 LSTMs.
<a id="S0024"></a> Source: p.2 S0024
Original: Similar to ELMo, their model is els are available at https://github.com/ feature-based and not deeply bidirectional.
中文: 与ELMo类似,其型号为els,可在https://github.com/ special-based,而非深度双向.
<a id="S0025"></a> Source: p.2 S0025
Original: Fedus google-research/bert. et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod- 2 Related Work els.
中文: Fedus google-research/bert.等 (2018)显示,可以使用clze任务来提高文本生成mod-2相關Work els的稳健性.
<a id="S0026"></a> Source: p.2 S0026
Original: There is a long history of pre-training general language representations, and we briefly review the 2.2 Unsupervised Fine-tuning Approaches most widely-used approaches in this section.
中文: 培训前通用语言表述有很长的历史,我们简要地回顾本节中最广泛使用的2.2个未经监督的微调方法。
<a id="S0027"></a> Source: p.2 S0027
Original: As with the feature-based approaches, the first 2.1 Unsupervised Feature-based Approaches works in this direction only pre-trained word em- Learning widely applicable representations of bedding parameters from unlabeled text (Colwords has been an active area of research for lobert and Weston, 2008). decades, including non-neural (Brown et al., 1992; More recently, sentence or document encoders Ando and Zhang, 2005; Blitzer et al., 2006) and which produce contextual token representations neural (Mikolov et al., 2013; Pennington et al., have been pre-trained from unlabeled text and 2014) methods.
中文: 与基于特征的方法一样,首个 2.1 不受监督的基于特征的方法只在这方面进行预先培训的词em- 从无标签的文本中学习广泛适用的被褥参数说明(Colwords一直是lobert和Weston的一个积极研究领域,2008年)。 数十年,包括非神经病(Brown等人,1992年;最近,句子或文件编码器Ando和Zhang,2005年;Blitzer等人,2006年),这些神经病产生上下文符号表示法(Mikolov等人,2013年;Pennington等人,2013年;Pennington等人,从未贴标签的文字和2014年的方法中预先接受了培训。
<a id="S0028"></a> Source: p.2 S0028
Original: Pre-trained word embeddings fine-tuned for a supervised downstream task (Dai are an integral part of modern NLP systems, of- and Le, 2015; Howard and Ruder, 2018; Radford fering significant improvements over embeddings et al., 2018).
中文: 预先训练过的词嵌入被监督的下游任务经过了微调(Dai是现代NLP系统的一个组成部分,of-和Le,2015;Howard和Ruder,2018;Radford ferring firing over 嵌入等,2018).
<a id="S0029"></a> Source: p.2 S0029
Original: The advantage of these approaches learned from scratch (Turian et al., 2010).
中文: 这些方法的优点是从零开始的(Turian等人,2010年)。
<a id="S0030"></a> Source: p.2 S0030
Original: To pre- is that few parameters need to be learned from train word embedding vectors, left-to-right lan- scratch.
中文: 要预-,需要从列车单词嵌入向量,从左到右的lan-抓取中学习的参数很少.
<a id="S0031"></a> Source: p.2 S0031
Original: At least partly due to this advantage, guage modeling objectives have been used (Mnih OpenAI GPT (Radford et al., 2018) achieved preand Hinton, 2009), as well as objectives to dis- viously state-of-the-art results on many sentencecriminate correct from incorrect words in left and level tasks from the GLUE benchmark (Wang right context (Mikolov et al., 2013). et al., 2018a).
中文: 至少有部分由于这一优势,使用了Guage模型目标(Mnih OpenAI GPT(Radford等,2018年)实现事前和Hinton,2009年),以及从GLUE基准(Wang Right上下文(Mikolov等,2013年)等,2018年a)中从左上下级任务错误中得出许多错误的句子最新结果的目标)。
<a id="S0032"></a> Source: p.3 S0032
Original: NSP Mask LM Mask LM MNLI NER SQuAD Start/End Span C T ... T T T ’ ... T ’ C T ... T T T ’ ... T ’ 1 N [SEP] 1 M 1 N [SEP] 1 M BERT BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ [CLS] Tok 1 ...
中文: NSP Mask LM Mask LM MNLi NER SQuAD起步/结束 Span C T. T. T. T. T. T. T. N. [SEP] 1 M. N. [SEP] 1 M BERT BERT BERT E. [CLS] E. [SEP] 1 [CLS] Tok 1.
<a id="S0033"></a> Source: p.3 S0033
Original: TokM Masked Sentence A Masked Sentence B Question Paragraph Unlabeled Sentence A and B Pair Question Answer Pair Pre-training Fine-Tuning Figure 1: Overall pre-training and fine-tuning procedures for BERT.
中文: TokM 蒙面刑 A 蒙面刑 B 问题段落未标注的句子 A 和 B 对等问题回答 平分前训练 罚款 图1:BERT的总体预训和微调程序.
<a id="S0034"></a> Source: p.3 S0034
Original: Apart from output layers, the same architectures are used in both pre-training and fine-tuning.
中文: 除了输出层外,同样的架构也被用于预训和微调.
<a id="S0035"></a> Source: p.3 S0035
Original: The same pre-trained model parameters are used to initialize models for different down-stream tasks.
中文: 同样的预先训练的模型参数被用来初始化不同下游任务的模型.
<a id="S0036"></a> Source: p.3 S0036
Original: During fine-tuning, all parameters are fine-tuned. [CLS] is a special symbol added in front of every input example, and [SEP] is a special separator token (e.g. separating questions/answers). ing and auto-encoder objectives have been used mal difference between the pre-trained architecfor pre-training such models (Howard and Ruder, ture and the final downstream architecture. 2018; Radford et al., 2018; Dai and Le, 2015).
中文: 在微调期间,所有参数都会被微调. [CLS]是每个输入例前所添加的特殊符号,而[SEP]是特殊的分隔符符(例如分离问答). 预先训练的此类模型(Howard和Ruder、ture和最后下游建筑)之间的差别不大。 2018; Radford等, 2018; Dai and Le, 2015).
<a id="S0037"></a> Source: p.3 S0037
Original: Model Architecture BERT’s model architec- 2.3 Transfer Learning from Supervised Data ture is a multi-layer bidirectional Transformer encoder based on the original implementation de- There has also been work showing effective transscribed in Vaswani et al. (2017) and released in fer from supervised tasks with large datasets, such the tensor2tensor library.1 Because the use as natural language inference (Conneau et al., of Transformers has become common and our im- 2017) and machine translation (McCann et al., plementation is almost identical to the original, 2017).
中文: Model Architecture BERT的模型Architec - 2.3 从监督数据图中学习传输是多层双向变形器编码器,基于原始执行de - 还有一些工作显示,在Vaswani等人(2017年)中作了有效的转录,并用大型数据集,如 " lator2tensor " 图书馆,1 从监督任务中释放出来。 因为"变形金刚"(Conneau等,"变形金刚"等)和"机器翻译"(McCann等,"平分"与"2017"的原作几乎完全相同)作为自然语言推论(Conneau等,"变形金刚"已经很常见了,而我们的"im-2017").
<a id="S0038"></a> Source: p.3 S0038
Original: Computer vision research has also demonwe will omit an exhaustive background descripstrated the importance of transfer learning from tion of the model architecture and refer readers to large pre-trained models, where an effective recipe Vaswani et al. (2017) as well as excellent guides is to fine-tune models pre-trained with Imasuch as “The Annotated Transformer.”2 geNet (Deng et al., 2009; Yosinski et al., 2014).
中文: 计算机视觉研究也让We忽略了一个详尽无遗的背景,抹去从模型结构中学习的重要性,并将读者引荐到经过预先培训的大型模型,其中有效的食谱Vaswani等人(2017年)以及出色的指南是微调经过Ima(例如“附加说明的变形器”)2 的模型(Deng等人,2009年;Yosinski等人,2014年)。
<a id="S0039"></a> Source: p.3 S0039
Original: In this work, we denote the number of layers 3 BERT (i.e., Transformer blocks) as L, the hidden size as H, and the number of self-attention heads as A.3 We introduce BERT and its detailed implementa- We primarily report results on two model sizes: tion in this section.
中文: 在这部作品中,我们表示第3层BERT(即变形板块)为L,隐藏大小为H,自念头为A.3. 我们介绍BERT及其详细执行a 我们主要报告两个模型规模的结果:在本节中Tion。
<a id="S0040"></a> Source: p.3 S0040
Original: There are two steps in our BERT (L=12, H=768, A=12, Total Param- BASE framework: pre-training and fine-tuning.
中文: 我们的BERT(L=12,H=768,A=12,Total Param-BASE框架)有两个步骤:预训和微调.
<a id="S0041"></a> Source: p.3 S0041
Original: Dur- eters=110M) and BERT (L=24, H=1024, LARGE ing pre-training, the model is trained on unlabeled A=16, Total Parameters=340M). data over different pre-training tasks.
中文: Dur-eters=110M)和BERT(L=24,H=1024,LARGE ing预训,该型号在无标签的A=16,Total参数=340M上进行了训练. 不同培训前任务的数据。
<a id="S0042"></a> Source: p.3 S0042
Original: For fine- BERT was chosen to have the same model BASE tuning, the BERT model is first initialized with size as OpenAI GPT for comparison purposes. the pre-trained parameters, and all of the param- Critically, however, the BERT Transformer uses eters are fine-tuned using labeled data from the bidirectional self-attention, while the GPT Transdownstream tasks.
中文: 对于fine-BERT被选用相同的型号BASE调音,BERT模式首先以大小为OpenAI GPT而初始化,以作比较. 训练前的参数, 和所有参数... 然而,关键的是,BERT Transformer使用eters,使用双向自意的标注数据进行微调,而GPT跨下游任务.
<a id="S0043"></a> Source: p.3 S0043
Original: Each downstream task has sepformer uses constrained self-attention where every arate fine-tuned models, even though they are ini- token can only attend to context to its left.4 tialized with the same pre-trained parameters.
中文: 每个下游任务都有Sepfors使用受限制的自觉,每个速率微调的模型,即使它们是ini-象征,只能注意左侧上下文. 4以相同的预训参数拨号.
<a id="S0044"></a> Source: p.3 S0044
Original: The question-answering example in Figure 1 will serve 1https://github.com/tensorflow/tensor2tensor as a running example for this section. 2http://nlp.seas.harvard.edu/2018/04/03/attention.html 3In all cases we set the feed-forward/filter size to be 4H, A distinctive feature of BERT is its unified ari.e., 3072 for the H = 768 and 4096 for the H = 1024. chitecture across different tasks.
中文: 图1中的答题例子将作为本节的运行示例。 2 http://nlp.seas.harvard.edu/2018/04/03/atenty.html (中文(简体) ). 3 在所有情况下,我们规定向导/过滤器的尺寸为4H,BERT的一个显著特征是其统一的ari.,即:H=768;H=1024为4096。 穿梭在不同的任务上
<a id="S0045"></a> Source: p.3 S0045
Original: There is mini- 4We note that in the literature the bidirectional Trans-
中文: 我们注意到,在文献中,双向跨
<a id="S0046"></a> Source: p.4 S0046
Original: Input/Output Representations To make BERT In order to train a deep bidirectional representahandle a variety of down-stream tasks, our input tion, we simply mask some percentage of the input representation is able to unambiguously represent tokens at random, and then predict those masked both a single sentence and a pair of sentences tokens.
中文: 投入/产出 使贝特 为了训练出深双向代表 各种下流任务,我们的输入tion,我们只是遮掩了一定比例的输入表示能够随机明确表示符号,然后预测那些被遮掩的既能单句又能一对句子表示符号.
<a id="S0047"></a> Source: p.4 S0047
Original: We refer to this procedure as a “masked (e.g., (cid:104) Question, Answer (cid:105)) in one token sequence.
中文: 我们把这一程序称为“一个象征性的顺序(例如(cid:104)问答(cid:105))。
<a id="S0048"></a> Source: p.4 S0048
Original: LM” (MLM), although it is often referred to as a Throughout this work, a “sentence” can be an arbi- Cloze task in the literature (Taylor, 1953).
中文: LM”(MLM),虽然在这项工作中经常被称为“判决”,但“判决”在文献中可能是一种仲裁-克洛兹任务(Taylor,1953年)。
<a id="S0049"></a> Source: p.4 S0049
Original: In this trary span of contiguous text, rather than an actual case, the final hidden vectors corresponding to the linguistic sentence. A “sequence” refers to the in- mask tokens are fed into an output softmax over put token sequence to BERT, which may be a sin- the vocabulary, as in a standard LM.
中文: 在这个相接文字的三长段中,而不是一个实际的情况,最终隐藏的向量对应语言句子. “序列”是指将口罩中的活字符号输入输出软max,而不是将活字符号序列放入BERT, 这可能是一种罪恶—— 词汇,如标准LM。
<a id="S0050"></a> Source: p.4 S0050
Original: In all of our gle sentence or two sentences packed together. experiments, we mask 15% of all WordPiece to- We use WordPiece embeddings (Wu et al., kens in each sequence at random.
中文: 在我们所有的gle句子 或两句集合在一起。 我们用WordPiece嵌入法(Wu等人,每个序列的肯斯随机) 来掩盖所有WordPiece的15%。
<a id="S0051"></a> Source: p.4 S0051
Original: In contrast to 2016) with a 30,000 token vocabulary.
中文: 与2016年形成对比),有3万个符号词汇.
<a id="S0052"></a> Source: p.4 S0052
Original: The first denoising auto-encoders (Vincent et al., 2008), we token of every sequence is always a special clas- only predict the masked words rather than reconsification token ([CLS]).
中文: 第一个去诺的自动编码器(Vincent等,2008年),我们每个序列的符号总是一个特殊的clas-只预测被遮蔽的单词,而不是再编码符([CLS]).
<a id="S0053"></a> Source: p.4 S0053
Original: The final hidden state structing the entire input. corresponding to this token is used as the ag- Although this allows us to obtain a bidirecgregate sequence representation for classification tional pre-trained model, a downside is that we tasks.
中文: 最终隐藏状态构建了全部输入. 与这个符号相对应的符号被作为ag——虽然这允许我们获得一个对分类的trophical expressed models的活化序列表示,但是一个缺点是我们的任务.
<a id="S0054"></a> Source: p.4 S0054
Original: Sentence pairs are packed together into a are creating a mismatch between pre-training and single sequence.
中文: 句子对被打包成一正使预训和单序不匹配.
<a id="S0055"></a> Source: p.4 S0055
Original: We differentiate the sentences in fine-tuning, since the [MASK] token does not aptwo ways.
中文: 我们用微调来区分句子,因为[MASK]符号不是双向的。
<a id="S0056"></a> Source: p.4 S0056
Original: First, we separate them with a special pear during fine-tuning.
中文: 首先,我们在微调时用一棵特殊的梨来分离它们.
<a id="S0057"></a> Source: p.4 S0057
Original: Second, we add a learned embed- not always replace “masked” words with the acding to every token indicating whether it belongs tual [MASK] token.
中文: 第二,我们添加一个有学问的嵌入 -- -- 并不总是将“被蒙上”的词取而代之的是每一个表示它是否属于图尔[MASK]活字。
<a id="S0058"></a> Source: p.4 S0058
Original: The training data generator to sentence A or sentence B.
中文: 培训数据生成器用于判决A或判决B。
<a id="S0059"></a> Source: p.4 S0059
Original: As shown in Figure 1, chooses 15% of the token positions at random for we denote input embedding as E, the final hidden prediction.
中文: 如图1所示,我们随机选择15%的象征位置,因为我们表示输入嵌入为E,即最后隐藏的预测.
<a id="S0060"></a> Source: p.4 S0060
Original: If the i-th token is chosen, we replace vector of the special [CLS] token as C ∈ RH , the i-th token with (1) the [MASK] token 80% of and the final hidden vector for the ith input token the time (2) a random token 10% of the time (3) as T ∈ RH . the unchanged i-th token 10% of the time.
中文: 如果选择了i-th 令牌,我们将特殊 [CLS] 令牌的向量替换为 C QQ RH, i-th 令牌为 (1) [MASK] 令牌 80% 和 最终隐藏的向量 以时间 (2) 作为随机令牌 10% 时间 (3) 作为 T QQ RH . 不变的 I-th 令牌 10% 时间 .
<a id="S0061"></a> Source: p.4 S0061
Original: Then, i For a given token, its input representation is T i will be used to predict the original token with constructed by summing the corresponding token, cross entropy loss.
中文: 然后,i 对于一个给定的符号,它的输入表示是T i会被用来通过对相应的符号进行相接来预测所构造的原始符号,交叉的 en损.
<a id="S0062"></a> Source: p.4 S0062
Original: We compare variations of this segment, and position embeddings. A visualiza- procedure in Appendix C.2. tion of this construction can be seen in Figure 2.
中文: 我们比较这个段的变异和位置嵌入。 图2显示了附录C.2中的可视化程序。
<a id="S0063"></a> Source: p.4 S0063
Original: Task #2: Next Sentence Prediction (NSP) 3.1 Pre-training BERT Many important downstream tasks such as Question Answering (QA) and Natural Language Infer- Unlike Peters et al. (2018a) and Radford et al. ence (NLI) are based on understanding the rela- (2018), we do not use traditional left-to-right or tionship between two sentences, which is not diright-to-left language models to pre-train BERT. rectly captured by language modeling.
中文: 任务 2: 下句预测( NSP) 3. 1 培训前贝特 许多重要的下游任务,如"问题回答"(QA)和"自然语言"(Natural Language Infer-)与Peters等(2018a)和"拉德福德等"(NLI)是基于对rela-(2018)的理解,我们不使用两句之间传统的"从左到右"或"通"来进行预训BERT. 由语言建模直截了当地捕获.
<a id="S0064"></a> Source: p.4 S0064
Original: In order Instead, we pre-train BERT using two unsuperto train a model that understands sentence relavised tasks, described in this section.
中文: 取而代之的是,我们预训BERT使用两个非超能力训练一个能理解句子重排任务的模式,在本节中描述.
<a id="S0065"></a> Source: p.4 S0065
Original: This step tionships, we pre-train for a binarized next senis presented in the left part of Figure 1. tence prediction task that can be trivially gener- Task #1: Masked LM Intuitively, it is reason- ated from any monolingual corpus.
中文: 这艘步道飞船 我们为图1左部分中呈现的 二进制的下一艘赛尼号进行预训 坚斯的预测任务 可能是微不足道的基因 - 任务 # 1: 蒙面 LM 直觉上,这是理由 从任何单一语言的书本。
<a id="S0066"></a> Source: p.4 S0066
Original: Specifically, able to believe that a deep bidirectional model is when choosing the sentences A and B for each prestrictly more powerful than either a left-to-right training example, 50% of the time B is the actual model or the shallow concatenation of a left-to- next sentence that follows A (labeled as IsNext), right and a right-to-left model.
中文: 具体来说,能够相信深双向模式是,在为每个前行选择比从左到右的训练例子更强的句子A和B时,50%的时间B是实际模式,或者是A后行左到下句(被标记为IsNext),右到左的句子的浅接.
<a id="S0067"></a> Source: p.4 S0067
Original: Unfortunately, and 50% of the time it is a random sentence from standard conditional language models can only be the corpus (labeled as NotNext).
中文: 不幸的是,在标准有条件语言模型中作为随机句子的50%时间只能是本体(被标记为"NotNext").
<a id="S0068"></a> Source: p.4 S0068
Original: As we show trained left-to-right or right-to-left, since bidirec- in Figure 1, C is used for next sentence predictional conditioning would allow each word to in- tion (NSP).5 Despite its simplicity, we demondirectly “see itself”, and the model could trivially strate in Section 5.1 that pre-training towards this predict the target word in a multi-layered context. task is very beneficial to both QA and NLI. 6 former is often referred to as a “Transformer encoder” while 5The final model achieves 97%-98% accuracy on NSP. the left-context-only version is referred to as a “Transformer 6The vector C is not a meaningful sentence representation decoder” since it can be used for text generation. without fine-tuning, since it was trained with NSP.
中文: 当我们显示训练有素的从左到右或从右到左时,由于图1中的bodirec-用于下一句预测条件,每个词都可以被输入(NSP)。 尽管它很简单,但我们直接地“看到自己”,而且该模型在5.1节中可能略微僵化,该节在多层次的环境下为预测目标词进行预先培训。 任务对质量保证和NLI都非常有益。 6 前者常被称作“转换编码器”,而5 最终模型在NSP上达到97%-98%的精确度. 仅用左文本的版本被称为“Transformer 6,矢量 C不是一个有意义的句子表示解码器”,因为它可用于文本生成。 没有微调,因为它是训练NSP。
<a id="S0069"></a> Source: p.5 S0069
Original: Input [CLS] my dog is cute [SEP] he likes play ##ing [SEP] Token E E E E E E E E E E E Embeddings [CLS] my dog is cute [SEP] he likes play ##ing [SEP] Segment E E E E E E E E E E E Embeddings A A A A A A B B B B B Position E E E E E E E E E E E Embeddings 0 1 2 3 4 5 6 7 8 9 10 Figure 2: BERT input representation.
中文: [CLS]我的狗很可爱 [SEP]他喜欢玩 [SEP] 托肯 E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E Embeddings [CLS] 我的狗很可爱 [SEP] 他喜欢玩 [SEP] [SEP] E部分 E E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E部分 E
<a id="S0070"></a> Source: p.5 S0070
Original: The input embeddings are the sum of the token embeddings, the segmentation embeddings and the position embeddings.
中文: 入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入出入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入
<a id="S0071"></a> Source: p.5 S0071
Original: The NSP task is closely related to representation- (4) a degenerate text-∅ pair in text classification learning objectives used in Jernite et al. (2017) and or sequence tagging.
中文: 国家战略计划的任务与表述-(4)在Jernite等人(2017年)和或序列标记中使用的文本分类学习目标中一种变质的文本-X对关系密切。
<a id="S0072"></a> Source: p.5 S0072
Original: At the output, the token rep- Logeswaran and Lee (2018).
中文: 出品时,代号为"Logeswaran"和"Lee" (2018).
<a id="S0073"></a> Source: p.5 S0073
Original: However, in prior resentations are fed into an output layer for tokenwork, only sentence embeddings are transferred to level tasks, such as sequence tagging or question down-stream tasks, where BERT transfers all pa- answering, and the [CLS] representation is fed rameters to initialize end-task model parameters. into an output layer for classification, such as entailment or sentiment analysis.
中文: 然而,在之前的怨恨中,会被输入到一个输出层以用于表示工作,只有句子嵌入被转移到等级任务,例如序列标记或向下流任务提问,BERT将所有pa-应答,而[CLS]代表被给入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入出入出入出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出 进入一个输出层进行分类,例如内涵分析或情绪分析。
<a id="S0074"></a> Source: p.5 S0074
Original: Pre-training data The pre-training procedure Compared to pre-training, fine-tuning is relalargely follows the existing literature on language tively inexpensive.
中文: 培训前数据 与培训前相比,培训前程序调整在很大程度上沿用了现有关于语言的廉价文献。
<a id="S0075"></a> Source: p.5 S0075
Original: All of the results in the pamodel pre-training.
中文: 建模预训中的所有成绩.
<a id="S0076"></a> Source: p.5 S0076
Original: For the pre-training corpus we per can be replicated in at most 1 hour on a sinuse the BooksCorpus (800M words) (Zhu et al., gle Cloud TPU, or a few hours on a GPU, starting 2015) and English Wikipedia (2,500M words). from the exact same pre-trained model.7 We de- For Wikipedia we extract only the text passages scribe the task-specific details in the correspondand ignore lists, tables, and headers.
中文: 对于预训课程,我们每人最多可以在1小时内被复制到"活塞"(BooksCorpus (800M words) (Zhu等,gle Cloud TPU,或者在GPU上几个小时,从2015年开始)和英语维基百科(2,500M words)上. 完全相同的预训型号。 为了维基百科,我们只提取文本段落,在对应和忽略列表、表格和页眉中记录特定任务的细节。
<a id="S0077"></a> Source: p.5 S0077
Original: It is critiing subsections of Section 4.
中文: 这是第4节中令人痛心的分节。
<a id="S0078"></a> Source: p.5 S0078
Original: More details can be cal to use a document-level corpus rather than a found in Appendix A.5. shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to 4 Experiments extract long contiguous sequences.
中文: 更多的细节可以用来使用文件级体,而不是在附录A.5.中找到的. Shuffled 句子级体,如"亿字基准"(Chelba等,2013年),以便4个实验提取出长相相相接的序列.
<a id="S0079"></a> Source: p.5 S0079
Original: In this section, we present BERT fine-tuning re- 3.2 Fine-tuning BERT sults on 11 NLP tasks.
中文: 在本节中,我们介绍BERT微调再调整 - 3.2微调BERT sults 11 NLP任务.
<a id="S0080"></a> Source: p.5 S0080
Original: Fine-tuning is straightforward since the self- 4.1 GLUE attention mechanism in the Transformer al- The General Language Understanding Evaluation lows BERT to model many downstream tasks— (GLUE) benchmark (Wang et al., 2018a) is a colwhether they involve single text or text pairs—by lection of diverse natural language understanding swapping out the appropriate inputs and outputs. tasks.
中文: 精细调整是直截了当的,因为变形器 al-The General Language understanding Revolution中的自 4.1 GLUE 注意机制降低了BERT的分量,以模拟许多下游任务——(GLUE)基准(Wang等,2018年a)——是一种串联,无论它们涉及单一文本还是文本配对——通过各种自然语言理解的取出适当的投入和产出来拼接。 任务。
<a id="S0081"></a> Source: p.5 S0081
Original: Detailed descriptions of GLUE datasets are For applications involving text pairs, a common included in Appendix B.1. pattern is to independently encode text pairs be- To fine-tune on GLUE, we represent the input fore applying bidirectional cross attention, such sequence (for single sentence or sentence pairs) as Parikh et al. (2016); Seo et al. (2017).
中文: GLUE数据集的详细描述针对涉及文本对的应用程序,附录B.1中包含的常见模式是独立编码文本对为-. 为了对GLUE进行微调,我们代表了应用双向交叉注意力的输入前缀,如Parikh等(2016年);Seo等(2017年)。
<a id="S0082"></a> Source: p.5 S0082
Original: BERT as described in Section 3, and use the final hidinstead uses the self-attention mechanism to unify den vector C ∈ RH corresponding to the first these two stages, as encoding a concatenated text input token ([CLS]) as the aggregate representapair with self-attention effectively includes bidition.
中文: 如第3节所描述的BERT,并使用最终隐藏机制使用自我注意机制,将密度矢量C QQ RH统一起来,与这两个阶段的第一阶段相对应,因为将一个被拼接的文本输入符([CLS])编码为自注意的集合代表符实际上包括出价.
<a id="S0083"></a> Source: p.5 S0083
Original: The only new parameters introduced during rectional cross attention between two sentences. fine-tuning are classification layer weights W ∈ For each task, we simply plug in the task- RK×H , where K is the number of labels.
中文: 复出时唯一引入的新参数在两句之间交叉注意. 微调是分类层权重W QQ 对于每一项任务,我们只是插入任务- RKxH,其中K是标签的数量.
<a id="S0084"></a> Source: p.5 S0084
Original: We comspecific inputs and outputs into BERT and finepute a standard classification loss with C and W , tune all the parameters end-to-end.
中文: 我们为BERT提供具体投入和产出,并对C和W的标准分类损失进行微调,调整所有参数的端到端。
<a id="S0085"></a> Source: p.5 S0085
Original: At the ini.e., log(softmax(CW T )). put, sentence A and sentence B from pre-training are analogous to (1) sentence pairs in paraphras- 7For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev ing, (2) hypothesis-premise pairs in entailment, (3) F1 score of 91.0%. question-passage pairs in question answering, and 8See (10) in https://gluebenchmark.com/faq.
中文: 在ini.,即:log(softmax(CW T)). 将A句和B句放在训练前类似于(1) 第7段的句子对等。 例如,BERT SQuAD模型可以在大约30分钟内在一个Cloud TPU上进行训练,以实现 Dev ing,(2) 假设-假设对等,(3) F1分91.0%。 回答中的问题-通过对,以及https://gluebenchmark.com/faq中的8-10。
<a id="S0086"></a> Source: p.6 S0086
Original: System MNLI-(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE Average 392k 363k 108k 67k 8.5k 5.7k 3.5k 2.5k - Pre-OpenAI SOTA 80.6/80.1 66.1 82.3 93.2 35.0 81.0 86.0 61.7 74.0 BiLSTM+ELMo+Attn 76.4/76.1 64.8 79.8 90.4 36.0 73.3 84.9 56.8 71.0 OpenAI GPT 82.1/81.4 70.3 87.4 91.3 45.4 80.0 82.3 56.0 75.1 BERT 84.6/83.4 71.2 90.5 93.5 52.1 85.8 88.9 66.4 79.6 BASE BERT 86.7/85.9 72.1 92.7 94.9 60.5 86.5 89.3 70.1 82.1 LARGE Table 1: GLUE Test results, scored by the evaluation server (https://gluebenchmark.com/leaderboard).
中文: 系统 MNLI-(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE 平均 392k 363k 108k 67k 8.5k 5.7k 3.5k 2.5k - 前开放AI SOTA 80.6/80.1 66.1 82.3 93.2 35.0 86.0 61.0 74.7 74.0 BilSTM+ELM+64.4/76.1 64.79.8 90.4 36.0 73.3 84.9 56.8 71.0 OpenAI GPT 82.1/81.4 70.3 87.4 91 43.4 82.0.5 56.0 75.1 BERT 84.6/83.4 71.2 90.5 93.5 52.1 85.8 88.9 66.4 79.6 BASE BERT 86.7/85.9 72.1 92.7 94.9 60.5 86.5 89.3 70.1 LARGE 表1:GLUE 测试结果,由评价服务器评分(https://gluebenchmark.com/leaderboard).
<a id="S0087"></a> Source: p.6 S0087
Original: The number below each task denotes the number of training examples.
中文: 以下各项任务的数目表示培训实例的数目。
<a id="S0088"></a> Source: p.6 S0088
Original: The “Average” column is slightly different than the official GLUE score, since we exclude the problematic WNLI set.8 BERT and OpenAI GPT are singlemodel, single task.
中文: “Average”一栏与官方的GLUE分数略有不同,因为我们排除了有问题的WNLI设定。 8 BERT和OpenAI GPT是单一模式,单一任务。
<a id="S0089"></a> Source: p.6 S0089
Original: F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks.
中文: F1分被报告为QQP和MRPC,斯皮尔曼相关被报告为STS-B,而准确分被报告为其他任务.
<a id="S0090"></a> Source: p.6 S0090
Original: We exclude entries that use BERT as one of their components.
中文: 我们排除了使用BERT作为其组件之一的条目.
<a id="S0091"></a> Source: p.6 S0091
Original: We use a batch size of 32 and fine-tune for 3 Wikipedia containing the answer, the task is to epochs over the data for all GLUE tasks.
中文: 我们使用一个32的批量大小和微调的3个维基百科包含答案,任务就是对所有GLUE任务的数据进行划时代.
<a id="S0092"></a> Source: p.6 S0092
Original: For each predict the answer text span in the passage. task, we selected the best fine-tuning learning rate As shown in Figure 1, in the question answer- (among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set. ing task, we represent the input question and pas- Additionally, for BERT LARGE we found that fine- sage as a single packed sequence, with the questuning was sometimes unstable on small datasets, tion using the A embedding and the passage using so we ran several random restarts and selected the the B embedding.
中文: 每一个都预言答案文本在段落中跨度。 如图1所示,我们选择了最佳微调学习率。 任务,我们代表 输入问题和帕斯 - 此外,对于BERT LARGE,我们发现精细的贤者作为单一的被打包的序列,在小数据集上,追求有时不稳定,使用A嵌入和通过,所以我们运行了几次随机重启,选择了B嵌入.
<a id="S0093"></a> Source: p.6 S0093
Original: We only introduce a start vecbest model on the Dev set.
中文: 我们只在Dev片场引入了最佳的开始模型.
<a id="S0094"></a> Source: p.6 S0094
Original: With random restarts, tor S ∈ RH and an end vector E ∈ RH during we use the same pre-trained checkpoint but per- fine-tuning.
中文: 随着随机重启,Tor S QQ RH 和末向量 E QQ RH 在我们使用相同的预先训练的检查点但每个微调.
<a id="S0095"></a> Source: p.6 S0095
Original: The probability of word i being the form different fine-tuning data shuffling and clas- start of the answer span is computed as a dot prodsifier layer initialization.9 uct between T and S followed by a softmax over i Results are presented in Table 1.
中文: 单词一作为不同微调数据打乱和clas-起始的答题的概率是作为点增生层初始化计算出。 9 T和S之间的uct,然后是软max相接i的结果见表1。
<a id="S0096"></a> Source: p.6 S0096
Original: Both all of the words in the paragraph: P = eS·Ti . i (cid:80) eS·Tj BERT and BERT outperform all sys- j BASE LARGE The analogous formula is used for the end of the tems on all tasks by a substantial margin, obtaining answer span.
中文: 该段中的两个词:P=eS=Ti. (cid:80) eS-Tj BERT和BERT都超过了所有sys-j BASE LARGE 在所有任务的结束时,以相当的差幅使用类似的公式,并获得答案。
<a id="S0097"></a> Source: p.6 S0097
Original: The score of a candidate span from 4.5% and 7.0% respective average accuracy imposition i to position j is defined as S·T + E·T , i j provement over the prior state of the art.
中文: 候选人的得分从4.5%和7.0%不等,分别对定位j的平均精度要求i被定义为S-T + E-T, i j 的证明高于先前的状态.
<a id="S0098"></a> Source: p.6 S0098
Original: Note that and the maximum scoring span where j ≥ i is BERT and OpenAI GPT are nearly identical BASE used as a prediction.
中文: 请注意,j ≥i为BERT而OpenAI GPT作为预测使用的BASE几乎完全相同,最大得分跨度也相同。
<a id="S0099"></a> Source: p.6 S0099
Original: The training objective is the in terms of model architecture apart from the atsum of the log-likelihoods of the correct start and tention masking.
中文: 培训目标是在模型架构方面,除正确起步和留念口罩的对数相接外。
<a id="S0100"></a> Source: p.6 S0100
Original: For the largest and most widely end positions.
中文: 对于最大和最广泛的端口位置。
<a id="S0101"></a> Source: p.6 S0101
Original: We fine-tune for 3 epochs with a reported GLUE task, MNLI, BERT obtains a 4.6% learning rate of 5e-5 and a batch size of 32. absolute accuracy improvement.
中文: 我们精细调整了3个世纪的GLUE任务, MNLI, BERT获得4.6%的学习率为5e-5,批量规模为32. 绝对准确性提高。
<a id="S0102"></a> Source: p.6 S0102
Original: On the official Table 2 shows top leaderboard entries as well GLUE leaderboard10, BERT obtains a score LARGE as results from top published systems (Seo et al., of 80.5, compared to OpenAI GPT, which obtains 2017; Clark and Gardner, 2018; Peters et al., 72.8 as of the date of writing. 2018a; Hu et al., 2018).
中文: 在"官方表2"上,显示顶级领跑板条目以及GLUE领跑板10,BERT通过顶级发布系统(Seo等,80.5分;而OpenAI GPT则获得2017分;Clark等,2018分;Peters等,72.8分)获得分数. 2018a; 胡等, 2018).
<a id="S0103"></a> Source: p.6 S0103
Original: The top results from the We find that BERT significantly outper- LARGE SQuAD leaderboard do not have up-to-date public forms BERT across all tasks, especially those BASE system descriptions available,11 and are allowed to with very little training data.
中文: 最高结果来自我们发现,BERT在LARGE SQuAD领导板上明显超越了所有任务,特别是现有的BASE系统描述,11 没有最新的公共表格BERT,而且很少培训数据。
<a id="S0104"></a> Source: p.6 S0104
Original: The effect of model use any public data when training their systems. size is explored more thoroughly in Section 5.2.
中文: 模型的效果在培训其系统时使用任何公共数据。 第5.2节更详尽地探讨了规模问题。
<a id="S0105"></a> Source: p.6 S0105
Original: We therefore use modest data augmentation in our system by first fine-tuning on TriviaQA (Joshi 4.2 SQuAD v1.1 et al., 2017) befor fine-tuning on SQuAD.
中文: 因此,我们通过对TriviaQA(Joshi 4.2 SQuAD v1.1 et al., 2017)进行首次微调,在系统中使用微调数据来进行微调。
<a id="S0106"></a> Source: p.6 S0106
Original: The Stanford Question Answering Dataset Our best performing system outperforms the top (SQuAD v1.1) is a collection of 100k crowdleaderboard system by +1.5 F1 in ensembling and sourced question/answer pairs (Rajpurkar et al., +1.3 F1 as a single system.
中文: Stanford Question Answering Dataset Our best performance system servers over the top(SQuAD v1.1)是一款由 +1.5 F1 组成的百克众筹板系统在综艺和源问答对上(Rajpurkar等,+1.3 F1作为单系统)所组成的集合.
<a id="S0107"></a> Source: p.6 S0107
Original: Given a question and a passage from BERT model outperforms the top ensemble sys- 9The GLUE data set distribution does not include the Test tem in terms of F1 score.
中文: 鉴于一个问题和BERT模型的一段话超越了最顶尖的综艺 sys-9 GLUE数据集分布不包括从F1分数来看的Test tem.
<a id="S0108"></a> Source: p.6 S0108
Original: Without TriviaQA finelabels, and we only made a single GLUE evaluation server submission for each of BERT BASE and BERT LARGE . 11QANet is described in Yu et al. (2018), but the system 10https://gluebenchmark.com/leaderboard has improved substantially after publication.
中文: 没有TriviaQA的精细标签,我们只为BERT BASE和BERT LARGE的每个版本做了一个单一的GLUE评价服务器提交. 11QANet在Yu等 (2018)中被描述,但系统10https://gluebenchmark.com/leaderboard在发布后得到了大幅改进.
<a id="S0109"></a> Source: p.7 S0109
Original: System Dev Test System Dev Test EM F1 EM F1 ESIM+GloVe 51.9 52.7 Top Leaderboard Systems (Dec 10th, 2018) ESIM+ELMo 59.1 59.2 Human - - 82.3 91.2 OpenAI GPT - 78.0 #1 Ensemble - nlnet - - 86.0 91.7 BERT 81.6 - #2 Ensemble - QANet - - 84.5 90.5 BASE BERT 86.6 86.3 LARGE Published Human (expert)† - 85.0 BiDAF+ELMo (Single) - 85.6 - 85.8 R.M.
中文: 系统 Dev Test System Dev Test EM F1 EM F1 ESIM+GloVe 51.9 52.7 顶级导板系统(Dec 10th, 2018) ESIM+ELMo 59.1 59.2 Human - - 82.3 91.2 OpenAI GPT - 78.0 #1 Ensemble - nlnet - 86.0 9.1.7 BERT 81.6 - #2 Ensemble - QANet - 84.5 90.5 BASE BERT 86.6 86.3 LARGE 出版人(专家) † - 85.0 BiDAF+ELMo (Single) - 85.6 - 85.8 R.M.
<a id="S0110"></a> Source: p.7 S0110
Original: Reader (Ensemble) 81.2 87.9 82.3 88.5 Human (5 annotations)† - 88.0 Ours Table 4: SWAG Dev and Test accuracies. †Human per- BERT (Single) 80.8 88.5 - - BASE BERT (Single) 84.1 90.9 - - formance is measured with 100 samples, as reported in LARGE BERT (Ensemble) 85.8 91.8 - - the SWAG paper.
中文: 阅读器(集) 81.2 87.9 82.3 88.5 人类(5个说明) † - 88.0 我们的表4: SWAG Dev and Test Cracies. † 人/贝特(单一) 80.8 88.5 - - BASE贝特(单一) 84.1 90.9 - - 如LARGE BERT(集成)85.8 91.8 - SWAG文件所报告,用100个样本测量成型。
<a id="S0111"></a> Source: p.7 S0111
Original: LARGE BERT (Sgl.+TriviaQA) 84.2 91.1 85.1 91.8 LARGE BERT (Ens.+TriviaQA) 86.2 92.2 87.4 93.2 LARGE sˆ = max S·T + E·T .
中文: LARGE BERT(Sgl.+TriviaQA) 84.2 91.1 85.1 91.8 LARGE BERT(Ens.+TriviaQA) 86.2 92.2 87.4 93.2 LARGE sˆ=最大 S-T + E-T.
<a id="S0112"></a> Source: p.7 S0112
Original: We predict a non-null i,j j≥i i j Table 2: SQuAD 1.1 results.
中文: 我们预测一个非null i,j j≥i i j 表2: SQuAD 1.1结果.
<a id="S0113"></a> Source: p.7 S0113
Original: The BERT ensemble answer when sˆ > s + τ , where the threshi,j null is 7x systems which use different pre-training checkold τ is selected on the dev set to maximize F1. points and fine-tuning seeds.
中文: 当 sˆ > s + τ时, BERT 综艺解答, 其中 treshi, j 无效为 7x 系统, 使用不同的训练前检查符 τ 在 dev 集上选择, 以最大化 F1 分并微调种子 .
<a id="S0114"></a> Source: p.7 S0114
Original: We did not use TriviaQA data for this model.
中文: 我们没有使用 TriviaQA 数据为这个模型。
<a id="S0115"></a> Source: p.7 S0115
Original: We fine-tuned for 2 epochs with a learning rate of 5e-5 System Dev Test and a batch size of 48.
中文: 我们对两个时代进行了微调,学习率为5e-5系统Dev Test,批量尺寸为48.
<a id="S0116"></a> Source: p.7 S0116
Original: EM F1 EM F1 The results compared to prior leaderboard en- Top Leaderboard Systems (Dec 10th, 2018) Human 86.3 89.0 86.9 89.5 tries and top published work (Sun et al., 2018; #1 Single - MIR-MRC (F-Net) - - 74.8 78.0 Wang et al., 2018b) are shown in Table 3, exclud- #2 Single - nlnet - - 74.2 77.1 ing systems that use BERT as one of their com- Published ponents.
中文: EM F1 EM F1 与前导板 en-Top Leaderboard Systems(Dec 10th, 2018)相比的结果 Human 86.3 89.0 86.9 89.5 尝试和顶级出版作品(Sun等, 2018;#1 Single-MIR-MRC (F-Net)-74.8 78.0) Wang et al., 2018b)在表3中列出,不包括-2 单-nlnet--74.2 77.1 使用BERT作为其com所出版的活塞之一的活塞.
<a id="S0117"></a> Source: p.7 S0117
Original: We observe a +5.1 F1 improvement over unet (Ensemble) - - 71.4 74.9 SLQA+ (Single) - 71.4 74.4 the previous best system.
中文: 我们观测到一个+ 5.1 F1 优于 unet(组合) - 71.4 74.9 SLQA+(单一) - 71.4 74.4 之前最好的系统.
<a id="S0118"></a> Source: p.7 S0118
Original: Ours 4.4 SWAG BERT (Single) 78.7 81.9 80.0 83.1 LARGE The Situations With Adversarial Generations Table 3: SQuAD 2.0 results.
中文: 我们的4.4 SWAG BERT(单一) 78.7 81.9 80.0 83.1 LARGE 表3:SQuAD 2.0结果。
<a id="S0119"></a> Source: p.7 S0119
Original: We exclude entries that (SWAG) dataset contains 113k sentence-pair comuse BERT as one of their components. pletion examples that evaluate grounded commonsense inference (Zellers et al., 2018).
中文: 我们排除(SWAG)数据集中包含113k个句子-pair comuse BERT作为其组件之一的条目. 评估基于常识推论的多功能实例(Zellers等,2018年)。
<a id="S0120"></a> Source: p.7 S0120
Original: Given a sentence, the task is to choose the most plausible contuning data, we only lose 0.1-0.4 F1, still outpertinuation among four choices. forming all existing systems by a wide margin.12 When fine-tuning on the SWAG dataset, we 4.3 SQuAD v2.0 construct four input sequences, each containing the concatenation of the given sentence (sentence The SQuAD 2.0 task extends the SQuAD 1.1 A) and a possible continuation (sentence B).
中文: 有一句话,任务就是选择最可信的调和数据,我们只损失了0.1-0.4 F1,仍然超越了四个选择。 将所有现有系统扩大范围。 在对SWAG数据集进行微调时,我们4.3 SQuAD v2.0 构造了四个输入序列,每个序列包含给定句子的调和(sentence The SQuAD 2.0任务延长了SQuAD 1.1 A)并可能延续(sentence B).
<a id="S0121"></a> Source: p.7 S0121
Original: The problem definition by allowing for the possibility only task-specific parameters introduced is a vecthat no short answer exists in the provided parator whose dot product with the [CLS] token repgraph, making the problem more realistic. resentation C denotes a score for each choice We use a simple approach to extend the SQuAD which is normalized with a softmax layer. v1.1 BERT model for this task.
中文: 通过只允许引入任务特定参数的可能性而给出的问题定义是一个vec,即所提供派生物中不存在一个短答,其点出物与[CLS]符号地圖相通,使问题更加现实. 怨恨 C 表示每个选择的分数 我们使用一个简单的方法来扩展SQuAD,它与软马克斯层相正统. v1.1 此项任务的BERT模型.
<a id="S0122"></a> Source: p.7 S0122
Original: We treat ques- We fine-tune the model for 3 epochs with a tions that do not have an answer as having an anlearning rate of 2e-5 and a batch size of 16.
中文: 我们治疗克丝... 我们微调了3个纪元的模型,其分数没有答案,因为学习率为2e-5,批量大小为16.
<a id="S0123"></a> Source: p.7 S0123
Original: Reswer span with start and end at the [CLS] tosults are presented in Table 4.
中文: 表4列示了[CLS]断层起止的回转间隔。
<a id="S0124"></a> Source: p.7 S0124
Original: The probability space for the start and end LARGE performs the authors’ baseline ESIM+ELMo sysanswer span positions is extended to include the tem by +27.1% and OpenAI GPT by 8.3%. position of the [CLS] token.
中文: 开始和结束的概率空间 LARGE 执行作者的基线 ESIM+ELMo sysanswer 跨度位置被扩展为+27.1%的 Tem 和 8.3%的 OpenAI GBT 。 [CLS]符号的位置。
<a id="S0125"></a> Source: p.7 S0125
Original: For prediction, we compare the score of the no-answer span: s = null 5 Ablation Studies S·C + E·C to the score of the best non-null span In this section, we perform ablation experiments 12The TriviaQA data we used consists of paragraphs from over a number of facets of BERT in order to better TriviaQA-Wiki formed of the first 400 tokens in documents, that contain at least one of the provided possible answers. understand their relative importance.
中文: 对于预测,我们比较无答间距的分数: s = 无效5 Ablus Research S-C + E-C 与最佳非核间距的分数 在本节中,我们进行通融实验 12 我们使用的TriviaQA数据包括来自BERT多个方面的段落,以便更好的是TriviaQA-Wiki由文档中的前400个符号所形成,其中至少包含一个可能提供的答案. 了解它们的相对重要性。
<a id="S0126"></a> Source: p.8 S0126
Original: Dev Set results are still far worse than those of the pre- Tasks MNLI-m QNLI MRPC SST-2 SQuAD trained bidirectional models.
中文: 德维 设定结果仍然比任务前的MNLI-m QNLI MRPC SST-2 SQuAD训练双向模式更差.
<a id="S0127"></a> Source: p.8 S0127
Original: The BiLSTM hurts (Acc) (Acc) (Acc) (Acc) (F1) performance on the GLUE tasks.
中文: BILSTM伤害 (Acc) (Acc) (Acc) (Acc) (F1) 在GLUE任务上的表现.
<a id="S0128"></a> Source: p.8 S0128
Original: BERT 84.4 88.4 86.7 92.7 88.5 BASE We recognize that it would also be possible to No NSP 83.9 84.9 86.5 92.6 87.9 LTR & No NSP 82.1 84.3 77.5 92.1 77.8 train separate LTR and RTL models and represent + BiLSTM 82.1 84.1 75.7 91.6 84.9 each token as the concatenation of the two models, as ELMo does.
中文: BERT 84.4 88.4 86.7 92.7 88.5 BASE 我们认识到,也可以使用NSP 83.9 84.9 86.5 92.6 87.9 LTR & No NSP 82.1 84.3 77.5 92.1 77.8次列车分别使用LTR和RTL两种车型,并代表+ BiLSTM 82.1 84.1 75.7 91.6 84.9 每一个符号作为两个模型的调和,就像ELMo一样.
<a id="S0129"></a> Source: p.8 S0129
Original: However: (a) this is twice as Table 5: Ablation over the pre-training tasks using the expensive as a single bidirectional model; (b) this BERT architecture. “No NSP” is trained without BASE is non-intuitive for tasks like QA, since the RTL the next sentence prediction task. “LTR & No NSP” is trained as a left-to-right LM without the next sentence model would not be able to condition the answer prediction, like OpenAI GPT. “+ BiLSTM” adds a ran- on the question; (c) this it is strictly less powerful domly initialized BiLSTM on top of the “LTR + No than a deep bidirectional model, since it can use NSP” model during fine-tuning. both left and right context at every layer. 5.2 Effect of Model Size ablation studies can be found in Appendix C.
中文: 然而:(a) 这是表5的两倍:(a) 使用昂贵的单双向模式对培训前任务进行对比;(b) 这种BERT架构。 “无核生化”在没有BASE的情况下培训,对QA等任务不具有直观性,因为RTL是下句预测任务。 “LTR & No NSP”作为左到右的LM培训,没有下句模型将无法像OpenAI GPT那样对答案预测进行条件化。 “+ BiLSTM”在这个问题上增加了一跑; (c) 这在“LTR + NO”模型上方的多姆化初始化BiLSTM绝对不如深双向模型,因为它可以在微调时使用NSP模型。 每层左右上下文。 5.2 模型大小衰减研究的效果见附录C。
<a id="S0130"></a> Source: p.8 S0130
Original: In this section, we explore the effect of model size 5.1 Effect of Pre-training Tasks on fine-tuning task accuracy.
中文: 在本节中,我们探讨模式尺寸5.1对精细调整任务准确性的影响。
<a id="S0131"></a> Source: p.8 S0131
Original: We trained a number of BERT models with a differing number of layers, We demonstrate the importance of the deep bidihidden units, and attention heads, while otherwise rectionality of BERT by evaluating two preusing the same hyperparameters and training protraining objectives using exactly the same precedure as described previously. training data, fine-tuning scheme, and hyperpa- Results on selected GLUE tasks are shown in rameters as BERT : BASE Table 6.
中文: 我们训练了一些具有不同层次的BERT模型,我们展示了深层Badihidden单元和注意力头的重要性,而BERT则通过评估两个预先使用同一超参数以及使用与前述完全相同的先入为主的训练目标来进行再演绎。 培训数据、微调办法和超高级 选定GLUE任务的结果以拉面显示为BERT:BASE表6。
<a id="S0132"></a> Source: p.8 S0132
Original: In this table, we report the average Dev No NSP: A bidirectional model which is trained Set accuracy from 5 random restarts of fine-tuning. using the “masked LM” (MLM) but without the We can see that larger models lead to a strict ac- “next sentence prediction” (NSP) task. curacy improvement across all four datasets, even LTR & No NSP: A left-context-only model which for MRPC which only has 3,600 labeled trainis trained using a standard Left-to-Right (LTR) ing examples, and is substantially different from LM, rather than an MLM.
中文: 在本表中,我们报告平均Dev No NSP:一种双向模型,它从5个随机重启微调中训练出来。 使用“假LM”(MLM),但没有“我们可以看到,更大的模型会导致严格的AC-“下句预测”(NSP)任务。 ceracy改进了所有四个数据集,甚至LTR & No NSP:一种仅用左文本的模型,对于MRPC来说,它只有3,600个标记为trainis的训练,使用标准左到右(LTR)的示例,并且与LM,而不是MLM有很大不同.
<a id="S0133"></a> Source: p.8 S0133
Original: The left-only constraint the pre-training tasks.
中文: 仅由左所制约的训练前任务.
<a id="S0134"></a> Source: p.8 S0134
Original: It is also perhaps surpriswas also applied at fine-tuning, because removing ing that we are able to achieve such significant it introduced a pre-train/fine-tune mismatch that improvements on top of models which are aldegraded downstream performance.
中文: 它也许也被应用在微调中,因为去除我们能够实现如此重大的成就,它引入了在改进下游性能的模型之上的训练前/精益求精的不匹配。
<a id="S0135"></a> Source: p.8 S0135
Original: Additionally, ready quite large relative to the existing literature. this model was pre-trained without the NSP task.
中文: 此外,相对于现有文献而言,准备量相当大。 这一型号未经国家战略计划任务而经过预先培训。
<a id="S0136"></a> Source: p.8 S0136
Original: For example, the largest Transformer explored in This is directly comparable to OpenAI GPT, but Vaswani et al. (2017) is (L=6, H=1024, A=16) using our larger training dataset, our input repre- with 100M parameters for the encoder, and the sentation, and our fine-tuning scheme. largest Transformer we have found in the literature We first examine the impact brought by the NSP is (L=64, H=512, A=2) with 235M parameters task.
中文: 例如,本作所探索的最大变形器可直接与OpenAI GPT相媲美,但Vaswani等人(2017年)是(L=6,H=1024,A=16)使用我们更大的训练数据集,我们的输入还原——配有编码器的100M参数,发送,以及我们的微调方案. 我们在文献中发现的最大变形器 我们首先研究NSP带来的影响是(L=64,H=512,A=2),有235M参数任务.
<a id="S0137"></a> Source: p.8 S0137
Original: In Table 5, we show that removing NSP (Al-Rfou et al., 2018).
中文: 在表5中,我们显示删除NSP(Al-Rfou等,2018年)。
<a id="S0138"></a> Source: p.8 S0138
Original: By contrast, BERT BASE hurts performance significantly on QNLI, MNLI, contains 110M parameters and BERT con- LARGE and SQuAD 1.1.
中文: 相形之下,BERT BASE在QNLI,MNLI上显著地伤害了性能,包含110M参数和BERT con-LARGE和SQuAD 1.1.
<a id="S0139"></a> Source: p.8 S0139
Original: Next, we evaluate the impact tains 340M parameters. of training bidirectional representations by com- It has long been known that increasing the paring “No NSP” to “LTR & No NSP”.
中文: 接下来,我们评估340M的撞击参数。 培训的双向表述 众所周知,将“无国家战略计划”改为“无国家战略计划”。
<a id="S0140"></a> Source: p.8 S0140
Original: The LTR model size will lead to continual improvements model performs worse than the MLM model on all on large-scale tasks such as machine translation tasks, with large drops on MRPC and SQuAD. and language modeling, which is demonstrated For SQuAD it is intuitively clear that a LTR by the LM perplexity of held-out training data model will perform poorly at token predictions, shown in Table 6.
中文: LTR模型大小将导致持续改进模型在机器翻译任务等大型任务上表现得比MLM模型差,MRPC和SQuAD上出现大跌. 对SQuAD来说,语言模型的显示是直观的,即LM对被搁置的培训数据模型的LTR的迷惑性在象征性预测中效果不佳,见表6。
<a id="S0141"></a> Source: p.8 S0141
Original: However, we believe that since the token-level hidden states have no right- this is the first work to demonstrate convincside context.
中文: 然而,我们认为,由于象征级隐藏状态没有权利——这是第一个展示说服力背景的工作.
<a id="S0142"></a> Source: p.8 S0142
Original: In order to make a good faith at- ingly that scaling to extreme model sizes also tempt at strengthening the LTR system, we added leads to large improvements on very small scale a randomly initialized BiLSTM on top.
中文: 为了让一个良好的信念在—— 渐渐地, 缩放到极端模型大小 也诱导加强 LTR 系统, 我们添加了引领 在非常小的尺度上,
<a id="S0143"></a> Source: p.8 S0143
Original: This does tasks, provided that the model has been suffisignificantly improve results on SQuAD, but the ciently pre-trained.
中文: 这样做可以完成任务,前提是该模型已经大大地改进了SQuAD的成果,但经过了预先培训。
<a id="S0144"></a> Source: p.9 S0144
Original: mixed results on the downstream task impact of System Dev F1 Test F1 increasing the pre-trained bi-LM size from two ELMo (Peters et al., 2018a) 95.7 92.2 to four layers and Melamud et al. (2016) men- CVT (Clark et al., 2018) - 92.6 CSE (Akbik et al., 2018) - 93.1 tioned in passing that increasing hidden dimension size from 200 to 600 helped, but increasing Fine-tuning approach BERT 96.6 92.8 further to 1,000 did not bring further improve- LARGE BERT 96.4 92.4 BASE ments.
中文: System Dev F1 Test F1对下游任务影响的结果好坏参半,将预先训练的双LM尺寸从两个ELMo(Peters等人,2018年a)95.7 92.2升至四个层和Melamud等人(2016年) men-CVT(Clark等人,2018年) - 92.6 CSE (Akbik等,2018年) - 93.1转动,将隐藏尺寸从200个增加到600个有所帮助,但将微调方法BERT 96.6 92.8再加到1000个并没有带来进一步的改进 -- LARGE BERT 96.4 92.4 BASE ments.
<a id="S0145"></a> Source: p.9 S0145
Original: Both of these prior works used a feature- Feature-based approach (BERT ) BASE based approach — we hypothesize that when the Embeddings 91.0 model is fine-tuned directly on the downstream Second-to-Last Hidden 95.6 - Last Hidden 94.9 tasks and uses only a very small number of ran- Weighted Sum Last Four Hidden 95.9 domly initialized additional parameters, the task- Concat Last Four Hidden 96.1 specific models can benefit from the larger, more Weighted Sum All 12 Layers 95.5 expressive pre-trained representations even when Table 7: CoNLL-2003 Named Entity Recognition redownstream task data is very small. sults.
中文: 之前的两部作品都使用了一个特性... 基于地物的方法(BERT)基于BASE的方法——我们假设,当嵌入式91.0模型直接被微调到下游的"第二至最后隐藏"95.6——最后隐藏的94.9任务并使用极少数跑-. 加权总和 Last Four Hidden 95.9 Domly初始化了附加参数,任务- Concat Last Four Hidden 96.1 具体模型可以受益于更大,更加权总和 所有 12 层 95.5 表达式预受训练,即使表 7: CoNLL-2003 命名实体再下游任务数据很小. ul.
<a id="S0146"></a> Source: p.9 S0146
Original: Hyperparameters were selected using the Dev set.
中文: 使用 Dev 集选择了超参数 。
<a id="S0147"></a> Source: p.9 S0147
Original: The reported Dev and Test scores are averaged over 5.3 Feature-based Approach with BERT 5 random restarts using those hyperparameters.
中文: 所报告的Dev和Test分数平均超过5.3个基于地物的方法,BERT 5使用这些超参数随机重启。
<a id="S0148"></a> Source: p.9 S0148
Original: All of the BERT results presented so far have used the fine-tuning approach, where a simple classification layer is added to the pre-trained model, and layer in the output.
中文: 迄今介绍的所有BERT结果都采用了微调方法,即将一个简单的分级层添加到预先训练的模型中,并在输出中加入分层.
<a id="S0149"></a> Source: p.9 S0149
Original: We use the representation of all parameters are jointly fine-tuned on a down- the first sub-token as the input to the token-level stream task.
中文: 我们使用所有参数的表示值 在一个下调上共同微调—— 第一个子调值作为符号级流任务的投入.
<a id="S0150"></a> Source: p.9 S0150
Original: However, the feature-based approach, classifier over the NER label set. where fixed features are extracted from the pre- To ablate the fine-tuning approach, we apply the trained model, has certain advantages.
中文: 然而,基于特性的方法,在NER标签集之上进行分级. 从预 为了放弃微调方法,我们采用了经过训练的模型,具有某些优点。
<a id="S0151"></a> Source: p.9 S0151
Original: First, not feature-based approach by extracting the activaall tasks can be easily represented by a Transtions from one or more layers without fine-tuning former encoder architecture, and therefore require any parameters of BERT.
中文: 首先,不通过取出Activaall任务来进行以特征为基础的方法,很容易被一个从一层或多层取出而无需对前编码器架构进行微调的转录所代表,因此需要BERT的任何参数.
<a id="S0152"></a> Source: p.9 S0152
Original: These contextual ema task-specific model architecture to be added. beddings are used as input to a randomly initial- Second, there are major computational benefits ized two-layer 768-dimensional BiLSTM before to pre-compute an expensive representation of the the classification layer. training data once and then run many experiments Results are presented in Table 7.
中文: 这些背景的ema任务特有模式架构需要添加. 被褥被用作随机初始的输入 -- 其次,在预先计算出一个昂贵的分类层代表之前,存在将双层768维比LSTM分解的主要计算效益. 培训数据一次又一次进行许多实验,结果见表7。
<a id="S0153"></a> Source: p.9 S0153
Original: BERT with cheaper models on top of this representation.
中文: BERT在这个代表的外加更便宜的模型.
<a id="S0154"></a> Source: p.9 S0154
Original: LARGE performs competitively with state-of-the-art meth- In this section, we compare the two approaches ods.
中文: LARGE与最先进的冰毒竞争 在本节中,我们比较两种方法。
<a id="S0155"></a> Source: p.9 S0155
Original: The best performing method concatenates the by applying BERT to the CoNLL-2003 Named token representations from the top four hidden lay- Entity Recognition (NER) task (Tjong Kim Sang ers of the pre-trained Transformer, which is only and De Meulder, 2003).
中文: 最佳表现方法通过将BERT应用到CoNLL-2003上,从前4个隐藏实体识别(NER)任务(Tjong Kim Sang ers of the pre-trained Transformer,仅此而已和De Meulder,2003年)中命名了符号表示.
<a id="S0156"></a> Source: p.9 S0156
Original: In the input to BERT, we 0.3 F1 behind fine-tuning the entire model.
中文: 在给BERT的输入中,我们0.3 F1在精细调整整个模型后.
<a id="S0157"></a> Source: p.9 S0157
Original: This use a case-preserving WordPiece model, and we demonstrates that BERT is effective for both fineinclude the maximal document context provided tuning and feature-based approaches. by the data.
中文: 这使用了保存案例的WordPiece模型,我们表明,BERT对两种精细都有效,包括所提供的最大文档背景和基于特征的方法。 根据数据。
<a id="S0158"></a> Source: p.9 S0158
Original: Following standard practice, we formulate this as a tagging task but do not use a CRF 6 Conclusion Hyperparams Dev Set Accuracy Recent empirical improvements due to transfer #L #H #A LM (ppl) MNLI-m MRPC SST-2 learning with language models have demonstrated 3 768 12 5.84 77.9 79.8 88.4 6 768 3 5.24 80.6 82.2 90.7 that rich, unsupervised pre-training is an integral 6 768 12 4.68 81.9 84.8 91.3 part of many language understanding systems.
中文: 按照标准做法,我们将此作为标记任务,但不使用通用报告格式6的结论Hyperparams Dev Set精确度。 最近由于转让#L#H#A LM(ppl) MNLI-m MRPC SST-2语言模型的学习,经验改进了3 768 12 5.84 77.9 79.8 88.4 6 768 3 5.24 80.6 82.2。 90.7 丰富而不受监督的预训是许多语言理解系统的一部分,是其中的一个组成部分。
<a id="S0159"></a> Source: p.9 S0159
Original: In 12 768 12 3.99 84.4 86.7 92.9 particular, these results enable even low-resource 12 1024 16 3.54 85.7 86.9 93.3 24 1024 16 3.23 86.6 87.8 93.7 tasks to benefit from deep unidirectional architectures.
中文: 12 768 12 3.99 84.4 86.7 (英语). 92.9 特别是,这些成果使即使是低资源12 1024 16 3.54 85.7 86.9 93.3 24 1024 16 3.23 86.6 87.8 93.7 项任务能够受益于深层单向结构。
<a id="S0160"></a> Source: p.9 S0160
Original: Our major contribution is further general- Table 6: Ablation over BERT model size. #L = the izing these findings to deep bidirectional architecnumber of layers; #H = hidden size; #A = number of at- tures, allowing the same pre-trained model to suctention heads. “LM (ppl)” is the masked LM perplexity cessfully tackle a broad set of NLP tasks. of held-out training data.
中文: 我们的主要贡献是进一步的一般表6:超出BERT模型规模。 #L = 这些发现的大小为深双向拱形地层数;#H = 隐藏大小;#A = 在- 取出相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相 “LM(ppl)”是掩盖的LM模糊性,必须处理一系列广泛的NLP任务。 培训数据。
<a id="S0161"></a> Source: p.10 S0161
Original: References Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018.
中文: 参考:凯文·克拉克(英语:Kevin Clark, Minh-Thang Luong),克里斯多夫·德·曼宁(英语:Christopher D Manning)和克多克·勒. 2018.
<a id="S0162"></a> Source: p.10 S0162
Original: Semi-supervised se- Alan Akbik, Duncan Blythe, and Roland Vollgraf. quence modeling with cross-view training.
中文: 半被监督的se-艾伦·阿克比克(Alan Akbik),邓肯·布莱斯(Duncan Blythe)和罗兰·沃尔格拉夫(Roland Vollgraf.)以交叉取景训练为定型.
<a id="S0163"></a> Source: p.10 S0163
Original: Contextual string embeddings for sequence ceedings of the 2018 Conference on Empirical Methlabeling.
中文: 2018年经验性 Methlabeling会议序列跳跃的上下文字符串嵌入.
<a id="S0164"></a> Source: p.10 S0164
Original: In Proceedings of the 27th International ods in Natural Language Processing, pages 1914– Conference on Computational Linguistics, pages 1925. 1638–1649.
中文: 《自然语言处理第27届国际日记》,第1914页 -- -- 计算语言学会议,第1925页。
<a id="S0165"></a> Source: p.10 S0165
Original: Ronan Collobert and Jason Weston. 2008. A unified Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy architecture for natural language processing: Deep Guo, and Llion Jones. 2018.
中文: 罗南·科洛伯特和杰森·韦斯顿. 2008 (英语). 一个统一的拉米·阿尔-Rfou,多克克克·乔克,诺亚·康斯坦特,曼迪建筑用于自然语言处理:Deep Guo,和Llion Jones. 2018.
<a id="S0166"></a> Source: p.10 S0166
Original: Character-level lan- neural networks with multitask learning.
中文: 有多任务学习的字符级lan-神经网络.
<a id="S0167"></a> Source: p.10 S0167
Original: In Proguage modeling with deeper self-attention. arXiv ceedings of the 25th international conference on preprint arXiv:1808.04444.
中文: 在"Proguage"中以更深的自觉来建模. arXiv 第25届国际会议预印版 arXiv:1808.0444.
<a id="S0168"></a> Source: p.10 S0168
Original: Rie Kubota Ando and Tong Zhang. 2005. A framework Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo¨ıc for learning predictive structures from multiple tasks Barrault, and Antoine Bordes. 2017.
中文: (原始内容存档于2017-10-12). Rie Kubota Ando and Tong Zhang. 2005. 一个框架 Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo QQc 用于从多个任务中学习预测结构 Barrault,以及Antoine Bordes. 2017 (英语).
<a id="S0169"></a> Source: p.10 S0169
Original: Journal of Machine Learning learning of universal sentence representations from Research, 6(Nov):1817–1853. natural language inference data.
中文: "机器学习杂志","研究界普遍句子表述",6(Nov):1817–1853. 自然语言推论数据.
<a id="S0170"></a> Source: p.10 S0170
Original: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copen- Luisa Bentivogli, Bernardo Magnini, Ido Dagan, hagen, Denmark.
中文: 《2017年自然语言处理经验方法会议纪要》,第670-680页,Copen-Luisa Bentivogli、Bernardo Magnini、Ido Dagan、Hagen,丹麦。
<a id="S0171"></a> Source: p.10 S0171
Original: Association for Computational Hoa Trang Dang, and Danilo Giampiccolo. 2009.
中文: 计算协会 Hoa Trang Dang, and Danilo Giampicolo, 2009.
<a id="S0172"></a> Source: p.10 S0172
Original: The fifth PASCAL recognizing textual entailment challenge.
中文: 第五个PASCAL承认在文字上存在挑战。
<a id="S0173"></a> Source: p.10 S0173
Original: In Advances in neural informa- John Blitzer, Ryan McDonald, and Fernando Pereira. tion processing systems, pages 3079–3087. 2006.
中文: (原始内容存档于2013-10-12). In Advances in neural informationa - John Blitzer, Ryan McDonald, and Fernando Pereira. 通化处理系统, 第3079-3087页. 2006.
<a id="S0174"></a> Source: p.10 S0174
Original: Domain adaptation with structural correspondence learning.
中文: 域与结构函授学习相适应.
<a id="S0175"></a> Source: p.10 S0175
Original: Feience on empirical methods in natural language pro- Fei. 2009.
中文: Feience 关于自然语言中的经验方法 pro-Fei. 2009.
<a id="S0176"></a> Source: p.10 S0176
Original: ImageNet: A Large-Scale Hierarchical cessing, pages 120–128.
中文: ImageNet:一个大尺度的分级失败,第120-128页.
<a id="S0177"></a> Source: p.10 S0177
Original: Association for Computa- Image Database.
中文: Computa-图像数据库协会.
<a id="S0178"></a> Source: p.10 S0178
Original: William B Dolan and Chris Brockett. 2005.
中文: 威廉·B·多兰和克里斯·布洛克特. 2005.
<a id="S0179"></a> Source: p.10 S0179
Original: Bowman, Gabor Angeli, Christopher Potts, cally constructing a corpus of sentential paraphrases. and Christopher D.
中文: 鲍曼, Gabor Angeli, Christopher Potts, Calty 构筑了一套具有传记性的口语. (原始内容存档于2018-09-26). and Christopher D.
<a id="S0180"></a> Source: p.10 S0180
Original: Manning. 2015. A large anno- In Proceedings of the Third International Workshop tated corpus for learning natural language inference. on Paraphrasing (IWP2005).
中文: Manning. 2015. 第三次国际研讨会记录中学习自然语言推论的大型无声词集. 注释(IWP2005)。
<a id="S0181"></a> Source: p.10 S0181
Original: Association for Computational Linguistics.
中文: 计算语言学协会.
<a id="S0182"></a> Source: p.10 S0182
Original: William Fedus, Ian Goodfellow, and Andrew M Dai. 2018.
中文: 威廉·费杜斯(William Fedus),伊恩·古德费洛(Ian Goodfellow)和安德鲁·M·戴(Andrew M Dai. 2018.
<a id="S0183"></a> Source: p.10 S0183
Original: Maskgan: Better text generation via filling in Peter F Brown, Peter V Desouza, Robert L Mercer, the . arXiv preprint arXiv:1801.07736.
中文: 马斯克干:通过填写彼得·F·布朗,彼得·V·德苏扎,罗伯特·勒·默瑟等. arXiv preprint arXiv:1801.07736.
<a id="S0184"></a> Source: p.10 S0184
Original: Vincent J Della Pietra, and Jenifer C Lai. 1992.
中文: Vincent J Della Pietra, and Jenifer C Lai. 1992 (英语).
<a id="S0185"></a> Source: p.10 S0185
Original: Class-based n-gram models of natural language.
中文: 以班为主的n-gram自然语言模型.
<a id="S0186"></a> Source: p.10 S0186
Original: Bridging Computational linguistics, 18(4):467–479. nonlinearities and stochastic regularizers with gaussian error linear units.
中文: 桥接计算语言学, 18(4): 467–479. 非线性与有gaussian误差线性单位的相通调节器.
<a id="S0187"></a> Source: p.10 S0187
Original: Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016.
中文: 丹尼尔·瑟(Daniel Cer),莫娜·迪亚布(Mona Diab),埃内克·阿吉尔(Eneko Agirre),伊尼戈·洛佩斯-菲利克斯·希尔(Inigo Lopez-Felix Hill),克庆贤·乔(Kyunghyun Cho)和安娜·克霍宁(Anna Korhonen. 2016.
<a id="S0188"></a> Source: p.10 S0188
Original: Semeval-2017 Learning distributed representations of sentences task 1: Semantic textual similarity multilingual and from unlabelled data.
中文: 塞梅瓦尔2017年 学习分布式的句子表述任务一:语义文字相似多语种并取自无标签的数据.
<a id="S0189"></a> Source: p.10 S0189
Original: In Proceedings of the 2016 crosslingual focused evaluation.
中文: 2016年跨语言重点评价记录.
<a id="S0190"></a> Source: p.10 S0190
Original: In Proceedings Conference of the North American Chapter of the of the 11th International Workshop on Semantic Association for Computational Linguistics: Human Evaluation (SemEval-2017), pages 1–14, Vancou- Language Technologies.
中文: 在"计算语言学语义协会:人类评价"(SemEval-2017)第11届国际研讨会北美分会的议事录会议上,第1至14页,万古-语言技术.
<a id="S0191"></a> Source: p.10 S0191
Original: Association for Computational Lintional Linguistics. guistics.
中文: 计算入声语言学协会. 吉斯语。
<a id="S0192"></a> Source: p.10 S0192
Original: Jeremy Howard and Sebastian Ruder. 2018.
中文: 杰里米·霍华德和塞巴斯蒂安·鲁德. 2018.
<a id="S0193"></a> Source: p.10 S0193
Original: Universal Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, language model fine-tuning for text classification.
中文: Universal Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, 语言模型为文本分类进行微调.
<a id="S0194"></a> Source: p.10 S0194
Original: In Thorsten Brants, Phillipp Koehn, and Tony Robin- ACL.
中文: 在"索斯滕"(Thorsten Blants),"菲利普·克恩"(Phillipp Koehn),"托尼·罗宾"(Tony Robin-ACL.
<a id="S0195"></a> Source: p.10 S0195
Original: Association for Computational Linguistics. son. 2013.
中文: 计算语言学协会,儿子,2013年。
<a id="S0196"></a> Source: p.10 S0196
Original: One billion word benchmark for measuring progress in statistical language modeling. arXiv Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, preprint arXiv:1312.3005.
中文: 衡量统计语言建模进展的10亿字基准. arXiv Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Quiu,预印arXiv:1312.3005.
<a id="S0197"></a> Source: p.10 S0197
Original: Reinforced mnemonic reader for machine reading comprehen- Z.
中文: 强化了机器读取读取器
<a id="S0198"></a> Source: p.10 S0198
Original: Bowman, and David Son- Christopher Clark and Matt Gardner. 2018.
中文: 鲍曼,还有大卫·桑-克里斯托弗·克拉克和马特·加德纳. 2018.
<a id="S0199"></a> Source: p.10 S0199
Original: Discourse-based objectives for fast unand effective multi-paragraph reading comprehen- supervised sentence representation learning.
中文: 基于演讲的目标,即快速而无效果的多段读取 Comprehen 监督的句子表达学习。
<a id="S0200"></a> Source: p.11 S0200
Original: Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Matthew Peters, Mark Neumann, Luke Zettlemoyer, Zettlemoyer. 2017.
中文: 曼达尔·乔希,恩索尔·崔,丹尼尔·斯·韦尔德,以及卢克·马修·彼得斯,马克·诺伊曼,卢克·泽特勒莫耶,泽特勒莫耶. 2017.
<a id="S0201"></a> Source: p.11 S0201
Original: Triviaqa: A large scale distantly and Wen-tau Yih. 2018b.
中文: Triviaqa:一幅相去甚远和文-陶·易. 2018b.
<a id="S0202"></a> Source: p.11 S0202
Original: Dissecting contextual supervised challenge dataset for reading comprehen- word embeddings: Architecture and representation. sion.
中文: 解析用于读取Comprehen-单词嵌入的上下文所监管的挑战数据集:架构和表示. 锡安
<a id="S0203"></a> Source: p.11 S0203
Original: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, 1499–1509.
中文: 在"2018年自然语言处理经验方法会议纪要"中,页面有:Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, 1499–1509.
<a id="S0204"></a> Source: p.11 S0204
Original: Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.
中文: 理查德·泽梅尔,拉克尔·乌尔塔松,安东尼奥·托拉尔巴,和桑贾·菲德勒. 2015.
<a id="S0205"></a> Source: p.11 S0205
Original: In Alec Radford, Karthik Narasimhan, Tim Salimans, and Advances in neural information processing systems, Ilya Sutskever. 2018.
中文: 在Alec Radford,Karthik Narasimhan,Tim Salimans,以及神经信息处理系统的进步,Ilya Sutskever. 2018 (英语).
<a id="S0206"></a> Source: p.11 S0206
Original: Improving language underpages 3294–3302. standing with unsupervised learning.
中文: 改进语言下页3294–3302. 站立在无监督的学习.
<a id="S0207"></a> Source: p.11 S0207
Original: Distributed representations of sentences and documents.
中文: 分发判决和文件的说明。
<a id="S0208"></a> Source: p.11 S0208
Original: In Inter- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and national Conference on Machine Learning, pages Percy Liang. 2016.
中文: 在Inter-Pranav Rajpurkar中,张建章,康斯坦丁·洛皮雷夫,以及全国机器学习会议,Percy Liang. 2016.
<a id="S0209"></a> Source: p.11 S0209
Original: Squad: 100,000+ questions for 1188–1196. machine comprehension of text.
中文: Squad: 100 000+问题,用于1188–1196. 机器理解文本.
<a id="S0210"></a> Source: p.11 S0210
Original: In Proceedings of the 2016 Conference on Empirical Methods in Nat- Hector J Levesque, Ernest Davis, and Leora Morgenural Language Processing, pages 2383–2392. stern. 2011.
中文: 《2016年Nat-Hector J Levesque、Ernest Davis和Leora Morgenural Languages的经验方法会议记录》,第2383-2392页。
<a id="S0211"></a> Source: p.11 S0211
Original: In Aaai spring symposium: Logical formalizations of Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and commonsense reasoning, volume 46, page 47.
中文: 在"亚爱"春季座谈会上:明俊徐的逻辑形式化,Aniruddha Kembhavi, Ali Farhadi,和常识推理 第46卷 第47页.
<a id="S0212"></a> Source: p.11 S0212
Original: Bidirectional attention flow for machine comprehension.
中文: 双向注意力流 机器理解。
<a id="S0213"></a> Source: p.11 S0213
Original: Lajanugen Logeswaran and Honglak Lee. 2018.
中文: 拉雅努根·洛热斯瓦兰和洪拉克·李. 2018.
<a id="S0214"></a> Source: p.11 S0214
Original: An efficient framework for learning sentence represen- Richard Socher, Alex Perelygin, Jean Wu, Jason tations.
中文: 学习句子的高效框架 recresen-Richard Socher, Alex Perelygin, Jean Wu, Jason tuts.
<a id="S0215"></a> Source: p.11 S0215
Original: In International Conference on Learning Chuang, Christopher D Manning, Andrew Ng, and Representations.
中文: 在学习Chuang国际会议上,克里斯托弗·德·曼宁、安德鲁·恩格和代表处。
<a id="S0216"></a> Source: p.11 S0216
Original: Recursive deep models for semantic compositionality over a sentiment tree- Bryan McCann, James Bradbury, Caiming Xiong, and bank.
中文: 语义成分的回溯性深层模型 凌驾于情感树上 布莱恩·麦坎 詹姆斯·布拉德伯里 凯明·西翁和银行
<a id="S0217"></a> Source: p.11 S0217
Original: In Proceedings of the 2013 conference on Richard Socher. 2017.
中文: 在2013年关于理查德·索彻的会议记录中. 2017.
<a id="S0218"></a> Source: p.11 S0218
Original: Learned in translation: Conempirical methods in natural language processing, textualized word vectors.
中文: 在翻译方面学到了:自然语言处理中的Connempical方法,文本化的单词矢量.
<a id="S0219"></a> Source: p.11 S0219
Original: Oren Melamud, Jacob Goldberger, and Ido Dagan.
中文: 奥伦·梅拉穆德,雅各布·戈德伯格,和伊多·达甘.
<a id="S0220"></a> Source: p.11 S0220
Original: Fu Sun, Linyang Li, Xipeng Qiu, and Yang Liu. 2016. context2vec: Learning generic context em- 2018. U-net: Machine reading comprehension bedding with bidirectional LSTM.
中文: 傅晨; 林阳李; 西平邱; 杨刘. 2016. 上下文2vec:学习通用上下文em-2018. U-net:机器读取理解被褥 双向LSTM.
<a id="S0221"></a> Source: p.11 S0221
Original: In CoNLL. with unanswerable questions. arXiv preprint Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- arXiv:1810.06638. rado, and Jeff Dean. 2013.
中文: 在CONLL中,有无法回答的问题. arXiv预印托马斯·米科洛夫,伊利亚·苏特斯克韦尔,凯·陈,格雷格·S·科尔-arXiv:181.06638. rado,和杰夫·迪恩. 2013.
<a id="S0222"></a> Source: p.11 S0222
Original: Distributed representa- Wilson L Taylor. 1953.
中文: 1953年 威尔逊·L·泰勒
<a id="S0223"></a> Source: p.11 S0223
Original: Cloze procedure: A new tions of words and phrases and their compositionaltool for measuring readability.
中文: 克洛兹程序:用于测量可读性的新词语和短语及其组成工具。
<a id="S0224"></a> Source: p.11 S0224
Original: In Advances in Neural Information Processing 30(4):415–433.
中文: 在神经信息处理方面的进步 30(4):415–433.
<a id="S0225"></a> Source: p.11 S0225
Original: Erik F Tjong Kim Sang and Fien De Meulder.
中文: Erik F Tjong Kim Sang和费恩·德·默德.
<a id="S0226"></a> Source: p.11 S0226
Original: Andriy Mnih and Geoffrey E Hinton. 2009. A scal- 2003.
中文: 安德里·姆尼和杰弗里·艾·欣顿. 2009 (英语). 一个2003年的Scal。
<a id="S0227"></a> Source: p.11 S0227
Original: Introduction to the conll-2003 shared task: able hierarchical distributed language model.
中文: Conll-2003共同任务简介:能分级分布的语言模式.
<a id="S0228"></a> Source: p.11 S0228
Original: In Language-independent named entity recognition.
中文: 在语言独立命名的实体识别中.
<a id="S0229"></a> Source: p.11 S0229
Original: Bot- CoNLL. tou, editors, Advances in Neural Information Pro- Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. cessing Systems 21, pages 1081–1088.
中文: Bot-CONLL. tou,编辑,神经信息预告 Pro-Joseph Turian,列夫·拉蒂诺夫和约斯华·本吉奥. 2010. Sessing Systems 21, 第1081–1088页.
<a id="S0230"></a> Source: p.11 S0230
Original: Curran As- Word representations: A simple and general method sociates, Inc. for semi-supervised learning.
中文: Curran As-Word differences:一个简单而通俗的方法 sociates, Inc. 用于半监督学习.
<a id="S0231"></a> Source: p.11 S0231
Original: In Proceedings of the Ankur P Parikh, Oscar Ta¨ckstro¨m, Dipanjan Das, and 48th Annual Meeting of the Association for Compu- Jakob Uszkoreit. 2016. A decomposable attention tational Linguistics, ACL ’10, pages 384–394. model for natural language inference.
中文: Ankur Parikh、Oscar Ta'ckstro'm、Dipanjan Das和Compu-Jakob Uszkoreit协会第48次年会记录。 2016 (英语). 可分解的注意语法,ACL ' 10, 第384–394页. 自然语言推论的模型。
<a id="S0232"></a> Source: p.11 S0232
Original: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Jeffrey Pennington, Richard Socher, and Christo- Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz pher D.
中文: 阿希什·瓦斯瓦尼,诺姆·沙泽尔,尼基·帕尔马尔,雅可布·杰弗里·彭宁顿,理查德·索彻,和克丽斯多-乌斯克克赖特,利翁·琼斯,艾丹·恩·戈麦斯,卢卡斯茨 Pher D.
<a id="S0233"></a> Source: p.11 S0233
Original: Glove: Global vectors for Kaiser, and Illia Polosukhin. 2017.
中文: 手套:为Kaiser提供全球矢量,和Illia Polosukhin. 2017.
<a id="S0234"></a> Source: p.11 S0234
Original: In Advances in Neural Information Proural Language Processing (EMNLP), pages 1532– cessing Systems, pages 6000–6010. 1543.
中文: 在"神经信息Proural Languages"(EMNLP)中,第1532页-上塞系统,第6000-6010页. 1543.
<a id="S0235"></a> Source: p.11 S0235
Original: Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Matthew Peters, Waleed Ammar, Chandra Bhagavat- Pierre-Antoine Manzagol. 2008.
中文: Pascal Vincent, Hugo Larochelle, Yoshua Bengio,以及Matthew Peters, Waleed Ammar, Chandra Bhagavat-Pierre-Antoine Manzagol. 2008. 互联网档案馆的存檔,存档日期2008-12-21.
<a id="S0236"></a> Source: p.11 S0236
Original: Extracting and ula, and Russell Power. 2017.
中文: 取出和ULA,还有Russell Power. 2017.
<a id="S0237"></a> Source: p.11 S0237
Original: Semi-supervised se- composing robust features with denoising autoenquence tagging with bidirectional language models. coders.
中文: 半监督的se-编组了强健的特性,并用双向语言模型去诺化自定义标记. 密码员
<a id="S0238"></a> Source: p.11 S0238
Original: In Proceedings of the 25th international In ACL. conference on Machine learning, pages 1096–1103.
中文: 《第25届国际法学会关于机器学习的会议记录》,第1096-1103页。
<a id="S0239"></a> Source: p.11 S0239
Original: Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Alex Wang, Amanpreet Singh, Julian Michael, Fe- Zettlemoyer. 2018a.
中文: 马修·彼得斯,马克·诺伊曼,莫希特·伊耶尔,马特·加德纳,克里斯托弗·克拉克,肯顿·李,以及卢克·亚历克·王,阿曼普雷特·辛格,朱利安·迈克尔,费-泽特尔莫耶. 2018a.
<a id="S0240"></a> Source: p.11 S0240
Original: Deep contextualized word rep- lix Hill, Omer Levy, and Samuel Bowman. 2018a. resentations.
中文: 深层背景化名词 rep-lix Hill, Omer Levy, 和塞缪尔·鲍曼. 2018a. 不满.
<a id="S0241"></a> Source: p.11 S0241
Original: Glue: A multi-task benchmark and analysis platform
中文: Glue: 多任务基准和分析平台
<a id="S0242"></a> Source: p.12 S0242
Original: In Proceedings • Additional details for our experiments are of the 2018 EMNLP Workshop BlackboxNLP: An- presented in Appendix B; and alyzing and Interpreting Neural Networks for NLP, pages 353–355. • Additional ablation studies are presented in Appendix C.
中文: 在"议事录"中,我们实验的其他细节有:2018年EMNLP Work BlackboxNLP:An-在附录B中介绍;为NLP分析和解释神经网络,353–355页. • 附录C中载有其他的消化研究。
<a id="S0243"></a> Source: p.12 S0243
Original: Multigranularity hierarchical attention fusion networks We present additional ablation studies for for reading comprehension and question answering.
中文: 多相分级关注聚变网络 我们提出更多关于阅读理解和回答问题的研究。
<a id="S0244"></a> Source: p.12 S0244
Original: BERT including: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: – Effect of Number of Training Steps; and Long Papers).
中文: BERT,包括: 计算语言学协会第56届年会记录(第1卷:培训步骤次数的影响;和长篇论文)。
<a id="S0245"></a> Source: p.12 S0245
Original: Association for Computational Linguistics. – Ablation for Different Masking Procedures.
中文: 计算语言学协会。
<a id="S0246"></a> Source: p.12 S0246
Original: Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018.
中文: Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018 (英语).
<a id="S0247"></a> Source: p.12 S0247
Original: Neural network acceptability judg- A Additional Details for BERT ments. arXiv preprint arXiv:1805.12471. A.1 Illustration of the Pre-training Tasks Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus We provide examples of the pre-training tasks in for sentence understanding through inference.
中文: 神经网络可接受性judg - BERT颗粒的附加细节. arXiv preprint arXiv:1805.12471. A.1 预训任务说明 阿迪娜·威廉斯,尼基塔·南吉亚和塞缪尔·R·鲍曼. 2018 (英语). 涵盖面广的质疑书 我们举例说明通过推论理解判决的预先培训任务。
<a id="S0248"></a> Source: p.12 S0248
Original: Masked LM and the Masking Procedure As- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V suming the unlabeled sentence is my dog is Le, Mohammad Norouzi, Wolfgang Macherey, hairy, and during the random masking procedure Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.
中文: 蒙面LM和蒙面程序 As-Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V 概括未加标签的句子是:我的狗是Le, Mohammad Norouzi, Wolfgang Macherey, 有毛, 在随机蒙面程序期间有Maxim Krikun,袁曹,秦高,克勞斯·馬切雷等. 2016.
<a id="S0249"></a> Source: p.12 S0249
Original: Google’s neural ma- we chose the 4-th token (which corresponding to chine translation system: Bridging the gap between hairy), our masking procedure can be further ilhuman and machine translation. arXiv preprint lustrated by arXiv:1609.08144. • 80% of the time: Replace the word with the Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod [MASK] token, e.g., my dog is hairy → Lipson. 2014.
中文: Google的神经元Ma——我们选择了第4个符号(这与中国的翻译系统相对应:缩小毛发之间的鸿沟),我们的面具程序可以进一步非人和机器翻译. arXiv预印由arXiv:1609.08144. ^ 80%时间:用Jason Yosinski, Jeff Clune, Yoshua Bengio,和Hod [MASK] sorry来代替这个词,例如我的狗有毛-Lipson. 2014 (中文(简体) ).
<a id="S0250"></a> Source: p.12 S0250
Original: How transferable are features in deep neural networks?
中文: 深度神经网络的特征如何可转移?
<a id="S0251"></a> Source: p.12 S0251
Original: In Advances in neural information my dog is [MASK] processing systems, pages 3320–3328. • 10% of the time: Replace the word with a Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui random word, e.g., my dog is hairy → my Zhao, Kai Chen, Mohammad Norouzi, and Quoc V dog is apple Le. 2018.
中文: 在神经信息方面的进步 我的狗是[MASK]处理系统,第3320–3328页。 ^ 10% 时间:用亚当斯·魏羽,大卫·达温,明-唐·罗荣,瑞随机等词取而代之,例如:我的狗有毛-我的赵克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克
<a id="S0252"></a> Source: p.12 S0252
Original: QANet: Combining local convolution with global self-attention for reading comprehen- • 10% of the time: Keep the word unsion.
中文: QANet:将本地化的进取与全球自觉阅读Comprehen结合——-10%时间:保留单词放出.
<a id="S0253"></a> Source: p.12 S0253
Original: In ICLR. changed, e.g., my dog is hairy → my dog Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin is hairy.
中文: 在ICLR中变化了,例如我的狗有毛——我的狗Rowan Zellers,Yonatan Bisk,Roy Schwartz,和Yejin有毛.
<a id="S0254"></a> Source: p.12 S0254
Original: The purpose of this is to bias the Choi. 2018.
中文: 此举的目的是偏向"崔". 2018.
<a id="S0255"></a> Source: p.12 S0255
Original: Swag: A large-scale adversarial dataset representation towards the actual observed for grounded commonsense inference.
中文: Swag:针对基于常识推论的实际观察到的大规模对抗数据集表示。
<a id="S0256"></a> Source: p.12 S0256
Original: In Proceedword. ings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
中文: 继续说 2018年自然语言处理经验方法会议 (EMNLP).
<a id="S0257"></a> Source: p.12 S0257
Original: The advantage of this procedure is that the Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- Transformer encoder does not know which words dinov, Raquel Urtasun, Antonio Torralba, and Sanja it will be asked to predict or which have been re- Fidler. 2015.
中文: 这个程序的优点是: 尤昆·朱 瑞安·基罗斯 里奇·泽梅尔 罗斯兰·萨拉克胡特 变形器编码器不知道哪个词是dinov,Raquel Urtasun,Antonio Torralba,以及Sanja,它将被要求来预测,或者被重新用到Fidler. 2015 (英语).
<a id="S0258"></a> Source: p.12 S0258
Original: Aligning books and movies: Towards placed by random words, so it is forced to keep story-like visual explanations by watching movies a distributional contextual representation of evand reading books.
中文: 校正书籍和电影:朝向被随机文字所放置的方向,因此它被迫通过观看电影来保持类似故事的视觉解释,以发行上下文来代表ev和阅读书籍.
<a id="S0259"></a> Source: p.12 S0259
Original: In Proceedings of the IEEE international conference on computer vision, pages ery input token.
中文: 在IEEE计算机视觉国际会议的会议记录中,页刻入符号。
<a id="S0260"></a> Source: p.12 S0260
Original: Additionally, because random 19–27. replacement only occurs for 1.5% of all tokens (i.e., 10% of 15%), this does not seem to harm Appendix for “BERT: Pre-training of the model’s language understanding capability.
中文: 此外,由于随机的19–27. 替换只用于1.5%的所有代币(即15%的10%),这似乎不影响附录“BERT:模型语言理解能力预训 ” 。
<a id="S0261"></a> Source: p.12 S0261
Original: In Deep Bidirectional Transformers for Section C.2, we evaluate the impact this proce- Language Understanding” dure.
中文: 在C.2节的 " 深度双向变形器 " 中,我们评价这种亲子语言理解的影响。
<a id="S0262"></a> Source: p.12 S0262
Original: We organize the appendix into three sections: Compared to standard langauge model training, the masked LM only make predictions on 15% of • Additional implementation details for BERT tokens in each batch, which suggests that more are presented in Appendix A; pre-training steps may be required for the model
中文: 我们把附录分成三节: 与标准装潢模型培训相比,被遮住的LM只预测15% – 每批BERT标志的额外实施细节,这说明附录A中有更多的内容;该模型可能需要培训前步骤.
<a id="S0263"></a> Source: p.13 S0263
Original: EN Figure 3: Differences in pre-training model architectures.
中文: EN 图3:培训前模式架构的差异。
<a id="S0264"></a> Source: p.13 S0264
Original: OpenAI GPT uses a left-to-right Transformer.
中文: OpenAI GPT使用从左到右的变形器.
<a id="S0265"></a> Source: p.13 S0265
Original: ELMo uses the concatenation of independently trained left-to-right and right-toleft LSTMs to generate features for downstream tasks.
中文: ELMo使用被独立训练出左到右和右到左的LSTMs的集合来生成下游任务的特性.
<a id="S0266"></a> Source: p.13 S0266
Original: Among the three, only BERT representations are jointly conditioned on both left and right context in all layers.
中文: 在三个方面中,只有BERT的表述在所有层面都以左上下文和右上下文为共同条件。
<a id="S0267"></a> Source: p.13 S0267
Original: In addition to the architecture differences, BERT and OpenAI GPT are fine-tuning approaches, while ELMo is a feature-based approach. to converge.
中文: 除了架构差异之外,BERT和OpenAI GPT都是微调方法,而ELMo则是基于地物的方法. {\fn华文仿宋\fs16\1cHD1D1D1}会合
<a id="S0268"></a> Source: p.13 S0268
Original: In Section C.1 we demonstrate that epochs over the 3.3 billion word corpus.
中文: 在C.1节中,我们展示了33亿字体的时代。
<a id="S0269"></a> Source: p.13 S0269
Original: We MLM does converge marginally slower than a left- use Adam with learning rate of 1e-4, β = 0.9, 1 to-right model (which predicts every token), but β = 0.999, L2 weight decay of 0.01, learning 2 the empirical improvements of the MLM model rate warmup over the first 10,000 steps, and linear far outweigh the increased training cost. decay of the learning rate.
中文: 我们的MLM确实比左用亚当的学习速率稍慢一点, β = 0.9, 1到右用模型(它预测每个符号), 但是β = 0.999, L2 重量衰减 0.01, 学习 2 MLM模型在前一万个步骤上的经验性改进, 线性远远超过增加的训练成本. 学习率的衰减。
<a id="S0270"></a> Source: p.13 S0270
Original: We use a dropout probability of 0.1 on all layers.
中文: 我们使用0.1的退学概率 所有层次。
<a id="S0271"></a> Source: p.13 S0271
Original: We use a gelu acti- Next Sentence Prediction The next sentence vation (Hendrycks and Gimpel, 2016) rather than prediction task can be illustrated in the following the standard relu, following OpenAI GPT.
中文: 我们使用Gelu acti - 下一个判决预测 下句活字(Hendrycks and Gimpel, 2016),而非预测任务,可以在OpenAI GPT之后的以下标准relu中说明.
<a id="S0272"></a> Source: p.13 S0272
Original: The examples. training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction Input = [CLS] the man went to [MASK] store [SEP] likelihood. he bought a gallon [MASK] milk [SEP] Training of BERT was performed on 4 BASE Label = IsNext Cloud TPUs in Pod configuration (16 TPU chips total).13 Training of BERT was performed LARGE on 16 Cloud TPUs (64 TPU chips total).
中文: 诸相例. 训练损失是平均被遮住的LM概率和下句的平均值预测输入=[CLS] 男子去[MASK]存储[SEP]概率的总和. 他买了一加仑 [MASK] 牛奶 [SEP] 对BERT进行了4个BASE标签=IsNext Cloud TPU在Pod配置方面的培训(共16个TPU芯片)。 13 对BERT进行了16个Cloud TPU(共64个TPU芯片)的培训。
<a id="S0273"></a> Source: p.13 S0273
Original: Each pre- Input = [CLS] the man [MASK] to the store [SEP] training took 4 days to complete. penguin [MASK] are flight ##less birds [SEP] Longer sequences are disproportionately expen- Label = NotNext sive because attention is quadratic to the sequence length.
中文: 每一次预输入=[CLS] 男 [MASK]去商店[SEP]训练需要4天才能完成. 企鹅[MASK]是飞行□□的无花鸟[SEP]. 较长的序列多为外延-Label = 非外延活性,因为注意力对序列长度是四相的.
<a id="S0274"></a> Source: p.13 S0274
Original: To speed up pretraing in our experiments, A.2 Pre-training Procedure we pre-train the model with sequence length of To generate each training input sequence, we sam- 128 for 90% of the steps.
中文: 为了加快我们实验的预导,A.2预导程序, 我们预导训练模型的序列长度 以生成每个训练输入序列, 我们萨姆 -128为90%的步骤。
<a id="S0275"></a> Source: p.13 S0275
Original: Then, we train the rest ple two spans of text from the corpus, which we 10% of the steps of sequence of 512 to learn the refer to as “sentences” even though they are typ- positional embeddings. ically much longer than single sentences (but can be shorter also).
中文: 然后,我们从本体中训练其余的两段文字,我们用512的顺序步骤的10%来学习所谓的“判决”,尽管它们是打字位置嵌入。 本质上比单句长得多(但也可以更短).
<a id="S0276"></a> Source: p.13 S0276
Original: The first sentence receives the A A.3 Fine-tuning Procedure embedding and the second receives the B embed- For fine-tuning, most model hyperparameters are ding. 50% of the time B is the actual next sentence the same as in pre-training, with the exception of that follows A and 50% of the time it is a random the batch size, learning rate, and number of trainsentence, which is done for the “next sentence pre- ing epochs.
中文: 第一句是A.3精细调整程序嵌入,第二句是B嵌入 -- 对于微调,大多数型号的超参数都为丁. B时间的50%是实际的下句与预训相同,但A时间之后的50%是随机的分批大小,学习率,以及列车判决次数除外,这是针对"下句预训"的.
<a id="S0277"></a> Source: p.13 S0277
Original: The dropout probability was always diction” task.
中文: 辍学的概率总是词典“任务。
<a id="S0278"></a> Source: p.13 S0278
Original: They are sampled such that the com- kept at 0.1.
中文: 它们被取样后,相机保持在0.1。
<a id="S0279"></a> Source: p.13 S0279
Original: The optimal hyperparameter values bined length is ≤ 512 tokens.
中文: 最佳的超参数值是 \ 512 个符号 。
<a id="S0280"></a> Source: p.13 S0280
Original: The LM masking is are task-specific, but we found the following range applied after WordPiece tokenization with a uni- of possible values to work well across all tasks: form masking rate of 15%, and no special consideration given to partial word pieces. • Batch size: 16, 32 We train with batch size of 256 sequences (256 13https://cloudplatform.googleblog.com/2018/06/Cloudsequences * 512 tokens = 128,000 tokens/batch) TPU-now-offers-preemptible-pricing-and-globalfor 1,000,000 steps, which is approximately 40 availability.html
中文: LM 遮掩是任务特定,但我们发现WordPiece 标注后应用了以下范围,其中单值可能在所有任务中都有效: 形式遮掩率为15%,没有特别考虑部分单词块. • 批号:16 32 我们用256个序列(256 13https://cloudplatreform.googleblog.com/2018/06/Cloudsequences *512个令牌=12.8万个令牌/批发)的分批量列车 TPU-now-offers-prefulable-price-和-global 为1,000,000个步骤,大约40个可用. 页面存档备份
<a id="S0281"></a> Source: p.14 S0281
Original: • Learning rate (Adam): 5e-5, 3e-5, 2e-5 To isolate the effect of these differences, we per- • Number of epochs: 2, 3, 4 form ablation experiments in Section 5.1 which demonstrate that the majority of the improvements We also observed that large data sets (e.g., are in fact coming from the two pre-training tasks 100k+ labeled training examples) were far less and the bidirectionality they enable. sensitive to hyperparameter choice than small data sets.
中文: • 学习率(亚当):5e-5、3e-5、2e-5 为了隔离这些差异的影响,我们按 -- -- 年数:2、3、4在第5.1节中形成衰减实验,这表明大多数改进 我们还注意到,大型数据集(例如,事实上来自两个培训前任务100k+标有标签的培训实例)要少得多,它们能够双向操作。 与小数据集相比,对超参数选择敏感。
<a id="S0282"></a> Source: p.14 S0282
Original: Fine-tuning is typically very fast, so it is rea- A.5 Illustrations of Fine-tuning on Different sonable to simply run an exhaustive search over Tasks the above parameters and choose the model that The illustration of fine-tuning BERT on different performs best on the development set. tasks can be seen in Figure 4.
中文: 微调一般非常快,因此,对不同子句的微调说明是Rea -- A.5, 简单详尽地搜索以上参数,并选择不同子句上微调BERT的插图在开发集上表现最好的模式。 任务见图4。
<a id="S0283"></a> Source: p.14 S0283
Original: Our task-specific models are formed by incorporating BERT with A.4 Comparison of BERT, ELMo ,and one additional output layer, so a minimal num- OpenAI GPT ber of parameters need to be learned from scratch.
中文: 我们的任务特异性模型是通过将BERT与A.4比较BERT,ELMo组成,并增加一个输出层,因此需要从零开始学习一个最小的num-OpenAI GPT标注参数.
<a id="S0284"></a> Source: p.14 S0284
Original: Here we studies the differences in recent popular Among the tasks, (a) and (b) are sequence-level representation learning models including ELMo, tasks while (c) and (d) are token-level tasks.
中文: 在此,我们研究最近流行的差别,在各项任务中,(a)和(b)是包括ELMo在内的序列级代表学习模式,而(c)和(d)是象征性的一级任务。
<a id="S0285"></a> Source: p.14 S0285
Original: The comparisons be- the figure, E represents the input embedding, T i tween the model architectures are shown visually represents the contextual representation of token i, in Figure 3.
中文: 比较是 - 图, E 代表输入嵌入, T i tween 模型架构显示为可视化的表示符i的上下文代表,见图3.
<a id="S0286"></a> Source: p.14 S0286
Original: Note that in addition to the architec- [CLS] is the special symbol for classification outture differences, BERT and OpenAI GPT are fine- put, and [SEP] is the special symbol to separate tuning approaches, while ELMo is a feature-based non-consecutive token sequences. approach. B Detailed Experimental Setup The most comparable existing pre-training method to BERT is OpenAI GPT, which trains a B.1 Detailed Descriptions for the GLUE left-to-right Transformer LM on a large text cor- Benchmark Experiments. pus.
中文: 注意:除了Architec-[CLS]是分类外出差异的特殊符号外,BERT和OpenAI GPT是细放的,[SEP]是分离调制方法的特殊符号,而ELMo则是以特征为基础的非相交符号序列. 办法。 B 详细实验设置 BERT现存最可比较的预训方法为OpenAI GPT,该方法在大型文本corp-Bork实验上为GLUE左向右变形器LM训练了一款B. 脓.
<a id="S0287"></a> Source: p.14 S0287
Original: In fact, many of the design decisions in BERT Our GLUE results in Table1 are obtained were intentionally made to make it as close to from https://gluebenchmark.com/ GPT as possible so that the two methods could be leaderboard and https://blog. minimally compared.
中文: 事实上,BERT Our GLUE中的许多设计决定是在表1中得到的,是有意使这些决定接近于https://gluebenchmark.com/. GPT 尽可能使两种方法成为主板和https://blog. 进行最小的比较.
<a id="S0288"></a> Source: p.14 S0288
Original: The core argument of this openai.com/language-unsupervised. work is that the bi-directionality and the two pre- The GLUE benchmark includes the following training tasks presented in Section 3.1 account for datasets, the descriptions of which were originally the majority of the empirical improvements, but summarized in Wang et al. (2018a): we do note that there are several other differences between how BERT and GPT were trained: MNLI Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classifi- • GPT is trained on the BooksCorpus (800M cation task (Williams et al., 2018).
中文: 这个openai.com/语言无监督的核心论点. 工作是双向和双先 GLUE基准包括第3.1节说明数据集的下列培训任务,这些数据集的描述最初是经验改进的大部分,但在Wang等人(2018年a)中作了总结: 我们确实注意到,BERT和GPT的培训方式还有其他几种不同: MNLI多基因自然语言推论是一个大规模,多源性入门类- – GPT在"BooksCorpus"(800M cation reduction (Williams等, 2018年))的培训.
<a id="S0289"></a> Source: p.14 S0289
Original: Given a pair of words); BERT is trained on the BooksCor- sentences, the goal is to predict whether the secpus (800M words) and Wikipedia (2,500M ond sentence is an entailment, contradiction, or words). neutral with respect to the first one. • GPT uses a sentence separator ([SEP]) and QQP Quora Question Pairs is a binary classificlassifier token ([CLS]) which are only in- cation task where the goal is to determine if two troduced at fine-tuning time; BERT learns questions asked on Quora are semantically equiv- [SEP], [CLS] and sentence A/B embed- alent (Chen et al., 2018). dings during pre-training.
中文: 给一对词;BERT接受"BooksCor-句子"的培训,目标是预测secpus(800M词)和"维基百科"(2,500M上句子)是必然,矛盾,还是单词. 对于第一个,是中立的。 • GPT使用句子分隔符([SEP])和QQP Quora Question Pairs是二进制分级符号([CLS]),它只是用于确定在微调时是否引入两个分级;BERT学习Quora上所询问的问题在表态上是等同的-[SEP],[CLS]和句子A/B嵌入-arent(Chen等,2018年)的分级任务. 预训期间的叮当.
<a id="S0290"></a> Source: p.14 S0290
Original: QNLI Question Natural Language Inference is a version of the Stanford Question Answering • GPT was trained for 1M steps with a batch Dataset (Rajpurkar et al., 2016) which has been size of 32,000 words; BERT was trained for converted to a binary classification task (Wang 1M steps with a batch size of 128,000 words. et al., 2018a).
中文: QNLI问题自然语言推论(英語:QNLI Question Natural Language Information)是斯坦福问题解答法的一款版本,GPT曾被训练为1M步骤并有批量数据集(Rajpurkar等,2016年),其尺寸为32,000个字;BERT被训练为二进制分级任务(Wang 1M步骤,批量为128,000个字. 等, 2018a).
<a id="S0291"></a> Source: p.14 S0291
Original: The positive examples are (ques- • GPT used the same learning rate of 5e-5 for tion, sentence) pairs which do contain the correct all fine-tuning experiments; BERT chooses a answer, and the negative examples are (question, task-specific fine-tuning learning rate which sentence) from the same paragraph which do not performs the best on the development set. contain the answer.
中文: 正面的例子有(ques--) GPT使用相同的5e-5的学习率来进行tion,句子)一对,它们确实包含正确的所有微调实验;BERT选择一个答案,而负面的例子则是(问题,任务特定的微调学习率哪句)从同一段落中不能在开发集中表现得最好. 包含答案。
<a id="S0292"></a> Source: p.15 S0292
Original: Class Class Label Label C T 1 ... T N T [SEP] T 1 ’ ... T M ’ C T 1 T 2 ... T N BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E [CLS] E 1 E 2 ... E N [CLS] T 1 ok ... T N ok [SEP] T 1 ok ... T M ok CCLLSS TTookk 11 Tok 2 ...
中文: 分类标签C T 1 ... T N T [SEP] T 1 ' ... T M ' T 1 T 2... T BERT BERT BERT E [CLS] E 1. E. E. E. M. [CLS] E. [CLS] E. E. E. [CLS] T. 1 好... T. [SEP] T. M. OK [[CCLLSS] T. K. 11 Tok 2...
<a id="S0293"></a> Source: p.15 S0293
Original: Tok N Sentence 1 Sentence 2 Single Sentence Start/End Span O B-PER ... O C T 1 ... T N T [SEP] T 1 ’ ... T M ’ C T 1 T 2 ... T N BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E [CLS] E 1 E 2 ... E N [CLS] T 1 ok ... T N ok [SEP] T 1 ok ... T M ok [CLS] Tok 1 Tok 2 ...
中文: Tok N 句子 1 句子 2 单句开始/结束 Span O B-PER... O C T 1... T N T T [SEP] T 1 ' T. T. M BERT BERT E [CLS] E 1 [CLS] E. E. [SEP] E. M. [CLS] E. E. E. E. E. E. E. E. E. E. E. E. E. E. E. [CLS] T. OK [SEP] T. OK 1 ok. [CLS] T. M. [CLS] Tok 1 Tok 2...
<a id="S0294"></a> Source: p.15 S0294
Original: Tok N Question Paragraph Single Sentence Figure 4: Illustrations of Fine-tuning BERT on Different Tasks.
中文: Tok N问题段落单句图4:关于不同任务的精细调整BERT的说明.
<a id="S0295"></a> Source: p.15 S0295
Original: SST-2 The Stanford Sentiment Treebank is a for whether the sentences in the pair are semantibinary single-sentence classification task consist- cally equivalent (Dolan and Brockett, 2005). ing of sentences extracted from movie reviews RTE Recognizing Textual Entailment is a biwith human annotations of their sentiment (Socher nary entailment task similar to MNLI, but with et al., 2013). much less training data (Bentivogli et al., 2009).14 CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where WNLI Winograd NLI is a small natural lanthe goal is to predict whether an English sentence guage inference dataset (Levesque et al., 2011). is linguistically “acceptable” or not (Warstadt The GLUE webpage notes that there are issues et al., 2018). with the construction of this dataset, 15 and every trained system that’s been submitted to GLUE has STS-B The Semantic Textual Similarity Benchperformed worse than the 65.1 baseline accuracy mark is a collection of sentence pairs drawn from of predicting the majority class.
中文: SST-2 Stanford Sentimment Treebank是用于说明对等的句子是否是分泌性单刑分类任务由-cally等同而来(Dolan and Brockett, 2005). 从电影评论中摘取的句子 RTE Enceptulation Textual Entailment 是人类对其情绪的描述(Socher nary completement required required required required to computing to MNLI, but with et al., 2013)的双行本. 培训数据要少得多(Bentivogli等人,2009年)。 语言可接受性(Corpus of Language Acceptility)是一个二进制的单判分类任务,WNLI Winograd NLI是一个小的自然线条,目标是预测一个英语句子guage推断数据集(Levesque等,2011年). 在语言上是否“可接受”(Warstadt) GLUE网页指出存在问题等,2018年. 随着这个数据集的构建,15个和每个提交给GLUE的训练有素的系统都有STS-B 语义文字相似率座椅表现比65.1基线精度标记差,这是从预测多数类中抽取的一对句子.
<a id="S0296"></a> Source: p.15 S0296
Original: We therefore exnews headlines and other sources (Cer et al., clude this set to be fair to OpenAI GPT.
中文: 因此,我们新闻头条和其他来源(Cer等人,这集对OpenAI GPT来说是公平的。
<a id="S0297"></a> Source: p.15 S0297
Original: They were annotated with a score from 1 GLUE submission, we always predicted the mato 5 denoting how similar the two sentences are in terms of semantic meaning. 14Note that we only report single-task fine-tuning results in this paper. A multitask fine-tuning approach could poten- MRPC Microsoft Research Paraphrase Corpus tially push the performance even further.
中文: 他们从1GLUE提交书中得到了一个分数,我们总是预言了mato 5在语义意义上两句的相似性. 14 注意本文件只报告单一任务微调结果。 多任务微调方法可以将性能推得更进一步。
<a id="S0298"></a> Source: p.15 S0298
Original: For example, we did observe substantial improvements on RTE from multiconsists of sentence pairs automatically extracted task training with MNLI. from online news sources, with human annotations 15https://gluebenchmark.com/faq
中文: 例如,我们确实从句子对等的多共通分子中观察到了RTE的重大改进,这些组合自动从MNLI中提取任务训练. 从在线新闻来源, 与人的说明 15https://gluebenchmark.com/faq (中文(简体) ).
<a id="S0299"></a> Source: p.16 S0299
Original: jority class. C Additional Ablation Studies C.1 Effect of Number of Training Steps Figure 5 presents MNLI Dev accuracy after finetuning from a checkpoint that has been pre-trained for k steps.
中文: 热恋课。 C. 额外校正研究 C.1 培训步骤次数的影响 图5显示MNLI Dev精准度,该精准度在经过K级预训的检查站后得到微调。
<a id="S0300"></a> Source: p.16 S0300
Original: This allows us to answer the following questions: 1.
中文: 这使我们能够回答以下问题: 1.
<a id="S0301"></a> Source: p.16 S0301
Original: Question: Does BERT really need such a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?
中文: 问:BERT是否真的需要如此大量的预训(128,000个字/批*1,000,000个步骤)来达到高微调精度?
<a id="S0302"></a> Source: p.16 S0302
Original: Answer: Yes, BERT achieves almost BASE 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps. 2.
中文: 回答:是的,BERT在MNLI上接受1M步骤训练时,比起500k步骤,几乎达到BASE1.0%的额外精度. 2. 联合国
<a id="S0303"></a> Source: p.16 S0303
Original: Question: Does MLM pre-training converge slower than LTR pre-training, since only 15% of words are predicted in each batch rather than every word?
中文: 问题:MLM预训是否比LTR预训更慢地汇合,因为每批中只有15%的词被预测出,而不是每个词?
<a id="S0304"></a> Source: p.16 S0304
Original: Answer: The MLM model does converge slightly slower than the LTR model.
中文: 答:MLM模型的聚合速度确实比LTR模型稍慢.
<a id="S0305"></a> Source: p.16 S0305
Original: However, in terms of absolute accuracy the MLM model begins to outperform the LTR model almost immediately. C.2 Ablation for Different Masking Procedures In Section 3.1, we mention that BERT uses a mixed strategy for masking the target tokens when pre-training with the masked language model (MLM) objective.
中文: 然而,在绝对准确性方面,MLM模型几乎立即开始超过LTR模型. 在第3.1节中,我们提到,BERT使用混合策略,在使用被蒙蔽的语言模型(MLM)目标进行预训时掩盖目标符号。
<a id="S0306"></a> Source: p.16 S0306
Original: The following is an ablation study to evaluate the effect of different masking strategies. 84 82 80 78 76 200 400 600 800 1,000 Pre-training Steps (Thousands) ycaruccAveDILNM Note that the purpose of the masking strategies is to reduce the mismatch between pre-training and fine-tuning, as the [MASK] symbol never appears during the fine-tuning stage.
中文: 以下为评估不同口罩策略的效果而作的消毒研究. 84 82 80 78 76 200 400 600 800 培训前步骤(以千计) 注意:蒙面策略的目的是要减少预训和微调之间的不匹配,因为[MASK]符号在微调阶段从未出现.
<a id="S0307"></a> Source: p.16 S0307
Original: We report the Dev results for both MNLI and NER.
中文: 我们报告MNLI和NER的Dev结果。
<a id="S0308"></a> Source: p.16 S0308
Original: For NER, we report both fine-tuning and feature-based approaches, as we expect the mismatch will be amplified for the feature-based approach as the model will not have the chance to adjust the representations.
中文: 就NER而言,我们既报告微调方法,也报告基于特征的方法,因为我们预计,基于特征的方法的不匹配将扩大,因为模型将没有机会调整表述方式。
<a id="S0309"></a> Source: p.16 S0309
Original: Masking Rates Dev Set Results MASK SAME RND MNLI NER Fine-tune Fine-tune Feature-based 80% 10% 10% 84.2 95.4 94.9 100% 0% 0% 84.3 94.9 94.0 80% 0% 20% 84.1 95.2 94.6 80% 20% 0% 84.4 95.2 94.7 0% 20% 80% 83.7 94.8 94.6 0% 0% 100% 83.6 94.9 94.6 Table 8: Ablation over different masking strategies.
中文: 刷新率 设计结果 MASK SAME RND MNLI 精细调试 NER 精细调试特性 80% 10% 10% 10% 84.2 95.4 100% 100% 84.3 94.9 94.9 94.0 80% 20% 84.1 95.2 94.6 80% 20% 84.4 95.2 94.0 20% 80% 83.7 94.8 94.6 0% 0% 100% 100% 100% 83.6 94.9 94.6 表8:不同口罩策略上的偏差.
<a id="S0310"></a> Source: p.16 S0310
Original: In the table, MASK means that we replace the target token with the [MASK] symbol for MLM; SAME means that we keep the target token as is; RND means that we replace the target token with another random token.
中文: 在表格中,MASK表示我们用MLM的[MASK]符号来替换目标符;SAME表示我们保留目标符;RND表示我们用另一个随机符来替换目标符.
<a id="S0311"></a> Source: p.16 S0311
Original: The numbers in the left part of the table represent the probabilities of the specific strategies used during MLM pre-training (BERT uses 80%, 10%, 10%).
中文: 表左部分的数字代表了MLM预训期间使用的具体策略的概率(BERT使用80%,10%,10%).
<a id="S0312"></a> Source: p.16 S0312
Original: The right part of the paper represents the Dev set results.
中文: 论文的正确部分代表了Dev设定的结果.
<a id="S0313"></a> Source: p.16 S0313
Original: For the feature-based approach, we concatenate the last 4 layers of BERT as the features, which was shown to be the best approach in Section 5.3.
中文: 对于以特征为基础的方法,我们把BERT的最后4层作为特征,这在第5.3节中被证明是最佳方法.
<a id="S0314"></a> Source: p.16 S0314
Original: From the table it can be seen that fine-tuning is surprisingly robust to different masking strategies.
中文: 从表中可以看出,微调对不同的遮掩策略的力度惊人地大。
<a id="S0315"></a> Source: p.16 S0315
Original: However, as expected, using only the MASK strategy was problematic when applying the featurebased approach to NER.
中文: 然而,正如预期的那样,在对净入学率适用基于特征的方法时,仅使用MASK战略是有问题的。
<a id="S0316"></a> Source: p.16 S0316
Original: Interestingly, using only the RND strategy performs much worse than our strategy as well.
中文: 有趣的是,仅仅使用RND战略的成绩也比我们的战略差得多。
<a id="S0317"></a> Source: p.16 S0317
Original: BERTBASE (Masked LM) BERTBASE (Left-to-Right) Figure 5: Ablation over number of training steps.
中文: BERTBASE(MASKED LM) BERTBASE(从左到右) 图5:超过培训步骤的数量。
<a id="S0318"></a> Source: p.16 S0318
Original: This shows the MNLI accuracy after fine-tuning, starting from model parameters that have been pre-trained for k steps.
中文: 这显示了微调后MNLI的精度,从已经为 k 级预训的模型参数开始.