Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI - 中英文对照
translated: 2026-07-16
title: "Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI" aliases: - "GPT" source: "https://openai.com/research/language-unsupervised" arxiv: "" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning - nlp
Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI alec@openai.com karthikn@openai.com tim@openai.com ilyasu@openai.com Abstract Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification.
中文: 提高语言理解,由Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI alec@openai.com karthikn@openai.com tim@openai.com ilyasu@openai.com编写 文摘自然语言理解包含各种各样的任务,如文字涵义,问答,语义相似性评估和文件分类等.
<a id="S0002"></a> Source: p.1 S0002
Original: Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately.
中文: 虽然大而无标签的文字公司是丰富的,但用于学习这些具体任务的有标签的数据却很少,使得经过歧视性培训的模型难以充分发挥作用。
<a id="S0003"></a> Source: p.1 S0003
Original: We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.
中文: 我们表明,在这些任务上可以取得很大进展,具体做法是,对各种无标签文本的语言模式进行先入为主的培训,然后对每项具体任务进行歧视性的微调。
<a id="S0004"></a> Source: p.1 S0004
Original: In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture.
中文: 与前作不同的是,我们在微调时利用任务感输入转换来实现有效的转移,同时要求最小程度地改变模型架构.
<a id="S0005"></a> Source: p.1 S0005
Original: We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding.
中文: 我们显示了我们在各种自然语言理解基准方面的做法的有效性。
<a id="S0006"></a> Source: p.1 S0006
Original: Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.
中文: 我们的一般任务不可知模型优于经过歧视性培训的模型,这些模型使用专门为每项任务而设计的建筑,在研究的12项任务中,有9项大大改进了最新艺术。
<a id="S0007"></a> Source: p.1 S0007
Original: For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI). 1 Introduction The ability to learn effectively from raw text is crucial to alleviating the dependence on supervised learning in natural language processing (NLP).
中文: 例如,我们在常识推理(Stries Cloze Test)方面实现了8.9%的绝对改进,在回答问题(RACE)方面实现了5.7%的绝对改进,在文字方面实现了1.5%的绝对改进。 导言 有效学习原始文字的能力对于减轻对自然语言处理(NLP)中受监督的学习的依赖至关重要.
<a id="S0008"></a> Source: p.1 S0008
Original: Most deep learning methods require substantial amounts of manually labeled data, which restricts their applicability in many domains that suffer from a dearth of annotated resources [61].
中文: 大多数深层学习方法都需要大量人工标注的数据,这限制了这些数据在许多领域的适用性,这些领域缺乏附加说明的资源[61]。
<a id="S0009"></a> Source: p.1 S0009
Original: In these situations, models that can leverage linguistic information from unlabeled data provide a valuable alternative to gathering more annotation, which can be time-consuming and expensive.
中文: 在这种情况下,能够利用无标签数据的语言信息的模式为收集更多注释提供了宝贵的替代方法,这些注释可能耗时而昂贵。
<a id="S0010"></a> Source: p.1 S0010
Original: Further, even in cases where considerable supervision is available, learning good representations in an unsupervised fashion can provide a significant performance boost.
中文: 此外,即使在有相当程度监督的情况下,以不受监督的方式学习良好的表现方式也能大大地促进业绩。
<a id="S0011"></a> Source: p.1 S0011
Original: The most compelling evidence for this so far has been the extensive use of pretrained word embeddings [10, 39, 42] to improve performance on a range of NLP tasks [8, 11, 26, 45].
中文: 迄今为止,这方面最令人信服的证据是广泛使用预先训练过的字词嵌入[10、39、42],以提高一系列NLP任务[8、11、26、45]的性能。
<a id="S0012"></a> Source: p.1 S0012
Original: Leveraging more than word-level information from unlabeled text, however, is challenging for two main reasons.
中文: 然而,从未加标签的文本中获取超过文字层面的信息具有挑战性,主要原因有二.
<a id="S0013"></a> Source: p.1 S0013
Original: First, it is unclear what type of optimization objectives are most effective at learning text representations that are useful for transfer.
中文: 首先,不清楚哪种类型的优化目标在学习文本表述方面最为有效,而文本表述对转移有用.
<a id="S0014"></a> Source: p.1 S0014
Original: Recent research has looked at various objectives such as language modeling [44], machine translation [38], and discourse coherence [22], with each method outperforming the others on different tasks.1 Second, there is no consensus on the most effective way to transfer these learned representations to the target task.
中文: 最近的研究研究了各种目标,如语言模型[44]、机器翻译[38]和语句一致性[22],每种方法在不同任务上都比其他方法要好。 第二,对于将这些学到的意见转移到目标任务上的最有效方式没有达成共识。
<a id="S0015"></a> Source: p.1 S0015
Original: Existing techniques involve a combination of making task-specific changes to the model architecture [43, 44], using intricate learning schemes [21] and adding auxiliary learning objectives [50].
中文: 现有技术包括结合对示范架构进行针对具体任务的改变[43,44],采用复杂的学习计划[21]并增加辅助学习目标[50].
<a id="S0016"></a> Source: p.1 S0016
Original: These uncertainties have made it difficult to develop effective semi-supervised learning approaches for language processing. 1https://gluebenchmark.com/leaderboard Preprint.
中文: 这些不确定因素使得很难为语言处理制定有效的半监督学习方法. 1https://gluebenchmark.com/ 领导板预印.
<a id="S0017"></a> Source: p.2 S0017
Original: In this paper, we explore a semi-supervised approach for language understanding tasks using a combination of unsupervised pre-training and supervised fine-tuning.
中文: 在本文中,我们探索了一种半监督的方法,利用无监督的预训和有监督的微调相结合,完成语言理解任务.
<a id="S0018"></a> Source: p.2 S0018
Original: Our goal is to learn a universal representation that transfers with little adaptation to a wide range of tasks.
中文: 我们的目标是学习一种普遍代表性,这种代表性的转让很少适应范围广泛的任务。
<a id="S0019"></a> Source: p.2 S0019
Original: We assume access to a large corpus of unlabeled text and several datasets with manually annotated training examples (target tasks).
中文: 我们假定可以查阅大量未加标签的文本和几个带有人工附加说明的培训实例(目标任务)的数据集。
<a id="S0020"></a> Source: p.2 S0020
Original: Our setup does not require these target tasks to be in the same domain as the unlabeled corpus.
中文: 我们的设置并不要求这些目标任务与未被标记的元件在同一域.
<a id="S0021"></a> Source: p.2 S0021
Original: We employ a two-stage training procedure.
中文: 我们采用两个阶段的培训程序。
<a id="S0022"></a> Source: p.2 S0022
Original: First, we use a language modeling objective on the unlabeled data to learn the initial parameters of a neural network model.
中文: 首先,我们使用未标注数据上的语言建模目标来学习神经网络模型的初始参数.
<a id="S0023"></a> Source: p.2 S0023
Original: Subsequently, we adapt these parameters to a target task using the corresponding supervised objective.
中文: 随后,我们利用相应的监督目标,使这些参数适应目标任务。
<a id="S0024"></a> Source: p.2 S0024
Original: For our model architecture, we use the Transformer [62], which has been shown to perform strongly on various tasks such as machine translation [62], document generation [34], and syntactic parsing [29].
中文: 对于我们的模型架构,我们使用变形器[62],已经显示它在机器翻译[62],文件生成[34],和合成解析[29]等各种任务上表现很强.
<a id="S0025"></a> Source: p.2 S0025
Original: This model choice provides us with a more structured memory for handling long-term dependencies in text, compared to alternatives like recurrent networks, resulting in robust transfer performance across diverse tasks.
中文: 这种模式选择为我们提供了处理文本长期依赖性的更结构化的内存,与经常性网络等替代方案相比,这导致了不同任务的强力转移性能.
<a id="S0026"></a> Source: p.2 S0026
Original: During transfer, we utilize task-specific input adaptations derived from traversal-style approaches [52], which process structured text input as a single contiguous sequence of tokens.
中文: 在转让期间,我们利用由转录式方法产生的针对具体任务的投入调整[52],将结构化的文本输入作为单一的相接符号序列处理。
<a id="S0027"></a> Source: p.2 S0027
Original: As we demonstrate in our experiments, these adaptations enable us to fine-tune effectively with minimal changes to the architecture of the pre-trained model.
中文: 正如我们在实验中所表明的那样,这些改造使我们能够通过对预先训练过的模型结构的微小改变而有效地调整。
<a id="S0028"></a> Source: p.2 S0028
Original: We evaluate our approach on four types of language understanding tasks – natural language inference, question answering, semantic similarity, and text classification.
中文: 我们对四种语言理解任务——自然语言推论、问题回答、语义相似性以及文本分类——进行评估。
<a id="S0029"></a> Source: p.2 S0029
Original: Our general task-agnostic model outperforms discriminatively trained models that employ architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.
中文: 我们的一般任务不可知模型超越了经过歧视性培训的模型,这些模型采用了专门为每项任务设计的建筑,在所研究的12项任务中,有9项大大改进了最新艺术。
<a id="S0030"></a> Source: p.2 S0030
Original: For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test) [40], 5.7% on question answering (RACE) [30], 1.5% on textual entailment (MultiNLI) [66] and 5.5% on the recently introduced GLUE multi-task benchmark [64].
中文: 例如,我们在常识推理(Stries Cloze Test)[40]、回答问题(RACE)[30]5.7%、文字暗示(MultiNLI)[66]1.5%和最近推出的GLUE多任务基准[64]5.5%方面实现了绝对改进。
<a id="S0031"></a> Source: p.2 S0031
Original: We also analyzed zero-shot behaviors of the pre-trained model on four different settings and demonstrate that it acquires useful linguistic knowledge for downstream tasks. 2 Related Work Semi-supervised learning for NLP Our work broadly falls under the category of semi-supervised learning for natural language.
中文: 我们还在四个不同的场合上分析预训模型的零发行为,并表明它为下游任务获得了有用的语言知识. 2 国家劳工局半监督学习的相关工作 我们的工作大体属于自然语言半监督学习范畴.
<a id="S0032"></a> Source: p.2 S0032
Original: This paradigm has attracted significant interest, with applications to tasks like sequence labeling [24, 33, 57] or text classification [41, 70].
中文: 这个范式引起了人们的极大兴趣,适用于序列标签[24,33,57]或文本分类[41,70]等任务.
<a id="S0033"></a> Source: p.2 S0033
Original: The earliest approaches used unlabeled data to compute word-level or phrase-level statistics, which were then used as features in a supervised model [33].
中文: 最早的方法使用无标签数据来计算词级或短语级统计,然后作为被监管模型的特征[33].
<a id="S0034"></a> Source: p.2 S0034
Original: Over the last few years, researchers have demonstrated the benefits of using word embeddings [11, 39, 42], which are trained on unlabeled corpora, to improve performance on a variety of tasks [8, 11, 26, 45].
中文: 在过去几年中,研究人员展示了使用被培训的无标签公司字嵌入法[11,39,42]的好处,以提高各种任务[8,11,26,45]的性能.
<a id="S0035"></a> Source: p.2 S0035
Original: These approaches, however, mainly transfer word-level information, whereas we aim to capture higher-level semantics.
中文: 然而,这些方法主要传递词层信息,而我们的目标是捕捉更高层次的语义.
<a id="S0036"></a> Source: p.2 S0036
Original: Recent approaches have investigated learning and utilizing more than word-level semantics from unlabeled data.
中文: 最近的方法研究了学习和利用的不仅仅是无标签数据的词级语义。
<a id="S0037"></a> Source: p.2 S0037
Original: Phrase-level or sentence-level embeddings, which can be trained using an unlabeled corpus, have been used to encode text into suitable vector representations for various target tasks [28, 32, 1, 36, 22, 12, 56, 31].
中文: 可以使用无标签符来训练的词层或句层嵌入器,被用来将文本编码成适合用于各种目标任务的矢量表示 [28, 32, 1, 36, 22, 12, 56, 31].
<a id="S0038"></a> Source: p.2 S0038
Original: Unsupervised pre-training Unsupervised pre-training is a special case of semi-supervised learning where the goal is to find a good initialization point instead of modifying the supervised learning objective.
中文: 未受监督的预训未受监督的预训是半受监督的学习的特例,目标是找到良好的初始化点,而不是修改被监督的学习目标.
<a id="S0039"></a> Source: p.2 S0039
Original: Early works explored the use of the technique in image classification [20, 49, 63] and regression tasks [3].
中文: 早期的作品探索了该技术在图像分类[20,49,63]和回归任务[3]中的应用.
<a id="S0040"></a> Source: p.2 S0040
Original: Subsequent research [15] demonstrated that pre-training acts as a regularization scheme, enabling better generalization in deep neural networks.
中文: 随后的研究[15]表明,培训前是一种规范化计划,有助于在深层神经网络中更好地概括。
<a id="S0041"></a> Source: p.2 S0041
Original: In recent work, the method has been used to help train deep neural networks on various tasks like image classification [69], speech recognition [68], entity disambiguation [17] and machine translation [48].
中文: 在近期的作品中,该方法被用于帮助训练深层神经网络执行各种任务,如图像分类[69],语音识别[68],实体分化[17]和机器翻译[48]等.
<a id="S0042"></a> Source: p.2 S0042
Original: The closest line of work to ours involves pre-training a neural network using a language modeling objective and then fine-tuning it on a target task with supervision.
中文: 与我们最接近的工作是利用语言建模目标对神经网络进行预先培训,然后在监督下对目标任务进行微调。
<a id="S0043"></a> Source: p.2 S0043
Original: Dai et al. [13] and Howard and Ruder [21] follow this method to improve text classification.
中文: Dai等 [13]和Howard和Ruder [21] 采用这种方法来改进文本分类.
<a id="S0044"></a> Source: p.2 S0044
Original: However, although the pre-training phase helps capture some linguistic information, their usage of LSTM models restricts their prediction ability to a short range.
中文: 然而,虽然培训前阶段有助于获取一些语言信息,但他们对LSTM模型的使用将预测能力限制在了很短的范围内.
<a id="S0045"></a> Source: p.2 S0045
Original: In contrast, our choice of transformer networks allows us to capture longerrange linguistic structure, as demonstrated in our experiments.
中文: 与此相对照,我们选择变压器网络,使我们能够捕捉更远距离的语言结构,正如我们的实验所证明的那样.
<a id="S0046"></a> Source: p.2 S0046
Original: Further, we also demonstrate the effectiveness of our model on a wider range of tasks including natural language inference, paraphrase detection and story completion.
中文: 此外,我们还展示了我们的模式在更广泛的任务方面的效力,包括自然语言推论、解说探测和故事完成。
<a id="S0047"></a> Source: p.2 S0047
Original: Other approaches [43, 44, 38] use hidden representations from a 2
中文: 其他办法[43、44、38]使用2中的隐性表述
<a id="S0048"></a> Source: p.3 S0048
Original: pre-trained language or machine translation model as auxiliary features while training a supervised model on the target task.
中文: 预训语言或机器翻译模型作为辅助功能,同时就目标任务培训一个有监督的模型.
<a id="S0049"></a> Source: p.3 S0049
Original: This involves a substantial amount of new parameters for each separate target task, whereas we require minimal changes to our model architecture during transfer.
中文: 这涉及每项单独目标任务的大量新参数,而我们在转移过程中需要对我们的模式结构作最小的改动。
<a id="S0050"></a> Source: p.3 S0050
Original: Auxiliary training objectives Adding auxiliary unsupervised training objectives is an alternative form of semi-supervised learning.
中文: 辅助培训目标 添加辅助无监督培训目标是半监督学习的一种替代形式.
<a id="S0051"></a> Source: p.3 S0051
Original: Early work by Collobert and Weston [10] used a wide variety of auxiliary NLP tasks such as POS tagging, chunking, named entity recognition, and language modeling to improve semantic role labeling.
中文: Collobert和Weston的早期工作[10]利用了POS标记,块分,命名实体识别等多种辅助NLP任务,以及语言模型来改进语义角色标签.
<a id="S0052"></a> Source: p.3 S0052
Original: More recently, Rei [50] added an auxiliary language modeling objective to their target task objective and demonstrated performance gains on sequence labeling tasks.
中文: 最近,Rei [50]在他们的目标任务目标中增加了一个辅助语言模型目标,并展示了序列标签任务的绩效收益.
<a id="S0053"></a> Source: p.3 S0053
Original: Our experiments also use an auxiliary objective, but as we show, unsupervised pre-training already learns several linguistic aspects relevant to target tasks. 3 Framework Our training procedure consists of two stages.
中文: 我们的实验也使用辅助目标,但正如我们所显示的,未经监督的预训已经学习了与目标任务相关的几个语言方面. 3 框架 我们的训练程序包括两个阶段。
<a id="S0054"></a> Source: p.3 S0054
Original: The first stage is learning a high-capacity language model on a large corpus of text.
中文: 第一阶段是在一大批文字上学习高能语言模型.
<a id="S0055"></a> Source: p.3 S0055
Original: This is followed by a fine-tuning stage, where we adapt the model to a discriminative task with labeled data. 3.1 Unsupervised pre-training Given an unsupervised corpus of tokens U = {u , . . . , u }, we use a standard language modeling 1 n objective to maximize the following likelihood: (cid:88) L (U) = log P (u |u , . . . , u ; Θ) (1) 1 i i−k i−1 i where k is the size of the context window, and the conditional probability P is modeled using a neural network with parameters Θ.
中文: 接下来是微调阶段,我们用标有标签的数据使模型适应歧视性的任务. 3.1 鉴于U = {u,.,.,.u}的无监督符号,我们使用一个标准语言模式,以达到1 n 的目标,以最大限度地实现以下可能性:(cid:88 L (U) = log P (u |u,.,.,.; ) (1) 1 i i-k i- 1 i,其中 k 为上下文窗口的大小,有条件的 P 是使用带有参数的神经网络进行模型。
<a id="S0056"></a> Source: p.3 S0056
Original: These parameters are trained using stochastic gradient descent [51].
中文: 这些参数使用分层梯度下降法进行训练[51].
<a id="S0057"></a> Source: p.3 S0057
Original: In our experiments, we use a multi-layer Transformer decoder [34] for the language model, which is a variant of the transformer [62].
中文: 在我们的实验中,我们使用多层变形器解码器[34]用于语言模型,这是变形器[62]的一个变体.
<a id="S0058"></a> Source: p.3 S0058
Original: This model applies a multi-headed self-attention operation over the input context tokens followed by position-wise feedforward layers to produce an output distribution over target tokens: h = U W + W 0 e p h l = transformer_block(h l−1 )∀i ∈ [1, n] (2) P (u) = softmax(h W T ) n e where U = (u , . . . , u ) is the context vector of tokens, n is the number of layers, W is the token −k −1 e embedding matrix, and W is the position embedding matrix. p 3.2 Supervised fine-tuning After training the model with the objective in Eq. 1, we adapt the parameters to the supervised target task.
中文: 本模型在输入上下文符号上应用多头自留式操作,然后是位置方向的向导层,以在目标符号上产生输出分布: h = U W + W 0 e p h l = 变压器 block(h l−1) ∀ ∀ [1, n] (2) P (u) = 软max (h W T) n e , 其中 U = (u,., u) 为上下文向量, n 为地层数, W 为正向- k− 1 e 嵌入矩阵, W 为位置嵌入矩阵. 页: 1 监督微调 在以Eq. 1为目标对模型进行培训后,我们调整参数,使之适应受监督的目标任务。
<a id="S0059"></a> Source: p.3 S0059
Original: We assume a labeled dataset C, where each instance consists of a sequence of input tokens, x1, . . . , xm, along with a label y.
中文: 我们假设一个有标签的数据集C,其中每个实例包含一个输入符的序列,x1,.,xm,以及标签y.
<a id="S0060"></a> Source: p.3 S0060
Original: The inputs are passed through our pre-trained model to obtain the final transformer block’s activation hm, which is then fed into an added linear output layer with l parameters W to predict y: y P (y|x1, . . . , xm) = softmax(hmW ). (3) l y This gives us the following objective to maximize: (cid:88) L (C) = log P (y|x1, . . . , xm). (4) 2 (x,y) We additionally found that including language modeling as an auxiliary objective to the fine-tuning helped learning by (a) improving generalization of the supervised model, and (b) accelerating convergence.
中文: 输入通过我们预先训练过的模型来获得最后变压器块的活化hm,然后输入一个附加的线性输出层,其中包含l参数W以预测y:y:y-P(y-Xx1,.,xm)=软max(hmW.). (3)l y 这给了我们以下最大化的目标:(cid:88)L(C)=log P(y-Xx1,.,.,xm.,xy). (4) 2(x,y) 我们进一步发现,将语言模型作为微调的辅助目标,通过(a) 改进被监督模型的通化,以及(b) 加速趋同。
<a id="S0061"></a> Source: p.3 S0061
Original: This is in line with prior work [50, 43], who also observed improved performance with such an auxiliary objective.
中文: 这与先前的工作[50、43]是一致的,后者也观察到这种辅助目标提高了业绩。
<a id="S0062"></a> Source: p.3 S0062
Original: Specifically, we optimize the following objective (with weight λ): L (C) = L (C) + λ ∗ L (C) (5) 3 2 1 Overall, the only extra parameters we require during fine-tuning are W , and embeddings for delimiter y tokens (described below in Section 3.3). 3
中文: 具体地说,我们优化了以下目标(有重量+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 3个
<a id="S0063"></a> Source: p.4 S0063
Original: Figure 1: (left) Transformer architecture and training objectives used in this work. (right) Input transformations for fine-tuning on different tasks.
中文: 图1: (左)本作采用的变形器架构和培训目标. (权利) 对不同任务进行微调的输入转换.
<a id="S0064"></a> Source: p.4 S0064
Original: We convert all structured inputs into token sequences to be processed by our pre-trained model, followed by a linear+softmax layer. 3.3 Task-specific input transformations For some tasks, like text classification, we can directly fine-tune our model as described above.
中文: 我们把所有结构化的输入转换成符号序列,由我们预先训练的模型处理,然后是线性+软max层. 3.3 针对具体任务的投入转变 对于一些任务,如文本分类,我们可以直接微调上述模式.
<a id="S0065"></a> Source: p.4 S0065
Original: Certain other tasks, like question answering or textual entailment, have structured inputs such as ordered sentence pairs, or triplets of document, question, and answers.
中文: 某些其他任务,如回答问题或文字问题,有结构化的投入,如命令的句子对,或文件、问答的三重性。
<a id="S0066"></a> Source: p.4 S0066
Original: Since our pre-trained model was trained on contiguous sequences of text, we require some modifications to apply it to these tasks.
中文: 由于我们经过预先训练的模型接受了关于相接相接的文本序列的培训,我们需要进行一些修改,以便将它应用于这些任务。
<a id="S0067"></a> Source: p.4 S0067
Original: Previous work proposed learning task specific architectures on top of transferred representations [44].
中文: 先前的工作除了转移的表述外,还提议学习任务的具体结构[44]。
<a id="S0068"></a> Source: p.4 S0068
Original: Such an approach re-introduces a significant amount of task-specific customization and does not use transfer learning for these additional architectural components.
中文: 这种做法重新引入了大量针对具体任务的定制,不使用这些额外的建筑构件的转移学习.
<a id="S0069"></a> Source: p.4 S0069
Original: Instead, we use a traversal-style approach [52], where we convert structured inputs into an ordered sequence that our pre-trained model can process.
中文: 取而代之的是,我们使用一种倒转式的方法[52],将结构化的输入转换成我们预先训练过的模型可以处理的顺序.
<a id="S0070"></a> Source: p.4 S0070
Original: These input transformations allow us to avoid making extensive changes to the architecture across tasks.
中文: 这些投入转变使我们能够避免对各种任务的架构进行广泛的改变。
<a id="S0071"></a> Source: p.4 S0071
Original: We provide a brief description of these input transformations below and Figure 1 provides a visual illustration.
中文: 下面我们简要描述这些输入转换,图1提供了可视化的图解.
<a id="S0072"></a> Source: p.4 S0072
Original: All transformations include adding randomly initialized start and end tokens ((cid:104)s(cid:105), (cid:104)e(cid:105)).
中文: 所有变换包括加入随机初始化起站牌和末站牌((cid:104)s(cid:105),(cid:104)e(cid:105)).
<a id="S0073"></a> Source: p.4 S0073
Original: Textual entailment For entailment tasks, we concatenate the premise p and hypothesis h token sequences, with a delimiter token ($) in between.
中文: 为了完成各项任务,我们将前提p和假设h符号序列合并起来,并在两者之间加上一个划定符($)。
<a id="S0074"></a> Source: p.4 S0074
Original: Similarity For similarity tasks, there is no inherent ordering of the two sentences being compared.
中文: 相似性 就相似性任务而言,对这两个句子没有进行内在的比较。
<a id="S0075"></a> Source: p.4 S0075
Original: To reflect this, we modify the input sequence to contain both possible sentence orderings (with a delimiter in between) and process each independently to produce two sequence representations hm l which are added element-wise before being fed into the linear output layer.
中文: 为了反映这一点,我们修改输入序列,以便既包含可能的句子顺序(介于两者之间),又独立处理每个句子,以产生两个序列表示hml,在被输入到线性输出层之前按元素添加到hml。
<a id="S0076"></a> Source: p.4 S0076
Original: Question Answering and Commonsense Reasoning For these tasks, we are given a context document z, a question q, and a set of possible answers {a }.
中文: 问答和常识 对于这些任务,我们得到了一个上下文文件z,一个问题q,以及一组可能的答案{a}.
<a id="S0077"></a> Source: p.4 S0077
Original: We concatenate the document context k and question with each possible answer, adding a delimiter token in between to get [z; q; $; a ].
中文: 我们将文档上下文k与每个可能的答案合并,并在两者之间添加一个分隔符以获得[z; q; $; a]。
<a id="S0078"></a> Source: p.4 S0078
Original: Each k of these sequences are processed independently with our model and then normalized via a softmax layer to produce an output distribution over possible answers. 4 Experiments 4.1 Setup Unsupervised pre-training We use the BooksCorpus dataset [71] for training the language model.
中文: 这些序列中的每一个 k 都与我们的模型独立处理,然后通过软马克斯层实现常态化,以在可能的答案上产生输出分布. 4 实验 4.1 设置无监督预训 我们使用 BooksCorpus 数据集 [71] 来培训语言模型.
<a id="S0079"></a> Source: p.4 S0079
Original: It contains over 7,000 unique unpublished books from a variety of genres including Adventure, Fantasy, and Romance.
中文: 该书收录了来自"冒险","幻想","浪漫"等多家流派的7000多本独特的未出版书籍.
<a id="S0080"></a> Source: p.4 S0080
Original: Crucially, it contains long stretches of contiguous text, which allows the generative model to learn to condition on long-range information.
中文: 关键的是,它包含了相接文字的长长长度,这使得基因模型可以学习以长距离信息为条件.
<a id="S0081"></a> Source: p.4 S0081
Original: An alternative dataset, the 1B Word Benchmark, which is used by a similar approach, ELMo [44], is approximately the same size 4
中文: 另一种数据集,即类似方法使用的1B Word基准,ELMo[44],大小大致相同。
<a id="S0082"></a> Source: p.5 S0082
Original: Table 1: A list of the different tasks and datasets used in our experiments.
中文: 表1:我们实验中使用的不同任务和数据集列表.
<a id="S0083"></a> Source: p.5 S0083
Original: Task Datasets Natural language inference SNLI [5], MultiNLI [66], Question NLI [64], RTE [4], SciTail [25] Question Answering RACE [30], Story Cloze [40] Sentence similarity MSR Paraphrase Corpus [14], Quora Question Pairs [9], STS Benchmark [6] Classification Stanford Sentiment Treebank-2 [54], CoLA [65] but is shuffled at a sentence level - destroying long-range structure.
中文: 任务数据集 自然语言推断 SNLI [5], MultiNLI [66], question NLI [64], RTE [4], Scitail [25] 回答 RACE [30], Story Cloze [40] 句子相近 MSR 参数 Corpus [14], Quora 质询对等 [9], STS 基准 [6] 分类 Stanford Sentimment Treebank-2 [54], CoLA [65] 但被打乱了句子-摧毁了远程结构.
<a id="S0084"></a> Source: p.5 S0084
Original: Our language model achieves a very low token level perplexity of 18.4 on this corpus.
中文: 我们的语言模式 实现了一个非常低的标志 水平的模糊度 18.4在这个本体。
<a id="S0085"></a> Source: p.5 S0085
Original: Model specifications Our model largely follows the original transformer work [62].
中文: 示范规格 我们的型号基本上遵循了原来的变压器工作[62].
<a id="S0086"></a> Source: p.5 S0086
Original: We trained a 12-layer decoder-only transformer with masked self-attention heads (768 dimensional states and 12 attention heads).
中文: 我们训练了一台12层的解码器专用变压器,并有面具自留心头(768个维态和12个注意力头).
<a id="S0087"></a> Source: p.5 S0087
Original: For the position-wise feed-forward networks, we used 3072 dimensional inner states.
中文: 对于位置方向的进取网络,我们使用了3072维的内态.
<a id="S0088"></a> Source: p.5 S0088
Original: We used the Adam optimization scheme [27] with a max learning rate of 2.5e-4.
中文: 我们使用了亚当优化计划[27] 最大学习率为2.5e-4.
<a id="S0089"></a> Source: p.5 S0089
Original: The learning rate was increased linearly from zero over the first 2000 updates and annealed to 0 using a cosine schedule.
中文: 在2000年的首次更新中,学习率从零线性地提高到了0线性,使用同位素表。
<a id="S0090"></a> Source: p.5 S0090
Original: We train for 100 epochs on minibatches of 64 randomly sampled, contiguous sequences of 512 tokens.
中文: 我们训练100个世纪 在64个随机抽取的小型战舰上 512个标志的相接序列
<a id="S0091"></a> Source: p.5 S0091
Original: Since layernorm [2] is used extensively throughout the model, a simple weight initialization of N (0, 0.02) was sufficient.
中文: 由于图层诺姆[2]在整个模型中被广泛使用,简单的N(0,0.02)的重量初始化就足够了.
<a id="S0092"></a> Source: p.5 S0092
Original: We used a bytepair encoding (BPE) vocabulary with 40,000 merges [53] and residual, embedding, and attention dropouts with a rate of 0.1 for regularization.
中文: 我们使用了一个字节编码(BPE)词汇,有40,000个合并[53],剩余,嵌入,以及注意力的失学率0.1用于规范化.
<a id="S0093"></a> Source: p.5 S0093
Original: We also employed a modified version of L2 regularization proposed in [37], with w = 0.01 on all non bias or gain weights.
中文: 我们还采用了[37]中提议的L2规范化修改版,所有非偏差或增重的w=0.01。
<a id="S0094"></a> Source: p.5 S0094
Original: For the activation function, we used the Gaussian Error Linear Unit (GELU) [18].
中文: 对于活化功能,我们使用了高斯误差线性单元(GELU)[18].
<a id="S0095"></a> Source: p.5 S0095
Original: We used learned position embeddings instead of the sinusoidal version proposed in the original work.
中文: 我们使用了学习的嵌入位置,而不是原作中提议的鼻音版本。
<a id="S0096"></a> Source: p.5 S0096
Original: We use the ftfy library2 to clean the raw text in BooksCorpus, standardize some punctuation and whitespace, and use the spaCy tokenizer.3 Fine-tuning details Unless specified, we reuse the hyperparameter settings from unsupervised pre-training.
中文: 我们使用 ftfy 库2 来清理 BooksCorpus 中的原始文本,将一些平分和白空格标准化,并使用 sparCy 标注器。 微调细节 除非指定, 我们重新使用超参数设置 从不受监督的预训。
<a id="S0097"></a> Source: p.5 S0097
Original: We add dropout to the classifier with a rate of 0.1.
中文: 我们把退学率加到0.1
<a id="S0098"></a> Source: p.5 S0098
Original: For most tasks, we use a learning rate of 6.25e-5 and a batchsize of 32.
中文: 对于大多数任务,我们使用6.25e-5的学习率,分批使用32.
<a id="S0099"></a> Source: p.5 S0099
Original: Our model finetunes quickly and 3 epochs of training was sufficient for most cases.
中文: 我们的模式迅速微调,三个时代的训练对多数情况都足够。
<a id="S0100"></a> Source: p.5 S0100
Original: We use a linear learning rate decay schedule with warmup over 0.2% of training. λ was set to 0.5. 4.2 Supervised fine-tuning We perform experiments on a variety of supervised tasks including natural language inference, question answering, semantic similarity, and text classification.
中文: 我们使用线性学习率衰减计划 热身超过0.2%的训练 确定为0.5.4.2 监督微调 我们进行各种监督任务的实验,包括自然语言推论,问题回答,语义相似,以及文字分类.
<a id="S0101"></a> Source: p.5 S0101
Original: Some of these tasks are available as part of the recently released GLUE multi-task benchmark [64], which we make use of.
中文: 其中一些任务可作为最近发布的GLUE多任务基准的一部分[64]提供,我们使用这些基准。
<a id="S0102"></a> Source: p.5 S0102
Original: Figure 1 provides an overview of all the tasks and datasets.
中文: 图1概述了所有任务和数据集。
<a id="S0103"></a> Source: p.5 S0103
Original: Natural Language Inference The task of natural language inference (NLI), also known as recognizing textual entailment, involves reading a pair of sentences and judging the relationship between them from one of entailment, contradiction or neutral.
中文: 自然语言推论 自然语言推论(NLI)的任务,也称为承认文字内涵,涉及读取一对句子,并从引出,矛盾或中性的角度判断它们之间的关系.
<a id="S0104"></a> Source: p.5 S0104
Original: Although there has been a lot of recent interest [58, 35, 44], the task remains challenging due to the presence of a wide variety of phenomena like lexical entailment, coreference, and lexical and syntactic ambiguity.
中文: 虽然最近引起了许多兴趣[58,35,44],但由于存在各种现象,如从词法上引出、相互参照以及从词法上和从语法上模糊不清,这项任务仍然具有挑战性。
<a id="S0105"></a> Source: p.5 S0105
Original: We evaluate on five datasets with diverse sources, including image captions (SNLI), transcribed speech, popular fiction, and government reports (MNLI), Wikipedia articles (QNLI), science exams (SciTail) or news articles (RTE).
中文: 我们评价五个有不同来源的数据集,包括图像标题(SNLI),转写语音,通俗小说,以及政府报告(MNLI),维基百科文章(QNLI),科学考试(SciTail)或新闻文章(RTE).
<a id="S0106"></a> Source: p.5 S0106
Original: Table 2 details various results on the different NLI tasks for our model and previous state-of-the-art approaches.
中文: 表2详细列出了我国模式和以往最新办法的不同国家综合行动任务的各种结果。
<a id="S0107"></a> Source: p.5 S0107
Original: Our method significantly outperforms the baselines on four of the five datasets, achieving absolute improvements of upto 1.5% on MNLI, 5% on SciTail, 5.8% on QNLI and 0.6% on SNLI over the previous best results.
中文: 我们的方法大大超过了五个数据集中的四个数据集的基线,实现了在MNLI上最高1.5%,在SciTail上5%,在QNLI上5.8%,在SNLI上0.6%的绝对改进,超过了之前的最佳结果.
<a id="S0108"></a> Source: p.5 S0108
Original: This demonstrates our model’s ability to better reason over multiple sentences, and handle aspects of linguistic ambiguity.
中文: 这说明我们的模型有能力更好地解释多个句子,
<a id="S0109"></a> Source: p.5 S0109
Original: On RTE, one of the smaller datasets we evaluate on (2490 examples), we achieve an accuracy of 56%, which is below the 61.7% reported by a multi-task biLSTM model.
中文: 在RTE上,我们评价的一个较小的数据集(2490例),我们实现了56%的精度,低于一个多任务双LSTM模型报告的61.7%.
<a id="S0110"></a> Source: p.5 S0110
Original: Given the strong performance of our approach on larger NLI datasets, it is likely our model will benefit from multi-task training as well but we have not explored this currently. 2https://ftfy.readthedocs.io/en/latest/ 3https://spacy.io/ 5
中文: 鉴于我们在较大的国家液化物联网数据集方面做法的有力表现,我们的模型可能也将受益于多任务培训,但我们目前尚未探讨这一问题。 2https://ftfy.readthedocs.io/en/latest/ 3https://spacy.io/ 5 (中文(简体) ).
<a id="S0111"></a> Source: p.6 S0111
Original: Table 2: Experimental results on natural language inference tasks, comparing our model with current state-of-the-art methods. 5x indicates an ensemble of 5 models.
中文: 表2:自然语言推断任务的实验结果,将我们的模型与目前最先进的方法进行比较. 5x表示由5个型号组成的组合.
<a id="S0112"></a> Source: p.6 S0112
Original: All datasets use accuracy as the evaluation metric.
中文: 所有数据集都以准确性作为评价尺度。
<a id="S0113"></a> Source: p.6 S0113
Original: Method MNLI-m MNLI-mm SNLI SciTail QNLI RTE ESIM + ELMo [44] (5x) - - 89.3 - - - CAFE [58] (5x) 80.2 79.0 89.3 - - - Stochastic Answer Network [35] (3x) 80.6 80.1 - - - - CAFE [58] 78.7 77.9 88.5 83.3 GenSen [64] 71.4 71.3 - - 82.3 59.2 Multi-task BiLSTM + Attn [64] 72.2 72.1 - - 82.1 61.7 Finetuned Transformer LM (ours) 82.1 81.4 89.9 88.3 88.1 56.0 Table 3: Results on question answering and commonsense reasoning, comparing our model with current state-of-the-art methods.. 9x means an ensemble of 9 models.
中文: MNLI-m MNLI-mm SNLI SciTail QNLI RTE ESIM + ELMo [44] (5x) - - 89.3 (5x) - - - CAFE [58] 80.2 79.0 89.3 - - - - Stochastic Answer Network [35 (3x) 80.6 80.1 - - - - - - - CAFE [58] 78.7 77.9 88.5 83.3 GenSen [64] 71.4 71.3 - 82.3 - 82.3 59.2 多任务 BiLSTM + Atn [64] 72.2 72.1 - 82.1 61.7 微调变压器 LM (ours) 82.1 81.4 89.9 88.3 88.1 56.0 表3:问答结果和常识推理,将我们的模型与目前最先进的方法进行比较. 9x指由9个模型组成的组合.
<a id="S0114"></a> Source: p.6 S0114
Original: Method Story Cloze RACE-m RACE-h RACE val-LS-skip [55] 76.5 - - - Hidden Coherence Model [7] 77.6 - - - Dynamic Fusion Net [67] (9x) - 55.6 49.4 51.2 BiAttention MRU [59] (9x) - 60.2 50.3 53.3 Finetuned Transformer LM (ours) 86.5 62.9 57.4 59.0 Question answering and commonsense reasoning Another task that requires aspects of single and multi-sentence reasoning is question answering.
中文: 方法故事 Cloze RACE-m RACE-h RACE val-LS-skip [55] 76.5 - - - - - 隐藏一致性模型 [7] 77.6 - - - - - 动态聚合网 [67] (9x) - 55.6 49.4 51.2 BiAtenting MRU [59] (9x) - 60.2 50.3 53.3 微调变压器 LM(我们) 86.5 62.9 57.4 59.0 (韩语) 问题回答和常识推理 另一个需要单一和多判决推理各个方面的任务就是问题回答.
<a id="S0115"></a> Source: p.6 S0115
Original: We use the recently released RACE dataset [30], consisting of English passages with associated questions from middle and high school exams.
中文: 我们使用最近发布的RACE数据集[30],由英文段落和初中和高中考试相关问题组成.
<a id="S0116"></a> Source: p.6 S0116
Original: This corpus has been shown to contain more reasoning type questions that other datasets like CNN [19] or SQuaD [47], providing the perfect evaluation for our model which is trained to handle long-range contexts.
中文: 已经证明这本书中包含更多的推理类型问题,例如CNN[19]或SQuaD[47]等其他数据集,为我们模型提供了完美的评价,该模型经过了处理远程环境的培训。
<a id="S0117"></a> Source: p.6 S0117
Original: In addition, we evaluate on the Story Cloze Test [40], which involves selecting the correct ending to multi-sentence stories from two options.
中文: 此外,我们还在"故事克洛兹测试"(Story Cloze Test)上评价了[40],这涉及到从两个选项中选择正确的结局到多判决故事.
<a id="S0118"></a> Source: p.6 S0118
Original: On these tasks, our model again outperforms the previous best results by significant margins - up to 8.9% on Story Cloze, and 5.7% overall on RACE.
中文: 在这些任务上,我们的模型再次以显著的差幅超越了之前的最佳结果——在"故事克洛兹"上高达8.9%,在RACE上则高达5.7%。
<a id="S0119"></a> Source: p.6 S0119
Original: This demonstrates the ability of our model to handle long-range contexts effectively.
中文: 这表明我们的模式有能力有效地处理长期环境。
<a id="S0120"></a> Source: p.6 S0120
Original: Semantic Similarity Semantic similarity (or paraphrase detection) tasks involve predicting whether two sentences are semantically equivalent or not.
中文: 语义相似性语义相似性(或译出词检测)的任务涉及预测两个句子在语义上是否等同.
<a id="S0121"></a> Source: p.6 S0121
Original: The challenges lie in recognizing rephrasing of concepts, understanding negation, and handling syntactic ambiguity.
中文: 挑战在于承认概念的重写、理解否定和处理协同模糊。
<a id="S0122"></a> Source: p.6 S0122
Original: We use three datasets for this task – the Microsoft Paraphrase corpus (MRPC) [14] (collected from news sources), the Quora Question Pairs (QQP) dataset [9], and the Semantic Textual Similarity benchmark (STS-B) [6].
中文: 我们使用三个数据集来完成这项任务 — 微软参数(MRPC)14,Quora Question Pairs(QQP)数据集[9],以及语义文字相似性基准(STS-B)[6].
<a id="S0123"></a> Source: p.6 S0123
Original: We obtain state-of-the-art results on two of the three semantic similarity tasks (Table 4) with a 1 point absolute gain on STS-B.
中文: 我们以STS-B的1分绝对得分获得三个语义相似任务中的两个(表4)的最新结果.
<a id="S0124"></a> Source: p.6 S0124
Original: The performance delta on QQP is significant, with a 4.2% absolute improvement over Single-task BiLSTM + ELMo + Attn.
中文: 在QQP上的性能Delta显著出众,与单任务BiLSTM + ELMo + Attn相比有4.2%的绝对改进.
<a id="S0125"></a> Source: p.6 S0125
Original: Classification Finally, we also evaluate on two different text classification tasks.
中文: 最后,我们还评估了两个不同的文本分类任务。
<a id="S0126"></a> Source: p.6 S0126
Original: The Corpus of Linguistic Acceptability (CoLA) [65] contains expert judgements on whether a sentence is grammatical or not, and tests the innate linguistic bias of trained models.
中文: 语言可接受性公司(CoLA)[65] 载有对某一句子是否语法问题的专家判断,并测试训练有素的模型固有的语言偏差.
<a id="S0127"></a> Source: p.6 S0127
Original: The Stanford Sentiment Treebank (SST-2) [54], on the other hand, is a standard binary classification task.
中文: 而"斯坦福感想树"(SST-2)[54]则是标准二进制分类任务.
<a id="S0128"></a> Source: p.6 S0128
Original: Our model obtains an score of 45.4 on CoLA, which is an especially big jump over the previous best result of 35.0, showcasing the innate linguistic bias learned by our model.
中文: 我们的模型在CoLA上获得45.4分的分数,这比以前35.0的最好成绩特别大,展现出我们模型学到的先天语言偏差.
<a id="S0129"></a> Source: p.6 S0129
Original: The model also achieves 91.3% accuracy on SST-2, which is competitive with the state-of-the-art results.
中文: 该模型在SST-2上也实现了91.3%的精度,与最先进的成绩相竞争.
<a id="S0130"></a> Source: p.6 S0130
Original: We also achieve an overall score of 72.8 on the GLUE benchmark, which is significantly better than the previous best of 68.9. 6
中文: 我们还在GLUE基准上达到总分72.8分,大大优于之前68.9分中的最佳成绩. 6个
<a id="S0131"></a> Source: p.7 S0131
Original: Table 4: Semantic similarity and classification results, comparing our model with current state-of-theart methods.
中文: 表4:语义相似和分类结果,将我们的模型同目前最先进的方法进行比较.
<a id="S0132"></a> Source: p.7 S0132
Original: All task evaluations in this table were done using the GLUE benchmark. (mc= Mathews correlation, acc=Accuracy, pc=Pearson correlation) Method Classification Semantic Similarity GLUE CoLA SST2 MRPC STSB QQP (mc) (acc) (F1) (pc) (F1) Sparse byte mLSTM [16] - 93.2 - - - - TF-KLD [23] - - 86.0 - - - ECNU (mixed ensemble) [60] - - - 81.0 - - Single-task BiLSTM + ELMo + Attn [64] 35.0 90.2 80.2 55.5 66.1 64.8 Multi-task BiLSTM + ELMo + Attn [64] 18.9 91.6 83.5 72.8 63.3 68.9 Finetuned Transformer LM (ours) 45.4 91.3 82.3 82.0 70.3 72.8 Overall, our approach achieves new state-of-the-art results in 9 out of the 12 datasets we evaluate on, outperforming ensembles in many cases.
中文: 本表中的所有任务评价都是使用GLUE基准进行的. (mc=Mathews conference, acc=准确性, pc=Pearson conference) 方法分类语义相似性 GLUE CoLA SST2 MRPC STSB QQP (mc)(ac)(F1)(pc)(F1) Sparse byte mLSTM [16] - 93.2 - - - - - - - - - TF-KLD [23] - 86.0 - - - - - - - ECNU(混合合唱团) [60] - - - - 81.0 - - 单任务 BiLSTM + ELMo + Atn [64] 35.0 90.2 55.5 66.1 64.8 多任务 BILSTM + EL Mo + Atn [64] 18..9 91.6 83.5 63.3 68.9 精细变形器 LM(ours) 45.4 9.1.3 82.382.30 70.3 72.8 总的来说,我们的方法在我们评价的12个数据集中,有9个取得了新的最新成果,在许多情况下,这些数据集的成绩超过了总体。
<a id="S0133"></a> Source: p.7 S0133
Original: Our results also indicate that our approach works well across datasets of different sizes, from smaller datasets such as STS-B (≈5.7k training examples) – to the largest one – SNLI (≈550k training examples). 5 Analysis Impact of number of layers transferred We observed the impact of transferring a variable number of layers from unsupervised pre-training to the supervised target task.
中文: 我们的结果还表明,我们的方法在大小不同的数据集之间运作良好,从较小的数据集,如STS-B(XQ5.7k培训实例)到最大的一个(SNLI)(XQ550k培训实例)。 5 转让层数的影响 我们看到将不同数量的分层从无监督的前期培训转移到受监督的目标任务所产生的影响。
<a id="S0134"></a> Source: p.7 S0134
Original: Figure 2(left) illustrates the performance of our approach on MultiNLI and RACE as a function of the number of layers transferred.
中文: 图2(左)显示我们在多NLI和RACE方面作为转移层数函数的表现。
<a id="S0135"></a> Source: p.7 S0135
Original: We observe the standard result that transferring embeddings improves performance and that each transformer layer provides further benefits up to 9% for full transfer on MultiNLI.
中文: 我们观测到的标准结果是,转接嵌入物能提高性能,每个变压器层在MultiNLI上为全转接提供高达9%的进一步好处.
<a id="S0136"></a> Source: p.7 S0136
Original: This indicates that each layer in the pre-trained model contains useful functionality for solving target tasks.
中文: 这表明预训模型中的每个层都包含了解决目标任务的有用功能.
<a id="S0137"></a> Source: p.7 S0137
Original: Figure 2: (left) Effect of transferring increasing number of layers from the pre-trained language model on RACE and MultiNLI. (right) Plot showing the evolution of zero-shot performance on different tasks as a function of LM pre-training updates.
中文: 图2: (左)从RACE和MultiNLI的预训语言模型中转移越来越多的地层的影响. (权利) 作为LM预训更新的一个功能,在不同的任务上显示零发性能的演变.
<a id="S0138"></a> Source: p.7 S0138
Original: Performance per task is normalized between a random guess baseline and the current state-of-the-art with a single model.
中文: 每个任务的性能在随机猜想基线和目前采用单一模型的先进水平之间实现常态化.
<a id="S0139"></a> Source: p.7 S0139
Original: Zero-shot Behaviors We’d like to better understand why language model pre-training of transformers is effective. A hypothesis is that the underlying generative model learns to perform many of the tasks we evaluate on in order to improve its language modeling capability and that the more structured 7
中文: 零射出的行为 我们想知道为什么变压器的语文模型预训是有效的。 一种假设是,基础基因模型学会执行我们评价的许多任务,以提高其语言模型能力,结构更加严密的7
<a id="S0140"></a> Source: p.8 S0140
Original: Table 5: Analysis of various model ablations on different tasks.
中文: 表5:关于不同任务的各种模型分析。
<a id="S0141"></a> Source: p.8 S0141
Original: Avg. score is a unweighted average of all the results. (mc= Mathews correlation, acc=Accuracy, pc=Pearson correlation) Method Avg.
中文: Avg.分数是所有结果的未加权平均值. (mc=马修斯相关,acc=精确,pc=皮尔逊相关) Method Avg.
<a id="S0142"></a> Source: p.8 S0142
Original: Score CoLA SST2 MRPC STSB QQP MNLI QNLI RTE (mc) (acc) (F1) (pc) (F1) (acc) (acc) (acc) Transformer w/ aux LM (full) 74.7 45.4 91.3 82.3 82.0 70.3 81.8 88.1 56.0 Transformer w/o pre-training 59.9 18.9 84.0 79.4 30.9 65.5 75.7 71.2 53.8 Transformer w/o aux LM 75.0 47.9 92.0 84.9 83.2 69.8 81.1 86.9 54.4 LSTM w/ aux LM 69.1 30.3 90.5 83.2 71.8 68.1 73.7 81.1 54.6 attentional memory of the transformer assists in transfer compared to LSTMs.
中文: 分数CoLA SST2 MRPC STSB QQP MNLI QNLI RTE (mc)(ac)(F1)(pc)(f1)(ac)(ac) 变压器 w/ aux LM (完整) 74.7 45.4 91.3 82.3 82.0 80.3 56.0 变压器 w/o预训 59.9 18.8 84.0 79.4 30.9 65.5 75.7 71.2 53.8 变压器 w/oux LM 75.0 47.9 92.0 84.9 84.9 83.2 69.8 81.1 86.9 54.4 LSTM w/aux LM 69.1 30.3 90.5 83.2 71.8.1 73.7 81.1 54.6 变压器的注意内存与LSTMs相比,有助于转录.
<a id="S0143"></a> Source: p.8 S0143
Original: We designed a series of heuristic solutions that use the underlying generative model to perform tasks without supervised finetuning.
中文: 我们设计了一系列的热力学解决方案,这些解决方案利用基本的基因模型来完成任务,而不进行监督的微调.
<a id="S0144"></a> Source: p.8 S0144
Original: We visualize the effectiveness of these heuristic solutions over the course of generative pre-training in Fig 2(right).
中文: 我们通过图2(right)的基因前训练,来想象这些热力学解决方案的有效性.
<a id="S0145"></a> Source: p.8 S0145
Original: We observe the performance of these heuristics is stable and steadily increases over training suggesting that generative pretraining supports the learning of a wide variety of task relevant functionality.
中文: 我们观察到,与培训相比,这些heuristics的性能是稳定和稳步地增加的,这表明基因前培训有助于学习各种与任务有关的功能。
<a id="S0146"></a> Source: p.8 S0146
Original: We also observe the LSTM exhibits higher variance in its zero-shot performance suggesting that the inductive bias of the Transformer architecture assists in transfer.
中文: 我们还观察到LSTTM在零射出性能上表现出了更高的差异,这表明变形器架构的诱导偏差有助于传输.
<a id="S0147"></a> Source: p.8 S0147
Original: For CoLA (linguistic acceptability), examples are scored as the average token log-probability the generative model assigns and predictions are made by thresholding.
中文: 对于CoLA(语言可接受性),实例被评为基因模型指定和预测通过阈值得出的平均符号对数概率.
<a id="S0148"></a> Source: p.8 S0148
Original: For SST-2 (sentiment analysis), we append the token very to each example and restrict the language model’s output distribution to only the words positive and negative and guess the token it assigns higher probability to as the prediction.
中文: 对于 SST-2 (sentiment analysis) , 我们把符号附加到每个例子中, 并且将语言模型的输出分布限制在只有正负两个单词, 并猜测它给作为预测的概率更高。
<a id="S0149"></a> Source: p.8 S0149
Original: For RACE (question answering), we pick the answer the generative model assigns the highest average token log-probability when conditioned on the document and question.
中文: 对于 RACE(问题回答),我们选择一个答案,基因模型在对文件和问题作条件时指定了最高的平均符号日志概率。
<a id="S0150"></a> Source: p.8 S0150
Original: For DPRD [46] (winograd schemas), we replace the definite pronoun with the two possible referrents and predict the resolution that the generative model assigns higher average token log-probability to the rest of the sequence after the substitution.
中文: 对于DPRD46,我们用两个可能的推荐词来取代确定的代词,并预测基因模型给替代后的其余序列分配了更高的平均符号对数概率的分辨率.
<a id="S0151"></a> Source: p.8 S0151
Original: Ablation studies We perform three different ablation studies (Table 5).
中文: 阅读研究 我们进行了三项不同的消化研究(表5)。
<a id="S0152"></a> Source: p.8 S0152
Original: First, we examine the performance of our method without the auxiliary LM objective during fine-tuning.
中文: 首先,我们在微调时,在没有辅助LM目标的情况下,检查我们方法的性能.
<a id="S0153"></a> Source: p.8 S0153
Original: We observe that the auxiliary objective helps on the NLI tasks and QQP.
中文: 我们观察到,辅助目标有助于NLI任务和QQP.
<a id="S0154"></a> Source: p.8 S0154
Original: Overall, the trend suggests that larger datasets benefit from the auxiliary objective but smaller datasets do not.
中文: 总的来说,趋势表明,较大的数据集受益于辅助目标,但较小的数据集却没有。
<a id="S0155"></a> Source: p.8 S0155
Original: Second, we analyze the effect of the Transformer by comparing it with a single layer 2048 unit LSTM using the same framework.
中文: 第二,我们分析变形器的效果,把它与使用同一种框架的单层2048个单元的LSTM进行比较.
<a id="S0156"></a> Source: p.8 S0156
Original: We observe a 5.6 average score drop when using the LSTM instead of the Transformer.
中文: 我们在使用LSTM而不是"变形器"时观察到平均得分下降5.6分.
<a id="S0157"></a> Source: p.8 S0157
Original: The LSTM only outperforms the Transformer on one dataset – MRPC.
中文: LSTM只在一个数据集上超越了变形器 — MRPC.
<a id="S0158"></a> Source: p.8 S0158
Original: Finally, we also compare with our transformer architecture directly trained on supervised target tasks, without pre-training.
中文: 最后,我们还与直接接受监督目标任务培训的变压器结构进行比较,不进行预先培训。
<a id="S0159"></a> Source: p.8 S0159
Original: We observe that the lack of pre-training hurts performance across all the tasks, resulting in a 14.8% decrease compared to our full model. 6 Conclusion We introduced a framework for achieving strong natural language understanding with a single task-agnostic model through generative pre-training and discriminative fine-tuning.
中文: 我们发现,由于缺乏预训,所有任务的业绩都受到影响,因此与我们的全部模式相比,下降了14.8%。 6 结论 我们采用了一个框架,通过基因前培训和歧视性的微调,用单一的任务不可知模型实现强烈的自然语言理解。
<a id="S0160"></a> Source: p.8 S0160
Original: By pre-training on a diverse corpus with long stretches of contiguous text our model acquires significant world knowledge and ability to process long-range dependencies which are then successfully transferred to solving discriminative tasks such as question answering, semantic similarity assessment, entailment determination, and text classification, improving the state of the art on 9 of the 12 datasets we study.
中文: 通过对具有长长毗连文本的多种实体进行预先培训,我们的模型获得了重要的世界知识和能力,可以处理长期依赖性,然后成功地转移到解决诸如回答问题、语义相似性评估、必然性确定和文本分类等歧视性任务上,从而改进了我们研究的12个数据集中的9个数据集的艺术水平。
<a id="S0161"></a> Source: p.8 S0161
Original: Using unsupervised (pre-)training to boost performance on discriminative tasks has long been an important goal of Machine Learning research.
中文: 利用不受监督的(前)培训来提升歧视性任务的表现,长期以来一直是机器学习研究的一个重要目标.
<a id="S0162"></a> Source: p.8 S0162
Original: Our work suggests that achieving significant performance gains is indeed possible, and offers hints as to what models (Transformers) and data sets (text with long range dependencies) work best with this approach.
中文: 我们的工作表明,取得重大业绩收益确实是可能的,并提示哪些模型(转换器)和数据集(具有长距离依赖性的文本)最能采用这种方法。
<a id="S0163"></a> Source: p.8 S0163
Original: We hope that this will help enable new research into unsupervised learning, for both natural language understanding and other domains, further improving our understanding of how and when unsupervised learning works.
中文: 我们希望,这将有助于对自然语言理解和其他领域的无监督学习进行新的研究,进一步提高我们对无监督学习方式和时间的理解。
<a id="S0164"></a> Source: p.8 S0164
Original: Ma. A simple but tough-to-beat baseline for sentence embeddings. 2016. 8
中文: 妈,句子嵌入的基线很简单,但很难打 2016. 8 (英语).
<a id="S0165"></a> Source: p.9 S0165
Original: Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [3] Y.
中文: 层层正常化. arXiv preprint arXiv:1607.06450, 2016. [3] (英语). 没错
<a id="S0166"></a> Source: p.9 S0166
Original: Greedy layer-wise training of deep networks.
中文: 贪心地分层训练深网.
<a id="S0167"></a> Source: p.9 S0167
Original: In Advances in neural information processing systems, pages 153–160, 2007. [4] L.
中文: "神经信息处理系统的进步", 第153-160页, 2007. [4]. 维基月球在线解说-李克文陨石坑 Andersson, L.
<a id="S0168"></a> Source: p.9 S0168
Original: The fifth pascal recognizing textual entailment challenge.
中文: 第五波斯卡尔承认文字上的挑战。
<a id="S0169"></a> Source: p.9 S0169
Original: Manning. A large annotated corpus for learning natural language inference.
中文: 曼宁语. 学习自然语言推论的大型注释教程.
<a id="S0170"></a> Source: p.9 S0170
Original: Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. [7] S.
中文: Semeval-2017任务一:语义文字相似性-多语言和跨语言重点评价. arXiv预印 arXiv:1708.00055, 2017. [7]. 师曰.
<a id="S0171"></a> Source: p.9 S0171
Original: Story comprehension for predicting what happens next.
中文: 故事理解预测接下来会发生什么。
<a id="S0172"></a> Source: p.9 S0172
Original: In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614, 2017. [8] D.
中文: 2017年自然语言处理经验方法会议纪要, 第1603–1614页, 2017. [8]. D类.
<a id="S0173"></a> Source: p.9 S0173
Original: Manning. A fast and accurate dependency parser using neural networks.
中文: 曼宁语. 利用神经网络快速准确的依赖解析器.
<a id="S0174"></a> Source: p.9 S0174
Original: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740–750, 2014. [9] Z.
中文: 2014年自然语言处理经验方法会议纪要(EMNLP),第740-750页,2014 [9] 兹.
<a id="S0175"></a> Source: p.9 S0175
Original: Quora question pairs. https://data.quora.com/First-Quora- Dataset-Release-Question-Pairs, 2018. [10] R.
中文: 克罗拉问题对. https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs, 2018. [10] R.
<a id="S0176"></a> Source: p.9 S0176
Original: Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning.
中文: 韦斯顿. 自然语言处理的统一架构:具有多任务学习的深层神经网络.
<a id="S0177"></a> Source: p.9 S0177
Original: In Proceedings of the 25th international conference on Machine learning, pages 160–167.
中文: 《第25次机器学习国际会议记录》,第160-167页。
<a id="S0178"></a> Source: p.9 S0178
Original: Natural language processing (almost) from scratch.
中文: 自然语言处理(从零开始).
<a id="S0179"></a> Source: p.9 S0179
Original: Journal of Machine Learning Research, 12(Aug):2493–2537, 2011. [12] A.
中文: 机器学习研究杂志,12 (Aug):2493–2537, 2011. [12]. A类.
<a id="S0180"></a> Source: p.9 S0180
Original: Supervised learning of universal sentence representations from natural language inference data.
中文: 从自然语言推论数据中监督学习通用句子表达.
<a id="S0181"></a> Source: p.9 S0181
Original: In Advances in Neural Information Processing Systems, pages 3079–3087, 2015. [14] W. B.
中文: In Advances in Neural Information Systems, pages 3079–3087, 2015 [14] W. B.
<a id="S0182"></a> Source: p.9 S0182
Original: Automatically constructing a corpus of sentential paraphrases.
中文: 自动地构建出一系列的感人语句.
<a id="S0183"></a> Source: p.9 S0183
Original: In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [15] D.
中文: 2005年第三次国际审校讲习班会议记录[15] D。
<a id="S0184"></a> Source: p.9 S0184
Original: Why does unsupervised pre-training help deep learning?
中文: 为何无监督的预训会帮助深入学习?
<a id="S0185"></a> Source: p.9 S0185
Original: Journal of Machine Learning Research, 11(Feb):625–660, 2010. [16] S.
中文: 机器学习研究杂志,11(Feb):625–660, 2010. [16]. 师曰.
<a id="S0186"></a> Source: p.9 S0186
Original: Gpu kernels for block-sparse weights. 2017. [17] Z.
中文: Gpu内核用于块-sparse重量. 2017. [17] Z (中文(简体) ).
<a id="S0187"></a> Source: p.9 S0187
Original: Learning entity representation for entity disambiguation.
中文: 学习实体代表实体分裂。
<a id="S0188"></a> Source: p.9 S0188
Original: In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 30–34, 2013. [18] D.
中文: 计算语言学协会第51届年会记录(第2卷:短篇论文),第2卷,第30至34页,2013 [18] D.
<a id="S0189"></a> Source: p.9 S0189
Original: Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016. [19] K. M.
中文: 将非线性与有条纹的正则器与高斯误差线性单位相接. arXiv preprint arXiv:1606.08415, 2016. [19] K. M.
<a id="S0190"></a> Source: p.9 S0190
Original: Teaching machines to read and comprehend.
中文: 教出能读能懂的机器.
<a id="S0191"></a> Source: p.9 S0191
Original: In Advances in Neural Information Processing Systems, pages 1693– 1701, 2015. [20] G. E.
中文: In Advances in Neural Information Systems, pages 1693–1701, 2015 [20] G. E.
<a id="S0192"></a> Source: p.9 S0192
Original: Teh. A fast learning algorithm for deep belief nets.
中文: 铁. 深度信仰网快速学习算法.
<a id="S0193"></a> Source: p.9 S0193
Original: Neural computation, 18(7):1527–1554, 2006. [21] J.
中文: 神经计算,18(7):1527–1554, 2006. [21] J.
<a id="S0194"></a> Source: p.9 S0194
Original: Universal language model fine-tuning for text classification.
中文: 通用语言模型微调文本分类.
<a id="S0195"></a> Source: p.9 S0195
Original: Association for Computational Linguistics (ACL), 2018. [22] Y.
中文: 计算语言学协会(ACL),2018. [22] Y.
<a id="S0196"></a> Source: p.9 S0196
Original: Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017. [23] Y.
中文: 基于演讲的目标为快速无监督的句子表现学习. arXiv预印 arXiv:1705.00557, 2017. [23] Y.
<a id="S0197"></a> Source: p.9 S0197
Original: Discriminative improvements to distributional sentence similarity.
中文: 对分布式句子相似性的歧视性改进.
<a id="S0198"></a> Source: p.9 S0198
Original: In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 891–896, 2013. 9
中文: 《2013年自然语言处理经验方法会议纪要》,第891-896页。
<a id="S0199"></a> Source: p.10 S0199
Original: Semi-supervised conditional random fields for improved sequence segmentation and labeling.
中文: 半监督有条件的随机字段,用于改进序列分和标签.
<a id="S0200"></a> Source: p.10 S0200
Original: In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 209–216.
中文: 第21次计算语言学国际会议和计算语言学协会第44次年会记录,第209-216页。
<a id="S0201"></a> Source: p.10 S0201
Original: Association for Computational Linguistics, 2006. [25] T.
中文: 计算语言学协会, 2006. [25] T.
<a id="S0202"></a> Source: p.10 S0202
Original: Scitail: A textual entailment dataset from science question answering.
中文: 剪尾:从科学问题回答中得出的文字内涵数据集.
<a id="S0203"></a> Source: p.10 S0203
Original: Convolutional neural networks for sentence classification.
中文: 判决分类的革命神经网络.
<a id="S0204"></a> Source: p.10 S0204
Original: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [28] R.
中文: 亚当:一种花样优化的方法. arXiv preprint arXiv:1412.6980, 2014. [28] R.
<a id="S0205"></a> Source: p.10 S0205
Original: In Advances in neural information processing systems, pages 3294–3302, 2015. [29] N.
中文: In Advances in 神经信息处理系统, 第3294–3302页, 2015. [29]. 编号
<a id="S0206"></a> Source: p.10 S0206
Original: Constituency parsing with a self-attentive encoder.
中文: 选区解析用自意编码器.
<a id="S0207"></a> Source: p.10 S0207
Original: Race: Large-scale reading comprehension dataset from examinations.
中文: 种族:通过考试获得的大规模阅读理解数据集.
<a id="S0208"></a> Source: p.10 S0208
Original: Unsupervised machine translation using monolingual corpora only.
中文: 无人监督的机器翻译只使用单一语言的corpora.
<a id="S0209"></a> Source: p.10 S0209
Original: Distributed representations of sentences and documents.
中文: 分发判决和文件的说明。
<a id="S0210"></a> Source: p.10 S0210
Original: In International Conference on Machine Learning, pages 1188–1196, 2014. [33] P.
中文: 国际机器学习会议,第1188-1196页,2014 [33] P.
<a id="S0211"></a> Source: p.10 S0211
Original: Semi-supervised learning for natural language.
中文: 半监督学习自然语言.
<a id="S0212"></a> Source: p.10 S0212
Original: PhD thesis, Massachusetts Institute of Technology, 2005. [34] P. J.
中文: 2005年麻省理工学院博士论文[34] P.J.
<a id="S0213"></a> Source: p.10 S0213
Original: Generating wikipedia by summarizing long sequences.
中文: 通过总结长序来生成维基百科.
<a id="S0214"></a> Source: p.10 S0214
Original: Stochastic answer networks for natural language inference. arXiv preprint arXiv:1804.07888, 2018. [36] L.
中文: Stochastic 回答网络 自然语言推论. arXiv preprint arXiv:1804.078888, 2018. [36] L.
<a id="S0215"></a> Source: p.10 S0215
Original: An efficient framework for learning sentence representations.
中文: 学习判刑表述的有效框架。
<a id="S0216"></a> Source: p.10 S0216
Original: Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 2017. [38] B.
中文: 修复重衰变常态在Adam. arXiv预印 arXiv:1711.05101, 2017. [38]. B类.
<a id="S0217"></a> Source: p.10 S0217
Original: Learned in translation: Contextualized word vectors.
中文: 翻译学:背景化字向量.
<a id="S0218"></a> Source: p.10 S0218
Original: In Advances in Neural Information Processing Systems, pages 6297–6308, 2017. [39] T.
中文: (原始内容存档于2017-09-26). In Advances in Neural Information Systems, page 6297–6308, 2017. [39]. T.
<a id="S0219"></a> Source: p.10 S0219
Original: Distributed representations of words and phrases and their compositionality.
中文: 分布的表达词和短语及其组成.
<a id="S0220"></a> Source: p.10 S0220
Original: In Advances in neural information processing systems, pages 3111–3119, 2013. [40] N.
中文: 在神经信息处理系统的进步, 第3111–3119页, 2013. [40]. 编号
<a id="S0221"></a> Source: p.10 S0221
Original: Lsdsem 2017 shared task: The story cloze test.
中文: Lsdsem 2017 共享任务:故事血块测试.
<a id="S0222"></a> Source: p.10 S0222
Original: In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51, 2017. [41] K.
中文: 《关于将语言、感知和演讲层次语义模型联系起来的第二次讲习班记录》,第46-51页,2017. [41] K.
<a id="S0223"></a> Source: p.10 S0223
Original: Semi-supervised text classification using em.
中文: 使用em进行半监督文本分类.
<a id="S0224"></a> Source: p.10 S0224
Original: Semi-Supervised Learning, pages 33–56, 2006. [42] J.
中文: 半监督学习, 第33-56页, 2006. [42] J.
<a id="S0225"></a> Source: p.10 S0225
Original: Glove: Global vectors for word representation.
中文: 手套:用于文字表达的全球矢量.
<a id="S0226"></a> Source: p.10 S0226
Original: In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. [43] M. E.
中文: 2014年自然语言处理经验方法会议纪要(EMNLP),第1532–1543页,2014 [43] M. E.
<a id="S0227"></a> Source: p.10 S0227
Original: Semi-supervised sequence tagging with bidirectional language models.
中文: 用双向语言模型进行半监督序列标记.
<a id="S0228"></a> Source: p.10 S0228
Original: Deep contextualized word representations.
中文: 深为上下文的单词表达.
<a id="S0229"></a> Source: p.10 S0229
Original: When and why are pre-trained word embeddings useful for neural machine translation?
中文: 神经机能翻译何时及为何使用预训词嵌入?.
<a id="S0230"></a> Source: p.11 S0230
Original: Resolving complex cases of definite pronouns: the winograd schema challenge.
中文: 解决确定代名词的复杂案例:花花岗岩计划挑战.
<a id="S0231"></a> Source: p.11 S0231
Original: In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789.
中文: 2012年自然语言处理和计算自然语言学习经验方法联席会议纪要,第777-789页.
<a id="S0232"></a> Source: p.11 S0232
Original: Association for Computational Linguistics, 2012. [47] P.
中文: 计算语言学协会, 2012. [47] P.
<a id="S0233"></a> Source: p.11 S0233
Original: Squad: 100,000+ questions for machine comprehension of text.
中文: 分队:100,000+问题,用于机器理解文本.
<a id="S0234"></a> Source: p.11 S0234
Original: Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683, 2016. [49] M.
中文: 无监督的预训,用于序列学习. arXiv preprint arXiv:1611.02683,2016. [49] (英语). M.
<a id="S0235"></a> Source: p.11 S0235
Original: Efficient learning of sparse representations with an energy-based model.
中文: 以能动模式高效地学习稀有的表达方式.
<a id="S0236"></a> Source: p.11 S0236
Original: In Advances in neural information processing systems, pages 1137–1144, 2007. [50] M.
中文: 神经信息处理系统的进展,第1137-1144页,2007 [50] M.
<a id="S0237"></a> Source: p.11 S0237
Original: Semi-supervised multitask learning for sequence labeling.
中文: 半监督多任务学习,用于序列标签.
<a id="S0238"></a> Source: p.11 S0238
Original: Monro. A stochastic approximation method.
中文: 门罗. 斯多克近似法.
<a id="S0239"></a> Source: p.11 S0239
Original: The annals of mathematical statistics, pages 400–407, 1951. [52] T.
中文: 数学统计史料, 第400–407页, 1951. [52] T.
<a id="S0240"></a> Source: p.11 S0240
Original: Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664, 2015. [53] R.
中文: 与神经相通的原因 arXiv preprint arXiv:1509.0664,2015 [53] R.
<a id="S0241"></a> Source: p.11 S0241
Original: Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [54] R.
中文: 神经机译有分词单位的稀有词. arXiv preprint arXiv:1508.07909, 2015. [54] R.
<a id="S0242"></a> Source: p.11 S0242
Original: Recursive deep models for semantic compositionality over a sentiment treebank.
中文: 语义成分的回溯性深层模型 比情感树库。
<a id="S0243"></a> Source: p.11 S0243
Original: In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013. [55] S.
中文: 2013年自然语言处理经验方法会议纪要,第1631-1642页,2013 [55] 师曰.
<a id="S0244"></a> Source: p.11 S0244
Original: Riedl. A simple and effective approach to the story cloze test. arXiv preprint arXiv:1803.05547, 2018. [56] S.
中文: 瑞德尔 一个简单而有效的方法 故事的血浆测试。 arXiv preprint arXiv:1803.05547,2018. [56] (中文(简体) ). 师曰.
<a id="S0245"></a> Source: p.11 S0245
Original: Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079, 2018. [57] J.
中文: 通过大规模多任务学习,学习通用分布式的句子表述. arXiv preprint arXiv:1804.00079, 2018. [57] J.
<a id="S0246"></a> Source: p.11 S0246
Original: Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data.
中文: 使用千兆字缩放无标签数据进行半监督相继标签和分解.
<a id="S0247"></a> Source: p.11 S0247
Original: Proceedings of ACL-08: HLT, pages 665–673, 2008. [58] Y.
中文: ACL-08记录:HLT, 第665-673页, 2008. [58] Y.
<a id="S0248"></a> Source: p.11 S0248
Original: Hui. A compare-propagate architecture with alignment factorization for natural language inference. arXiv preprint arXiv:1801.00102, 2017. [59] Y.
中文: 慧来. 与自然语言推论的对等因子化相比较的平面结构. arXiv preprint arXiv:1801.00102,2017. [59] (中文(简体) ). 没错
<a id="S0249"></a> Source: p.11 S0249
Original: Multi-range reasoning for machine comprehension. arXiv preprint arXiv:1803.09074, 2018. [60] J.
中文: 机器理解的多程推理. arXiv预印 arXiv:1803.09074,2018. [60] J.
<a id="S0250"></a> Source: p.11 S0250
Original: Ecnu at semeval-2017 task 1: Leverage kernel-based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity.
中文: Ecnu at semeval-2017 job 1:以内核为基础的传统nlp特征和神经网络来构建多语种和跨语言语义文字相似性的通用模型.
<a id="S0251"></a> Source: p.11 S0251
Original: In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 191–197, 2017. [61] Y.
中文: 第11届语义学评价国际研讨会纪要(SemEval-2017),第191-197页,2017. [61]. 没错
<a id="S0252"></a> Source: p.11 S0252
Original: Opportunities and challenges in working with low-resource languages.
中文: 与低资源语言合作的机会和挑战。
<a id="S0253"></a> Source: p.11 S0253
Original: In Advances in Neural Information Processing Systems, pages 6000–6010, 2017. [63] P.
中文: In Advances in Neural Information Systems, pages 6000–6010, 2017. [63] P.
<a id="S0254"></a> Source: p.11 S0254
Original: Extracting and composing robust features with denoising autoencoders.
中文: 取出并编组带有去诺化自编码器的坚固特性.
<a id="S0255"></a> Source: p.11 S0255
Original: In Proceedings of the 25th international conference on Machine learning, pages 1096–1103.
中文: 《第25届机器学习国际会议记录》,第1096-1103页。
<a id="S0256"></a> Source: p.11 S0256
Original: Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [65] A.
中文: Glue:用于自然语言理解的多任务基准和分析平台. arXiv preprint arXiv:1804.07461, 2018. [65] (中文(简体) ). A类.
<a id="S0257"></a> Source: p.11 S0257
Original: Corpus of linguistic acceptability. http://nyu-mll.github.io/cola, 2018. [66] A.
中文: 语言可接受性。 http://nyu-mll.github.io/cola, 2018. [66] A.
<a id="S0258"></a> Source: p.11 S0258
Original: Bowman. A broad-coverage challenge corpus for sentence understanding through inference.
中文: 鲍曼. 通过推论理解判决的广覆盖质疑书。
<a id="S0259"></a> Source: p.11 S0259
Original: Towards human-level machine reading comprehension: Reasoning and inference with multiple strategies. arXiv preprint arXiv:1711.04964, 2017. 11
中文: 向人级机器读取理解:理性和对多种策略的推论. arXiv预印 arXiv:1711.04964,2017. 11 (英语).
<a id="S0260"></a> Source: p.12 S0260
Original: Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition.
中文: 预训和微调的作用 取决于背景 dbn-hmms 现实世界语音识别.
<a id="S0261"></a> Source: p.12 S0261
Original: NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010. [69] R.
中文: NIPS深度学习和无监督地特长学习讲习班,2010 [69] R.
<a id="S0262"></a> Source: p.12 S0262
Original: Split-brain autoencoders: Unsupervised learning by cross-channel prediction.
中文: 分化-大脑自发编码器:由跨通道预测进行无监督学习.
<a id="S0263"></a> Source: p.12 S0263
Original: In CVPR, volume 1, page 6, 2017. [70] X.
中文: 在CVPR中,第1卷,第6页,2017. [70] X.
<a id="S0264"></a> Source: p.12 S0264
Original: Semi-supervised learning literature survey. 2005. [71] Y.
中文: 半监督学习文献调查. 2005. [71] Y.
<a id="S0265"></a> Source: p.12 S0265
Original: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.
中文: 校正书籍和电影:通过观看电影和阅读书籍,走向故事般的视觉解释.
<a id="S0266"></a> Source: p.12 S0266
Original: In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015. 12
中文: 《IEEE计算机视野国际会议记录》,第19-27页,2015年。 第12条