Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or - 中英文对照
translated: 2026-07-16
title: "Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or" aliases: - "Transformer" - "arXiv:1706.03762" source: "https://arxiv.org/abs/1706.03762" arxiv: "1706.03762" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning - nlp
Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works.
中文: 只要提供了适当的归属,Google特此准许仅将本文中的表格和数字用于新闻或学术著作。
<a id="S0002"></a> Source: p.1 S0002
Original: Attention Is All You Need Ashish Vaswani∗ Noam Shazeer∗ Niki Parmar∗ Jakob Uszkoreit∗ Google Brain Google Brain Google Research Google Research avaswani@google.com noam@google.com nikip@google.com usz@google.com Llion Jones∗ Aidan N.
中文: All You need Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Google Brain Google Google研究 Avaswani@google.com noam@google.com nikip@google.com usz@google.com Llion Jones * Aidan N.
<a id="S0003"></a> Source: p.1 S0003
Original: Gomez∗ † Łukasz Kaiser∗ Google Research University of Toronto Google Brain llion@google.com aidan@cs.toronto.edu lukaszkaiser@google.com Illia Polosukhin∗ ‡ illia.polosukhin@gmail.com Abstract The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder.
中文: 多伦多谷歌研究大学 Google Brain lion@google.com aidan@cs.toronto.edu lukaszkaiser@google.com Illia Polosukhin * QQ hilia.polosukhin@gmail.com 摘要 主导序列转录模型基于复杂的反复或相生神经网络,其中包括了编码器和解码器.
<a id="S0004"></a> Source: p.1 S0004
Original: The best performing models also connect the encoder and decoder through an attention mechanism.
中文: 性能最好的模型也通过关注机制将编码器和解码器相接.
<a id="S0005"></a> Source: p.1 S0005
Original: We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
中文: 我们提出一个新的简单的网络架构,即"变形器",它完全基于关注机制,完全摆脱了再现和进化.
<a id="S0006"></a> Source: p.1 S0006
Original: Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train.
中文: 对两个机器翻译任务的实验表明,这些模型在质量上比较优越,同时比较相容,训练所需的时间也大大缩短.
<a id="S0007"></a> Source: p.1 S0007
Original: Our model achieves 28.4 BLEU on the WMT 2014 Englishto-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU.
中文: 我们的模式在2014年的WPT英德翻译任务上实现了28.4 BLEU,比包括综艺节目在内的现有最佳成果提高了2 BLEU以上.
<a id="S0008"></a> Source: p.1 S0008
Original: On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature.
中文: 在WMT 2014英法翻译任务上,我们的模式在8个GPU训练了3.5天后,确定了41.8分的新单模最先进的BLEU分数,是文献中最佳模型培训费用的一小部分.
<a id="S0009"></a> Source: p.1 S0009
Original: We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data. ∗Equal contribution.
中文: 我们显示,变形器将它成功地应用到英国选区,用大量有限的培训数据来分析,从而很好地概括了其他任务。 * 平等捐款。
<a id="S0010"></a> Source: p.1 S0010
Original: Jakob proposed replacing RNNs with self-attention and started the effort to evaluate this idea.
中文: Jakob提议用自我关注取代RNNs,并开始努力评价这一想法.
<a id="S0011"></a> Source: p.1 S0011
Original: Ashish, with Illia, designed and implemented the first Transformer models and has been crucially involved in every aspect of this work.
中文: Ashish与Illia一起设计和实施了第一个变形金刚模型,并积极参与了这项工作的每一个方面。
<a id="S0012"></a> Source: p.1 S0012
Original: Noam proposed scaled dot-product attention, multi-head attention and the parameter-free position representation and became the other person involved in nearly every detail.
中文: 诺姆提议扩大点产品关注度,多头关注度和无参数位置代表度,并成为几乎每个细节都涉及的另一人.
<a id="S0013"></a> Source: p.1 S0013
Original: Niki designed, implemented, tuned and evaluated countless model variants in our original codebase and tensor2tensor.
中文: Niki设计,执行,调制和评估了我们最初的代码库和收音机中无数的模型变体.
<a id="S0014"></a> Source: p.1 S0014
Original: Llion also experimented with novel model variants, was responsible for our initial codebase, and efficient inference and visualizations.
中文: Llion还实验了新颖的模型变体, 负责我们最初的代码库, 以及高效的推论和可视化。
<a id="S0015"></a> Source: p.1 S0015
Original: Lukasz and Aidan spent countless long days designing various parts of and implementing tensor2tensor, replacing our earlier codebase, greatly improving results and massively accelerating our research. †Work performed while at Google Brain. ‡Work performed while at Google Research. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. 3202 guA 2 ]LC.sc[ 7v26730.6071:viXra
中文: Lukasz和Aidan花费了无数长日,设计并执行了各种抗热剂,取代了我们早先的编码基础,大大改进了成果并大规模地加快了我们的研究工作。 在谷歌脑部工作 在Google Research工作期间, 第31届神经信息处理系统会议(NIPS 2017),CA. CA. 3202 guA 2]LC.sc [7v26730.6071:viXra
<a id="S0016"></a> Source: p.2 S0016
Original: 1 Introduction Recurrent neural networks, long short-term memory [13] and gated recurrent [7] neural networks in particular, have been firmly established as state of the art approaches in sequence modeling and transduction problems such as language modeling and machine translation [35, 2, 5].
中文: 1 经常神经网络,长期短期记忆[13],并特别有门的经常神经网络[7] 已牢固地确立为诸如语言建模和机器翻译等测序和转录问题的先进方法[35,2,5].
<a id="S0017"></a> Source: p.2 S0017
Original: Numerous efforts have since continued to push the boundaries of recurrent language models and encoder-decoder architectures [38, 24, 15].
中文: 此后,许多努力继续推进了反复出现的语言模型和编码器-解码器架构的界限[38,24,15].
<a id="S0018"></a> Source: p.2 S0018
Original: Recurrent models typically factor computation along the symbol positions of the input and output sequences.
中文: 经常模型一般按照输入和输出序列的符号位置来计算因子.
<a id="S0019"></a> Source: p.2 S0019
Original: Aligning the positions to steps in computation time, they generate a sequence of hidden states h , as a function of the previous hidden state h and the input for position t.
中文: 在计算时间中将位置与步骤相匹配,它们产生隐藏状态h的序列,作为之前隐藏状态h和位置t的输入函数.
<a id="S0020"></a> Source: p.2 S0020
Original: This inherently t t−1 sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths, as memory constraints limit batching across examples.
中文: 这种固有的t-1相继性质排除了培训实例中的平行性,在较长的序列长度中,这种平行性变得至关重要,因为内存限制限制了对实例的分批处理。
<a id="S0021"></a> Source: p.2 S0021
Original: Recent work has achieved significant improvements in computational efficiency through factorization tricks [21] and conditional computation [32], while also improving model performance in case of the latter.
中文: 最近的工作通过因子化技巧[21]和有条件的计算[32]在计算效率上取得了显著的提高,同时在后者的情况下也改善了模型性能.
<a id="S0022"></a> Source: p.2 S0022
Original: The fundamental constraint of sequential computation, however, remains.
中文: 然而,顺序计算的根本制约因素依然存在。
<a id="S0023"></a> Source: p.2 S0023
Original: Attention mechanisms have become an integral part of compelling sequence modeling and transduction models in various tasks, allowing modeling of dependencies without regard to their distance in the input or output sequences [2, 19].
中文: 注意机制已经成为各种任务中令人信服的序列模型和转录模型的一个组成部分,使得依赖物的模型可以不考虑它们在输入或输出序列中的距离[2,19].
<a id="S0024"></a> Source: p.2 S0024
Original: In all but a few cases [27], however, such attention mechanisms are used in conjunction with a recurrent network.
中文: 然而,除少数情况外,所有其他情况[27],这种注意机制都与经常性网络结合使用。
<a id="S0025"></a> Source: p.2 S0025
Original: In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output.
中文: 在这部作品中,我们提出"变形器",一种模式架构避免了再现,而是完全依靠关注机制来吸引投入和产出之间的全球依赖.
<a id="S0026"></a> Source: p.2 S0026
Original: The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality after being trained for as little as twelve hours on eight P100 GPUs. 2 Background The goal of reducing sequential computation also forms the foundation of the Extended Neural GPU [16], ByteNet [18] and ConvS2S [9], all of which use convolutional neural networks as basic building block, computing hidden representations in parallel for all input and output positions.
中文: "变形器"在8个P100GPU上接受了长达12小时的训练后,可以显著地实现更平行化,并可以在翻译质量上达到一个新的水平. 2 背景情况 减少相继计算的目标也构成了扩展神经GPU [16],ByteNet [18]和ConvS2S [9]的基础,它们都以入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入
<a id="S0027"></a> Source: p.2 S0027
Original: In these models, the number of operations required to relate signals from two arbitrary input or output positions grows in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet.
中文: 在这些模型中,连接两个任意输入或输出位置的信号所需的操作数量在位置之间的距离上增加,ConvS2S的线性操作和ByteNet的对数操作.
<a id="S0028"></a> Source: p.2 S0028
Original: This makes it more difficult to learn dependencies between distant positions [12].
中文: 这使得在遥远的位置之间学习依赖性更加困难[12].
<a id="S0029"></a> Source: p.2 S0029
Original: In the Transformer this is reduced to a constant number of operations, albeit at the cost of reduced effective resolution due to averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as described in section 3.2.
中文: 在 " 变形器 " 中,这已减少到经常的作业次数,尽管由于平均注意力加权位置而降低了有效分辨率,但正如第3.2节所描述的那样,我们用多头注意力抵消了这种影响。
<a id="S0030"></a> Source: p.2 S0030
Original: Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.
中文: 自取自取,有时也叫取自自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取自取
<a id="S0031"></a> Source: p.2 S0031
Original: Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations [4, 27, 28, 22].
中文: 在各种任务中,包括阅读理解、抽象归纳、文字含义和学习任务独立的句子表述(第4、27、28、22段),成功地使用了自我注意。
<a id="S0032"></a> Source: p.2 S0032
Original: End-to-end memory networks are based on a recurrent attention mechanism instead of sequencealigned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks [34].
中文: 端到端内存网络基于反复关注机制,而不是序列相接再现,在简单语言问答和语言建模任务上表现良好[34].
<a id="S0033"></a> Source: p.2 S0033
Original: To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequencealigned RNNs or convolution.
中文: 然而,据我们所知,变形器是第一个完全依靠自觉来计算其输入和输出的表示而不用序列相通的RNNs或演化的转录模型.
<a id="S0034"></a> Source: p.2 S0034
Original: In the following sections, we will describe the Transformer, motivate self-attention and discuss its advantages over models such as [17, 18] and [9]. 3 Model Architecture Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].
中文: 在以下各节中,我们将描述变形器,激发自我关注,并讨论其优于模型,如[17,18]和[9]. 3 Model Architecture 最具竞争力的神经序列转录模型具有编码器-解码器结构[5,2,35].
<a id="S0035"></a> Source: p.2 S0035
Original: Here, the encoder maps an input sequence of symbol representations (x , ..., x ) to a sequence 1 n of continuous representations z = (z , ..., z ).
中文: 在这里,编码器将符号表示(x,.,x)的输入序列映射到连续表示z=(z,.,z.)的序列上.
<a id="S0036"></a> Source: p.2 S0036
Original: Given z, the decoder then generates an output 1 n sequence (y , ..., y ) of symbols one element at a time.
中文: Given z,解码器然后一次生成一个符号一个元素的输出 1 n 序列(y,.,y).
<a id="S0037"></a> Source: p.2 S0037
Original: At each step the model is auto-regressive 1 m [10], consuming the previously generated symbols as additional input when generating the next. 2
中文: 在每一步中,模型都是自递回式的 1 m [10],在生成下一个时消耗了先前产生的符号作为附加输入. 2个
<a id="S0038"></a> Source: p.3 S0038
Original: Figure 1: The Transformer - model architecture.
中文: 图1:"变形"-模型架构.
<a id="S0039"></a> Source: p.3 S0039
Original: The Transformer follows this overall architecture using stacked self-attention and point-wise, fully connected layers for both the encoder and decoder, shown in the left and right halves of Figure 1, respectively. 3.1 Encoder and Decoder Stacks Encoder: The encoder is composed of a stack of N = 6 identical layers.
中文: 变形器遵循这个整体架构,采用堆叠自意和点向,完全连接的地层,用于编码器和解码器,分别以图1的左半部分和右半部分显示. 3.1 编码器和解码器堆叠式编码器:编码器由一叠由N=6等同层所组成.
<a id="S0040"></a> Source: p.3 S0040
Original: The first is a multi-head self-attention mechanism, and the second is a simple, positionwise fully connected feed-forward network.
中文: 第一种是多头自通机制,第二种是简单,位置明智地完全连接到向导网络.
<a id="S0041"></a> Source: p.3 S0041
Original: We employ a residual connection [11] around each of the two sub-layers, followed by layer normalization [1].
中文: 我们使用一个残余的连接[11] 围绕两个子层,然后层态化[1].
<a id="S0042"></a> Source: p.3 S0042
Original: That is, the output of each sub-layer is LayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer itself.
中文: 即每个子层的输出是LayerNorm(x + sublayer(x)),其中子层(x)是子层本身所执行的函数.
<a id="S0043"></a> Source: p.3 S0043
Original: To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers, produce outputs of dimension d = 512. model Decoder: The decoder is also composed of a stack of N = 6 identical layers.
中文: 为了方便这些剩余连接,模型中的所有子层以及嵌入层都会产生维度d=512的输出. 模型解码器 : 解码器还由一叠由N=6个同层相接而成.
<a id="S0044"></a> Source: p.3 S0044
Original: In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack.
中文: 除了每个编码器层的两个子层外,解码器还插入了第三个子层,在编码器堆栈的输出上进行多头的注意.
<a id="S0045"></a> Source: p.3 S0045
Original: Similar to the encoder, we employ residual connections around each of the sub-layers, followed by layer normalization.
中文: 与编码器相类似,我们在每个子层周围采用残余连接,然后进行层态化.
<a id="S0046"></a> Source: p.3 S0046
Original: We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions.
中文: 我们还修改了解码器堆栈中的自留分层来防止位置前往后续位置.
<a id="S0047"></a> Source: p.3 S0047
Original: This masking, combined with fact that the output embeddings are offset by one position, ensures that the predictions for position i can depend only on the known outputs at positions less than i. 3.2 Attention An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors.
中文: 这种遮掩,加上输出嵌入被一个位置所抵消的事实,确保了对我位置的预测只能依靠低于3.2位的已知输出. 注意 注意功能可以被描述为将查询和一组密钥值对映射到输出,其中查询,密钥,值和输出都是向量.
<a id="S0048"></a> Source: p.3 S0048
Original: The output is computed as a weighted sum 3
中文: 产出按加权和3计算
<a id="S0049"></a> Source: p.4 S0049
Original: Scaled Dot-Product Attention Multi-Head Attention Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key. 3.2.1 Scaled Dot-Product Attention We call our particular attention "Scaled Dot-Product Attention" (Figure 2).
中文: 缩放 Dot-Product 注意多头图2: (左)缩放 Dot-Product 注意多头图. (右)多头注意由平行运行的几个注意层组成. 中,其中每个值的权重由查询与相应密钥相容的函数计算。 3.2.1 规模化点-生产注意 我们特别提请注意“规模点生产注意”(图2)。
<a id="S0050"></a> Source: p.4 S0050
Original: The input consists of queries and keys of dimension d k , a√nd values of dimension d v .
中文: 输入内容由维度dk的查询和键,维度dv的a√nd值.
<a id="S0051"></a> Source: p.4 S0051
Original: We compute the dot products of the query with all keys, divide each by d , and apply a softmax function to obtain the weights on the k values.
中文: 我们用所有键计算查询的点产品, 将每个键除以 d , 并应用一个软max 函数来获取 k 值的权重 。
<a id="S0052"></a> Source: p.4 S0052
Original: In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q.
中文: 在实际操作中,我们同时计算一组查询的注意力功能,组合成矩阵Q.
<a id="S0053"></a> Source: p.4 S0053
Original: The keys and values are also packed together into matrices K and V .
中文: 键和值还被组合成矩阵K和矩阵V。
<a id="S0054"></a> Source: p.4 S0054
Original: We compute the matrix of outputs as: QKT Attention(Q, K, V ) = softmax( √ )V (1) d k The two most commonly used attention functions are additive attention [2], and dot-product (multiplicative) attention.
中文: 我们计算输出矩阵为: QKT 注意 (Q, K, V) = s软max (~) V (1) d k 两种最常用的注意功能是: 添加式注意 [2] 和 点-产物 (多相) 注意.
<a id="S0055"></a> Source: p.4 S0055
Original: Dot-product attention is identical to our algorithm, except for the scaling factor of √1 .
中文: 点-产品注意度与我们的算法相同,除了缩放系数为"% 1".
<a id="S0056"></a> Source: p.4 S0056
Original: Additive attention computes the compatibility function using a feed-forward network with dk a single hidden layer.
中文: 添加注意会使用有 dk 单个隐藏层的向导网络来计算兼容功能.
<a id="S0057"></a> Source: p.4 S0057
Original: While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code.
中文: 虽然两者在理论复杂性上相似,但点产品关注在实践中要快得多,空间效率更高,因为它可以使用高度优化的矩阵乘法来实施.
<a id="S0058"></a> Source: p.4 S0058
Original: While for small values of d the two mechanisms perform similarly, additive attention outperforms k dot product attention without scaling for larger values of d [3].
中文: 虽然两种机制的d小值具有类似作用,但添加剂的注意量超过了k点产品注意量,而没有缩小较大的d[3]值。
<a id="S0059"></a> Source: p.4 S0059
Original: We suspect that for large values of k d , the dot products grow large in magnitude, pushing the softmax function into regions where it has k extremely small gradients 4.
中文: 我们怀疑,对于K d 的大型值,点出产物的体积越来越大,将软马克函数推入了有K极小梯度4的区域.
<a id="S0060"></a> Source: p.4 S0060
Original: To counteract this effect, we scale the dot products by √1 . dk 3.2.2 Multi-Head Attention Instead of performing a single attention function with d -dimensional keys, values and queries, model we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to d , d and d dimensions, respectively.
中文: 为了抵消这种影响,我们用"% 1"(dk 3.2.2多头注意)来缩放点产品,而不是用d维键、值和查询来履行单一的注意功能,我们发现模型将查询、值和值h的时间线性地分别投射到d、d和d维。
<a id="S0061"></a> Source: p.4 S0061
Original: On each of these projected versions of k k v queries, keys and values we then perform the attention function in parallel, yielding d -dimensional v 4To illustrate why the dot products get large, assume that the components of q and k are independent random variables with mean 0 and variance 1.
中文: 在这些预测的 k k v 查询、 键和值的每个版本上, 我们都会平行执行注意功能, 生成 d - 维 v 4 To 说明点产品为什么变大, 假设 q 和 k 的组件是独立的随机变量, 平均值为 0, 差异为 1 。
<a id="S0062"></a> Source: p.4 S0062
Original: Then their dot product, q · k = (cid:80)dk q k , has mean 0 and variance d . i=1 i i k 4
中文: 然后它们的点产品 q → k = (cid: 80)dk q k, 表示 0 和 相差 d. i = 1 i i k 4
<a id="S0063"></a> Source: p.5 S0063
Original: These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.
中文: 如图2所显示的,这些数值被压缩并再次预测,得出了最后数值。
<a id="S0064"></a> Source: p.5 S0064
Original: Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.
中文: 多头注意使模型能够在不同位置共同处理来自不同代表子空间的信息.
<a id="S0065"></a> Source: p.5 S0065
Original: With a single attention head, averaging inhibits this.
中文: 只有一个注意头, 平均抑制了这一点。
<a id="S0066"></a> Source: p.5 S0066
Original: MultiHead(Q, K, V ) = Concat(head , ..., head )W O 1 h where head = Attention(QW Q, KW K, V W V ) i i i i Where the projections are parameter matrices W Q ∈ Rdmodel×dk , W K ∈ Rdmodel×dk , W V ∈ Rdmodel×dv i i i and W O ∈ Rhdv×dmodel.
中文: Multihead(Q, K, V) = Concat(head, ..., head) W O 1 h 其中head = 注意 (QW Q, KW K, V W V V V) i i 预测是参数矩阵 W Q Q Rdmodel×dk, W K ∈ Rdmodel×dk, W V ∈ Rdmodel×dv i and W O ∈ Rhdv×dmodel.
<a id="S0067"></a> Source: p.5 S0067
Original: In this work we employ h = 8 parallel attention layers, or heads.
中文: 在这项工作中,我们使用h=8平行的注意力层,或头.
<a id="S0068"></a> Source: p.5 S0068
Original: For each of these we use d = d = d /h = 64.
中文: 每一个都使用d=d=d=d/h=64.
<a id="S0069"></a> Source: p.5 S0069
Original: Due to the reduced dimension of each head, the total computational cost k v model is similar to that of single-head attention with full dimensionality. 3.2.3 Applications of Attention in our Model The Transformer uses multi-head attention in three different ways: • In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder.
中文: 由于每个头的尺寸减少,计算总成本克v模型与全尺寸的单头注意相类似. 3.2.3 在我们的"变形金刚"模型中注意的应用多头注意用三种不同的方式: 在"编码器-解码器注意"分层中,查询来自之前的解码器分层,内存键和值来自编码器的输出.
<a id="S0070"></a> Source: p.5 S0070
Original: This allows every position in the decoder to attend over all positions in the input sequence.
中文: 这使得解码器中的每一个位置都能在输入序列中的所有位置上参加.
<a id="S0071"></a> Source: p.5 S0071
Original: This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models such as [38, 2, 9]. • The encoder contains self-attention layers.
中文: 这模仿了典型的编码器-解码器注意机制在序列到序列模型中,如[38, 2, 9]. ^ 编码器包含自意分层.
<a id="S0072"></a> Source: p.5 S0072
Original: In a self-attention layer all of the keys, values and queries come from the same place, in this case, the output of the previous layer in the encoder.
中文: 在自留层中,所有的密钥,值和查询都来自同一个地方,在这种情况下,编码器中前一层的输出.
<a id="S0073"></a> Source: p.5 S0073
Original: Each position in the encoder can attend to all positions in the previous layer of the encoder. • Similarly, self-attention layers in the decoder allow each position in the decoder to attend to all positions in the decoder up to and including that position.
中文: 编码器中的每个位置都可以关注编码器上一层的所有位置. • 同样地,解码器中的自留层使解码器中的每个位置都能够处理解码器中直至并包括该位置的所有位置.
<a id="S0074"></a> Source: p.5 S0074
Original: We need to prevent leftward information flow in the decoder to preserve the auto-regressive property.
中文: 要防止解码器中左倾信息流,以保存自发后退地产.
<a id="S0075"></a> Source: p.5 S0075
Original: We implement this inside of scaled dot-product attention by masking out (setting to −∞) all values in the input of the softmax which correspond to illegal connections.
中文: 我们通过掩盖(设定为)与非法联系相对应的软马克斯输入中的所有值,在规模化的点产品关注范围内实施这一措施。
<a id="S0076"></a> Source: p.5 S0076
Original: See Figure 2. 3.3 Position-wise Feed-Forward Networks In addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully connected feed-forward network, which is applied to each position separately and identically.
中文: 见图2.3.3 位置方向饲料前进网络 除了注意子层外,我们编码器和解码器的每个层都包含一个完全连接的向导线网,这个线网分别和平等地应用于每个位置.
<a id="S0077"></a> Source: p.5 S0077
Original: This consists of two linear transformations with a ReLU activation in between.
中文: 这包括两个线性转换,中间有一个再LU活化.
<a id="S0078"></a> Source: p.5 S0078
Original: FFN(x) = max(0, xW + b )W + b (2) 1 1 2 2 While the linear transformations are the same across different positions, they use different parameters from layer to layer.
中文: FFN(x) = max(0), xW + b) W + b (2) 1 1 2 2 虽然线性变换在不同位置上是相同的,但它们使用不同的分层参数.
<a id="S0079"></a> Source: p.5 S0079
Original: Another way of describing this is as two convolutions with kernel size 1.
中文: 另一种描述方式是作为内核尺寸为1的两组进取.
<a id="S0080"></a> Source: p.5 S0080
Original: The dimensionality of input and output is d = 512, and the inner-layer has dimensionality model d = 2048. ff 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension d .
中文: 输入和输出的维度为d=512,内层有维度模型d=2048. 对应3.4 嵌入和Softmax 和其他序列转录模型类似,我们使用所学入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出出
<a id="S0081"></a> Source: p.5 S0081
Original: We also use the usual learned linear transformodel mation and softmax function to convert the decoder output to predicted next-token probabilities.
中文: 我们还使用通常所学的线性变形元模和软磁函数来将解码器输出转换为预测下个托肯概率.
<a id="S0082"></a> Source: p.5 S0082
Original: In our model, we share the same weight matrix between the two embedding layers and the pre-√softmax linear transformation, similar to [30].
中文: 在我们的模型中,我们分享了两个嵌入层和前-软max线性变换之间的相同重量矩阵,类似于[30].
<a id="S0083"></a> Source: p.5 S0083
Original: In the embedding layers, we multiply those weights by d . model 5
中文: 在嵌入层中,我们把这些重量乘以d. 型号5
<a id="S0084"></a> Source: p.6 S0084
Original: Table 1: Maximum path lengths, per-layer complexity and minimum number of sequential operations for different layer types. n is the sequence length, d is the representation dimension, k is the kernel size of convolutions and r the size of the neighborhood in restricted self-attention.
中文: 表1:不同地层类型的最大路径长度,每层复杂度和相继操作的最小数量. n为序列长度,d为表达维度,k为内核大小的演化和r相邻的大小在受限自觉.
<a id="S0085"></a> Source: p.6 S0085
Original: Layer Type Complexity per Layer Sequential Maximum Path Length Operations Self-Attention O(n2 · d) O(1) O(1) Recurrent O(n · d2) O(n) O(n) Convolutional O(k · n · d2) O(1) O(log (n)) k Self-Attention (restricted) O(r · n · d) O(1) O(n/r) 3.5 Positional Encoding Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence.
中文: 每层地层类型复杂度 顺序最大路径长度操作 自留 O(n2-d) O(1) O(1) 经常性 O(n-d2) O(n) O(n) 演化 O(k-n-d2) O(1) O(log(n)) k 自留 (受限制) O(r-n-d) O(1) O(n/r) 3.5 定位编码 由于我们的模型不包含重现和演化,为了使模型能够利用序列的顺序,我们必须在序列中注入一些关于符号相对或绝对位置的信息.
<a id="S0086"></a> Source: p.6 S0086
Original: To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks.
中文: 为此,在编码器和解码器堆栈底部的输入嵌入物上再加入"位置编码".
<a id="S0087"></a> Source: p.6 S0087
Original: The positional encodings have the same dimension d model as the embeddings, so that the two can be summed.
中文: 位置编码的维度d型号与嵌入型号相同,可以将两者相加.
<a id="S0088"></a> Source: p.6 S0088
Original: There are many choices of positional encodings, learned and fixed [9].
中文: 位置编码有许多选择,学习和固定[9].
<a id="S0089"></a> Source: p.6 S0089
Original: In this work, we use sine and cosine functions of different frequencies: P E = sin(pos/100002i/dmodel) (pos,2i) P E = cos(pos/100002i/dmodel) (pos,2i+1) where pos is the position and i is the dimension.
中文: 在这部作品中,我们使用不同频率的正弦和相弦函数: P E = sin(pos/100002i/dmodel)(pos 2i) P E = cos(pos/100002i/dmodel)(pos 2i+1) 其中pos是位置而i是维度.
<a id="S0090"></a> Source: p.6 S0090
Original: That is, each dimension of the positional encoding corresponds to a sinusoid.
中文: 也就是说,位置编码的每个维度都对应一个sinusoid.
<a id="S0091"></a> Source: p.6 S0091
Original: The wavelengths form a geometric progression from 2π to 10000 · 2π.
中文: 这些波长形成从2π到10000 → 2π的几何相上演.
<a id="S0092"></a> Source: p.6 S0092
Original: We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, P E can be represented as a linear function of pos+k P E . pos We also experimented with using learned positional embeddings [9] instead, and found that the two versions produced nearly identical results (see Table 3 row (E)).
中文: 我们之所以选择这一功能,是因为我们假设可以让模型很容易地通过相对位置来学习,因为对于任何固定的相冲克,PE都可以被作为 pos+k P E 的线性函数来表示. pos我们还试验使用已学习到的位置嵌入[9],发现两个版本的结果几乎相同(见表3行(E))。
<a id="S0093"></a> Source: p.6 S0093
Original: We chose the sinusoidal version because it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training. 4 Why Self-Attention In this section we compare various aspects of self-attention layers to the recurrent and convolutional layers commonly used for mapping one variable-length sequence of symbol representations (x , ..., x ) to another sequence of equal length (z , ..., z ), with x , z ∈ Rd, such as a hidden 1 n 1 n i i layer in a typical sequence transduction encoder or decoder.
中文: 我们选择了sinusoidal版本,因为它可能允许模型推断出序列长度比训练中遇到的长度长. 4 为什么是自留 在本节中,我们将自留层的各个方面与通常用于将一个可变长的符号表示序列(x,.,x)与另一个等长的序列(z,.,z)相比较,与x,z的Rd相比较,例如典型序列转录编码器或解码器中隐藏的1 n 1 n i i 层.
<a id="S0094"></a> Source: p.6 S0094
Original: Motivating our use of self-attention we consider three desiderata.
中文: 激励我们使用自我关注,我们考虑三个去除。
<a id="S0095"></a> Source: p.6 S0095
Original: One is the total computational complexity per layer.
中文: 一是每层计算总复杂度.
<a id="S0096"></a> Source: p.6 S0096
Original: Another is the amount of computation that can be parallelized, as measured by the minimum number of sequential operations required.
中文: 另一种是可平行的计算量,以所需最小相继操作数来计量.
<a id="S0097"></a> Source: p.6 S0097
Original: The third is the path length between long-range dependencies in the network.
中文: 第三是网络中远程依赖之间的路径长度.
<a id="S0098"></a> Source: p.6 S0098
Original: Learning long-range dependencies is a key challenge in many sequence transduction tasks.
中文: 学习远距离依赖性是许多顺序转换任务的关键挑战.
<a id="S0099"></a> Source: p.6 S0099
Original: One key factor affecting the ability to learn such dependencies is the length of the paths forward and backward signals have to traverse in the network.
中文: 影响了解这种依赖性能力的一个关键因素是前进路径的长度和落后信号必须在网络中穿行。
<a id="S0100"></a> Source: p.6 S0100
Original: The shorter these paths between any combination of positions in the input and output sequences, the easier it is to learn long-range dependencies [12].
中文: 这些路径在输入序列和输出序列中任何位置组合之间的时间越短,就越容易学习远程依赖性[12].
<a id="S0101"></a> Source: p.6 S0101
Original: Hence we also compare the maximum path length between any two input and output positions in networks composed of the different layer types.
中文: 因此,我们还比较了由不同层类型组成的网络中任何两个输入和输出位置的最大路径长度。
<a id="S0102"></a> Source: p.6 S0102
Original: As noted in Table 1, a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.
中文: 如表1所指出,自留层将所有位置与连续执行的操作数量相接,而经常层则需要O(n)顺序操作。
<a id="S0103"></a> Source: p.6 S0103
Original: In terms of computational complexity, self-attention layers are faster than recurrent layers when the sequence 6
中文: 在计算复杂度方面,当序列 6 时,自觉分层比再生分层快
<a id="S0104"></a> Source: p.7 S0104
Original: length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models in machine translations, such as word-piece [38] and byte-pair [31] representations.
中文: n 的长度小于表示维度d,最常见的是机器翻译中最先进的模型所使用的句子表示,如单词 [38]和字节-pair [31] 表示.
<a id="S0105"></a> Source: p.7 S0105
Original: To improve computational performance for tasks involving very long sequences, self-attention could be restricted to considering only a neighborhood of size r in the input sequence centered around the respective output position.
中文: 为了改进涉及非常长的序列的任务的计算性能,自我注意可仅限于考虑在围绕各自输出位置的输入序列中,只考虑大小为r的相邻.
<a id="S0106"></a> Source: p.7 S0106
Original: This would increase the maximum path length to O(n/r).
中文: 这将增加最大路径长度到 O(n/r) 。
<a id="S0107"></a> Source: p.7 S0107
Original: We plan to investigate this approach further in future work. A single convolutional layer with kernel width k < n does not connect all pairs of input and output positions.
中文: 我们计划在今后的工作中进一步调查这一做法。 内核宽度为k < n的单向演化层不连接所有对输入和输出位置.
<a id="S0108"></a> Source: p.7 S0108
Original: Doing so requires a stack of O(n/k) convolutional layers in the case of contiguous kernels, or O(log (n)) in the case of dilated convolutions [18], increasing the length of the longest paths k between any two positions in the network.
中文: 这样做需要在相接内核的情况下需要一叠O(n/k)分生层,在分生分生后则需要一叠O(log(n))分生层,增加网络中任意两个位置之间最长的路径k的长度.
<a id="S0109"></a> Source: p.7 S0109
Original: Convolutional layers are generally more expensive than recurrent layers, by a factor of k.
中文: 革命地层一般比再生地层更昂贵,以k为因子.
<a id="S0110"></a> Source: p.7 S0110
Original: Separable convolutions [6], however, decrease the complexity considerably, to O(k · n · d + n · d2).
中文: 但可分化的分化[6]将复杂程度大幅降低到O(k-n-d-d+n-d2).
<a id="S0111"></a> Source: p.7 S0111
Original: Even with k = n, however, the complexity of a separable convolution is equal to the combination of a self-attention layer and a point-wise feed-forward layer, the approach we take in our model.
中文: 然而,即使用 k = n 来表示,一个可分化的相接体的复杂程度,也等于一个自取自取的地层和一个点入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入入
<a id="S0112"></a> Source: p.7 S0112
Original: As side benefit, self-attention could yield more interpretable models.
中文: 作为附带好处,自觉可产生更能解释的模式。
<a id="S0113"></a> Source: p.7 S0113
Original: We inspect attention distributions from our models and present and discuss examples in the appendix.
中文: 我们从模型中检查注意的分布情况,并在附录中介绍和讨论例子。
<a id="S0114"></a> Source: p.7 S0114
Original: Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences. 5 Training This section describes the training regime for our models. 5.1 Training Data and Batching We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs.
中文: 不仅个别的注意力头明确学会执行不同的任务,许多似乎表现出了与句子的合成和语义结构有关的行为. 5 培训 本节介绍我们模式的培训制度。 5.1 培训数据和打击 我们接受了由大约450万个句子组成的标准WPT 2014英德数据集的培训.
<a id="S0115"></a> Source: p.7 S0115
Original: Sentences were encoded using byte-pair encoding [3], which has a shared sourcetarget vocabulary of about 37000 tokens.
中文: 句子被用字节-pair编码[3]来编码,该词有约37000个令牌的共享源目标词汇.
<a id="S0116"></a> Source: p.7 S0116
Original: For English-French, we used the significantly larger WMT 2014 English-French dataset consisting of 36M sentences and split tokens into a 32000 word-piece vocabulary [38].
中文: 对英语-法语来说,我们使用了显著更大的WPT 2014英语-法语数据集,由36M句子和分出符符组成为32000个字形词汇[38].
<a id="S0117"></a> Source: p.7 S0117
Original: Sentence pairs were batched together by approximate sequence length.
中文: 句子对数按大致顺序长度进行分批。
<a id="S0118"></a> Source: p.7 S0118
Original: Each training batch contained a set of sentence pairs containing approximately 25000 source tokens and 25000 target tokens. 5.2 Hardware and Schedule We trained our models on one machine with 8 NVIDIA P100 GPUs.
中文: 每批训练中包含一组句子对,包含约25000个源令牌和25000个目标令牌. 5.2 硬件和时间表 我们用8台NVIDIA P100 GPU的一台机器训练我们的模型.
<a id="S0119"></a> Source: p.7 S0119
Original: For our base models using the hyperparameters described throughout the paper, each training step took about 0.4 seconds.
中文: 对于我们使用整个论文中描述的超参数的基本模型来说,每个训练步骤大约用了0.4秒。
<a id="S0120"></a> Source: p.7 S0120
Original: We trained the base models for a total of 100,000 steps or 12 hours.
中文: 我们总共训练了10万步或12小时的基地模型.
<a id="S0121"></a> Source: p.7 S0121
Original: For our big models,(described on the bottom line of table 3), step time was 1.0 seconds.
中文: 对我们的大模型来说(在表3的底线上描述),步骤时间是1.0秒.
<a id="S0122"></a> Source: p.7 S0122
Original: The big models were trained for 300,000 steps (3.5 days). 5.3 Optimizer We used the Adam optimizer [20] with β = 0.9, β = 0.98 and ϵ = 10−9.
中文: 大模型接受了30万步(3.5天)的培训. 5.3 优化剂 我们使用了 Adam 优化器 [20] 有 β = 0.9, β = 0.98和 + 10 - 9.
<a id="S0123"></a> Source: p.7 S0123
Original: We varied the learning 1 2 rate over the course of training, according to the formula: lrate = d−0.5 · min(step_num−0.5, step_num · warmup_steps−1.5) (3) model This corresponds to increasing the learning rate linearly for the first warmup_steps training steps, and decreasing it thereafter proportionally to the inverse square root of the step number.
中文: 在训练过程中,我们按照公式改变学习率12:larat=d-0.5=min(step num-0.5,step-num-warup-steps-1.5)(3)模式 这相当于对第一个取暖步骤的学习率线性地提高,然后与步数倒数平方根成正比地降低。
<a id="S0124"></a> Source: p.7 S0124
Original: We used warmup_steps = 4000. 5.4 Regularization We employ three types of regularization during training: 7
中文: 我们用取暖的步数=4000 5.4 正规化 在培训期间,我们采用三种形式:
<a id="S0125"></a> Source: p.8 S0125
Original: Table 2: The Transformer achieves better BLEU scores than previous state-of-the-art models on the English-to-German and English-to-French newstest2014 tests at a fraction of the training cost.
中文: 表2:"变形金刚"在英语对德和英语对法新闻测试2014年测试中,以培训成本的一分之一的成绩,比之前的"最先进"模式获得了更好的BLEU分数.
<a id="S0126"></a> Source: p.8 S0126
Original: BLEU Training Cost (FLOPs) Model EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 Deep-Att + PosUnk [39] 39.2 1.0 · 1020 GNMT + RL [38] 24.6 39.92 2.3 · 1019 1.4 · 1020 ConvS2S [9] 25.16 40.46 9.6 · 1018 1.5 · 1020 MoE [32] 26.03 40.56 2.0 · 1019 1.2 · 1020 Deep-Att + PosUnk Ensemble [39] 40.4 8.0 · 1020 GNMT + RL Ensemble [38] 26.30 41.16 1.8 · 1020 1.1 · 1021 ConvS2S Ensemble [9] 26.36 41.29 7.7 · 1019 1.2 · 1021 Transformer (base model) 27.3 38.1 3.3 · 1018 Transformer (big) 28.4 41.8 2.3 · 1019 Residual Dropout We apply dropout [33] to the output of each sub-layer, before it is added to the sub-layer input and normalized.
中文: BLEU培训成本(FLOPs) 型号 EN-DE EN-FR EN-DE EN-FR ByteNet [18] 23.75 Deep-Att + Posunk [39] 39.2 1.0 → 1020 GNMT + RL [38] 24.6 39.92 2.3 → 1019 1.4 → 1020 ConvS2S [9] 25.16 40.46 9.6 → 1018 1.5 → 1020 MoE [32] 26.03.56 2.0 → 1019 → 1020 Dep-Att + Posunk Ensemble [39] 40.4.0 → 1020 GNMT + Rsemble [38] 26.30.41.16 1.8 1020 1.1 → 1021 ConvS2S 环 [9] 26.36 41.29 7.7 → 10191.2 1021 变形器(基本型) 27.3 38.1.3 → 1018 1018 变形器(大) 28. 我们在每个子层的输出上应用退出[33],然后添加到子层输入并实现正态.
<a id="S0127"></a> Source: p.8 S0127
Original: In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks.
中文: 此外,我们在编码器和解码器堆栈的嵌入和位置编码中,都应用了退出.
<a id="S0128"></a> Source: p.8 S0128
Original: For the base model, we use a rate of P = 0.1. drop Label Smoothing During training, we employed label smoothing of value ϵ = 0.1 [36].
中文: 对于基准模型,我们使用P=0.1的速率。 在训练期间,我们使用Q=0.1 [36]的平滑标签。
<a id="S0129"></a> Source: p.8 S0129
Original: This ls hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score. 6 Results 6.1 Machine Translation On the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big) in Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4.
中文: 这伤害了疑惑,因为模型学会了更不确定,但提高了准确度和BLEU分数. 6.1 机器翻译 在WMT 2014的英德翻译任务上,大变压器模型(表2中的变压器(大))比以往报告的最佳模型(包括综艺)多出2.0多BLEU,确立了28.4.
<a id="S0130"></a> Source: p.8 S0130
Original: The configuration of this model is listed in the bottom line of Table 3.
中文: 该模型的配置列于表3下行.
<a id="S0131"></a> Source: p.8 S0131
Original: Even our base model surpasses all previously published models and ensembles, at a fraction of the training cost of any of the competitive models.
中文: 即使是我们的基础模型也超越了所有以前公布的模型和综艺节目,其成本是任何竞争性模型培训费用的一小部分.
<a id="S0132"></a> Source: p.8 S0132
Original: On the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0, outperforming all of the previously published single models, at less than 1/4 the training cost of the previous state-of-the-art model.
中文: 在WMT 2014英语至法语翻译任务上,我们大模式的BLEU得分达到41.0分,比以前公布的所有单模式都高,低于之前最先进的模式的1/4培训费用.
<a id="S0133"></a> Source: p.8 S0133
Original: The Transformer (big) model trained for English-to-French used dropout rate P = 0.1, instead of 0.3. drop For the base models, we used a single model obtained by averaging the last 5 checkpoints, which were written at 10-minute intervals.
中文: 为英语对法语培训的变形器(大)模型使用辍学率P=0.1,而不是0.3. 对于基准模型,我们使用了通过平均最后5个检查站获得的单一模型,这些检查站每10分钟写一次。
<a id="S0134"></a> Source: p.8 S0134
Original: For the big models, we averaged the last 20 checkpoints.
中文: 对于大模型,我们平均过去20个检查站。
<a id="S0135"></a> Source: p.8 S0135
Original: We used beam search with a beam size of 4 and length penalty α = 0.6 [38].
中文: 我们用光束搜索,光束尺寸为4,长度罚分为α=0.6 [38].
<a id="S0136"></a> Source: p.8 S0136
Original: These hyperparameters were chosen after experimentation on the development set.
中文: 这些超参数是在试验开发装置后选定的。
<a id="S0137"></a> Source: p.8 S0137
Original: We set the maximum output length during inference to input length + 50, but terminate early when possible [38].
中文: 我们在推论中将最大输出长度设定为输入长度+50,但可能时提前终止[38].
<a id="S0138"></a> Source: p.8 S0138
Original: Table 2 summarizes our results and compares our translation quality and training costs to other model architectures from the literature.
中文: 表2总结了我们的成果,并将翻译质量和培训费用与文献中的其他示范架构进行比较。
<a id="S0139"></a> Source: p.8 S0139
Original: We estimate the number of floating point operations used to train a model by multiplying the training time, the number of GPUs used, and an estimate of the sustained single-precision floating-point capacity of each GPU 5. 6.2 Model Variations To evaluate the importance of different components of the Transformer, we varied our base model in different ways, measuring the change in performance on English-to-German translation on the 5We used values of 2.8, 3.7, 6.0 and 9.5 TFLOPS for K80, K40, M40 and P100, respectively. 8
中文: 我们通过乘以训练时间,使用的GPU数量,以及估计每个GPU 5.的持续单精度浮点容量,来估计用于训练一个模型的浮点操作次数. 6.2 模式变化 为了评估变形器不同组成部分的重要性,我们以不同的方式改变了我们的基础模型,在5We的英文至德文翻译上测量了性能的变化,在K80,K40,M40和P100上分别使用了2.8,3.7,6.0和9.5的TFLOPS值.
<a id="S0140"></a> Source: p.9 S0140
Original: Table 3: Variations on the Transformer architecture.
中文: 表3:变形器架构的变化.
<a id="S0141"></a> Source: p.9 S0141
Original: Unlisted values are identical to those of the base model.
中文: 未列出的数值与基准模型的数值相同。
<a id="S0142"></a> Source: p.9 S0142
Original: All metrics are on the English-to-German translation development set, newstest2013.
中文: 所有度量衡都在英德译名开发集"新闻测试2013"上.
<a id="S0143"></a> Source: p.9 S0143
Original: Listed perplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to per-word perplexities. train PPL BLEU params N d d h d d P ϵ model ff k v drop ls steps (dev) (dev) ×106 base 6 512 2048 8 64 64 0.1 0.1 100K 4.92 25.8 65 1 512 512 5.29 24.9 4 128 128 5.00 25.5 (A) 16 32 32 4.91 25.8 32 16 16 5.01 25.4 16 5.16 25.1 58 (B) 32 5.01 25.4 60 2 6.11 23.7 36 4 5.19 25.3 50 8 4.88 25.5 80 (C) 256 32 32 5.75 24.5 28 1024 128 128 4.66 26.0 168 1024 5.12 25.4 53 4096 4.75 26.2 90 0.0 5.77 24.6 0.2 4.95 25.5 (D) 0.0 4.67 25.3 0.2 5.47 25.7 (E) positional embedding instead of sinusoids 4.92 25.7 big 6 1024 4096 16 0.3 300K 4.33 26.4 213 development set, newstest2013.
中文: 根据我们的字节-pair编码,列出的迷惑是一字一字,不应与一字一字的迷惑相提并论. 列车PPL BLEU params N d d d d d → P 型号 ff k v dropp Is step (dev) (dev) ×106 Base 6 512 2048 8 64 64 64 0.1 100 K 4.92 25.8 65 1 512 529 24.9 4 128 5.00 25.5 (A) 16 32 32 4.91 25.8 32 16 5.01 25.4 16 5.16 25.1 58 (B) 32 5.01 25.4 60 6.11 23.7 36 4.5.19 25.3 50 888 25.5 80 (C) 256 32 5.5 24.5 28 1024 128 128 4.66 26.0 168 1024 5.12 25.4 53 4096 4.75 26.2 90 0.05.77 24.6 0.4.95.5 (D) 4.67 25.3 0.5.47 25.7 (E) 定位嵌入式而非鼻膜 4.92 25.7 大 6 1024 4096 16 4.3 300. 33 26.4 213 发展集,新闻测试2013.
<a id="S0144"></a> Source: p.9 S0144
Original: We used beam search as described in the previous section, but no checkpoint averaging.
中文: 我们使用了上一节所述的光束搜查,但没有平均检查点。
<a id="S0145"></a> Source: p.9 S0145
Original: In Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions, keeping the amount of computation constant, as described in Section 3.2.2.
中文: 在表3行(A)中,如第3.2.2节所述,我们改变注意头数和注意键和价值维度,使计算量保持不变。
<a id="S0146"></a> Source: p.9 S0146
Original: While single-head attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.
中文: 虽然单头的注意力比最好的设定要差0.9 BLEU,但质量也会随着头数过多而下降.
<a id="S0147"></a> Source: p.9 S0147
Original: In Table 3 rows (B), we observe that reducing the attention key size d hurts model quality.
中文: 在表3行(B)中,我们看到减少注意键大小d会损害模型质量。
<a id="S0148"></a> Source: p.9 S0148
Original: This k suggests that determining compatibility is not easy and that a more sophisticated compatibility function than dot product may be beneficial.
中文: 这个k表示,确定相容性并不容易,比"点"产品更精密的相容性功能可能是有益的.
<a id="S0149"></a> Source: p.9 S0149
Original: We further observe in rows (C) and (D) that, as expected, bigger models are better, and dropout is very helpful in avoiding over-fitting.
中文: 我们进一步在行(C)和(D)中观察到,如所预期,更大的模型更好,辍学非常有助于避免过度调整。
<a id="S0150"></a> Source: p.9 S0150
Original: In row (E) we replace our sinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical results to the base model. 6.3 English Constituency Parsing To evaluate if the Transformer can generalize to other tasks we performed experiments on English constituency parsing.
中文: 在行(E)中,我们用学习到的位置嵌入来代替我们的鼻音定位编码[9],并观察到与基模型几乎相同的结果. 6.3 英语选区解析 为了评估变形金刚 能否概括到其他任务上 我们在英语组别解析方面做了实验
<a id="S0151"></a> Source: p.9 S0151
Original: This task presents specific challenges: the output is subject to strong structural constraints and is significantly longer than the input.
中文: 这项任务提出了具体的挑战:产出受到强有力的结构制约,而且比投入要长得多。
<a id="S0152"></a> Source: p.9 S0152
Original: Furthermore, RNN sequence-to-sequence models have not been able to attain state-of-the-art results in small-data regimes [37].
中文: 此外,RNN序列至序列模型未能在小数据制度中取得最先进的结果[37].
<a id="S0153"></a> Source: p.9 S0153
Original: We trained a 4-layer transformer with d = 1024 on the Wall Street Journal (WSJ) portion of the model Penn Treebank [25], about 40K training sentences.
中文: 我们在"华尔街日报"(WSJ)的Penn Treebank [25]模型部分训练了一台带有d=1024的4层变压器,大约40K的训练句.
<a id="S0154"></a> Source: p.9 S0154
Original: We also trained it in a semi-supervised setting, using the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences [37].
中文: 我们还在半监督环境下对其进行培训,使用更大的高自信和约17M句子的BerkleyParser Corpora[37].
<a id="S0155"></a> Source: p.9 S0155
Original: We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens for the semi-supervised setting.
中文: 我们只使用WSJ设置的词汇为16K令牌,半监督设置的词汇为32K令牌.
<a id="S0156"></a> Source: p.9 S0156
Original: We performed only a small number of experiments to select the dropout, both attention and residual (section 5.4), learning rates and beam size on the Section 22 development set, all other parameters remained unchanged from the English-to-German base translation model.
中文: 我们仅进行了少量的实验,以选择辍学者,包括注意力和残余(第5.4节)、学习率和第22节开发集的梁体大小,所有其他参数与英德基础翻译模型相比保持不变。
<a id="S0157"></a> Source: p.10 S0157
Original: Table 4: The Transformer generalizes well to English constituency parsing (Results are on Section 23 of WSJ) Parser Training WSJ 23 F1 Vinyals & Kaiser el al. (2014) [37] WSJ only, discriminative 88.3 Petrov et al. (2006) [29] WSJ only, discriminative 90.4 Zhu et al. (2013) [40] WSJ only, discriminative 90.4 Dyer et al. (2016) [8] WSJ only, discriminative 91.7 Transformer (4 layers) WSJ only, discriminative 91.3 Zhu et al. (2013) [40] semi-supervised 91.3 Huang & Harper (2009) [14] semi-supervised 91.3 McClosky et al. (2006) [26] semi-supervised 92.1 Vinyals & Kaiser el al. (2014) [37] semi-supervised 92.1 Transformer (4 layers) semi-supervised 92.7 Luong et al. (2015) [23] multi-task 93.0 Dyer et al. (2016) [8] generative 93.3 increased the maximum output length to input length + 300.
中文: 表4:变形人对英国选区的剖析(见WSJ第23节) 变形人一般化(见WSJ第23节) 变形人训练(见WSJ第23节) 变形人训练(见WSJ第23节) 变形人训练(见WSJ第23节) 变形人训练(见WSJ第23节) 变形人训练(见WSJ第23节) 变相人训练(见WSJ第23节) 变相人训练(见WSJ第23节) 变相人训练(见WSJ第23节) 变相人训练(见WSJ第23节) 变相人训练(见WSJ第23节) 变相人训练(见于第23节) 变相人训练(见WSJ第23节) 变相人训练(见于2013年) 变相训练(见于2013年) 变相训练(见于2013年) [40) 变相训练(见于2013年) 变相训练(见于2013年)
<a id="S0158"></a> Source: p.10 S0158
Original: We used a beam size of 21 and α = 0.3 for both WSJ only and the semi-supervised setting.
中文: 我们使用的光束尺寸为21和α=0.3,仅用于WSJ和半监督设置。
<a id="S0159"></a> Source: p.10 S0159
Original: Our results in Table 4 show that despite the lack of task-specific tuning our model performs surprisingly well, yielding better results than all previously reported models with the exception of the Recurrent Neural Network Grammar [8].
中文: 我们见表4的结果显示,尽管没有针对具体任务的调整,但我们的模式表现得令人惊讶,除了经常神经网络语法外,结果比以前报告的所有模式都好。
<a id="S0160"></a> Source: p.10 S0160
Original: In contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the Berkeley- Parser [29] even when training only on the WSJ training set of 40K sentences. 7 Conclusion In this work, we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.
中文: 与RNN序列至序列模型[37]形成对比的是,"变形器"比"伯克利-派瑟"(Berkeley-Parser)[29]甚至只在WSJ训练集40K句子上进行训练. 7 结论 在这部作品中,我们提出了"变形器",第一个完全基于注意的序列转录模型,用多头自意来取代了编码器-解码器架构中最常用的反复层.
<a id="S0161"></a> Source: p.10 S0161
Original: For translation tasks, the Transformer can be trained significantly faster than architectures based on recurrent or convolutional layers.
中文: 对于翻译任务,"变形器"的训练速度可以大大快于基于反复或革命层的架构.
<a id="S0162"></a> Source: p.10 S0162
Original: On both WMT 2014 English-to-German and WMT 2014 English-to-French translation tasks, we achieve a new state of the art.
中文: 在WMT 2014英语对德语和WMT 2014英语对法语翻译任务上,我们实现了一个新的艺术状态.
<a id="S0163"></a> Source: p.10 S0163
Original: In the former task our best model outperforms even all previously reported ensembles.
中文: 在前一个任务中,我们的最佳模式甚至超过了以前报告的所有综艺节目。
<a id="S0164"></a> Source: p.10 S0164
Original: We are excited about the future of attention-based models and plan to apply them to other tasks.
中文: 我们对关注型模式的未来感到兴奋,并计划将这些模式应用于其他任务。
<a id="S0165"></a> Source: p.10 S0165
Original: We plan to extend the Transformer to problems involving input and output modalities other than text and to investigate local, restricted attention mechanisms to efficiently handle large inputs and outputs such as images, audio and video.
中文: 我们计划将变换器扩展至涉及文本以外的投入和产出方式的问题,并调查局部的有限关注机制,以高效处理图像、音频和视频等大量投入和产出。
<a id="S0166"></a> Source: p.10 S0166
Original: Making generation less sequential is another research goals of ours.
中文: 减少世代相传是我们的另一个研究目标。
<a id="S0167"></a> Source: p.10 S0167
Original: The code we used to train and evaluate our models is available at https://github.com/ tensorflow/tensor2tensor.
中文: 我们用来训练和评价我们的模型的代码见https://github.com/ latersorflow/tensor2tensor。
<a id="S0168"></a> Source: p.10 S0168
Original: Acknowledgements We are grateful to Nal Kalchbrenner and Stephan Gouws for their fruitful comments, corrections and inspiration.
中文: 鸣谢 我们感谢纳尔·卡尔克布伦纳和斯捷芬·古斯富有成果地发表了评论、作了改正并给予启发。
<a id="S0169"></a> Source: p.10 S0169
Original: References [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
中文: 参考文献 [1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
<a id="S0170"></a> Source: p.10 S0170
Original: Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
中文: 层层正常化. arXiv预印版arXiv:1607.06450,2016. [2]. 德兹米特里·巴赫达瑙(德語:Dzmitry Bahdanau),克京亨·乔(德語:Chyunghyun Cho)和约斯华·本乔(德語:Yoshua Bengio).
<a id="S0171"></a> Source: p.10 S0171
Original: Neural machine translation by jointly learning to align and translate.
中文: 神经机能翻译通过共同学习对齐和翻译.
<a id="S0172"></a> Source: p.10 S0172
Original: CoRR, abs/1409.0473, 2014. [3] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V.
中文: CoRR, abs/1409.0473, 2014. [3] (英语). 德尼·布里兹,安娜·戈地,明-唐·罗荣桓,和克克·克克.
<a id="S0173"></a> Source: p.10 S0173
Original: Massive exploration of neural machine translation architectures.
中文: 大规模探索神经机能翻译架构.
<a id="S0174"></a> Source: p.10 S0174
Original: CoRR, abs/1703.03906, 2017. [4] Jianpeng Cheng, Li Dong, and Mirella Lapata.
中文: CoRR, abs/1703.03906, 2017. [4] (英语). (原始内容存档于2013-10-12). Jianpeng Cheng, Li Dong, and Mirella Lapata.
<a id="S0175"></a> Source: p.10 S0175
Original: Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.06733, 2016. 10
中文: 机器读取的长期短期内存-网络. arXiv preprint arXiv:1601.06733, 2016. 10 (英语).
<a id="S0176"></a> Source: p.11 S0176
Original: [5] Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
中文: [5] 克庆贤 Cho, Bart van Merienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, 和 Yoshua Bengio. (原始内容存档于2018-12-12).
<a id="S0177"></a> Source: p.11 S0177
Original: Learning phrase representations using rnn encoder-decoder for statistical machine translation.
中文: 学习短语表达法使用rnn编码器-解码器进行统计机器翻译.
<a id="S0178"></a> Source: p.11 S0178
Original: CoRR, abs/1406.1078, 2014. [6] Francois Chollet.
中文: CoRR, abs/1406.1078, 2014. [6] (英语). 弗朗索瓦·肖莱特.
<a id="S0179"></a> Source: p.11 S0179
Original: Xception: Deep learning with depthwise separable convolutions. arXiv preprint arXiv:1610.02357, 2016. [7] Junyoung Chung, Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio.
中文: 受想行识:以深智相分分相取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取 arXiv preprint arXiv:1610.02357, 2016. [7] (英语). 钟俊英, 恰格拉尔·居尔采尔, 邱京贤,和约斯华·本乔.
<a id="S0180"></a> Source: p.11 S0180
Original: Empirical evaluation of gated recurrent neural networks on sequence modeling.
中文: 在序列建模方面对入门反复神经网络进行经验评价.
<a id="S0181"></a> Source: p.11 S0181
Original: CoRR, abs/1412.3555, 2014. [8] Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A.
中文: CoRR, abs/1412.3555, 2014. [8] (中文(简体) ). 克里斯·戴尔,阿迪古纳·昆科罗,米格尔·巴列斯特罗斯,和诺亚·A.
<a id="S0182"></a> Source: p.11 S0182
Original: In Proc. of NAACL, 2016. [9] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N.
中文: 在NAACL的Proc., 2016. [9] (英语). 乔纳斯·盖林(Jonas Gehring),迈克尔·奥利(Michael Auli),大卫·格兰吉耶(David Grangier),德尼·亚拉特斯(Denis Yarats),和亚恩·N.
<a id="S0183"></a> Source: p.11 S0183
Original: Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122v2, 2017. [10] Alex Graves.
中文: 进取的顺序 学习的顺序。 arXiv preprint arXiv:1705.03122v2, 2017. [10] 亚历克斯·格雷夫斯.
<a id="S0184"></a> Source: p.11 S0184
Original: Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013. [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
中文: 以反复出现的神经网络生成序列. arXiv preprint arXiv:1308.0850, 2013. [11] (中文(简体) ). Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
<a id="S0185"></a> Source: p.11 S0185
Original: Deep residual learning for image recognition.
中文: 深残留学习以进行图像识别.
<a id="S0186"></a> Source: p.11 S0186
Original: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. [12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, and Jürgen Schmidhuber.
中文: 在IEEE计算机视野和模式识别会议记录中,第770–778页,2016. [12] Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, 和 Jürgen Schmidhuber.
<a id="S0187"></a> Source: p.11 S0187
Original: Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001. [13] Sepp Hochreiter and Jürgen Schmidhuber.
中文: 经常性蚊帐的逐步流动:学习长期依赖性的困难,2001年。 [13] 塞普·霍克赖特和于尔根·施密德胡伯.
<a id="S0188"></a> Source: p.11 S0188
Original: Neural computation, 9(8):1735–1780, 1997. [14] Zhongqiang Huang and Mary Harper.
中文: 神经计算,9(8):1735–1780,1997 [14]. 黄中强与玛丽·哈珀.
<a id="S0189"></a> Source: p.11 S0189
Original: Self-training PCFG grammars with latent annotations across languages.
中文: 自学PCFG语法,有各种语言的潜在说明.
<a id="S0190"></a> Source: p.11 S0190
Original: In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 832–841.
中文: 2009年自然语言处理经验方法会议记录,第832-841页。
<a id="S0191"></a> Source: p.11 S0191
Original: ACL, August 2009. [15] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu.
中文: ACL,2009年8月. [15] (中文(简体) ). 拉法尔·乔泽福维奇,奥里奥尔·维尼亚尔斯,迈克·舒斯特,诺姆·沙泽尔,和吴永辉.
<a id="S0192"></a> Source: p.11 S0192
Original: Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. [16] Łukasz Kaiser and Samy Bengio.
中文: 探索语言建模的极限. arXiv preprint arXiv:1602.02410, 2016. [16] (英语). Quakasz Kaiser and Samy Bengio.
<a id="S0193"></a> Source: p.11 S0193
Original: In Advances in Neural Information Processing Systems, (NIPS), 2016. [17] Łukasz Kaiser and Ilya Sutskever.
中文: In Advances in Neural Information Processing Systems, (NIPS), 2016. [17] (英语). Quakasz Kaiser and Ilya Sutskever.
<a id="S0194"></a> Source: p.11 S0194
Original: In International Conference on Learning Representations (ICLR), 2016. [18] Nal Kalchbrenner, Lasse Espeholt, Karen Simonyan, Aaron van den Oord, Alex Graves, and Koray Kavukcuoglu.
中文: 2016年国际学习代表会议。 [18] 纳尔·卡尔克布伦纳(英语:Nal Kalchbrenner),拉塞·埃斯佩霍尔特(英语:Lasse Espeholt),凯伦·西蒙扬(英语:Karen Simonyan),亚伦·范登·奥德(英语:Aaron van den Oord),亚历克斯·格雷夫斯(英语:Alex Graves)和克赖·卡武克库奥格卢(英语:Koray Kavukcuoglu).
<a id="S0195"></a> Source: p.11 S0195
Original: Neural machine translation in linear time. arXiv preprint arXiv:1610.10099v2, 2017. [19] Yoon Kim, Carl Denton, Luong Hoang, and Alexander M.
中文: 神经机能翻译在线性时间. arXiv预印 arXiv:1610.10099v2,2017. [19] (中文(简体) ). 尹·金,卡尔·登顿,罗荣桓,和亚历山大·M.
<a id="S0196"></a> Source: p.11 S0196
Original: In International Conference on Learning Representations, 2017. [20] Diederik Kingma and Jimmy Ba.
中文: (原始内容存档于2017-10-13). In International Conference on Learning Delections, 2017. [20] Diederik Kingma and Jimmy Ba.
<a id="S0197"></a> Source: p.11 S0197
Original: Adam: A method for stochastic optimization.
中文: 亚当:一种花样优化的方法.
<a id="S0198"></a> Source: p.11 S0198
Original: In ICLR, 2015. [21] Oleksii Kuchaiev and Boris Ginsburg.
中文: 在ICLR, 2015 [21] Oleksii Kuchaiev和鲍里斯·金斯堡.
<a id="S0199"></a> Source: p.11 S0199
Original: Factorization tricks for LSTM networks. arXiv preprint arXiv:1703.10722, 2017. [22] Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130, 2017. [23] Minh-Thang Luong, Quoc V.
中文: LSTM网络的因数化技巧. arXiv preprint arXiv:1703.10722, 2017. [22] 周汉林, 明伟峰, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, Yoshua Bengio. 结构化自动句子嵌入. arXiv preprint arXiv:1703.03130, 2017. [23] (中文(简体) ). 明通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通通.
<a id="S0200"></a> Source: p.11 S0200
Original: Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser.
中文: 勒,伊利亚·萨特斯克韦尔,奥里奥尔·维尼亚尔斯,和卢卡斯克·凯泽.
<a id="S0201"></a> Source: p.11 S0201
Original: Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114, 2015. [24] Minh-Thang Luong, Hieu Pham, and Christopher D Manning.
中文: 多任务序列来序列学习. arXiv preprint arXiv:15:11.06114, 2015. [24] 明-通·吕荣,平平,和克多克·德·曼宁.
<a id="S0202"></a> Source: p.11 S0202
Original: Effective approaches to attentionbased neural machine translation. arXiv preprint arXiv:1508.04025, 2015. 11
中文: 以注意力为基础的神经机理翻译的有效方法. arXiv preprint arXiv:1508.04025, 2015. 11 (英语).
<a id="S0203"></a> Source: p.12 S0203
Original: [25] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini.
中文: [25] 米切尔·P·马库斯,玛丽·安·马辛基耶维奇,和比阿特丽斯·桑托里尼.
<a id="S0204"></a> Source: p.12 S0204
Original: Building a large annotated corpus of english: The penn treebank.
中文: 打造一大批注解英文:"笔下树".
<a id="S0205"></a> Source: p.12 S0205
Original: Computational linguistics, 19(2):313–330, 1993. [26] David McClosky, Eugene Charniak, and Mark Johnson.
中文: 计算语言学, 19(2):313–330, 1993. [26]. 大卫·麦克洛斯基,尤金·查尼亚克,和马克·约翰逊.
<a id="S0206"></a> Source: p.12 S0206
Original: In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, pages 152–159.
中文: 《NAACL人文技术会议纪要》,主要会议,第152-159页。
<a id="S0207"></a> Source: p.12 S0207
Original: ACL, June 2006. [27] Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model.
中文: ACL,2006年6月. [27] Ankur Parikh, Oscar Täckström, Dipanjan Das,和雅各布·乌斯克赖特. 一个可分解的注意力模型.
<a id="S0208"></a> Source: p.12 S0208
Original: In Empirical Methods in Natural Language Processing, 2016. [28] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304, 2017. [29] Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein.
中文: 在"自然语言处理的经验方法"中,2016. [28] Romain Paulus, Caiming Xiong, and Richard Socher. 抽象归纳的深度强化模型. arXiv preprint arXiv:1705.04304,2017. [29] 斯拉夫·彼得罗夫,莱昂·巴雷特,罗曼·蒂鲍斯,和丹·克莱因.
<a id="S0209"></a> Source: p.12 S0209
Original: Learning accurate, compact, and interpretable tree annotation.
中文: 学习准确,紧凑,可解释的树注释.
<a id="S0210"></a> Source: p.12 S0210
Original: In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 433–440.
中文: 《第二十一届国际计算语言学会议和第四十四届国际法协会年会记录》,第433至440页。
<a id="S0211"></a> Source: p.12 S0211
Original: ACL, July 2006. [30] Ofir Press and Lior Wolf.
中文: ACL, 2006年7月. [30] Ofir Press and Lior Wolf. (中文(简体) ).
<a id="S0212"></a> Source: p.12 S0212
Original: Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859, 2016. [31] Rico Sennrich, Barry Haddow, and Alexandra Birch.
中文: 使用输出嵌入来改进语言模型. arXiv preprint arXiv:1608.05859, 2016. [31] (中文(简体) ). 瑞克·塞恩里奇,巴里·哈多和亚历山德拉·伯克.
<a id="S0213"></a> Source: p.12 S0213
Original: Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.
中文: 神经机能翻译有子词单位的稀有词. arXiv preprint arXiv:1508.07909, 2015. [32] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton,和杰夫·迪恩.
<a id="S0214"></a> Source: p.12 S0214
Original: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017. [33] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.
中文: 奇特的大型神经网络:稀有的混合专家层. arXiv preprint arXiv:1701.06538, 2017. [33]. 尼蒂什·斯里瓦斯塔瓦,杰弗里·埃·欣通,亚历克·克里齐赫夫斯基,伊利亚·苏茨克韦尔,和鲁斯兰·萨拉胡特季诺夫.
<a id="S0215"></a> Source: p.12 S0215
Original: Dropout: a simple way to prevent neural networks from overfitting.
中文: 退学:防止神经网络过度适应的简单方法.
<a id="S0216"></a> Source: p.12 S0216
Original: Journal of Machine Learning Research, 15(1):1929–1958, 2014. [34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus.
中文: 机器学习研究杂志, 15(1):1929–1958, 2014. [34] Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus.
<a id="S0217"></a> Source: p.12 S0217
Original: Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2440–2448.
中文: Garnett,编辑,神经信息处理系统的进步 28, 第2440–2448页.
<a id="S0218"></a> Source: p.12 S0218
Original: Curran Associates, Inc., 2015. [35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le.
中文: Curran Associates, Inc., 2015. [35] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le.
<a id="S0219"></a> Source: p.12 S0219
Original: Sequence to sequence learning with neural networks.
中文: 与神经网络同步学习的顺序.
<a id="S0220"></a> Source: p.12 S0220
Original: In Advances in Neural Information Processing Systems, pages 3104–3112, 2014. [36] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.
中文: 《神经信息处理系统预付款》,第3104-3112页,2014年。 [36] 克里斯蒂安·斯盖迪,文森特·范霍克,谢尔盖·约夫,约纳顿·斯伦斯,和兹比格涅夫·沃克纳.
<a id="S0221"></a> Source: p.12 S0221
Original: Rethinking the inception architecture for computer vision.
中文: 重新思考计算机视觉的初始架构.
<a id="S0222"></a> Source: p.12 S0222
Original: CoRR, abs/1512.00567, 2015. [37] Vinyals & Kaiser, Koo, Petrov, Sutskever, and Hinton.
中文: CoRR, abs/1512.00567, 2015 [37] (英语). Vinyals & Kaiser, Koo, Petrov, Sutskever, 和欣顿.
<a id="S0223"></a> Source: p.12 S0223
Original: In Advances in Neural Information Processing Systems, 2015. [38] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al.
中文: (原始内容存档于2015-10-25). In Advances in Neural Information Systems, 2015. [38]. 吴永辉; 迈克·舒斯特; 陈志芬; 高克活; 克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活克活
<a id="S0224"></a> Source: p.12 S0224
Original: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. [39] Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu.
中文: 谷歌神经机翻译系统:缩小人与机翻译差距. arXiv preprint arXiv:1609.08144, 2016. [39] (中文(中国大陆) ). Jee Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu.
<a id="S0225"></a> Source: p.12 S0225
Original: Deep recurrent models with fast-forward connections for neural machine translation.
中文: 具有快速前向连接的神经机能翻译深层再生模型.
<a id="S0226"></a> Source: p.12 S0226
Original: CoRR, abs/1606.04199, 2016. [40] Muhua Zhu, Yue Zhang, Wenliang Chen, Min Zhang, and Jingbo Zhu.
中文: CoRR, abs/1606.04199, 2016. [40]. 慕花 Zhu, 岳翔, 陈文亮, 敏翔, 静波等.
<a id="S0227"></a> Source: p.12 S0227
Original: Fast and accurate shift-reduce constituent parsing.
中文: 快速而准确的切换-减少成分解析.
<a id="S0228"></a> Source: p.12 S0228
Original: In Proceedings of the 51st Annual Meeting of the ACL (Volume 1: Long Papers), pages 434–443.
中文: ACL 第51届年会记录(第一卷:长文),第434-443页。
<a id="S0229"></a> Source: p.13 S0229
Original: Input-Input Layer5 Attention Visualizations tI tI si si ni ni siht siht tirips tirips taht taht a a ytirojam ytirojam fo fo naciremA naciremA stnemnrevog stnemnrevog evah evah dessap dessap wen wen swal swal ecnis ecnis 9002 9002 gnikam gnikam eht eht noitartsiger noitartsiger ro ro gnitov gnitov ssecorp ssecorp erom erom tluciffid tluciffid . . >SOE< >SOE< >dap< >dap< >dap< >dap< >dap< >dap< >dap< >dap< >dap< >dap< >dap< >dap< Figure 3: An example of the attention mechanism following long-distance dependencies in the encoder self-attention in layer 5 of 6.
中文: 输入层5 注意可视化 tI si si ni ni sikht tirips taht a ytirojam ytirojam fo fo fo naciremA naciremA stnemnrevog stnemnrevog evah dessap wen swal swal ecnis ecnis 9002 gnikam eht nitartchiger noittiger ro gnitov gnitov ssecorp ero erom erom eromiffidt tuffid 6. > SOE < SOE < dap < dap < dap < dap > dap > dap > dap > dap < dap > dap < dap < dap < dap < dap < dap < dap < dap < dap < dap > > 图3 > 图中5-自留机制自留的5-自留机制实例
<a id="S0230"></a> Source: p.13 S0230
Original: Many of the attention heads attend to a distant dependency of the verb ‘making’, completing the phrase ‘making...more difficult’.
中文: 许多关注者关注动词“制作”的遥远依赖性,
<a id="S0231"></a> Source: p.13 S0231
Original: Attentions here shown only for the word ‘making’.
中文: 此处只提到“制作”一词。
<a id="S0232"></a> Source: p.13 S0232
Original: Different colors represent different heads.
中文: 不同的颜色代表不同的头.
<a id="S0233"></a> Source: p.14 S0233
Original: Input-Input Layer5 ehT ehT waL waL lliw lliw reven reven eb eb tcefrep tcefrep , , tub tub sti sti noitacilppa noitacilppa dluohs dluohs eb eb tsuj tsuj - siht siht si si tahw tahw ew ew era era gnissim gnissim , , ni ni ym ym noinipo noinipo . . >SOE< >SOE< >dap< >dap< Input-Input Layer5 ehT ehT waL waL lliw lliw reven reven eb eb tcefrep tcefrep , , tub tub sti sti noitacilppa noitacilppa dluohs dluohs eb eb tsuj tsuj - siht siht si si tahw tahw ew ew era era gnissim gnissim , , ni ni ym ym noinipo noinipo . . >SOE< >SOE< >dap< >dap< Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution.
中文: 输入层5 ehT ehT waL waL lliw ew ew 时代 gnissim gnissim, ni ym ym noinipo noinipo. > SOE < sOE < dap < dap > 输入层5 ehT waL walppa dluohs dluohs eb eb tsuj tsuj – sit siht waht thw waht waht-tsuj – tu-tim nim-tsuj → sim-tim-tsui-t-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-timi-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-t
<a id="S0234"></a> Source: p.14 S0234
Original: Bottom: Isolated attentions from just the word ‘its’ for attention heads 5 and 6.
中文: 底部:仅用 " its " 一词来表示注意力的孤立的注意力取而代之的是5和6。
<a id="S0235"></a> Source: p.14 S0235
Original: Note that the attentions are very sharp for this word. 14
中文: 注意注意,注意这个字很尖锐. 页:1
<a id="S0236"></a> Source: p.15 S0236
Original: Input-Input Layer5 ehT ehT waL waL lliw lliw reven reven eb eb tcefrep tcefrep , , tub tub sti sti noitacilppa noitacilppa dluohs dluohs eb eb tsuj tsuj - siht siht si si tahw tahw ew ew era era gnissim gnissim , , ni ni ym ym noinipo noinipo . . >SOE< >SOE< >dap< >dap< Input-Input Layer5 ehT ehT waL waL lliw lliw reven reven eb eb tcefrep tcefrep , , tub tub sti sti noitacilppa noitacilppa dluohs dluohs eb eb tsuj tsuj - siht siht si si tahw tahw ew ew era era gnissim gnissim , , ni ni ym ym noinipo noinipo . . >SOE< >SOE< >dap< >dap< Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the sentence.
中文: 输入层5 ehT ehT waL waL wal waiw waiw ew 时代 gnissim gnissim, ni ym ym ninipo noinipo. > SOE < sOE < dap < dap > 输入层5 ehT waL waL walppa dluohs dluohs eb eb seuj tsuj-tsuj-tsuj-tim nim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-tim-t
<a id="S0237"></a> Source: p.15 S0237
Original: We give two such examples above, from two different heads from the encoder self-attention at layer 5 of 6.
中文: 我们在上面举出两个这样的例子,分别来自编码器在6层第5层自我注意的两个不同的头.
<a id="S0238"></a> Source: p.15 S0238
Original: The heads clearly learned to perform different tasks. 15
中文: 头目们显然学会了执行不同的任务。 15个