Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA - 中英文对照
translated: 2026-07-16
title: "Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA" aliases: - "word2vec" - "arXiv:1301.3781" source: "https://arxiv.org/abs/1301.3781" arxiv: "1301.3781" created: 2026-07-16 type: paper-translation status: translated tags: - paper - ml - deep-learning - nlp
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA tmikolov@google.com kaichen@google.com Greg Corrado Jeffrey Dean Google Inc., Mountain View, CA Google Inc., Mountain View, CA gcorrado@google.com jeff@google.com Abstract We propose two novel model architectures for computing continuous vector representations of words from very large data sets.
中文: 对矢量空间Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA tmikolov@google.com kaichen@google.com Greg Corrado Jeffrey Dean Google Inc., Mountain View, CA gcorrado@google.com JEP@gogle.com 的文字表达方式的有效估计 摘要 我们提出两个新颖的模型架构,用于计算来自非常大数据集的词的连续矢量表示.
<a id="S0002"></a> Source: p.1 S0002
Original: The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks.
中文: 这些表现的质量用一个词来测量相似性任务,结果与以前基于不同类型神经网络的最佳表现技术相比较.
<a id="S0003"></a> Source: p.1 S0003
Original: We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set.
中文: 我们看到精度以更低的计算成本大幅提高,即从16亿字数据集中学习高品质的字向量需要不到一天的时间.
<a id="S0004"></a> Source: p.1 S0004
Original: Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. 1 Introduction Many current NLP systems and techniques treat words as atomic units - there is no notion of similarity between words, as these are represented as indices in a vocabulary.
中文: 此外,我们显示,这些矢量提供了我们测试集中最先进的性能,用于测量同分词和语义词的相似性。 1 导言 许多目前的NLP系统和技术把单词当作原子单位——没有单词之间的相似性概念,因为这些单词在词汇中作为指数来表示.
<a id="S0005"></a> Source: p.1 S0005
Original: This choice has several good reasons - simplicity, robustness and the observation that simple models trained on huge amounts of data outperform complex systems trained on less data.
中文: 这种选择有几个很好的理由 -- -- 简单、稳健和观察,在大量数据方面训练的简单模型比在较少数据方面训练的复杂系统要好。
<a id="S0006"></a> Source: p.1 S0006
Original: An example is the popular N-gram model used for statistical language modeling - today, it is possible to train N-grams on virtually all available data (trillions of words [3]).
中文: 例如,统计语言模型的流行N-gram模型----今天,有可能在几乎所有现有数据上对N-gram进行培训(千字[3])。
<a id="S0007"></a> Source: p.1 S0007
Original: However, the simple techniques are at their limits in many tasks.
中文: 然而,在许多任务中,简单技术处于极限。
<a id="S0008"></a> Source: p.1 S0008
Original: For example, the amount of relevant in-domain data for automatic speech recognition is limited - the performance is usually dominated by the size of high quality transcribed speech data (often just millions of words).
中文: 例如,用于自动语音识别的相关域内数据数量有限——性能通常由高品质的转录语音数据(往往只有上百万个单词)的大小所主导.
<a id="S0009"></a> Source: p.1 S0009
Original: In machine translation, the existing corpora for many languages contain only a few billions of words or less.
中文: 在机器翻译中,许多语言现有的corpora只包含有几十亿个或更少的字.
<a id="S0010"></a> Source: p.1 S0010
Original: Thus, there are situations where simple scaling up of the basic techniques will not result in any significant progress, and we have to focus on more advanced techniques.
中文: 因此,在某些情况下,简单扩大基本技术不会取得任何重大的进展,我们必须注重更先进的技术。
<a id="S0011"></a> Source: p.1 S0011
Original: With progress of machine learning techniques in recent years, it has become possible to train more complex models on much larger data set, and they typically outperform the simple models.
中文: 随着近年来机器学习技术的进步,有可能在更大的数据集上训练出更复杂的模型,这些模型通常比简单的模型要好.
<a id="S0012"></a> Source: p.1 S0012
Original: Probably the most successful concept is to use distributed representations of words [10].
中文: 可能最成功的概念是使用分布式的文字表述[10].
<a id="S0013"></a> Source: p.1 S0013
Original: For example, neural network based language models significantly outperform N-gram models [1, 27, 17]. 1.1 Goals of the Paper The main goal of this paper is to introduce techniques that can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary.
中文: 例如,基于神经网络的语言模型显著地超过了N-gram模型[1,27,17]. 1.1 论文的目标 本文的主要目标是引入技术,用于从庞大的数据集中学习高品质的单词矢量,有数十亿个单词,词汇中有上百万个单词.
<a id="S0014"></a> Source: p.1 S0014
Original: As far as we know, none of the previously proposed architectures has been successfully trained on more 1 3102 peS 7 ]LC.sc[ 3v1873.1031:viXra
中文: 据我们所知,以前提议的建筑都没有成功培训超过1 3102 peS 7]LC.sc[3v18731.031:viXra
<a id="S0015"></a> Source: p.2 S0015
Original: than a few hundred of millions of words, with a modest dimensionality of the word vectors between 50 - 100.
中文: 超过数以亿计的字,字向量在50-100之间有适度的维度.
<a id="S0016"></a> Source: p.2 S0016
Original: We use recently proposed techniques for measuring the quality of the resulting vector representations, with the expectation that not only will similar words tend to be close to each other, but that words can have multiple degrees of similarity [20].
中文: 我们使用最近提出的技术来测量由此产生的向量表示的质量,期望不仅类似词会倾向于相近,而且词可以具有多等相近性[20].
<a id="S0017"></a> Source: p.2 S0017
Original: This has been observed earlier in the context of inflectional languages - for example, nouns can have multiple word endings, and if we search for similar words in a subspace of the original vector space, it is possible to find words that have similar endings [13, 14].
中文: 这一点在早期被观察到于非词性语言上——例如名词可以有多个词尾,而如果我们在原始矢量空间的子空间中搜索出相类似的词,就可以找到有相类似的词尾[13,14].
<a id="S0018"></a> Source: p.2 S0018
Original: Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities.
中文: 令人惊讶的是,人们发现,字面表述的相似性超出了简单的综合规律。
<a id="S0019"></a> Source: p.2 S0019
Original: Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector(”King”) - vector(”Man”) + vector(”Woman”) results in a vector that is closest to the vector representation of the word Queen [20].
中文: 使用在向量单词上进行简单代数操作的单词抵消技术,例如显示向量("King")-向量("Man")+向量("Woman")导致的向量最接近于向量表示单词Queen[20].
<a id="S0020"></a> Source: p.2 S0020
Original: In this paper, we try to maximize accuracy of these vector operations by developing new model architectures that preserve the linear regularities among words.
中文: 在本文中,我们试图通过开发新的模式架构来保持文字间的线性规律,来使这些矢量操作的精确度最大化.
<a id="S0021"></a> Source: p.2 S0021
Original: We design a new comprehensive test set for measuring both syntactic and semantic regularities1, and show that many such regularities can be learned with high accuracy.
中文: 我们设计了一套新的综合测试,既测量综合规律又测量语义规律1,并表明许多这样的规律可以高精度地学习.
<a id="S0022"></a> Source: p.2 S0022
Original: Moreover, we discuss how training time and accuracy depends on the dimensionality of the word vectors and on the amount of the training data. 1.2 Previous Work Representation of words as continuous vectors has a long history [10, 26, 8]. A very popular model architecture for estimating neural network language model (NNLM) was proposed in [1], where a feedforward neural network with a linear projection layer and a non-linear hidden layer was used to learn jointly the word vector representation and a statistical language model.
中文: 此外,我们讨论了培训时间和准确性如何取决于 " 矢量 " 一词的维度以及培训数据的数量。 1.2 以往作为连续矢量的单词的工作表现有很长的历史[10、26、8]。 [1] 提出了一个非常受欢迎的用于估计神经网络语言模型(NNLM)的模型架构,其中使用了具有线性投影层和非线性隐藏层的向导神经网络,共同学习矢量表示和统计语言模型.
<a id="S0023"></a> Source: p.2 S0023
Original: This work has been followed by many others.
中文: 这项工作之后还有许多其他工作。
<a id="S0024"></a> Source: p.2 S0024
Original: Another interesting architecture of NNLM was presented in [13, 14], where the word vectors are first learned using neural network with a single hidden layer.
中文: 另一个有趣的NNLM架构在[13,14]中被呈现出来,其中"矢量"一词最早是使用单层隐藏的神经网络来学习的.
<a id="S0025"></a> Source: p.2 S0025
Original: The word vectors are then used to train the NNLM.
中文: 然后用矢量来训练NNLM.
<a id="S0026"></a> Source: p.2 S0026
Original: Thus, the word vectors are learned even without constructing the full NNLM.
中文: 因此,即使没有构建完整的NNLM,也学到矢量一词.
<a id="S0027"></a> Source: p.2 S0027
Original: In this work, we directly extend this architecture, and focus just on the first step where the word vectors are learned using a simple model.
中文: 在这部作品中,我们直接地扩展了这个架构,并且只专注于用一个简单的模型来学习单词矢量的第一步.
<a id="S0028"></a> Source: p.2 S0028
Original: It was later shown that the word vectors can be used to significantly improve and simplify many NLP applications [4, 5, 29].
中文: 后来显示"矢量"一词可以被用来显著地改进和简化许多NLP应用[4,5,29].
<a id="S0029"></a> Source: p.2 S0029
Original: Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 29, 23, 19, 9], and some of the resulting word vectors were made available for future research and comparison2.
中文: 对"矢量"一词本身的估算是使用不同的模型架构进行的,并接受了各种corpora[4, 29, 23, 19, 9]的训练,所产生的部分"矢量"一词被提供给了未来的研究和比较2.
<a id="S0030"></a> Source: p.2 S0030
Original: However, as far as we know, these architectures were significantly more computationally expensive for training than the one proposed in [13], with the exception of certain version of log-bilinear model where diagonal weight matrices are used [23]. 2 Model Architectures Many different types of models were proposed for estimating continuous representations of words, including the well-known Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA).
中文: 然而,据我们所知,这些建筑在计算上比[13]中提议的高得多的培训费用,但使用对角重量矩阵的某些版本的对角双线模型除外[23]. 2 模型架构 提出了许多不同类型的模型来估计连续表达词,包括著名的Latetnt语义分析(LSA)和Latett Drichlet分配(LDA).
<a id="S0031"></a> Source: p.2 S0031
Original: In this paper, we focus on distributed representations of words learned by neural networks, as it was previously shown that they perform significantly better than LSA for preserving linear regularities among words [20, 31]; LDA moreover becomes computationally very expensive on large data sets.
中文: 在本文中,我们侧重于神经网络所学词语的分布式表达,因为以前已经表明,这些词语的性能明显优于LSA,以在单词中保持线性规律性[20,31];此外,LDA在大数据集上变得非常昂贵的计算.
<a id="S0032"></a> Source: p.2 S0032
Original: Similar to [18], to compare different model architectures we define first the computational complexity of a model as the number of parameters that need to be accessed to fully train the model.
中文: 与[18]类似,为了比较不同的模型架构,我们首先将模型的计算复杂性定义为需要访问以充分训练模型的参数数量.
<a id="S0033"></a> Source: p.2 S0033
Original: Next, we will try to maximize the accuracy, while minimizing the computational complexity. 1The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2http://ronan.collobert.com/senna/ http://metaoptimize.com/projects/wordreprs/ http://www.fit.vutbr.cz/˜imikolov/rnnlm/ http://ai.stanford.edu/˜ehhuang/ 2
中文: 其次,我们将尽量提高准确性,同时尽量减少计算的复杂性。 1 测试集见www.fit.vutbr.cz/ mikolov/rnnlm/word-test.v1.txt。 2 http://ronan.collobert.com/senna/ http://metaopimize.com/projects/wordreprs/ 页面存档备份,存于互联网档案馆. http://www.fit.vutbr.cz/ mikolov/rnnlm/ http://ai.stanford.edu/̃hhuang/ 2.
<a id="S0034"></a> Source: p.3 S0034
Original: For all the following models, the training complexity is proportional to O = E × T × Q, (1) where E is number of the training epochs, T is the number of the words in the training set and Q is defined further for each model architecture.
中文: 对于以下所有型号,训练复杂度与O=E×T×Q成正比,(1)在E为训练纪元数的情况下,T为训练集中的单词数,而Q则为每个示范架构进一步定义.
<a id="S0035"></a> Source: p.3 S0035
Original: Common choice is E = 3 − 50 and T up to one billion.
中文: 常见的选择是E=3-50,T最高可达10亿.
<a id="S0036"></a> Source: p.3 S0036
Original: All models are trained using stochastic gradient descent and backpropagation [26]. 2.1 Feedforward Neural Net Language Model (NNLM) The probabilistic feedforward neural network language model has been proposed in [1].
中文: 所有模型都采用分层梯度下垂和回向传播[26]来训练. 2.1 FeedForward神经网络语言模型(NNLM) [1]中提出了概率向导神经网络语言模型.
<a id="S0037"></a> Source: p.3 S0037
Original: It consists of input, projection, hidden and output layers.
中文: 它由输入,投影,隐藏和输出层组成.
<a id="S0038"></a> Source: p.3 S0038
Original: At the input layer, N previous words are encoded using 1-of-V coding, where V is size of the vocabulary.
中文: 在输入层上,前作的N字被用1-of-V编码来编码,其中V是词汇的大小.
<a id="S0039"></a> Source: p.3 S0039
Original: The input layer is then projected to a projection layer P that has dimensionality N × D, using a shared projection matrix.
中文: 然后将输入层投射到具有维度N×D的投影层P,使用共享投影矩阵.
<a id="S0040"></a> Source: p.3 S0040
Original: As only N inputs are active at any given time, composition of the projection layer is a relatively cheap operation.
中文: 由于在任何特定时间只有N输入是活性的,投影层的构成是一个相对廉价的操作.
<a id="S0041"></a> Source: p.3 S0041
Original: The NNLM architecture becomes complex for computation between the projection and the hidden layer, as values in the projection layer are dense.
中文: NNLM架构在投影和隐藏地层之间计算变得复杂了,因为投影地层中的值是密集的.
<a id="S0042"></a> Source: p.3 S0042
Original: For a common choice of N = 10, the size of the projection layer (P ) might be 500 to 2000, while the hidden layer size H is typically 500 to 1000 units.
中文: 对于N=10的共同选择,投影层(P)的大小可能是500到2000年,而隐藏层的大小H一般是500到1000个单位.
<a id="S0043"></a> Source: p.3 S0043
Original: Moreover, the hidden layer is used to compute probability distribution over all the words in the vocabulary, resulting in an output layer with dimensionality V .
中文: 此外,隐藏层用于计算词汇中所有单词的概率分布,从而形成一个带有维度V.的输出层.
<a id="S0044"></a> Source: p.3 S0044
Original: Thus, the computational complexity per each training example is Q = N × D + N × D × H + H × V, (2) where the dominating term is H × V .
中文: 因此,每个训练例的计算复杂性为Q = N × D + N → D → H + H → V, (2) 其中主要术语为 H × V.
<a id="S0045"></a> Source: p.3 S0045
Original: However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [25, 23, 18], or avoiding normalized models completely by using models that are not normalized during training [4, 9].
中文: 然而,为避免出现这种情况,提出了几个切实可行的解决办法;或者使用分级版本的软马克斯[25,23,18],或者通过使用在训练期间没有正常化的模型来完全避免正常化的模型[4,9].
<a id="S0046"></a> Source: p.3 S0046
Original: With binary tree representations of the vocabulary, the number of output units that need to be evaluated can go down to around log (V ).
中文: 由于词汇的二进制树表示,需要评价的输出单位数可以下至周围的log(V).
<a id="S0047"></a> Source: p.3 S0047
Original: Thus, most of the complexity is caused by the term N × D × H. 2 In our models, we use hierarchical softmax where the vocabulary is represented as a Huffman binary tree.
中文: 因此,复杂性大多由N×D×H 2. 在我们的模型中,我们使用分级软max来表示词汇为Huffman 二进制树.
<a id="S0048"></a> Source: p.3 S0048
Original: This follows previous observations that the frequency of words works well for obtaining classes in neural net language models [16].
中文: 此前的观察认为,单词的频率对于在神经网络语言模型中获得课程很有效[16].
<a id="S0049"></a> Source: p.3 S0049
Original: Huffman trees assign short binary codes to frequent words, and this further reduces the number of output units that need to be evaluated: while balanced binary tree would require log (V ) outputs to be evaluated, the Huffman tree based hierarchical softmax requires 2 only about log (U nigram perplexity(V )).
中文: Huffman树为频繁字指定了短的二进制代码,这进一步减少了需要评价的输出单位数量:虽然平衡的二进制树需要评价log(V)输出,而Huffman树的分级软max只需要2个关于log(U nigram perplexity(V)).
<a id="S0050"></a> Source: p.3 S0050
Original: For example when the vocabulary size is one million 2 words, this results in about two times speedup in evaluation.
中文: 例如,当词汇大小为一百万个2个单词时,这导致评价速度的大约2倍.
<a id="S0051"></a> Source: p.3 S0051
Original: While this is not crucial speedup for neural network LMs as the computational bottleneck is in the N ×D ×H term, we will later propose architectures that do not have hidden layers and thus depend heavily on the efficiency of the softmax normalization. 2.2 Recurrent Neural Net Language Model (RNNLM) Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model N ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural networks [15, 2].
中文: 虽然这对神经网络LMS来说并不是至关重要的加速,因为计算瓶颈在N×D×H术语中,但我们以后会提出没有隐藏地层的架构,从而严重依赖软max正常化的效率. 2.2 经常神经网络语言模型(RNNLM) 经常神经网络基于语言模型被提出来克服向上NNLM的某些局限性,例如需要指定上下文长度(模式N的顺序),也因为理论上RNN能够有效地代表比浅神经网络更复杂的模式[15,2].
<a id="S0052"></a> Source: p.3 S0052
Original: The RNN model does not have a projection layer; only input, hidden and output layer.
中文: RNN模型没有投影层;只有输入,隐藏和输出层.
<a id="S0053"></a> Source: p.3 S0053
Original: What is special for this type of model is the recurrent matrix that connects hidden layer to itself, using time-delayed connections.
中文: 这类模型的特殊之处在于,经常的矩阵将隐藏层连接到自己,使用被延迟的连接.
<a id="S0054"></a> Source: p.3 S0054
Original: This allows the recurrent model to form some kind of short term memory, as information from the past can be represented by the hidden layer state that gets updated based on the current input and the state of the hidden layer in the previous time step.
中文: 这使得经常模型可以形成某种短期内存,因为来自过去的信息可以被基于当前输入和上个时间步中隐藏层状态而得到更新的隐藏层状态所代表.
<a id="S0055"></a> Source: p.3 S0055
Original: The complexity per training example of the RNN model is Q = H × H + H × V, (3) where the word representations D have the same dimensionality as the hidden layer H.
中文: RNN模型的每个训练实例的复杂性为Q = H × H + H × V,(3)其中"表示"D"字与"隐藏地层"H有相同的维度.
<a id="S0056"></a> Source: p.3 S0056
Original: Again, the term H × V can be efficiently reduced to H × log (V ) by using hierarchical softmax.
中文: 同样地,通过使用分级软max,H×V一词可以被高效地简化为H×log(V).
<a id="S0057"></a> Source: p.3 S0057
Original: Most of the 2 complexity then comes from H × H. 3
中文: 那么,2个复杂性大多来自H×H. 3
<a id="S0058"></a> Source: p.4 S0058
Original: 2.3 Parallel Training of Neural Networks To train models on huge data sets, we have implemented several models on top of a large-scale distributed framework called DistBelief [6], including the feedforward NNLM and the new models proposed in this paper.
中文: 2.3 神经网络的平行培训 为了在庞大的数据集上训练模型,我们已经在名为DistBelief的大规模分布式框架上实施了几个模型,包括向上提供NNLM和本文中提议的新模型.
<a id="S0059"></a> Source: p.4 S0059
Original: The framework allows us to run multiple replicas of the same model in parallel, and each replica synchronizes its gradient updates through a centralized server that keeps all the parameters.
中文: 框架允许我们并行运行多个相同模型的复制品,每个复制品通过保存所有参数的集中服务器来同步其梯度更新.
<a id="S0060"></a> Source: p.4 S0060
Original: For this parallel training, we use mini-batch asynchronous gradient descent with an adaptive learning rate procedure called Adagrad [7].
中文: 对于这种平行训练,我们使用一个叫作"Adagrad"的适应性学习速率程序的小型批量同步梯度下降.
<a id="S0061"></a> Source: p.4 S0061
Original: Under this framework, it is common to use one hundred or more model replicas, each using many CPU cores at different machines in a data center. 3 New Log-linear Models In this section, we propose two new model architectures for learning distributed representations of words that try to minimize computational complexity.
中文: 在这个框架下,常见的是使用100个或更多的模型复制件,每个模型在数据中心的不同机器上使用许多CPU核心. 3 新的日志线性模型 在本节中,我们提出两个新的模式架构,用于学习文字分布式表达,试图将计算的复杂性降到最低.
<a id="S0062"></a> Source: p.4 S0062
Original: The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model.
中文: 从上一节主要观察到,复杂性多由模型中非线性隐藏层所造成.
<a id="S0063"></a> Source: p.4 S0063
Original: While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.
中文: 虽然这让神经网络变得如此有吸引力,但我们决定探索更简单的模型,这些模型可能无法像神经网络那样精确地代表数据,但有可能被训练得更高效得多的数据.
<a id="S0064"></a> Source: p.4 S0064
Original: The new architectures directly follow those proposed in our earlier work [13, 14], where it was found that neural network language model can be successfully trained in two steps: first, continuous word vectors are learned using simple model, and then the N-gram NNLM is trained on top of these distributed representations of words.
中文: 新的架构直接沿用了我们早期作品[13,14]中所提出的架构,其中发现神经网络语言模型可以在两个步骤中成功训练:第一,使用简单的模型来学习连续字向量,而后N-gram NNLM在这些分布式的文字表现上被训练.
<a id="S0065"></a> Source: p.4 S0065
Original: While there has been later substantial amount of work that focuses on learning word vectors, we consider the approach proposed in [13] to be the simplest one.
中文: 虽然后来有大量的工作侧重于学习单词矢量,但我们认为[13]中提议的方法是最简单的。
<a id="S0066"></a> Source: p.4 S0066
Original: Note that related models have been proposed also much earlier [26, 8]. 3.1 Continuous Bag-of-Words Model The first proposed architecture is similar to the feedforward NNLM, where the non-linear hidden layer is removed and the projection layer is shared for all words (not just the projection matrix); thus, all words get projected into the same position (their vectors are averaged).
中文: 请注意,也早在[26、8]之前就提出了相关的模型。 3.1 连续词包模型 第一个拟议架构类似于feedforward NNLM,其中去掉非线性隐藏层并共享所有单词的投影层(而不只是投影矩阵);因此,所有单词都会被投影到同一个位置(它们的矢量被平均).
<a id="S0067"></a> Source: p.4 S0067
Original: We call this architecture a bag-of-words model as the order of words in the history does not influence the projection.
中文: 我们称这个建筑为"一袋一字"模型,因为历史上的文字顺序并不影响投影.
<a id="S0068"></a> Source: p.4 S0068
Original: Furthermore, we also use words from the future; we have obtained the best performance on the task introduced in the next section by building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word.
中文: 此外,我们还使用来自未来的词语;我们通过在输入中建立一个包含4个未来和4个历史词的日志线性分类器,在下一节介绍的任务上取得了最佳业绩,培训标准是正确划分当前(中间)词。
<a id="S0069"></a> Source: p.4 S0069
Original: Training complexity is then Q = N × D + D × log (V ). (4) 2 We denote this model further as CBOW, as unlike standard bag-of-words model, it uses continuous distributed representation of the context.
中文: 然后训练复杂度为Q = N × D + D × log (V). (4) 2 我们进一步表示这个模型为CBOW,因为它与标准字袋模型不同,它使用上下文的连续分布表示.
<a id="S0070"></a> Source: p.4 S0070
Original: The model architecture is shown at Figure 1.
中文: 模型架构见图一.
<a id="S0071"></a> Source: p.4 S0071
Original: Note that the weight matrix between the input and the projection layer is shared for all word positions in the same way as in the NNLM. 3.2 Continuous Skip-gram Model The second architecture is similar to CBOW, but instead of predicting the current word based on the context, it tries to maximize classification of a word based on another word in the same sentence.
中文: 注意输入层和投影层之间的权重矩阵以与NNLM相同的方式为所有单词位置共享. 3.2 连续跳格克模型 第二个架构与CBOW相似,但与其根据上下文预测当前单词,不如尝试根据同句中另一个单词对单词进行最大限度的分类.
<a id="S0072"></a> Source: p.4 S0072
Original: More precisely, we use each current word as an input to a log-linear classifier with continuous projection layer, and predict words within a certain range before and after the current word.
中文: 更确切地说,我们使用每个当前单词作为输入一个具有连续投影层的对数线分级器,并预测当前单词前后一定范围内的单词.
<a id="S0073"></a> Source: p.4 S0073
Original: We found that increasing the range improves quality of the resulting word vectors, but it also increases the computational complexity.
中文: 我们发现,扩大范围提高了所生成的单词矢量的质量,但也增加了计算的复杂性.
<a id="S0074"></a> Source: p.4 S0074
Original: Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples.
中文: 由于较远的词通常与现在的词比相近的词关系更小,所以我们通过从我们的培训例子中从这些词中取出较少的样本来减少对较远的词的分量.
<a id="S0075"></a> Source: p.4 S0075
Original: The training complexity of this architecture is proportional to Q = C × (D + D × log (V )), (5) 2 where C is the maximum distance of the words.
中文: 此架构的训练复杂度与Q=C×(D+D×log (V))成正比,(5)2 C是词的最大相距.
<a id="S0076"></a> Source: p.4 S0076
Original: Thus, if we choose C = 5, for each training word we will select randomly a number R in range < 1; C >, and then use R words from history and 4
中文: 因此,如果我们为每个训练词选择 C = 5,我们将随机选择一个在 < 1; C > 范围内的数字R,然后使用历史中的 R 和 4
<a id="S0077"></a> Source: p.5 S0077
Original: INPUT PROJECTION OUTPUT INPUT PROJECTION OUTPUT w(t-2) w(t-2) w(t-1) w(t-1) SUM w(t) w(t) w(t+1) w(t+1) w(t+2) w(t+2) CBOW Skip-gram Figure 1: New model architectures.
中文: INPUT PROJECTION OUPUT INPUT PROJECTION OUT W(t-2) W(t-2) W(t-2) W(t-2) W(t-2) SUM (t-1) SUM (t+1) W(t+1) w(t+2) w(t+2) CBOW Skipp-gram 图1:新模型架构.
<a id="S0078"></a> Source: p.5 S0078
Original: The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word. R words from the future of the current word as correct labels.
中文: CBOW架构根据上下文对当前单词进行预测,Skip-gram根据当前单词对周围单词进行预测. R从当前单词的未来作为正确的标签.
<a id="S0079"></a> Source: p.5 S0079
Original: This will require us to do R × 2 word classifications, with the current word as input, and each of the R + R words as output.
中文: 这将要求我们进行R×2字分类,以当前单词作为输入,而每个R+R单词作为输出.
<a id="S0080"></a> Source: p.5 S0080
Original: In the following experiments, we use C = 10. 4 Results To compare the quality of different versions of word vectors, previous papers typically use a table showing example words and their most similar words, and understand them intuitively.
中文: 在以下实验中,我们使用C=10. 4 结果 为了比较不同版本的单词矢量的质量,前篇论文一般使用显示实例词及其最相近的词的表格,并直观地理解.
<a id="S0081"></a> Source: p.5 S0081
Original: Although it is easy to show that word France is similar to Italy and perhaps some other countries, it is much more challenging when subjecting those vectors in a more complex similarity task, as follows.
中文: 虽然法国一词很容易与意大利以及也许其他一些国家相提并论,但是,如果将这些载体置于一个更为复杂的类似任务中,那么它就更具挑战性,具体如下。
<a id="S0082"></a> Source: p.5 S0082
Original: We follow previous observation that there can be many different types of similarities between words, for example, word big is similar to bigger in the same sense that small is similar to smaller.
中文: 我们沿用了之前的观察,认为词之间可能有多种不同种类的相似之处,例如,词大与大类似,同样意义上的小与小相似.
<a id="S0083"></a> Source: p.5 S0083
Original: Example of another type of relationship can be word pairs big - biggest and small - smallest [20].
中文: 另一种类型关系的例子可以是单词对大-大小-最小[20].
<a id="S0084"></a> Source: p.5 S0084
Original: We further denote two pairs of words with the same relationship as a question, as we can ask: ”What is the word that is similar to small in the same sense as biggest is similar to big?” Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words.
中文: 我们进一步表示与一个问题有着相同关系的两对字,我们可以问道:"在与最大同义的意义上,什么字与小字相似? 有点令人惊讶的是,这些问题可以通过简单的代数操作来解答,并带有单词的矢量表示.
<a id="S0085"></a> Source: p.5 S0085
Original: To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector X = vector(”biggest”) − vector(”big”) + vector(”small”).
中文: 要找到一个与最大同义的小字相类似的词,我们可以简单地计算出矢量X=矢量("大")-矢量("大")+矢量("小").
<a id="S0086"></a> Source: p.5 S0086
Original: Then, we search in the vector space for the word closest to X measured by cosine distance, and use it as the answer to the question (we discard the input question words during this search).
中文: 然后,我们在矢量空间中搜索最接近以余弦相距测量的X的单词,并将其作为问题的答案(我们在此搜索中丢弃输入问题单词).
<a id="S0087"></a> Source: p.5 S0087
Original: When the word vectors are well trained, it is possible to find the correct answer (word smallest) using this method.
中文: 当"矢量"一词训练有素时,可以使用这种方法找到正确的答案(单词最小).
<a id="S0088"></a> Source: p.5 S0088
Original: Finally, we found that when we train high dimensional word vectors on a large amount of data, the resulting vectors can be used to answer very subtle semantic relationships between words, such as a city and the country it belongs to, e.g.
中文: 最后,我们发现,当我们在大量数据上训练出高维度的单词矢量时,所产生的矢量可以用来回答词之间的非常微妙的语义关系,比如一个城市和它所属的国家.
<a id="S0089"></a> Source: p.5 S0089
Original: France is to Paris as Germany is to Berlin.
中文: 法国是巴黎,德国是柏林。
<a id="S0090"></a> Source: p.5 S0090
Original: Word vectors with such semantic relationships could be used to improve many existing NLP applications, such as machine translation, information retrieval and question answering systems, and may enable other future applications yet to be invented. 5
中文: 具有这种语义关系的单词矢量可用于改进许多现有的NLP应用,如机器翻译,信息检索和问答系统,并可能使其他未来应用尚未被发明. 页:1
<a id="S0091"></a> Source: p.6 S0091
Original: Table 1: Examples of five types of semantic and nine types of syntactic questions in the Semantic- Syntactic Word Relationship test set.
中文: 表1:语义-词义关系测试集中五种语义和九种语义问题的例子.
<a id="S0092"></a> Source: p.6 S0092
Original: Type of relationship Word Pair 1 Word Pair 2 Common capital city Athens Greece Oslo Norway All capital cities Astana Kazakhstan Harare Zimbabwe Currency Angola kwanza Iran rial City-in-state Chicago Illinois Stockton California Man-Woman brother sister grandson granddaughter Adjective to adverb apparent apparently rapid rapidly Opposite possibly impossibly ethical unethical Comparative great greater tough tougher Superlative easy easiest lucky luckiest Present Participle think thinking read reading Nationality adjective Switzerland Swiss Cambodia Cambodian Past tense walking walked swimming swam Plural nouns mouse mice dollar dollars Plural verbs work works speak speaks 4.1 Task Description To measure quality of the word vectors, we define a comprehensive test set that contains five types of semantic questions, and nine types of syntactic questions.
中文: 所有首都城市 阿斯塔纳 哈萨克 哈拉雷 津巴布韦 货币 安哥拉克旺扎 伊朗州 芝加哥 伊利诺伊州 斯托克通 加利福尼亚州 男-女 兄弟孙女 形容词 明显快速的对面 可能无法实现道德伦理 比较更强悍 更幸运的 为了衡量单词向量的质量,我们定义了一套包含五种语义问题和九种合成问题的综合测试集.
<a id="S0093"></a> Source: p.6 S0093
Original: Two examples from each category are shown in Table 1.
中文: 每个类别的两个例子见表1。
<a id="S0094"></a> Source: p.6 S0094
Original: Overall, there are 8869 semantic and 10675 syntactic questions.
中文: 总体而言,共有8869个语义问题和10675个合成问题.
<a id="S0095"></a> Source: p.6 S0095
Original: The questions in each category were created in two steps: first, a list of similar word pairs was created manually.
中文: 每个类别中的问题分两步创建:一是手动创建出一个类似单词对的列表.
<a id="S0096"></a> Source: p.6 S0096
Original: Then, a large list of questions is formed by connecting two word pairs.
中文: 然后,通过连接两个单词对来形成一个很大的问题列表.
<a id="S0097"></a> Source: p.6 S0097
Original: For example, we made a list of 68 large American cities and the states they belong to, and formed about 2.5K questions by picking two word pairs at random.
中文: 例如,我们列出了68个美国大城市和它们所属的州,并通过随机选择两个单词对来形成大约2.5K个问题.
<a id="S0098"></a> Source: p.6 S0098
Original: We have included in our test set only single token words, thus multi-word entities are not present (such as New York).
中文: 我们的测试中只收录了单一的标语,因此多字实体没有出现(例如纽约)。
<a id="S0099"></a> Source: p.6 S0099
Original: We evaluate the overall accuracy for all question types, and for each question type separately (semantic, syntactic).
中文: 我们分别评价所有问题类型和每个问题类型(语义、综合)的总体准确性。
<a id="S0100"></a> Source: p.6 S0100
Original: Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes.
中文: 只有在与使用上述方法计算出的向量最接近的词与问题中的正确词完全相同的情况下,才假定回答正确;因此同义词被算作错误.
<a id="S0101"></a> Source: p.6 S0101
Original: This also means that reaching 100% accuracy is likely to be impossible, as the current models do not have any input information about word morphology.
中文: 这也意味着达到100%的准确度很可能是不可能的,因为目前的模型没有关于词形学的任何输入信息.
<a id="S0102"></a> Source: p.6 S0102
Original: However, we believe that usefulness of the word vectors for certain applications should be positively correlated with this accuracy metric.
中文: 然而,我们认为,矢量一词对某些应用的有用性应当与这一精确度指标有正比关系。
<a id="S0103"></a> Source: p.6 S0103
Original: Further progress can be achieved by incorporating information about structure of words, especially for the syntactic questions. 4.2 Maximization of Accuracy We have used a Google News corpus for training the word vectors.
中文: 通过纳入有关词语结构的信息,特别是综合问题的信息,可以取得进一步进展。 4.2 使准确性最大化 我们用Google新闻机 来训练矢量这个词
<a id="S0104"></a> Source: p.6 S0104
Original: We have restricted the vocabulary size to 1 million most frequent words.
中文: 我们把词汇的大小限制在100万个最常用的词上。
<a id="S0105"></a> Source: p.6 S0105
Original: Clearly, we are facing time constrained optimization problem, as it can be expected that both using more data and higher dimensional word vectors will improve the accuracy.
中文: 显然,我们正面临时间所限的优化问题,因为可以预期,使用更多的数据和更维度的字向量将提高准确性。
<a id="S0106"></a> Source: p.6 S0106
Original: To estimate the best choice of model architecture for obtaining as good as possible results quickly, we have first evaluated models trained on subsets of the training data, with vocabulary restricted to the most frequent 30k words.
中文: 为了估计对模型架构的最佳选择,以便尽快取得尽可能好的结果,我们首先对培训数据子集进行了培训,词汇限制在最常见的30克字上。
<a id="S0107"></a> Source: p.6 S0107
Original: The results using the CBOW architecture with different choice of word vector dimensionality and increasing amount of the training data are shown in Table 2.
中文: 使用CBOW架构得出的结果在字向量维度和不断增加的培训数据方面有不同的选择,如表2所示.
<a id="S0108"></a> Source: p.6 S0108
Original: It can be seen that after some point, adding more dimensions or adding more training data provides diminishing improvements.
中文: 可以看出,经过一段时间后,增加更多维度或增加更多培训数据可提供不断减少的改进。
<a id="S0109"></a> Source: p.6 S0109
Original: So, we have to increase both vector dimensionality and the amount of the training data together.
中文: 因此,我们必须同时增加矢量维度和培训数据的数量。
<a id="S0110"></a> Source: p.6 S0110
Original: While this observation might seem trivial, it must be noted that it is currently popular to train word vectors on relatively large amounts of data, but with insufficient size 6
中文: 虽然这种观察似乎微不足道,但必须指出的是,目前很流行的做法是在数量相对较多的数据上训练字向量,但尺寸不足。
<a id="S0111"></a> Source: p.7 S0111
Original: Table 2: Accuracy on subset of the Semantic-Syntactic Word Relationship test set, using word vectors from the CBOW architecture with limited vocabulary.
中文: 表2:语义-同义词关系测试集子集上的精确度,使用来自词汇有限的CBOW架构的词矢量.
<a id="S0112"></a> Source: p.7 S0112
Original: Only questions containing words from the most frequent 30k words are used.
中文: 只使用包含最常出自30k字的问题.
<a id="S0113"></a> Source: p.7 S0113
Original: Dimensionality / Training words 24M 49M 98M 196M 391M 783M 50 13.4 15.7 18.6 19.1 22.5 23.2 100 19.4 23.1 27.8 28.7 33.4 32.2 300 23.2 29.2 35.3 38.6 43.7 45.9 600 24.0 30.1 36.5 40.8 46.6 50.4 Table 3: Comparison of architectures using models trained on the same data, with 640-dimensional word vectors.
中文: 尺寸 / 训练词 24M 49M 98M 196M 391M 783M 50 13.4 15.7 18.6 19.1 22.5 23.2 100 19.4 23.1 27.8 28.7 33.4 32.2 300 23.2 29.2 35.3 38.6 43.7 45.9 24.0 30.1 36.5 40.8 46.6 50.4 表3:使用同一数据所训练的模型进行架构比较,有640维字向量.
<a id="S0114"></a> Source: p.7 S0114
Original: The accuracies are reported on our Semantic-Syntactic Word Relationship test set, and on the syntactic relationship test set of [20] Model Semantic-Syntactic Word Relationship test set MSR Word Relatedness Architecture Semantic Accuracy [%] Syntactic Accuracy [%] Test Set [20] RNNLM 9 36 35 NNLM 23 53 47 CBOW 24 64 61 Skip-gram 55 59 56 (such as 50 - 100).
中文: 我们的语义-同义词关系测试集,以及同义词关系测试集[20]模型语义-同义词关系测试集 MSR 词理相通性 建筑语理 [%] 词理相通性 [%] 测试集 [20] RNNLM 9 36 35 NNLM 23 53 47 CBOW 24 64 61 skipp-gram 55 59 56(如50-100).
<a id="S0115"></a> Source: p.7 S0115
Original: Given Equation 4, increasing amount of training data twice results in about the same increase of computational complexity as increasing vector size twice.
中文: 鉴于方程式4,培训数据的增加两次导致计算复杂性的提高与矢量大小的提高两次相同。
<a id="S0116"></a> Source: p.7 S0116
Original: For the experiments reported in Tables 2 and 4, we used three training epochs with stochastic gradient descent and backpropagation.
中文: 在表2和表4中报告的实验中,我们使用了三个具有划时代梯度下降和反扩散的训练时代。
<a id="S0117"></a> Source: p.7 S0117
Original: We chose starting learning rate 0.025 and decreased it linearly, so that it approaches zero at the end of the last training epoch. 4.3 Comparison of Model Architectures First we compare different model architectures for deriving the word vectors using the same training data and using the same dimensionality of 640 of the word vectors.
中文: 我们选择了开始学习 0.025 并线性地降低它, 这样它接近0 在最后的训练时代结束时。 4.3 模型架构的比较 首先,我们比较不同的模型架构,以便使用相同的训练数据,并使用640个矢量的等同维度得出单词矢量.
<a id="S0118"></a> Source: p.7 S0118
Original: In the further experiments, we use full set of questions in the new Semantic-Syntactic Word Relationship test set, i.e. unrestricted to the 30k vocabulary.
中文: 在进一步的实验中,我们在新的语义-同义词关系测试集中使用了完整的一组问题,即不受30k词汇限制.
<a id="S0119"></a> Source: p.7 S0119
Original: We also include results on a test set introduced in [20] that focuses on syntactic similarity between words3.
中文: 我们还包括了在[20]中引入的一套测试结果,该套测试侧重于词3之间的协同相似性.
<a id="S0120"></a> Source: p.7 S0120
Original: The training data consists of several LDC corpora and is described in detail in [18] (320M words, 82K vocabulary).
中文: 培训数据由几个最不发达国家公司组成,在18中作了详细说明。
<a id="S0121"></a> Source: p.7 S0121
Original: We used these data to provide a comparison to a previously trained recurrent neural network language model that took about 8 weeks to train on a single CPU.
中文: 我们利用这些数据来提供一种与之前训练的神经网络经常性语言模型的比较,这个模型花了大约8周的时间来训练一个CPU.
<a id="S0122"></a> Source: p.7 S0122
Original: We trained a feedforward NNLM with the same number of 640 hidden units using the DistBelief parallel training [6], using a history of 8 previous words (thus, the NNLM has more parameters than the RNNLM, as the projection layer has size 640 × 8).
中文: 我们使用 DistBelief 并行训练[6] 训练了同样数量为640个隐藏单元的向导NNLM,使用过去8个单词的历史(因此NNLM的参数比RNNLM多,因为投影层有640×8).
<a id="S0123"></a> Source: p.7 S0123
Original: In Table 3, it can be seen that the word vectors from the RNN (as used in [20]) perform well mostly on the syntactic questions.
中文: 在表3中,可以看出来自RNN的"矢量"(如20]所使用)一词大多在合成问题上表现良好.
<a id="S0124"></a> Source: p.7 S0124
Original: The NNLM vectors perform significantly better than the RNN - this is not surprising, as the word vectors in the RNNLM are directly connected to a non-linear hidden layer.
中文: NNLM向量的性能明显好于RNN——这并不奇怪,因为RNNLM中的"向量"一词直接连接到一个非线性隐藏地层.
<a id="S0125"></a> Source: p.7 S0125
Original: The CBOW architecture works better than the NNLM on the syntactic tasks, and about the same on the semantic one.
中文: CBOW架构在综合任务上比NNLM效果更好,在语义上则大致相同.
<a id="S0126"></a> Source: p.7 S0126
Original: Finally, the Skip-gram architecture works slightly worse on the syntactic task than the CBOW model (but still better than the NNLM), and much better on the semantic part of the test than all the other models.
中文: 最后,Skip-gram架构在综合任务上比CBOW模型(但依然比NNLM更好)稍差,在测试的语义部分上比所有其他模型要好得多.
<a id="S0127"></a> Source: p.7 S0127
Original: Next, we evaluated our models trained using one CPU only and compared the results against publicly available word vectors.
中文: 接下来,我们只用一个CPU来评估我们训练的模型,并将结果与公开的单词矢量进行比较。
<a id="S0128"></a> Source: p.7 S0128
Original: The CBOW model was trained on subset 3We thank Geoff Zweig for providing us the test set. 7
中文: CBOW模型在子集3上接受了训练 我们感谢Geoff Zweig为我们提供了测试集. 第7条
<a id="S0129"></a> Source: p.8 S0129
Original: Table 4: Comparison of publicly available word vectors on the Semantic-Syntactic Word Relationship test set, and word vectors from our models.
中文: 表4:语义-同义词关系测试集上公开的单词矢量的比较,以及我们模型中的单词矢量.
<a id="S0130"></a> Source: p.8 S0130
Original: Model Vector Training Accuracy [%] Dimensionality words Semantic Syntactic Total Collobert-Weston NNLM 50 660M 9.3 12.3 11.0 Turian NNLM 50 37M 1.4 2.6 2.1 Turian NNLM 200 37M 1.4 2.2 1.8 Mnih NNLM 50 37M 1.8 9.1 5.8 Mnih NNLM 100 37M 3.3 13.2 8.8 Mikolov RNNLM 80 320M 4.9 18.4 12.7 Mikolov RNNLM 640 320M 8.6 36.5 24.6 Huang NNLM 50 990M 13.3 11.6 12.3 Our NNLM 20 6B 12.9 26.4 20.3 Our NNLM 50 6B 27.9 55.8 43.2 Our NNLM 100 6B 34.2 64.5 50.8 CBOW 300 783M 15.5 53.1 36.1 Skip-gram 300 783M 50.0 55.9 53.3 Table 5: Comparison of models trained for three epochs on the same data and models trained for one epoch.
中文: 模式矢量培训准确性(%) 维度词: Semantic Syntic Total Collobert-Weston NNLM 50 660M 9.3 12.3 11.0 Turian NNLM 50 37M 1.4 2.6 2.1 Turian NNLM 200 1.8 Mnih NNLM 50 37M 9.1 3.3 13.2 8.8 Mikolov RNNLM 80 320M 18.4 12.7 Mikolov RNNLM 640 320M 8.6 24.6 Huang NNLM 50 990M 13.3 11.6 12.3 Our NNLM 20 6B 12.9 26.4 20.3 Our NNLM 50 6B 27.8 55.8 43.2 Our NNLM 100 6B 34.2 64.5 50.8 CBOW 300 783M 15.5 53.1 36.1 跳-gram 300 783M 50.0 55.9 53.3 表5:为三个时代培训的模型与为同一个时代培训的模型的比较。
<a id="S0131"></a> Source: p.8 S0131
Original: Accuracy is reported on the full Semantic-Syntactic data set.
中文: 准确性在完整的语义-综合数据集上报告.
<a id="S0132"></a> Source: p.8 S0132
Original: Model Vector Training Accuracy [%] Training time Dimensionality words [days] Semantic Syntactic Total 3 epoch CBOW 300 783M 15.5 53.1 36.1 1 3 epoch Skip-gram 300 783M 50.0 55.9 53.3 3 1 epoch CBOW 300 783M 13.8 49.9 33.6 0.3 1 epoch CBOW 300 1.6B 16.1 52.6 36.1 0.6 1 epoch CBOW 600 783M 15.4 53.3 36.2 0.7 1 epoch Skip-gram 300 783M 45.6 52.2 49.2 1 1 epoch Skip-gram 300 1.6B 52.2 55.1 53.8 2 1 epoch Skip-gram 600 783M 56.7 54.5 55.5 2.5 of the Google News data in about a day, while training time for the Skip-gram model was about three days.
中文: 模式矢量培训准确性(%) 训练时间 维度词[日] 语义学总和 3个字母 CBOW 300 783M 15.5 53.1 36.1 1 3个字母 CBOW 300 783M 500.0 55.9 53.3 3 1个字母 CBOW 300 783M 13.8 49.9 33.6 0.3 1个字母 CBOW 300 16.1 52.6 36.1 1个字母 CBOW 600 783M 15.4 53.3 36.2 0.7 1个字母 CBOW 300 783M 45.6 52.2 49.2 1个字母 CBOW 300 783M 52.2 B 52.2 55.1 53.8 1个字母 CBOW 600 783M 56.7 54.5 55.5 2.5 相隔一日左右的"Google News"数据,而"Skip-gram"模型的培训时间约为三天.
<a id="S0133"></a> Source: p.8 S0133
Original: For experiments reported further, we used just one training epoch (again, we decrease the learning rate linearly so that it approaches zero at the end of training).
中文: 对于进一步报道的实验,我们只使用了一个培训时代(再次,我们线性地降低学习率,使其在培训结束时接近零)。
<a id="S0134"></a> Source: p.8 S0134
Original: Training a model on twice as much data using one epoch gives comparable or better results than iterating over the same data for three epochs, as is shown in Table 5, and provides additional small speedup. 4.4 Large Scale Parallel Training of Models As mentioned earlier, we have implemented various models in a distributed framework called DistBelief.
中文: 如表5所示,利用一个时代对数据进行两倍于同一时代数据的模型培训,比三个时代的相同数据显示的可比较或更好的结果,并提供了额外的小速度。 4.4 模型的大规模平行培训 如前所述,我们在称为DistBelief的分布式框架内实施了各种模式。
<a id="S0135"></a> Source: p.8 S0135
Original: Below we report the results of several models trained on the Google News 6B data set, with mini-batch asynchronous gradient descent and the adaptive learning rate procedure called Adagrad [7].
中文: 下面我们报告在Google News 6B数据集上接受训练的几个模型的结果,小批量的同步梯度下降和称为Adagrad的适应性学习速率程序[7].
<a id="S0136"></a> Source: p.8 S0136
Original: We used 50 to 100 model replicas during the training.
中文: 我们在训练期间使用了50到100个模型复制品.
<a id="S0137"></a> Source: p.9 S0137
Original: Table 6: Comparison of models trained using the DistBelief distributed framework.
中文: 表6:使用DistBelief分布式框架培训的模型比较。
<a id="S0138"></a> Source: p.9 S0138
Original: Note that training of NNLM with 1000-dimensional vectors would take too long to complete.
中文: 请注意,使用1000维向量对NNLM进行培训需要太长的时间才能完成.
<a id="S0139"></a> Source: p.9 S0139
Original: Model Vector Training Accuracy [%] Training time Dimensionality words [days x CPU cores] Semantic Syntactic Total NNLM 100 6B 34.2 64.5 50.8 14 x 180 CBOW 1000 6B 57.3 68.9 63.7 2 x 140 Skip-gram 1000 6B 66.1 65.1 65.6 2.5 x 125 Table 7: Comparison and combination of models on the Microsoft Sentence Completion Challenge.
中文: 模式矢量培训准确性(%) 训练时间 维度词 [days x CPU cores] 语义共通NNLM 100 6B 34.2 64.5 50.8 14 x 180 CBOW 1000 6B 57.3 68.9 63.7 2 x 140 Skip-gram 1000 6B 66.1 65.1 65.6 2.5 x 125 表7: 关于微软判决完成挑战的模型比较和组合.
<a id="S0140"></a> Source: p.9 S0140
Original: Architecture Accuracy [%] 4-gram [32] 39 Average LSA similarity [32] 49 Log-bilinear model [24] 54.8 RNNLMs [19] 55.4 Skip-gram 48.0 Skip-gram + RNNLMs 58.9 estimate since the data center machines are shared with other production tasks, and the usage can fluctuate quite a bit.
中文: 结构精确度[%] 4-克 [32] 39 LSA平均相似度 [32] 49 Log-bilinear model [24] 54.8 RNNLMs [19] 55.4 Skip-gram 48.0 Skip-gram + RNNLMs 58.9 估计,因为数据中心的机器与其他生产任务共享,使用率可以有相当的波动.
<a id="S0141"></a> Source: p.9 S0141
Original: Note that due to the overhead of the distributed framework, the CPU usage of the CBOW model and the Skip-gram model are much closer to each other than their single-machine implementations.
中文: 请注意,由于分布式框架的间接费用,CPU对CBOW模型和Skip-gram模型的用法比起它们的单机执行更相近.
<a id="S0142"></a> Source: p.9 S0142
Original: The result are reported in Table 6. 4.5 Microsoft Research Sentence Completion Challenge The Microsoft Sentence Completion Challenge has been recently introduced as a task for advancing language modeling and other NLP techniques [32].
中文: 表6. 4.5 微软研究判决完成挑战 微软判决完成挑战最近作为推进语言模型和其他NLP技术的一项任务[32]。
<a id="S0143"></a> Source: p.9 S0143
Original: This task consists of 1040 sentences, where one word is missing in each sentence and the goal is to select word that is the most coherent with the rest of the sentence, given a list of five reasonable choices.
中文: 这项任务由1040个句子组成,每个句子中都缺少一个单词,目标是选择与句子其余部分最相通的单词,给出一个包含五个合理选择的列表.
<a id="S0144"></a> Source: p.9 S0144
Original: Performance of several techniques has been already reported on this set, including N-gram models, LSA-based model [32], log-bilinear model [24] and a combination of recurrent neural networks that currently holds the state of the art performance of 55.4% accuracy on this benchmark [19].
中文: 已经有关于该套技术的几种技术的性能的报告,包括N-克克模型,基于LSA的模型[32],对数比线模型[24],以及目前在这个基准上保持55.4%精度的常态神经网络组合[19].
<a id="S0145"></a> Source: p.9 S0145
Original: We have explored the performance of Skip-gram architecture on this task.
中文: 我们探索了Skip-gram架构在这项任务上的性能.
<a id="S0146"></a> Source: p.9 S0146
Original: First, we train the 640dimensional model on 50M words provided in [32].
中文: 首先,我们用[32]中提供的50M字来训练640维模型.
<a id="S0147"></a> Source: p.9 S0147
Original: Then, we compute score of each sentence in the test set by using the unknown word at the input, and predict all surrounding words in a sentence.
中文: 然后,我们用输入时的未知单词来计算每句的得分,并在一句中预测出所有周围的单词.
<a id="S0148"></a> Source: p.9 S0148
Original: The final sentence score is then the sum of these individual predictions.
中文: 最终的句子分数是这些个别预测的总和.
<a id="S0149"></a> Source: p.9 S0149
Original: Using the sentence scores, we choose the most likely sentence. A short summary of some previous results together with the new results is presented in Table 7.
中文: 使用句子分数,我们选择最可能的句子. 表7简要汇总了以前的一些成果以及新的成果。
<a id="S0150"></a> Source: p.9 S0150
Original: While the Skip-gram model itself does not perform on this task better than LSA similarity, the scores from this model are complementary to scores obtained with RNNLMs, and a weighted combination leads to a new state of the art result 58.9% accuracy (59.2% on the development part of the set and 58.7% on the test part of the set). 5 Examples of the Learned Relationships Table 8 shows words that follow various relationships.
中文: 虽然Skip-gram模型本身在这个任务上的表现并不比LSA相似性好,但从这个模型得到的分数与用RNNLMs获得的分数是互补的,而加权组合导致新状态的精度达到58.9%(在集的开发部分为59.2%,在集的测试部分为58.7%). 5 表8举例说明了各种关系之后的词。
<a id="S0151"></a> Source: p.9 S0151
Original: We follow the approach described above: the relationship is defined by subtracting two word vectors, and the result is added to another word.
中文: 我们遵循上述方法:通过去掉两个单词向量来定义关系,结果被添加到另一个单词中.
<a id="S0152"></a> Source: p.9 S0152
Original: Thus for example, Paris - France + Italy = Rome.
中文: 例如,巴黎-法国+意大利=罗马。
<a id="S0153"></a> Source: p.9 S0153
Original: As it can be seen, accuracy is quite good, although there is clearly a lot of room for further improvements (note that using our accuracy metric that 9
中文: 可以看出,准确性相当好,尽管显然有许多改进的余地(注意使用我们的准确度衡量标准,即9)
<a id="S0154"></a> Source: p.10 S0154
Original: Table 8: Examples of the word pair relationships, using the best word vectors from Table 4 (Skipgram model trained on 783M words with 300 dimensionality).
中文: 表8:单词对等关系的例子,使用表4中最好的单词矢量(Skipgram模型训练了783M字有300个维度).
<a id="S0155"></a> Source: p.10 S0155
Original: Relationship Example 1 Example 2 Example 3 France - Paris Italy: Rome Japan: Tokyo Florida: Tallahassee big - bigger small: larger cold: colder quick: quicker Miami - Florida Baltimore: Maryland Dallas: Texas Kona: Hawaii Einstein - scientist Messi: midfielder Mozart: violinist Picasso: painter Sarkozy - France Berlusconi: Italy Merkel: Germany Koizumi: Japan copper - Cu zinc: Zn gold: Au uranium: plutonium Berlusconi - Silvio Sarkozy: Nicolas Putin: Medvedev Obama: Barack Microsoft - Windows Google: Android IBM: Linux Apple: iPhone Microsoft - Ballmer Google: Yahoo IBM: McNealy Apple: Jobs Japan - sushi Germany: bratwurst France: tapas USA: pizza assumes exact match, the results in Table 8 would score only about 60%).
中文: 关系例 1 例 2 例 法国 - 巴黎 意大利:罗马 日本:东京 佛罗里达:塔拉哈斯西 大 -- -- 大 -- -- 小:更冷 -- -- 快:更快 迈阿密 - 佛罗里达 巴尔的摩:马里兰 达拉斯:德克萨克 科纳:夏威夷 爱因斯坦 - 科学家 梅西:中场选手 莫扎特:小提琴家 毕加索:画家 萨科齐 - 法国:贝卢斯科尼 默克尔:德国 小泉:日本铜 -- Cu锌:Zn金:Au铀:钚 Berlusconi - 西尔维奥·萨科尼 - 尼古拉斯·普京:梅德韦杰夫·奥巴马:巴拉克·微软 - Windows:Android IBM:Linux Apple:iPhone - Ballmer Google:Yahoo IBM:M:麦克尼利 苹果:日本 - Jobsions 日本 - Sushurst France: Tapas USA:比萨克 假设准确匹配,表8中的结果只有60%左右).
<a id="S0156"></a> Source: p.10 S0156
Original: We believe that word vectors trained on even larger data sets with larger dimensionality will perform significantly better, and will enable the development of new innovative applications.
中文: 我们认为,在更大范围的数据集上培训的字向量将大为改进,并能够开发新的创新应用。
<a id="S0157"></a> Source: p.10 S0157
Original: Another way to improve accuracy is to provide more than one example of the relationship.
中文: 提高准确性的另一个方法是提供不止一个关系的例子.
<a id="S0158"></a> Source: p.10 S0158
Original: By using ten examples instead of one to form the relationship vector (we average the individual vectors together), we have observed improvement of accuracy of our best models by about 10% absolutely on the semantic-syntactic test.
中文: 通过使用十个例子而不是一个例子来形成关系向量(我们平均单个向量在一起),我们看到我们最佳模型的精度在语义-合成测试中绝对提高了10%左右.
<a id="S0159"></a> Source: p.10 S0159
Original: It is also possible to apply the vector operations to solve different tasks.
中文: 也可以应用向量操作来解决不同的任务.
<a id="S0160"></a> Source: p.10 S0160
Original: For example, we have observed good accuracy for selecting out-of-the-list words, by computing average vector for a list of words, and finding the most distant word vector.
中文: 例如,我们通过计算一个单词列表的平均矢量,并找到最遥远的单词矢量,观察到选择列表外单词的准确性很好.
<a id="S0161"></a> Source: p.10 S0161
Original: This is a popular type of problems in certain human intelligence tests.
中文: 这是某些人类智能测试中流行的问题类型.
<a id="S0162"></a> Source: p.10 S0162
Original: Clearly, there is still a lot of discoveries to be made using these techniques. 6 Conclusion In this paper we studied the quality of vector representations of words derived by various models on a collection of syntactic and semantic language tasks.
中文: 显然,使用这些技术仍然有许多发现。 6 结论 在本文中,我们研究了各种模型所衍生出语言的矢量表示的质量,这些语言在合成和语义语言任务的集合上.
<a id="S0163"></a> Source: p.10 S0163
Original: We observed that it is possible to train high quality word vectors using very simple model architectures, compared to the popular neural network models (both feedforward and recurrent).
中文: 我们观察到,与流行的神经网络模型相比,使用非常简单的模型架构来训练高品质的单词矢量是有可能的(无论是向后还是反复).
<a id="S0164"></a> Source: p.10 S0164
Original: Because of the much lower computational complexity, it is possible to compute very accurate high dimensional word vectors from a much larger data set.
中文: 由于计算复杂度要低得多,所以可以从一个更大的数据集中计算出非常精确的高维字向量.
<a id="S0165"></a> Source: p.10 S0165
Original: Using the DistBelief distributed framework, it should be possible to train the CBOW and Skip-gram models even on corpora with one trillion words, for basically unlimited size of the vocabulary.
中文: 使用DistBelief分布式框架,甚至在Corpora上用一万亿字来训练CBOW和Skip-gram模型,对于基本无限大小的词汇来说也是可能的.
<a id="S0166"></a> Source: p.10 S0166
Original: That is several orders of magnitude larger than the best previously published results for similar models.
中文: 这比以前公布的类似模型的最佳结果还多。
<a id="S0167"></a> Source: p.10 S0167
Original: An interesting task where the word vectors have recently been shown to significantly outperform the previous state of the art is the SemEval-2012 Task 2 [11].
中文: 一个有趣的任务就是SemEval-2012任务2[11],其中最近显示的矢量一词明显地超过了之前的状态.
<a id="S0168"></a> Source: p.10 S0168
Original: The publicly available RNN vectors were used together with other techniques to achieve over 50% increase in Spearman’s rank correlation over the previous best result [31].
中文: 公开的RNN矢量与其他技术一起被使用,使斯皮尔曼的分级关系比以往的最佳结果提高50%以上[31]。
<a id="S0169"></a> Source: p.10 S0169
Original: The neural network based word vectors were previously applied to many other NLP tasks, for example sentiment analysis [12] and paraphrase detection [28].
中文: 神经网络基于单词矢量之前被应用于许多其他NLP任务,例如情绪分析[12]和参数检测[28].
<a id="S0170"></a> Source: p.10 S0170
Original: It can be expected that these applications can benefit from the model architectures described in this paper.
中文: 可以预期,这些应用能够从本文所描述的模型架构中获益.
<a id="S0171"></a> Source: p.10 S0171
Original: Our ongoing work shows that the word vectors can be successfully applied to automatic extension of facts in Knowledge Bases, and also for verification of correctness of existing facts.
中文: 我们正在进行的工作表明,“矢量”一词可以成功地应用于知识库中事实的自动延伸,也可用于核查现有事实的正确性。
<a id="S0172"></a> Source: p.10 S0172
Original: Results from machine translation experiments also look very promising.
中文: 机器翻译实验的结果也看起来很有希望。
<a id="S0173"></a> Source: p.10 S0173
Original: In the future, it would be also interesting to compare our techniques to Latent Relational Analysis [30] and others.
中文: 今后,将我们的技术与Latetnt关系分析[30]等作一比较,也会很有趣.
<a id="S0174"></a> Source: p.10 S0174
Original: We believe that our comprehensive test set will help the research community to improve the existing techniques for estimating the word vectors.
中文: 我们相信,我们的综合测试集将帮助研究界改进估计矢量一词的现有技术.
<a id="S0175"></a> Source: p.10 S0175
Original: We also expect that high quality word vectors will become an important building block for future NLP applications. 10
中文: 我们还期望高质量的单词矢量将成为未来NLP应用的重要基石. 10个
<a id="S0176"></a> Source: p.11 S0176
Original: 7 Follow-Up Work After the initial version of this paper was written, we published single-machine multi-threaded C++ code for computing the word vectors, using both the continuous bag-of-words and skip-gram architectures4.
中文: 7个后续工作 在本文的初始版本被写出后,我们发布了用于计算单词向量的单机多线程 C++ 代码,同时使用连续的"活字袋"和"跳格克"架构.
<a id="S0177"></a> Source: p.11 S0177
Original: The training speed is significantly higher than reported earlier in this paper, i.e. it is in the order of billions of words per hour for typical hyperparameter choices.
中文: 训练速度大大高于本文之前所报道的速度,即对于典型的超参数选择,训练速度约为每小时数十亿字.
<a id="S0178"></a> Source: p.11 S0178
Original: We also published more than 1.4 million vectors that represent named entities, trained on more than 100 billion words.
中文: 我们还出版了140多万个媒介,这些媒介代表了被命名的实体,经过了1 000多亿字的培训。
<a id="S0179"></a> Source: p.11 S0179
Original: Some of our follow-up work will be published in an upcoming NIPS 2013 paper [21].
中文: 我们的一些后续工作将发表在即将出版的NIPS 2013文件上[21].
<a id="S0180"></a> Source: p.11 S0180
Original: Vincent. A neural probabilistic language model.
中文: 文殊师利. 神经概率语言模型.
<a id="S0181"></a> Source: p.11 S0181
Original: Journal of Machine Learning Research, 3:1137-1155, 2003. [2] Y.
中文: 机器学习研究杂志, 2003: 3: 1137-1155, [2]. 没错
<a id="S0182"></a> Source: p.11 S0182
Original: In: Large-Scale Kernel Machines, MIT Press, 2007. [3] T.
中文: 载于:大型克内尔机,麻省理工学院出版社,2007年 [3]. T.
<a id="S0183"></a> Source: p.11 S0183
Original: Large language models in machine translation.
中文: 机器翻译中的大语言模型.
<a id="S0184"></a> Source: p.11 S0184
Original: In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007. [4] R.
中文: 2007年自然语言处理和计算语言学习经验方法联席会议纪要 [4]. R. (英语).
<a id="S0185"></a> Source: p.11 S0185
Original: Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning.
中文: 韦斯顿. 自然语言处理统一架构:有多任务学习的深层神经网络.
<a id="S0186"></a> Source: p.11 S0186
Original: In International Conference on Machine Learning, ICML, 2008. [5] R.
中文: 在国际机器学习会议上,ICML,2008 [5]. R. (英语).
<a id="S0187"></a> Source: p.11 S0187
Original: Natural Language Processing (Almost) from Scratch.
中文: 自然语言处理(几乎)出自"Scratch".
<a id="S0188"></a> Source: p.11 S0188
Original: Journal of Machine Learning Research, 12:2493- 2537, 2011. [6] J.
中文: 机器学习研究杂志,12:2493-2537, 2011. [6] J.
<a id="S0189"></a> Source: p.11 S0189
Original: Ng., Large Scale Distributed Deep Networks, NIPS, 2012. [7] J.C.
中文: (原始内容存档于2017-10-21). Ng., Class supplized Deep Networks, NIPS, 2012. [7] J.C.
<a id="S0190"></a> Source: p.11 S0190
Original: Adaptive subgradient methods for online learning and stochastic optimization.
中文: 适应性下级方法,用于在线学习和花样优化.
<a id="S0191"></a> Source: p.11 S0191
Original: Journal of Machine Learning Research, 2011. [8] J.
中文: 机器学习研究杂志, 2011. [8] J.
<a id="S0192"></a> Source: p.11 S0192
Original: Cognitive Science, 14, 179-211, 1990. [9] Eric H.
中文: 认知科学, 14, 179-211, 1990 [9]. (原始内容存档于2018-10-21). Eric H.
<a id="S0193"></a> Source: p.11 S0193
Original: Improving Word Representations via Global Context and Multiple Word Prototypes.
中文: 通过"全球背景"和"多字原型"来改进"文字表现".
<a id="S0194"></a> Source: p.11 S0194
Original: Association for Computational Linguistics, 2012. [10] G.E.
中文: 计算语言学协会, 2012. [10] G.E.
<a id="S0195"></a> Source: p.11 S0195
Original: In: Parallel distributed processing: Explorations in the microstructure of cognition.
中文: 载于:平行分布式处理:认知微观结构中的探索.
<a id="S0196"></a> Source: p.11 S0196
Original: Volume 1: Foundations, MIT Press, 1986. [11] D.A.
中文: 第1卷:基金会,麻省理工学院出版社,1986. [11] D.
<a id="S0197"></a> Source: p.11 S0197
Original: Semeval-2012 task 2: Measuring degrees of relational similarity.
中文: Semeval-2012任务2:衡量关系相似程度.
<a id="S0198"></a> Source: p.11 S0198
Original: In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012. [12] A.L.
中文: 载于:第6届语义评价国际研讨会纪要(SemEval 2012),2012. [12] A.L.
<a id="S0199"></a> Source: p.11 S0199
Original: Learning word vectors for sentiment analysis.
中文: 学习单词向量用于情绪分析.
<a id="S0200"></a> Source: p.11 S0200
Original: Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007. [14] T.
中文: 捷克语语音识别语言模型,硕士论文,布尔诺科技大学,2007 [14]. T.
<a id="S0201"></a> Source: p.11 S0201
Original: Neural network based language models for higly inflective languages, In: Proc.
中文: 以神经网络为基础的语言模型用于 higg inflective languages, In: Proc.
<a id="S0202"></a> Source: p.11 S0202
Original: Recurrent neural network based language model, In: Proceedings of Interspeech, 2010. [16] T.
中文: 经常神经网络基于语言模型,In: Course of Interspeech, 2010. [16]. T.
<a id="S0203"></a> Source: p.11 S0203
Original: Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011. [17] T.
中文: 经常性神经网络语言模型的扩展, In: Corps of ICASSP 2011 [17]. T.
<a id="S0204"></a> Source: p.11 S0204
Original: Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011. 4The code is available at https://code.google.com/p/word2vec/ 11
中文: 高级语言建模技术的经验评价和组合,In: Course of Interspeech, 2011. 4 编码见https://code.google.com/p/word2vec/11。
<a id="S0205"></a> Source: p.12 S0205
Original: Strategies for Training Large Scale Neural Network Language Models, In: Proc.
中文: 大型神经网络语言模型培训策略, in: Proc.
<a id="S0206"></a> Source: p.12 S0206
Original: Automatic Speech Recognition and Understanding, 2011. [19] T.
中文: 自动语音识别与理解, 2011. [19] T.
<a id="S0207"></a> Source: p.12 S0207
Original: Statistical Language Models based on Neural Networks.
中文: 基于神经网络的统计语言模型.
<a id="S0208"></a> Source: p.12 S0208
Original: PhD thesis, Brno University of Technology, 2012. [20] T.
中文: 2012年布尔诺科技大学博士论文, [20] T.
<a id="S0209"></a> Source: p.12 S0209
Original: Linguistic Regularities in Continuous Space Word Representations.
中文: 连续空间文字表述语言规律.
<a id="S0210"></a> Source: p.12 S0210
Original: Distributed Representations of Words and Phrases and their Compositionality.
中文: 分布式表达词和口语及其构成性.
<a id="S0211"></a> Source: p.12 S0211
Original: Three new graphical models for statistical language modelling.
中文: 统计语言建模的三个新的图形模型.
<a id="S0212"></a> Source: p.12 S0212
Original: Hinton. A Scalable Hierarchical Distributed Language Model.
中文: 兴通. 一个可缩放的分级语言分布模型.
<a id="S0213"></a> Source: p.12 S0213
Original: Advances in Neural Information Processing Systems 21, MIT Press, 2009. [24] A.
中文: 神经信息处理系统的进步,21,麻省理工学院出版社,2009. [24] A.
<a id="S0214"></a> Source: p.12 S0214
Original: Teh. A fast and simple algorithm for training neural probabilistic language models.
中文: 铁. 用于训练神经概率语言模型的快速而简单的算法.
<a id="S0215"></a> Source: p.12 S0215
Original: Hierarchical Probabilistic Neural Network Language Model.
中文: 分级概率神经网络语言模型.
<a id="S0216"></a> Source: p.12 S0216
Original: Learning internal representations by backpropagating errors.
中文: 通过背传错误来学习内部表现.
<a id="S0217"></a> Source: p.12 S0217
Original: Computer Speech and Language, vol. 21, 2007. [28] R.
中文: 计算机语音和语言, 第21卷, 2007. [28] R.
<a id="S0218"></a> Source: p.12 S0218
Original: Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection.
中文: 用于参数化检测的动态集合和不重覆递归自动编码器。
<a id="S0219"></a> Source: p.12 S0219
Original: Word Representations: A Simple and General Method for Semi-Supervised Learning.
中文: 单词表达:半监督学习的简单而通俗的方法.
<a id="S0220"></a> Source: p.12 S0220
Original: Association for Computational Linguistics, 2010. [30] P. D.
中文: 计算语言学协会, 2010. [30] P. D.
<a id="S0221"></a> Source: p.12 S0221
Original: Measuring Semantic Similarity by Latent Relational Analysis.
中文: 由Latetnt关系分析测量语义相似性.
<a id="S0222"></a> Source: p.12 S0222
Original: International Joint Conference on Artificial Intelligence, 2005. [31] A.
中文: 国际人工智能联席会议,2005. [31]. A类.
<a id="S0223"></a> Source: p.12 S0223
Original: Combining Heterogeneous Models for Measuring Relational Similarity.
中文: 结合异相模型来测量关系相似性.
<a id="S0224"></a> Source: p.12 S0224
Original: The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011. 12
中文: 微软研究判决完成挑战,微软研究技术报告 MSR-TR-2011-129, 2011. 12