Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA
--- title: "Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA" aliases: - "word2vec" - "arXiv:1301.3781" source: "https://arxiv.org/abs/1301.3781" arxiv: "1301.3781" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - nlp
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen Google Inc., Mountain View, CA Google Inc., Mountain View, CA tmikolov@google.com kaichen@google.com Greg Corrado Jeffrey Dean Google Inc., Mountain View, CA Google Inc., Mountain View, CA gcorrado@google.com jeff@google.com Abstract We propose two novel model architectures for computing continuous vector representations of words from very large data sets.
<a id="S0002"></a> Source: p.1 S0002
The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks.
<a id="S0003"></a> Source: p.1 S0003
We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set.
<a id="S0004"></a> Source: p.1 S0004
Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities. 1 Introduction Many current NLP systems and techniques treat words as atomic units - there is no notion of similarity between words, as these are represented as indices in a vocabulary.
<a id="S0005"></a> Source: p.1 S0005
This choice has several good reasons - simplicity, robustness and the observation that simple models trained on huge amounts of data outperform complex systems trained on less data.
<a id="S0006"></a> Source: p.1 S0006
An example is the popular N-gram model used for statistical language modeling - today, it is possible to train N-grams on virtually all available data (trillions of words [3]).
<a id="S0007"></a> Source: p.1 S0007
However, the simple techniques are at their limits in many tasks.
<a id="S0008"></a> Source: p.1 S0008
For example, the amount of relevant in-domain data for automatic speech recognition is limited - the performance is usually dominated by the size of high quality transcribed speech data (often just millions of words).
<a id="S0009"></a> Source: p.1 S0009
In machine translation, the existing corpora for many languages contain only a few billions of words or less.
<a id="S0010"></a> Source: p.1 S0010
Thus, there are situations where simple scaling up of the basic techniques will not result in any significant progress, and we have to focus on more advanced techniques.
<a id="S0011"></a> Source: p.1 S0011
With progress of machine learning techniques in recent years, it has become possible to train more complex models on much larger data set, and they typically outperform the simple models.
<a id="S0012"></a> Source: p.1 S0012
Probably the most successful concept is to use distributed representations of words [10].
<a id="S0013"></a> Source: p.1 S0013
For example, neural network based language models significantly outperform N-gram models [1, 27, 17]. 1.1 Goals of the Paper The main goal of this paper is to introduce techniques that can be used for learning high-quality word vectors from huge data sets with billions of words, and with millions of words in the vocabulary.
<a id="S0014"></a> Source: p.1 S0014
As far as we know, none of the previously proposed architectures has been successfully trained on more 1 3102 peS 7 ]LC.sc[ 3v1873.1031:viXra
Page 2
<a id="S0015"></a> Source: p.2 S0015
than a few hundred of millions of words, with a modest dimensionality of the word vectors between 50 - 100.
<a id="S0016"></a> Source: p.2 S0016
We use recently proposed techniques for measuring the quality of the resulting vector representations, with the expectation that not only will similar words tend to be close to each other, but that words can have multiple degrees of similarity [20].
<a id="S0017"></a> Source: p.2 S0017
This has been observed earlier in the context of inflectional languages - for example, nouns can have multiple word endings, and if we search for similar words in a subspace of the original vector space, it is possible to find words that have similar endings [13, 14].
<a id="S0018"></a> Source: p.2 S0018
Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities.
<a id="S0019"></a> Source: p.2 S0019
Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that vector(”King”) - vector(”Man”) + vector(”Woman”) results in a vector that is closest to the vector representation of the word Queen [20].
<a id="S0020"></a> Source: p.2 S0020
In this paper, we try to maximize accuracy of these vector operations by developing new model architectures that preserve the linear regularities among words.
<a id="S0021"></a> Source: p.2 S0021
We design a new comprehensive test set for measuring both syntactic and semantic regularities1, and show that many such regularities can be learned with high accuracy.
<a id="S0022"></a> Source: p.2 S0022
Moreover, we discuss how training time and accuracy depends on the dimensionality of the word vectors and on the amount of the training data. 1.2 Previous Work Representation of words as continuous vectors has a long history [10, 26, 8]. A very popular model architecture for estimating neural network language model (NNLM) was proposed in [1], where a feedforward neural network with a linear projection layer and a non-linear hidden layer was used to learn jointly the word vector representation and a statistical language model.
<a id="S0023"></a> Source: p.2 S0023
This work has been followed by many others.
<a id="S0024"></a> Source: p.2 S0024
Another interesting architecture of NNLM was presented in [13, 14], where the word vectors are first learned using neural network with a single hidden layer.
<a id="S0025"></a> Source: p.2 S0025
The word vectors are then used to train the NNLM.
<a id="S0026"></a> Source: p.2 S0026
Thus, the word vectors are learned even without constructing the full NNLM.
<a id="S0027"></a> Source: p.2 S0027
In this work, we directly extend this architecture, and focus just on the first step where the word vectors are learned using a simple model.
<a id="S0028"></a> Source: p.2 S0028
It was later shown that the word vectors can be used to significantly improve and simplify many NLP applications [4, 5, 29].
<a id="S0029"></a> Source: p.2 S0029
Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 29, 23, 19, 9], and some of the resulting word vectors were made available for future research and comparison2.
<a id="S0030"></a> Source: p.2 S0030
However, as far as we know, these architectures were significantly more computationally expensive for training than the one proposed in [13], with the exception of certain version of log-bilinear model where diagonal weight matrices are used [23]. 2 Model Architectures Many different types of models were proposed for estimating continuous representations of words, including the well-known Latent Semantic Analysis (LSA) and Latent Dirichlet Allocation (LDA).
<a id="S0031"></a> Source: p.2 S0031
In this paper, we focus on distributed representations of words learned by neural networks, as it was previously shown that they perform significantly better than LSA for preserving linear regularities among words [20, 31]; LDA moreover becomes computationally very expensive on large data sets.
<a id="S0032"></a> Source: p.2 S0032
Similar to [18], to compare different model architectures we define first the computational complexity of a model as the number of parameters that need to be accessed to fully train the model.
<a id="S0033"></a> Source: p.2 S0033
Next, we will try to maximize the accuracy, while minimizing the computational complexity. 1The test set is available at www.fit.vutbr.cz/˜imikolov/rnnlm/word-test.v1.txt 2http://ronan.collobert.com/senna/ http://metaoptimize.com/projects/wordreprs/ http://www.fit.vutbr.cz/˜imikolov/rnnlm/ http://ai.stanford.edu/˜ehhuang/ 2
Page 3
<a id="S0034"></a> Source: p.3 S0034
For all the following models, the training complexity is proportional to O = E × T × Q, (1) where E is number of the training epochs, T is the number of the words in the training set and Q is defined further for each model architecture.
<a id="S0035"></a> Source: p.3 S0035
Common choice is E = 3 − 50 and T up to one billion.
<a id="S0036"></a> Source: p.3 S0036
All models are trained using stochastic gradient descent and backpropagation [26]. 2.1 Feedforward Neural Net Language Model (NNLM) The probabilistic feedforward neural network language model has been proposed in [1].
<a id="S0037"></a> Source: p.3 S0037
It consists of input, projection, hidden and output layers.
<a id="S0038"></a> Source: p.3 S0038
At the input layer, N previous words are encoded using 1-of-V coding, where V is size of the vocabulary.
<a id="S0039"></a> Source: p.3 S0039
The input layer is then projected to a projection layer P that has dimensionality N × D, using a shared projection matrix.
<a id="S0040"></a> Source: p.3 S0040
As only N inputs are active at any given time, composition of the projection layer is a relatively cheap operation.
<a id="S0041"></a> Source: p.3 S0041
The NNLM architecture becomes complex for computation between the projection and the hidden layer, as values in the projection layer are dense.
<a id="S0042"></a> Source: p.3 S0042
For a common choice of N = 10, the size of the projection layer (P ) might be 500 to 2000, while the hidden layer size H is typically 500 to 1000 units.
<a id="S0043"></a> Source: p.3 S0043
Moreover, the hidden layer is used to compute probability distribution over all the words in the vocabulary, resulting in an output layer with dimensionality V .
<a id="S0044"></a> Source: p.3 S0044
Thus, the computational complexity per each training example is Q = N × D + N × D × H + H × V, (2) where the dominating term is H × V .
<a id="S0045"></a> Source: p.3 S0045
However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [25, 23, 18], or avoiding normalized models completely by using models that are not normalized during training [4, 9].
<a id="S0046"></a> Source: p.3 S0046
With binary tree representations of the vocabulary, the number of output units that need to be evaluated can go down to around log (V ).
<a id="S0047"></a> Source: p.3 S0047
Thus, most of the complexity is caused by the term N × D × H. 2 In our models, we use hierarchical softmax where the vocabulary is represented as a Huffman binary tree.
<a id="S0048"></a> Source: p.3 S0048
This follows previous observations that the frequency of words works well for obtaining classes in neural net language models [16].
<a id="S0049"></a> Source: p.3 S0049
Huffman trees assign short binary codes to frequent words, and this further reduces the number of output units that need to be evaluated: while balanced binary tree would require log (V ) outputs to be evaluated, the Huffman tree based hierarchical softmax requires 2 only about log (U nigram perplexity(V )).
<a id="S0050"></a> Source: p.3 S0050
For example when the vocabulary size is one million 2 words, this results in about two times speedup in evaluation.
<a id="S0051"></a> Source: p.3 S0051
While this is not crucial speedup for neural network LMs as the computational bottleneck is in the N ×D ×H term, we will later propose architectures that do not have hidden layers and thus depend heavily on the efficiency of the softmax normalization. 2.2 Recurrent Neural Net Language Model (RNNLM) Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model N ), and because theoretically RNNs can efficiently represent more complex patterns than the shallow neural networks [15, 2].
<a id="S0052"></a> Source: p.3 S0052
The RNN model does not have a projection layer; only input, hidden and output layer.
<a id="S0053"></a> Source: p.3 S0053
What is special for this type of model is the recurrent matrix that connects hidden layer to itself, using time-delayed connections.
<a id="S0054"></a> Source: p.3 S0054
This allows the recurrent model to form some kind of short term memory, as information from the past can be represented by the hidden layer state that gets updated based on the current input and the state of the hidden layer in the previous time step.
<a id="S0055"></a> Source: p.3 S0055
The complexity per training example of the RNN model is Q = H × H + H × V, (3) where the word representations D have the same dimensionality as the hidden layer H.
<a id="S0056"></a> Source: p.3 S0056
Again, the term H × V can be efficiently reduced to H × log (V ) by using hierarchical softmax.
<a id="S0057"></a> Source: p.3 S0057
Most of the 2 complexity then comes from H × H. 3
Page 4
<a id="S0058"></a> Source: p.4 S0058
2.3 Parallel Training of Neural Networks To train models on huge data sets, we have implemented several models on top of a large-scale distributed framework called DistBelief [6], including the feedforward NNLM and the new models proposed in this paper.
<a id="S0059"></a> Source: p.4 S0059
The framework allows us to run multiple replicas of the same model in parallel, and each replica synchronizes its gradient updates through a centralized server that keeps all the parameters.
<a id="S0060"></a> Source: p.4 S0060
For this parallel training, we use mini-batch asynchronous gradient descent with an adaptive learning rate procedure called Adagrad [7].
<a id="S0061"></a> Source: p.4 S0061
Under this framework, it is common to use one hundred or more model replicas, each using many CPU cores at different machines in a data center. 3 New Log-linear Models In this section, we propose two new model architectures for learning distributed representations of words that try to minimize computational complexity.
<a id="S0062"></a> Source: p.4 S0062
The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model.
<a id="S0063"></a> Source: p.4 S0063
While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.
<a id="S0064"></a> Source: p.4 S0064
The new architectures directly follow those proposed in our earlier work [13, 14], where it was found that neural network language model can be successfully trained in two steps: first, continuous word vectors are learned using simple model, and then the N-gram NNLM is trained on top of these distributed representations of words.
<a id="S0065"></a> Source: p.4 S0065
While there has been later substantial amount of work that focuses on learning word vectors, we consider the approach proposed in [13] to be the simplest one.
<a id="S0066"></a> Source: p.4 S0066
Note that related models have been proposed also much earlier [26, 8]. 3.1 Continuous Bag-of-Words Model The first proposed architecture is similar to the feedforward NNLM, where the non-linear hidden layer is removed and the projection layer is shared for all words (not just the projection matrix); thus, all words get projected into the same position (their vectors are averaged).
<a id="S0067"></a> Source: p.4 S0067
We call this architecture a bag-of-words model as the order of words in the history does not influence the projection.
<a id="S0068"></a> Source: p.4 S0068
Furthermore, we also use words from the future; we have obtained the best performance on the task introduced in the next section by building a log-linear classifier with four future and four history words at the input, where the training criterion is to correctly classify the current (middle) word.
<a id="S0069"></a> Source: p.4 S0069
Training complexity is then Q = N × D + D × log (V ). (4) 2 We denote this model further as CBOW, as unlike standard bag-of-words model, it uses continuous distributed representation of the context.
<a id="S0070"></a> Source: p.4 S0070
The model architecture is shown at Figure 1.
<a id="S0071"></a> Source: p.4 S0071
Note that the weight matrix between the input and the projection layer is shared for all word positions in the same way as in the NNLM. 3.2 Continuous Skip-gram Model The second architecture is similar to CBOW, but instead of predicting the current word based on the context, it tries to maximize classification of a word based on another word in the same sentence.
<a id="S0072"></a> Source: p.4 S0072
More precisely, we use each current word as an input to a log-linear classifier with continuous projection layer, and predict words within a certain range before and after the current word.
<a id="S0073"></a> Source: p.4 S0073
We found that increasing the range improves quality of the resulting word vectors, but it also increases the computational complexity.
<a id="S0074"></a> Source: p.4 S0074
Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples.
<a id="S0075"></a> Source: p.4 S0075
The training complexity of this architecture is proportional to Q = C × (D + D × log (V )), (5) 2 where C is the maximum distance of the words.
<a id="S0076"></a> Source: p.4 S0076
Thus, if we choose C = 5, for each training word we will select randomly a number R in range < 1; C >, and then use R words from history and 4
Page 5
<a id="S0077"></a> Source: p.5 S0077
INPUT PROJECTION OUTPUT INPUT PROJECTION OUTPUT w(t-2) w(t-2) w(t-1) w(t-1) SUM w(t) w(t) w(t+1) w(t+1) w(t+2) w(t+2) CBOW Skip-gram Figure 1: New model architectures.
<a id="S0078"></a> Source: p.5 S0078
The CBOW architecture predicts the current word based on the context, and the Skip-gram predicts surrounding words given the current word. R words from the future of the current word as correct labels.
<a id="S0079"></a> Source: p.5 S0079
This will require us to do R × 2 word classifications, with the current word as input, and each of the R + R words as output.
<a id="S0080"></a> Source: p.5 S0080
In the following experiments, we use C = 10. 4 Results To compare the quality of different versions of word vectors, previous papers typically use a table showing example words and their most similar words, and understand them intuitively.
<a id="S0081"></a> Source: p.5 S0081
Although it is easy to show that word France is similar to Italy and perhaps some other countries, it is much more challenging when subjecting those vectors in a more complex similarity task, as follows.
<a id="S0082"></a> Source: p.5 S0082
We follow previous observation that there can be many different types of similarities between words, for example, word big is similar to bigger in the same sense that small is similar to smaller.
<a id="S0083"></a> Source: p.5 S0083
Example of another type of relationship can be word pairs big - biggest and small - smallest [20].
<a id="S0084"></a> Source: p.5 S0084
We further denote two pairs of words with the same relationship as a question, as we can ask: ”What is the word that is similar to small in the same sense as biggest is similar to big?” Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words.
<a id="S0085"></a> Source: p.5 S0085
To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector X = vector(”biggest”) − vector(”big”) + vector(”small”).
<a id="S0086"></a> Source: p.5 S0086
Then, we search in the vector space for the word closest to X measured by cosine distance, and use it as the answer to the question (we discard the input question words during this search).
<a id="S0087"></a> Source: p.5 S0087
When the word vectors are well trained, it is possible to find the correct answer (word smallest) using this method.
<a id="S0088"></a> Source: p.5 S0088
Finally, we found that when we train high dimensional word vectors on a large amount of data, the resulting vectors can be used to answer very subtle semantic relationships between words, such as a city and the country it belongs to, e.g.
<a id="S0089"></a> Source: p.5 S0089
France is to Paris as Germany is to Berlin.
<a id="S0090"></a> Source: p.5 S0090
Word vectors with such semantic relationships could be used to improve many existing NLP applications, such as machine translation, information retrieval and question answering systems, and may enable other future applications yet to be invented. 5
Page 6
<a id="S0091"></a> Source: p.6 S0091
Table 1: Examples of five types of semantic and nine types of syntactic questions in the Semantic- Syntactic Word Relationship test set.
<a id="S0092"></a> Source: p.6 S0092
Type of relationship Word Pair 1 Word Pair 2 Common capital city Athens Greece Oslo Norway All capital cities Astana Kazakhstan Harare Zimbabwe Currency Angola kwanza Iran rial City-in-state Chicago Illinois Stockton California Man-Woman brother sister grandson granddaughter Adjective to adverb apparent apparently rapid rapidly Opposite possibly impossibly ethical unethical Comparative great greater tough tougher Superlative easy easiest lucky luckiest Present Participle think thinking read reading Nationality adjective Switzerland Swiss Cambodia Cambodian Past tense walking walked swimming swam Plural nouns mouse mice dollar dollars Plural verbs work works speak speaks 4.1 Task Description To measure quality of the word vectors, we define a comprehensive test set that contains five types of semantic questions, and nine types of syntactic questions.
<a id="S0093"></a> Source: p.6 S0093
Two examples from each category are shown in Table 1.
<a id="S0094"></a> Source: p.6 S0094
Overall, there are 8869 semantic and 10675 syntactic questions.
<a id="S0095"></a> Source: p.6 S0095
The questions in each category were created in two steps: first, a list of similar word pairs was created manually.
<a id="S0096"></a> Source: p.6 S0096
Then, a large list of questions is formed by connecting two word pairs.
<a id="S0097"></a> Source: p.6 S0097
For example, we made a list of 68 large American cities and the states they belong to, and formed about 2.5K questions by picking two word pairs at random.
<a id="S0098"></a> Source: p.6 S0098
We have included in our test set only single token words, thus multi-word entities are not present (such as New York).
<a id="S0099"></a> Source: p.6 S0099
We evaluate the overall accuracy for all question types, and for each question type separately (semantic, syntactic).
<a id="S0100"></a> Source: p.6 S0100
Question is assumed to be correctly answered only if the closest word to the vector computed using the above method is exactly the same as the correct word in the question; synonyms are thus counted as mistakes.
<a id="S0101"></a> Source: p.6 S0101
This also means that reaching 100% accuracy is likely to be impossible, as the current models do not have any input information about word morphology.
<a id="S0102"></a> Source: p.6 S0102
However, we believe that usefulness of the word vectors for certain applications should be positively correlated with this accuracy metric.
<a id="S0103"></a> Source: p.6 S0103
Further progress can be achieved by incorporating information about structure of words, especially for the syntactic questions. 4.2 Maximization of Accuracy We have used a Google News corpus for training the word vectors.
<a id="S0104"></a> Source: p.6 S0104
We have restricted the vocabulary size to 1 million most frequent words.
<a id="S0105"></a> Source: p.6 S0105
Clearly, we are facing time constrained optimization problem, as it can be expected that both using more data and higher dimensional word vectors will improve the accuracy.
<a id="S0106"></a> Source: p.6 S0106
To estimate the best choice of model architecture for obtaining as good as possible results quickly, we have first evaluated models trained on subsets of the training data, with vocabulary restricted to the most frequent 30k words.
<a id="S0107"></a> Source: p.6 S0107
The results using the CBOW architecture with different choice of word vector dimensionality and increasing amount of the training data are shown in Table 2.
<a id="S0108"></a> Source: p.6 S0108
It can be seen that after some point, adding more dimensions or adding more training data provides diminishing improvements.
<a id="S0109"></a> Source: p.6 S0109
So, we have to increase both vector dimensionality and the amount of the training data together.
<a id="S0110"></a> Source: p.6 S0110
While this observation might seem trivial, it must be noted that it is currently popular to train word vectors on relatively large amounts of data, but with insufficient size 6
Page 7
<a id="S0111"></a> Source: p.7 S0111
Table 2: Accuracy on subset of the Semantic-Syntactic Word Relationship test set, using word vectors from the CBOW architecture with limited vocabulary.
<a id="S0112"></a> Source: p.7 S0112
Only questions containing words from the most frequent 30k words are used.
<a id="S0113"></a> Source: p.7 S0113
Dimensionality / Training words 24M 49M 98M 196M 391M 783M 50 13.4 15.7 18.6 19.1 22.5 23.2 100 19.4 23.1 27.8 28.7 33.4 32.2 300 23.2 29.2 35.3 38.6 43.7 45.9 600 24.0 30.1 36.5 40.8 46.6 50.4 Table 3: Comparison of architectures using models trained on the same data, with 640-dimensional word vectors.
<a id="S0114"></a> Source: p.7 S0114
The accuracies are reported on our Semantic-Syntactic Word Relationship test set, and on the syntactic relationship test set of [20] Model Semantic-Syntactic Word Relationship test set MSR Word Relatedness Architecture Semantic Accuracy [%] Syntactic Accuracy [%] Test Set [20] RNNLM 9 36 35 NNLM 23 53 47 CBOW 24 64 61 Skip-gram 55 59 56 (such as 50 - 100).
<a id="S0115"></a> Source: p.7 S0115
Given Equation 4, increasing amount of training data twice results in about the same increase of computational complexity as increasing vector size twice.
<a id="S0116"></a> Source: p.7 S0116
For the experiments reported in Tables 2 and 4, we used three training epochs with stochastic gradient descent and backpropagation.
<a id="S0117"></a> Source: p.7 S0117
We chose starting learning rate 0.025 and decreased it linearly, so that it approaches zero at the end of the last training epoch. 4.3 Comparison of Model Architectures First we compare different model architectures for deriving the word vectors using the same training data and using the same dimensionality of 640 of the word vectors.
<a id="S0118"></a> Source: p.7 S0118
In the further experiments, we use full set of questions in the new Semantic-Syntactic Word Relationship test set, i.e. unrestricted to the 30k vocabulary.
<a id="S0119"></a> Source: p.7 S0119
We also include results on a test set introduced in [20] that focuses on syntactic similarity between words3.
<a id="S0120"></a> Source: p.7 S0120
The training data consists of several LDC corpora and is described in detail in [18] (320M words, 82K vocabulary).
<a id="S0121"></a> Source: p.7 S0121
We used these data to provide a comparison to a previously trained recurrent neural network language model that took about 8 weeks to train on a single CPU.
<a id="S0122"></a> Source: p.7 S0122
We trained a feedforward NNLM with the same number of 640 hidden units using the DistBelief parallel training [6], using a history of 8 previous words (thus, the NNLM has more parameters than the RNNLM, as the projection layer has size 640 × 8).
<a id="S0123"></a> Source: p.7 S0123
In Table 3, it can be seen that the word vectors from the RNN (as used in [20]) perform well mostly on the syntactic questions.
<a id="S0124"></a> Source: p.7 S0124
The NNLM vectors perform significantly better than the RNN - this is not surprising, as the word vectors in the RNNLM are directly connected to a non-linear hidden layer.
<a id="S0125"></a> Source: p.7 S0125
The CBOW architecture works better than the NNLM on the syntactic tasks, and about the same on the semantic one.
<a id="S0126"></a> Source: p.7 S0126
Finally, the Skip-gram architecture works slightly worse on the syntactic task than the CBOW model (but still better than the NNLM), and much better on the semantic part of the test than all the other models.
<a id="S0127"></a> Source: p.7 S0127
Next, we evaluated our models trained using one CPU only and compared the results against publicly available word vectors.
<a id="S0128"></a> Source: p.7 S0128
The CBOW model was trained on subset 3We thank Geoff Zweig for providing us the test set. 7
Page 8
<a id="S0129"></a> Source: p.8 S0129
Table 4: Comparison of publicly available word vectors on the Semantic-Syntactic Word Relationship test set, and word vectors from our models.
<a id="S0130"></a> Source: p.8 S0130
Model Vector Training Accuracy [%] Dimensionality words Semantic Syntactic Total Collobert-Weston NNLM 50 660M 9.3 12.3 11.0 Turian NNLM 50 37M 1.4 2.6 2.1 Turian NNLM 200 37M 1.4 2.2 1.8 Mnih NNLM 50 37M 1.8 9.1 5.8 Mnih NNLM 100 37M 3.3 13.2 8.8 Mikolov RNNLM 80 320M 4.9 18.4 12.7 Mikolov RNNLM 640 320M 8.6 36.5 24.6 Huang NNLM 50 990M 13.3 11.6 12.3 Our NNLM 20 6B 12.9 26.4 20.3 Our NNLM 50 6B 27.9 55.8 43.2 Our NNLM 100 6B 34.2 64.5 50.8 CBOW 300 783M 15.5 53.1 36.1 Skip-gram 300 783M 50.0 55.9 53.3 Table 5: Comparison of models trained for three epochs on the same data and models trained for one epoch.
<a id="S0131"></a> Source: p.8 S0131
Accuracy is reported on the full Semantic-Syntactic data set.
<a id="S0132"></a> Source: p.8 S0132
Model Vector Training Accuracy [%] Training time Dimensionality words [days] Semantic Syntactic Total 3 epoch CBOW 300 783M 15.5 53.1 36.1 1 3 epoch Skip-gram 300 783M 50.0 55.9 53.3 3 1 epoch CBOW 300 783M 13.8 49.9 33.6 0.3 1 epoch CBOW 300 1.6B 16.1 52.6 36.1 0.6 1 epoch CBOW 600 783M 15.4 53.3 36.2 0.7 1 epoch Skip-gram 300 783M 45.6 52.2 49.2 1 1 epoch Skip-gram 300 1.6B 52.2 55.1 53.8 2 1 epoch Skip-gram 600 783M 56.7 54.5 55.5 2.5 of the Google News data in about a day, while training time for the Skip-gram model was about three days.
<a id="S0133"></a> Source: p.8 S0133
For experiments reported further, we used just one training epoch (again, we decrease the learning rate linearly so that it approaches zero at the end of training).
<a id="S0134"></a> Source: p.8 S0134
Training a model on twice as much data using one epoch gives comparable or better results than iterating over the same data for three epochs, as is shown in Table 5, and provides additional small speedup. 4.4 Large Scale Parallel Training of Models As mentioned earlier, we have implemented various models in a distributed framework called DistBelief.
<a id="S0135"></a> Source: p.8 S0135
Below we report the results of several models trained on the Google News 6B data set, with mini-batch asynchronous gradient descent and the adaptive learning rate procedure called Adagrad [7].
<a id="S0136"></a> Source: p.8 S0136
We used 50 to 100 model replicas during the training.
Page 9
<a id="S0137"></a> Source: p.9 S0137
Table 6: Comparison of models trained using the DistBelief distributed framework.
<a id="S0138"></a> Source: p.9 S0138
Note that training of NNLM with 1000-dimensional vectors would take too long to complete.
<a id="S0139"></a> Source: p.9 S0139
Model Vector Training Accuracy [%] Training time Dimensionality words [days x CPU cores] Semantic Syntactic Total NNLM 100 6B 34.2 64.5 50.8 14 x 180 CBOW 1000 6B 57.3 68.9 63.7 2 x 140 Skip-gram 1000 6B 66.1 65.1 65.6 2.5 x 125 Table 7: Comparison and combination of models on the Microsoft Sentence Completion Challenge.
<a id="S0140"></a> Source: p.9 S0140
Architecture Accuracy [%] 4-gram [32] 39 Average LSA similarity [32] 49 Log-bilinear model [24] 54.8 RNNLMs [19] 55.4 Skip-gram 48.0 Skip-gram + RNNLMs 58.9 estimate since the data center machines are shared with other production tasks, and the usage can fluctuate quite a bit.
<a id="S0141"></a> Source: p.9 S0141
Note that due to the overhead of the distributed framework, the CPU usage of the CBOW model and the Skip-gram model are much closer to each other than their single-machine implementations.
<a id="S0142"></a> Source: p.9 S0142
The result are reported in Table 6. 4.5 Microsoft Research Sentence Completion Challenge The Microsoft Sentence Completion Challenge has been recently introduced as a task for advancing language modeling and other NLP techniques [32].
<a id="S0143"></a> Source: p.9 S0143
This task consists of 1040 sentences, where one word is missing in each sentence and the goal is to select word that is the most coherent with the rest of the sentence, given a list of five reasonable choices.
<a id="S0144"></a> Source: p.9 S0144
Performance of several techniques has been already reported on this set, including N-gram models, LSA-based model [32], log-bilinear model [24] and a combination of recurrent neural networks that currently holds the state of the art performance of 55.4% accuracy on this benchmark [19].
<a id="S0145"></a> Source: p.9 S0145
We have explored the performance of Skip-gram architecture on this task.
<a id="S0146"></a> Source: p.9 S0146
First, we train the 640dimensional model on 50M words provided in [32].
<a id="S0147"></a> Source: p.9 S0147
Then, we compute score of each sentence in the test set by using the unknown word at the input, and predict all surrounding words in a sentence.
<a id="S0148"></a> Source: p.9 S0148
The final sentence score is then the sum of these individual predictions.
<a id="S0149"></a> Source: p.9 S0149
Using the sentence scores, we choose the most likely sentence. A short summary of some previous results together with the new results is presented in Table 7.
<a id="S0150"></a> Source: p.9 S0150
While the Skip-gram model itself does not perform on this task better than LSA similarity, the scores from this model are complementary to scores obtained with RNNLMs, and a weighted combination leads to a new state of the art result 58.9% accuracy (59.2% on the development part of the set and 58.7% on the test part of the set). 5 Examples of the Learned Relationships Table 8 shows words that follow various relationships.
<a id="S0151"></a> Source: p.9 S0151
We follow the approach described above: the relationship is defined by subtracting two word vectors, and the result is added to another word.
<a id="S0152"></a> Source: p.9 S0152
Thus for example, Paris - France + Italy = Rome.
<a id="S0153"></a> Source: p.9 S0153
As it can be seen, accuracy is quite good, although there is clearly a lot of room for further improvements (note that using our accuracy metric that 9
Page 10
<a id="S0154"></a> Source: p.10 S0154
Table 8: Examples of the word pair relationships, using the best word vectors from Table 4 (Skipgram model trained on 783M words with 300 dimensionality).
<a id="S0155"></a> Source: p.10 S0155
Relationship Example 1 Example 2 Example 3 France - Paris Italy: Rome Japan: Tokyo Florida: Tallahassee big - bigger small: larger cold: colder quick: quicker Miami - Florida Baltimore: Maryland Dallas: Texas Kona: Hawaii Einstein - scientist Messi: midfielder Mozart: violinist Picasso: painter Sarkozy - France Berlusconi: Italy Merkel: Germany Koizumi: Japan copper - Cu zinc: Zn gold: Au uranium: plutonium Berlusconi - Silvio Sarkozy: Nicolas Putin: Medvedev Obama: Barack Microsoft - Windows Google: Android IBM: Linux Apple: iPhone Microsoft - Ballmer Google: Yahoo IBM: McNealy Apple: Jobs Japan - sushi Germany: bratwurst France: tapas USA: pizza assumes exact match, the results in Table 8 would score only about 60%).
<a id="S0156"></a> Source: p.10 S0156
We believe that word vectors trained on even larger data sets with larger dimensionality will perform significantly better, and will enable the development of new innovative applications.
<a id="S0157"></a> Source: p.10 S0157
Another way to improve accuracy is to provide more than one example of the relationship.
<a id="S0158"></a> Source: p.10 S0158
By using ten examples instead of one to form the relationship vector (we average the individual vectors together), we have observed improvement of accuracy of our best models by about 10% absolutely on the semantic-syntactic test.
<a id="S0159"></a> Source: p.10 S0159
It is also possible to apply the vector operations to solve different tasks.
<a id="S0160"></a> Source: p.10 S0160
For example, we have observed good accuracy for selecting out-of-the-list words, by computing average vector for a list of words, and finding the most distant word vector.
<a id="S0161"></a> Source: p.10 S0161
This is a popular type of problems in certain human intelligence tests.
<a id="S0162"></a> Source: p.10 S0162
Clearly, there is still a lot of discoveries to be made using these techniques. 6 Conclusion In this paper we studied the quality of vector representations of words derived by various models on a collection of syntactic and semantic language tasks.
<a id="S0163"></a> Source: p.10 S0163
We observed that it is possible to train high quality word vectors using very simple model architectures, compared to the popular neural network models (both feedforward and recurrent).
<a id="S0164"></a> Source: p.10 S0164
Because of the much lower computational complexity, it is possible to compute very accurate high dimensional word vectors from a much larger data set.
<a id="S0165"></a> Source: p.10 S0165
Using the DistBelief distributed framework, it should be possible to train the CBOW and Skip-gram models even on corpora with one trillion words, for basically unlimited size of the vocabulary.
<a id="S0166"></a> Source: p.10 S0166
That is several orders of magnitude larger than the best previously published results for similar models.
<a id="S0167"></a> Source: p.10 S0167
An interesting task where the word vectors have recently been shown to significantly outperform the previous state of the art is the SemEval-2012 Task 2 [11].
<a id="S0168"></a> Source: p.10 S0168
The publicly available RNN vectors were used together with other techniques to achieve over 50% increase in Spearman’s rank correlation over the previous best result [31].
<a id="S0169"></a> Source: p.10 S0169
The neural network based word vectors were previously applied to many other NLP tasks, for example sentiment analysis [12] and paraphrase detection [28].
<a id="S0170"></a> Source: p.10 S0170
It can be expected that these applications can benefit from the model architectures described in this paper.
<a id="S0171"></a> Source: p.10 S0171
Our ongoing work shows that the word vectors can be successfully applied to automatic extension of facts in Knowledge Bases, and also for verification of correctness of existing facts.
<a id="S0172"></a> Source: p.10 S0172
Results from machine translation experiments also look very promising.
<a id="S0173"></a> Source: p.10 S0173
In the future, it would be also interesting to compare our techniques to Latent Relational Analysis [30] and others.
<a id="S0174"></a> Source: p.10 S0174
We believe that our comprehensive test set will help the research community to improve the existing techniques for estimating the word vectors.
<a id="S0175"></a> Source: p.10 S0175
We also expect that high quality word vectors will become an important building block for future NLP applications. 10
Page 11
<a id="S0176"></a> Source: p.11 S0176
7 Follow-Up Work After the initial version of this paper was written, we published single-machine multi-threaded C++ code for computing the word vectors, using both the continuous bag-of-words and skip-gram architectures4.
<a id="S0177"></a> Source: p.11 S0177
The training speed is significantly higher than reported earlier in this paper, i.e. it is in the order of billions of words per hour for typical hyperparameter choices.
<a id="S0178"></a> Source: p.11 S0178
We also published more than 1.4 million vectors that represent named entities, trained on more than 100 billion words.
<a id="S0179"></a> Source: p.11 S0179
Some of our follow-up work will be published in an upcoming NIPS 2013 paper [21].
<a id="S0180"></a> Source: p.11 S0180
Vincent. A neural probabilistic language model.
<a id="S0181"></a> Source: p.11 S0181
Journal of Machine Learning Research, 3:1137-1155, 2003. [2] Y.
<a id="S0182"></a> Source: p.11 S0182
In: Large-Scale Kernel Machines, MIT Press, 2007. [3] T.
<a id="S0183"></a> Source: p.11 S0183
Large language models in machine translation.
<a id="S0184"></a> Source: p.11 S0184
In Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, 2007. [4] R.
<a id="S0185"></a> Source: p.11 S0185
Weston. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning.
<a id="S0186"></a> Source: p.11 S0186
In International Conference on Machine Learning, ICML, 2008. [5] R.
<a id="S0187"></a> Source: p.11 S0187
Natural Language Processing (Almost) from Scratch.
<a id="S0188"></a> Source: p.11 S0188
Journal of Machine Learning Research, 12:2493- 2537, 2011. [6] J.
<a id="S0189"></a> Source: p.11 S0189
Ng., Large Scale Distributed Deep Networks, NIPS, 2012. [7] J.C.
<a id="S0190"></a> Source: p.11 S0190
Adaptive subgradient methods for online learning and stochastic optimization.
<a id="S0191"></a> Source: p.11 S0191
Journal of Machine Learning Research, 2011. [8] J.
<a id="S0192"></a> Source: p.11 S0192
Cognitive Science, 14, 179-211, 1990. [9] Eric H.
<a id="S0193"></a> Source: p.11 S0193
Improving Word Representations via Global Context and Multiple Word Prototypes.
<a id="S0194"></a> Source: p.11 S0194
Association for Computational Linguistics, 2012. [10] G.E.
<a id="S0195"></a> Source: p.11 S0195
In: Parallel distributed processing: Explorations in the microstructure of cognition.
<a id="S0196"></a> Source: p.11 S0196
Volume 1: Foundations, MIT Press, 1986. [11] D.A.
<a id="S0197"></a> Source: p.11 S0197
Semeval-2012 task 2: Measuring degrees of relational similarity.
<a id="S0198"></a> Source: p.11 S0198
In: Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012), 2012. [12] A.L.
<a id="S0199"></a> Source: p.11 S0199
Learning word vectors for sentiment analysis.
<a id="S0200"></a> Source: p.11 S0200
Language Modeling for Speech Recognition in Czech, Masters thesis, Brno University of Technology, 2007. [14] T.
<a id="S0201"></a> Source: p.11 S0201
Neural network based language models for higly inflective languages, In: Proc.
<a id="S0202"></a> Source: p.11 S0202
Recurrent neural network based language model, In: Proceedings of Interspeech, 2010. [16] T.
<a id="S0203"></a> Source: p.11 S0203
Extensions of recurrent neural network language model, In: Proceedings of ICASSP 2011. [17] T.
<a id="S0204"></a> Source: p.11 S0204
Empirical Evaluation and Combination of Advanced Language Modeling Techniques, In: Proceedings of Interspeech, 2011. 4The code is available at https://code.google.com/p/word2vec/ 11
Page 12
<a id="S0205"></a> Source: p.12 S0205
Strategies for Training Large Scale Neural Network Language Models, In: Proc.
<a id="S0206"></a> Source: p.12 S0206
Automatic Speech Recognition and Understanding, 2011. [19] T.
<a id="S0207"></a> Source: p.12 S0207
Statistical Language Models based on Neural Networks.
<a id="S0208"></a> Source: p.12 S0208
PhD thesis, Brno University of Technology, 2012. [20] T.
<a id="S0209"></a> Source: p.12 S0209
Linguistic Regularities in Continuous Space Word Representations.
<a id="S0210"></a> Source: p.12 S0210
Distributed Representations of Words and Phrases and their Compositionality.
<a id="S0211"></a> Source: p.12 S0211
Three new graphical models for statistical language modelling.
<a id="S0212"></a> Source: p.12 S0212
Hinton. A Scalable Hierarchical Distributed Language Model.
<a id="S0213"></a> Source: p.12 S0213
Advances in Neural Information Processing Systems 21, MIT Press, 2009. [24] A.
<a id="S0214"></a> Source: p.12 S0214
Teh. A fast and simple algorithm for training neural probabilistic language models.
<a id="S0215"></a> Source: p.12 S0215
Hierarchical Probabilistic Neural Network Language Model.
<a id="S0216"></a> Source: p.12 S0216
Learning internal representations by backpropagating errors.
<a id="S0217"></a> Source: p.12 S0217
Computer Speech and Language, vol. 21, 2007. [28] R.
<a id="S0218"></a> Source: p.12 S0218
Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection.
<a id="S0219"></a> Source: p.12 S0219
Word Representations: A Simple and General Method for Semi-Supervised Learning.
<a id="S0220"></a> Source: p.12 S0220
Association for Computational Linguistics, 2010. [30] P. D.
<a id="S0221"></a> Source: p.12 S0221
Measuring Semantic Similarity by Latent Relational Analysis.
<a id="S0222"></a> Source: p.12 S0222
International Joint Conference on Artificial Intelligence, 2005. [31] A.
<a id="S0223"></a> Source: p.12 S0223
Combining Heterogeneous Models for Measuring Relational Similarity.
<a id="S0224"></a> Source: p.12 S0224
The Microsoft Research Sentence Completion Challenge, Microsoft Research Technical Report MSR-TR-2011-129, 2011. 12