Published as a conference paper at ICLR 2015 NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE Dzmitry Bahdanau Jacobs University Bremen,
--- title: "Published as a conference paper at ICLR 2015 NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE Dzmitry Bahdanau Jacobs University Bremen," aliases: - "Attention" - "arXiv:1409.0473" source: "https://arxiv.org/abs/1409.0473" arxiv: "1409.0473" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - nlp
Published as a conference paper at ICLR 2015 NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE Dzmitry Bahdanau Jacobs University Bremen,
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Published as a conference paper at ICLR 2015 NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE Dzmitry Bahdanau Jacobs University Bremen, Germany KyungHyun Cho Yoshua Bengio∗ Universite´ de Montre´al ABSTRACT Neural machine translation is a recently proposed approach to machine translation.
<a id="S0002"></a> Source: p.1 S0002
Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance.
<a id="S0003"></a> Source: p.1 S0003
The models proposed recently for neural machine translation often belong to a family of encoder–decoders and encode a source sentence into a fixed-length vector from which a decoder generates a translation.
<a id="S0004"></a> Source: p.1 S0004
In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder–decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly.
<a id="S0005"></a> Source: p.1 S0005
With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation.
<a id="S0006"></a> Source: p.1 S0006
Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition. 1 INTRODUCTION Neural machine translation is a newly emerging approach to machine translation, recently proposed by Kalchbrenner and Blunsom (2013), Sutskever et al. (2014) and Cho et al. (2014b).
<a id="S0007"></a> Source: p.1 S0007
Unlike the traditional phrase-based translation system (see, e.g., Koehn et al., 2003) which consists of many small sub-components that are tuned separately, neural machine translation attempts to build and train a single, large neural network that reads a sentence and outputs a correct translation.
<a id="S0008"></a> Source: p.1 S0008
Most of the proposed neural machine translation models belong to a family of encoder– decoders (Sutskever et al., 2014; Cho et al., 2014a), with an encoder and a decoder for each language, or involve a language-specific encoder applied to each sentence whose outputs are then compared (Hermann and Blunsom, 2014).
<a id="S0009"></a> Source: p.1 S0009
An encoder neural network reads and encodes a source sentence into a fixed-length vector. A decoder then outputs a translation from the encoded vector.
<a id="S0010"></a> Source: p.1 S0010
The whole encoder–decoder system, which consists of the encoder and the decoder for a language pair, is jointly trained to maximize the probability of a correct translation given a source sentence. A potential issue with this encoder–decoder approach is that a neural network needs to be able to compress all the necessary information of a source sentence into a fixed-length vector.
<a id="S0011"></a> Source: p.1 S0011
This may make it difficult for the neural network to cope with long sentences, especially those that are longer than the sentences in the training corpus.
<a id="S0012"></a> Source: p.1 S0012
Cho et al. (2014b) showed that indeed the performance of a basic encoder–decoder deteriorates rapidly as the length of an input sentence increases.
<a id="S0013"></a> Source: p.1 S0013
In order to address this issue, we introduce an extension to the encoder–decoder model which learns to align and translate jointly.
<a id="S0014"></a> Source: p.1 S0014
Each time the proposed model generates a word in a translation, it (soft-)searches for a set of positions in a source sentence where the most relevant information is concentrated.
<a id="S0015"></a> Source: p.1 S0015
The model then predicts a target word based on the context vectors associated with these source positions and all the previous generated target words. ∗CIFAR Senior Fellow 1 6102 yaM 91 ]LC.sc[ 7v3740.9041:viXra
Page 2
<a id="S0016"></a> Source: p.2 S0016
Published as a conference paper at ICLR 2015 The most important distinguishing feature of this approach from the basic encoder–decoder is that it does not attempt to encode a whole input sentence into a single fixed-length vector.
<a id="S0017"></a> Source: p.2 S0017
Instead, it encodes the input sentence into a sequence of vectors and chooses a subset of these vectors adaptively while decoding the translation.
<a id="S0018"></a> Source: p.2 S0018
This frees a neural translation model from having to squash all the information of a source sentence, regardless of its length, into a fixed-length vector.
<a id="S0019"></a> Source: p.2 S0019
We show this allows a model to cope better with long sentences.
<a id="S0020"></a> Source: p.2 S0020
In this paper, we show that the proposed approach of jointly learning to align and translate achieves significantly improved translation performance over the basic encoder–decoder approach.
<a id="S0021"></a> Source: p.2 S0021
The improvement is more apparent with longer sentences, but can be observed with sentences of any length.
<a id="S0022"></a> Source: p.2 S0022
On the task of English-to-French translation, the proposed approach achieves, with a single model, a translation performance comparable, or close, to the conventional phrase-based system.
<a id="S0023"></a> Source: p.2 S0023
Furthermore, qualitative analysis reveals that the proposed model finds a linguistically plausible (soft-)alignment between a source sentence and the corresponding target sentence. 2 BACKGROUND: NEURAL MACHINE TRANSLATION From a probabilistic perspective, translation is equivalent to finding a target sentence y that maximizes the conditional probability of y given a source sentence x, i.e., arg max p(y | x).
<a id="S0024"></a> Source: p.2 S0024
In y neural machine translation, we fit a parameterized model to maximize the conditional probability of sentence pairs using a parallel training corpus.
<a id="S0025"></a> Source: p.2 S0025
Once the conditional distribution is learned by a translation model, given a source sentence a corresponding translation can be generated by searching for the sentence that maximizes the conditional probability.
<a id="S0026"></a> Source: p.2 S0026
Recently, a number of papers have proposed the use of neural networks to directly learn this conditional distribution (see, e.g., Kalchbrenner and Blunsom, 2013; Cho et al., 2014a; Sutskever et al., 2014; Cho et al., 2014b; Forcada and N˜ eco, 1997).
<a id="S0027"></a> Source: p.2 S0027
This neural machine translation approach typically consists of two components, the first of which encodes a source sentence x and the second decodes to a target sentence y.
<a id="S0028"></a> Source: p.2 S0028
For instance, two recurrent neural networks (RNN) were used by (Cho et al., 2014a) and (Sutskever et al., 2014) to encode a variable-length source sentence into a fixed-length vector and to decode the vector into a variable-length target sentence.
<a id="S0029"></a> Source: p.2 S0029
Despite being a quite new approach, neural machine translation has already shown promising results.
<a id="S0030"></a> Source: p.2 S0030
Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long shortterm memory (LSTM) units achieves close to the state-of-the-art performance of the conventional phrase-based machine translation system on an English-to-French translation task.1 Adding neural components to existing translation systems, for instance, to score the phrase pairs in the phrase table (Cho et al., 2014a) or to re-rank candidate translations (Sutskever et al., 2014), has allowed to surpass the previous state-of-the-art performance level. 2.1 RNN ENCODER–DECODER Here, we describe briefly the underlying framework, called RNN Encoder–Decoder, proposed by Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns to align and translate simultaneously.
<a id="S0031"></a> Source: p.2 S0031
In the Encoder–Decoder framework, an encoder reads the input sentence, a sequence of vectors x = (x , · · · , x ), into a vector c.2 The most common approach is to use an RNN such that 1 Tx h = f (x , h ) (1) t t t−1 and c = q ({h , · · · , h }) , 1 Tx where h ∈ Rn is a hidden state at time t, and c is a vector generated from the sequence of the t hidden states. f and q are some nonlinear functions.
<a id="S0032"></a> Source: p.2 S0032
Sutskever et al. (2014) used an LSTM as f and q ({h , · · · , h }) = h , for instance. 1 T T 1 We mean by the state-of-the-art performance, the performance of the conventional phrase-based system without using any neural network-based component. 2 Although most of the previous works (see, e.g., Cho et al., 2014a; Sutskever et al., 2014; Kalchbrenner and Blunsom, 2013) used to encode a variable-length input sentence into a fixed-length vector, it is not necessary, and even it may be beneficial to have a variable-length vector, as we will show later. 2
Page 3
<a id="S0033"></a> Source: p.3 S0033
Published as a conference paper at ICLR 2015 The decoder is often trained to predict the next word y given the context vector c and all the t(cid:48) previously predicted words {y , · · · , y }.
<a id="S0034"></a> Source: p.3 S0034
In other words, the decoder defines a probability over 1 t(cid:48)−1 the translation y by decomposing the joint probability into the ordered conditionals: T (cid:89) p(y) = p(y | {y , · · · , y } , c), (2) t 1 t−1 t=1 (cid:0) (cid:1) where y = y , · · · , y .
<a id="S0035"></a> Source: p.3 S0035
With an RNN, each conditional probability is modeled as 1 Ty p(y | {y , · · · , y } , c) = g(y , s , c), (3) t 1 t−1 t−1 t where g is a nonlinear, potentially multi-layered, function that outputs the probability of y , and s is t t the hidden state of the RNN.
<a id="S0036"></a> Source: p.3 S0036
It should be noted that other architectures such as a hybrid of an RNN and a de-convolutional neural network can be used (Kalchbrenner and Blunsom, 2013). 3 LEARNING TO ALIGN AND TRANSLATE In this section, we propose a novel architecture for neural machine translation.
<a id="S0037"></a> Source: p.3 S0037
The new architecture consists of a bidirectional RNN as an encoder (Sec. 3.2) and a decoder that emulates searching through a source sentence during decoding a translation (Sec. 3.1). 3.1 DECODER: GENERAL DESCRIPTION In a new model architecture, we define each conditional probability in Eq. (2) as: y y t-1 t p(y |y , . . . , y , x) = g(y , s , c ), (4) i 1 i−1 i−1 i i s s t-1 t where s is an RNN hidden state for time i, computed by i s = f (s , y , c ). + i i−1 i−1 i It should be noted that unlike the existing encoder–decoder ap- α t,1 αt,T proach (see Eq. (2)), here the probability is conditioned on a distinct αt,2 αt,3 context vector c for each target word y . i i h h h h 1 2 3 T The context vector c depends on a sequence of annotations i (h , · · · , h ) to which an encoder maps the input sentence.
<a id="S0038"></a> Source: p.3 S0038
Each ann 1 otation T h x i contains information about the whole input sequence h 1 h 2 h 3 h T with a strong focus on the parts surrounding the i-th word of the x x x x input sequence.
<a id="S0039"></a> Source: p.3 S0039
We explain in detail how the annotations are com- 1 2 3 T puted in the next section.
<a id="S0040"></a> Source: p.3 S0040
Figure 1: The graphical illus- The context vector c i is, then, computed as a weighted sum of these tration of the proposed model annotations h i : trying to generate the t-th target word y given a source c i = (cid:88) Tx α ij h j . (5) sentence (x 1 t , x 2 , . . . , x T ). j=1 The weight α of each annotation h is computed by ij j exp (e ) α = ij , (6) ij (cid:80)Tx exp (e ) k=1 ik where e = a(s , h ) ij i−1 j is an alignment model which scores how well the inputs around position j and the output at position i match.
<a id="S0041"></a> Source: p.3 S0041
The score is based on the RNN hidden state s (just before emitting y , Eq. (4)) and the i−1 i j-th annotation h of the input sentence. j We parametrize the alignment model a as a feedforward neural network which is jointly trained with all the other components of the proposed system.
<a id="S0042"></a> Source: p.3 S0042
Note that unlike in traditional machine translation, 3
Page 4
<a id="S0043"></a> Source: p.4 S0043
Published as a conference paper at ICLR 2015 the alignment is not considered to be a latent variable.
<a id="S0044"></a> Source: p.4 S0044
Instead, the alignment model directly computes a soft alignment, which allows the gradient of the cost function to be backpropagated through.
<a id="S0045"></a> Source: p.4 S0045
This gradient can be used to train the alignment model as well as the whole translation model jointly.
<a id="S0046"></a> Source: p.4 S0046
We can understand the approach of taking a weighted sum of all the annotations as computing an expected annotation, where the expectation is over possible alignments.
<a id="S0047"></a> Source: p.4 S0047
Let α be a probability that ij the target word y is aligned to, or translated from, a source word x .
<a id="S0048"></a> Source: p.4 S0048
Then, the i-th context vector i j c is the expected annotation over all the annotations with probabilities α . i ij The probability α , or its associated energy e , reflects the importance of the annotation h with ij ij j respect to the previous hidden state s in deciding the next state s and generating y .
<a id="S0049"></a> Source: p.4 S0049
Intuitively, i−1 i i this implements a mechanism of attention in the decoder.
<a id="S0050"></a> Source: p.4 S0050
The decoder decides parts of the source sentence to pay attention to.
<a id="S0051"></a> Source: p.4 S0051
By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixedlength vector.
<a id="S0052"></a> Source: p.4 S0052
With this new approach the information can be spread throughout the sequence of annotations, which can be selectively retrieved by the decoder accordingly. 3.2 ENCODER: BIDIRECTIONAL RNN FOR ANNOTATING SEQUENCES The usual RNN, described in Eq. (1), reads an input sequence x in order starting from the first symbol x to the last one x .
<a id="S0053"></a> Source: p.4 S0053
However, in the proposed scheme, we would like the annotation 1 Tx of each word to summarize not only the preceding words, but also the following words.
<a id="S0054"></a> Source: p.4 S0054
Hence, we propose to use a bidirectional RNN (BiRNN, Schuster and Paliwal, 1997), which has been successfully used recently in speech recognition (see, e.g., Graves et al., 2013). →− A BiRNN consists of forward and backward RNN’s.
<a id="S0055"></a> Source: p.4 S0055
The forward RNN f reads the input sequence →− →− as it is ordered (from x to x ) and calculates a sequence of forward hidden states ( h , · · · , h ). ←− 1 Tx 1 Tx The backward RNN f reads the sequence in the reverse order (from x to x ), resulting in a ←− ←− Tx 1 sequence of backward hidden states ( h , · · · , h ). 1 Tx →− We obtain an annotation for each word x by concatenating the forward hidden state h and the j j ←− (cid:104)→− ←− (cid:105)(cid:62) backward one h , i.e., h = h (cid:62); h (cid:62) .
<a id="S0056"></a> Source: p.4 S0056
In this way, the annotation h contains the summaries j j j j j of both the preceding words and the following words.
<a id="S0057"></a> Source: p.4 S0057
Due to the tendency of RNNs to better represent recent inputs, the annotation h will be focused on the words around x .
<a id="S0058"></a> Source: p.4 S0058
This sequence j j of annotations is used by the decoder and the alignment model later to compute the context vector (Eqs. (5)–(6)).
<a id="S0059"></a> Source: p.4 S0059
See Fig. 1 for the graphical illustration of the proposed model. 4 EXPERIMENT SETTINGS We evaluate the proposed approach on the task of English-to-French translation.
<a id="S0060"></a> Source: p.4 S0060
We use the bilingual, parallel corpora provided by ACL WMT ’14.3 As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a).
<a id="S0061"></a> Source: p.4 S0061
We use the same training procedures and the same dataset for both models.4 4.1 DATASET WMT ’14 contains the following English-French parallel corpora: Europarl (61M words), news commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively, totaling 850M words.
<a id="S0062"></a> Source: p.4 S0062
Following the procedure described in Cho et al. (2014a), we reduce the size of the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011).5 We do not use any monolingual data other than the mentioned parallel corpora, although it may be possible to use a much larger monolingual corpus to pretrain an encoder.
<a id="S0063"></a> Source: p.4 S0063
We concatenate news-test- 3 http://www.statmt.org/wmt14/translation-task.html 4 Implementations are available at https://github.com/lisa-groundhog/GroundHog. 5 Available online at http://www-lium.univ-lemans.fr/˜schwenk/cslm_joint_paper/. 4
Page 5
<a id="S0064"></a> Source: p.5 S0064
Published as a conference paper at ICLR 2015 30 25 20 15 10 5 0 0 10 20 30 40 50 60 Sentence length erocsUELB Figure 2: The BLEU scores of the generated translations on the test set with respect to the lengths of the sen- RNNsearch-50 tences.
<a id="S0065"></a> Source: p.5 S0065
The results are on RNNsearch-30 the full test set which in- RNNenc-50 cludes sentences having un- RNNenc-30 known words to the models. 2012 and news-test-2013 to make a development (validation) set, and evaluate the models on the test set (news-test-2014) from WMT ’14, which consists of 3003 sentences not present in the training data.
<a id="S0066"></a> Source: p.5 S0066
After a usual tokenization6, we use a shortlist of 30,000 most frequent words in each language to train our models.
<a id="S0067"></a> Source: p.5 S0067
Any word not included in the shortlist is mapped to a special token ([UNK]).
<a id="S0068"></a> Source: p.5 S0068
We do not apply any other special preprocessing, such as lowercasing or stemming, to the data. 4.2 MODELS We train two types of models.
<a id="S0069"></a> Source: p.5 S0069
The first one is an RNN Encoder–Decoder (RNNencdec, Cho et al., 2014a), and the other is the proposed model, to which we refer as RNNsearch.
<a id="S0070"></a> Source: p.5 S0070
We train each model twice: first with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then with the sentences of length up to 50 word (RNNencdec-50, RNNsearch-50).
<a id="S0071"></a> Source: p.5 S0071
The encoder and decoder of the RNNencdec have 1000 hidden units each.7 The encoder of the RNNsearch consists of forward and backward recurrent neural networks (RNN) each having 1000 hidden units.
<a id="S0072"></a> Source: p.5 S0072
In both cases, we use a multilayer network with a single maxout (Goodfellow et al., 2013) hidden layer to compute the conditional probability of each target word (Pascanu et al., 2014).
<a id="S0073"></a> Source: p.5 S0073
We use a minibatch stochastic gradient descent (SGD) algorithm together with Adadelta (Zeiler, 2012) to train each model.
<a id="S0074"></a> Source: p.5 S0074
Each SGD update direction is computed using a minibatch of 80 sentences.
<a id="S0075"></a> Source: p.5 S0075
We trained each model for approximately 5 days.
<a id="S0076"></a> Source: p.5 S0076
Once a model is trained, we use a beam search to find a translation that approximately maximizes the conditional probability (see, e.g., Graves, 2012; Boulanger-Lewandowski et al., 2013).
<a id="S0077"></a> Source: p.5 S0077
Sutskever et al. (2014) used this approach to generate translations from their neural machine translation model.
<a id="S0078"></a> Source: p.5 S0078
For more details on the architectures of the models and training procedure used in the experiments, see Appendices A and B. 5 RESULTS 5.1 QUANTITATIVE RESULTS In Table 1, we list the translation performances measured in BLEU score.
<a id="S0079"></a> Source: p.5 S0079
It is clear from the table that in all the cases, the proposed RNNsearch outperforms the conventional RNNencdec.
<a id="S0080"></a> Source: p.5 S0080
More importantly, the performance of the RNNsearch is as high as that of the conventional phrase-based translation system (Moses), when only the sentences consisting of known words are considered.
<a id="S0081"></a> Source: p.5 S0081
This is a significant achievement, considering that Moses uses a separate monolingual corpus (418M words) in addition to the parallel corpora we used to train the RNNsearch and RNNencdec. 6 We used the tokenization script from the open-source machine translation package, Moses. 7 In this paper, by a ’hidden unit’, we always mean the gated hidden unit (see Appendix A.1.1). 5
Page 6
<a id="S0082"></a> Source: p.6 S0082
Published as a conference paper at ICLR 2015 ehT tnemeerga no eht naeporuE cimonocE aerA saw dengis ni tsuguA 2991 . >dne< L' accord sur la zone économique européenne a été signé en août 1992 . <end> tI dluohs eb deton taht eht eniram tnemnorivne si eht tsael nwonk fo stnemnorivne . >dne< Il convient de noter que l' environnement marin est le moins connu de l' environnement . <end> (a) (b) noitcurtseD fo eht tnempiuqe snaem taht airyS nac on regnol ecudorp wen lacimehc snopaew . >dne< La destruction de l' équipement signifie que la Syrie ne peut plus produire de nouvelles armes chimiques . <end> " sihT lliw egnahc ym erutuf htiw ym ylimaf , " eht nam dias . >dne< " Cela va changer mon avenir avec ma famille " , a dit l' homme . <end> (c) (d) Figure 3: Four sample alignments found by RNNsearch-50.
<a id="S0083"></a> Source: p.6 S0083
The x-axis and y-axis of each plot correspond to the words in the source sentence (English) and the generated translation (French), respectively.
<a id="S0084"></a> Source: p.6 S0084
Each pixel shows the weight α of the annotation of the j-th source word for the i-th ij target word (see Eq. (6)), in grayscale (0: black, 1: white). (a) an arbitrary sentence. (b–d) three randomly selected samples among the sentences without any unknown words and of length between 10 and 20 words from the test set.
<a id="S0085"></a> Source: p.6 S0085
One of the motivations behind the proposed approach was the use of a fixed-length context vector in the basic encoder–decoder approach.
<a id="S0086"></a> Source: p.6 S0086
We conjectured that this limitation may make the basic encoder–decoder approach to underperform with long sentences.
<a id="S0087"></a> Source: p.6 S0087
In Fig. 2, we see that the performance of RNNencdec dramatically drops as the length of the sentences increases.
<a id="S0088"></a> Source: p.6 S0088
On the other hand, both RNNsearch-30 and RNNsearch-50 are more robust to the length of the sentences.
<a id="S0089"></a> Source: p.6 S0089
RNNsearch- 50, especially, shows no performance deterioration even with sentences of length 50 or more.
<a id="S0090"></a> Source: p.6 S0090
This superiority of the proposed model over the basic encoder–decoder is further confirmed by the fact that the RNNsearch-30 even outperforms RNNencdec-50 (see Table 1). 6
Page 7
<a id="S0091"></a> Source: p.7 S0091
Published as a conference paper at ICLR 2015 Table 1: BLEU scores of the trained models com- Model All No UNK◦ puted on the test set.
<a id="S0092"></a> Source: p.7 S0092
The second and third columns show respectively the scores on all the sentences and, RNNencdec-30 13.93 24.19 on the sentences without any unknown word in them- RNNsearch-30 21.50 31.44 selves and in the reference translations.
<a id="S0093"></a> Source: p.7 S0093
Note that RNNencdec-50 17.82 26.71 RNNsearch-50(cid:63) was trained much longer until the RNNsearch-50 26.75 34.16 performance on the development set stopped improv- RNNsearch-50(cid:63) 28.45 36.15 ing. (◦) We disallowed the models to generate [UNK] Moses 33.30 35.63 tokens when only the sentences having no unknown words were evaluated (last column). 5.2 QUALITATIVE ANALYSIS 5.2.1 ALIGNMENT The proposed approach provides an intuitive way to inspect the (soft-)alignment between the words in a generated translation and those in a source sentence.
<a id="S0094"></a> Source: p.7 S0094
This is done by visualizing the annotation weights α from Eq. (6), as in Fig. 3.
<a id="S0095"></a> Source: p.7 S0095
Each row of a matrix in each plot indicates the weights ij associated with the annotations.
<a id="S0096"></a> Source: p.7 S0096
From this we see which positions in the source sentence were considered more important when generating the target word.
<a id="S0097"></a> Source: p.7 S0097
We can see from the alignments in Fig. 3 that the alignment of words between English and French is largely monotonic.
<a id="S0098"></a> Source: p.7 S0098
We see strong weights along the diagonal of each matrix.
<a id="S0099"></a> Source: p.7 S0099
However, we also observe a number of non-trivial, non-monotonic alignments.
<a id="S0100"></a> Source: p.7 S0100
Adjectives and nouns are typically ordered differently between French and English, and we see an example in Fig. 3 (a).
<a id="S0101"></a> Source: p.7 S0101
From this figure, we see that the model correctly translates a phrase [European Economic Area] into [zone e´conomique europe´en].
<a id="S0102"></a> Source: p.7 S0102
The RNNsearch was able to correctly align [zone] with [Area], jumping over the two words ([European] and [Economic]), and then looked one word back at a time to complete the whole phrase [zone e´conomique europe´enne].
<a id="S0103"></a> Source: p.7 S0103
The strength of the soft-alignment, opposed to a hard-alignment, is evident, for instance, from Fig. 3 (d).
<a id="S0104"></a> Source: p.7 S0104
Consider the source phrase [the man] which was translated into [l’ homme].
<a id="S0105"></a> Source: p.7 S0105
Any hard alignment will map [the] to [l’] and [man] to [homme].
<a id="S0106"></a> Source: p.7 S0106
This is not helpful for translation, as one must consider the word following [the] to determine whether it should be translated into [le], [la], [les] or [l’].
<a id="S0107"></a> Source: p.7 S0107
Our soft-alignment solves this issue naturally by letting the model look at both [the] and [man], and in this example, we see that the model was able to correctly translate [the] into [l’].
<a id="S0108"></a> Source: p.7 S0108
We observe similar behaviors in all the presented cases in Fig. 3.
<a id="S0109"></a> Source: p.7 S0109
An additional benefit of the soft alignment is that it naturally deals with source and target phrases of different lengths, without requiring a counter-intuitive way of mapping some words to or from nowhere ([NULL]) (see, e.g., Chapters 4 and 5 of Koehn, 2010). 5.2.2 LONG SENTENCES As clearly visible from Fig. 2 the proposed model (RNNsearch) is much better than the conventional model (RNNencdec) at translating long sentences.
<a id="S0110"></a> Source: p.7 S0110
This is likely due to the fact that the RNNsearch does not require encoding a long sentence into a fixed-length vector perfectly, but only accurately encoding the parts of the input sentence that surround a particular word.
<a id="S0111"></a> Source: p.7 S0111
As an example, consider this source sentence from the test set: An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
<a id="S0112"></a> Source: p.7 S0112
The RNNencdec-50 translated this sentence into: Un privilege d’admission est le droit d’un me´decin de reconnaˆıtre un patient a l’hoˆpital ou un centre me´dical d’un diagnostic ou de prendre un diagnostic en fonction de son e´tat de sante´. 7
Page 8
<a id="S0113"></a> Source: p.8 S0113
Published as a conference paper at ICLR 2015 The RNNencdec-50 correctly translated the source sentence until [a medical center].
<a id="S0114"></a> Source: p.8 S0114
However, from there on (underlined), it deviated from the original meaning of the source sentence.
<a id="S0115"></a> Source: p.8 S0115
For instance, it replaced [based on his status as a health care worker at a hospital] in the source sentence with [en fonction de son e´tat de sante´] (“based on his state of health”).
<a id="S0116"></a> Source: p.8 S0116
On the other hand, the RNNsearch-50 generated the following correct translation, preserving the whole meaning of the input sentence without omitting any details: Un privilege d’admission est le droit d’un me´decin d’admettre un patient a un hoˆpital ou un centre me´dical pour effectuer un diagnostic ou une proce´dure, selon son statut de travailleur des soins de sante´ a` l’hoˆpital.
<a id="S0117"></a> Source: p.8 S0117
Let us consider another sentence from the test set: This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added.
<a id="S0118"></a> Source: p.8 S0118
The translation by the RNNencdec-50 is Ce type d’expe´rience fait partie des initiatives du Disney pour ”prolonger la dure´e de vie de ses nouvelles et de de´velopper des liens avec les lecteurs nume´riques qui deviennent plus complexes.
<a id="S0119"></a> Source: p.8 S0119
As with the previous example, the RNNencdec began deviating from the actual meaning of the source sentence after generating approximately 30 words (see the underlined phrase).
<a id="S0120"></a> Source: p.8 S0120
After that point, the quality of the translation deteriorates, with basic mistakes such as the lack of a closing quotation mark.
<a id="S0121"></a> Source: p.8 S0121
Again, the RNNsearch-50 was able to translate this long sentence correctly: Ce genre d’expe´rience fait partie des efforts de Disney pour ”prolonger la dure´e de vie de ses se´ries et cre´er de nouvelles relations avec des publics via des plateformes nume´riques de plus en plus importantes”, a-t-il ajoute´.
<a id="S0122"></a> Source: p.8 S0122
In conjunction with the quantitative results presented already, these qualitative observations confirm our hypotheses that the RNNsearch architecture enables far more reliable translation of long sentences than the standard RNNencdec model.
<a id="S0123"></a> Source: p.8 S0123
In Appendix C, we provide a few more sample translations of long source sentences generated by the RNNencdec-50, RNNsearch-50 and Google Translate along with the reference translations. 6 RELATED WORK 6.1 LEARNING TO ALIGN A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis.
<a id="S0124"></a> Source: p.8 S0124
Handwriting synthesis is a task where the model is asked to generate handwriting of a given sequence of characters.
<a id="S0125"></a> Source: p.8 S0125
In his work, he used a mixture of Gaussian kernels to compute the weights of the annotations, where the location, width and mixture coefficient of each kernel was predicted from an alignment model.
<a id="S0126"></a> Source: p.8 S0126
More specifically, his alignment was restricted to predict the location such that the location increases monotonically.
<a id="S0127"></a> Source: p.8 S0127
The main difference from our approach is that, in (Graves, 2013), the modes of the weights of the annotations only move in one direction.
<a id="S0128"></a> Source: p.8 S0128
In the context of machine translation, this is a severe limitation, as (long-distance) reordering is often needed to generate a grammatically correct translation (for instance, English-to-German).
<a id="S0129"></a> Source: p.8 S0129
Our approach, on the other hand, requires computing the annotation weight of every word in the source sentence for each word in the translation.
<a id="S0130"></a> Source: p.8 S0130
This drawback is not severe with the task of translation in which most of input and output sentences are only 15–40 words.
<a id="S0131"></a> Source: p.8 S0131
However, this may limit the applicability of the proposed scheme to other tasks. 8
Page 9
<a id="S0132"></a> Source: p.9 S0132
Published as a conference paper at ICLR 2015 6.2 NEURAL NETWORKS FOR MACHINE TRANSLATION Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a fixed number of the preceding words, neural networks have widely been used in machine translation.
<a id="S0133"></a> Source: p.9 S0133
However, the role of neural networks has been largely limited to simply providing a single feature to an existing statistical machine translation system or to re-rank a list of candidate translations provided by an existing system.
<a id="S0134"></a> Source: p.9 S0134
For instance, Schwenk (2012) proposed using a feedforward neural network to compute the score of a pair of source and target phrases and to use the score as an additional feature in the phrase-based statistical machine translation system.
<a id="S0135"></a> Source: p.9 S0135
More recently, Kalchbrenner and Blunsom (2013) and Devlin et al. (2014) reported the successful use of the neural networks as a sub-component of the existing translation system.
<a id="S0136"></a> Source: p.9 S0136
Traditionally, a neural network trained as a target-side language model has been used to rescore or rerank a list of candidate translations (see, e.g., Schwenk et al., 2006).
<a id="S0137"></a> Source: p.9 S0137
Although the above approaches were shown to improve the translation performance over the stateof-the-art machine translation systems, we are more interested in a more ambitious objective of designing a completely new translation system based on neural networks.
<a id="S0138"></a> Source: p.9 S0138
The neural machine translation approach we consider in this paper is therefore a radical departure from these earlier works.
<a id="S0139"></a> Source: p.9 S0139
Rather than using a neural network as a part of the existing system, our model works on its own and generates a translation from a source sentence directly. 7 CONCLUSION The conventional approach to neural machine translation, called an encoder–decoder approach, encodes a whole input sentence into a fixed-length vector from which a translation will be decoded.
<a id="S0140"></a> Source: p.9 S0140
We conjectured that the use of a fixed-length context vector is problematic for translating long sentences, based on a recent empirical study reported by Cho et al. (2014b) and Pouget-Abadie et al. (2014).
<a id="S0141"></a> Source: p.9 S0141
In this paper, we proposed a novel architecture that addresses this issue.
<a id="S0142"></a> Source: p.9 S0142
We extended the basic encoder–decoder by letting a model (soft-)search for a set of input words, or their annotations computed by an encoder, when generating each target word.
<a id="S0143"></a> Source: p.9 S0143
This frees the model from having to encode a whole source sentence into a fixed-length vector, and also lets the model focus only on information relevant to the generation of the next target word.
<a id="S0144"></a> Source: p.9 S0144
This has a major positive impact on the ability of the neural machine translation system to yield good results on longer sentences.
<a id="S0145"></a> Source: p.9 S0145
Unlike with the traditional machine translation systems, all of the pieces of the translation system, including the alignment mechanism, are jointly trained towards a better log-probability of producing correct translations.
<a id="S0146"></a> Source: p.9 S0146
We tested the proposed model, called RNNsearch, on the task of English-to-French translation.
<a id="S0147"></a> Source: p.9 S0147
The experiment revealed that the proposed RNNsearch outperforms the conventional encoder–decoder model (RNNencdec) significantly, regardless of the sentence length and that it is much more robust to the length of a source sentence.
<a id="S0148"></a> Source: p.9 S0148
From the qualitative analysis where we investigated the (soft-)alignment generated by the RNNsearch, we were able to conclude that the model can correctly align each target word with the relevant words, or their annotations, in the source sentence as it generated a correct translation.
<a id="S0149"></a> Source: p.9 S0149
Perhaps more importantly, the proposed approach achieved a translation performance comparable to the existing phrase-based statistical machine translation.
<a id="S0150"></a> Source: p.9 S0150
It is a striking result, considering that the proposed architecture, or the whole family of neural machine translation, has only been proposed as recently as this year.
<a id="S0151"></a> Source: p.9 S0151
We believe the architecture proposed here is a promising step toward better machine translation and a better understanding of natural languages in general.
<a id="S0152"></a> Source: p.9 S0152
One of challenges left for the future is to better handle unknown, or rare words.
<a id="S0153"></a> Source: p.9 S0153
This will be required for the model to be more widely used and to match the performance of current state-of-the-art machine translation systems in all contexts. 9
Page 10
<a id="S0154"></a> Source: p.10 S0154
Published as a conference paper at ICLR 2015 ACKNOWLEDGMENTS The authors would like to thank the developers of Theano (Bergstra et al., 2010; Bastien et al., 2012).
<a id="S0155"></a> Source: p.10 S0155
We acknowledge the support of the following agencies for research funding and computing support: NSERC, Calcul Que´bec, Compute Canada, the Canada Research Chairs and CIFAR.
<a id="S0156"></a> Source: p.10 S0156
Bahdanau thanks the support from Planet Intelligent Systems GmbH.
<a id="S0157"></a> Source: p.10 S0157
We also thank Felix Hill, Bart van Merrie´nboer, Jean Pouget-Abadie, Coline Devin and Tae-Ho Kim.
<a id="S0158"></a> Source: p.10 S0158
REFERENCES Axelrod, A., He, X., and Gao, J. (2011).
<a id="S0159"></a> Source: p.10 S0159
Domain adaptation via pseudo in-domain data selection.
<a id="S0160"></a> Source: p.10 S0160
In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–362.
<a id="S0161"></a> Source: p.10 S0161
Association for Computational Linguistics.
<a id="S0162"></a> Source: p.10 S0162
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012).
<a id="S0163"></a> Source: p.10 S0163
Theano: new features and speed improvements.
<a id="S0164"></a> Source: p.10 S0164
Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop.
<a id="S0165"></a> Source: p.10 S0165
Bengio, Y., Simard, P., and Frasconi, P. (1994).
<a id="S0166"></a> Source: p.10 S0166
Learning long-term dependencies with gradient descent is difficult.
<a id="S0167"></a> Source: p.10 S0167
IEEE Transactions on Neural Networks, 5(2), 157–166.
<a id="S0168"></a> Source: p.10 S0168
Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. J.
<a id="S0169"></a> Source: p.10 S0169
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde- Farley, D., and Bengio, Y. (2010).
<a id="S0170"></a> Source: p.10 S0170
Theano: a CPU and GPU math expression compiler.
<a id="S0171"></a> Source: p.10 S0171
In Proceedings of the Python for Scientific Computing Conference (SciPy).
<a id="S0172"></a> Source: p.10 S0172
Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2013).
<a id="S0173"></a> Source: p.10 S0173
Audio chord recognition with recurrent neural networks.
<a id="S0174"></a> Source: p.10 S0174
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014a).
<a id="S0175"></a> Source: p.10 S0175
Learning phrase representations using RNN encoder-decoder for statistical machine translation.
<a id="S0176"></a> Source: p.10 S0176
In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014). to appear.
<a id="S0177"></a> Source: p.10 S0177
Cho, K., van Merrie¨nboer, B., Bahdanau, D., and Bengio, Y. (2014b).
<a id="S0178"></a> Source: p.10 S0178
On the properties of neural machine translation: Encoder–Decoder approaches.
<a id="S0179"></a> Source: p.10 S0179
In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear.
<a id="S0180"></a> Source: p.10 S0180
Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014).
<a id="S0181"></a> Source: p.10 S0181
Fast and robust neural network joint models for statistical machine translation.
<a id="S0182"></a> Source: p.10 S0182
In Association for Computational Linguistics.
<a id="S0183"></a> Source: p.10 S0183
Forcada, M. L. and N˜ eco, R. P. (1997).
<a id="S0184"></a> Source: p.10 S0184
Recursive hetero-associative memories for translation.
<a id="S0185"></a> Source: p.10 S0185
Cabestany, editors, Biological and Artificial Computation: From Neuroscience to Technology, volume 1240 of Lecture Notes in Computer Science, pages 453–462.
<a id="S0186"></a> Source: p.10 S0186
Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013).
<a id="S0187"></a> Source: p.10 S0187
In Proceedings of The 30th International Conference on Machine Learning, pages 1319– 1327.
<a id="S0188"></a> Source: p.10 S0188
Sequence transduction with recurrent neural networks.
<a id="S0189"></a> Source: p.10 S0189
In Proceedings of the 29th International Conference on Machine Learning (ICML 2012).
<a id="S0190"></a> Source: p.10 S0190
Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs.NE].
<a id="S0191"></a> Source: p.10 S0191
Graves, A., Jaitly, N., and Mohamed, A.-R. (2013).
<a id="S0192"></a> Source: p.10 S0192
Hybrid speech recognition with deep bidirectional LSTM.
<a id="S0193"></a> Source: p.10 S0193
In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. 10
Page 11
<a id="S0194"></a> Source: p.11 S0194
Published as a conference paper at ICLR 2015 Hermann, K. and Blunsom, P. (2014).
<a id="S0195"></a> Source: p.11 S0195
Multilingual distributed representations without word alignment.
<a id="S0196"></a> Source: p.11 S0196
In Proceedings of the Second International Conference on Learning Representations (ICLR 2014).
<a id="S0197"></a> Source: p.11 S0197
Untersuchungen zu dynamischen neuronalen Netzen.
<a id="S0198"></a> Source: p.11 S0198
Diploma thesis, Institut fu¨r Informatik, Lehrstuhl Prof.
<a id="S0199"></a> Source: p.11 S0199
Brauer, Technische Universita¨t Mu¨nchen.
<a id="S0200"></a> Source: p.11 S0200
Hochreiter, S. and Schmidhuber, J. (1997).
<a id="S0201"></a> Source: p.11 S0201
Kalchbrenner, N. and Blunsom, P. (2013).
<a id="S0202"></a> Source: p.11 S0202
Recurrent continuous translation models.
<a id="S0203"></a> Source: p.11 S0203
In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709.
<a id="S0204"></a> Source: p.11 S0204
Association for Computational Linguistics.
<a id="S0205"></a> Source: p.11 S0205
Cambridge University Press, New York, NY, USA.
<a id="S0206"></a> Source: p.11 S0206
Koehn, P., Och, F. J., and Marcu, D. (2003).
<a id="S0207"></a> Source: p.11 S0207
In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA.
<a id="S0208"></a> Source: p.11 S0208
Association for Computational Linguistics.
<a id="S0209"></a> Source: p.11 S0209
Pascanu, R., Mikolov, T., and Bengio, Y. (2013a).
<a id="S0210"></a> Source: p.11 S0210
On the difficulty of training recurrent neural networks.
<a id="S0211"></a> Source: p.11 S0211
Pascanu, R., Mikolov, T., and Bengio, Y. (2013b).
<a id="S0212"></a> Source: p.11 S0212
On the difficulty of training recurrent neural networks.
<a id="S0213"></a> Source: p.11 S0213
In Proceedings of the 30th International Conference on Machine Learning (ICML 2013).
<a id="S0214"></a> Source: p.11 S0214
Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. (2014).
<a id="S0215"></a> Source: p.11 S0215
How to construct deep recurrent neural networks.
<a id="S0216"></a> Source: p.11 S0216
In Proceedings of the Second International Conference on Learning Representations (ICLR 2014).
<a id="S0217"></a> Source: p.11 S0217
Pouget-Abadie, J., Bahdanau, D., van Merrie¨nboer, B., Cho, K., and Bengio, Y. (2014).
<a id="S0218"></a> Source: p.11 S0218
Overcoming the curse of sentence length for neural machine translation using automatic segmentation.
<a id="S0219"></a> Source: p.11 S0219
In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear.
<a id="S0220"></a> Source: p.11 S0220
Bidirectional recurrent neural networks.
<a id="S0221"></a> Source: p.11 S0221
Signal Processing, IEEE Transactions on, 45(11), 2673–2681.
<a id="S0222"></a> Source: p.11 S0222
Continuous space translation models for phrase-based statistical machine translation.
<a id="S0223"></a> Source: p.11 S0223
Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080.
<a id="S0224"></a> Source: p.11 S0224
Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006).
<a id="S0225"></a> Source: p.11 S0225
Continuous space language models for statistical machine translation.
<a id="S0226"></a> Source: p.11 S0226
In Proceedings of the COLING/ACL on Main conference poster sessions, pages 723–730.
<a id="S0227"></a> Source: p.11 S0227
Association for Computational Linguistics.
<a id="S0228"></a> Source: p.11 S0228
Sutskever, I., Vinyals, O., and Le, Q. (2014).
<a id="S0229"></a> Source: p.11 S0229
Sequence to sequence learning with neural networks.
<a id="S0230"></a> Source: p.11 S0230
In Advances in Neural Information Processing Systems (NIPS 2014).
<a id="S0231"></a> Source: p.11 S0231
ADADELTA: An adaptive learning rate method. arXiv:1212.5701 [cs.LG]. 11
Page 12
<a id="S0232"></a> Source: p.12 S0232
Published as a conference paper at ICLR 2015 A MODEL ARCHITECTURE A.1 ARCHITECTURAL CHOICES The proposed scheme in Section 3 is a general framework where one can freely define, for instance, the activation functions f of recurrent neural networks (RNN) and the alignment model a.
<a id="S0233"></a> Source: p.12 S0233
Here, we describe the choices we made for the experiments in this paper. A.1.1 RECURRENT NEURAL NETWORK For the activation function f of an RNN, we use the gated hidden unit recently proposed by Cho et al. (2014a).
<a id="S0234"></a> Source: p.12 S0234
The gated hidden unit is an alternative to the conventional simple units such as an element-wise tanh.
<a id="S0235"></a> Source: p.12 S0235
This gated unit is similar to a long short-term memory (LSTM) unit proposed earlier by Hochreiter and Schmidhuber (1997), sharing with it the ability to better model and learn long-term dependencies.
<a id="S0236"></a> Source: p.12 S0236
This is made possible by having computation paths in the unfolded RNN for which the product of derivatives is close to 1.
<a id="S0237"></a> Source: p.12 S0237
These paths allow gradients to flow backward easily without suffering too much from the vanishing effect (Hochreiter, 1991; Bengio et al., 1994; Pascanu et al., 2013a).
<a id="S0238"></a> Source: p.12 S0238
It is therefore possible to use LSTM units instead of the gated hidden unit described here, as was done in a similar context by Sutskever et al. (2014).
<a id="S0239"></a> Source: p.12 S0239
The new state s of the RNN employing n gated hidden units8 is computed by i s = f (s , y , c ) = (1 − z ) ◦ s + z ◦ s˜ , i i−1 i−1 i i i−1 i i where ◦ is an element-wise multiplication, and z is the output of the update gates (see below).
<a id="S0240"></a> Source: p.12 S0240
The i proposed updated state s˜ is computed by i s˜ = tanh (W e(y ) + U [r ◦ s ] + Cc ) , i i−1 i i−1 i where e(y ) ∈ Rm is an m-dimensional embedding of a word y , and r is the output of the i−1 i−1 i reset gates (see below).
<a id="S0241"></a> Source: p.12 S0241
When y is represented as a 1-of-K vector, e(y ) is simply a column of an i i embedding matrix E ∈ Rm×K.
<a id="S0242"></a> Source: p.12 S0242
Whenever possible, we omit bias terms to make the equations less cluttered.
<a id="S0243"></a> Source: p.12 S0243
The update gates z allow each hidden unit to maintain its previous activation, and the reset gates r i i control how much and what information from the previous state should be reset.
<a id="S0244"></a> Source: p.12 S0244
We compute them by z = σ (W e(y ) + U s + C c ) , i z i−1 z i−1 z i r = σ (W e(y ) + U s + C c ) , i r i−1 r i−1 r i where σ (·) is a logistic sigmoid function.
<a id="S0245"></a> Source: p.12 S0245
At each step of the decoder, we compute the output probability (Eq. (4)) as a multi-layered function (Pascanu et al., 2014).
<a id="S0246"></a> Source: p.12 S0246
We use a single hidden layer of maxout units (Goodfellow et al., 2013) and normalize the output probabilities (one for each word) with a softmax function (see Eq. (6)). A.1.2 ALIGNMENT MODEL The alignment model should be designed considering that the model needs to be evaluated T × T x y times for each sentence pair of lengths T and T .
<a id="S0247"></a> Source: p.12 S0247
In order to reduce computation, we use a singlex y layer multilayer perceptron such that a(s , h ) = v(cid:62) tanh (W s + U h ) , i−1 j a a i−1 a j where W ∈ Rn×n, U ∈ Rn×2n and v ∈ Rn are the weight matrices.
<a id="S0248"></a> Source: p.12 S0248
Since U h does not a a a a j depend on i, we can pre-compute it in advance to minimize the computational cost. 8 Here, we show the formula of the decoder.
<a id="S0249"></a> Source: p.12 S0249
The same formula can be used in the encoder by simply ignoring the context vector c and the related terms. i 12
Page 13
<a id="S0250"></a> Source: p.13 S0250
Published as a conference paper at ICLR 2015 A.2 DETAILED DESCRIPTION OF THE MODEL A.2.1 ENCODER In this section, we describe in detail the architecture of the proposed model (RNNsearch) used in the experiments (see Sec. 4–5).
<a id="S0251"></a> Source: p.13 S0251
From here on, we omit all bias terms in order to increase readability.
<a id="S0252"></a> Source: p.13 S0252
The model takes a source sentence of 1-of-K coded word vectors as input x = (x , . . . , x ), x ∈ RKx 1 Tx i and outputs a translated sentence of 1-of-K coded word vectors y = (y , . . . , y ), y ∈ RKy , 1 Ty i where K and K are the vocabulary sizes of source and target languages, respectively. T and T x y x y respectively denote the lengths of source and target sentences.
<a id="S0253"></a> Source: p.13 S0253
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed: (cid:40) →− →− →− →− →− (1 − z ) ◦ h + z ◦ h , if i > 0 h = i i−1 i i i 0 , if i = 0 where →− (cid:16)−→ →− (cid:104)→− →− (cid:105)(cid:17) h = tanh W Ex + U r ◦ h i i i i−1 →− (cid:16)−→ →− →− (cid:17) z =σ W Ex + U h i z i z i−1 →− (cid:16)−→ →− →− (cid:17) r =σ W Ex + U h . i r i r i−1 −→ −→ −→ →− →− →− E ∈ Rm×Kx is the word embedding matrix. W , W z , W r ∈ Rn×m, U , U z , U r ∈ Rn×n are weight matrices. m and n are the word embedding dimensionality and the number of hidden units, respectively. σ(·) is as usual a logistic sigmoid function. ←− ←− The backward states ( h , · · · , h ) are computed similarly.
<a id="S0254"></a> Source: p.13 S0254
We share the word embedding matrix 1 Tx E between the forward and backward RNNs, unlike the weight matrices.
<a id="S0255"></a> Source: p.13 S0255
We concatenate the forward and backward states to to obtain the annotations (h , h , · · · , h ), 1 2 Tx where (cid:34) →− (cid:35) h h i = ←− i (7) h i A.2.2 DECODER The hidden state s of the decoder given the annotations from the encoder is computed by i s =(1 − z ) ◦ s + z ◦ s˜ , i i i−1 i i where s˜ = tanh (W Ey + U [r ◦ s ] + Cc ) i i−1 i i−1 i z =σ (W Ey + U s + C c ) i z i−1 z i−1 z i r =σ (W Ey + U s + C c ) i r i−1 r i−1 r i E is the word embedding matrix for the target language. W, W , W ∈ Rn×m, U, U , U ∈ Rn×n, z r z r and C, C , C ∈ Rn×2n are weights.
<a id="S0256"></a> Source: p.13 S0256
Again, m and n are the word embedding dimensionality z r and the number of hidden units, respectively.
<a id="S0257"></a> Source: p.13 S0257
The initial hidden state s is computed by s = 0 0 (cid:16) ←− (cid:17) tanh W h , where W ∈ Rn×n. s 1 s The context vector c are recomputed at each step by the alignment model: i (cid:88) Tx c = α h , i ij j j=1 13
Page 14
<a id="S0258"></a> Source: p.14 S0258
Published as a conference paper at ICLR 2015 Model Updates (×105) Epochs Hours GPU Train NLL Dev.
<a id="S0259"></a> Source: p.14 S0259
NLL RNNenc-30 8.46 6.4 109 TITAN BLACK 28.1 53.0 RNNenc-50 6.00 4.5 108 Quadro K-6000 44.0 43.6 RNNsearch-30 4.71 3.6 113 TITAN BLACK 26.7 47.2 RNNsearch-50 2.88 2.2 111 Quadro K-6000 40.7 38.1 RNNsearch-50(cid:63) 6.67 5.0 252 Quadro K-6000 36.7 35.2 Table 2: Learning statistics and relevant information.
<a id="S0260"></a> Source: p.14 S0260
Each update corresponds to updating the parameters once using a single minibatch.
<a id="S0261"></a> Source: p.14 S0261
One epoch is one pass through the training set.
<a id="S0262"></a> Source: p.14 S0262
NLL is the average conditional log-probabilities of the sentences in either the training set or the development set.
<a id="S0263"></a> Source: p.14 S0263
Note that the lengths of the sentences differ. where exp (e ) α = ij ij (cid:80)Tx exp (e ) k=1 ik e =v(cid:62) tanh (W s + U h ) , ij a a i−1 a j and h is the j-th annotation in the source sentence (see Eq. (7)). v ∈ Rn(cid:48), W ∈ Rn(cid:48)×n and j a a U ∈ Rn(cid:48)×2n are weight matrices.
<a id="S0264"></a> Source: p.14 S0264
Note that the model becomes RNN Encoder–Decoder (Cho a →− et al., 2014a), if we fix c to h . i Tx With the decoder state s , the context c and the last generated word y , we define the probability i−1 i i−1 of a target word y as i p(y |s , y , c ) ∝ exp (cid:0) y(cid:62)W t (cid:1) , i i i−1 i i o i where t = (cid:2) max (cid:8) t˜ , t˜ (cid:9)(cid:3)(cid:62) i i,2j−1 i,2j j=1,...,l and t˜ is the k-th element of a vector t˜ which is computed by i,k i t˜ =U s + V Ey + C c . i o i−1 o i−1 o i W o ∈ RKy×l, U o ∈ R2l×n, V o ∈ R2l×m and C o ∈ R2l×2n are weight matrices.
<a id="S0265"></a> Source: p.14 S0265
This can be understood as having a deep output (Pascanu et al., 2014) with a single maxout hidden layer (Goodfellow et al., 2013). A.2.3 MODEL SIZE For all the models used in this paper, the size of a hidden layer n is 1000, the word embedding dimensionality m is 620 and the size of the maxout hidden layer in the deep output l is 500.
<a id="S0266"></a> Source: p.14 S0266
The number of hidden units in the alignment model n(cid:48) is 1000. B TRAINING PROCEDURE B.1 PARAMETER INITIALIZATION ←− ←− ←− →− →− →− We initialized the recurrent weight matrices U, U , U , U , U , U , U , U and U as random orz r z r z r thogonal matrices.
<a id="S0267"></a> Source: p.14 S0267
For W and U , we initialized them by sampling each element from the Gaussian a a distribution of mean 0 and variance 0.0012.
<a id="S0268"></a> Source: p.14 S0268
All the elements of V and all the bias vectors were inia tialized to zero.
<a id="S0269"></a> Source: p.14 S0269
Any other weight matrix was initialized by sampling from the Gaussian distribution of mean 0 and variance 0.012. B.2 TRAINING We used the stochastic gradient descent (SGD) algorithm.
<a id="S0270"></a> Source: p.14 S0270
Adadelta (Zeiler, 2012) was used to automatically adapt the learning rate of each parameter ((cid:15) = 10−6 and ρ = 0.95).
Page 15
<a id="S0271"></a> Source: p.15 S0271
Published as a conference paper at ICLR 2015 normalized the L -norm of the gradient of the cost function each time to be at most a predefined 2 threshold of 1, when the norm was larger than the threshold (Pascanu et al., 2013b).
<a id="S0272"></a> Source: p.15 S0272
Each SGD update direction was computed with a minibatch of 80 sentences.
<a id="S0273"></a> Source: p.15 S0273
At each update our implementation requires time proportional to the length of the longest sentence in a minibatch.
<a id="S0274"></a> Source: p.15 S0274
Hence, to minimize the waste of computation, before every 20-th update, we retrieved 1600 sentence pairs, sorted them according to the lengths and split them into 20 minibatches.
<a id="S0275"></a> Source: p.15 S0275
The training data was shuffled once before training and was traversed sequentially in this manner.
<a id="S0276"></a> Source: p.15 S0276
In Tables 2 we present the statistics related to training all the models used in the experiments. C TRANSLATIONS OF LONG SENTENCES Source An admitting privilege is the right of a doctor to admit a patient to a hospital or a medical centre to carry out a diagnosis or a procedure, based on his status as a health care worker at a hospital.
<a id="S0277"></a> Source: p.15 S0277
Reference Le privile`ge d’admission est le droit d’un me´decin, en vertu de son statut de membre soignant d’un hoˆpital, d’admettre un patient dans un hoˆpital ou un centre me´dical afin d’y de´livrer un diagnostic ou un traitement.
<a id="S0278"></a> Source: p.15 S0278
RNNenc-50 Un privilege d’admission est le droit d’un me´decin de reconnaˆıtre un patient a l’hoˆpital ou un centre me´dical d’un diagnostic ou de prendre un diagnostic en fonction de son e´tat de sante´.
<a id="S0279"></a> Source: p.15 S0279
RNNsearch-50 Un privilege d’admission est le droit d’un me´decin d’admettre un patient a un hoˆpital ou un centre me´dical pour effectuer un diagnostic ou une proce´dure, selon son statut de travailleur des soins de sante´ a` l’hoˆpital.
<a id="S0280"></a> Source: p.15 S0280
Google Un privile`ge admettre est le droit d’un me´decin d’admettre un patient dans un hoˆpital ou un Translate centre me´dical pour effectuer un diagnostic ou une proce´dure, fonde´e sur sa situation en tant que travailleur de soins de sante´ dans un hoˆpital.
<a id="S0281"></a> Source: p.15 S0281
Source This kind of experience is part of Disney’s efforts to ”extend the lifetime of its series and build new relationships with audiences via digital platforms that are becoming ever more important,” he added.
<a id="S0282"></a> Source: p.15 S0282
Reference Ce type d’expe´rience entre dans le cadre des efforts de Disney pour ”e´tendre la dure´e de vie de ses se´ries et construire de nouvelles relations avec son public graˆce a` des plateformes nume´riques qui sont de plus en plus importantes”, a-t-il ajoute´.
<a id="S0283"></a> Source: p.15 S0283
RNNenc-50 Ce type d’expe´rience fait partie des initiatives du Disney pour ”prolonger la dure´e de vie de ses nouvelles et de de´velopper des liens avec les lecteurs nume´riques qui deviennent plus complexes.
<a id="S0284"></a> Source: p.15 S0284
RNNsearch-50 Ce genre d’expe´rience fait partie des efforts de Disney pour ”prolonger la dure´e de vie de ses se´ries et cre´er de nouvelles relations avec des publics via des plateformes nume´riques de plus en plus importantes”, a-t-il ajoute´.
<a id="S0285"></a> Source: p.15 S0285
Google Ce genre d’expe´rience fait partie des efforts de Disney a` “e´tendre la dure´e de vie de sa se´rie et Translate construire de nouvelles relations avec le public par le biais des plates-formes nume´riques qui deviennent de plus en plus important”, at-il ajoute´.
<a id="S0286"></a> Source: p.15 S0286
Source In a press conference on Thursday, Mr Blair stated that there was nothing in this video that might constitute a ”reasonable motive” that could lead to criminal charges being brought against the mayor.
<a id="S0287"></a> Source: p.15 S0287
Reference En confe´rence de presse, jeudi, M.
<a id="S0288"></a> Source: p.15 S0288
Blair a affirme´ qu’il n’y avait rien dans cette vide´o qui puisse constituer des ”motifs raisonnables” pouvant mener au de´poˆt d’une accusation criminelle contre le maire.
<a id="S0289"></a> Source: p.15 S0289
RNNenc-50 Lors de la confe´rence de presse de jeudi, M.
<a id="S0290"></a> Source: p.15 S0290
Blair a dit qu’il n’y avait rien dans cette vide´o qui pourrait constituer une ”motivation raisonnable” pouvant entraˆıner des accusations criminelles porte´es contre le maire.
<a id="S0291"></a> Source: p.15 S0291
RNNsearch-50 Lors d’une confe´rence de presse jeudi, M.
<a id="S0292"></a> Source: p.15 S0292
Blair a de´clare´ qu’il n’y avait rien dans cette vide´o qui pourrait constituer un ”motif raisonnable” qui pourrait conduire a` des accusations criminelles contre le maire.
<a id="S0293"></a> Source: p.15 S0293
Google Lors d’une confe´rence de presse jeudi, M.
<a id="S0294"></a> Source: p.15 S0294
Blair a de´clare´ qu’il n’y avait rien dans cette vido Translate qui pourrait constituer un ”motif raisonnable” qui pourrait mener a` des accusations criminelles portes contre le maire.
<a id="S0295"></a> Source: p.15 S0295
Table 3: The translations generated by RNNenc-50 and RNNsearch-50 from long source sentences (30 words or more) selected from the test set.
<a id="S0296"></a> Source: p.15 S0296
For each source sentence, we also show the goldstandard translation.
<a id="S0297"></a> Source: p.15 S0297
The translations by Google Translate were made on 27 August 2014. 15