Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI
--- title: "Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI" aliases: - "GPT" source: "https://openai.com/research/language-unsupervised" arxiv: "" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - nlp
Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
Improving Language Understanding by Generative Pre-Training Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever OpenAI OpenAI OpenAI OpenAI alec@openai.com karthikn@openai.com tim@openai.com ilyasu@openai.com Abstract Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification.
<a id="S0002"></a> Source: p.1 S0002
Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately.
<a id="S0003"></a> Source: p.1 S0003
We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.
<a id="S0004"></a> Source: p.1 S0004
In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture.
<a id="S0005"></a> Source: p.1 S0005
We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding.
<a id="S0006"></a> Source: p.1 S0006
Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.
<a id="S0007"></a> Source: p.1 S0007
For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI). 1 Introduction The ability to learn effectively from raw text is crucial to alleviating the dependence on supervised learning in natural language processing (NLP).
<a id="S0008"></a> Source: p.1 S0008
Most deep learning methods require substantial amounts of manually labeled data, which restricts their applicability in many domains that suffer from a dearth of annotated resources [61].
<a id="S0009"></a> Source: p.1 S0009
In these situations, models that can leverage linguistic information from unlabeled data provide a valuable alternative to gathering more annotation, which can be time-consuming and expensive.
<a id="S0010"></a> Source: p.1 S0010
Further, even in cases where considerable supervision is available, learning good representations in an unsupervised fashion can provide a significant performance boost.
<a id="S0011"></a> Source: p.1 S0011
The most compelling evidence for this so far has been the extensive use of pretrained word embeddings [10, 39, 42] to improve performance on a range of NLP tasks [8, 11, 26, 45].
<a id="S0012"></a> Source: p.1 S0012
Leveraging more than word-level information from unlabeled text, however, is challenging for two main reasons.
<a id="S0013"></a> Source: p.1 S0013
First, it is unclear what type of optimization objectives are most effective at learning text representations that are useful for transfer.
<a id="S0014"></a> Source: p.1 S0014
Recent research has looked at various objectives such as language modeling [44], machine translation [38], and discourse coherence [22], with each method outperforming the others on different tasks.1 Second, there is no consensus on the most effective way to transfer these learned representations to the target task.
<a id="S0015"></a> Source: p.1 S0015
Existing techniques involve a combination of making task-specific changes to the model architecture [43, 44], using intricate learning schemes [21] and adding auxiliary learning objectives [50].
<a id="S0016"></a> Source: p.1 S0016
These uncertainties have made it difficult to develop effective semi-supervised learning approaches for language processing. 1https://gluebenchmark.com/leaderboard Preprint.
Page 2
<a id="S0017"></a> Source: p.2 S0017
In this paper, we explore a semi-supervised approach for language understanding tasks using a combination of unsupervised pre-training and supervised fine-tuning.
<a id="S0018"></a> Source: p.2 S0018
Our goal is to learn a universal representation that transfers with little adaptation to a wide range of tasks.
<a id="S0019"></a> Source: p.2 S0019
We assume access to a large corpus of unlabeled text and several datasets with manually annotated training examples (target tasks).
<a id="S0020"></a> Source: p.2 S0020
Our setup does not require these target tasks to be in the same domain as the unlabeled corpus.
<a id="S0021"></a> Source: p.2 S0021
We employ a two-stage training procedure.
<a id="S0022"></a> Source: p.2 S0022
First, we use a language modeling objective on the unlabeled data to learn the initial parameters of a neural network model.
<a id="S0023"></a> Source: p.2 S0023
Subsequently, we adapt these parameters to a target task using the corresponding supervised objective.
<a id="S0024"></a> Source: p.2 S0024
For our model architecture, we use the Transformer [62], which has been shown to perform strongly on various tasks such as machine translation [62], document generation [34], and syntactic parsing [29].
<a id="S0025"></a> Source: p.2 S0025
This model choice provides us with a more structured memory for handling long-term dependencies in text, compared to alternatives like recurrent networks, resulting in robust transfer performance across diverse tasks.
<a id="S0026"></a> Source: p.2 S0026
During transfer, we utilize task-specific input adaptations derived from traversal-style approaches [52], which process structured text input as a single contiguous sequence of tokens.
<a id="S0027"></a> Source: p.2 S0027
As we demonstrate in our experiments, these adaptations enable us to fine-tune effectively with minimal changes to the architecture of the pre-trained model.
<a id="S0028"></a> Source: p.2 S0028
We evaluate our approach on four types of language understanding tasks – natural language inference, question answering, semantic similarity, and text classification.
<a id="S0029"></a> Source: p.2 S0029
Our general task-agnostic model outperforms discriminatively trained models that employ architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied.
<a id="S0030"></a> Source: p.2 S0030
For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test) [40], 5.7% on question answering (RACE) [30], 1.5% on textual entailment (MultiNLI) [66] and 5.5% on the recently introduced GLUE multi-task benchmark [64].
<a id="S0031"></a> Source: p.2 S0031
We also analyzed zero-shot behaviors of the pre-trained model on four different settings and demonstrate that it acquires useful linguistic knowledge for downstream tasks. 2 Related Work Semi-supervised learning for NLP Our work broadly falls under the category of semi-supervised learning for natural language.
<a id="S0032"></a> Source: p.2 S0032
This paradigm has attracted significant interest, with applications to tasks like sequence labeling [24, 33, 57] or text classification [41, 70].
<a id="S0033"></a> Source: p.2 S0033
The earliest approaches used unlabeled data to compute word-level or phrase-level statistics, which were then used as features in a supervised model [33].
<a id="S0034"></a> Source: p.2 S0034
Over the last few years, researchers have demonstrated the benefits of using word embeddings [11, 39, 42], which are trained on unlabeled corpora, to improve performance on a variety of tasks [8, 11, 26, 45].
<a id="S0035"></a> Source: p.2 S0035
These approaches, however, mainly transfer word-level information, whereas we aim to capture higher-level semantics.
<a id="S0036"></a> Source: p.2 S0036
Recent approaches have investigated learning and utilizing more than word-level semantics from unlabeled data.
<a id="S0037"></a> Source: p.2 S0037
Phrase-level or sentence-level embeddings, which can be trained using an unlabeled corpus, have been used to encode text into suitable vector representations for various target tasks [28, 32, 1, 36, 22, 12, 56, 31].
<a id="S0038"></a> Source: p.2 S0038
Unsupervised pre-training Unsupervised pre-training is a special case of semi-supervised learning where the goal is to find a good initialization point instead of modifying the supervised learning objective.
<a id="S0039"></a> Source: p.2 S0039
Early works explored the use of the technique in image classification [20, 49, 63] and regression tasks [3].
<a id="S0040"></a> Source: p.2 S0040
Subsequent research [15] demonstrated that pre-training acts as a regularization scheme, enabling better generalization in deep neural networks.
<a id="S0041"></a> Source: p.2 S0041
In recent work, the method has been used to help train deep neural networks on various tasks like image classification [69], speech recognition [68], entity disambiguation [17] and machine translation [48].
<a id="S0042"></a> Source: p.2 S0042
The closest line of work to ours involves pre-training a neural network using a language modeling objective and then fine-tuning it on a target task with supervision.
<a id="S0043"></a> Source: p.2 S0043
Dai et al. [13] and Howard and Ruder [21] follow this method to improve text classification.
<a id="S0044"></a> Source: p.2 S0044
However, although the pre-training phase helps capture some linguistic information, their usage of LSTM models restricts their prediction ability to a short range.
<a id="S0045"></a> Source: p.2 S0045
In contrast, our choice of transformer networks allows us to capture longerrange linguistic structure, as demonstrated in our experiments.
<a id="S0046"></a> Source: p.2 S0046
Further, we also demonstrate the effectiveness of our model on a wider range of tasks including natural language inference, paraphrase detection and story completion.
<a id="S0047"></a> Source: p.2 S0047
Other approaches [43, 44, 38] use hidden representations from a 2
Page 3
<a id="S0048"></a> Source: p.3 S0048
pre-trained language or machine translation model as auxiliary features while training a supervised model on the target task.
<a id="S0049"></a> Source: p.3 S0049
This involves a substantial amount of new parameters for each separate target task, whereas we require minimal changes to our model architecture during transfer.
<a id="S0050"></a> Source: p.3 S0050
Auxiliary training objectives Adding auxiliary unsupervised training objectives is an alternative form of semi-supervised learning.
<a id="S0051"></a> Source: p.3 S0051
Early work by Collobert and Weston [10] used a wide variety of auxiliary NLP tasks such as POS tagging, chunking, named entity recognition, and language modeling to improve semantic role labeling.
<a id="S0052"></a> Source: p.3 S0052
More recently, Rei [50] added an auxiliary language modeling objective to their target task objective and demonstrated performance gains on sequence labeling tasks.
<a id="S0053"></a> Source: p.3 S0053
Our experiments also use an auxiliary objective, but as we show, unsupervised pre-training already learns several linguistic aspects relevant to target tasks. 3 Framework Our training procedure consists of two stages.
<a id="S0054"></a> Source: p.3 S0054
The first stage is learning a high-capacity language model on a large corpus of text.
<a id="S0055"></a> Source: p.3 S0055
This is followed by a fine-tuning stage, where we adapt the model to a discriminative task with labeled data. 3.1 Unsupervised pre-training Given an unsupervised corpus of tokens U = {u , . . . , u }, we use a standard language modeling 1 n objective to maximize the following likelihood: (cid:88) L (U) = log P (u |u , . . . , u ; Θ) (1) 1 i i−k i−1 i where k is the size of the context window, and the conditional probability P is modeled using a neural network with parameters Θ.
<a id="S0056"></a> Source: p.3 S0056
These parameters are trained using stochastic gradient descent [51].
<a id="S0057"></a> Source: p.3 S0057
In our experiments, we use a multi-layer Transformer decoder [34] for the language model, which is a variant of the transformer [62].
<a id="S0058"></a> Source: p.3 S0058
This model applies a multi-headed self-attention operation over the input context tokens followed by position-wise feedforward layers to produce an output distribution over target tokens: h = U W + W 0 e p h l = transformer_block(h l−1 )∀i ∈ [1, n] (2) P (u) = softmax(h W T ) n e where U = (u , . . . , u ) is the context vector of tokens, n is the number of layers, W is the token −k −1 e embedding matrix, and W is the position embedding matrix. p 3.2 Supervised fine-tuning After training the model with the objective in Eq. 1, we adapt the parameters to the supervised target task.
<a id="S0059"></a> Source: p.3 S0059
We assume a labeled dataset C, where each instance consists of a sequence of input tokens, x1, . . . , xm, along with a label y.
<a id="S0060"></a> Source: p.3 S0060
The inputs are passed through our pre-trained model to obtain the final transformer block’s activation hm, which is then fed into an added linear output layer with l parameters W to predict y: y P (y|x1, . . . , xm) = softmax(hmW ). (3) l y This gives us the following objective to maximize: (cid:88) L (C) = log P (y|x1, . . . , xm). (4) 2 (x,y) We additionally found that including language modeling as an auxiliary objective to the fine-tuning helped learning by (a) improving generalization of the supervised model, and (b) accelerating convergence.
<a id="S0061"></a> Source: p.3 S0061
This is in line with prior work [50, 43], who also observed improved performance with such an auxiliary objective.
<a id="S0062"></a> Source: p.3 S0062
Specifically, we optimize the following objective (with weight λ): L (C) = L (C) + λ ∗ L (C) (5) 3 2 1 Overall, the only extra parameters we require during fine-tuning are W , and embeddings for delimiter y tokens (described below in Section 3.3). 3
Page 4
<a id="S0063"></a> Source: p.4 S0063
Figure 1: (left) Transformer architecture and training objectives used in this work. (right) Input transformations for fine-tuning on different tasks.
<a id="S0064"></a> Source: p.4 S0064
We convert all structured inputs into token sequences to be processed by our pre-trained model, followed by a linear+softmax layer. 3.3 Task-specific input transformations For some tasks, like text classification, we can directly fine-tune our model as described above.
<a id="S0065"></a> Source: p.4 S0065
Certain other tasks, like question answering or textual entailment, have structured inputs such as ordered sentence pairs, or triplets of document, question, and answers.
<a id="S0066"></a> Source: p.4 S0066
Since our pre-trained model was trained on contiguous sequences of text, we require some modifications to apply it to these tasks.
<a id="S0067"></a> Source: p.4 S0067
Previous work proposed learning task specific architectures on top of transferred representations [44].
<a id="S0068"></a> Source: p.4 S0068
Such an approach re-introduces a significant amount of task-specific customization and does not use transfer learning for these additional architectural components.
<a id="S0069"></a> Source: p.4 S0069
Instead, we use a traversal-style approach [52], where we convert structured inputs into an ordered sequence that our pre-trained model can process.
<a id="S0070"></a> Source: p.4 S0070
These input transformations allow us to avoid making extensive changes to the architecture across tasks.
<a id="S0071"></a> Source: p.4 S0071
We provide a brief description of these input transformations below and Figure 1 provides a visual illustration.
<a id="S0072"></a> Source: p.4 S0072
All transformations include adding randomly initialized start and end tokens ((cid:104)s(cid:105), (cid:104)e(cid:105)).
<a id="S0073"></a> Source: p.4 S0073
Textual entailment For entailment tasks, we concatenate the premise p and hypothesis h token sequences, with a delimiter token ($) in between.
<a id="S0074"></a> Source: p.4 S0074
Similarity For similarity tasks, there is no inherent ordering of the two sentences being compared.
<a id="S0075"></a> Source: p.4 S0075
To reflect this, we modify the input sequence to contain both possible sentence orderings (with a delimiter in between) and process each independently to produce two sequence representations hm l which are added element-wise before being fed into the linear output layer.
<a id="S0076"></a> Source: p.4 S0076
Question Answering and Commonsense Reasoning For these tasks, we are given a context document z, a question q, and a set of possible answers {a }.
<a id="S0077"></a> Source: p.4 S0077
We concatenate the document context k and question with each possible answer, adding a delimiter token in between to get [z; q; $; a ].
<a id="S0078"></a> Source: p.4 S0078
Each k of these sequences are processed independently with our model and then normalized via a softmax layer to produce an output distribution over possible answers. 4 Experiments 4.1 Setup Unsupervised pre-training We use the BooksCorpus dataset [71] for training the language model.
<a id="S0079"></a> Source: p.4 S0079
It contains over 7,000 unique unpublished books from a variety of genres including Adventure, Fantasy, and Romance.
<a id="S0080"></a> Source: p.4 S0080
Crucially, it contains long stretches of contiguous text, which allows the generative model to learn to condition on long-range information.
<a id="S0081"></a> Source: p.4 S0081
An alternative dataset, the 1B Word Benchmark, which is used by a similar approach, ELMo [44], is approximately the same size 4
Page 5
<a id="S0082"></a> Source: p.5 S0082
Table 1: A list of the different tasks and datasets used in our experiments.
<a id="S0083"></a> Source: p.5 S0083
Task Datasets Natural language inference SNLI [5], MultiNLI [66], Question NLI [64], RTE [4], SciTail [25] Question Answering RACE [30], Story Cloze [40] Sentence similarity MSR Paraphrase Corpus [14], Quora Question Pairs [9], STS Benchmark [6] Classification Stanford Sentiment Treebank-2 [54], CoLA [65] but is shuffled at a sentence level - destroying long-range structure.
<a id="S0084"></a> Source: p.5 S0084
Our language model achieves a very low token level perplexity of 18.4 on this corpus.
<a id="S0085"></a> Source: p.5 S0085
Model specifications Our model largely follows the original transformer work [62].
<a id="S0086"></a> Source: p.5 S0086
We trained a 12-layer decoder-only transformer with masked self-attention heads (768 dimensional states and 12 attention heads).
<a id="S0087"></a> Source: p.5 S0087
For the position-wise feed-forward networks, we used 3072 dimensional inner states.
<a id="S0088"></a> Source: p.5 S0088
We used the Adam optimization scheme [27] with a max learning rate of 2.5e-4.
<a id="S0089"></a> Source: p.5 S0089
The learning rate was increased linearly from zero over the first 2000 updates and annealed to 0 using a cosine schedule.
<a id="S0090"></a> Source: p.5 S0090
We train for 100 epochs on minibatches of 64 randomly sampled, contiguous sequences of 512 tokens.
<a id="S0091"></a> Source: p.5 S0091
Since layernorm [2] is used extensively throughout the model, a simple weight initialization of N (0, 0.02) was sufficient.
<a id="S0092"></a> Source: p.5 S0092
We used a bytepair encoding (BPE) vocabulary with 40,000 merges [53] and residual, embedding, and attention dropouts with a rate of 0.1 for regularization.
<a id="S0093"></a> Source: p.5 S0093
We also employed a modified version of L2 regularization proposed in [37], with w = 0.01 on all non bias or gain weights.
<a id="S0094"></a> Source: p.5 S0094
For the activation function, we used the Gaussian Error Linear Unit (GELU) [18].
<a id="S0095"></a> Source: p.5 S0095
We used learned position embeddings instead of the sinusoidal version proposed in the original work.
<a id="S0096"></a> Source: p.5 S0096
We use the ftfy library2 to clean the raw text in BooksCorpus, standardize some punctuation and whitespace, and use the spaCy tokenizer.3 Fine-tuning details Unless specified, we reuse the hyperparameter settings from unsupervised pre-training.
<a id="S0097"></a> Source: p.5 S0097
We add dropout to the classifier with a rate of 0.1.
<a id="S0098"></a> Source: p.5 S0098
For most tasks, we use a learning rate of 6.25e-5 and a batchsize of 32.
<a id="S0099"></a> Source: p.5 S0099
Our model finetunes quickly and 3 epochs of training was sufficient for most cases.
<a id="S0100"></a> Source: p.5 S0100
We use a linear learning rate decay schedule with warmup over 0.2% of training. λ was set to 0.5. 4.2 Supervised fine-tuning We perform experiments on a variety of supervised tasks including natural language inference, question answering, semantic similarity, and text classification.
<a id="S0101"></a> Source: p.5 S0101
Some of these tasks are available as part of the recently released GLUE multi-task benchmark [64], which we make use of.
<a id="S0102"></a> Source: p.5 S0102
Figure 1 provides an overview of all the tasks and datasets.
<a id="S0103"></a> Source: p.5 S0103
Natural Language Inference The task of natural language inference (NLI), also known as recognizing textual entailment, involves reading a pair of sentences and judging the relationship between them from one of entailment, contradiction or neutral.
<a id="S0104"></a> Source: p.5 S0104
Although there has been a lot of recent interest [58, 35, 44], the task remains challenging due to the presence of a wide variety of phenomena like lexical entailment, coreference, and lexical and syntactic ambiguity.
<a id="S0105"></a> Source: p.5 S0105
We evaluate on five datasets with diverse sources, including image captions (SNLI), transcribed speech, popular fiction, and government reports (MNLI), Wikipedia articles (QNLI), science exams (SciTail) or news articles (RTE).
<a id="S0106"></a> Source: p.5 S0106
Table 2 details various results on the different NLI tasks for our model and previous state-of-the-art approaches.
<a id="S0107"></a> Source: p.5 S0107
Our method significantly outperforms the baselines on four of the five datasets, achieving absolute improvements of upto 1.5% on MNLI, 5% on SciTail, 5.8% on QNLI and 0.6% on SNLI over the previous best results.
<a id="S0108"></a> Source: p.5 S0108
This demonstrates our model’s ability to better reason over multiple sentences, and handle aspects of linguistic ambiguity.
<a id="S0109"></a> Source: p.5 S0109
On RTE, one of the smaller datasets we evaluate on (2490 examples), we achieve an accuracy of 56%, which is below the 61.7% reported by a multi-task biLSTM model.
<a id="S0110"></a> Source: p.5 S0110
Given the strong performance of our approach on larger NLI datasets, it is likely our model will benefit from multi-task training as well but we have not explored this currently. 2https://ftfy.readthedocs.io/en/latest/ 3https://spacy.io/ 5
Page 6
<a id="S0111"></a> Source: p.6 S0111
Table 2: Experimental results on natural language inference tasks, comparing our model with current state-of-the-art methods. 5x indicates an ensemble of 5 models.
<a id="S0112"></a> Source: p.6 S0112
All datasets use accuracy as the evaluation metric.
<a id="S0113"></a> Source: p.6 S0113
Method MNLI-m MNLI-mm SNLI SciTail QNLI RTE ESIM + ELMo [44] (5x) - - 89.3 - - - CAFE [58] (5x) 80.2 79.0 89.3 - - - Stochastic Answer Network [35] (3x) 80.6 80.1 - - - - CAFE [58] 78.7 77.9 88.5 83.3 GenSen [64] 71.4 71.3 - - 82.3 59.2 Multi-task BiLSTM + Attn [64] 72.2 72.1 - - 82.1 61.7 Finetuned Transformer LM (ours) 82.1 81.4 89.9 88.3 88.1 56.0 Table 3: Results on question answering and commonsense reasoning, comparing our model with current state-of-the-art methods.. 9x means an ensemble of 9 models.
<a id="S0114"></a> Source: p.6 S0114
Method Story Cloze RACE-m RACE-h RACE val-LS-skip [55] 76.5 - - - Hidden Coherence Model [7] 77.6 - - - Dynamic Fusion Net [67] (9x) - 55.6 49.4 51.2 BiAttention MRU [59] (9x) - 60.2 50.3 53.3 Finetuned Transformer LM (ours) 86.5 62.9 57.4 59.0 Question answering and commonsense reasoning Another task that requires aspects of single and multi-sentence reasoning is question answering.
<a id="S0115"></a> Source: p.6 S0115
We use the recently released RACE dataset [30], consisting of English passages with associated questions from middle and high school exams.
<a id="S0116"></a> Source: p.6 S0116
This corpus has been shown to contain more reasoning type questions that other datasets like CNN [19] or SQuaD [47], providing the perfect evaluation for our model which is trained to handle long-range contexts.
<a id="S0117"></a> Source: p.6 S0117
In addition, we evaluate on the Story Cloze Test [40], which involves selecting the correct ending to multi-sentence stories from two options.
<a id="S0118"></a> Source: p.6 S0118
On these tasks, our model again outperforms the previous best results by significant margins - up to 8.9% on Story Cloze, and 5.7% overall on RACE.
<a id="S0119"></a> Source: p.6 S0119
This demonstrates the ability of our model to handle long-range contexts effectively.
<a id="S0120"></a> Source: p.6 S0120
Semantic Similarity Semantic similarity (or paraphrase detection) tasks involve predicting whether two sentences are semantically equivalent or not.
<a id="S0121"></a> Source: p.6 S0121
The challenges lie in recognizing rephrasing of concepts, understanding negation, and handling syntactic ambiguity.
<a id="S0122"></a> Source: p.6 S0122
We use three datasets for this task – the Microsoft Paraphrase corpus (MRPC) [14] (collected from news sources), the Quora Question Pairs (QQP) dataset [9], and the Semantic Textual Similarity benchmark (STS-B) [6].
<a id="S0123"></a> Source: p.6 S0123
We obtain state-of-the-art results on two of the three semantic similarity tasks (Table 4) with a 1 point absolute gain on STS-B.
<a id="S0124"></a> Source: p.6 S0124
The performance delta on QQP is significant, with a 4.2% absolute improvement over Single-task BiLSTM + ELMo + Attn.
<a id="S0125"></a> Source: p.6 S0125
Classification Finally, we also evaluate on two different text classification tasks.
<a id="S0126"></a> Source: p.6 S0126
The Corpus of Linguistic Acceptability (CoLA) [65] contains expert judgements on whether a sentence is grammatical or not, and tests the innate linguistic bias of trained models.
<a id="S0127"></a> Source: p.6 S0127
The Stanford Sentiment Treebank (SST-2) [54], on the other hand, is a standard binary classification task.
<a id="S0128"></a> Source: p.6 S0128
Our model obtains an score of 45.4 on CoLA, which is an especially big jump over the previous best result of 35.0, showcasing the innate linguistic bias learned by our model.
<a id="S0129"></a> Source: p.6 S0129
The model also achieves 91.3% accuracy on SST-2, which is competitive with the state-of-the-art results.
<a id="S0130"></a> Source: p.6 S0130
We also achieve an overall score of 72.8 on the GLUE benchmark, which is significantly better than the previous best of 68.9. 6
Page 7
<a id="S0131"></a> Source: p.7 S0131
Table 4: Semantic similarity and classification results, comparing our model with current state-of-theart methods.
<a id="S0132"></a> Source: p.7 S0132
All task evaluations in this table were done using the GLUE benchmark. (mc= Mathews correlation, acc=Accuracy, pc=Pearson correlation) Method Classification Semantic Similarity GLUE CoLA SST2 MRPC STSB QQP (mc) (acc) (F1) (pc) (F1) Sparse byte mLSTM [16] - 93.2 - - - - TF-KLD [23] - - 86.0 - - - ECNU (mixed ensemble) [60] - - - 81.0 - - Single-task BiLSTM + ELMo + Attn [64] 35.0 90.2 80.2 55.5 66.1 64.8 Multi-task BiLSTM + ELMo + Attn [64] 18.9 91.6 83.5 72.8 63.3 68.9 Finetuned Transformer LM (ours) 45.4 91.3 82.3 82.0 70.3 72.8 Overall, our approach achieves new state-of-the-art results in 9 out of the 12 datasets we evaluate on, outperforming ensembles in many cases.
<a id="S0133"></a> Source: p.7 S0133
Our results also indicate that our approach works well across datasets of different sizes, from smaller datasets such as STS-B (≈5.7k training examples) – to the largest one – SNLI (≈550k training examples). 5 Analysis Impact of number of layers transferred We observed the impact of transferring a variable number of layers from unsupervised pre-training to the supervised target task.
<a id="S0134"></a> Source: p.7 S0134
Figure 2(left) illustrates the performance of our approach on MultiNLI and RACE as a function of the number of layers transferred.
<a id="S0135"></a> Source: p.7 S0135
We observe the standard result that transferring embeddings improves performance and that each transformer layer provides further benefits up to 9% for full transfer on MultiNLI.
<a id="S0136"></a> Source: p.7 S0136
This indicates that each layer in the pre-trained model contains useful functionality for solving target tasks.
<a id="S0137"></a> Source: p.7 S0137
Figure 2: (left) Effect of transferring increasing number of layers from the pre-trained language model on RACE and MultiNLI. (right) Plot showing the evolution of zero-shot performance on different tasks as a function of LM pre-training updates.
<a id="S0138"></a> Source: p.7 S0138
Performance per task is normalized between a random guess baseline and the current state-of-the-art with a single model.
<a id="S0139"></a> Source: p.7 S0139
Zero-shot Behaviors We’d like to better understand why language model pre-training of transformers is effective. A hypothesis is that the underlying generative model learns to perform many of the tasks we evaluate on in order to improve its language modeling capability and that the more structured 7
Page 8
<a id="S0140"></a> Source: p.8 S0140
Table 5: Analysis of various model ablations on different tasks.
<a id="S0141"></a> Source: p.8 S0141
Avg. score is a unweighted average of all the results. (mc= Mathews correlation, acc=Accuracy, pc=Pearson correlation) Method Avg.
<a id="S0142"></a> Source: p.8 S0142
Score CoLA SST2 MRPC STSB QQP MNLI QNLI RTE (mc) (acc) (F1) (pc) (F1) (acc) (acc) (acc) Transformer w/ aux LM (full) 74.7 45.4 91.3 82.3 82.0 70.3 81.8 88.1 56.0 Transformer w/o pre-training 59.9 18.9 84.0 79.4 30.9 65.5 75.7 71.2 53.8 Transformer w/o aux LM 75.0 47.9 92.0 84.9 83.2 69.8 81.1 86.9 54.4 LSTM w/ aux LM 69.1 30.3 90.5 83.2 71.8 68.1 73.7 81.1 54.6 attentional memory of the transformer assists in transfer compared to LSTMs.
<a id="S0143"></a> Source: p.8 S0143
We designed a series of heuristic solutions that use the underlying generative model to perform tasks without supervised finetuning.
<a id="S0144"></a> Source: p.8 S0144
We visualize the effectiveness of these heuristic solutions over the course of generative pre-training in Fig 2(right).
<a id="S0145"></a> Source: p.8 S0145
We observe the performance of these heuristics is stable and steadily increases over training suggesting that generative pretraining supports the learning of a wide variety of task relevant functionality.
<a id="S0146"></a> Source: p.8 S0146
We also observe the LSTM exhibits higher variance in its zero-shot performance suggesting that the inductive bias of the Transformer architecture assists in transfer.
<a id="S0147"></a> Source: p.8 S0147
For CoLA (linguistic acceptability), examples are scored as the average token log-probability the generative model assigns and predictions are made by thresholding.
<a id="S0148"></a> Source: p.8 S0148
For SST-2 (sentiment analysis), we append the token very to each example and restrict the language model’s output distribution to only the words positive and negative and guess the token it assigns higher probability to as the prediction.
<a id="S0149"></a> Source: p.8 S0149
For RACE (question answering), we pick the answer the generative model assigns the highest average token log-probability when conditioned on the document and question.
<a id="S0150"></a> Source: p.8 S0150
For DPRD [46] (winograd schemas), we replace the definite pronoun with the two possible referrents and predict the resolution that the generative model assigns higher average token log-probability to the rest of the sequence after the substitution.
<a id="S0151"></a> Source: p.8 S0151
Ablation studies We perform three different ablation studies (Table 5).
<a id="S0152"></a> Source: p.8 S0152
First, we examine the performance of our method without the auxiliary LM objective during fine-tuning.
<a id="S0153"></a> Source: p.8 S0153
We observe that the auxiliary objective helps on the NLI tasks and QQP.
<a id="S0154"></a> Source: p.8 S0154
Overall, the trend suggests that larger datasets benefit from the auxiliary objective but smaller datasets do not.
<a id="S0155"></a> Source: p.8 S0155
Second, we analyze the effect of the Transformer by comparing it with a single layer 2048 unit LSTM using the same framework.
<a id="S0156"></a> Source: p.8 S0156
We observe a 5.6 average score drop when using the LSTM instead of the Transformer.
<a id="S0157"></a> Source: p.8 S0157
The LSTM only outperforms the Transformer on one dataset – MRPC.
<a id="S0158"></a> Source: p.8 S0158
Finally, we also compare with our transformer architecture directly trained on supervised target tasks, without pre-training.
<a id="S0159"></a> Source: p.8 S0159
We observe that the lack of pre-training hurts performance across all the tasks, resulting in a 14.8% decrease compared to our full model. 6 Conclusion We introduced a framework for achieving strong natural language understanding with a single task-agnostic model through generative pre-training and discriminative fine-tuning.
<a id="S0160"></a> Source: p.8 S0160
By pre-training on a diverse corpus with long stretches of contiguous text our model acquires significant world knowledge and ability to process long-range dependencies which are then successfully transferred to solving discriminative tasks such as question answering, semantic similarity assessment, entailment determination, and text classification, improving the state of the art on 9 of the 12 datasets we study.
<a id="S0161"></a> Source: p.8 S0161
Using unsupervised (pre-)training to boost performance on discriminative tasks has long been an important goal of Machine Learning research.
<a id="S0162"></a> Source: p.8 S0162
Our work suggests that achieving significant performance gains is indeed possible, and offers hints as to what models (Transformers) and data sets (text with long range dependencies) work best with this approach.
<a id="S0163"></a> Source: p.8 S0163
We hope that this will help enable new research into unsupervised learning, for both natural language understanding and other domains, further improving our understanding of how and when unsupervised learning works.
<a id="S0164"></a> Source: p.8 S0164
Ma. A simple but tough-to-beat baseline for sentence embeddings. 2016. 8
Page 9
<a id="S0165"></a> Source: p.9 S0165
Layer normalization. arXiv preprint arXiv:1607.06450, 2016. [3] Y.
<a id="S0166"></a> Source: p.9 S0166
Greedy layer-wise training of deep networks.
<a id="S0167"></a> Source: p.9 S0167
In Advances in neural information processing systems, pages 153–160, 2007. [4] L.
<a id="S0168"></a> Source: p.9 S0168
The fifth pascal recognizing textual entailment challenge.
<a id="S0169"></a> Source: p.9 S0169
Manning. A large annotated corpus for learning natural language inference.
<a id="S0170"></a> Source: p.9 S0170
Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055, 2017. [7] S.
<a id="S0171"></a> Source: p.9 S0171
Story comprehension for predicting what happens next.
<a id="S0172"></a> Source: p.9 S0172
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1603–1614, 2017. [8] D.
<a id="S0173"></a> Source: p.9 S0173
Manning. A fast and accurate dependency parser using neural networks.
<a id="S0174"></a> Source: p.9 S0174
In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740–750, 2014. [9] Z.
<a id="S0175"></a> Source: p.9 S0175
Quora question pairs. https://data.quora.com/First-Quora- Dataset-Release-Question-Pairs, 2018. [10] R.
<a id="S0176"></a> Source: p.9 S0176
Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning.
<a id="S0177"></a> Source: p.9 S0177
In Proceedings of the 25th international conference on Machine learning, pages 160–167.
<a id="S0178"></a> Source: p.9 S0178
Natural language processing (almost) from scratch.
<a id="S0179"></a> Source: p.9 S0179
Journal of Machine Learning Research, 12(Aug):2493–2537, 2011. [12] A.
<a id="S0180"></a> Source: p.9 S0180
Supervised learning of universal sentence representations from natural language inference data.
<a id="S0181"></a> Source: p.9 S0181
In Advances in Neural Information Processing Systems, pages 3079–3087, 2015. [14] W. B.
<a id="S0182"></a> Source: p.9 S0182
Automatically constructing a corpus of sentential paraphrases.
<a id="S0183"></a> Source: p.9 S0183
In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. [15] D.
<a id="S0184"></a> Source: p.9 S0184
Why does unsupervised pre-training help deep learning?
<a id="S0185"></a> Source: p.9 S0185
Journal of Machine Learning Research, 11(Feb):625–660, 2010. [16] S.
<a id="S0186"></a> Source: p.9 S0186
Gpu kernels for block-sparse weights. 2017. [17] Z.
<a id="S0187"></a> Source: p.9 S0187
Learning entity representation for entity disambiguation.
<a id="S0188"></a> Source: p.9 S0188
In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 30–34, 2013. [18] D.
<a id="S0189"></a> Source: p.9 S0189
Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXiv:1606.08415, 2016. [19] K. M.
<a id="S0190"></a> Source: p.9 S0190
Teaching machines to read and comprehend.
<a id="S0191"></a> Source: p.9 S0191
In Advances in Neural Information Processing Systems, pages 1693– 1701, 2015. [20] G. E.
<a id="S0192"></a> Source: p.9 S0192
Teh. A fast learning algorithm for deep belief nets.
<a id="S0193"></a> Source: p.9 S0193
Neural computation, 18(7):1527–1554, 2006. [21] J.
<a id="S0194"></a> Source: p.9 S0194
Universal language model fine-tuning for text classification.
<a id="S0195"></a> Source: p.9 S0195
Association for Computational Linguistics (ACL), 2018. [22] Y.
<a id="S0196"></a> Source: p.9 S0196
Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017. [23] Y.
<a id="S0197"></a> Source: p.9 S0197
Discriminative improvements to distributional sentence similarity.
<a id="S0198"></a> Source: p.9 S0198
In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 891–896, 2013. 9
Page 10
<a id="S0199"></a> Source: p.10 S0199
Semi-supervised conditional random fields for improved sequence segmentation and labeling.
<a id="S0200"></a> Source: p.10 S0200
In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 209–216.
<a id="S0201"></a> Source: p.10 S0201
Association for Computational Linguistics, 2006. [25] T.
<a id="S0202"></a> Source: p.10 S0202
Scitail: A textual entailment dataset from science question answering.
<a id="S0203"></a> Source: p.10 S0203
Convolutional neural networks for sentence classification.
<a id="S0204"></a> Source: p.10 S0204
Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [28] R.
<a id="S0205"></a> Source: p.10 S0205
In Advances in neural information processing systems, pages 3294–3302, 2015. [29] N.
<a id="S0206"></a> Source: p.10 S0206
Constituency parsing with a self-attentive encoder.
<a id="S0207"></a> Source: p.10 S0207
Race: Large-scale reading comprehension dataset from examinations.
<a id="S0208"></a> Source: p.10 S0208
Unsupervised machine translation using monolingual corpora only.
<a id="S0209"></a> Source: p.10 S0209
Distributed representations of sentences and documents.
<a id="S0210"></a> Source: p.10 S0210
In International Conference on Machine Learning, pages 1188–1196, 2014. [33] P.
<a id="S0211"></a> Source: p.10 S0211
Semi-supervised learning for natural language.
<a id="S0212"></a> Source: p.10 S0212
PhD thesis, Massachusetts Institute of Technology, 2005. [34] P. J.
<a id="S0213"></a> Source: p.10 S0213
Generating wikipedia by summarizing long sequences.
<a id="S0214"></a> Source: p.10 S0214
Stochastic answer networks for natural language inference. arXiv preprint arXiv:1804.07888, 2018. [36] L.
<a id="S0215"></a> Source: p.10 S0215
An efficient framework for learning sentence representations.
<a id="S0216"></a> Source: p.10 S0216
Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101, 2017. [38] B.
<a id="S0217"></a> Source: p.10 S0217
Learned in translation: Contextualized word vectors.
<a id="S0218"></a> Source: p.10 S0218
In Advances in Neural Information Processing Systems, pages 6297–6308, 2017. [39] T.
<a id="S0219"></a> Source: p.10 S0219
Distributed representations of words and phrases and their compositionality.
<a id="S0220"></a> Source: p.10 S0220
In Advances in neural information processing systems, pages 3111–3119, 2013. [40] N.
<a id="S0221"></a> Source: p.10 S0221
Lsdsem 2017 shared task: The story cloze test.
<a id="S0222"></a> Source: p.10 S0222
In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51, 2017. [41] K.
<a id="S0223"></a> Source: p.10 S0223
Semi-supervised text classification using em.
<a id="S0224"></a> Source: p.10 S0224
Semi-Supervised Learning, pages 33–56, 2006. [42] J.
<a id="S0225"></a> Source: p.10 S0225
Glove: Global vectors for word representation.
<a id="S0226"></a> Source: p.10 S0226
In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. [43] M. E.
<a id="S0227"></a> Source: p.10 S0227
Semi-supervised sequence tagging with bidirectional language models.
<a id="S0228"></a> Source: p.10 S0228
Deep contextualized word representations.
<a id="S0229"></a> Source: p.10 S0229
When and why are pre-trained word embeddings useful for neural machine translation?
Page 11
<a id="S0230"></a> Source: p.11 S0230
Resolving complex cases of definite pronouns: the winograd schema challenge.
<a id="S0231"></a> Source: p.11 S0231
In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789.
<a id="S0232"></a> Source: p.11 S0232
Association for Computational Linguistics, 2012. [47] P.
<a id="S0233"></a> Source: p.11 S0233
Squad: 100,000+ questions for machine comprehension of text.
<a id="S0234"></a> Source: p.11 S0234
Unsupervised pretraining for sequence to sequence learning. arXiv preprint arXiv:1611.02683, 2016. [49] M.
<a id="S0235"></a> Source: p.11 S0235
Efficient learning of sparse representations with an energy-based model.
<a id="S0236"></a> Source: p.11 S0236
In Advances in neural information processing systems, pages 1137–1144, 2007. [50] M.
<a id="S0237"></a> Source: p.11 S0237
Semi-supervised multitask learning for sequence labeling.
<a id="S0238"></a> Source: p.11 S0238
Monro. A stochastic approximation method.
<a id="S0239"></a> Source: p.11 S0239
The annals of mathematical statistics, pages 400–407, 1951. [52] T.
<a id="S0240"></a> Source: p.11 S0240
Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664, 2015. [53] R.
<a id="S0241"></a> Source: p.11 S0241
Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. [54] R.
<a id="S0242"></a> Source: p.11 S0242
Recursive deep models for semantic compositionality over a sentiment treebank.
<a id="S0243"></a> Source: p.11 S0243
In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1642, 2013. [55] S.
<a id="S0244"></a> Source: p.11 S0244
Riedl. A simple and effective approach to the story cloze test. arXiv preprint arXiv:1803.05547, 2018. [56] S.
<a id="S0245"></a> Source: p.11 S0245
Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079, 2018. [57] J.
<a id="S0246"></a> Source: p.11 S0246
Semi-supervised sequential labeling and segmentation using giga-word scale unlabeled data.
<a id="S0247"></a> Source: p.11 S0247
Proceedings of ACL-08: HLT, pages 665–673, 2008. [58] Y.
<a id="S0248"></a> Source: p.11 S0248
Hui. A compare-propagate architecture with alignment factorization for natural language inference. arXiv preprint arXiv:1801.00102, 2017. [59] Y.
<a id="S0249"></a> Source: p.11 S0249
Multi-range reasoning for machine comprehension. arXiv preprint arXiv:1803.09074, 2018. [60] J.
<a id="S0250"></a> Source: p.11 S0250
Ecnu at semeval-2017 task 1: Leverage kernel-based traditional nlp features and neural networks to build a universal model for multilingual and cross-lingual semantic textual similarity.
<a id="S0251"></a> Source: p.11 S0251
In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 191–197, 2017. [61] Y.
<a id="S0252"></a> Source: p.11 S0252
Opportunities and challenges in working with low-resource languages.
<a id="S0253"></a> Source: p.11 S0253
In Advances in Neural Information Processing Systems, pages 6000–6010, 2017. [63] P.
<a id="S0254"></a> Source: p.11 S0254
Extracting and composing robust features with denoising autoencoders.
<a id="S0255"></a> Source: p.11 S0255
In Proceedings of the 25th international conference on Machine learning, pages 1096–1103.
<a id="S0256"></a> Source: p.11 S0256
Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018. [65] A.
<a id="S0257"></a> Source: p.11 S0257
Corpus of linguistic acceptability. http://nyu-mll.github.io/cola, 2018. [66] A.
<a id="S0258"></a> Source: p.11 S0258
Bowman. A broad-coverage challenge corpus for sentence understanding through inference.
<a id="S0259"></a> Source: p.11 S0259
Towards human-level machine reading comprehension: Reasoning and inference with multiple strategies. arXiv preprint arXiv:1711.04964, 2017. 11
Page 12
<a id="S0260"></a> Source: p.12 S0260
Roles of pre-training and fine-tuning in context-dependent dbn-hmms for real-world speech recognition.
<a id="S0261"></a> Source: p.12 S0261
NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2010. [69] R.
<a id="S0262"></a> Source: p.12 S0262
Split-brain autoencoders: Unsupervised learning by cross-channel prediction.
<a id="S0263"></a> Source: p.12 S0263
In CVPR, volume 1, page 6, 2017. [70] X.
<a id="S0264"></a> Source: p.12 S0264
Semi-supervised learning literature survey. 2005. [71] Y.
<a id="S0265"></a> Source: p.12 S0265
Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.
<a id="S0266"></a> Source: p.12 S0266
In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015. 12