BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language
--- title: "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language" aliases: - "BERT" - "arXiv:1810.04805" source: "https://arxiv.org/abs/1810.04805" arxiv: "1810.04805" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - nlp
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova Google AI Language {jacobdevlin,mingweichang,kentonl,kristout}@google.com Abstract There are two existing strategies for applying pre-trained language representations to down- We introduce a new language representastream tasks: feature-based and fine-tuning.
<a id="S0002"></a> Source: p.1 S0002
The tion model called BERT, which stands for feature-based approach, such as ELMo (Peters Bidirectional Encoder Representations from Transformers.
<a id="S0003"></a> Source: p.1 S0003
Unlike recent language repre- et al., 2018a), uses task-specific architectures that sentation models (Peters et al., 2018a; Rad- include the pre-trained representations as addiford et al., 2018), BERT is designed to pre- tional features.
<a id="S0004"></a> Source: p.1 S0004
The fine-tuning approach, such as train deep bidirectional representations from the Generative Pre-trained Transformer (OpenAI unlabeled text by jointly conditioning on both GPT) (Radford et al., 2018), introduces minimal left and right context in all layers.
<a id="S0005"></a> Source: p.1 S0005
As a retask-specific parameters, and is trained on the sult, the pre-trained BERT model can be finedownstream tasks by simply fine-tuning all pretuned with just one additional output layer trained parameters.
<a id="S0006"></a> Source: p.1 S0006
The two approaches share the to create state-of-the-art models for a wide range of tasks, such as question answering and same objective function during pre-training, where language inference, without substantial task- they use unidirectional language models to learn specific architecture modifications. general language representations.
<a id="S0007"></a> Source: p.1 S0007
BERT is conceptually simple and empirically We argue that current techniques restrict the powerful.
<a id="S0008"></a> Source: p.1 S0008
It obtains new state-of-the-art re- power of the pre-trained representations, espesults on eleven natural language processing cially for the fine-tuning approaches.
<a id="S0009"></a> Source: p.1 S0009
The matasks, including pushing the GLUE score to jor limitation is that standard language models are 80.5% (7.7% point absolute improvement), unidirectional, and this limits the choice of archi- MultiNLI accuracy to 86.7% (4.6% absolute tectures that can be used during pre-training.
<a id="S0010"></a> Source: p.1 S0010
For improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute im- example, in OpenAI GPT, the authors use a left-toprovement) and SQuAD v2.0 Test F1 to 83.1 right architecture, where every token can only at- (5.1 point absolute improvement). tend to previous tokens in the self-attention layers of the Transformer (Vaswani et al., 2017).
<a id="S0011"></a> Source: p.1 S0011
Such re- 1 Introduction strictions are sub-optimal for sentence-level tasks, Language model pre-training has been shown to and could be very harmful when applying finebe effective for improving many natural language tuning based approaches to token-level tasks such processing tasks (Dai and Le, 2015; Peters et al., as question answering, where it is crucial to incor- 2018a; Radford et al., 2018; Howard and Ruder, porate context from both directions. 2018).
<a id="S0012"></a> Source: p.1 S0012
These include sentence-level tasks such as In this paper, we improve the fine-tuning based natural language inference (Bowman et al., 2015; approaches by proposing BERT: Bidirectional Williams et al., 2018) and paraphrasing (Dolan Encoder Representations from Transformers. and Brockett, 2005), which aim to predict the re- BERT alleviates the previously mentioned unidilationships between sentences by analyzing them rectionality constraint by using a “masked lanholistically, as well as token-level tasks such as guage model” (MLM) pre-training objective, innamed entity recognition and question answering, spired by the Cloze task (Taylor, 1953).
<a id="S0013"></a> Source: p.1 S0013
The where models are required to produce fine-grained masked language model randomly masks some of output at the token level (Tjong Kim Sang and the tokens from the input, and the objective is to De Meulder, 2003; Rajpurkar et al., 2016). predict the original vocabulary id of the masked 9102 yaM 42 ]LC.sc[ 2v50840.0181:viXra
Page 2
<a id="S0014"></a> Source: p.2 S0014
Unlike left-to- These approaches have been generalized to right language model pre-training, the MLM ob- coarser granularities, such as sentence embedjective enables the representation to fuse the left dings (Kiros et al., 2015; Logeswaran and Lee, and the right context, which allows us to pre- 2018) or paragraph embeddings (Le and Mikolov, train a deep bidirectional Transformer.
<a id="S0015"></a> Source: p.2 S0015
To train sentence representations, prior tion to the masked language model, we also use work has used objectives to rank candidate next a “next sentence prediction” task that jointly pre- sentences (Jernite et al., 2017; Logeswaran and trains text-pair representations.
<a id="S0016"></a> Source: p.2 S0016
The contributions Lee, 2018), left-to-right generation of next senof our paper are as follows: tence words given a representation of the previous sentence (Kiros et al., 2015), or denoising auto- • We demonstrate the importance of bidirectional encoder derived objectives (Hill et al., 2016). pre-training for language representations.
<a id="S0017"></a> Source: p.2 S0017
Un- ELMo and its predecessor (Peters et al., 2017, like Radford et al. (2018), which uses unidirec- 2018a) generalize traditional word embedding retional language models for pre-training, BERT search along a different dimension.
<a id="S0018"></a> Source: p.2 S0018
They extract uses masked language models to enable precontext-sensitive features from a left-to-right and a trained deep bidirectional representations.
<a id="S0019"></a> Source: p.2 S0019
The contextual repis also in contrast to Peters et al. (2018a), which resentation of each token is the concatenation of uses a shallow concatenation of independently the left-to-right and right-to-left representations. trained left-to-right and right-to-left LMs.
<a id="S0020"></a> Source: p.2 S0020
When integrating contextual word embeddings • We show that pre-trained representations reduce with existing task-specific architectures, ELMo the need for many heavily-engineered task- advances the state of the art for several major NLP specific architectures.
<a id="S0021"></a> Source: p.2 S0021
BERT is the first fine- benchmarks (Peters et al., 2018a) including questuning based representation model that achieves tion answering (Rajpurkar et al., 2016), sentiment state-of-the-art performance on a large suite analysis (Socher et al., 2013), and named entity of sentence-level and token-level tasks, outper- recognition (Tjong Kim Sang and De Meulder, forming many task-specific architectures. 2003).
<a id="S0022"></a> Source: p.2 S0022
Melamud et al. (2016) proposed learning contextual representations through a task to pre- • BERT advances the state of the art for eleven dict a single word from both left and right context NLP tasks.
<a id="S0023"></a> Source: p.2 S0023
The code and pre-trained modusing LSTMs.
<a id="S0024"></a> Source: p.2 S0024
Similar to ELMo, their model is els are available at https://github.com/ feature-based and not deeply bidirectional.
<a id="S0025"></a> Source: p.2 S0025
Fedus google-research/bert. et al. (2018) shows that the cloze task can be used to improve the robustness of text generation mod- 2 Related Work els.
<a id="S0026"></a> Source: p.2 S0026
There is a long history of pre-training general language representations, and we briefly review the 2.2 Unsupervised Fine-tuning Approaches most widely-used approaches in this section.
<a id="S0027"></a> Source: p.2 S0027
As with the feature-based approaches, the first 2.1 Unsupervised Feature-based Approaches works in this direction only pre-trained word em- Learning widely applicable representations of bedding parameters from unlabeled text (Colwords has been an active area of research for lobert and Weston, 2008). decades, including non-neural (Brown et al., 1992; More recently, sentence or document encoders Ando and Zhang, 2005; Blitzer et al., 2006) and which produce contextual token representations neural (Mikolov et al., 2013; Pennington et al., have been pre-trained from unlabeled text and 2014) methods.
<a id="S0028"></a> Source: p.2 S0028
Pre-trained word embeddings fine-tuned for a supervised downstream task (Dai are an integral part of modern NLP systems, of- and Le, 2015; Howard and Ruder, 2018; Radford fering significant improvements over embeddings et al., 2018).
<a id="S0029"></a> Source: p.2 S0029
The advantage of these approaches learned from scratch (Turian et al., 2010).
<a id="S0030"></a> Source: p.2 S0030
To pre- is that few parameters need to be learned from train word embedding vectors, left-to-right lan- scratch.
<a id="S0031"></a> Source: p.2 S0031
At least partly due to this advantage, guage modeling objectives have been used (Mnih OpenAI GPT (Radford et al., 2018) achieved preand Hinton, 2009), as well as objectives to dis- viously state-of-the-art results on many sentencecriminate correct from incorrect words in left and level tasks from the GLUE benchmark (Wang right context (Mikolov et al., 2013). et al., 2018a).
Page 3
<a id="S0032"></a> Source: p.3 S0032
NSP Mask LM Mask LM MNLI NER SQuAD Start/End Span C T ... T T T ’ ... T ’ C T ... T T T ’ ... T ’ 1 N [SEP] 1 M 1 N [SEP] 1 M BERT BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ [CLS] Tok 1 ...
<a id="S0033"></a> Source: p.3 S0033
TokM Masked Sentence A Masked Sentence B Question Paragraph Unlabeled Sentence A and B Pair Question Answer Pair Pre-training Fine-Tuning Figure 1: Overall pre-training and fine-tuning procedures for BERT.
<a id="S0034"></a> Source: p.3 S0034
Apart from output layers, the same architectures are used in both pre-training and fine-tuning.
<a id="S0035"></a> Source: p.3 S0035
The same pre-trained model parameters are used to initialize models for different down-stream tasks.
<a id="S0036"></a> Source: p.3 S0036
During fine-tuning, all parameters are fine-tuned. [CLS] is a special symbol added in front of every input example, and [SEP] is a special separator token (e.g. separating questions/answers). ing and auto-encoder objectives have been used mal difference between the pre-trained architecfor pre-training such models (Howard and Ruder, ture and the final downstream architecture. 2018; Radford et al., 2018; Dai and Le, 2015).
<a id="S0037"></a> Source: p.3 S0037
Model Architecture BERT’s model architec- 2.3 Transfer Learning from Supervised Data ture is a multi-layer bidirectional Transformer encoder based on the original implementation de- There has also been work showing effective transscribed in Vaswani et al. (2017) and released in fer from supervised tasks with large datasets, such the tensor2tensor library.1 Because the use as natural language inference (Conneau et al., of Transformers has become common and our im- 2017) and machine translation (McCann et al., plementation is almost identical to the original, 2017).
<a id="S0038"></a> Source: p.3 S0038
Computer vision research has also demonwe will omit an exhaustive background descripstrated the importance of transfer learning from tion of the model architecture and refer readers to large pre-trained models, where an effective recipe Vaswani et al. (2017) as well as excellent guides is to fine-tune models pre-trained with Imasuch as “The Annotated Transformer.”2 geNet (Deng et al., 2009; Yosinski et al., 2014).
<a id="S0039"></a> Source: p.3 S0039
In this work, we denote the number of layers 3 BERT (i.e., Transformer blocks) as L, the hidden size as H, and the number of self-attention heads as A.3 We introduce BERT and its detailed implementa- We primarily report results on two model sizes: tion in this section.
<a id="S0040"></a> Source: p.3 S0040
There are two steps in our BERT (L=12, H=768, A=12, Total Param- BASE framework: pre-training and fine-tuning.
<a id="S0041"></a> Source: p.3 S0041
Dur- eters=110M) and BERT (L=24, H=1024, LARGE ing pre-training, the model is trained on unlabeled A=16, Total Parameters=340M). data over different pre-training tasks.
<a id="S0042"></a> Source: p.3 S0042
For fine- BERT was chosen to have the same model BASE tuning, the BERT model is first initialized with size as OpenAI GPT for comparison purposes. the pre-trained parameters, and all of the param- Critically, however, the BERT Transformer uses eters are fine-tuned using labeled data from the bidirectional self-attention, while the GPT Transdownstream tasks.
<a id="S0043"></a> Source: p.3 S0043
Each downstream task has sepformer uses constrained self-attention where every arate fine-tuned models, even though they are ini- token can only attend to context to its left.4 tialized with the same pre-trained parameters.
<a id="S0044"></a> Source: p.3 S0044
The question-answering example in Figure 1 will serve 1https://github.com/tensorflow/tensor2tensor as a running example for this section. 2http://nlp.seas.harvard.edu/2018/04/03/attention.html 3In all cases we set the feed-forward/filter size to be 4H, A distinctive feature of BERT is its unified ari.e., 3072 for the H = 768 and 4096 for the H = 1024. chitecture across different tasks.
<a id="S0045"></a> Source: p.3 S0045
There is mini- 4We note that in the literature the bidirectional Trans-
Page 4
<a id="S0046"></a> Source: p.4 S0046
Input/Output Representations To make BERT In order to train a deep bidirectional representahandle a variety of down-stream tasks, our input tion, we simply mask some percentage of the input representation is able to unambiguously represent tokens at random, and then predict those masked both a single sentence and a pair of sentences tokens.
<a id="S0047"></a> Source: p.4 S0047
We refer to this procedure as a “masked (e.g., (cid:104) Question, Answer (cid:105)) in one token sequence.
<a id="S0048"></a> Source: p.4 S0048
LM” (MLM), although it is often referred to as a Throughout this work, a “sentence” can be an arbi- Cloze task in the literature (Taylor, 1953).
<a id="S0049"></a> Source: p.4 S0049
In this trary span of contiguous text, rather than an actual case, the final hidden vectors corresponding to the linguistic sentence. A “sequence” refers to the in- mask tokens are fed into an output softmax over put token sequence to BERT, which may be a sin- the vocabulary, as in a standard LM.
<a id="S0050"></a> Source: p.4 S0050
In all of our gle sentence or two sentences packed together. experiments, we mask 15% of all WordPiece to- We use WordPiece embeddings (Wu et al., kens in each sequence at random.
<a id="S0051"></a> Source: p.4 S0051
In contrast to 2016) with a 30,000 token vocabulary.
<a id="S0052"></a> Source: p.4 S0052
The first denoising auto-encoders (Vincent et al., 2008), we token of every sequence is always a special clas- only predict the masked words rather than reconsification token ([CLS]).
<a id="S0053"></a> Source: p.4 S0053
The final hidden state structing the entire input. corresponding to this token is used as the ag- Although this allows us to obtain a bidirecgregate sequence representation for classification tional pre-trained model, a downside is that we tasks.
<a id="S0054"></a> Source: p.4 S0054
Sentence pairs are packed together into a are creating a mismatch between pre-training and single sequence.
<a id="S0055"></a> Source: p.4 S0055
We differentiate the sentences in fine-tuning, since the [MASK] token does not aptwo ways.
<a id="S0056"></a> Source: p.4 S0056
First, we separate them with a special pear during fine-tuning.
<a id="S0057"></a> Source: p.4 S0057
Second, we add a learned embed- not always replace “masked” words with the acding to every token indicating whether it belongs tual [MASK] token.
<a id="S0058"></a> Source: p.4 S0058
The training data generator to sentence A or sentence B.
<a id="S0059"></a> Source: p.4 S0059
As shown in Figure 1, chooses 15% of the token positions at random for we denote input embedding as E, the final hidden prediction.
<a id="S0060"></a> Source: p.4 S0060
If the i-th token is chosen, we replace vector of the special [CLS] token as C ∈ RH , the i-th token with (1) the [MASK] token 80% of and the final hidden vector for the ith input token the time (2) a random token 10% of the time (3) as T ∈ RH . the unchanged i-th token 10% of the time.
<a id="S0061"></a> Source: p.4 S0061
Then, i For a given token, its input representation is T i will be used to predict the original token with constructed by summing the corresponding token, cross entropy loss.
<a id="S0062"></a> Source: p.4 S0062
We compare variations of this segment, and position embeddings. A visualiza- procedure in Appendix C.2. tion of this construction can be seen in Figure 2.
<a id="S0063"></a> Source: p.4 S0063
Task #2: Next Sentence Prediction (NSP) 3.1 Pre-training BERT Many important downstream tasks such as Question Answering (QA) and Natural Language Infer- Unlike Peters et al. (2018a) and Radford et al. ence (NLI) are based on understanding the rela- (2018), we do not use traditional left-to-right or tionship between two sentences, which is not diright-to-left language models to pre-train BERT. rectly captured by language modeling.
<a id="S0064"></a> Source: p.4 S0064
In order Instead, we pre-train BERT using two unsuperto train a model that understands sentence relavised tasks, described in this section.
<a id="S0065"></a> Source: p.4 S0065
This step tionships, we pre-train for a binarized next senis presented in the left part of Figure 1. tence prediction task that can be trivially gener- Task #1: Masked LM Intuitively, it is reason- ated from any monolingual corpus.
<a id="S0066"></a> Source: p.4 S0066
Specifically, able to believe that a deep bidirectional model is when choosing the sentences A and B for each prestrictly more powerful than either a left-to-right training example, 50% of the time B is the actual model or the shallow concatenation of a left-to- next sentence that follows A (labeled as IsNext), right and a right-to-left model.
<a id="S0067"></a> Source: p.4 S0067
Unfortunately, and 50% of the time it is a random sentence from standard conditional language models can only be the corpus (labeled as NotNext).
<a id="S0068"></a> Source: p.4 S0068
As we show trained left-to-right or right-to-left, since bidirec- in Figure 1, C is used for next sentence predictional conditioning would allow each word to in- tion (NSP).5 Despite its simplicity, we demondirectly “see itself”, and the model could trivially strate in Section 5.1 that pre-training towards this predict the target word in a multi-layered context. task is very beneficial to both QA and NLI. 6 former is often referred to as a “Transformer encoder” while 5The final model achieves 97%-98% accuracy on NSP. the left-context-only version is referred to as a “Transformer 6The vector C is not a meaningful sentence representation decoder” since it can be used for text generation. without fine-tuning, since it was trained with NSP.
Page 5
<a id="S0069"></a> Source: p.5 S0069
Input [CLS] my dog is cute [SEP] he likes play ##ing [SEP] Token E E E E E E E E E E E Embeddings [CLS] my dog is cute [SEP] he likes play ##ing [SEP] Segment E E E E E E E E E E E Embeddings A A A A A A B B B B B Position E E E E E E E E E E E Embeddings 0 1 2 3 4 5 6 7 8 9 10 Figure 2: BERT input representation.
<a id="S0070"></a> Source: p.5 S0070
The input embeddings are the sum of the token embeddings, the segmentation embeddings and the position embeddings.
<a id="S0071"></a> Source: p.5 S0071
The NSP task is closely related to representation- (4) a degenerate text-∅ pair in text classification learning objectives used in Jernite et al. (2017) and or sequence tagging.
<a id="S0072"></a> Source: p.5 S0072
At the output, the token rep- Logeswaran and Lee (2018).
<a id="S0073"></a> Source: p.5 S0073
However, in prior resentations are fed into an output layer for tokenwork, only sentence embeddings are transferred to level tasks, such as sequence tagging or question down-stream tasks, where BERT transfers all pa- answering, and the [CLS] representation is fed rameters to initialize end-task model parameters. into an output layer for classification, such as entailment or sentiment analysis.
<a id="S0074"></a> Source: p.5 S0074
Pre-training data The pre-training procedure Compared to pre-training, fine-tuning is relalargely follows the existing literature on language tively inexpensive.
<a id="S0075"></a> Source: p.5 S0075
All of the results in the pamodel pre-training.
<a id="S0076"></a> Source: p.5 S0076
For the pre-training corpus we per can be replicated in at most 1 hour on a sinuse the BooksCorpus (800M words) (Zhu et al., gle Cloud TPU, or a few hours on a GPU, starting 2015) and English Wikipedia (2,500M words). from the exact same pre-trained model.7 We de- For Wikipedia we extract only the text passages scribe the task-specific details in the correspondand ignore lists, tables, and headers.
<a id="S0077"></a> Source: p.5 S0077
It is critiing subsections of Section 4.
<a id="S0078"></a> Source: p.5 S0078
More details can be cal to use a document-level corpus rather than a found in Appendix A.5. shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to 4 Experiments extract long contiguous sequences.
<a id="S0079"></a> Source: p.5 S0079
In this section, we present BERT fine-tuning re- 3.2 Fine-tuning BERT sults on 11 NLP tasks.
<a id="S0080"></a> Source: p.5 S0080
Fine-tuning is straightforward since the self- 4.1 GLUE attention mechanism in the Transformer al- The General Language Understanding Evaluation lows BERT to model many downstream tasks— (GLUE) benchmark (Wang et al., 2018a) is a colwhether they involve single text or text pairs—by lection of diverse natural language understanding swapping out the appropriate inputs and outputs. tasks.
<a id="S0081"></a> Source: p.5 S0081
Detailed descriptions of GLUE datasets are For applications involving text pairs, a common included in Appendix B.1. pattern is to independently encode text pairs be- To fine-tune on GLUE, we represent the input fore applying bidirectional cross attention, such sequence (for single sentence or sentence pairs) as Parikh et al. (2016); Seo et al. (2017).
<a id="S0082"></a> Source: p.5 S0082
BERT as described in Section 3, and use the final hidinstead uses the self-attention mechanism to unify den vector C ∈ RH corresponding to the first these two stages, as encoding a concatenated text input token ([CLS]) as the aggregate representapair with self-attention effectively includes bidition.
<a id="S0083"></a> Source: p.5 S0083
The only new parameters introduced during rectional cross attention between two sentences. fine-tuning are classification layer weights W ∈ For each task, we simply plug in the task- RK×H , where K is the number of labels.
<a id="S0084"></a> Source: p.5 S0084
We comspecific inputs and outputs into BERT and finepute a standard classification loss with C and W , tune all the parameters end-to-end.
<a id="S0085"></a> Source: p.5 S0085
At the ini.e., log(softmax(CW T )). put, sentence A and sentence B from pre-training are analogous to (1) sentence pairs in paraphras- 7For example, the BERT SQuAD model can be trained in around 30 minutes on a single Cloud TPU to achieve a Dev ing, (2) hypothesis-premise pairs in entailment, (3) F1 score of 91.0%. question-passage pairs in question answering, and 8See (10) in https://gluebenchmark.com/faq.
Page 6
<a id="S0086"></a> Source: p.6 S0086
System MNLI-(m/mm) QQP QNLI SST-2 CoLA STS-B MRPC RTE Average 392k 363k 108k 67k 8.5k 5.7k 3.5k 2.5k - Pre-OpenAI SOTA 80.6/80.1 66.1 82.3 93.2 35.0 81.0 86.0 61.7 74.0 BiLSTM+ELMo+Attn 76.4/76.1 64.8 79.8 90.4 36.0 73.3 84.9 56.8 71.0 OpenAI GPT 82.1/81.4 70.3 87.4 91.3 45.4 80.0 82.3 56.0 75.1 BERT 84.6/83.4 71.2 90.5 93.5 52.1 85.8 88.9 66.4 79.6 BASE BERT 86.7/85.9 72.1 92.7 94.9 60.5 86.5 89.3 70.1 82.1 LARGE Table 1: GLUE Test results, scored by the evaluation server (https://gluebenchmark.com/leaderboard).
<a id="S0087"></a> Source: p.6 S0087
The number below each task denotes the number of training examples.
<a id="S0088"></a> Source: p.6 S0088
The “Average” column is slightly different than the official GLUE score, since we exclude the problematic WNLI set.8 BERT and OpenAI GPT are singlemodel, single task.
<a id="S0089"></a> Source: p.6 S0089
F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks.
<a id="S0090"></a> Source: p.6 S0090
We exclude entries that use BERT as one of their components.
<a id="S0091"></a> Source: p.6 S0091
We use a batch size of 32 and fine-tune for 3 Wikipedia containing the answer, the task is to epochs over the data for all GLUE tasks.
<a id="S0092"></a> Source: p.6 S0092
For each predict the answer text span in the passage. task, we selected the best fine-tuning learning rate As shown in Figure 1, in the question answer- (among 5e-5, 4e-5, 3e-5, and 2e-5) on the Dev set. ing task, we represent the input question and pas- Additionally, for BERT LARGE we found that fine- sage as a single packed sequence, with the questuning was sometimes unstable on small datasets, tion using the A embedding and the passage using so we ran several random restarts and selected the the B embedding.
<a id="S0093"></a> Source: p.6 S0093
We only introduce a start vecbest model on the Dev set.
<a id="S0094"></a> Source: p.6 S0094
With random restarts, tor S ∈ RH and an end vector E ∈ RH during we use the same pre-trained checkpoint but per- fine-tuning.
<a id="S0095"></a> Source: p.6 S0095
The probability of word i being the form different fine-tuning data shuffling and clas- start of the answer span is computed as a dot prodsifier layer initialization.9 uct between T and S followed by a softmax over i Results are presented in Table 1.
<a id="S0096"></a> Source: p.6 S0096
Both all of the words in the paragraph: P = eS·Ti . i (cid:80) eS·Tj BERT and BERT outperform all sys- j BASE LARGE The analogous formula is used for the end of the tems on all tasks by a substantial margin, obtaining answer span.
<a id="S0097"></a> Source: p.6 S0097
The score of a candidate span from 4.5% and 7.0% respective average accuracy imposition i to position j is defined as S·T + E·T , i j provement over the prior state of the art.
<a id="S0098"></a> Source: p.6 S0098
Note that and the maximum scoring span where j ≥ i is BERT and OpenAI GPT are nearly identical BASE used as a prediction.
<a id="S0099"></a> Source: p.6 S0099
The training objective is the in terms of model architecture apart from the atsum of the log-likelihoods of the correct start and tention masking.
<a id="S0100"></a> Source: p.6 S0100
For the largest and most widely end positions.
<a id="S0101"></a> Source: p.6 S0101
We fine-tune for 3 epochs with a reported GLUE task, MNLI, BERT obtains a 4.6% learning rate of 5e-5 and a batch size of 32. absolute accuracy improvement.
<a id="S0102"></a> Source: p.6 S0102
On the official Table 2 shows top leaderboard entries as well GLUE leaderboard10, BERT obtains a score LARGE as results from top published systems (Seo et al., of 80.5, compared to OpenAI GPT, which obtains 2017; Clark and Gardner, 2018; Peters et al., 72.8 as of the date of writing. 2018a; Hu et al., 2018).
<a id="S0103"></a> Source: p.6 S0103
The top results from the We find that BERT significantly outper- LARGE SQuAD leaderboard do not have up-to-date public forms BERT across all tasks, especially those BASE system descriptions available,11 and are allowed to with very little training data.
<a id="S0104"></a> Source: p.6 S0104
The effect of model use any public data when training their systems. size is explored more thoroughly in Section 5.2.
<a id="S0105"></a> Source: p.6 S0105
We therefore use modest data augmentation in our system by first fine-tuning on TriviaQA (Joshi 4.2 SQuAD v1.1 et al., 2017) befor fine-tuning on SQuAD.
<a id="S0106"></a> Source: p.6 S0106
The Stanford Question Answering Dataset Our best performing system outperforms the top (SQuAD v1.1) is a collection of 100k crowdleaderboard system by +1.5 F1 in ensembling and sourced question/answer pairs (Rajpurkar et al., +1.3 F1 as a single system.
<a id="S0107"></a> Source: p.6 S0107
Given a question and a passage from BERT model outperforms the top ensemble sys- 9The GLUE data set distribution does not include the Test tem in terms of F1 score.
<a id="S0108"></a> Source: p.6 S0108
Without TriviaQA finelabels, and we only made a single GLUE evaluation server submission for each of BERT BASE and BERT LARGE . 11QANet is described in Yu et al. (2018), but the system 10https://gluebenchmark.com/leaderboard has improved substantially after publication.
Page 7
<a id="S0109"></a> Source: p.7 S0109
System Dev Test System Dev Test EM F1 EM F1 ESIM+GloVe 51.9 52.7 Top Leaderboard Systems (Dec 10th, 2018) ESIM+ELMo 59.1 59.2 Human - - 82.3 91.2 OpenAI GPT - 78.0 #1 Ensemble - nlnet - - 86.0 91.7 BERT 81.6 - #2 Ensemble - QANet - - 84.5 90.5 BASE BERT 86.6 86.3 LARGE Published Human (expert)† - 85.0 BiDAF+ELMo (Single) - 85.6 - 85.8 R.M.
<a id="S0110"></a> Source: p.7 S0110
Reader (Ensemble) 81.2 87.9 82.3 88.5 Human (5 annotations)† - 88.0 Ours Table 4: SWAG Dev and Test accuracies. †Human per- BERT (Single) 80.8 88.5 - - BASE BERT (Single) 84.1 90.9 - - formance is measured with 100 samples, as reported in LARGE BERT (Ensemble) 85.8 91.8 - - the SWAG paper.
<a id="S0111"></a> Source: p.7 S0111
LARGE BERT (Sgl.+TriviaQA) 84.2 91.1 85.1 91.8 LARGE BERT (Ens.+TriviaQA) 86.2 92.2 87.4 93.2 LARGE sˆ = max S·T + E·T .
<a id="S0112"></a> Source: p.7 S0112
We predict a non-null i,j j≥i i j Table 2: SQuAD 1.1 results.
<a id="S0113"></a> Source: p.7 S0113
The BERT ensemble answer when sˆ > s + τ , where the threshi,j null is 7x systems which use different pre-training checkold τ is selected on the dev set to maximize F1. points and fine-tuning seeds.
<a id="S0114"></a> Source: p.7 S0114
We did not use TriviaQA data for this model.
<a id="S0115"></a> Source: p.7 S0115
We fine-tuned for 2 epochs with a learning rate of 5e-5 System Dev Test and a batch size of 48.
<a id="S0116"></a> Source: p.7 S0116
EM F1 EM F1 The results compared to prior leaderboard en- Top Leaderboard Systems (Dec 10th, 2018) Human 86.3 89.0 86.9 89.5 tries and top published work (Sun et al., 2018; #1 Single - MIR-MRC (F-Net) - - 74.8 78.0 Wang et al., 2018b) are shown in Table 3, exclud- #2 Single - nlnet - - 74.2 77.1 ing systems that use BERT as one of their com- Published ponents.
<a id="S0117"></a> Source: p.7 S0117
We observe a +5.1 F1 improvement over unet (Ensemble) - - 71.4 74.9 SLQA+ (Single) - 71.4 74.4 the previous best system.
<a id="S0118"></a> Source: p.7 S0118
Ours 4.4 SWAG BERT (Single) 78.7 81.9 80.0 83.1 LARGE The Situations With Adversarial Generations Table 3: SQuAD 2.0 results.
<a id="S0119"></a> Source: p.7 S0119
We exclude entries that (SWAG) dataset contains 113k sentence-pair comuse BERT as one of their components. pletion examples that evaluate grounded commonsense inference (Zellers et al., 2018).
<a id="S0120"></a> Source: p.7 S0120
Given a sentence, the task is to choose the most plausible contuning data, we only lose 0.1-0.4 F1, still outpertinuation among four choices. forming all existing systems by a wide margin.12 When fine-tuning on the SWAG dataset, we 4.3 SQuAD v2.0 construct four input sequences, each containing the concatenation of the given sentence (sentence The SQuAD 2.0 task extends the SQuAD 1.1 A) and a possible continuation (sentence B).
<a id="S0121"></a> Source: p.7 S0121
The problem definition by allowing for the possibility only task-specific parameters introduced is a vecthat no short answer exists in the provided parator whose dot product with the [CLS] token repgraph, making the problem more realistic. resentation C denotes a score for each choice We use a simple approach to extend the SQuAD which is normalized with a softmax layer. v1.1 BERT model for this task.
<a id="S0122"></a> Source: p.7 S0122
We treat ques- We fine-tune the model for 3 epochs with a tions that do not have an answer as having an anlearning rate of 2e-5 and a batch size of 16.
<a id="S0123"></a> Source: p.7 S0123
Reswer span with start and end at the [CLS] tosults are presented in Table 4.
<a id="S0124"></a> Source: p.7 S0124
The probability space for the start and end LARGE performs the authors’ baseline ESIM+ELMo sysanswer span positions is extended to include the tem by +27.1% and OpenAI GPT by 8.3%. position of the [CLS] token.
<a id="S0125"></a> Source: p.7 S0125
For prediction, we compare the score of the no-answer span: s = null 5 Ablation Studies S·C + E·C to the score of the best non-null span In this section, we perform ablation experiments 12The TriviaQA data we used consists of paragraphs from over a number of facets of BERT in order to better TriviaQA-Wiki formed of the first 400 tokens in documents, that contain at least one of the provided possible answers. understand their relative importance.
Page 8
<a id="S0126"></a> Source: p.8 S0126
Dev Set results are still far worse than those of the pre- Tasks MNLI-m QNLI MRPC SST-2 SQuAD trained bidirectional models.
<a id="S0127"></a> Source: p.8 S0127
The BiLSTM hurts (Acc) (Acc) (Acc) (Acc) (F1) performance on the GLUE tasks.
<a id="S0128"></a> Source: p.8 S0128
BERT 84.4 88.4 86.7 92.7 88.5 BASE We recognize that it would also be possible to No NSP 83.9 84.9 86.5 92.6 87.9 LTR & No NSP 82.1 84.3 77.5 92.1 77.8 train separate LTR and RTL models and represent + BiLSTM 82.1 84.1 75.7 91.6 84.9 each token as the concatenation of the two models, as ELMo does.
<a id="S0129"></a> Source: p.8 S0129
However: (a) this is twice as Table 5: Ablation over the pre-training tasks using the expensive as a single bidirectional model; (b) this BERT architecture. “No NSP” is trained without BASE is non-intuitive for tasks like QA, since the RTL the next sentence prediction task. “LTR & No NSP” is trained as a left-to-right LM without the next sentence model would not be able to condition the answer prediction, like OpenAI GPT. “+ BiLSTM” adds a ran- on the question; (c) this it is strictly less powerful domly initialized BiLSTM on top of the “LTR + No than a deep bidirectional model, since it can use NSP” model during fine-tuning. both left and right context at every layer. 5.2 Effect of Model Size ablation studies can be found in Appendix C.
<a id="S0130"></a> Source: p.8 S0130
In this section, we explore the effect of model size 5.1 Effect of Pre-training Tasks on fine-tuning task accuracy.
<a id="S0131"></a> Source: p.8 S0131
We trained a number of BERT models with a differing number of layers, We demonstrate the importance of the deep bidihidden units, and attention heads, while otherwise rectionality of BERT by evaluating two preusing the same hyperparameters and training protraining objectives using exactly the same precedure as described previously. training data, fine-tuning scheme, and hyperpa- Results on selected GLUE tasks are shown in rameters as BERT : BASE Table 6.
<a id="S0132"></a> Source: p.8 S0132
In this table, we report the average Dev No NSP: A bidirectional model which is trained Set accuracy from 5 random restarts of fine-tuning. using the “masked LM” (MLM) but without the We can see that larger models lead to a strict ac- “next sentence prediction” (NSP) task. curacy improvement across all four datasets, even LTR & No NSP: A left-context-only model which for MRPC which only has 3,600 labeled trainis trained using a standard Left-to-Right (LTR) ing examples, and is substantially different from LM, rather than an MLM.
<a id="S0133"></a> Source: p.8 S0133
The left-only constraint the pre-training tasks.
<a id="S0134"></a> Source: p.8 S0134
It is also perhaps surpriswas also applied at fine-tuning, because removing ing that we are able to achieve such significant it introduced a pre-train/fine-tune mismatch that improvements on top of models which are aldegraded downstream performance.
<a id="S0135"></a> Source: p.8 S0135
Additionally, ready quite large relative to the existing literature. this model was pre-trained without the NSP task.
<a id="S0136"></a> Source: p.8 S0136
For example, the largest Transformer explored in This is directly comparable to OpenAI GPT, but Vaswani et al. (2017) is (L=6, H=1024, A=16) using our larger training dataset, our input repre- with 100M parameters for the encoder, and the sentation, and our fine-tuning scheme. largest Transformer we have found in the literature We first examine the impact brought by the NSP is (L=64, H=512, A=2) with 235M parameters task.
<a id="S0137"></a> Source: p.8 S0137
In Table 5, we show that removing NSP (Al-Rfou et al., 2018).
<a id="S0138"></a> Source: p.8 S0138
By contrast, BERT BASE hurts performance significantly on QNLI, MNLI, contains 110M parameters and BERT con- LARGE and SQuAD 1.1.
<a id="S0139"></a> Source: p.8 S0139
Next, we evaluate the impact tains 340M parameters. of training bidirectional representations by com- It has long been known that increasing the paring “No NSP” to “LTR & No NSP”.
<a id="S0140"></a> Source: p.8 S0140
The LTR model size will lead to continual improvements model performs worse than the MLM model on all on large-scale tasks such as machine translation tasks, with large drops on MRPC and SQuAD. and language modeling, which is demonstrated For SQuAD it is intuitively clear that a LTR by the LM perplexity of held-out training data model will perform poorly at token predictions, shown in Table 6.
<a id="S0141"></a> Source: p.8 S0141
However, we believe that since the token-level hidden states have no right- this is the first work to demonstrate convincside context.
<a id="S0142"></a> Source: p.8 S0142
In order to make a good faith at- ingly that scaling to extreme model sizes also tempt at strengthening the LTR system, we added leads to large improvements on very small scale a randomly initialized BiLSTM on top.
<a id="S0143"></a> Source: p.8 S0143
This does tasks, provided that the model has been suffisignificantly improve results on SQuAD, but the ciently pre-trained.
Page 9
<a id="S0144"></a> Source: p.9 S0144
mixed results on the downstream task impact of System Dev F1 Test F1 increasing the pre-trained bi-LM size from two ELMo (Peters et al., 2018a) 95.7 92.2 to four layers and Melamud et al. (2016) men- CVT (Clark et al., 2018) - 92.6 CSE (Akbik et al., 2018) - 93.1 tioned in passing that increasing hidden dimension size from 200 to 600 helped, but increasing Fine-tuning approach BERT 96.6 92.8 further to 1,000 did not bring further improve- LARGE BERT 96.4 92.4 BASE ments.
<a id="S0145"></a> Source: p.9 S0145
Both of these prior works used a feature- Feature-based approach (BERT ) BASE based approach — we hypothesize that when the Embeddings 91.0 model is fine-tuned directly on the downstream Second-to-Last Hidden 95.6 - Last Hidden 94.9 tasks and uses only a very small number of ran- Weighted Sum Last Four Hidden 95.9 domly initialized additional parameters, the task- Concat Last Four Hidden 96.1 specific models can benefit from the larger, more Weighted Sum All 12 Layers 95.5 expressive pre-trained representations even when Table 7: CoNLL-2003 Named Entity Recognition redownstream task data is very small. sults.
<a id="S0146"></a> Source: p.9 S0146
Hyperparameters were selected using the Dev set.
<a id="S0147"></a> Source: p.9 S0147
The reported Dev and Test scores are averaged over 5.3 Feature-based Approach with BERT 5 random restarts using those hyperparameters.
<a id="S0148"></a> Source: p.9 S0148
All of the BERT results presented so far have used the fine-tuning approach, where a simple classification layer is added to the pre-trained model, and layer in the output.
<a id="S0149"></a> Source: p.9 S0149
We use the representation of all parameters are jointly fine-tuned on a down- the first sub-token as the input to the token-level stream task.
<a id="S0150"></a> Source: p.9 S0150
However, the feature-based approach, classifier over the NER label set. where fixed features are extracted from the pre- To ablate the fine-tuning approach, we apply the trained model, has certain advantages.
<a id="S0151"></a> Source: p.9 S0151
First, not feature-based approach by extracting the activaall tasks can be easily represented by a Transtions from one or more layers without fine-tuning former encoder architecture, and therefore require any parameters of BERT.
<a id="S0152"></a> Source: p.9 S0152
These contextual ema task-specific model architecture to be added. beddings are used as input to a randomly initial- Second, there are major computational benefits ized two-layer 768-dimensional BiLSTM before to pre-compute an expensive representation of the the classification layer. training data once and then run many experiments Results are presented in Table 7.
<a id="S0153"></a> Source: p.9 S0153
BERT with cheaper models on top of this representation.
<a id="S0154"></a> Source: p.9 S0154
LARGE performs competitively with state-of-the-art meth- In this section, we compare the two approaches ods.
<a id="S0155"></a> Source: p.9 S0155
The best performing method concatenates the by applying BERT to the CoNLL-2003 Named token representations from the top four hidden lay- Entity Recognition (NER) task (Tjong Kim Sang ers of the pre-trained Transformer, which is only and De Meulder, 2003).
<a id="S0156"></a> Source: p.9 S0156
In the input to BERT, we 0.3 F1 behind fine-tuning the entire model.
<a id="S0157"></a> Source: p.9 S0157
This use a case-preserving WordPiece model, and we demonstrates that BERT is effective for both fineinclude the maximal document context provided tuning and feature-based approaches. by the data.
<a id="S0158"></a> Source: p.9 S0158
Following standard practice, we formulate this as a tagging task but do not use a CRF 6 Conclusion Hyperparams Dev Set Accuracy Recent empirical improvements due to transfer #L #H #A LM (ppl) MNLI-m MRPC SST-2 learning with language models have demonstrated 3 768 12 5.84 77.9 79.8 88.4 6 768 3 5.24 80.6 82.2 90.7 that rich, unsupervised pre-training is an integral 6 768 12 4.68 81.9 84.8 91.3 part of many language understanding systems.
<a id="S0159"></a> Source: p.9 S0159
In 12 768 12 3.99 84.4 86.7 92.9 particular, these results enable even low-resource 12 1024 16 3.54 85.7 86.9 93.3 24 1024 16 3.23 86.6 87.8 93.7 tasks to benefit from deep unidirectional architectures.
<a id="S0160"></a> Source: p.9 S0160
Our major contribution is further general- Table 6: Ablation over BERT model size. #L = the izing these findings to deep bidirectional architecnumber of layers; #H = hidden size; #A = number of at- tures, allowing the same pre-trained model to suctention heads. “LM (ppl)” is the masked LM perplexity cessfully tackle a broad set of NLP tasks. of held-out training data.
Page 10
<a id="S0161"></a> Source: p.10 S0161
References Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018.
<a id="S0162"></a> Source: p.10 S0162
Semi-supervised se- Alan Akbik, Duncan Blythe, and Roland Vollgraf. quence modeling with cross-view training.
<a id="S0163"></a> Source: p.10 S0163
Contextual string embeddings for sequence ceedings of the 2018 Conference on Empirical Methlabeling.
<a id="S0164"></a> Source: p.10 S0164
In Proceedings of the 27th International ods in Natural Language Processing, pages 1914– Conference on Computational Linguistics, pages 1925. 1638–1649.
<a id="S0165"></a> Source: p.10 S0165
Ronan Collobert and Jason Weston. 2008. A unified Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy architecture for natural language processing: Deep Guo, and Llion Jones. 2018.
<a id="S0166"></a> Source: p.10 S0166
Character-level lan- neural networks with multitask learning.
<a id="S0167"></a> Source: p.10 S0167
In Proguage modeling with deeper self-attention. arXiv ceedings of the 25th international conference on preprint arXiv:1808.04444.
<a id="S0168"></a> Source: p.10 S0168
Rie Kubota Ando and Tong Zhang. 2005. A framework Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo¨ıc for learning predictive structures from multiple tasks Barrault, and Antoine Bordes. 2017.
<a id="S0169"></a> Source: p.10 S0169
Journal of Machine Learning learning of universal sentence representations from Research, 6(Nov):1817–1853. natural language inference data.
<a id="S0170"></a> Source: p.10 S0170
In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copen- Luisa Bentivogli, Bernardo Magnini, Ido Dagan, hagen, Denmark.
<a id="S0171"></a> Source: p.10 S0171
Association for Computational Hoa Trang Dang, and Danilo Giampiccolo. 2009.
<a id="S0172"></a> Source: p.10 S0172
The fifth PASCAL recognizing textual entailment challenge.
<a id="S0173"></a> Source: p.10 S0173
In Advances in neural informa- John Blitzer, Ryan McDonald, and Fernando Pereira. tion processing systems, pages 3079–3087. 2006.
<a id="S0174"></a> Source: p.10 S0174
Domain adaptation with structural correspondence learning.
<a id="S0175"></a> Source: p.10 S0175
Feience on empirical methods in natural language pro- Fei. 2009.
<a id="S0176"></a> Source: p.10 S0176
ImageNet: A Large-Scale Hierarchical cessing, pages 120–128.
<a id="S0177"></a> Source: p.10 S0177
Association for Computa- Image Database.
<a id="S0178"></a> Source: p.10 S0178
William B Dolan and Chris Brockett. 2005.
<a id="S0179"></a> Source: p.10 S0179
Bowman, Gabor Angeli, Christopher Potts, cally constructing a corpus of sentential paraphrases. and Christopher D.
<a id="S0180"></a> Source: p.10 S0180
Manning. 2015. A large anno- In Proceedings of the Third International Workshop tated corpus for learning natural language inference. on Paraphrasing (IWP2005).
<a id="S0181"></a> Source: p.10 S0181
Association for Computational Linguistics.
<a id="S0182"></a> Source: p.10 S0182
William Fedus, Ian Goodfellow, and Andrew M Dai. 2018.
<a id="S0183"></a> Source: p.10 S0183
Maskgan: Better text generation via filling in Peter F Brown, Peter V Desouza, Robert L Mercer, the . arXiv preprint arXiv:1801.07736.
<a id="S0184"></a> Source: p.10 S0184
Vincent J Della Pietra, and Jenifer C Lai. 1992.
<a id="S0185"></a> Source: p.10 S0185
Class-based n-gram models of natural language.
<a id="S0186"></a> Source: p.10 S0186
Bridging Computational linguistics, 18(4):467–479. nonlinearities and stochastic regularizers with gaussian error linear units.
<a id="S0187"></a> Source: p.10 S0187
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016.
<a id="S0188"></a> Source: p.10 S0188
Semeval-2017 Learning distributed representations of sentences task 1: Semantic textual similarity multilingual and from unlabelled data.
<a id="S0189"></a> Source: p.10 S0189
In Proceedings of the 2016 crosslingual focused evaluation.
<a id="S0190"></a> Source: p.10 S0190
In Proceedings Conference of the North American Chapter of the of the 11th International Workshop on Semantic Association for Computational Linguistics: Human Evaluation (SemEval-2017), pages 1–14, Vancou- Language Technologies.
<a id="S0191"></a> Source: p.10 S0191
Association for Computational Lintional Linguistics. guistics.
<a id="S0192"></a> Source: p.10 S0192
Jeremy Howard and Sebastian Ruder. 2018.
<a id="S0193"></a> Source: p.10 S0193
Universal Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, language model fine-tuning for text classification.
<a id="S0194"></a> Source: p.10 S0194
In Thorsten Brants, Phillipp Koehn, and Tony Robin- ACL.
<a id="S0195"></a> Source: p.10 S0195
Association for Computational Linguistics. son. 2013.
<a id="S0196"></a> Source: p.10 S0196
One billion word benchmark for measuring progress in statistical language modeling. arXiv Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, preprint arXiv:1312.3005.
<a id="S0197"></a> Source: p.10 S0197
Reinforced mnemonic reader for machine reading comprehen- Z.
<a id="S0198"></a> Source: p.10 S0198
Bowman, and David Son- Christopher Clark and Matt Gardner. 2018.
<a id="S0199"></a> Source: p.10 S0199
Discourse-based objectives for fast unand effective multi-paragraph reading comprehen- supervised sentence representation learning.
Page 11
<a id="S0200"></a> Source: p.11 S0200
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Matthew Peters, Mark Neumann, Luke Zettlemoyer, Zettlemoyer. 2017.
<a id="S0201"></a> Source: p.11 S0201
Triviaqa: A large scale distantly and Wen-tau Yih. 2018b.
<a id="S0202"></a> Source: p.11 S0202
Dissecting contextual supervised challenge dataset for reading comprehen- word embeddings: Architecture and representation. sion.
<a id="S0203"></a> Source: p.11 S0203
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, 1499–1509.
<a id="S0204"></a> Source: p.11 S0204
Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.
<a id="S0205"></a> Source: p.11 S0205
In Alec Radford, Karthik Narasimhan, Tim Salimans, and Advances in neural information processing systems, Ilya Sutskever. 2018.
<a id="S0206"></a> Source: p.11 S0206
Improving language underpages 3294–3302. standing with unsupervised learning.
<a id="S0207"></a> Source: p.11 S0207
Distributed representations of sentences and documents.
<a id="S0208"></a> Source: p.11 S0208
In Inter- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and national Conference on Machine Learning, pages Percy Liang. 2016.
<a id="S0209"></a> Source: p.11 S0209
Squad: 100,000+ questions for 1188–1196. machine comprehension of text.
<a id="S0210"></a> Source: p.11 S0210
In Proceedings of the 2016 Conference on Empirical Methods in Nat- Hector J Levesque, Ernest Davis, and Leora Morgenural Language Processing, pages 2383–2392. stern. 2011.
<a id="S0211"></a> Source: p.11 S0211
In Aaai spring symposium: Logical formalizations of Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and commonsense reasoning, volume 46, page 47.
<a id="S0212"></a> Source: p.11 S0212
Bidirectional attention flow for machine comprehension.
<a id="S0213"></a> Source: p.11 S0213
Lajanugen Logeswaran and Honglak Lee. 2018.
<a id="S0214"></a> Source: p.11 S0214
An efficient framework for learning sentence represen- Richard Socher, Alex Perelygin, Jean Wu, Jason tations.
<a id="S0215"></a> Source: p.11 S0215
In International Conference on Learning Chuang, Christopher D Manning, Andrew Ng, and Representations.
<a id="S0216"></a> Source: p.11 S0216
Recursive deep models for semantic compositionality over a sentiment tree- Bryan McCann, James Bradbury, Caiming Xiong, and bank.
<a id="S0217"></a> Source: p.11 S0217
In Proceedings of the 2013 conference on Richard Socher. 2017.
<a id="S0218"></a> Source: p.11 S0218
Learned in translation: Conempirical methods in natural language processing, textualized word vectors.
<a id="S0219"></a> Source: p.11 S0219
Oren Melamud, Jacob Goldberger, and Ido Dagan.
<a id="S0220"></a> Source: p.11 S0220
Fu Sun, Linyang Li, Xipeng Qiu, and Yang Liu. 2016. context2vec: Learning generic context em- 2018. U-net: Machine reading comprehension bedding with bidirectional LSTM.
<a id="S0221"></a> Source: p.11 S0221
In CoNLL. with unanswerable questions. arXiv preprint Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- arXiv:1810.06638. rado, and Jeff Dean. 2013.
<a id="S0222"></a> Source: p.11 S0222
Distributed representa- Wilson L Taylor. 1953.
<a id="S0223"></a> Source: p.11 S0223
Cloze procedure: A new tions of words and phrases and their compositionaltool for measuring readability.
<a id="S0224"></a> Source: p.11 S0224
In Advances in Neural Information Processing 30(4):415–433.
<a id="S0225"></a> Source: p.11 S0225
Erik F Tjong Kim Sang and Fien De Meulder.
<a id="S0226"></a> Source: p.11 S0226
Andriy Mnih and Geoffrey E Hinton. 2009. A scal- 2003.
<a id="S0227"></a> Source: p.11 S0227
Introduction to the conll-2003 shared task: able hierarchical distributed language model.
<a id="S0228"></a> Source: p.11 S0228
In Language-independent named entity recognition.
<a id="S0229"></a> Source: p.11 S0229
Bot- CoNLL. tou, editors, Advances in Neural Information Pro- Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. cessing Systems 21, pages 1081–1088.
<a id="S0230"></a> Source: p.11 S0230
Curran As- Word representations: A simple and general method sociates, Inc. for semi-supervised learning.
<a id="S0231"></a> Source: p.11 S0231
In Proceedings of the Ankur P Parikh, Oscar Ta¨ckstro¨m, Dipanjan Das, and 48th Annual Meeting of the Association for Compu- Jakob Uszkoreit. 2016. A decomposable attention tational Linguistics, ACL ’10, pages 384–394. model for natural language inference.
<a id="S0232"></a> Source: p.11 S0232
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Jeffrey Pennington, Richard Socher, and Christo- Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz pher D.
<a id="S0233"></a> Source: p.11 S0233
Glove: Global vectors for Kaiser, and Illia Polosukhin. 2017.
<a id="S0234"></a> Source: p.11 S0234
In Advances in Neural Information Proural Language Processing (EMNLP), pages 1532– cessing Systems, pages 6000–6010. 1543.
<a id="S0235"></a> Source: p.11 S0235
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Matthew Peters, Waleed Ammar, Chandra Bhagavat- Pierre-Antoine Manzagol. 2008.
<a id="S0236"></a> Source: p.11 S0236
Extracting and ula, and Russell Power. 2017.
<a id="S0237"></a> Source: p.11 S0237
Semi-supervised se- composing robust features with denoising autoenquence tagging with bidirectional language models. coders.
<a id="S0238"></a> Source: p.11 S0238
In Proceedings of the 25th international In ACL. conference on Machine learning, pages 1096–1103.
<a id="S0239"></a> Source: p.11 S0239
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Alex Wang, Amanpreet Singh, Julian Michael, Fe- Zettlemoyer. 2018a.
<a id="S0240"></a> Source: p.11 S0240
Deep contextualized word rep- lix Hill, Omer Levy, and Samuel Bowman. 2018a. resentations.
<a id="S0241"></a> Source: p.11 S0241
Glue: A multi-task benchmark and analysis platform
Page 12
<a id="S0242"></a> Source: p.12 S0242
In Proceedings • Additional details for our experiments are of the 2018 EMNLP Workshop BlackboxNLP: An- presented in Appendix B; and alyzing and Interpreting Neural Networks for NLP, pages 353–355. • Additional ablation studies are presented in Appendix C.
<a id="S0243"></a> Source: p.12 S0243
Multigranularity hierarchical attention fusion networks We present additional ablation studies for for reading comprehension and question answering.
<a id="S0244"></a> Source: p.12 S0244
BERT including: In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: – Effect of Number of Training Steps; and Long Papers).
<a id="S0245"></a> Source: p.12 S0245
Association for Computational Linguistics. – Ablation for Different Masking Procedures.
<a id="S0246"></a> Source: p.12 S0246
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018.
<a id="S0247"></a> Source: p.12 S0247
Neural network acceptability judg- A Additional Details for BERT ments. arXiv preprint arXiv:1805.12471. A.1 Illustration of the Pre-training Tasks Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus We provide examples of the pre-training tasks in for sentence understanding through inference.
<a id="S0248"></a> Source: p.12 S0248
Masked LM and the Masking Procedure As- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V suming the unlabeled sentence is my dog is Le, Mohammad Norouzi, Wolfgang Macherey, hairy, and during the random masking procedure Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.
<a id="S0249"></a> Source: p.12 S0249
Google’s neural ma- we chose the 4-th token (which corresponding to chine translation system: Bridging the gap between hairy), our masking procedure can be further ilhuman and machine translation. arXiv preprint lustrated by arXiv:1609.08144. • 80% of the time: Replace the word with the Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod [MASK] token, e.g., my dog is hairy → Lipson. 2014.
<a id="S0250"></a> Source: p.12 S0250
How transferable are features in deep neural networks?
<a id="S0251"></a> Source: p.12 S0251
In Advances in neural information my dog is [MASK] processing systems, pages 3320–3328. • 10% of the time: Replace the word with a Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui random word, e.g., my dog is hairy → my Zhao, Kai Chen, Mohammad Norouzi, and Quoc V dog is apple Le. 2018.
<a id="S0252"></a> Source: p.12 S0252
QANet: Combining local convolution with global self-attention for reading comprehen- • 10% of the time: Keep the word unsion.
<a id="S0253"></a> Source: p.12 S0253
In ICLR. changed, e.g., my dog is hairy → my dog Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin is hairy.
<a id="S0254"></a> Source: p.12 S0254
The purpose of this is to bias the Choi. 2018.
<a id="S0255"></a> Source: p.12 S0255
Swag: A large-scale adversarial dataset representation towards the actual observed for grounded commonsense inference.
<a id="S0256"></a> Source: p.12 S0256
In Proceedword. ings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
<a id="S0257"></a> Source: p.12 S0257
The advantage of this procedure is that the Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- Transformer encoder does not know which words dinov, Raquel Urtasun, Antonio Torralba, and Sanja it will be asked to predict or which have been re- Fidler. 2015.
<a id="S0258"></a> Source: p.12 S0258
Aligning books and movies: Towards placed by random words, so it is forced to keep story-like visual explanations by watching movies a distributional contextual representation of evand reading books.
<a id="S0259"></a> Source: p.12 S0259
In Proceedings of the IEEE international conference on computer vision, pages ery input token.
<a id="S0260"></a> Source: p.12 S0260
Additionally, because random 19–27. replacement only occurs for 1.5% of all tokens (i.e., 10% of 15%), this does not seem to harm Appendix for “BERT: Pre-training of the model’s language understanding capability.
<a id="S0261"></a> Source: p.12 S0261
In Deep Bidirectional Transformers for Section C.2, we evaluate the impact this proce- Language Understanding” dure.
<a id="S0262"></a> Source: p.12 S0262
We organize the appendix into three sections: Compared to standard langauge model training, the masked LM only make predictions on 15% of • Additional implementation details for BERT tokens in each batch, which suggests that more are presented in Appendix A; pre-training steps may be required for the model
Page 13
<a id="S0263"></a> Source: p.13 S0263
EN Figure 3: Differences in pre-training model architectures.
<a id="S0264"></a> Source: p.13 S0264
OpenAI GPT uses a left-to-right Transformer.
<a id="S0265"></a> Source: p.13 S0265
ELMo uses the concatenation of independently trained left-to-right and right-toleft LSTMs to generate features for downstream tasks.
<a id="S0266"></a> Source: p.13 S0266
Among the three, only BERT representations are jointly conditioned on both left and right context in all layers.
<a id="S0267"></a> Source: p.13 S0267
In addition to the architecture differences, BERT and OpenAI GPT are fine-tuning approaches, while ELMo is a feature-based approach. to converge.
<a id="S0268"></a> Source: p.13 S0268
In Section C.1 we demonstrate that epochs over the 3.3 billion word corpus.
<a id="S0269"></a> Source: p.13 S0269
We MLM does converge marginally slower than a left- use Adam with learning rate of 1e-4, β = 0.9, 1 to-right model (which predicts every token), but β = 0.999, L2 weight decay of 0.01, learning 2 the empirical improvements of the MLM model rate warmup over the first 10,000 steps, and linear far outweigh the increased training cost. decay of the learning rate.
<a id="S0270"></a> Source: p.13 S0270
We use a dropout probability of 0.1 on all layers.
<a id="S0271"></a> Source: p.13 S0271
We use a gelu acti- Next Sentence Prediction The next sentence vation (Hendrycks and Gimpel, 2016) rather than prediction task can be illustrated in the following the standard relu, following OpenAI GPT.
<a id="S0272"></a> Source: p.13 S0272
The examples. training loss is the sum of the mean masked LM likelihood and the mean next sentence prediction Input = [CLS] the man went to [MASK] store [SEP] likelihood. he bought a gallon [MASK] milk [SEP] Training of BERT was performed on 4 BASE Label = IsNext Cloud TPUs in Pod configuration (16 TPU chips total).13 Training of BERT was performed LARGE on 16 Cloud TPUs (64 TPU chips total).
<a id="S0273"></a> Source: p.13 S0273
Each pre- Input = [CLS] the man [MASK] to the store [SEP] training took 4 days to complete. penguin [MASK] are flight ##less birds [SEP] Longer sequences are disproportionately expen- Label = NotNext sive because attention is quadratic to the sequence length.
<a id="S0274"></a> Source: p.13 S0274
To speed up pretraing in our experiments, A.2 Pre-training Procedure we pre-train the model with sequence length of To generate each training input sequence, we sam- 128 for 90% of the steps.
<a id="S0275"></a> Source: p.13 S0275
Then, we train the rest ple two spans of text from the corpus, which we 10% of the steps of sequence of 512 to learn the refer to as “sentences” even though they are typ- positional embeddings. ically much longer than single sentences (but can be shorter also).
<a id="S0276"></a> Source: p.13 S0276
The first sentence receives the A A.3 Fine-tuning Procedure embedding and the second receives the B embed- For fine-tuning, most model hyperparameters are ding. 50% of the time B is the actual next sentence the same as in pre-training, with the exception of that follows A and 50% of the time it is a random the batch size, learning rate, and number of trainsentence, which is done for the “next sentence pre- ing epochs.
<a id="S0277"></a> Source: p.13 S0277
The dropout probability was always diction” task.
<a id="S0278"></a> Source: p.13 S0278
They are sampled such that the com- kept at 0.1.
<a id="S0279"></a> Source: p.13 S0279
The optimal hyperparameter values bined length is ≤ 512 tokens.
<a id="S0280"></a> Source: p.13 S0280
The LM masking is are task-specific, but we found the following range applied after WordPiece tokenization with a uni- of possible values to work well across all tasks: form masking rate of 15%, and no special consideration given to partial word pieces. • Batch size: 16, 32 We train with batch size of 256 sequences (256 13https://cloudplatform.googleblog.com/2018/06/Cloudsequences * 512 tokens = 128,000 tokens/batch) TPU-now-offers-preemptible-pricing-and-globalfor 1,000,000 steps, which is approximately 40 availability.html
Page 14
<a id="S0281"></a> Source: p.14 S0281
• Learning rate (Adam): 5e-5, 3e-5, 2e-5 To isolate the effect of these differences, we per- • Number of epochs: 2, 3, 4 form ablation experiments in Section 5.1 which demonstrate that the majority of the improvements We also observed that large data sets (e.g., are in fact coming from the two pre-training tasks 100k+ labeled training examples) were far less and the bidirectionality they enable. sensitive to hyperparameter choice than small data sets.
<a id="S0282"></a> Source: p.14 S0282
Fine-tuning is typically very fast, so it is rea- A.5 Illustrations of Fine-tuning on Different sonable to simply run an exhaustive search over Tasks the above parameters and choose the model that The illustration of fine-tuning BERT on different performs best on the development set. tasks can be seen in Figure 4.
<a id="S0283"></a> Source: p.14 S0283
Our task-specific models are formed by incorporating BERT with A.4 Comparison of BERT, ELMo ,and one additional output layer, so a minimal num- OpenAI GPT ber of parameters need to be learned from scratch.
<a id="S0284"></a> Source: p.14 S0284
Here we studies the differences in recent popular Among the tasks, (a) and (b) are sequence-level representation learning models including ELMo, tasks while (c) and (d) are token-level tasks.
<a id="S0285"></a> Source: p.14 S0285
The comparisons be- the figure, E represents the input embedding, T i tween the model architectures are shown visually represents the contextual representation of token i, in Figure 3.
<a id="S0286"></a> Source: p.14 S0286
Note that in addition to the architec- [CLS] is the special symbol for classification outture differences, BERT and OpenAI GPT are fine- put, and [SEP] is the special symbol to separate tuning approaches, while ELMo is a feature-based non-consecutive token sequences. approach. B Detailed Experimental Setup The most comparable existing pre-training method to BERT is OpenAI GPT, which trains a B.1 Detailed Descriptions for the GLUE left-to-right Transformer LM on a large text cor- Benchmark Experiments. pus.
<a id="S0287"></a> Source: p.14 S0287
In fact, many of the design decisions in BERT Our GLUE results in Table1 are obtained were intentionally made to make it as close to from https://gluebenchmark.com/ GPT as possible so that the two methods could be leaderboard and https://blog. minimally compared.
<a id="S0288"></a> Source: p.14 S0288
The core argument of this openai.com/language-unsupervised. work is that the bi-directionality and the two pre- The GLUE benchmark includes the following training tasks presented in Section 3.1 account for datasets, the descriptions of which were originally the majority of the empirical improvements, but summarized in Wang et al. (2018a): we do note that there are several other differences between how BERT and GPT were trained: MNLI Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classifi- • GPT is trained on the BooksCorpus (800M cation task (Williams et al., 2018).
<a id="S0289"></a> Source: p.14 S0289
Given a pair of words); BERT is trained on the BooksCor- sentences, the goal is to predict whether the secpus (800M words) and Wikipedia (2,500M ond sentence is an entailment, contradiction, or words). neutral with respect to the first one. • GPT uses a sentence separator ([SEP]) and QQP Quora Question Pairs is a binary classificlassifier token ([CLS]) which are only in- cation task where the goal is to determine if two troduced at fine-tuning time; BERT learns questions asked on Quora are semantically equiv- [SEP], [CLS] and sentence A/B embed- alent (Chen et al., 2018). dings during pre-training.
<a id="S0290"></a> Source: p.14 S0290
QNLI Question Natural Language Inference is a version of the Stanford Question Answering • GPT was trained for 1M steps with a batch Dataset (Rajpurkar et al., 2016) which has been size of 32,000 words; BERT was trained for converted to a binary classification task (Wang 1M steps with a batch size of 128,000 words. et al., 2018a).
<a id="S0291"></a> Source: p.14 S0291
The positive examples are (ques- • GPT used the same learning rate of 5e-5 for tion, sentence) pairs which do contain the correct all fine-tuning experiments; BERT chooses a answer, and the negative examples are (question, task-specific fine-tuning learning rate which sentence) from the same paragraph which do not performs the best on the development set. contain the answer.
Page 15
<a id="S0292"></a> Source: p.15 S0292
Class Class Label Label C T 1 ... T N T [SEP] T 1 ’ ... T M ’ C T 1 T 2 ... T N BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E [CLS] E 1 E 2 ... E N [CLS] T 1 ok ... T N ok [SEP] T 1 ok ... T M ok CCLLSS TTookk 11 Tok 2 ...
<a id="S0293"></a> Source: p.15 S0293
Tok N Sentence 1 Sentence 2 Single Sentence Start/End Span O B-PER ... O C T 1 ... T N T [SEP] T 1 ’ ... T M ’ C T 1 T 2 ... T N BERT BERT E[CLS] E 1 ... E N E [SEP] E 1 ’ ... E M ’ E [CLS] E 1 E 2 ... E N [CLS] T 1 ok ... T N ok [SEP] T 1 ok ... T M ok [CLS] Tok 1 Tok 2 ...
<a id="S0294"></a> Source: p.15 S0294
Tok N Question Paragraph Single Sentence Figure 4: Illustrations of Fine-tuning BERT on Different Tasks.
<a id="S0295"></a> Source: p.15 S0295
SST-2 The Stanford Sentiment Treebank is a for whether the sentences in the pair are semantibinary single-sentence classification task consist- cally equivalent (Dolan and Brockett, 2005). ing of sentences extracted from movie reviews RTE Recognizing Textual Entailment is a biwith human annotations of their sentiment (Socher nary entailment task similar to MNLI, but with et al., 2013). much less training data (Bentivogli et al., 2009).14 CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where WNLI Winograd NLI is a small natural lanthe goal is to predict whether an English sentence guage inference dataset (Levesque et al., 2011). is linguistically “acceptable” or not (Warstadt The GLUE webpage notes that there are issues et al., 2018). with the construction of this dataset, 15 and every trained system that’s been submitted to GLUE has STS-B The Semantic Textual Similarity Benchperformed worse than the 65.1 baseline accuracy mark is a collection of sentence pairs drawn from of predicting the majority class.
<a id="S0296"></a> Source: p.15 S0296
We therefore exnews headlines and other sources (Cer et al., clude this set to be fair to OpenAI GPT.
<a id="S0297"></a> Source: p.15 S0297
They were annotated with a score from 1 GLUE submission, we always predicted the mato 5 denoting how similar the two sentences are in terms of semantic meaning. 14Note that we only report single-task fine-tuning results in this paper. A multitask fine-tuning approach could poten- MRPC Microsoft Research Paraphrase Corpus tially push the performance even further.
<a id="S0298"></a> Source: p.15 S0298
For example, we did observe substantial improvements on RTE from multiconsists of sentence pairs automatically extracted task training with MNLI. from online news sources, with human annotations 15https://gluebenchmark.com/faq
Page 16
<a id="S0299"></a> Source: p.16 S0299
jority class. C Additional Ablation Studies C.1 Effect of Number of Training Steps Figure 5 presents MNLI Dev accuracy after finetuning from a checkpoint that has been pre-trained for k steps.
<a id="S0300"></a> Source: p.16 S0300
This allows us to answer the following questions: 1.
<a id="S0301"></a> Source: p.16 S0301
Question: Does BERT really need such a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?
<a id="S0302"></a> Source: p.16 S0302
Answer: Yes, BERT achieves almost BASE 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps. 2.
<a id="S0303"></a> Source: p.16 S0303
Question: Does MLM pre-training converge slower than LTR pre-training, since only 15% of words are predicted in each batch rather than every word?
<a id="S0304"></a> Source: p.16 S0304
Answer: The MLM model does converge slightly slower than the LTR model.
<a id="S0305"></a> Source: p.16 S0305
However, in terms of absolute accuracy the MLM model begins to outperform the LTR model almost immediately. C.2 Ablation for Different Masking Procedures In Section 3.1, we mention that BERT uses a mixed strategy for masking the target tokens when pre-training with the masked language model (MLM) objective.
<a id="S0306"></a> Source: p.16 S0306
The following is an ablation study to evaluate the effect of different masking strategies. 84 82 80 78 76 200 400 600 800 1,000 Pre-training Steps (Thousands) ycaruccAveDILNM Note that the purpose of the masking strategies is to reduce the mismatch between pre-training and fine-tuning, as the [MASK] symbol never appears during the fine-tuning stage.
<a id="S0307"></a> Source: p.16 S0307
We report the Dev results for both MNLI and NER.
<a id="S0308"></a> Source: p.16 S0308
For NER, we report both fine-tuning and feature-based approaches, as we expect the mismatch will be amplified for the feature-based approach as the model will not have the chance to adjust the representations.
<a id="S0309"></a> Source: p.16 S0309
Masking Rates Dev Set Results MASK SAME RND MNLI NER Fine-tune Fine-tune Feature-based 80% 10% 10% 84.2 95.4 94.9 100% 0% 0% 84.3 94.9 94.0 80% 0% 20% 84.1 95.2 94.6 80% 20% 0% 84.4 95.2 94.7 0% 20% 80% 83.7 94.8 94.6 0% 0% 100% 83.6 94.9 94.6 Table 8: Ablation over different masking strategies.
<a id="S0310"></a> Source: p.16 S0310
In the table, MASK means that we replace the target token with the [MASK] symbol for MLM; SAME means that we keep the target token as is; RND means that we replace the target token with another random token.
<a id="S0311"></a> Source: p.16 S0311
The numbers in the left part of the table represent the probabilities of the specific strategies used during MLM pre-training (BERT uses 80%, 10%, 10%).
<a id="S0312"></a> Source: p.16 S0312
The right part of the paper represents the Dev set results.
<a id="S0313"></a> Source: p.16 S0313
For the feature-based approach, we concatenate the last 4 layers of BERT as the features, which was shown to be the best approach in Section 5.3.
<a id="S0314"></a> Source: p.16 S0314
From the table it can be seen that fine-tuning is surprisingly robust to different masking strategies.
<a id="S0315"></a> Source: p.16 S0315
However, as expected, using only the MASK strategy was problematic when applying the featurebased approach to NER.
<a id="S0316"></a> Source: p.16 S0316
Interestingly, using only the RND strategy performs much worse than our strategy as well.
<a id="S0317"></a> Source: p.16 S0317
BERTBASE (Masked LM) BERTBASE (Left-to-Right) Figure 5: Ablation over number of training steps.
<a id="S0318"></a> Source: p.16 S0318
This shows the MNLI accuracy after fine-tuning, starting from model parameters that have been pre-trained for k steps.