LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen
--- title: "LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen" aliases: - "LoRA" - "arXiv:2106.09685" source: "https://arxiv.org/abs/2106.09685" arxiv: "2106.09685" created: 2026-07-16 type: paper-translation status: extraction-complete_translation-pending tags: - paper - ml - deep-learning - nlp
LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen
原文全文
Page 1
<a id="S0001"></a> Source: p.1 S0001
LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen Microsoft Corporation {edwardhu, yeshe, phwallis, zeyuana, yuanzhil, swang, luw, wzchen}@microsoft.com yuanzhil@andrew.cmu.edu (Version 2) ABSTRACT An important paradigm of natural language processing consists of large-scale pretraining on general domain data and adaptation to particular tasks or domains.
<a id="S0002"></a> Source: p.1 S0002
As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible.
<a id="S0003"></a> Source: p.1 S0003
Using GPT-3 175B as an example – deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive.
<a id="S0004"></a> Source: p.1 S0004
We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
<a id="S0005"></a> Source: p.1 S0005
Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.
<a id="S0006"></a> Source: p.1 S0006
LoRA performs on-par or better than finetuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency.
<a id="S0007"></a> Source: p.1 S0007
We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA.
<a id="S0008"></a> Source: p.1 S0008
We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA. 1 INTRODUCTION Many applications in natural language processing rely on adaptf(x) ing one large-scale, pre-trained language model to multiple down- h stream applications.
<a id="S0009"></a> Source: p.1 S0009
Such adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model.
<a id="S0010"></a> Source: p.1 S0010
The ma- Pretrained jor downside of fine-tuning is that the new model contains as many Pretrained 𝐵 = 0 Weights parameters as in the original model.
<a id="S0011"></a> Source: p.1 S0011
As larger models are trained Weights 𝑟 every few months, this changes from a mere “inconvenience” for 𝑊 ∈ ℝ𝑑×𝑑 𝑊 ∈ ℝ𝑑×𝑑 GPT-2 (Radford et al., b) or RoBERTa large (Liu et al., 2019) to a 𝐴 = 𝒩(0, 𝜎2) critical deployment challenge for GPT-3 (Brown et al., 2020) with 𝑑 𝑑 175 billion trainable parameters.1 x x Many sought to mitigate this by adapting only some parameters or Figure 1: Our reparametrizalearning external modules for new tasks.
<a id="S0012"></a> Source: p.1 S0012
We only train A and B. to store and load a small number of task-specific parameters in addition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed.
<a id="S0013"></a> Source: p.1 S0013
However, existing techniques ∗Equal contribution. 0Compared to V1, this draft includes better baselines, experiments on GLUE, and more on adapter latency. 1While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its performance significantly as shown in Appendix A. 1 1202 tcO 61 ]LC.sc[ 2v58690.6012:viXra
Page 2
<a id="S0014"></a> Source: p.2 S0014
often introduce inference latency (Houlsby et al., 2019; Rebuffi et al., 2017) by extending model depth or reduce the model’s usable sequence length (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021) (Section 3).
<a id="S0015"></a> Source: p.2 S0015
More importantly, these method often fail to match the fine-tuning baselines, posing a trade-off between efficiency and model quality.
<a id="S0016"></a> Source: p.2 S0016
We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.
<a id="S0017"></a> Source: p.2 S0017
We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach.
<a id="S0018"></a> Source: p.2 S0018
LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.
<a id="S0019"></a> Source: p.2 S0019
Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient.
<a id="S0020"></a> Source: p.2 S0020
LoRA possesses several key advantages. • A pre-trained model can be shared and used to build many small LoRA modules for different tasks.
<a id="S0021"></a> Source: p.2 S0021
We can freeze the shared model and efficiently switch tasks by replacing the matrices A and B in Figure 1, reducing the storage requirement and task-switching overhead significantly. • LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters.
<a id="S0022"></a> Source: p.2 S0022
Instead, we only optimize the injected, much smaller low-rank matrices. • Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction. • LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning.
<a id="S0023"></a> Source: p.2 S0023
Terminologies and Conventions We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions.
<a id="S0024"></a> Source: p.2 S0024
We call the input and output dimension size of a Transformer layer d .
<a id="S0025"></a> Source: p.2 S0025
We use W , W , W , and W to refer to the model q k v o query/key/value/output projection matrices in the self-attention module. W or W refers to a pre- 0 trained weight matrix and ∆W its accumulated gradient update during adaptation.
<a id="S0026"></a> Source: p.2 S0026
We use r to denote the rank of a LoRA module.
<a id="S0027"></a> Source: p.2 S0027
We follow the conventions set out by (Vaswani et al., 2017; Brown et al., 2020) and use Adam (Loshchilov & Hutter, 2019; Kingma & Ba, 2017) for model optimization and use a Transformer MLP feedforward dimension d = 4 × d . ffn model 2 PROBLEM STATEMENT While our proposal is agnostic to training objective, we focus on language modeling as our motivating use case.
<a id="S0028"></a> Source: p.2 S0028
Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-specific prompt.
<a id="S0029"></a> Source: p.2 S0029
Suppose we are given a pre-trained autoregressive language model P (y|x) parametrized by Φ. Φ For instance, P (y|x) can be a generic multi-task learner such as GPT (Radford et al., b; Brown Φ et al., 2020) based on the Transformer architecture (Vaswani et al., 2017).
<a id="S0030"></a> Source: p.2 S0030
Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL).
<a id="S0031"></a> Source: p.2 S0031
Each downstream task is represented by a training dataset of context-target pairs: Z = {(x , y )} , where both x and i i i=1,..,N i y are sequences of tokens.
<a id="S0032"></a> Source: p.2 S0032
For example, in NL2SQL, x is a natural language query and y its i i i corresponding SQL command; for summarization, x is the content of an article and y its summary. i i 2
Page 3
<a id="S0033"></a> Source: p.3 S0033
During full fine-tuning, the model is initialized to pre-trained weights Φ and updated to Φ + ∆Φ 0 0 by repeatedly following the gradient to maximize the conditional language modeling objective: |y| (cid:88) (cid:88) max log (P (y |x, y )) (1) Φ t <t Φ (x,y)∈Z t=1 One of the main drawbacks for full fine-tuning is that for each downstream task, we learn a different set of parameters ∆Φ whose dimension |∆Φ| equals |Φ |.
<a id="S0034"></a> Source: p.3 S0034
Thus, if the pre-trained model is large 0 (such as GPT-3 with |Φ | ≈ 175 Billion), storing and deploying many independent instances of 0 fine-tuned models can be challenging, if at all feasible.
<a id="S0035"></a> Source: p.3 S0035
In this paper, we adopt a more parameter-efficient approach, where the task-specific parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with |Θ| (cid:28) |Φ |.
<a id="S0036"></a> Source: p.3 S0036
The task of finding ∆Φ thus becomes optimizing over Θ: 0 |y| (cid:88) (cid:88) (cid:0) (cid:1) max log p (y |x, y ) (2) Θ Φ0+∆Φ(Θ) t <t (x,y)∈Z t=1 In the subsequent sections, we propose to use a low-rank representation to encode ∆Φ that is both compute- and memory-efficient.
<a id="S0037"></a> Source: p.3 S0037
When the pre-trained model is GPT-3 175B, the number of trainable parameters |Θ| can be as small as 0.01% of |Φ |. 0 3 AREN’T EXISTING SOLUTIONS GOOD ENOUGH?
<a id="S0038"></a> Source: p.3 S0038
The problem we set out to tackle is by no means new.
<a id="S0039"></a> Source: p.3 S0039
Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efficient.
<a id="S0040"></a> Source: p.3 S0040
See Section 6 for a survey of some of the well-known works.
<a id="S0041"></a> Source: p.3 S0041
Using language modeling as an example, there are two prominent strategies when it comes to efficient adaptations: adding adapter layers (Houlsby et al., 2019; Rebuffi et al., 2017; Pfeiffer et al., 2021; Ru¨ckle´ et al., 2020) or optimizing some forms of the input layer activations (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021).
<a id="S0042"></a> Source: p.3 S0042
However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.
<a id="S0043"></a> Source: p.3 S0043
Adapter Layers Introduce Inference Latency There are many variants of adapters.
<a id="S0044"></a> Source: p.3 S0044
We focus on the original design by Houlsby et al. (2019) which has two adapter layers per Transformer block and a more recent one by Lin et al. (2020) which has only one per block but with an additional LayerNorm (Ba et al., 2016).
<a id="S0045"></a> Source: p.3 S0045
While one can reduce the overall latency by pruning layers or exploiting multi-task settings (Ru¨ckle´ et al., 2020; Pfeiffer et al., 2021), there is no direct ways to bypass the extra compute in adapter layers.
<a id="S0046"></a> Source: p.3 S0046
This seems like a non-issue since adapter layers are designed to have few parameters (sometimes <1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add.
<a id="S0047"></a> Source: p.3 S0047
However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially.
<a id="S0048"></a> Source: p.3 S0048
This makes a difference in the online inference setting where the batch size is typically as small as one.
<a id="S0049"></a> Source: p.3 S0049
In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).
<a id="S0050"></a> Source: p.3 S0050
This problem gets worse when we need to shard the model as done in Shoeybi et al. (2020); Lepikhin et al. (2020), because the additional depth requires more synchronous GPU operations such as AllReduce and Broadcast, unless we store the adapter parameters redundantly many times.
<a id="S0051"></a> Source: p.3 S0051
Directly Optimizing the Prompt is Hard The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge.
<a id="S0052"></a> Source: p.3 S0052
We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper.
<a id="S0053"></a> Source: p.3 S0053
More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods.
<a id="S0054"></a> Source: p.3 S0054
We defer the study on task performance to Section 5. 3
Page 4
<a id="S0055"></a> Source: p.4 S0055
Batch Size 32 16 1 Sequence Length 512 256 128 |Θ| 0.5M 11M 11M Fine-Tune/LoRA 1449.4±0.8 338.0±0.6 19.8±2.7 AdapterL 1482.0±1.0 (+2.2%) 354.8±0.5 (+5.0%) 23.9±2.1 (+20.7%) AdapterH 1492.2±1.0 (+3.0%) 366.3±0.5 (+8.4%) 25.8±2.2 (+30.3%) Table 1: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, averaged over 100 trials.
<a id="S0056"></a> Source: p.4 S0056
We use an NVIDIA Quadro RTX8000. “|Θ|” denotes the number of trainable parameters in adapter layers.
<a id="S0057"></a> Source: p.4 S0057
AdapterL and AdapterH are two variants of adapter tuning, which we describe in Section 5.1.
<a id="S0058"></a> Source: p.4 S0058
The inference latency introduced by adapter layers can be significant in an online, short-sequence-length scenario.
<a id="S0059"></a> Source: p.4 S0059
See the full study in Appendix B. 4 OUR METHOD We describe the simple design of LoRA and its practical benefits.
<a id="S0060"></a> Source: p.4 S0060
The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case. 4.1 LOW-RANK-PARAMETRIZED UPDATE MATRICES A neural network contains many dense layers which perform matrix multiplication.
<a id="S0061"></a> Source: p.4 S0061
The weight matrices in these layers typically have full-rank.
<a id="S0062"></a> Source: p.4 S0062
When adapting to a specific task, Aghajanyan et al. (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace.
<a id="S0063"></a> Source: p.4 S0063
Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation.
<a id="S0064"></a> Source: p.4 S0064
For a pre-trained weight matrix W ∈ Rd×k, we constrain its update by representing the latter with a low-rank de- 0 composition W + ∆W = W + BA, where B ∈ Rd×r, A ∈ Rr×k, and the rank r (cid:28) min(d, k). 0 0 During training, W is frozen and does not receive gradient updates, while A and B contain trainable 0 parameters.
<a id="S0065"></a> Source: p.4 S0065
Note both W and ∆W = BA are multiplied with the same input, and their respective 0 output vectors are summed coordinate-wise.
<a id="S0066"></a> Source: p.4 S0066
For h = W x, our modified forward pass yields: 0 h = W x + ∆W x = W x + BAx (3) 0 0 We illustrate our reparametrization in Figure 1.
<a id="S0067"></a> Source: p.4 S0067
We use a random Gaussian initialization for A and zero for B, so ∆W = BA is zero at the beginning of training.
<a id="S0068"></a> Source: p.4 S0068
We then scale ∆W x by α , where α r is a constant in r.
<a id="S0069"></a> Source: p.4 S0069
When optimizing with Adam, tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately.
<a id="S0070"></a> Source: p.4 S0070
As a result, we simply set α to the first r we try and do not tune it.
<a id="S0071"></a> Source: p.4 S0071
This scaling helps to reduce the need to retune hyperparameters when we vary r (Yang & Hu, 2021). A Generalization of Full Fine-tuning. A more general form of fine-tuning allows the training of a subset of the pre-trained parameters.
<a id="S0072"></a> Source: p.4 S0072
LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation.
<a id="S0073"></a> Source: p.4 S0073
This means that when applying LoRA to all weight matrices and training all biases2, we roughly recover the expressiveness of full fine-tuning by setting the LoRA rank r to the rank of the pre-trained weight matrices.
<a id="S0074"></a> Source: p.4 S0074
In other words, as we increase the number of trainable parameters 3, training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and prefix-based methods to a model that cannot take long input sequences.
<a id="S0075"></a> Source: p.4 S0075
When deployed in production, we can explicitly compute and store W = W + BA and perform inference as usual.
<a id="S0076"></a> Source: p.4 S0076
Note that both W and BA are in Rd×k. 0 0 When we need to switch to another downstream task, we can recover W by subtracting BA and 0 then adding a different B(cid:48)A(cid:48), a quick operation with very little memory overhead.
<a id="S0077"></a> Source: p.4 S0077
Critically, this 2They represent a negligible number of parameters compared to weights. 3An inevitability when adapting to hard tasks. 4
Page 5
<a id="S0078"></a> Source: p.5 S0078
guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction. 4.2 APPLYING LORA TO TRANSFORMER In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters.
<a id="S0079"></a> Source: p.5 S0079
In the Transformer architecture, there are four weight matrices in the self-attention module (W , W , W , W ) and two in the MLP module.
<a id="S0080"></a> Source: p.5 S0080
We treat W (or W , W ) q k v o q k v as a single matrix of dimension d × d , even though the output dimension is usually sliced model model into attention heads.
<a id="S0081"></a> Source: p.5 S0081
We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency.We further study the effect on adapting different types of attention weight matrices in a Transformer in Section 7.1.
<a id="S0082"></a> Source: p.5 S0082
We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.
<a id="S0083"></a> Source: p.5 S0083
The most significant benefit comes from the reduction in memory and storage usage.
<a id="S0084"></a> Source: p.5 S0084
For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3 if r (cid:28) d as we do not need to store the optimizer states for the frozen model parameters.
<a id="S0085"></a> Source: p.5 S0085
On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB.
<a id="S0086"></a> Source: p.5 S0086
With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000× (from 350GB to 35MB)4.
<a id="S0087"></a> Source: p.5 S0087
This allows us to train with significantly fewer GPUs and avoid I/O bottlenecks.
<a id="S0088"></a> Source: p.5 S0088
Another benefit is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters.
<a id="S0089"></a> Source: p.5 S0089
This allows for the creation of many customized models that can be swapped in and out on the fly on machines that store the pre-trained weights in VRAM.
<a id="S0090"></a> Source: p.5 S0090
We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning5 as we do not need to calculate the gradient for the vast majority of the parameters.
<a id="S0091"></a> Source: p.5 S0091
For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency.
<a id="S0092"></a> Source: p.5 S0092
Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical. 5 EMPIRICAL EXPERIMENTS We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al., 2019), De- BERTa (He et al., 2021), and GPT-2 (Radford et al., b), before scaling up to GPT-3 175B (Brown et al., 2020).
<a id="S0093"></a> Source: p.5 S0093
Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG).
<a id="S0094"></a> Source: p.5 S0094
Specifically, we evaluate on the GLUE (Wang et al., 2019) benchmark for RoBERTa and DeBERTa.
<a id="S0095"></a> Source: p.5 S0095
We follow the setup of Li & Liang (2021) on GPT-2 for a direct comparison and add WikiSQL (Zhong et al., 2017) (NL to SQL queries) and SAMSum (Gliwa et al., 2019) (conversation summarization) for large-scale experiments on GPT-3.
<a id="S0096"></a> Source: p.5 S0096
See Appendix C for more details on the datasets we use.
<a id="S0097"></a> Source: p.5 S0097
We use NVIDIA Tesla V100 for all experiments. 5.1 BASELINES To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible.
<a id="S0098"></a> Source: p.5 S0098
This, however, means that some baselines might only appear in certain experiments.
<a id="S0099"></a> Source: p.5 S0099
Fine-Tuning (FT) is a common approach for adaptation.
<a id="S0100"></a> Source: p.5 S0100
During fine-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.A simple variant is to update only some layers while freezing others.
<a id="S0101"></a> Source: p.5 S0101
We include one such baseline reported in prior work (Li & Liang, 2021) on GPT-2, which adapts just the last two layers (FTTop2). 4We still need the 350GB model during deployment; however, storing 100 adapted models only requires 350GB + 35MB 100 ≈ 354GB as opposed to 100 350GB ≈ 35TB. 5For GPT-3 175B, the training throughput for full fine-tuning is 32.5 tokens/s per V100 GPU; with the same number of weight shards for model parallelism, the throughput is 43.1 tokens/s per V100 GPU for LoRA. 5
Page 6
<a id="S0102"></a> Source: p.6 S0102
Model & Method # Trainable Parameters MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Avg.
<a id="S0103"></a> Source: p.6 S0103
RoB (FT) 125.0M 87.6 94.8 90.2 63.6 92.8 91.9 78.7 91.2 86.4 base RoB (BitFit) 0.1M 84.7 93.7 92.7 62.0 91.8 84.0 81.5 90.8 85.2 base RoB (AdptD) 0.3M 87.1 94.2 88.5 60.8 93.1 90.2 71.5 89.7 84.4 base ±.0 ±.1 ±1.1 ±.4 ±.1 ±.0 ±2.7 ±.3 RoB (AdptD) 0.9M 87.3 94.7 88.4 62.6 93.0 90.6 75.9 90.3 85.4 base ±.1 ±.3 ±.1 ±.9 ±.2 ±.0 ±2.2 ±.1 RoB (LoRA) 0.3M 87.5 95.1 89.7 63.4 93.3 90.8 86.6 91.5 87.2 base ±.3 ±.2 ±.7 ±1.2 ±.3 ±.1 ±.7 ±.2 RoB (FT) 355.0M 90.2 96.4 90.9 68.0 94.7 92.2 86.6 92.4 88.9 large RoB (LoRA) 0.8M 90.6 96.2 90.9 68.2 94.9 91.6 87.4 92.6 89.0 large ±.2 ±.5 ±1.2 ±1.9 ±.3 ±.1 ±2.5 ±.2 RoB (AdptP)† 3.0M 90.2 96.1 90.2 68.3 94.8 91.9 83.8 92.1 88.4 large ±.3 ±.3 ±.7 ±1.0 ±.2 ±.1 ±2.9 ±.7 RoB (AdptP)† 0.8M 90.5 96.6 89.7 67.8 94.8 91.7 80.1 91.9 87.9 large ±.3 ±.2 ±1.2 ±2.5 ±.3 ±.2 ±2.9 ±.4 RoB (AdptH)† 6.0M 89.9 96.2 88.7 66.5 94.7 92.1 83.4 91.0 87.8 large ±.5 ±.3 ±2.9 ±4.4 ±.2 ±.1 ±1.1 ±1.7 RoB (AdptH)† 0.8M 90.3 96.3 87.7 66.3 94.7 91.5 72.9 91.5 86.4 large ±.3 ±.5 ±1.7 ±2.0 ±.2 ±.1 ±2.9 ±.5 RoB (LoRA)† 0.8M 90.6 96.2 90.2 68.2 94.8 91.6 85.2 92.3 88.6 large ±.2 ±.5 ±1.0 ±1.9 ±.3 ±.2 ±1.1 ±.5 DeB (FT) 1500.0M 91.8 97.2 92.0 72.0 96.0 92.7 93.9 92.9 91.1 XXL DeB (LoRA) 4.7M 91.9 96.9 92.6 72.4 96.0 92.9 94.9 93.0 91.3 XXL ±.2 ±.2 ±.6 ±1.1 ±.1 ±.1 ±.4 ±.2 Table 2: RoBERTa , RoBERTa , and DeBERTa with different adaptation methods on the base large XXL GLUE benchmark.
<a id="S0104"></a> Source: p.6 S0104
We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks.
<a id="S0105"></a> Source: p.6 S0105
Higher is better for all metrics. * indicates numbers published in prior works. † indicates runs configured in a setup similar to Houlsby et al. (2019) for a fair comparison.
<a id="S0106"></a> Source: p.6 S0106
Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else.
<a id="S0107"></a> Source: p.6 S0107
Contemporarily, this baseline has also been studied by BitFit (Zaken et al., 2021).
<a id="S0108"></a> Source: p.6 S0108
Prefix-embedding tuning (PreEmbed) inserts special tokens among the input tokens.
<a id="S0109"></a> Source: p.6 S0109
These special tokens have trainable word embeddings and are generally not in the model’s vocabulary.
<a id="S0110"></a> Source: p.6 S0110
Where to place such tokens can have an impact on performance.
<a id="S0111"></a> Source: p.6 S0111
We focus on “prefixing”, which prepends such tokens to the prompt, and “infixing”, which appends to the prompt; both are discussed in Li & Liang (2021).
<a id="S0112"></a> Source: p.6 S0112
We use l (resp. l ) denote the number of prefix (resp. infix) tokens.
<a id="S0113"></a> Source: p.6 S0113
The number of p i trainable parameters is |Θ| = d × (l + l ). model p i Prefix-layer tuning (PreLayer) is an extension to prefix-embedding tuning.
<a id="S0114"></a> Source: p.6 S0114
Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer.
<a id="S0115"></a> Source: p.6 S0115
The activations computed from previous layers are simply replaced by trainable ones.
<a id="S0116"></a> Source: p.6 S0116
The resulting number of trainable parameters is |Θ| = L × d × (l + l ), where L is the number of Transformer layers. model p i Adapter tuning as proposed in Houlsby et al. (2019) inserts adapter layers between the selfattention module (and the MLP module) and the subsequent residual connection.
<a id="S0117"></a> Source: p.6 S0117
There are two fully connected layers with biases in an adapter layer with a nonlinearity in between.
<a id="S0118"></a> Source: p.6 S0118
Recently, Lin et al. (2020) proposed a more efficient design with the adapter layer applied only after the MLP module and after a LayerNorm.
<a id="S0119"></a> Source: p.6 S0119
This is very similar to another deign proposed in Pfeiffer et al. (2021), which we call AdapterP.
<a id="S0120"></a> Source: p.6 S0120
We also include another baseline call AdapterDrop (Ru¨ckle´ et al., 2020) which drops some adapter layers for greater efficiency (AdapterD).
<a id="S0121"></a> Source: p.6 S0121
We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the first column.
<a id="S0122"></a> Source: p.6 S0122
In all cases, we have |Θ| = Lˆ × (2 × d × r + r + d ) + 2 × Lˆ × d where Lˆ Adpt model model LN model Adpt is the number of adapter layers and Lˆ the number of trainable LayerNorms (e.g., in AdapterL).
<a id="S0123"></a> Source: p.6 S0123
LN LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices.
<a id="S0124"></a> Source: p.6 S0124
As mentioned in Section 4.2, we only apply LoRA to W and W in most experiments for simplicity. q v The number of trainable parameters is determined by the rank r and the shape of the original weights: |Θ| = 2 × Lˆ × d × r, where Lˆ is the number of weight matrices we apply LoRA to.
Page 7
<a id="S0125"></a> Source: p.7 S0125
Model & Method # Trainable E2E NLG Challenge Parameters BLEU NIST MET ROUGE-L CIDEr GPT-2 M (FT) 354.92M 68.2 8.62 46.2 71.0 2.47 GPT-2 M (AdapterL) 0.37M 66.3 8.41 45.0 69.8 2.40 GPT-2 M (AdapterL) 11.09M 68.9 8.71 46.1 71.3 2.47 GPT-2 M (AdapterH) 11.09M 67.3 8.50 46.0 70.7 2.44 ±.6 ±.07 ±.2 ±.2 ±.01 GPT-2 M (FTTop2) 25.19M 68.1 8.59 46.0 70.8 2.41 GPT-2 M (PreLayer) 0.35M 69.7 8.81 46.1 71.4 2.49 GPT-2 M (LoRA) 0.35M 70.4 8.85 46.8 71.8 2.53 ±.1 ±.02 ±.2 ±.1 ±.02 GPT-2 L (FT) 774.03M 68.5 8.78 46.0 69.9 2.45 GPT-2 L (AdapterL) 0.88M 69.1 8.68 46.3 71.4 2.49 ±.1 ±.03 ±.0 ±.2 ±.0 GPT-2 L (AdapterL) 23.00M 68.9 8.70 46.1 71.3 2.45 ±.3 ±.04 ±.1 ±.2 ±.02 GPT-2 L (PreLayer)* 0.77M 70.3 8.85 46.2 71.7 2.47 GPT-2 L (LoRA) 0.77M 70.4 8.89 46.8 72.0 2.47 ±.1 ±.02 ±.2 ±.2 ±.02 Table 3: GPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG Challenge.
<a id="S0126"></a> Source: p.7 S0126
LoRA outperforms several baselines with comparable or fewer trainable parameters.
<a id="S0127"></a> Source: p.7 S0127
Confidence intervals are shown for experiments we ran. * indicates numbers published in prior works. 5.2 ROBERTA BASE/LARGE RoBERTa (Liu et al., 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al., 2019a) and boosted the latter’s task performance without introducing many more trainable parameters.
<a id="S0128"></a> Source: p.7 S0128
While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners.
<a id="S0129"></a> Source: p.7 S0129
We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace Transformers library (Wolf et al., 2020) and evaluate the performance of different efficient adaptation approaches on tasks from the GLUE benchmark.
<a id="S0130"></a> Source: p.7 S0130
We also replicate Houlsby et al. (2019) and Pfeiffer et al. (2021) according to their setup.
<a id="S0131"></a> Source: p.7 S0131
To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters.
<a id="S0132"></a> Source: p.7 S0132
First, we use the same batch size for all tasks and use a sequence length of 128 to match the adapter baselines.
<a id="S0133"></a> Source: p.7 S0133
Second, we initialize the model to the pre-trained model for MRPC, RTE, and STS-B, not a model already adapted to MNLI like the fine-tuning baseline.
<a id="S0134"></a> Source: p.7 S0134
Runs following this more restricted setup from Houlsby et al. (2019) are labeled with †.
<a id="S0135"></a> Source: p.7 S0135
The result is presented in Table 2 (Top Three Sections).
<a id="S0136"></a> Source: p.7 S0136
See Section D.1 for details on the hyperparameters used. 5.3 DEBERTA XXL DeBERTa (He et al., 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al., 2019) and SuperGLUE (Wang et al., 2020).
<a id="S0137"></a> Source: p.7 S0137
We evaluate if LoRA can still match the performance of a fully fine-tuned DeBERTa XXL (1.5B) on GLUE.
<a id="S0138"></a> Source: p.7 S0138
The result is presented in Table 2 (Bottom Section).
<a id="S0139"></a> Source: p.7 S0139
See Section D.2 for details on the hyperparameters used. 5.4 GPT-2 MEDIUM/LARGE Having shown that LoRA can be a competitive alternative to full fine-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al., b).
<a id="S0140"></a> Source: p.7 S0140
We keep our setup as close as possible to Li & Liang (2021) for a direct comparison.
<a id="S0141"></a> Source: p.7 S0141
Due to space constraint, we only present our result on E2E NLG Challenge (Table 3) in this section.
<a id="S0142"></a> Source: p.7 S0142
See Section F.1 for results on WebNLG (Gardent et al., 2017) and DART (Nan et al., 2020).
<a id="S0143"></a> Source: p.7 S0143
We include a list of the hyperparameters used in Section D.3. 7
Page 8
<a id="S0144"></a> Source: p.8 S0144
Trainable WikiSQL MNLI-m SAMSum Model&Method Parameters Acc. (%) Acc. (%) R1/R2/RL GPT-3 (FT) 175,255.8M 73.8 89.5 52.0/28.0/44.5 GPT-3 (BitFit) 14.2M 71.3 91.0 51.3/27.4/43.5 GPT-3 (PreEmbed) 3.2M 63.1 88.6 48.3/24.2/40.5 GPT-3 (PreLayer) 20.2M 70.1 89.5 50.8/27.3/43.5 GPT-3 (AdapterH) 7.1M 71.9 89.8 53.0/28.9/44.8 GPT-3 (AdapterH) 40.1M 73.2 91.5 53.2/29.0/45.1 GPT-3 (LoRA) 4.7M 73.4 91.7 53.8/29.8/45.9 GPT-3 (LoRA) 37.7M 74.0 91.6 53.4/29.2/45.1 Table 4: Performance of different adaptation methods on GPT-3 175B.
<a id="S0145"></a> Source: p.8 S0145
We report the logical form validation accuracy on WikiSQL, validation accuracy on MultiNLI-matched, and Rouge-1/2/L on SAMSum.
<a id="S0146"></a> Source: p.8 S0146
LoRA performs better than prior approaches, including full fine-tuning.
<a id="S0147"></a> Source: p.8 S0147
The results on WikiSQL have a fluctuation around ±0.5%, MNLI-m around ±0.1%, and SAMSum around ±0.2/±0.2/±0.1 for the three metrics. 5.5 SCALING UP TO GPT-3 175B As a final stress test for LoRA, we scale up to GPT-3 with 175 billion parameters.
<a id="S0148"></a> Source: p.8 S0148
Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry.
<a id="S0149"></a> Source: p.8 S0149
See Section D.4 for details on the hyperparameters used.
<a id="S0150"></a> Source: p.8 S0150
As shown in Table 4, LoRA matches or exceeds the fine-tuning baseline on all three datasets.
<a id="S0151"></a> Source: p.8 S0151
Note that not all methods benefit monotonically from having more trainable parameters, as shown in Figure 2.
<a id="S0152"></a> Source: p.8 S0152
We observe a significant performance drop when we use more than 256 special tokens for prefix-embedding tuning or more than 32 special tokens for prefix-layer tuning.
<a id="S0153"></a> Source: p.8 S0153
This corroborates similar observations in Li & Liang (2021).
<a id="S0154"></a> Source: p.8 S0154
While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution.
<a id="S0155"></a> Source: p.8 S0155
Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section F.3. 0.75 0.70 0.65 0.60 0.55 6 7 8 9 10 11 log10 # Trainable Parameters ycaruccA noitadilaV WikiSQL MultiNLI-matched 0.92 0.90 Method Fine-Tune 0.88 PrefixEmbed PrefixLayer 0.86 Adapter(H) LoRA 0.84 6 7 8 9 10 11 log10 # Trainable Parameters Figure 2: GPT-3 175B validation accuracy vs. number of trainable parameters of several adaptation methods on WikiSQL and MNLI-matched.
<a id="S0156"></a> Source: p.8 S0156
LoRA exhibits better scalability and task performance.
<a id="S0157"></a> Source: p.8 S0157
See Section F.2 for more details on the plotted data points. 6 RELATED WORKS Transformer Language Models.
<a id="S0158"></a> Source: p.8 S0158
Transformer (Vaswani et al., 2017) is a sequence-to-sequence architecture that makes heavy use of self-attention.
<a id="S0159"></a> Source: p.8 S0159
Radford et al. (a) applied it to autoregressive language modeling by using a stack of Transformer decoders.
<a id="S0160"></a> Source: p.8 S0160
Since then, Transformer-based language models have dominated NLP, achieving the state-of-the-art in many tasks. A new paradigm emerged with BERT (Devlin et al., 2019b) and GPT-2 (Radford et al., b) – both are large Transformer lan- 8
Page 9
<a id="S0161"></a> Source: p.9 S0161
guage models trained on a large amount of text – where fine-tuning on task-specific data after pretraining on general domain data provides a significant performance gain compared to training on task-specific data directly.
<a id="S0162"></a> Source: p.9 S0162
Training larger Transformers generally results in better performance and remains an active research direction.
<a id="S0163"></a> Source: p.9 S0163
GPT-3 (Brown et al., 2020) is the largest single Transformer language model trained to-date with 175B parameters.
<a id="S0164"></a> Source: p.9 S0164
While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).
<a id="S0165"></a> Source: p.9 S0165
This necessitates an empirical art of composing and formatting the prompt to maximize a model’s performance on a desired task, which is known as prompt engineering or prompt hacking.
<a id="S0166"></a> Source: p.9 S0166
Fine-tuning retrains a model pre-trained on general domains to a specific task Devlin et al. (2019b); Radford et al. (a).
<a id="S0167"></a> Source: p.9 S0167
Variants of it include learning just a subset of the parameters Devlin et al. (2019b); Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream performance.
<a id="S0168"></a> Source: p.9 S0168
However, the enormity of GPT-3 175B makes it challenging to perform fine-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.
<a id="S0169"></a> Source: p.9 S0169
Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al., 2019; Rebuffi et al., 2017; Lin et al., 2020).
<a id="S0170"></a> Source: p.9 S0170
Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates.
<a id="S0171"></a> Source: p.9 S0171
The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3). A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme.
<a id="S0172"></a> Source: p.9 S0172
Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efficiency, which we leave to future work.
<a id="S0173"></a> Source: p.9 S0173
More recently, many proposed optimizing the input word embeddings in lieu of fine-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021).
<a id="S0174"></a> Source: p.9 S0174
We include comparisons with Li & Liang (2021) in our experiment section.
<a id="S0175"></a> Source: p.9 S0175
However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.
<a id="S0176"></a> Source: p.9 S0176
Low-rank structure is very common in machine learning. A lot of machine learning problems have certain intrinsic low-rank structure (Li et al., 2016; Cai et al., 2010; Li et al., 2018b; Grasedyck et al., 2013).
<a id="S0177"></a> Source: p.9 S0177
Moreover, it is known that for many deep learning tasks, especially those with a heavily over-parametrized neural network, the learned neural network will enjoy low-rank properties after training (Oymak et al., 2019).
<a id="S0178"></a> Source: p.9 S0178
Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013; Povey et al., 2018; Zhang et al., 2014; Jaderberg et al., 2014; Zhao et al., 2016; Khodak et al., 2021; Denil et al., 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks.
<a id="S0179"></a> Source: p.9 S0179
In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (finite-width) neural tangent kernels (Allen-Zhu et al., 2019; Li & Liang, 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al., 2020; Allen-Zhu & Li, 2019; Allen-Zhu & Li, 2020a).
<a id="S0180"></a> Source: p.9 S0180
Another theoretical result in Allen-Zhu & Li (2020b) suggests that low-rank adaptations can be useful for adversarial training.
<a id="S0181"></a> Source: p.9 S0181
In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature. 7 UNDERSTANDING THE LOW-RANK UPDATES Given the empirical advantage of LoRA, we hope to further explain the properties of the low-rank adaptation learned from downstream tasks.
<a id="S0182"></a> Source: p.9 S0182
Note that the low-rank structure not only lowers the hardware barrier to entry which allows us to run multiple experiments in parallel, but also gives better interpretability of how the update weights are correlated with the pre-trained weights.
<a id="S0183"></a> Source: p.9 S0183
We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000×) without adversely affecting task performances.
<a id="S0184"></a> Source: p.9 S0184
We perform a sequence of empirical studies to answer the following questions: 1) Given a parameter budget constraint, which subset of weight matrices in a pre-trained Transformer should we adapt 9
Page 10
<a id="S0185"></a> Source: p.10 S0185
to maximize downstream performance? 2) Is the “optimal” adaptation matrix ∆W really rankdeficient?
<a id="S0186"></a> Source: p.10 S0186
If so, what is a good rank to use in practice? 3) What is the connection between ∆W and W ?
<a id="S0187"></a> Source: p.10 S0187
We believe that our answers to question (2) and (3) shed light on the fundamental principles of using pre-trained language models for downstream tasks, which is a critical topic in NLP. 7.1 WHICH WEIGHT MATRICES IN TRANSFORMER SHOULD WE APPLY LORA TO?
<a id="S0188"></a> Source: p.10 S0188
Given a limited parameter budget, which types of weights should we adapt with LoRA to obtain the best performance on downstream tasks?
<a id="S0189"></a> Source: p.10 S0189
As mentioned in Section 4.2, we only consider weight matrices in the self-attention module.
<a id="S0190"></a> Source: p.10 S0190
We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to r = 8 if we adapt one type of attention weights or r = 4 if we adapt two types, for all 96 layers.
<a id="S0191"></a> Source: p.10 S0191
The result is presented in Table 5. # of Trainable Parameters = 18M Weight Type W W W W W , W W , W W , W , W , W q k v o q k q v q k v o Rank r 8 8 8 8 4 4 2 WikiSQL (±0.5%) 70.4 70.0 73.0 73.2 71.4 73.7 73.7 MultiNLI (±0.1%) 91.0 90.8 91.0 91.3 91.3 91.3 91.7 Table 5: Validation accuracy on WikiSQL and MultiNLI after applying LoRA to different types of attention weights in GPT-3, given the same number of trainable parameters.
<a id="S0192"></a> Source: p.10 S0192
Adapting both W and q W gives the best performance overall.
<a id="S0193"></a> Source: p.10 S0193
We find the standard deviation across random seeds to be v consistent for a given dataset, which we report in the first column.
<a id="S0194"></a> Source: p.10 S0194
Note that putting all the parameters in ∆W or ∆W results in significantly lower performance, q k while adapting both W and W yields the best result.
<a id="S0195"></a> Source: p.10 S0195
This suggests that even a rank of four q v captures enough information in ∆W such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank. 7.2 WHAT IS THE OPTIMAL RANK r FOR LORA?
<a id="S0196"></a> Source: p.10 S0196
We turn our attention to the effect of rank r on model performance.
<a id="S0197"></a> Source: p.10 S0197
We adapt {W , W }, q v {W , W , W , W }, and just W for a comparison. q k v c q Weight Type r = 1 r = 2 r = 4 r = 8 r = 64 W 68.8 69.6 70.5 70.4 70.0 WikiSQL(±0.5%) q W , W 73.4 73.3 73.7 73.8 73.5 q v W , W , W , W 74.1 73.7 74.0 74.0 73.9 q k v o W 90.7 90.9 91.1 90.7 90.7 q MultiNLI (±0.1%) W , W 91.3 91.4 91.3 91.6 91.4 q v W , W , W , W 91.2 91.7 91.7 91.5 91.4 q k v o Table 6: Validation accuracy on WikiSQL and MultiNLI with different rank r.
<a id="S0198"></a> Source: p.10 S0198
To our surprise, a rank as small as one suffices for adapting both W and W on these datasets while training W alone q v q needs a larger r.
<a id="S0199"></a> Source: p.10 S0199
We conduct a similar experiment on GPT-2 in Section H.2.
<a id="S0200"></a> Source: p.10 S0200
Table 6 shows that, surprisingly, LoRA already performs competitively with a very small r (more so for {W , W } than just W ).
<a id="S0201"></a> Source: p.10 S0201
This suggests the update matrix ∆W could have a very small q v q “intrinsic rank”.6 To further support this finding, we check the overlap of the subspaces learned by different choices of r and by different random seeds.
<a id="S0202"></a> Source: p.10 S0202
We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient. 6However, we do not expect a small r to work for every task or dataset.
<a id="S0203"></a> Source: p.10 S0203
Consider the following thought experiment: if the downstream task were in a different language than the one used for pre-training, retraining the entire model (similar to LoRA with r = d ) could certainly outperform LoRA with a small r. model 10
Page 11
<a id="S0204"></a> Source: p.11 S0204
Subspace similarity between different r.
<a id="S0205"></a> Source: p.11 S0205
Given A and A which are the learned adaptar=8 r=64 tion matrices with rank r = 8 and 64 using the same pre-trained model, we perform singular value decomposition and obtain the right-singular unitary matrices U and U .7 We hope to an- Ar=8 Ar=64 swer: how much of the subspace spanned by the top i singular vectors in U (for 1 ≤ i ≤ 8) is Ar=8 contained in the subspace spanned by top j singular vectors of U (for 1 ≤ j ≤ 64)?
<a id="S0206"></a> Source: p.11 S0206
We mea- Ar=64 sure this quantity with a normalized subspace similarity based on the Grassmann distance (See Appendix G for a more formal discussion) ||U i(cid:62) U j ||2 φ(A , A , i, j) = Ar=8 Ar=64 F ∈ [0, 1] (4) r=8 r=64 min(i, j) where U i represents the columns of U corresponding to the top-i singular vectors.
<a id="S0207"></a> Source: p.11 S0207
Ar=8 Ar=8 φ(·) has a range of [0, 1], where 1 represents a complete overlap of subspaces and 0 a complete separation.
<a id="S0208"></a> Source: p.11 S0208
See Figure 3 for how φ changes as we vary i and j.
<a id="S0209"></a> Source: p.11 S0209
We only look at the 48th layer (out of 96) due to space constraint, but the conclusion holds for other layers as well, as shown in Section H.1. 1.0 0.8 0.6 0.4 0.2 0.0 1 6 21 81 32 92 53 04 64 25 85 j i 1 2 3 4 5 6 7 8 Wq 1 6 21 81 32 92 53 04 64 25 85 (Ar =64, Ar =8, i, j) Wv Wq Wv 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 j j j Figure 3: Subspace similarity between column vectors of A and A for both ∆W and ∆W . r=8 r=64 q v The third and the fourth figures zoom in on the lower-left triangle in the first two figures.
<a id="S0210"></a> Source: p.11 S0210
The top directions in r = 8 are included in r = 64, and vice versa.
<a id="S0211"></a> Source: p.11 S0211
We make an important observation from Figure 3.
<a id="S0212"></a> Source: p.11 S0212
Directions corresponding to the top singular vector overlap significantly between A and A , while others do not.
<a id="S0213"></a> Source: p.11 S0213
Specifically, ∆W (resp. ∆W ) of A r=8 r=64 v q r=8 and ∆W (resp. ∆W ) of A share a subspace of dimension 1 with normalized v q r=64 similarity > 0.5, providing an explanation of why r = 1 performs quite well in our downstream tasks for GPT-3.
<a id="S0214"></a> Source: p.11 S0214
Since both A and A are learned using the same pre-trained model, Figure 3 indicates that r=8 r=64 the top singular-vector directions of A and A are the most useful, while other directions r=8 r=64 potentially contain mostly random noises accumulated during training.
<a id="S0215"></a> Source: p.11 S0215
Hence, the adaptation matrix can indeed have a very low rank.
<a id="S0216"></a> Source: p.11 S0216
Subspace similarity between different random seeds.
<a id="S0217"></a> Source: p.11 S0217
We further confirm this by plotting the normalized subspace similarity between two randomly seeded runs with r = 64, shown in Figure 4. ∆W appears to have a higher “intrinsic rank” than ∆W , since more common singular value direcq v tions are learned by both runs for ∆W , which is in line with our empirical observation in Table 6. q As a comparison, we also plot two random Gaussian matrices, which do not share any common singular value directions with each other. 7.3 HOW DOES THE ADAPTATION MATRIX ∆W COMPARE TO W ?
<a id="S0218"></a> Source: p.11 S0218
We further investigate the relationship between ∆W and W .
<a id="S0219"></a> Source: p.11 S0219
In particular, does ∆W highly correlate with W ? (Or mathematically, is ∆W mostly contained in the top singular directions of W ?) Also, 7Note that a similar analysis can be carried out with B and the left-singular unitary matrices – we stick with A for our experiments. 11
Page 12
<a id="S0220"></a> Source: p.12 S0220
0.5 0.4 0.3 0.2 0.1 0.0 1 5 01 51 02 52 03 43 93 44 94 45 95 1 8 16 24 32 40 48 56 j i Wq 1 5 01 51 02 52 03 43 93 44 94 45 95 (Ar=64, A0r=64, i, j) Wv j 1 5 01 51 02 52 03 43 93 44 94 45 95 Random Gaussian j Figure 4: Left and Middle: Normalized subspace similarity between the column vectors of A r=64 from two random seeds, for both ∆W and ∆W in the 48-th layer.
<a id="S0221"></a> Source: p.12 S0221
Right: the same heat-map q v between the column vectors of two random Gaussian matrices.
<a id="S0222"></a> Source: p.12 S0222
See Section H.1 for other layers. how “large” is ∆W comparing to its corresponding directions in W ?
<a id="S0223"></a> Source: p.12 S0223
This can shed light on the underlying mechanism for adapting pre-trained language models.
<a id="S0224"></a> Source: p.12 S0224
To answer these questions, we project W onto the r-dimensional subspace of ∆W by computing U (cid:62)W V (cid:62), with U /V being the left/right singular-vector matrix of ∆W .
<a id="S0225"></a> Source: p.12 S0225
Then, we compare the Frobenius norm between (cid:107)U (cid:62)W V (cid:62)(cid:107) and (cid:107)W (cid:107) .
<a id="S0226"></a> Source: p.12 S0226
As a comparison, we also compute F F (cid:107)U (cid:62)W V (cid:62)(cid:107) by replacing U, V with the top r singular vectors of W or a random matrix. F r = 4 r = 64 ∆W W Random ∆W W Random q q q q ||U (cid:62)W V (cid:62)|| = 0.32 21.67 0.02 1.90 37.71 0.33 q F ||W || = 61.95 ||∆W || = 6.91 ||∆W || = 3.57 q F q F q F Table 7: The Frobenius norm of U (cid:62)W V (cid:62) where U and V are the left/right top r singular vector q directions of either (1) ∆W , (2) W , or (3) a random matrix.
<a id="S0227"></a> Source: p.12 S0227
The weight matrices are taken from q q the 48th layer of GPT-3.
<a id="S0228"></a> Source: p.12 S0228
We draw several conclusions from Table 7.
<a id="S0229"></a> Source: p.12 S0229
First, ∆W has a stronger correlation with W compared to a random matrix, indicating that ∆W amplifies some features that are already in W .
<a id="S0230"></a> Source: p.12 S0230
Second, instead of repeating the top singular directions of W , ∆W only amplifies directions that are not emphasized in W .
<a id="S0231"></a> Source: p.12 S0231
Third, the amplification factor is rather huge: 21.5 ≈ 6.91/0.32 for r = 4.
<a id="S0232"></a> Source: p.12 S0232
See Section H.4 for why r = 64 has a smaller amplification factor.
<a id="S0233"></a> Source: p.12 S0233
We also provide a visualization in Section H.3 for how the correlation changes as we include more top singular directions from W . q This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model. 8 CONCLUSION AND FUTURE WORK Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks.
<a id="S0234"></a> Source: p.12 S0234
We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.
<a id="S0235"></a> Source: p.12 S0235
Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters.
<a id="S0236"></a> Source: p.12 S0236
While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.
<a id="S0237"></a> Source: p.12 S0237
There are many directions for future works. 1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement. 2) The mechanism behind fine-tuning or LoRA is far from clear – how are features learned during pre-training transformed to do well on downstream tasks?
<a id="S0238"></a> Source: p.12 S0238
We believe that LoRA makes it more tractable to answer this than full fine- 12
Page 13
<a id="S0239"></a> Source: p.13 S0239
tuning. 3) We mostly depend on heuristics to select the weight matrices to apply LoRA to.
<a id="S0240"></a> Source: p.13 S0240
Are there more principled ways to do it? 4) Finally, the rank-deficiency of ∆W suggests that W could be rank-deficient as well, which can also be a source of inspiration for future works.
<a id="S0241"></a> Source: p.13 S0241
REFERENCES Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta.
<a id="S0242"></a> Source: p.13 S0242
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv:2012.13255 [cs], December 2020.
<a id="S0243"></a> Source: p.13 S0243
What Can ResNet Learn Efficiently, Going Beyond Kernels?
<a id="S0244"></a> Source: p.13 S0244
Full version available at http://arxiv.org/abs/1905.10337.
<a id="S0245"></a> Source: p.13 S0245
Backward feature correction: How deep learning performs deep learning. arXiv preprint arXiv:2001.04413, 2020a.
<a id="S0246"></a> Source: p.13 S0246
Feature purification: How adversarial training performs robust deep learning. arXiv preprint arXiv:2005.10190, 2020b.
<a id="S0247"></a> Source: p.13 S0247
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization.
<a id="S0248"></a> Source: p.13 S0248
Full version available at http://arxiv.org/abs/1811. 03962.
<a id="S0249"></a> Source: p.13 S0249
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E.
<a id="S0250"></a> Source: p.13 S0250
Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
<a id="S0251"></a> Source: p.13 S0251
Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
<a id="S0252"></a> Source: p.13 S0252
Language Models are Few-Shot Learners. arXiv:2005.14165 [cs], July 2020.
<a id="S0253"></a> Source: p.13 S0253
Jian-Feng Cai, Emmanuel J Cande`s, and Zuowei Shen. A singular value thresholding algorithm for matrix completion.
<a id="S0254"></a> Source: p.13 S0254
SIAM Journal on optimization, 20(4):1956–1982, 2010.
<a id="S0255"></a> Source: p.13 S0255
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.
<a id="S0256"></a> Source: p.13 S0256
Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.
<a id="S0257"></a> Source: p.13 S0257
Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017. doi: 10.18653/ v1/s17-2001.
<a id="S0258"></a> Source: p.13 S0258
URL http://dx.doi.org/10.18653/v1/S17-2001.
<a id="S0259"></a> Source: p.13 S0259
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning.
<a id="S0260"></a> Source: p.13 S0260
In Proceedings of the 25th international conference on Machine learning, ICML ’08, pp. 160–167, New York, NY, USA, July 2008.
<a id="S0261"></a> Source: p.13 S0261
ISBN 978-1-60558-205-4. doi: 10.1145/1390156.1390177.
<a id="S0262"></a> Source: p.13 S0262
URL https://doi.org/10.1145/1390156.1390177.
<a id="S0263"></a> Source: p.13 S0263
Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas.
<a id="S0264"></a> Source: p.13 S0264
Predicting parameters in deep learning, 2014.
<a id="S0265"></a> Source: p.13 S0265
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
<a id="S0266"></a> Source: p.13 S0266
Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.
<a id="S0267"></a> Source: p.13 S0267
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
<a id="S0268"></a> Source: p.13 S0268
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019b.
<a id="S0269"></a> Source: p.13 S0269
URL http://arxiv.org/abs/1810.04805. arXiv: 1810.04805.
<a id="S0270"></a> Source: p.13 S0270
Automatically constructing a corpus of sentential paraphrases.
<a id="S0271"></a> Source: p.13 S0271
In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
<a id="S0272"></a> Source: p.13 S0272
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini.
<a id="S0273"></a> Source: p.13 S0273
The webnlg challenge: Generating text from rdf data.
<a id="S0274"></a> Source: p.13 S0274
In Proceedings of the 10th International Conference on Natural Language Generation, pp. 124–133, 2017. 13
Page 14
<a id="S0275"></a> Source: p.14 S0275
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari.
<a id="S0276"></a> Source: p.14 S0276
When do neural networks outperform kernel methods? arXiv preprint arXiv:2006.13409, 2020.
<a id="S0277"></a> Source: p.14 S0277
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer.
<a id="S0278"></a> Source: p.14 S0278
Samsum corpus: A humanannotated dialogue dataset for abstractive summarization.
<a id="S0279"></a> Source: p.14 S0279
Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques.
<a id="S0280"></a> Source: p.14 S0280
Grassmann discriminant analysis: a unifying view on subspace-based learning.
<a id="S0281"></a> Source: p.14 S0281
URL https://doi.org/10.1145/1390156. 1390204.
<a id="S0282"></a> Source: p.14 S0282
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May.
<a id="S0283"></a> Source: p.14 S0283
WARP: Word-level Adversarial ReProgramming. arXiv:2101.00121 [cs], December 2020.
<a id="S0284"></a> Source: p.14 S0284
URL http://arxiv.org/abs/ 2101.00121. arXiv: 2101.00121.
<a id="S0285"></a> Source: p.14 S0285
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
<a id="S0286"></a> Source: p.14 S0286
Deberta: Decoding-enhanced bert with disentangled attention, 2021.
<a id="S0287"></a> Source: p.14 S0287
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
<a id="S0288"></a> Source: p.14 S0288
Parameter-Efficient Transfer Learning for NLP. arXiv:1902.00751 [cs, stat], June 2019.
<a id="S0289"></a> Source: p.14 S0289
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman.
<a id="S0290"></a> Source: p.14 S0290
Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
<a id="S0291"></a> Source: p.14 S0291
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo` Fusi.
<a id="S0292"></a> Source: p.14 S0292
Initialization and regularization of factorized neural layers, 2021.
<a id="S0293"></a> Source: p.14 S0293
Adam: A method for stochastic optimization, 2017.
<a id="S0294"></a> Source: p.14 S0294
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
<a id="S0295"></a> Source: p.14 S0295
Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
<a id="S0296"></a> Source: p.14 S0296
Brian Lester, Rami Al-Rfou, and Noah Constant.
<a id="S0297"></a> Source: p.14 S0297
The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691 [cs], April 2021.
<a id="S0298"></a> Source: p.14 S0298
URL http://arxiv.org/abs/2104.08691. arXiv: 2104.08691.
<a id="S0299"></a> Source: p.14 S0299
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.
<a id="S0300"></a> Source: p.14 S0300
Measuring the Intrinsic Dimension of Objective Landscapes. arXiv:1804.08838 [cs, stat], April 2018a.
<a id="S0301"></a> Source: p.14 S0301
URL http: //arxiv.org/abs/1804.08838. arXiv: 1804.08838.
<a id="S0302"></a> Source: p.14 S0302
Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190 [cs], January 2021.
<a id="S0303"></a> Source: p.14 S0303
Learning overparameterized neural networks via stochastic gradient descent on structured data.
<a id="S0304"></a> Source: p.14 S0304
In Advances in Neural Information Processing Systems, 2018.
<a id="S0305"></a> Source: p.14 S0305
Yuanzhi Li, Yingyu Liang, and Andrej Risteski.
<a id="S0306"></a> Source: p.14 S0306
Recovery guarantee of weighted low-rank approximation via alternating minimization.
<a id="S0307"></a> Source: p.14 S0307
In International Conference on Machine Learning, pp. 2358–2367.
<a id="S0308"></a> Source: p.14 S0308
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang.
<a id="S0309"></a> Source: p.14 S0309
Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations.
<a id="S0310"></a> Source: p.14 S0310
In Conference On Learning Theory, pp. 2–47.
<a id="S0311"></a> Source: p.14 S0311
Zhaojiang Lin, Andrea Madotto, and Pascale Fung.
<a id="S0312"></a> Source: p.14 S0312
Exploring versatile generative language model via parameter-efficient transfer learning.
<a id="S0313"></a> Source: p.14 S0313
In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 441–459, Online, November 2020.
<a id="S0314"></a> Source: p.14 S0314
Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.41.
<a id="S0315"></a> Source: p.14 S0315
URL https://aclanthology. org/2020.findings-emnlp.41. 14
Page 15
<a id="S0316"></a> Source: p.15 S0316
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.
<a id="S0317"></a> Source: p.15 S0317
GPT Understands, Too. arXiv:2103.10385 [cs], March 2021.
<a id="S0318"></a> Source: p.15 S0318
URL http://arxiv.org/abs/ 2103.10385. arXiv: 2103.10385.
<a id="S0319"></a> Source: p.15 S0319
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
<a id="S0320"></a> Source: p.15 S0320
Roberta: A robustly optimized bert pretraining approach, 2019.
<a id="S0321"></a> Source: p.15 S0321
Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
<a id="S0322"></a> Source: p.15 S0322
Decoupled weight decay regularization, 2019.
<a id="S0323"></a> Source: p.15 S0323
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.
<a id="S0324"></a> Source: p.15 S0324
Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
<a id="S0325"></a> Source: p.15 S0325
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al.
<a id="S0326"></a> Source: p.15 S0326
Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871, 2020.
<a id="S0327"></a> Source: p.15 S0327
Jekaterina Novikova, Ondˇrej Dusˇek, and Verena Rieser.
<a id="S0328"></a> Source: p.15 S0328
The e2e dataset: New challenges for endto-end generation. arXiv preprint arXiv:1706.09254, 2017.
<a id="S0329"></a> Source: p.15 S0329
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi.
<a id="S0330"></a> Source: p.15 S0330
Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392, 2019.
<a id="S0331"></a> Source: p.15 S0331
Jonas Pfeiffer, Aishwarya Kamath, Andreas Ru¨ckle´, Kyunghyun Cho, and Iryna Gurevych.
<a id="S0332"></a> Source: p.15 S0332
Adapterfusion: Non-destructive task composition for transfer learning, 2021.
<a id="S0333"></a> Source: p.15 S0333
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur.
<a id="S0334"></a> Source: p.15 S0334
Semi-orthogonal low-rank matrix factorization for deep neural networks.
<a id="S0335"></a> Source: p.15 S0335
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
<a id="S0336"></a> Source: p.15 S0336
Improving Language Understanding by Generative Pre-Training. pp. 12, a.
<a id="S0337"></a> Source: p.15 S0337
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
<a id="S0338"></a> Source: p.15 S0338
Language Models are Unsupervised Multitask Learners. pp. 24, b.
<a id="S0339"></a> Source: p.15 S0339
Pranav Rajpurkar, Robin Jia, and Percy Liang.
<a id="S0340"></a> Source: p.15 S0340
Know what you don’t know: Unanswerable questions for squad.
<a id="S0341"></a> Source: p.15 S0341
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.
<a id="S0342"></a> Source: p.15 S0342
Learning multiple visual domains with residual adapters. arXiv:1705.08045 [cs, stat], November 2017.
<a id="S0343"></a> Source: p.15 S0343
URL http://arxiv.org/ abs/1705.08045. arXiv: 1705.08045.
<a id="S0344"></a> Source: p.15 S0344
Andreas Ru¨ckle´, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych.
<a id="S0345"></a> Source: p.15 S0345
Adapterdrop: On the efficiency of adapters in transformers, 2020.
<a id="S0346"></a> Source: p.15 S0346
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran.
<a id="S0347"></a> Source: p.15 S0347
Lowrank matrix factorization for deep neural network training with high-dimensional output targets.
<a id="S0348"></a> Source: p.15 S0348
In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655– 6659.
<a id="S0349"></a> Source: p.15 S0349
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
<a id="S0350"></a> Source: p.15 S0350
Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.
<a id="S0351"></a> Source: p.15 S0351
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D.
<a id="S0352"></a> Source: p.15 S0352
Manning, Andrew Ng, and Christopher Potts.
<a id="S0353"></a> Source: p.15 S0353
Recursive deep models for semantic compositionality over a sentiment treebank.
<a id="S0354"></a> Source: p.15 S0354
In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA, October 2013.
<a id="S0355"></a> Source: p.15 S0355
Association for Computational Linguistics.
<a id="S0356"></a> Source: p.15 S0356
URL https://aclanthology.org/D13-1170. 15
Page 16
<a id="S0357"></a> Source: p.16 S0357
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
<a id="S0358"></a> Source: p.16 S0358
In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010, 2017.
<a id="S0359"></a> Source: p.16 S0359
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
<a id="S0360"></a> Source: p.16 S0360
Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
<a id="S0361"></a> Source: p.16 S0361
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
<a id="S0362"></a> Source: p.16 S0362
Superglue: A stickier benchmark for general-purpose language understanding systems, 2020.
<a id="S0363"></a> Source: p.16 S0363
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.
<a id="S0364"></a> Source: p.16 S0364
Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018.
<a id="S0365"></a> Source: p.16 S0365
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference.
<a id="S0366"></a> Source: p.16 S0366
In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122, New Orleans, Louisiana, June 2018.
<a id="S0367"></a> Source: p.16 S0367
Association for Computational Linguistics. doi: 10.18653/v1/N18-1101.
<a id="S0368"></a> Source: p.16 S0368
URL https://www.aclweb. org/anthology/N18-1101.
<a id="S0369"></a> Source: p.16 S0369
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Re´mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M.
<a id="S0370"></a> Source: p.16 S0370
Transformers: State-of-the-art natural language processing.
<a id="S0371"></a> Source: p.16 S0371
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, Online, October 2020.
<a id="S0372"></a> Source: p.16 S0372
Association for Computational Linguistics.
<a id="S0373"></a> Source: p.16 S0373
URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6.
<a id="S0374"></a> Source: p.16 S0374
Feature Learning in Infinite-Width Neural Networks. arXiv:2011.14522 [cond-mat], May 2021.
<a id="S0375"></a> Source: p.16 S0375
URL http://arxiv.org/abs/2011.14522. arXiv: 2011.14522.
<a id="S0376"></a> Source: p.16 S0376
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.
<a id="S0377"></a> Source: p.16 S0377
Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021.
<a id="S0378"></a> Source: p.16 S0378
Yu Zhang, Ekapol Chuangsuwanich, and James Glass.
<a id="S0379"></a> Source: p.16 S0379
Extracting deep neural network bottleneck features using low-rank matrix factorization.
<a id="S0380"></a> Source: p.16 S0380
In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 185–189.
<a id="S0381"></a> Source: p.16 S0381
Low-rank plus diagonal adaptation for deep neural networks.
<a id="S0382"></a> Source: p.16 S0382
In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5005–5009.
<a id="S0383"></a> Source: p.16 S0383
Victor Zhong, Caiming Xiong, and Richard Socher.
<a id="S0384"></a> Source: p.16 S0384
Seq2sql: Generating structured queries from natural language using reinforcement learning.
<a id="S0385"></a> Source: p.16 S0385
URL http:// arxiv.org/abs/1709.00103. A LARGE LANGUAGE MODELS STILL NEED PARAMETER UPDATES Few-shot learning, or prompt engineering, is very advantageous when we only have a handful of training samples.
<a id="S0386"></a> Source: p.16 S0386
However, in practice, we can often afford to curate a few thousand or more training examples for performance-sensitive applications.
<a id="S0387"></a> Source: p.16 S0387
As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.
<a id="S0388"></a> Source: p.16 S0388
We take the GPT-3 few-shot result on RTE from the GPT-3 paper (Brown et al., 2020).
<a id="S0389"></a> Source: p.16 S0389
For MNLI-matched, we use two demonstrations per class and six in-context examples in total. 16
Page 17
<a id="S0390"></a> Source: p.17 S0390
Acc./%) GPT-3 Few-Shot 40.6 69.0 GPT-3 Fine-Tuned 89.5 85.4 Table 8: Fine-tuning significantly outperforms few-shot learning on GPT-3 (Brown et al., 2020). B INFERENCE LATENCY INTRODUCED BY ADAPTER LAYERS Adapter layers are external modules added to a pre-trained model in a sequential manner, whereas our proposal, LoRA, can be seen as external modules added in a parallel manner.
<a id="S0391"></a> Source: p.17 S0391
Consequently, adapter layers must be computed in addition to the base model, inevitably introducing additional latency.
<a id="S0392"></a> Source: p.17 S0392
While as pointed out in Ru¨ckle´ et al. (2020), the latency introduced by adapter layers can be mitigated when the model batch size and/or sequence length is large enough to full utilize the hardware parallelism.
<a id="S0393"></a> Source: p.17 S0393
We confirm their observation with a similar latency study on GPT-2 medium and point out that there are scenarios, notably online inference where the batch size is small, where the added latency can be significant.
<a id="S0394"></a> Source: p.17 S0394
We measure the latency of a single forward pass on an NVIDIA Quadro RTX8000 by averaging over 100 trials.
<a id="S0395"></a> Source: p.17 S0395
We vary the input batch size, sequence length, and the adapter bottleneck dimension r.
<a id="S0396"></a> Source: p.17 S0396
We test two adapter designs: the original one by Houlsby et al. (2019), which we call AdapterH, and a recent, more efficient variant by Lin et al. (2020), which we call AdapterL.
<a id="S0397"></a> Source: p.17 S0397
See Section 5.1 for more details on the designs.
<a id="S0398"></a> Source: p.17 S0398
We plot the slow-down in percentage compared to the no-adapter baseline in Figure 5. 35 30 25 20 15 10 5 0 H r retpadA 0 01 001 052 Seq Len = 128 Seq Len = 256 Seq Len = 512 1 2 4 8 16 32 Batch Size L r retpadA 0 01 001 052 1 2 4 8 16 32 1 2 4 8 16 32 Batch Size Batch Size Figure 5: Percentage slow-down of inference latency compared to the no-adapter (r = 0) baseline.
<a id="S0399"></a> Source: p.17 S0399
The top row shows the result for AdapterH and the bottom row AdapterL.
<a id="S0400"></a> Source: p.17 S0400
Larger batch size and sequence length help to mitigate the latency, but the slow-down can be as high as over 30% in an online, short-sequence-length scenario.
<a id="S0401"></a> Source: p.17 S0401
We tweak the colormap for better visibility. C DATASET DETAILS GLUE Benchmark is a wide-ranging collection of natural language understanding tasks.
<a id="S0402"></a> Source: p.17 S0402
It includes MNLI (inference, Williams et al. (2018)), SST-2 (sentiment analysis, Socher et al. (2013)), MRPC (paraphrase detection, Dolan & Brockett (2005)), CoLA (linguistic acceptability, Warstadt et al. (2018)), QNLI (inference, Rajpurkar et al. (2018)), QQP8 (question-answering), RTE (inference), 8https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs 17
Page 18
<a id="S0403"></a> Source: p.18 S0403
and STS-B (textual similarity, Cer et al. (2017)).
<a id="S0404"></a> Source: p.18 S0404
The broad coverage makes GLUE benchmark a standard metric to evaluate NLU models such as RoBERTa and DeBERTa.
<a id="S0405"></a> Source: p.18 S0405
The individual datasets are released under different permissive licenses.
<a id="S0406"></a> Source: p.18 S0406
WikiSQL is introduced in Zhong et al. (2017) and contains 56, 355/8, 421 training/validation examples.
<a id="S0407"></a> Source: p.18 S0407
The task is to generate SQL queries from natural language questions and table schemata.
<a id="S0408"></a> Source: p.18 S0408
We encode context as x = {table schema, query} and target as y = {SQL}.
<a id="S0409"></a> Source: p.18 S0409
The dataset is release under the BSD 3-Clause License.
<a id="S0410"></a> Source: p.18 S0410
SAMSum is introduced in Gliwa et al. (2019) and contains 14, 732/819 training/test examples.
<a id="S0411"></a> Source: p.18 S0411
It consists of staged chat conversations between two people and corresponding abstractive summaries written by linguists.
<a id="S0412"></a> Source: p.18 S0412
We encode context as ”\n” concatenated utterances followed by a ”\n\n”, and target as y = {summary}.
<a id="S0413"></a> Source: p.18 S0413
The dataset is released under the non-commercial licence: Creative Commons BY-NC-ND 4.0.
<a id="S0414"></a> Source: p.18 S0414
E2E NLG Challenge was first introduced in Novikova et al. (2017) as a dataset for training end-toend, data-driven natural language generation systems and is commonly used for data-to-text evaluation.
<a id="S0415"></a> Source: p.18 S0415
The E2E dataset consists of roughly 42, 000 training, 4, 600 validation, and 4, 600 test examples from the restaurant domain.
<a id="S0416"></a> Source: p.18 S0416
Each source table used as input can have multiple references.
<a id="S0417"></a> Source: p.18 S0417
Each sample input (x, y) consists of a sequence of slot-value pairs, along with a corresponding natural language reference text.
<a id="S0418"></a> Source: p.18 S0418
The dataset is released under Creative Commons BY-NC-SA 4.0.
<a id="S0419"></a> Source: p.18 S0419
DART is an open-domain data-to-text dataset described in Nan et al. (2020).
<a id="S0420"></a> Source: p.18 S0420
DART inputs are structured as sequences of ENTITY — RELATION — ENTITY triples.
<a id="S0421"></a> Source: p.18 S0421
With 82K examples in total, DART is a significantly larger and more complex data-to-text task compared to E2E.
<a id="S0422"></a> Source: p.18 S0422
The dataset is released under the MIT license.
<a id="S0423"></a> Source: p.18 S0423
WebNLG is another commonly used dataset for data-to-text evaluation (Gardent et al., 2017).
<a id="S0424"></a> Source: p.18 S0424
With 22K examples in total WebNLG comprises 14 distinct categories, nine of which are seen during training.
<a id="S0425"></a> Source: p.18 S0425
Since five of the 14 total categories are not seen during training, but are represented in the test set, evaluation is typically broken out by “seen” categories (S), “unseen” categories (U) and “all” (A).
<a id="S0426"></a> Source: p.18 S0426
Each input example is represented by a sequence of SUBJECT — PROPERTY — OBJECT triples.
<a id="S0427"></a> Source: p.18 S0427
The dataset is released under Creative Commons BY-NC-SA 4.0. D HYPERPARAMETERS USED IN EXPERIMENTS D.1 ROBERTA We train using AdamW with a linear learning rate decay schedule.
<a id="S0428"></a> Source: p.18 S0428
We sweep learning rate, number of training epochs, and batch size for LoRA.
<a id="S0429"></a> Source: p.18 S0429
Following Liu et al. (2019), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.
<a id="S0430"></a> Source: p.18 S0430
We report the median over 5 random seeds; the result for each run is taken from the best epoch.
<a id="S0431"></a> Source: p.18 S0431
For a fair comparison with the setup in Houlsby et al. (2019) and Pfeiffer et al. (2021), we restrict the model sequence length to 128 and used a fixed batch size for all tasks.
<a id="S0432"></a> Source: p.18 S0432
Importantly, we start with the pre-trained RoBERTa large model when adapting to MRPC, RTE, and STS-B, instead of a model already adapted to MNLI.
<a id="S0433"></a> Source: p.18 S0433
The runs with this restricted setup are marked with †.
<a id="S0434"></a> Source: p.18 S0434
See the hyperparameters used in our runs in Table 9. D.2 DEBERTA We again train using AdamW with a linear learning rate decay schedule.
<a id="S0435"></a> Source: p.18 S0435
Following He et al. (2021), we tune learning rate, dropout probability, warm-up steps, and batch size.
<a id="S0436"></a> Source: p.18 S0436
We use the same model sequence length used by (He et al., 2021) to keep our comparison fair.
<a id="S0437"></a> Source: p.18 S0437
Following He et al. (2021), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.
<a id="S0438"></a> Source: p.18 S0438
We report the median over 5 random seeds; the result for each run is taken from the best epoch.
<a id="S0439"></a> Source: p.18 S0439
See the hyperparameters used in our runs in Table 10. 18
Page 19
<a id="S0440"></a> Source: p.19 S0440
Method Dataset MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Optimizer AdamW Warmup Ratio 0.06 LR Schedule Linear Batch Size 16 16 16 32 32 16 32 16 # Epochs 30 60 30 80 25 25 80 40 RoBERTa base Learning Rate 5E-04 5E-04 4E-04 4E-04 4E-04 5E-04 5E-04 4E-04 LoRA LoRA Config. r = r = 8 q v LoRA α 8 Max Seq.
<a id="S0441"></a> Source: p.19 S0441
Len. 512 Batch Size 4 4 4 4 4 4 8 8 # Epochs 10 10 20 20 10 20 20 30 RoBERTa large Learning Rate 3E-04 4E-04 3E-04 2E-04 2E-04 3E-04 4E-04 2E-04 LoRA LoRA Config. r = r = 8 q v LoRA α 16 Max Seq.
<a id="S0442"></a> Source: p.19 S0442
Len. 128 128 512 128 512 512 512 512 Batch Size 4 # Epochs 10 10 20 20 10 20 20 10 RoBERTa large Learning Rate 3E-04 4E-04 3E-04 2E-04 2E-04 3E-04 4E-04 2E-04 LoRA† LoRA Config. r = r = 8 q v LoRA α 16 Max Seq.
<a id="S0443"></a> Source: p.19 S0443
Len. 128 Batch Size 32 RoBERTa large # Epochs 10 20 20 20 10 20 20 20 AdptP (3M)† Learning Rate 3E-05 3E-05 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 64 Max Seq.
<a id="S0444"></a> Source: p.19 S0444
Len. 128 Batch Size 32 RoBERTa large # Epochs 5 20 20 20 10 20 20 20 AdptP (0.8M)† Learning Rate 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 16 Max Seq.
<a id="S0445"></a> Source: p.19 S0445
Len. 128 Batch Size 32 RoBERTa large # Epochs 10 5 10 10 5 20 20 10 AdptH (6M)† Learning Rate 3E-05 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 64 Max Seq.
<a id="S0446"></a> Source: p.19 S0446
Len. 128 Batch Size 32 RoBERTa large # Epochs 10 5 10 10 5 20 20 10 AdptH (0.8M)† Learning Rate 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 8 Max Seq.
<a id="S0447"></a> Source: p.19 S0447
Len. 128 Table 9: The hyperparameters we used for RoBERTa on the GLUE benchmark. D.3 GPT-2 We train all of our GPT-2 models using AdamW (Loshchilov & Hutter, 2017) with a linear learning rate schedule for 5 epochs.
<a id="S0448"></a> Source: p.19 S0448
We use the batch size, learning rate, and beam search beam size described in Li & Liang (2021).
<a id="S0449"></a> Source: p.19 S0449
Accordingly, we also tune the above hyperparameters for LoRA.
<a id="S0450"></a> Source: p.19 S0450
We report the mean over 3 random seeds; the result for each run is taken from the best epoch.
<a id="S0451"></a> Source: p.19 S0451
The hyperparameters used for LoRA in GPT-2 are listed in Table 11.
<a id="S0452"></a> Source: p.19 S0452
For those used for other baselines, see Li & Liang (2021). D.4 GPT-3 For all GPT-3 experiments, we train using AdamW (Loshchilov & Hutter, 2017) for 2 epochs with a batch size of 128 samples and a weight decay factor of 0.1.
Page 20
<a id="S0453"></a> Source: p.20 S0453
Method Dataset MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Optimizer AdamW Warmup Ratio 0.1 LR Schedule Linear Batch Size 8 8 32 4 6 8 4 4 # Epochs 5 16 30 10 8 11 11 10 DeBERTa XXL Learning Rate 1E-04 6E-05 2E-04 1E-04 1E-04 1E-04 2E-04 2E-04 LoRA Weight Decay 0 0.01 0.01 0 0.01 0.01 0.01 0.1 CLS Dropout 0.15 0 0 0.1 0.1 0.2 0.2 0.2 LoRA Config. r = r = 8 q v LoRA α 8 Max Seq.
<a id="S0454"></a> Source: p.20 S0454
Len. 256 128 128 64 512 320 320 128 Table 10: The hyperparameters for DeBERTa XXL on tasks included in the GLUE benchmark.
<a id="S0455"></a> Source: p.20 S0455
Dataset E2E WebNLG DART Training Optimizer AdamW Weight Decay 0.01 0.01 0.0 Dropout Prob 0.1 0.1 0.0 Batch Size 8 # Epoch 5 Warmup Steps 500 Learning Rate Schedule Linear Label Smooth 0.1 0.1 0.0 Learning Rate 0.0002 Adaptation r = r = 4 q v LoRA α 32 Inference Beam Size 10 Length Penalty 0.9 0.8 0.8 no repeat ngram size 4 Table 11: The hyperparameters for GPT-2 LoRA on E2E, WebNLG and DART.
<a id="S0456"></a> Source: p.20 S0456
WikiSQL (Zhong et al., 2017), 768 for MNLI (Williams et al., 2018), and 2048 for SAMSum (Gliwa et al., 2019).
<a id="S0457"></a> Source: p.20 S0457
We tune learning rate for all method-dataset combinations.
<a id="S0458"></a> Source: p.20 S0458
See Section D.4 for more details on the hyperparameters used.
<a id="S0459"></a> Source: p.20 S0459
For prefix-embedding tuning, we find the optimal l and l p i to be 256 and 8, respectively, totalling 3.2M trainable parameters.
<a id="S0460"></a> Source: p.20 S0460
We use l = 8 and l = 8 for p i prefix-layer tuning with 20.2M trainable parameters to obtain the overall best performance.
<a id="S0461"></a> Source: p.20 S0461
We present two parameter budgets for LoRA: 4.7M (r = r = 1 or r = 2) and 37.7M (r = r = 8 q v v q v or r = r = r = r = 2).
<a id="S0462"></a> Source: p.20 S0462
We report the best validation performance from each run.
<a id="S0463"></a> Source: p.20 S0463
The training q k v o hyperparameters used in our GPT-3 experiments are listed in Table 12. E COMBINING LORA WITH PREFIX TUNING LoRA can be naturally combined with existing prefix-based approaches.
<a id="S0464"></a> Source: p.20 S0464
In this section, we evaluate two combinations of LoRA and variants of prefix-tuning on WikiSQL and MNLI.
<a id="S0465"></a> Source: p.20 S0465
LoRA+PrefixEmbed (LoRA+PE) combines LoRA with prefix-embedding tuning, where we insert l + l special tokens whose embeddings are treated as trainable parameters.
<a id="S0466"></a> Source: p.20 S0466
For more on prefixp i embedding tuning, see Section 5.1.
<a id="S0467"></a> Source: p.20 S0467
LoRA+PrefixLayer (LoRA+PL) combines LoRA with prefix-layer tuning.
<a id="S0468"></a> Source: p.20 S0468
We also insert l + l p i special tokens; however, instead of letting the hidden representations of these tokens evolve natu- 20
Page 21
<a id="S0469"></a> Source: p.21 S0469
Hyperparameters Fine-Tune PreEmbed PreLayer BitFit AdapterH LoRA Optimizer AdamW Batch Size 128 # Epoch 2 Warmup Tokens 250,000 LR Schedule Linear Learning Rate 5.00E-06 5.00E-04 1.00E-04 1.6E-03 1.00E-04 2.00E-04 Table 12: The training hyperparameters used for different GPT-3 adaption methods.
<a id="S0470"></a> Source: p.21 S0470
We use the same hyperparameters for all datasets after tuning learning rate. rally, we replace them after every Transformer block with an input agnostic vector.
<a id="S0471"></a> Source: p.21 S0471
Thus, both the embeddings and subsequent Transformer block activations are treated as trainable parameters.
<a id="S0472"></a> Source: p.21 S0472
For more on prefix-layer tuning, see Section 5.1.
<a id="S0473"></a> Source: p.21 S0473
In Table 15, we show the evaluation results of LoRA+PE and LoRA+PL on WikiSQL and MultiNLI.
<a id="S0474"></a> Source: p.21 S0474
First of all, LoRA+PE significantly outperforms both LoRA and prefix-embedding tuning on WikiSQL, which indicates that LoRA is somewhat orthogonal to prefix-embedding tuning.
<a id="S0475"></a> Source: p.21 S0475
On MultiNLI, the combination of LoRA+PE doesn’t perform better than LoRA, possibly because LoRA on its own already achieves performance comparable to the human baseline.
<a id="S0476"></a> Source: p.21 S0476
Secondly, we notice that LoRA+PL performs slightly worse than LoRA even with more trainable parameters.
<a id="S0477"></a> Source: p.21 S0477
We attribute this to the fact that prefix-layer tuning is very sensitive to the choice of learning rate and thus makes the optimization of LoRA weights more difficult in LoRA+PL. F ADDITIONAL EMPIRICAL EXPERIMENTS F.1 ADDITIONAL EXPERIMENTS ON GPT-2 We also repeat our experiment on DART (Nan et al., 2020) and WebNLG (Gardent et al., 2017) following the setup of Li & Liang (2021).
<a id="S0478"></a> Source: p.21 S0478
Similar to our result on E2E NLG Challenge, reported in Section 5, LoRA performs better than or at least on-par with prefix-based approaches given the same number of trainable parameters.
<a id="S0479"></a> Source: p.21 S0479
Method # Trainable DART Parameters BLEU↑ MET↑ TER↓ GPT-2 Medium Fine-Tune 354M 46.2 0.39 0.46 AdapterL 0.37M 42.4 0.36 0.48 AdapterL 11M 45.2 0.38 0.46 FTTop2 24M 41.0 0.34 0.56 PrefLayer 0.35M 46.4 0.38 0.46 LoRA 0.35M 47.1 0.39 0.46 ±.2 GPT-2 Large Fine-Tune 774M 47.0 0.39 0.46 AdapterL 0.88M 45.7 0.38 0.46 ±.1 AdapterL 23M 47.1 0.39 0.45 ±.1 PrefLayer 0.77M 46.7 0.38 0.45 LoRA 0.77M 47.5 0.39 0.45 ±.1 Table 13: GPT-2 with different adaptation methods on DART.
<a id="S0480"></a> Source: p.21 S0480
The variances of MET and TER are less than 0.01 for all adaption approaches. 21
Page 22
<a id="S0481"></a> Source: p.22 S0481
Method WebNLG BLEU↑ MET↑ TER↓ U S A U S A U S A GPT-2 Medium Fine-Tune (354M) 27.7 64.2 46.5 .30 .45 .38 .76 .33 .53 AdapterL (0.37M) 45.1 54.5 50.2 .36 .39 .38 .46 .40 .43 AdapterL (11M) 48.3 60.4 54.9 .38 .43 .41 .45 .35 .39 FTTop2 (24M) 18.9 53.6 36.0 .23 .38 .31 .99 .49 .72 Prefix (0.35M) 45.6 62.9 55.1 .38 .44 .41 .49 .35 .40 LoRA (0.35M) 46.7 62.1 55.3 .38 .44 .41 .46 .33 .39 ±.4 ±.2 ±.2 GPT-2 Large Fine-Tune (774M) 43.1 65.3 55.5 .38 .46 .42 .53 .33 .42 AdapterL (0.88M) 49.8 61.1 56.0 .38 .43 .41 .44 .35 .39 ±.0 ±.0 ±.0 AdapterL (23M) 49.2 64.7 57.7 .39 .46 .43 .46 .33 .39 ±.1 ±.2 ±.1 Prefix (0.77M) 47.7 63.4 56.3 .39 .45 .42 .48 .34 .40 LoRA (0.77M) 48.4 64.0 57.0 .39 .45 .42 .45 .32 .38 ±.3 ±.3 ±.1 Table 14: GPT-2 with different adaptation methods on WebNLG.
<a id="S0482"></a> Source: p.22 S0482
The variances of MET and TER are less than 0.01 for all the experiments we ran. “U” indicates unseen categories, “S” indicates seen categories, and “A” indicates all categories in the test set of WebNLG. F.2 ADDITIONAL EXPERIMENTS ON GPT-3 We present additional runs on GPT-3 with different adaptation methods in Table 15.
<a id="S0483"></a> Source: p.22 S0483
The focus is on identifying the trade-off between performance and the number of trainable parameters. F.3 LOW-DATA REGIME To evaluate the performance of different adaptation approaches in the low-data regime. we randomly sample 100, 1k and 10k training examples from the full training set of MNLI to form the low-data MNLI-n tasks.
<a id="S0484"></a> Source: p.22 S0484
In Table 16, we show the performance of different adaptation approaches on MNLIn.
<a id="S0485"></a> Source: p.22 S0485
To our surprise, PrefixEmbed and PrefixLayer performs very poorly on MNLI-100 dataset, with PrefixEmbed performing only slightly better than random chance (37.6% vs. 33.3%).
<a id="S0486"></a> Source: p.22 S0486
PrefixLayer performs better than PrefixEmbed but is still significantly worse than Fine-Tune or LoRA on MNLI- 100.
<a id="S0487"></a> Source: p.22 S0487
The gap between prefix-based approaches and LoRA/Fine-tuning becomes smaller as we increase the number of training examples, which might suggest that prefix-based approaches are not suitable for low-data tasks in GPT-3.
<a id="S0488"></a> Source: p.22 S0488
LoRA achieves better performance than fine-tuning on both MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the (±0.3) variance due to random seeds.
<a id="S0489"></a> Source: p.22 S0489
The training hyperparameters of different adaptation approaches on MNLI-n are reported in Table 17.
<a id="S0490"></a> Source: p.22 S0490
We use a smaller learning rate for PrefixLayer on the MNLI-100 set, as the training loss does not decrease with a larger learning rate. G MEASURING SIMILARITY BETWEEN SUBSPACES In this paper we use the measure φ(A, B, i, j) = ψ(U i , U j ) = (cid:107)U A i(cid:62)UB(cid:107)2 F to measure the subspace A B min{i,j} similarity between two column orthonormal matrices U i ∈ Rd×i and U j ∈ Rd×j, obtained by A B taking columns of the left singular matrices of A and B.
<a id="S0491"></a> Source: p.22 S0491
We point out that this similarity is simply a reverse of the standard Projection Metric that measures distance between subspaces Ham & Lee (2008). 22
Page 23
<a id="S0492"></a> Source: p.23 S0492
Method Hyperparameters # Trainable Parameters WikiSQL MNLI-m Fine-Tune - 175B 73.8 89.5 l = 32, l = 8 0.4 M 55.9 84.9 p i l = 64, l = 8 0.9 M 58.7 88.1 p i PrefixEmbed l = 128, l = 8 1.7 M 60.6 88.0 p i l = 256, l = 8 3.2 M 63.1 88.6 p i l = 512, l = 8 6.4 M 55.9 85.8 p i l = 2, l = 2 5.1 M 68.5 89.2 p i l = 8, l = 0 10.1 M 69.8 88.2 p i PrefixLayer l = 8, l = 8 20.2 M 70.1 89.5 p i l = 32, l = 4 44.1 M 66.4 89.6 p i l = 64, l = 0 76.1 M 64.9 87.9 p i r = 1 7.1 M 71.9 89.8 r = 4 21.2 M 73.2 91.0 AdapterH r = 8 40.1 M 73.2 91.5 r = 16 77.9 M 73.2 91.5 r = 64 304.4 M 72.6 91.5 r = 2 4.7 M 73.4 91.7 v r = r = 1 4.7 M 73.4 91.3 q v LoRA r = r = 2 9.4 M 73.3 91.4 q v r = r = r = r = 1 9.4 M 74.1 91.2 q k v o r = r = 4 18.8 M 73.7 91.3 q v r = r = r = r = 2 18.8 M 73.7 91.7 q k v o r = r = 8 37.7 M 73.8 91.6 q v r = r = r = r = 4 37.7 M 74.0 91.7 q k v o r = r = 64 301.9 M 73.6 91.4 q v r = r = r = r = 64 603.8 M 73.9 91.4 q k v o r = r = 8, l = 8, l = 4 37.8 M 75.0 91.4 q v p i LoRA+PE r = r = 32, l = 8, l = 4 151.1 M 75.9 91.1 q v p i r = r = 64, l = 8, l = 4 302.1 M 76.2 91.3 q v p i LoRA+PL r = r = 8, l = 8, l = 4 52.8 M 72.9 90.2 q v p i Table 15: Hyperparameter analysis of different adaptation approaches on WikiSQL and MNLI.
<a id="S0493"></a> Source: p.23 S0493
Both prefix-embedding tuning (PrefixEmbed) and prefix-layer tuning (PrefixLayer) perform worse as we increase the number of trainable parameters, while LoRA’s performance stabilizes.
<a id="S0494"></a> Source: p.23 S0494
Performance is measured in validation accuracy.
<a id="S0495"></a> Source: p.23 S0495
Method MNLI(m)-100 MNLI(m)-1k MNLI(m)-10k MNLI(m)-392K GPT-3 (Fine-Tune) 60.2 85.8 88.9 89.5 GPT-3 (PrefixEmbed) 37.6 75.2 79.5 88.6 GPT-3 (PrefixLayer) 48.3 82.5 85.9 89.6 GPT-3 (LoRA) 63.8 85.6 89.2 91.7 Table 16: Validation accuracy of different methods on subsets of MNLI using GPT-3 175B.
<a id="S0496"></a> Source: p.23 S0496
MNLIn describes a subset with n training examples.
<a id="S0497"></a> Source: p.23 S0497
We evaluate with the full validation set.
<a id="S0498"></a> Source: p.23 S0498
LoRA performs exhibits favorable sample-efficiency compared to other methods, including fine-tuning.
<a id="S0499"></a> Source: p.23 S0499
To be concrete, let the singular values of U i(cid:62)U j to be σ , σ , · · · , σ where p = min{i, j}.
<a id="S0500"></a> Source: p.23 S0500
We A B 1 2 p know that the Projection Metric Ham & Lee (2008) is defined as: (cid:118) (cid:117) p d(U i , U j ) = (cid:117) (cid:116)p − (cid:88) σ2 ∈ [0, √ p] A B i i=1 23
Page 24
<a id="S0501"></a> Source: p.24 S0501
Hyperparameters Adaptation MNLI-100 MNLI-1k MNLI-10K MNLI-392K Optimizer - AdamW Warmup Tokens - 250,000 LR Schedule - Linear Batch Size - 20 20 100 128 # Epoch - 40 40 4 2 FineTune 5.00E-6 PrefixEmbed 2.00E-04 2.00E-04 4.00E-04 5.00E-04 Learning Rate PrefixLayer 5.00E-05 5.00E-05 5.00E-05 1.00E-04 LoRA 2.00E-4 PrefixEmbed l 16 32 64 256 p Adaptation- PrefixEmbed l 8 i Specific PrefixTune l = l = 8 p i LoRA r = r = 8 q v Table 17: The hyperparameters used for different GPT-3 adaptation methods on MNLI(m)-n. where our similarity is defined as: (cid:80)p σ2 1 (cid:16) (cid:17) φ(A, B, i, j) = ψ(U i , U j ) = i=1 i = 1 − d(U i , U j )2 A B p p A B This similarity satisfies that if U i and U j share the same column span, then φ(A, B, i, j) = 1.
<a id="S0502"></a> Source: p.24 S0502
If A B they are completely orthogonal, then φ(A, B, i, j) = 0.
<a id="S0503"></a> Source: p.24 S0503
Otherwise, φ(A, B, i, j) ∈ (0, 1). H ADDITIONAL EXPERIMENTS ON LOW-RANK MATRICES We present additional results from our investigation into the low-rank update matrices. H.1 CORRELATION BETWEEN LORA MODULES See Figure 6 and Figure 7 for how the results presented in Figure 3 and Figure 4 generalize to other layers. H.2 EFFECT OF r ON GPT-2 We repeat our experiment on the effect of r (Section 7.2) in GPT-2.
<a id="S0504"></a> Source: p.24 S0504
Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices of r after training for 26,000 steps.
<a id="S0505"></a> Source: p.24 S0505
The optimal rank for GPT-2 Medium is between 4 and 16 depending on the metric used, which is similar to that for GPT-3 175B.
<a id="S0506"></a> Source: p.24 S0506
Note that the relationship between model size and the optimal rank for adaptation is still an open question. H.3 CORRELATION BETWEEN W AND ∆W See Figure 8 for the normalized subspace similarity between W and ∆W with varying r.
<a id="S0507"></a> Source: p.24 S0507
Note again that ∆W does not contain the top singular directions of W , since the similarity between the top 4 directions in ∆W and the top-10% of those in W barely exceeds 0.2.
<a id="S0508"></a> Source: p.24 S0508
This gives evidence that ∆W contains those “task-specific” directions that are otherwise not emphasized in W .
<a id="S0509"></a> Source: p.24 S0509
An interesting next question to answer, is how “strong” do we need to amplify those task-specific directions, in order for the model adaptation to work well? 24
Page 25
<a id="S0510"></a> Source: p.25 S0510
1.0 0.8 0.6 0.4 0.2 0.0 1 reyaL i 1 2 3 4 5 6 7 8 Wq Wv Wq Wv 23 reyaL 46 reyaL i i 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 6 21 81 32 92 53 04 64 25 85 j 69 reyaL i 1 2 3 4 5 6 7 8 1 6 21 81 32 92 53 04 64 25 85 (Ar = 8, Ar = 64, i, j) 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 j j j Figure 6: Normalized subspace similarity between the column vectors of A and A for both r=8 r=64 ∆W and ∆W from the 1st, 32nd, 64th, and 96th layers in a 96-layer Transformer. q v H.4 AMPLIFICATION FACTOR One can naturally consider a feature amplification factor as the ratio (cid:107)∆W (cid:107)F , where U and V (cid:107)U(cid:62)W V (cid:62)(cid:107)F are the left- and right-singular matrices of the SVD decomposition of ∆W . (Recall U U (cid:62)W V (cid:62)V gives the “projection” of W onto the subspace spanned by ∆W .) Intuitively, when ∆W mostly contains task-specific directions, this quantity measures how much of them are amplified by ∆W .
<a id="S0511"></a> Source: p.25 S0511
As shown in Section 7.3, for r = 4, this amplification factor is as large as 20.
<a id="S0512"></a> Source: p.25 S0512
In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained model W ), that need to be amplified by a very large factor 20, in order to achieve our reported accuracy for the downstream specific task.
<a id="S0513"></a> Source: p.25 S0513
And, one should expect a very different set of feature directions to be amplified for each different downstream task.
<a id="S0514"></a> Source: p.25 S0514
One may notice, however, for r = 64, this amplification factor is only around 2, meaning that most directions learned in ∆W with r = 64 are not being amplified by much.
<a id="S0515"></a> Source: p.25 S0515
This should not be surprising, and in fact gives evidence (once again) that the intrinsic rank needed to represent the “task-specific directions” (thus for model adaptation) is low.
<a id="S0516"></a> Source: p.25 S0516
In contrast, those directions in the rank-4 version of ∆W (corresponding to r = 4) are amplified by a much larger factor 20. 25
Page 26
<a id="S0517"></a> Source: p.26 S0517
1 7 13 19 0.8 25 31 0.7 37 43 0.6 49 55 0.5 61 0.4 0.3 0.2 0.1 0.0 1 reyaL i Wq Wv 23 reyaL Wq Wv 1 6 11 61 12 62 13 63 14 64 15 65 16 1 7 13 19 25 31 37 43 49 55 61 j 46 reyaL i 1 6 11 61 12 62 13 63 14 64 15 65 16 j 1 6 11 61 12 62 13 63 14 64 15 65 16 j 69 reyaL 1 6 11 61 12 62 13 63 14 64 15 65 16 (Ar=64, A0r=64, i, j) j Figure 7: Normalized subspace similarity between the column vectors of A from two randomly r=64 seeded runs, for both ∆W and ∆W from the 1st, 32nd, 64th, and 96th layers in a 96-layer Transq v former.
<a id="S0518"></a> Source: p.26 S0518
Rank r val loss BLEU NIST METEOR ROUGE L CIDEr 1 1.23 68.72 8.7215 0.4565 0.7052 2.4329 2 1.21 69.17 8.7413 0.4590 0.7052 2.4639 4 1.18 70.38 8.8439 0.4689 0.7186 2.5349 8 1.17 69.57 8.7457 0.4636 0.7196 2.5196 16 1.16 69.61 8.7483 0.4629 0.7177 2.4985 32 1.16 69.33 8.7736 0.4642 0.7105 2.5255 64 1.16 69.24 8.7174 0.4651 0.7180 2.5070 128 1.16 68.73 8.6718 0.4628 0.7127 2.5030 256 1.16 68.92 8.6982 0.4629 0.7128 2.5012 512 1.16 68.78 8.6857 0.4637 0.7128 2.5025 1024 1.17 69.37 8.7495 0.4659 0.7149 2.5090 Table 18: Validation loss and test set metrics on E2E NLG Challenge achieved by LoRA with different rank r using GPT-2 Medium.
<a id="S0519"></a> Source: p.26 S0519
Unlike on GPT-3 where r = 1 suffices for many tasks, here the performance peaks at r = 16 for validation loss and r = 4 for BLEU, suggesting the GPT-2 Medium has a similar intrinsic rank for adaptation compared to GPT-3 175B.
<a id="S0520"></a> Source: p.26 S0520
Note that some of our hyperparameters are tuned on r = 4, which matches the parameter count of another baseline, and thus might not be optimal for other choices of r. 451 0.200 555 658 0.175 762 0.150 865 969 0.125 1072 0.100 1176 j i Wq Random (Wq, Ar = 4, i, j) (Wq, Ar = 8, i, j) (Wq, Ar = 64, i, j) (Wq, Arand, i, j) j j j Figure 8: Normalized subspace similarity between the singular directions of W and those of ∆W q q with varying r and a random baseline. ∆W amplifies directions that are important but not emphaq sized in W . ∆W with a larger r tends to pick up more directions that are already emphasized in W . 26