LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen - 中英文对照
LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen - 中英文对照
中英文对照
<a id="S0001"></a> Source: p.1 S0001
Original: LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS Edward Hu∗ Yelong Shen∗ Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen Microsoft Corporation {edwardhu, yeshe, phwallis, zeyuana, yuanzhil, swang, luw, wzchen}@microsoft.com yuanzhil@andrew.cmu.edu (Version 2) ABSTRACT An important paradigm of natural language processing consists of large-scale pretraining on general domain data and adaptation to particular tasks or domains.
中文: LORA: LARGE LAN-GUAGE MODELS Edward Hu Yelong Shen 菲利普·沃里斯·泽于安·艾伦-朱·袁日 李雪. 王·卢·王·魏祖陈微软公司 {edwardhu, yeshe, phallis, zeyuana, 人民币zhil, swang, luw, wzchen microsoft.com unzhil@andrew.cmu.edu (Version 2) ABSTRAT公司 自然语言处理的一个重要范式包括针对一般域数据的大规模预训并适应特定任务或域.
<a id="S0002"></a> Source: p.1 S0002
Original: As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible.
中文: 随着我们预先训练出更大的模型,完全的微调,将所有模型参数重排,变得不太可行.
<a id="S0003"></a> Source: p.1 S0003
Original: Using GPT-3 175B as an example – deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive.
中文: 以GPT-3 175B为例 — — 部署独立的有175B参数的微调模型实例是昂贵的。
<a id="S0004"></a> Source: p.1 S0004
Original: We propose Low-Rank Adaptation, or LoRA, which freezes the pretrained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks.
中文: 我们提出"低Rank适应"(Low-Rank Adaptation),即"LORA"(LORA),它冻结了预先训练过的模型重量,并将可训练的军衔分解矩阵注入"变形器"建筑的每一层,极大地减少了下游任务的可训练参数的数量.
<a id="S0005"></a> Source: p.1 S0005
Original: Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times.
中文: 相较于与亚当进行GPT-3 175B的微调,LORA可以将可列车参数数量减少一万人次并降低GPU内存要求3倍.
<a id="S0006"></a> Source: p.1 S0006
Original: LoRA performs on-par or better than finetuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency.
中文: LoRA在RoBERTa、DeBERTa、GPT-2和GPT-3上的模型质量进行平面或优于微调,尽管可训练参数较少,培训吞吐量较高,而且与适配器不同,没有额外的推断延迟。
<a id="S0007"></a> Source: p.1 S0007
Original: We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA.
中文: 我们还对语言模型改编中的军衔不足进行了实证调查,揭示了LORA的功效.
<a id="S0008"></a> Source: p.1 S0008
Original: We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA. 1 INTRODUCTION Many applications in natural language processing rely on adaptf(x) ing one large-scale, pre-trained language model to multiple down- h stream applications.
中文: 我们在https://github.com/microsoft/LORA. 1的网址上发布了一个软件包,促进LORA与PyTorch模型的整合,并为RoBERTa、DeBERTa和GPT-2提供我们的执行和示范检查站。 许多在自然语言处理中的应用依赖于适配f(x)将一个大规模,预先训练的语言模型接入了多个下-h流应用.
<a id="S0009"></a> Source: p.1 S0009
Original: Such adaptation is usually done via fine-tuning, which updates all the parameters of the pre-trained model.
中文: 这种改造通常通过微调完成,微调更新了预训模型的所有参数.
<a id="S0010"></a> Source: p.1 S0010
Original: The ma- Pretrained jor downside of fine-tuning is that the new model contains as many Pretrained 𝐵 = 0 Weights parameters as in the original model.
中文: 微调的ma-Pretrained jor下行是新型号包含与原型号一样多的Pretrained B = 0重量参数.
<a id="S0011"></a> Source: p.1 S0011
Original: As larger models are trained Weights 𝑟 every few months, this changes from a mere “inconvenience” for 𝑊 ∈ ℝ𝑑×𝑑 𝑊 ∈ ℝ𝑑×𝑑 GPT-2 (Radford et al., b) or RoBERTa large (Liu et al., 2019) to a 𝐴 = 𝒩(0, 𝜎2) critical deployment challenge for GPT-3 (Brown et al., 2020) with 𝑑 𝑑 175 billion trainable parameters.1 x x Many sought to mitigate this by adapting only some parameters or Figure 1: Our reparametrizalearning external modules for new tasks.
中文: 由于大型型号每几个月培训一次重量,这从W ∈ Rd×d W ∈ Rd×d GPT-2(拉德福德等,b)或RoBERTa大(Liu等,2019)的“不方便”变为GPT-3(Brown等,2020年)的A = N(0, 6.2)关键部署挑战(d 175亿可列车参数)。 1xx 许多人试图通过仅调整一些参数或图1:我们为新任务重新校正外部模块来缓解这种情况。
<a id="S0012"></a> Source: p.1 S0012
Original: We only train A and B. to store and load a small number of task-specific parameters in addition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed.
中文: 我们仅训练A和B,除了预先训练的每个任务模型外,储存并装入少量任务特定参数,极大地提升了部署时的业务效率.
<a id="S0013"></a> Source: p.1 S0013
Original: However, existing techniques ∗Equal contribution. 0Compared to V1, this draft includes better baselines, experiments on GLUE, and more on adapter latency. 1While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its performance significantly as shown in Appendix A. 1 1202 tcO 61 ]LC.sc[ 2v58690.6012:viXra
中文: 但是,现有技术 * 平等贡献。 0相比于V1,本草案包括更好的基线,GLUE上的实验,以及更多的适配器延迟. 1 虽然GPT-3 175B以少发学习方式实现非三角性能,但微调大大提升了它的性能,如附录A. 1 1202 tcO 61]LC.sc [2v58690.6012:viXra.
<a id="S0014"></a> Source: p.2 S0014
Original: often introduce inference latency (Houlsby et al., 2019; Rebuffi et al., 2017) by extending model depth or reduce the model’s usable sequence length (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021) (Section 3).
中文: 经常通过扩展模型深度或减少模型可用的序列长度引入推论纬度(Houlsby等,2019;Rebuffi等,2017年)(Li & Liang,2021;Lester等,2021;Hambardzumyan等,2020;Liu等,2021) (第3节).
<a id="S0015"></a> Source: p.2 S0015
Original: More importantly, these method often fail to match the fine-tuning baselines, posing a trade-off between efficiency and model quality.
中文: 更重要的是,这些方法往往不能与微调基线相匹配,在效率和模型质量之间造成了取舍。
<a id="S0016"></a> Source: p.2 S0016
Original: We take inspiration from Li et al. (2018a); Aghajanyan et al. (2020) which show that the learned over-parametrized models in fact reside on a low intrinsic dimension.
中文: 我们从Li等人(2018年a);Aghajanyan等人(2020年)那里得到启发,这些启发表明,学到的超参数模型事实上存在于低等内在维度上。
<a id="S0017"></a> Source: p.2 S0017
Original: We hypothesize that the change in weights during model adaptation also has a low “intrinsic rank”, leading to our proposed Low-Rank Adaptation (LoRA) approach.
中文: 我们假设,模型适应期间的权重变化也具有较低的“内在地位”,导致我们提议的低Rank适应办法。
<a id="S0018"></a> Source: p.2 S0018
Original: LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure 1.
中文: LoRA允许我们在神经网络中间接地训练出一些密集地层,方法是在适应过程中优化密集地层变化的分级矩阵,同时保持预受训练的重量被冻结,如图一所示.
<a id="S0019"></a> Source: p.2 S0019
Original: Using GPT-3 175B as an example, we show that a very low rank (i.e., r in Figure 1 can be one or two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute-efficient.
中文: 以GPT-3 175B为例,我们显示一个非常低的军衔(即图一中的r可以是一到两个)就足够了,即使全军衔(即d)高达12,288,使得LORA既能存储又能计算高效.
<a id="S0020"></a> Source: p.2 S0020
Original: LoRA possesses several key advantages. • A pre-trained model can be shared and used to build many small LoRA modules for different tasks.
中文: LORA拥有若干关键优势. ^ 可以共享并使用经过预先训练的模型来为不同的任务构建许多小型的LORA模块.
<a id="S0021"></a> Source: p.2 S0021
Original: We can freeze the shared model and efficiently switch tasks by replacing the matrices A and B in Figure 1, reducing the storage requirement and task-switching overhead significantly. • LoRA makes training more efficient and lowers the hardware barrier to entry by up to 3 times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters.
中文: 我们可以通过替换图1中的矩阵A和B来冻结共享模型并高效地切换任务,大大减少存储需要和任务切换管理费用. · LoRA使训练更有效率,在使用适应性优化器时将硬件进入屏障降低至3倍,因为我们不需要计算梯度或维持大多数参数的优化状态.
<a id="S0022"></a> Source: p.2 S0022
Original: Instead, we only optimize the injected, much smaller low-rank matrices. • Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing no inference latency compared to a fully fine-tuned model, by construction. • LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning.
中文: 相反,我们只是优化了 注射量小得多的低级矩阵。 二. 支助 我们的简单线性设计使我们能够在部署时将可被训练的基质与被冻结的重量合并起来,比起一个完全微调的模型,通过施工,引入了没有推论延迟. ^ LoRA对许多前作方法是正交的,可以与许多方法结合,如前作调和.
<a id="S0023"></a> Source: p.2 S0023
Original: Terminologies and Conventions We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions.
中文: 术语和公约 我们经常提到 " 变形器 " 架构,并使用传统术语来说明其层面。
<a id="S0024"></a> Source: p.2 S0024
Original: We call the input and output dimension size of a Transformer layer d .
中文: 我们称之为变形器d的输入和输出维度大小。
<a id="S0025"></a> Source: p.2 S0025
Original: We use W , W , W , and W to refer to the model q k v o query/key/value/output projection matrices in the self-attention module. W or W refers to a pre- 0 trained weight matrix and ∆W its accumulated gradient update during adaptation.
中文: 我们使用 W , W , W 和 W 来指代自注意模块中的 q k v o 查询/ key/值/输出投影矩阵. W或W指0前训练的重量矩阵和XQW在适应过程中的累积梯度更新.
<a id="S0026"></a> Source: p.2 S0026
Original: We use r to denote the rank of a LoRA module.
中文: 我们用r表示LORA模块的级别.
<a id="S0027"></a> Source: p.2 S0027
Original: We follow the conventions set out by (Vaswani et al., 2017; Brown et al., 2020) and use Adam (Loshchilov & Hutter, 2019; Kingma & Ba, 2017) for model optimization and use a Transformer MLP feedforward dimension d = 4 × d . ffn model 2 PROBLEM STATEMENT While our proposal is agnostic to training objective, we focus on language modeling as our motivating use case.
中文: 我们遵循(Vaswani等,2017年;Brown等,2020年)所制定的公约,并使用Adam(Loshchilov和Hutter,2019年;Kingma和Ba,2017年)进行模型优化,并使用变形器MLPforward维度d=4×d.ffn model 2 ProbleM语句. 虽然我们的建议对培训目标是不可知的,但我们侧重于语文模式,作为我们激励使用的案例。
<a id="S0028"></a> Source: p.2 S0028
Original: Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a task-specific prompt.
中文: 下文简要介绍语言建模问题,特别是给特定任务提示的有条件概率最大化.
<a id="S0029"></a> Source: p.2 S0029
Original: Suppose we are given a pre-trained autoregressive language model P (y|x) parametrized by Φ. Φ For instance, P (y|x) can be a generic multi-task learner such as GPT (Radford et al., b; Brown Φ et al., 2020) based on the Transformer architecture (Vaswani et al., 2017).
中文: 假设我们得到一个经过预先训练的自转式语言模型P(y|x)由 Φ 作参数化. 例如, P(y|x)可以是一个通用的多任务学习者,比如GPT(Radford等,b;Brown等,2020年),基于变形器架构(Vaswani等,2017年).
<a id="S0030"></a> Source: p.2 S0030
Original: Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language to SQL (NL2SQL).
中文: 考虑将这个预先训练的模型适应下游有条件的文本生成任务,如归纳,机器读取理解(MRC),以及自然语言到SQL(NL2SQL).
<a id="S0031"></a> Source: p.2 S0031
Original: Each downstream task is represented by a training dataset of context-target pairs: Z = {(x , y )} , where both x and i i i=1,..,N i y are sequences of tokens.
中文: 每个下游任务都由上下文-目标对的训练数据集来代表: Z = {(x,y)},其中x和i i = 1.,N i y都是指代的序列.
<a id="S0032"></a> Source: p.2 S0032
Original: For example, in NL2SQL, x is a natural language query and y its i i i corresponding SQL command; for summarization, x is the content of an article and y its summary. i i 2
中文: 例如,在NL2SQL中,x是一个自然语言查询,而它的iii对应的SQL命令;对于归纳,x是一篇文章的内容和y的汇总. (单位:千美元)
<a id="S0033"></a> Source: p.3 S0033
Original: During full fine-tuning, the model is initialized to pre-trained weights Φ and updated to Φ + ∆Φ 0 0 by repeatedly following the gradient to maximize the conditional language modeling objective: |y| (cid:88) (cid:88) max log (P (y |x, y )) (1) Φ t <t Φ (x,y)∈Z t=1 One of the main drawbacks for full fine-tuning is that for each downstream task, we learn a different set of parameters ∆Φ whose dimension |∆Φ| equals |Φ |.
中文: 在全面微调期间,该模型被初始化为前训练的权重QQQ并被更新到QQ+0,通过反复按照梯度来达到条件语言模型的最大化目标:QQYQ(cid:88)(cid:88)最大对数(P(y |x,y)) (1)QQt <t (x,y)∈Zt=1 充分微调的主要缺点是,对于每一项下游任务,我们学习了一套不同的参数QQQ,其维度等同QQQQ.
<a id="S0034"></a> Source: p.3 S0034
Original: Thus, if the pre-trained model is large 0 (such as GPT-3 with |Φ | ≈ 175 Billion), storing and deploying many independent instances of 0 fine-tuned models can be challenging, if at all feasible.
中文: 因此,如果预训型号为"0"(如GPT-3有QQ 175亿分之分),那么存储和部署许多独立型号为"0"的"微调型号"的"0"(微调型号)可能具有挑战性,如果可行的话.
<a id="S0035"></a> Source: p.3 S0035
Original: In this paper, we adopt a more parameter-efficient approach, where the task-specific parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with |Θ| (cid:28) |Φ |.
中文: 在本文中,我们采用了一个更具参数效率的方法,即任务特定参数加量 = = = (+) 由一组较小的参数与 = (cid:28) = = (+) 编码。
<a id="S0036"></a> Source: p.3 S0036
Original: The task of finding ∆Φ thus becomes optimizing over Θ: 0 |y| (cid:88) (cid:88) (cid:0) (cid:1) max log p (y |x, y ) (2) Θ Φ0+∆Φ(Θ) t <t (x,y)∈Z t=1 In the subsequent sections, we propose to use a low-rank representation to encode ∆Φ that is both compute- and memory-efficient.
中文: 因此,查找 QQ 的任务会比 : 0 = = === (代码: 88 (cid: o) (cid: 1) 匹配的日志 p (y ==x, y) (2) ===0 == (t (x, y) === 1) 在接下来的章节中,我们提议使用低等代表来编码QQ,这既具有计算效率,又具有内存效率.
<a id="S0037"></a> Source: p.3 S0037
Original: When the pre-trained model is GPT-3 175B, the number of trainable parameters |Θ| can be as small as 0.01% of |Φ |. 0 3 AREN’T EXISTING SOLUTIONS GOOD ENOUGH?
中文: 当训练前的型号是GPT-3 175B时,可训练参数QQ的数量可以小到QQ的0.01%. 0 3 AREN'T ExSISTING SOLUTIONS GOOW Enough?
<a id="S0038"></a> Source: p.3 S0038
Original: The problem we set out to tackle is by no means new.
中文: 我们所要解决的问题绝非新问题。
<a id="S0039"></a> Source: p.3 S0039
Original: Since the inception of transfer learning, dozens of works have sought to make model adaptation more parameter- and compute-efficient.
中文: 自转会学习开始以来,有数十部作品寻求使模型适应更加参数化和计算效率.
<a id="S0040"></a> Source: p.3 S0040
Original: See Section 6 for a survey of some of the well-known works.
中文: 关于一些有名的作品的调查,见第6节.
<a id="S0041"></a> Source: p.3 S0041
Original: Using language modeling as an example, there are two prominent strategies when it comes to efficient adaptations: adding adapter layers (Houlsby et al., 2019; Rebuffi et al., 2017; Pfeiffer et al., 2021; Ru¨ckle´ et al., 2020) or optimizing some forms of the input layer activations (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021).
中文: 以语言模型为例,在高效改造方面有两种突出策略:增加适配层(Houlsby等,2019;Rebuffi等,2017;Pfeiffer等,2021;Ru'cle'等,2020)或优化某些形式的输入层活化(Li & Liang,2021;Lester等,2021;Hambardzumyan等,2020;Liu等,2021).
<a id="S0042"></a> Source: p.3 S0042
Original: However, both strategies have their limitations, especially in a large-scale and latency-sensitive production scenario.
中文: 然而,这两种战略都有其局限性,特别是在大规模和长期性敏感的生产情景中。
<a id="S0043"></a> Source: p.3 S0043
Original: Adapter Layers Introduce Inference Latency There are many variants of adapters.
中文: 适应层引入推论纬度 适配器有很多变种.
<a id="S0044"></a> Source: p.3 S0044
Original: We focus on the original design by Houlsby et al. (2019) which has two adapter layers per Transformer block and a more recent one by Lin et al. (2020) which has only one per block but with an additional LayerNorm (Ba et al., 2016).
中文: 我们专注于由Houlsby等人(2019年)设计的最初设计,每个变形板块有两个适配层,而最近由Lin等人(2020年)设计,每个区块只有一个,但又增加了一个图层Norm(Ba等人,2016年)。
<a id="S0045"></a> Source: p.3 S0045
Original: While one can reduce the overall latency by pruning layers or exploiting multi-task settings (Ru¨ckle´ et al., 2020; Pfeiffer et al., 2021), there is no direct ways to bypass the extra compute in adapter layers.
中文: 虽然人们可以通过推倒地层或利用多任务设置来减少总体延迟(Ru'cle'等,2020年;Pfeiffer等,2021年),但没有直接绕过适配器地层中的额外计算.
<a id="S0046"></a> Source: p.3 S0046
Original: This seems like a non-issue since adapter layers are designed to have few parameters (sometimes <1% of the original model) by having a small bottleneck dimension, which limits the FLOPs they can add.
中文: 这似乎是一个非问题,因为适配器层被设计成只有很少的参数(有时为原模型的 <1%),因为有一个小的瓶颈维度,限制了它们可以添加的FLOP.
<a id="S0047"></a> Source: p.3 S0047
Original: However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially.
中文: 然而,大型神经网络依赖于硬件并行性来保持潜伏性低,适配层必须相继处理.
<a id="S0048"></a> Source: p.3 S0048
Original: This makes a difference in the online inference setting where the batch size is typically as small as one.
中文: 这在通常批量大小为一等分量的在线推论设置上有所区别.
<a id="S0049"></a> Source: p.3 S0049
Original: In a generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al., b) medium on a single GPU, we see a noticeable increase in latency when using adapters, even with a very small bottleneck dimension (Table 1).
中文: 在一个没有模型平行性的通用情景中,例如运行对GPT-2的推论(Radford等人,b)介质在一个单一的GPU上,我们看到使用适配器时的耐受度明显增加,即使有很小的瓶颈维度(表一).
<a id="S0050"></a> Source: p.3 S0050
Original: This problem gets worse when we need to shard the model as done in Shoeybi et al. (2020); Lepikhin et al. (2020), because the additional depth requires more synchronous GPU operations such as AllReduce and Broadcast, unless we store the adapter parameters redundantly many times.
中文: 这个问题在我们需要像Shoeybi等人(2020年);Lepikhin等人(2020年)所做的那样硬化模型时会变得更加严重,因为额外的深度需要AllReduce和Broadcast等更同步的GPU操作,除非我们多次重复存储适配器参数.
<a id="S0051"></a> Source: p.3 S0051
Original: Directly Optimizing the Prompt is Hard The other direction, as exemplified by prefix tuning (Li & Liang, 2021), faces a different challenge.
中文: 直接优化提示很困难 另一个方向,以前缀调音为例(Li & Liang,2021),面临着不同的挑战.
<a id="S0052"></a> Source: p.3 S0052
Original: We observe that prefix tuning is difficult to optimize and that its performance changes non-monotonically in trainable parameters, confirming similar observations in the original paper.
中文: 我们观察到前缀调音难以优化,其性能在可训练参数中发生非莫名的变化,证实了原始纸中的类似观察.
<a id="S0053"></a> Source: p.3 S0053
Original: More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available to process a downstream task, which we suspect makes tuning the prompt less performant compared to other methods.
中文: 更根本的是,为适应而保留一部分序列长度必然会减少可用于处理下游任务的序列长度,我们怀疑这会使快取的调试比起其他方法来更不起作用.
<a id="S0054"></a> Source: p.3 S0054
Original: We defer the study on task performance to Section 5. 3
中文: 我们把任务执行情况的研究推迟到第5节3。
<a id="S0055"></a> Source: p.4 S0055
Original: Batch Size 32 16 1 Sequence Length 512 256 128 |Θ| 0.5M 11M 11M Fine-Tune/LoRA 1449.4±0.8 338.0±0.6 19.8±2.7 AdapterL 1482.0±1.0 (+2.2%) 354.8±0.5 (+5.0%) 23.9±2.1 (+20.7%) AdapterH 1492.2±1.0 (+3.0%) 366.3±0.5 (+8.4%) 25.8±2.2 (+30.3%) Table 1: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, averaged over 100 trials.
中文: 批量尺寸 32 16 1 序列长度 512 256 128 * 0.5M 11M Fine-Tune/LORA 1449.4±08 338.0±0.6 19.8±2.7 适配器L 1482.0±1.0 (+2.2%) 354.8±0.5 (+5.0%) 23.9±2.1 (+20.7%) 适配器H 149.2.2±1.0 (+3.0%) 366.3±05 (+8.4%) 25.8±2.2 (+30.3%) 表1:以毫秒计的GPT-2介质中单次前行传入的强度,平均100多起试验.
<a id="S0056"></a> Source: p.4 S0056
Original: We use an NVIDIA Quadro RTX8000. “|Θ|” denotes the number of trainable parameters in adapter layers.
中文: 我们使用NVIDIA Quadro RTX8000。“|Θ|”表示适配器层中可训练参数的数量。
<a id="S0057"></a> Source: p.4 S0057
Original: AdapterL and AdapterH are two variants of adapter tuning, which we describe in Section 5.1.
中文: 适配器L和适配器H是适配器调音的两个变体,我们在第5.1节中对此进行了描述.
<a id="S0058"></a> Source: p.4 S0058
Original: The inference latency introduced by adapter layers can be significant in an online, short-sequence-length scenario.
中文: 适配器层引入的推论纬度在在线,短序列-长场景中可以显著.
<a id="S0059"></a> Source: p.4 S0059
Original: See the full study in Appendix B. 4 OUR METHOD We describe the simple design of LoRA and its practical benefits.
中文: 见附录B。 我们描述了LORA的简单设计及其实际好处.
<a id="S0060"></a> Source: p.4 S0060
Original: The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case. 4.1 LOW-RANK-PARAMETRIZED UPDATE MATRICES A neural network contains many dense layers which perform matrix multiplication.
中文: 本文概述的原则适用于深层学习模型中的任何密集层,尽管我们只注重在实验中的变形语模型中的某些分量,作为激励使用的案例. 4.1 低射程-射程更新材料 神经网络包含许多密集层,进行矩阵乘法.
<a id="S0061"></a> Source: p.4 S0061
Original: The weight matrices in these layers typically have full-rank.
中文: 这些地层中的权重矩阵一般都有全等分.
<a id="S0062"></a> Source: p.4 S0062
Original: When adapting to a specific task, Aghajanyan et al. (2020) shows that the pre-trained language models have a low “instrisic dimension” and can still learn efficiently despite a random projection to a smaller subspace.
中文: Aghajanyan等人(2020年)在适应一项具体任务时指出,经过预先训练的语言模型具有较低的“intrisic维度”,尽管随机投射到一个较小的子空间,但仍能有效地学习。
<a id="S0063"></a> Source: p.4 S0063
Original: Inspired by this, we hypothesize the updates to the weights also have a low “intrinsic rank” during adaptation.
中文: 受此启发,我们假设,在适应过程中,对重量的更新也具有较低的“内在排名”。
<a id="S0064"></a> Source: p.4 S0064
Original: For a pre-trained weight matrix W ∈ Rd×k, we constrain its update by representing the latter with a low-rank de- 0 composition W + ∆W = W + BA, where B ∈ Rd×r, A ∈ Rr×k, and the rank r (cid:28) min(d, k). 0 0 During training, W is frozen and does not receive gradient updates, while A and B contain trainable 0 parameters.
中文: 对于预训重量矩阵W QQ Rd×k,我们通过代表后者的低等去0成份W + QQW = W + BA来限制其更新,其中B → Rd×r,A → Rr×k,和分级r(cid:28 min(d,k). 0 在训练期间,W被冷冻并没有得到梯度更新,而A和B包含可被训练的0参数.
<a id="S0065"></a> Source: p.4 S0065
Original: Note both W and ∆W = BA are multiplied with the same input, and their respective 0 output vectors are summed coordinate-wise.
中文: 注意W和QQW = BA均以相同的输入相乘,而各自的0输出向量则以坐标相相取.
<a id="S0066"></a> Source: p.4 S0066
Original: For h = W x, our modified forward pass yields: 0 h = W x + ∆W x = W x + BAx (3) 0 0 We illustrate our reparametrization in Figure 1.
中文: 对于h = W x,我们经过修改的前传分数:0 h = W x + + → W x = W x + BAx (3) 0 我们用图1来说明我们的再补偿.
<a id="S0067"></a> Source: p.4 S0067
Original: We use a random Gaussian initialization for A and zero for B, so ∆W = BA is zero at the beginning of training.
中文: 我们使用一个随机的高斯初始化A和0为B,所以QQW=BA在训练开始时为0.
<a id="S0068"></a> Source: p.4 S0068
Original: We then scale ∆W x by α , where α r is a constant in r.
中文: 然后用α来缩放 α r 是一个常数
<a id="S0069"></a> Source: p.4 S0069
Original: When optimizing with Adam, tuning α is roughly the same as tuning the learning rate if we scale the initialization appropriately.
中文: 当与亚当进行最优化时,调取α与调取学习速率大致相同,如果我们适当缩放初始化.
<a id="S0070"></a> Source: p.4 S0070
Original: As a result, we simply set α to the first r we try and do not tune it.
中文: 因此,我们只是把α设定在第一个r,我们试图不调谐它。
<a id="S0071"></a> Source: p.4 S0071
Original: This scaling helps to reduce the need to retune hyperparameters when we vary r (Yang & Hu, 2021). A Generalization of Full Fine-tuning. A more general form of fine-tuning allows the training of a subset of the pre-trained parameters.
中文: 这个缩放有助于减少当我们变化为r(Yang & Hu, 2021)时重调高参数的需要. 全面精细化概括. 一种较为通俗的微调形式,可以训练出一组预先训练过的参数.
<a id="S0072"></a> Source: p.4 S0072
Original: LoRA takes a step further and does not require the accumulated gradient update to weight matrices to have full-rank during adaptation.
中文: LoRA走得更远,在适应过程中不需要积分梯度更新到权重矩阵即可完全排位.
<a id="S0073"></a> Source: p.4 S0073
Original: This means that when applying LoRA to all weight matrices and training all biases2, we roughly recover the expressiveness of full fine-tuning by setting the LoRA rank r to the rank of the pre-trained weight matrices.
中文: 这意味着,当将LORA应用到所有重量矩阵和培训所有偏差2时,我们通过将LORA的分级定为预训练重量矩阵的分级,大致恢复了完全微调的表现.
<a id="S0074"></a> Source: p.4 S0074
Original: In other words, as we increase the number of trainable parameters 3, training LoRA roughly converges to training the original model, while adapter-based methods converges to an MLP and prefix-based methods to a model that cannot take long input sequences.
中文: 换句话说,当我们增加可训练参数3的数量时,训练LORA大致集中到训练原始模型上,而以适配器为基础的方法集中到MLP和以前缀为基础的方法集中到一个不能需要长时间输入序列的模型上.
<a id="S0075"></a> Source: p.4 S0075
Original: When deployed in production, we can explicitly compute and store W = W + BA and perform inference as usual.
中文: 在投入生产时,可以明确计算并存储W=W+BA,并照常进行推论.
<a id="S0076"></a> Source: p.4 S0076
Original: Note that both W and BA are in Rd×k. 0 0 When we need to switch to another downstream task, we can recover W by subtracting BA and 0 then adding a different B(cid:48)A(cid:48), a quick operation with very little memory overhead.
中文: 请注意,W和BA都在Rd×k. 0 0 当我们需要切换到另一个下游任务时,我们可以通过减去BA和0来恢复W,然后添加一个不同的B(cid:48)A(cid:48),这是一种快速操作,其内存的上方很少.
<a id="S0077"></a> Source: p.4 S0077
Original: Critically, this 2They represent a negligible number of parameters compared to weights. 3An inevitability when adapting to hard tasks. 4
中文: 关键是,与重量相比,这2个参数的参数微不足道。 3 适应艰巨任务时的必然性。 页:1
<a id="S0078"></a> Source: p.5 S0078
Original: guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction. 4.2 APPLYING LORA TO TRANSFORMER In principle, we can apply LoRA to any subset of weight matrices in a neural network to reduce the number of trainable parameters.
中文: 保证在推论期间,我们不采用任何额外的延迟,而采用建筑的微调模型。 4.2 利用洛拉进行贸易 原则上,我们可以将LORA应用于神经网络中任何子集的重量矩阵来减少可受训练参数的数量.
<a id="S0079"></a> Source: p.5 S0079
Original: In the Transformer architecture, there are four weight matrices in the self-attention module (W , W , W , W ) and two in the MLP module.
中文: 在"变形器"架构中,自留心模块(W,W,W,W)有四个重量矩阵,而MLP模块则有两个.
<a id="S0080"></a> Source: p.5 S0080
Original: We treat W (or W , W ) q k v o q k v as a single matrix of dimension d × d , even though the output dimension is usually sliced model model into attention heads.
中文: 我们把W(或W,W) q k v o q k v 当作维度d×d的单个矩阵,尽管输出维度通常被切入了注意头.
<a id="S0081"></a> Source: p.5 S0081
Original: We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency.We further study the effect on adapting different types of attention weight matrices in a Transformer in Section 7.1.
中文: 我们的研究仅限于调整下游任务的注意力分量,并冻结MLP模块(因此它们没有接受下游任务培训),以做到简单和参数效率. 我们进一步研究第7.1节中的变形器对调整不同类型注意力重量矩阵的影响。
<a id="S0082"></a> Source: p.5 S0082
Original: We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.
中文: 我们把对MLP层、LayleNorm层和偏见的实证调查留给未来工作。
<a id="S0083"></a> Source: p.5 S0083
Original: The most significant benefit comes from the reduction in memory and storage usage.
中文: 最大的好处是内存和储存使用减少。
<a id="S0084"></a> Source: p.5 S0084
Original: For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3 if r (cid:28) d as we do not need to store the optimizer states for the frozen model parameters.
中文: 对于一个与亚当一起训练的大型变形器,如果r(cid:28) d,我们可将VRAM的使用减少至2/3,因为我们不需要存储被冻结的模型参数的优化状态.
<a id="S0085"></a> Source: p.5 S0085
Original: On GPT-3 175B, we reduce the VRAM consumption during training from 1.2TB to 350GB.
中文: 在GPT-3175B上,我们把训练期间的VRAM消耗从1.2TB减少到350GB.
<a id="S0086"></a> Source: p.5 S0086
Original: With r = 4 and only the query and value projection matrices being adapted, the checkpoint size is reduced by roughly 10,000× (from 350GB to 35MB)4.
中文: 随着r = 4 并仅调整了查询和数值预测矩阵,检查站规模被缩小了大约10 000x(从350GB减少到35MB)4。
<a id="S0087"></a> Source: p.5 S0087
Original: This allows us to train with significantly fewer GPUs and avoid I/O bottlenecks.
中文: 这使得我们能使用显著较少的GPU进行训练并避免I/O瓶颈.
<a id="S0088"></a> Source: p.5 S0088
Original: Another benefit is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights as opposed to all the parameters.
中文: 另一个好处是,我们可以在任务之间进行调换,同时以更低得多的成本部署,只交换LORA的重量,而不是所有参数。
<a id="S0089"></a> Source: p.5 S0089
Original: This allows for the creation of many customized models that can be swapped in and out on the fly on machines that store the pre-trained weights in VRAM.
中文: 这使得许多定制的模型得以被创建,这些模型可以在将预先训练过的重量存储于VRAM的机器上在苍蝇上互换.
<a id="S0090"></a> Source: p.5 S0090
Original: We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning5 as we do not need to calculate the gradient for the vast majority of the parameters.
中文: 我们还观察到,在GPT-3175B的训练中,比起全微调5,速度提高了25%,因为我们不需要计算绝大多数参数的梯度。
<a id="S0091"></a> Source: p.5 S0091
Original: For example, it is not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency.
中文: 例如,如果人们选择将A和B吸收到W中去去去去去消除额外的推论空想,那么将不同A和B的任务的输入分批处理就不是直接的了.
<a id="S0092"></a> Source: p.5 S0092
Original: Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical. 5 EMPIRICAL EXPERIMENTS We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al., 2019), De- BERTa (He et al., 2021), and GPT-2 (Radford et al., b), before scaling up to GPT-3 175B (Brown et al., 2020).
中文: 虽然不可能合并权重并动态地选择LORA模块,用于在不临界情况下的批量样本。 5 精神警告 我们评价洛拉在RoBERTa(Liu等,2019年)、De-BERTa(He等,2021年)和GPT-2(Radford等,b)上下游的任务业绩,然后扩大至GPT-3 175B(Brown等,2020年)。
<a id="S0093"></a> Source: p.5 S0093
Original: Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG).
中文: 我们的实验涵盖了从自然语言理解(NLU)到生成(NLG)的广泛任务.
<a id="S0094"></a> Source: p.5 S0094
Original: Specifically, we evaluate on the GLUE (Wang et al., 2019) benchmark for RoBERTa and DeBERTa.
中文: 具体来说,我们评价了罗贝塔和德贝塔的GLUE(Wang等,2019年)基准。
<a id="S0095"></a> Source: p.5 S0095
Original: We follow the setup of Li & Liang (2021) on GPT-2 for a direct comparison and add WikiSQL (Zhong et al., 2017) (NL to SQL queries) and SAMSum (Gliwa et al., 2019) (conversation summarization) for large-scale experiments on GPT-3.
中文: 我们跟踪李同良(2021)在GPT-2上的设置进行直接比较,并添加了WikisQL(Zhong等,2017年)(NL为SQL查询)和SAMSum(Gliwa等,2019年)(协商总结),用于GPT-3上的大规模实验.
<a id="S0096"></a> Source: p.5 S0096
Original: See Appendix C for more details on the datasets we use.
中文: 关于我们使用的数据集的更详细情况见附录C。
<a id="S0097"></a> Source: p.5 S0097
Original: We use NVIDIA Tesla V100 for all experiments. 5.1 BASELINES To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible.
中文: 我们用NVIDIA Tesla V100进行所有实验. 5.1 巴西 为了与其他基线作广义的比较,我们复制了先前工作所用的设置,并尽可能重新使用所报告的数量。
<a id="S0098"></a> Source: p.5 S0098
Original: This, however, means that some baselines might only appear in certain experiments.
中文: 然而,这意味着某些基线可能只出现在某些实验中。
<a id="S0099"></a> Source: p.5 S0099
Original: Fine-Tuning (FT) is a common approach for adaptation.
中文: Fine-Tuning(FT)是一种常见的适应方法.
<a id="S0100"></a> Source: p.5 S0100
Original: During fine-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.A simple variant is to update only some layers while freezing others.
中文: 在微调期间,该模型被初始化为预受训练的重量和偏差,所有模型参数都经过梯度更新. 一个简单的变体是只更新一些地层而冻结其他地层.
<a id="S0101"></a> Source: p.5 S0101
Original: We include one such baseline reported in prior work (Li & Liang, 2021) on GPT-2, which adapts just the last two layers (FTTop2). 4We still need the 350GB model during deployment; however, storing 100 adapted models only requires 350GB + 35MB 100 ≈ 354GB as opposed to 100 350GB ≈ 35TB. 5For GPT-3 175B, the training throughput for full fine-tuning is 32.5 tokens/s per V100 GPU; with the same number of weight shards for model parallelism, the throughput is 43.1 tokens/s per V100 GPU for LoRA. 5
中文: 我们列入了先前工作(Li & Liang, 2021年)中报告的关于GPT-2的基准,该基准仅适应后两层(FTTop2)。 4,在部署时我们仍然需要350GB型号;然而,存储100个被改造的型号只需要350GB + 35MB 100 → 354GB,而不是100 350GB → 35TB. 5 对于GPT-3 175B,完全微调的训练吞吐量为每V100 GPU32.5个活字符号/秒;对于模型平行主义,重量硬字符号相同,对于LORA,活字符号为每V100 GPU43.1个活字符号/秒. 5
<a id="S0102"></a> Source: p.6 S0102
Original: Model & Method # Trainable Parameters MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Avg.
中文: model & Method #可训练参数 MNLI SST-2 MRPC CoLA QQP RTE STS-B Avg.
<a id="S0103"></a> Source: p.6 S0103
Original: RoB (FT) 125.0M 87.6 94.8 90.2 63.6 92.8 91.9 78.7 91.2 86.4 base RoB (BitFit) 0.1M 84.7 93.7 92.7 62.0 91.8 84.0 81.5 90.8 85.2 base RoB (AdptD) 0.3M 87.1 94.2 88.5 60.8 93.1 90.2 71.5 89.7 84.4 base ±.0 ±.1 ±1.1 ±.4 ±.1 ±.0 ±2.7 ±.3 RoB (AdptD) 0.9M 87.3 94.7 88.4 62.6 93.0 90.6 75.9 90.3 85.4 base ±.1 ±.3 ±.1 ±.9 ±.2 ±.0 ±2.2 ±.1 RoB (LoRA) 0.3M 87.5 95.1 89.7 63.4 93.3 90.8 86.6 91.5 87.2 base ±.3 ±.2 ±.7 ±1.2 ±.3 ±.1 ±.7 ±.2 RoB (FT) 355.0M 90.2 96.4 90.9 68.0 94.7 92.2 86.6 92.4 88.9 large RoB (LoRA) 0.8M 90.6 96.2 90.9 68.2 94.9 91.6 87.4 92.6 89.0 large ±.2 ±.5 ±1.2 ±1.9 ±.3 ±.1 ±2.5 ±.2 RoB (AdptP)† 3.0M 90.2 96.1 90.2 68.3 94.8 91.9 83.8 92.1 88.4 large ±.3 ±.3 ±.7 ±1.0 ±.2 ±.1 ±2.9 ±.7 RoB (AdptP)† 0.8M 90.5 96.6 89.7 67.8 94.8 91.7 80.1 91.9 87.9 large ±.3 ±.2 ±1.2 ±2.5 ±.3 ±.2 ±2.9 ±.4 RoB (AdptH)† 6.0M 89.9 96.2 88.7 66.5 94.7 92.1 83.4 91.0 87.8 large ±.5 ±.3 ±2.9 ±4.4 ±.2 ±.1 ±1.1 ±1.7 RoB (AdptH)† 0.8M 90.3 96.3 87.7 66.3 94.7 91.5 72.9 91.5 86.4 large ±.3 ±.5 ±1.7 ±2.0 ±.2 ±.1 ±2.9 ±.5 RoB (LoRA)† 0.8M 90.6 96.2 90.2 68.2 94.8 91.6 85.2 92.3 88.6 large ±.2 ±.5 ±1.0 ±1.9 ±.3 ±.2 ±1.1 ±.5 DeB (FT) 1500.0M 91.8 97.2 92.0 72.0 96.0 92.7 93.9 92.9 91.1 XXL DeB (LoRA) 4.7M 91.9 96.9 92.6 72.4 96.0 92.9 94.9 93.0 91.3 XXL ±.2 ±.2 ±.6 ±1.1 ±.1 ±.1 ±.4 ±.2 Table 2: RoBERTa , RoBERTa , and DeBERTa with different adaptation methods on the base large XXL GLUE benchmark.
中文: RoB(FT) 125.0M 87.6 94.8 90.2 63.6 92.8 91.3 RoB(AdptD) 0.1M 84.7 92.7 62.7 62.8 62.8 84.0 8.8 4 BroB(AdptD) 0.3 M 87.1 94.2 94.2 89.7 84.4 +2 ±2 ±1 ±1 ±4 ±1 ±0.0.0 ±2 ±3 ±3 ±3 ±3 ±3 ±3 ±3 ±3 ±3 ±3 ±1 ±1 94.0.1 ±2 96.2 ±3 ±0 B 9 ±3 96.2 ±9 ±9 96.9 9 94-7 94-7 96.2 →6 Pt 大型 90.8.2 ±2 96. 4±2±1±1.7 RoB(AdptH)→0.8M 90.3 96.3 87.7 66.3 94.7 91.5 72.9 91.5 86.4 大±3 ±.5 ±1.7 ±2.0 ±2 ±1 ±2.9 ±.5 RoB(LORA)→0.8M 90.6 96.2 90.2 ±6 85.2 92.3 88.6 大±2 ±5 ±1.0±1.9 ±3 ±2 ±1.1 ±5 DEB(FT) 1500.0M 91.8 97.2 92.0 7 96.0 92.9 92.9 92.9 91.1 XXL DeB(LORA) 4.7M 91.9 96.9 92.6 72.4 96.0 92.9 94.9 93.9 93.9 93.0 91 XXL ±2 ±2 ±6 ±1.1 ±1 ±1 ±4 ±2 表2:在基准大XXL GLUE基准上采用不同适应方法的RoBERTa、RoBERTa和DeBERTa。
<a id="S0104"></a> Source: p.6 S0104
Original: We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks.
中文: 我们报告MNLI的总体精度(匹配和不匹配),Matthew的CoLA相关性,Pearson的STS-B相关性,以及其他任务的准确性.
<a id="S0105"></a> Source: p.6 S0105
Original: Higher is better for all metrics. * indicates numbers published in prior works. † indicates runs configured in a setup similar to Houlsby et al. (2019) for a fair comparison.
中文: 更高的水平对于所有衡量标准都更好。 * 表示以前著作中公布的数字。 † 表示运行在类似于Houlsby等人(2019年)的设置中进行公平比较.
<a id="S0106"></a> Source: p.6 S0106
Original: Bias-only or BitFit is a baseline where we only train the bias vectors while freezing everything else.
中文: 唯偏或BitFit是基准,我们只训练偏向向量,同时冻结其他一切.
<a id="S0107"></a> Source: p.6 S0107
Original: Contemporarily, this baseline has also been studied by BitFit (Zaken et al., 2021).
中文: 同时,BitFit也研究了这一基线(Zaken等人,2021年)。
<a id="S0108"></a> Source: p.6 S0108
Original: Prefix-embedding tuning (PreEmbed) inserts special tokens among the input tokens.
中文: 前缀嵌入调音(PreEmbed)在输入符中插入特殊符.
<a id="S0109"></a> Source: p.6 S0109
Original: These special tokens have trainable word embeddings and are generally not in the model’s vocabulary.
中文: 这些特殊标志有可训练的词嵌入并一般不在模型的词汇中.
<a id="S0110"></a> Source: p.6 S0110
Original: Where to place such tokens can have an impact on performance.
中文: 将这种标志放在哪里可对业绩产生影响。
<a id="S0111"></a> Source: p.6 S0111
Original: We focus on “prefixing”, which prepends such tokens to the prompt, and “infixing”, which appends to the prompt; both are discussed in Li & Liang (2021).
中文: 我们着重讨论“前置”和“后置”,两者都由李和梁(2021年)讨论。
<a id="S0112"></a> Source: p.6 S0112
Original: We use l (resp. l ) denote the number of prefix (resp. infix) tokens.
中文: 我们使用l(resp. l)表示前缀(resp. infix)符号的数量.
<a id="S0113"></a> Source: p.6 S0113
Original: The number of p i trainable parameters is |Θ| = d × (l + l ). model p i Prefix-layer tuning (PreLayer) is an extension to prefix-embedding tuning.
中文: p i可训练参数数为QQ = d × (l + l). model p i Prefix-layer turn (PreLayer)是前缀-嵌入式调制的扩展.
<a id="S0114"></a> Source: p.6 S0114
Original: Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, we learn the activations after every Transformer layer.
中文: 我们不是仅仅学习一些特殊符号的嵌入词(或等同地,嵌入层后活化),而是学习每个变形器层后活化.
<a id="S0115"></a> Source: p.6 S0115
Original: The activations computed from previous layers are simply replaced by trainable ones.
中文: 从前几层计算出来的活化装置只是被可训练装置所取代。
<a id="S0116"></a> Source: p.6 S0116
Original: The resulting number of trainable parameters is |Θ| = L × d × (l + l ), where L is the number of Transformer layers. model p i Adapter tuning as proposed in Houlsby et al. (2019) inserts adapter layers between the selfattention module (and the MLP module) and the subsequent residual connection.
中文: 由此得出的可列车参数为QQ=L×d×(l+l),其中L为变形器层数. 根据Houlsby等人(2019年)的建议,模型 p i 适配器调试在自留能模块(和MLP模块)和后续剩余连接之间插入适配器层.
<a id="S0117"></a> Source: p.6 S0117
Original: There are two fully connected layers with biases in an adapter layer with a nonlinearity in between.
中文: 适配器层中有两个完全相连的地层带有偏差,介于两者之间。
<a id="S0118"></a> Source: p.6 S0118
Original: Recently, Lin et al. (2020) proposed a more efficient design with the adapter layer applied only after the MLP module and after a LayerNorm.
中文: 最近,Lin等(2020年)提出一个更高效的设计,适配器层只在MLP模块后和LayerNorm后应用.
<a id="S0119"></a> Source: p.6 S0119
Original: This is very similar to another deign proposed in Pfeiffer et al. (2021), which we call AdapterP.
中文: 这与Pfeiffer等人(2021年)中提议的另一个代号非常相类似,我们称之为AdapterP.
<a id="S0120"></a> Source: p.6 S0120
Original: We also include another baseline call AdapterDrop (Ru¨ckle´ et al., 2020) which drops some adapter layers for greater efficiency (AdapterD).
中文: 我们还包括另一个基线呼叫AdapterDrop(Ru'cle' et al.,2020年),为提高效率而降下一些适配层(AdapterD)。
<a id="S0121"></a> Source: p.6 S0121
Original: We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the first column.
中文: 我们尽可能引用以往工作的数字,以尽量扩大我们与基线的比较;它们排在第一栏的星号(*)。
<a id="S0122"></a> Source: p.6 S0122
Original: In all cases, we have |Θ| = Lˆ × (2 × d × r + r + d ) + 2 × Lˆ × d where Lˆ Adpt model model LN model Adpt is the number of adapter layers and Lˆ the number of trainable LayerNorms (e.g., in AdapterL).
中文: 在所有情况下,我们都有 QQ = Lˆ (2 × d × r + r + d (d)) + 2 × Lˆ → d 其中 Lˆ adpt 型号 LN 型号 Adpt 为适配器层数, Lˆ 为可列车层数 (如 在 AdapterL).
<a id="S0123"></a> Source: p.6 S0123
Original: LN LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices.
中文: LN LoRA 与现有的重量矩阵平行地添加了可受训练的对等分解矩阵.
<a id="S0124"></a> Source: p.6 S0124
Original: As mentioned in Section 4.2, we only apply LoRA to W and W in most experiments for simplicity. q v The number of trainable parameters is determined by the rank r and the shape of the original weights: |Θ| = 2 × Lˆ × d × r, where Lˆ is the number of weight matrices we apply LoRA to.
中文: 正如第4.2节所提到,我们只在大多数实验中将LORA应用到W和W以简单化. 页:1 可列车参数的数量由等级 r 和原始重量的形状决定: |Θ| = 2 × Lˆ D× r,其中 Lˆ 为我们应用 LoRA 的重量矩阵数量.
<a id="S0125"></a> Source: p.7 S0125
Original: Model & Method # Trainable E2E NLG Challenge Parameters BLEU NIST MET ROUGE-L CIDEr GPT-2 M (FT) 354.92M 68.2 8.62 46.2 71.0 2.47 GPT-2 M (AdapterL) 0.37M 66.3 8.41 45.0 69.8 2.40 GPT-2 M (AdapterL) 11.09M 68.9 8.71 46.1 71.3 2.47 GPT-2 M (AdapterH) 11.09M 67.3 8.50 46.0 70.7 2.44 ±.6 ±.07 ±.2 ±.2 ±.01 GPT-2 M (FTTop2) 25.19M 68.1 8.59 46.0 70.8 2.41 GPT-2 M (PreLayer) 0.35M 69.7 8.81 46.1 71.4 2.49 GPT-2 M (LoRA) 0.35M 70.4 8.85 46.8 71.8 2.53 ±.1 ±.02 ±.2 ±.1 ±.02 GPT-2 L (FT) 774.03M 68.5 8.78 46.0 69.9 2.45 GPT-2 L (AdapterL) 0.88M 69.1 8.68 46.3 71.4 2.49 ±.1 ±.03 ±.0 ±.2 ±.0 GPT-2 L (AdapterL) 23.00M 68.9 8.70 46.1 71.3 2.45 ±.3 ±.04 ±.1 ±.2 ±.02 GPT-2 L (PreLayer)* 0.77M 70.3 8.85 46.2 71.7 2.47 GPT-2 L (LoRA) 0.77M 70.4 8.89 46.8 72.0 2.47 ±.1 ±.02 ±.2 ±.2 ±.02 Table 3: GPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG Challenge.
中文: 型号 & 方法#可训练E2E NLG挑战参数 BLEU NLG挑战参数 11.09M 67.3 8.50 70.7 2.44 ±6 ±07 ±2 ±2 M (FTOP2) 354.92M 68.2 8.62 GPT-2 M (AdapterL) 0.37 M 66.3 8.41 45.0 69.8 2.40 GPT-2 M (AdapterL) 11.09M 68.9 GPT-2 2.47 GPT-2 ±2 ±2 ±02 M (A) 不同方法 0.35 M 2.5 ±8 ±8 GPT 2.5 ±2 ±8 M 2.5 ±2 ±02 M 2.02 2.02 2.02 2.02 2.80 M 1. ±2 ±2 ±2 ±1 ±1 ±1 ±1 ±02 适应 GPT-2 L-FT 77.04 L.
<a id="S0126"></a> Source: p.7 S0126
Original: LoRA outperforms several baselines with comparable or fewer trainable parameters.
中文: LoRA的绩效超过了几个具有可比或较少可训练参数的基线。
<a id="S0127"></a> Source: p.7 S0127
Original: Confidence intervals are shown for experiments we ran. * indicates numbers published in prior works. 5.2 ROBERTA BASE/LARGE RoBERTa (Liu et al., 2019) optimized the pre-training recipe originally proposed in BERT (Devlin et al., 2019a) and boosted the latter’s task performance without introducing many more trainable parameters.
中文: 我们的实验显示信任的间隔 * 表示以前著作中公布的数字。 5.2 ROBERTA BASE/LARGE RoBERTA(Liu等人,2019年)优化了BERT(Devlin等人,2019年a)最初提出的培训前食谱,并提升了后者的任务性能,但没有引入更多的可训练参数.
<a id="S0128"></a> Source: p.7 S0128
Original: While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains a competitive and popular pre-trained model for its size among practitioners.
中文: 虽然近年来RoBERTa被GLUE基准(Wang等,2019年)等NLP领导板上更大得多的模型所超越,但对于其规模在从业人员中仍然是一个有竞争力和受欢迎的预先训练的模式.
<a id="S0129"></a> Source: p.7 S0129
Original: We take the pre-trained RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace Transformers library (Wolf et al., 2020) and evaluate the performance of different efficient adaptation approaches on tasks from the GLUE benchmark.
中文: 我们从HuggingFace变形器库(Wolf等人,2020年)取出经过预先训练的RoBERTa基地(125M)和RoBERTa大型(355M),从GLUE基准评价不同高效适应方法任务的表现.
<a id="S0130"></a> Source: p.7 S0130
Original: We also replicate Houlsby et al. (2019) and Pfeiffer et al. (2021) according to their setup.
中文: 我们还根据Houlsby等人(2019年)和Pfeiffer等人(2021年)的设置复制了它们.
<a id="S0131"></a> Source: p.7 S0131
Original: To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters.
中文: 为了确保进行公平的比较,我们在与适配器比较时对评估LORA的方法做了两个关键的改变.
<a id="S0132"></a> Source: p.7 S0132
Original: First, we use the same batch size for all tasks and use a sequence length of 128 to match the adapter baselines.
中文: 首先,我们对所有任务使用相同的批量尺寸,并使用128的序列长度来匹配适配器基线.
<a id="S0133"></a> Source: p.7 S0133
Original: Second, we initialize the model to the pre-trained model for MRPC, RTE, and STS-B, not a model already adapted to MNLI like the fine-tuning baseline.
中文: 其次,我们将模型初始化为MRPC,RTE,和STS-B的预训模型,而不是像微调基线那样已经适应了MNLI的模型.
<a id="S0134"></a> Source: p.7 S0134
Original: Runs following this more restricted setup from Houlsby et al. (2019) are labeled with †.
中文: 从Houlsby等人(2019年)开始的这种更受限制的设置之后的运行被标注为 .
<a id="S0135"></a> Source: p.7 S0135
Original: The result is presented in Table 2 (Top Three Sections).
中文: 其结果见表2(三节)。
<a id="S0136"></a> Source: p.7 S0136
Original: See Section D.1 for details on the hyperparameters used. 5.3 DEBERTA XXL DeBERTa (He et al., 2021) is a more recent variant of BERT that is trained on a much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al., 2019) and SuperGLUE (Wang et al., 2020).
中文: 关于使用的超参数,详见D.1节。 5.3 DEBERTA XXL DeBERTa(He等人,2021年)是BERT的一个较新的变体,在更大范围内接受培训,并在GLUE(Wang等人,2019年)和SuperGLUE(Wang等人,2020年)等基准上表现非常有竞争力.
<a id="S0137"></a> Source: p.7 S0137
Original: We evaluate if LoRA can still match the performance of a fully fine-tuned DeBERTa XXL (1.5B) on GLUE.
中文: 我们评价LORA是否能够在GLUE上保持完全微调的DeBERTa XXL(1.5B)的性能相匹配.
<a id="S0138"></a> Source: p.7 S0138
Original: The result is presented in Table 2 (Bottom Section).
中文: 其结果见表2(Bottom科)。
<a id="S0139"></a> Source: p.7 S0139
Original: See Section D.2 for details on the hyperparameters used. 5.4 GPT-2 MEDIUM/LARGE Having shown that LoRA can be a competitive alternative to full fine-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such as GPT-2 medium and large (Radford et al., b).
中文: 关于使用的超参数,详见D.2节。 5.4 GPT-2 中型/升 在证明LORA可以作为NLU完全微调的竞争性替代品后,我们希望回答如果LORA仍然在NLG模型上占上风,例如GPT-2中和大(Radford等人,b.).
<a id="S0140"></a> Source: p.7 S0140
Original: We keep our setup as close as possible to Li & Liang (2021) for a direct comparison.
中文: [永久失效連結] 我们尽量贴近李同良(2021年)直接比较.
<a id="S0141"></a> Source: p.7 S0141
Original: Due to space constraint, we only present our result on E2E NLG Challenge (Table 3) in this section.
中文: 由于空间的限制,我们只在本节中介绍我们对E2E NLG挑战(表3)的结果。
<a id="S0142"></a> Source: p.7 S0142
Original: See Section F.1 for results on WebNLG (Gardent et al., 2017) and DART (Nan et al., 2020).
中文: 关于WebNLG(Gardent等,2017年)和DART(Nan等,2020年)的成果,参见F.1节.
<a id="S0143"></a> Source: p.7 S0143
Original: We include a list of the hyperparameters used in Section D.3. 7
中文: 我们列出了D.3.7节所用的超参数。
<a id="S0144"></a> Source: p.8 S0144
Original: # Trainable WikiSQL MNLI-m SAMSum Model&Method Parameters Acc. (%) Acc. (%) R1/R2/RL GPT-3 (FT) 175,255.8M 73.8 89.5 52.0/28.0/44.5 GPT-3 (BitFit) 14.2M 71.3 91.0 51.3/27.4/43.5 GPT-3 (PreEmbed) 3.2M 63.1 88.6 48.3/24.2/40.5 GPT-3 (PreLayer) 20.2M 70.1 89.5 50.8/27.3/43.5 GPT-3 (AdapterH) 7.1M 71.9 89.8 53.0/28.9/44.8 GPT-3 (AdapterH) 40.1M 73.2 91.5 53.2/29.0/45.1 GPT-3 (LoRA) 4.7M 73.4 91.7 53.8/29.8/45.9 GPT-3 (LoRA) 37.7M 74.0 91.6 53.4/29.2/45.1 Table 4: Performance of different adaptation methods on GPT-3 175B.
中文: #可训练的WikiSQL MNLI-m SAMSUM Model and Method 参数 Acc. (%) Acc. (%) R1/R2/RL GPT-3(FT)175,255.8M 73.8 89.5 52.0/28.0/44.5 GPT-3(BitFit) 14.2M 71.3 91.0 51.3/27.4/43.5 GPT-3(PreEmbed) 3.2M 63.1 88.6 48.3/24.2/40.5 GPT-3(PreLayer) 20.2M 70.1 89.5 50.8/27.3/43.5 GPT-3(AdapterH) 7.1M 71.9 89.8 53.0/28.9/44.8 GPT-3(AdapterH) 40.1M 73.2 91.5 53.2/29.49.45.1 GPT-3(LORA) 4.7M 73.4 53.8/29.45.9 GPT-3(LORA) 37.7M 74.0. 91.2 5 53.29.2/45.1 表4:不同适应方法在GPT-3175B上的性能。
<a id="S0145"></a> Source: p.8 S0145
Original: We report the logical form validation accuracy on WikiSQL, validation accuracy on MultiNLI-matched, and Rouge-1/2/L on SAMSum.
中文: 我们在WikisQL上报告逻辑形式验证精度,在MultiNLI上报告验证精度,在SAMSum上报告Ruge-1/2/L.
<a id="S0146"></a> Source: p.8 S0146
Original: LoRA performs better than prior approaches, including full fine-tuning.
中文: LORA的表现优于以往的做法,包括全面微调.
<a id="S0147"></a> Source: p.8 S0147
Original: The results on WikiSQL have a fluctuation around ±0.5%, MNLI-m around ±0.1%, and SAMSum around ±0.2/±0.2/±0.1 for the three metrics. 5.5 SCALING UP TO GPT-3 175B As a final stress test for LoRA, we scale up to GPT-3 with 175 billion parameters.
中文: WikiSQL的结果显示,三个度量衡的波动率为±0.5%,MNLI-m约为±0.1%,SAMSum约为±0.2/±0.2/±0.1。 5.5 升至GPT-3 175B 作为LORA的最终应力测试,我们以1,750亿个参数被提升到GPT-3.
<a id="S0148"></a> Source: p.8 S0148
Original: Due to the high training cost, we only report the typical standard deviation for a given task over random seeds, as opposed to providing one for every entry.
中文: 由于训练成本高,我们只报告某项任务的典型标准偏差,而不是每个输入都提供一个.
<a id="S0149"></a> Source: p.8 S0149
Original: See Section D.4 for details on the hyperparameters used.
中文: 关于使用的超参数,详见D.4节。
<a id="S0150"></a> Source: p.8 S0150
Original: As shown in Table 4, LoRA matches or exceeds the fine-tuning baseline on all three datasets.
中文: 如表4所示,LORA匹配或超过所有三个数据集的微调基线。
<a id="S0151"></a> Source: p.8 S0151
Original: Note that not all methods benefit monotonically from having more trainable parameters, as shown in Figure 2.
中文: 请注意,如图2所示,并非所有方法都能够单调地受益于更多的可训练参数。
<a id="S0152"></a> Source: p.8 S0152
Original: We observe a significant performance drop when we use more than 256 special tokens for prefix-embedding tuning or more than 32 special tokens for prefix-layer tuning.
中文: 当我们使用超过256个特殊符来进行前缀嵌入式调取或超过32个特殊符来进行前缀层调取时,我们观察到了显著的性能下降.
<a id="S0153"></a> Source: p.8 S0153
Original: This corroborates similar observations in Li & Liang (2021).
中文: 这证实了李同良(2021年)的类似观察.
<a id="S0154"></a> Source: p.8 S0154
Original: While a thorough investigation into this phenomenon is out-of-scope for this work, we suspect that having more special tokens causes the input distribution to shift further away from the pre-training data distribution.
中文: 虽然对这一现象的彻底调查超出了这项工作的范围,但我们怀疑,由于有了更多的特殊标志,投入分配将进一步偏离培训前的数据分配。
<a id="S0155"></a> Source: p.8 S0155
Original: Separately, we investigate the performance of different adaptation approaches in the low-data regime in Section F.3. 0.75 0.70 0.65 0.60 0.55 6 7 8 9 10 11 log10 # Trainable Parameters ycaruccA noitadilaV WikiSQL MultiNLI-matched 0.92 0.90 Method Fine-Tune 0.88 PrefixEmbed PrefixLayer 0.86 Adapter(H) LoRA 0.84 6 7 8 9 10 11 log10 # Trainable Parameters Figure 2: GPT-3 175B validation accuracy vs. number of trainable parameters of several adaptation methods on WikiSQL and MNLI-matched.
中文: 另外,我们调查F.3.0.75 0.70 0.65 0.60 0.55 0.55 6 7 8 9 10 11 log10 #可训练参数 ycaruccA noitadilaV WikiSQL 多NLI相配0.92 0.90 方法 Fine-Tune 0.88 前缀 Embed 前缀 Layer 0.86 适配器(H) LORA 0.84 6 7 8 9 10 11 log10#可训练参数 图2:GPT-3 175B 验证精度与WikiSQL和 MNLI相匹配的几种可训练参数的数量。
<a id="S0156"></a> Source: p.8 S0156
Original: LoRA exhibits better scalability and task performance.
中文: LORA表现出更好的可扩展性和任务性能.
<a id="S0157"></a> Source: p.8 S0157
Original: See Section F.2 for more details on the plotted data points. 6 RELATED WORKS Transformer Language Models.
中文: 关于所绘制的数据点,详见F.2节。 6 相關的Works变形语言模型.
<a id="S0158"></a> Source: p.8 S0158
Original: Transformer (Vaswani et al., 2017) is a sequence-to-sequence architecture that makes heavy use of self-attention.
中文: 变形金刚(Vaswani等,2017年)是一款花序至花序建筑,重用自心.
<a id="S0159"></a> Source: p.8 S0159
Original: Radford et al. (a) applied it to autoregressive language modeling by using a stack of Transformer decoders.
中文: Radford等 (a)通过使用一叠变形器解码器,将其应用于自旋式语言模型.
<a id="S0160"></a> Source: p.8 S0160
Original: Since then, Transformer-based language models have dominated NLP, achieving the state-of-the-art in many tasks. A new paradigm emerged with BERT (Devlin et al., 2019b) and GPT-2 (Radford et al., b) – both are large Transformer lan- 8
中文: 此后以变形器为主的语言模型主导了NLP,在许多任务中实现了最先进的. BERT(Devlin等,2019年b)和GPT-2(Radford等,b)都出现了一种新的范式——两者都是大型变形器lan-8
<a id="S0161"></a> Source: p.9 S0161
Original: guage models trained on a large amount of text – where fine-tuning on task-specific data after pretraining on general domain data provides a significant performance gain compared to training on task-specific data directly.
中文: guage模型接受了大量文本的培训 — — 在对一般域数据进行预训后对任务特定数据进行微调,与直接对任务特定数据进行培训相比,它提供了显著的性能收益.
<a id="S0162"></a> Source: p.9 S0162
Original: Training larger Transformers generally results in better performance and remains an active research direction.
中文: 培训更大的变形人一般能提高业绩,而且仍然是积极的研究方向。
<a id="S0163"></a> Source: p.9 S0163
Original: GPT-3 (Brown et al., 2020) is the largest single Transformer language model trained to-date with 175B parameters.
中文: GPT-3(Brown等,2020年)是迄今为止接受过175B参数培训的最大的单变形语言模型.
<a id="S0164"></a> Source: p.9 S0164
Original: While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).
中文: 虽然GPT-3 175B可以仅用几个额外的训练实例来调整其行为,但结果很大程度上取决于输入快取(Brown等,2020年).
<a id="S0165"></a> Source: p.9 S0165
Original: This necessitates an empirical art of composing and formatting the prompt to maximize a model’s performance on a desired task, which is known as prompt engineering or prompt hacking.
中文: 这需要一种经验艺术,即编组和格式化的提示,以最大限度地发挥模型在理想任务上的性能,这被称为即时工程或即时黑客.
<a id="S0166"></a> Source: p.9 S0166
Original: Fine-tuning retrains a model pre-trained on general domains to a specific task Devlin et al. (2019b); Radford et al. (a).
中文: 精细调整一个在一般领域预先培训的模型,用于Devlin等人(2019年b);Radford等人(a)。
<a id="S0167"></a> Source: p.9 S0167
Original: Variants of it include learning just a subset of the parameters Devlin et al. (2019b); Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream performance.
中文: 它的变体包括仅仅学习一个子集的参数Devlin等 (2019b); Collobert & Weston (2008),然而从业者经常重新训练所有它们,以最大限度地发挥下游的性能.
<a id="S0168"></a> Source: p.9 S0168
Original: However, the enormity of GPT-3 175B makes it challenging to perform fine-tuning in the usual way due to the large checkpoint it produces and the high hardware barrier to entry since it has the same memory footprint as pre-training.
中文: 然而,由于GPT-3 175B的庞大性能使得由于它生产的大检查站和高硬件出入口障碍而难以以通常的方式进行微调,因为它的内存足迹与预训相同.
<a id="S0169"></a> Source: p.9 S0169
Original: Many have proposed inserting adapter layers between existing layers in a neural network (Houlsby et al., 2019; Rebuffi et al., 2017; Lin et al., 2020).
中文: 许多人建议在神经网络中现有地层之间插入适配层(Houlsby等,2019;Rebuffi等,2017;Lin等,2020)。
<a id="S0170"></a> Source: p.9 S0170
Original: Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates.
中文: 我们的方法使用类似的瓶颈结构对重量更新施加了低级限制.
<a id="S0171"></a> Source: p.9 S0171
Original: The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3). A comtenporary extension of adapter is COMPACTER (Mahabadi et al., 2021), which essentially parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme.
中文: 关键功能差异是,在推论期间,我们所学的权重可以与主权重相合并,因此没有引入任何耐久性,适配层的情况并非如此(第3节). 适配器的同位素延伸是COMPACTER(Mahabadi等,2021年),它基本上使适配器层无法使用克罗内克产品,并带有一些预先确定的分量方案.
<a id="S0172"></a> Source: p.9 S0172
Original: Similarly, combining LoRA with other tensor product-based methods could potentially improve its parameter efficiency, which we leave to future work.
中文: 同样地,将LORA与其他以收音机产品为基础的方法结合起来,有可能提高其参数效率,我们留给今后的工作。
<a id="S0173"></a> Source: p.9 S0173
Original: More recently, many proposed optimizing the input word embeddings in lieu of fine-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang, 2021; Lester et al., 2021; Hambardzumyan et al., 2020; Liu et al., 2021).
中文: 最近,许多人提议优化输入词嵌入来代替微调,类似于对即时工程的连续和可分化的概括(Li & Liang, 2021; Lester等, 2021; Hambardzumyan等, 2020; Liu等, 2021)。
<a id="S0174"></a> Source: p.9 S0174
Original: We include comparisons with Li & Liang (2021) in our experiment section.
中文: 我们的实验部分包含了与李同良(2021年)的比较.
<a id="S0175"></a> Source: p.9 S0175
Original: However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.
中文: 然而,这行作品只能通过在快取中使用更特殊的符号来扩大,当学习到位置嵌入时,它占用了任务符号可用的序列长度.
<a id="S0176"></a> Source: p.9 S0176
Original: Low-rank structure is very common in machine learning. A lot of machine learning problems have certain intrinsic low-rank structure (Li et al., 2016; Cai et al., 2010; Li et al., 2018b; Grasedyck et al., 2013).
中文: 低等结构在机器学习中非常常见. 许多机器学习问题都有某些内在的低等结构(Li等,2016;Cai等,2010;Li等,2018b;Grasedyck等,2013).
<a id="S0177"></a> Source: p.9 S0177
Original: Moreover, it is known that for many deep learning tasks, especially those with a heavily over-parametrized neural network, the learned neural network will enjoy low-rank properties after training (Oymak et al., 2019).
中文: 此外,众所周知,对于许多深层学习任务,特别是那些神经网络高度超标的学习任务,所学神经网络在训练后将享有低等属性(Oymak等,2019年).
<a id="S0178"></a> Source: p.9 S0178
Original: Some prior works even explicitly impose the low-rank constraint when training the original neural network (Sainath et al., 2013; Povey et al., 2018; Zhang et al., 2014; Jaderberg et al., 2014; Zhao et al., 2016; Khodak et al., 2021; Denil et al., 2014); however, to the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks.
中文: 一些前作在培训原始神经网络时甚至明确施加了低级约束(Sainath等,2013年;Povey等,2018年;Zhang等,2014年;Jaderberg等,2014年;Zhao等,2016年;Khodak等,2021年;Denil等,2014年);然而,据我们所知,这些作品都没有考虑低级更新到被冻结的模式上下游任务的适应.
<a id="S0179"></a> Source: p.9 S0179
Original: In theory literature, it is known that neural networks outperform other classical learning methods, including the corresponding (finite-width) neural tangent kernels (Allen-Zhu et al., 2019; Li & Liang, 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al., 2020; Allen-Zhu & Li, 2019; Allen-Zhu & Li, 2020a).
中文: 在理论文献中,已知神经网络超越了其他古典学习方法,包括相应的(无限-width)神经真核(Allen-Zhu等,2019;Li & Liang等,2018),而基础概念类具有一定的低等结构(Ghorbani等,2020;Allen-Zhu等,2019;Allen-Zhu等,2020a).
<a id="S0180"></a> Source: p.9 S0180
Original: Another theoretical result in Allen-Zhu & Li (2020b) suggests that low-rank adaptations can be useful for adversarial training.
中文: Allen-Zhu & Li (2020b)的另一个理论结果表明,低等级的适应对对抗训练可能有用.
<a id="S0181"></a> Source: p.9 S0181
Original: In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature. 7 UNDERSTANDING THE LOW-RANK UPDATES Given the empirical advantage of LoRA, we hope to further explain the properties of the low-rank adaptation learned from downstream tasks.
中文: 总之,我们认为,我们提议的低级适应更新受到文献的启发。 7 了解低调的更新 鉴于LORA的经验优势,我们希望进一步解释从下游任务中吸取的低等级适应的属性.
<a id="S0182"></a> Source: p.9 S0182
Original: Note that the low-rank structure not only lowers the hardware barrier to entry which allows us to run multiple experiments in parallel, but also gives better interpretability of how the update weights are correlated with the pre-trained weights.
中文: 请注意,低等结构不仅降低了硬件进入障碍,使我们能够平行地进行多项实验,而且还能更好地解释更新后重量如何与预先训练过的重量相关。
<a id="S0183"></a> Source: p.9 S0183
Original: We focus our study on GPT-3 175B, where we achieved the largest reduction of trainable parameters (up to 10,000×) without adversely affecting task performances.
中文: 我们的研究重点是GPT-3 175B, 在那里,我们实现了最大程度的可训练参数的削减(最多为10,000xx),而没有对任务执行产生不利影响。
<a id="S0184"></a> Source: p.9 S0184
Original: We perform a sequence of empirical studies to answer the following questions: 1) Given a parameter budget constraint, which subset of weight matrices in a pre-trained Transformer should we adapt 9
中文: 我们进行一系列的经验研究,以回答以下问题: (1) 鉴于一个参数预算限制,在经过预先培训的变形器中,哪个子集的权重矩阵,我们应该适应9
<a id="S0185"></a> Source: p.10 S0185
Original: to maximize downstream performance? 2) Is the “optimal” adaptation matrix ∆W really rankdeficient?
中文: 最大限度地提高下游的性能? 2) “最佳”适应矩阵QQW是否真的排位不足?
<a id="S0186"></a> Source: p.10 S0186
Original: If so, what is a good rank to use in practice? 3) What is the connection between ∆W and W ?
中文: 如果是的话,实际使用什么等级? 3) QQW和W有什么联系?
<a id="S0187"></a> Source: p.10 S0187
Original: We believe that our answers to question (2) and (3) shed light on the fundamental principles of using pre-trained language models for downstream tasks, which is a critical topic in NLP. 7.1 WHICH WEIGHT MATRICES IN TRANSFORMER SHOULD WE APPLY LORA TO?
中文: 我们认为,我们对问题(2)和(3)的答复阐明了在下游任务中采用预先训练的语言模型的基本原则,这是NLP中的一个关键话题. 7.1 在贸易中哪些Wheight Matrics 我们应该应用洛拉?
<a id="S0188"></a> Source: p.10 S0188
Original: Given a limited parameter budget, which types of weights should we adapt with LoRA to obtain the best performance on downstream tasks?
中文: 鉴于参数预算有限,我们应该与LORA一起调整哪些类型的权重,以取得下游任务的最佳业绩?
<a id="S0189"></a> Source: p.10 S0189
Original: As mentioned in Section 4.2, we only consider weight matrices in the self-attention module.
中文: 如第4.2节所提及,我们只考虑自觉模块中的权重矩阵。
<a id="S0190"></a> Source: p.10 S0190
Original: We set a parameter budget of 18M (roughly 35MB if stored in FP16) on GPT-3 175B, which corresponds to r = 8 if we adapt one type of attention weights or r = 4 if we adapt two types, for all 96 layers.
中文: 我们在GPT-3 175B上设定了18M的参数预算(如果存储在FP16中,大概为35MB),如果调整了一种类型的注意力权重,则对应r=8;如果调整了两种类型的注意力权重,则对应r=4。
<a id="S0191"></a> Source: p.10 S0191
Original: The result is presented in Table 5. # of Trainable Parameters = 18M Weight Type W W W W W , W W , W W , W , W , W q k v o q k q v q k v o Rank r 8 8 8 8 4 4 2 WikiSQL (±0.5%) 70.4 70.0 73.0 73.2 71.4 73.7 73.7 MultiNLI (±0.1%) 91.0 90.8 91.0 91.3 91.3 91.3 91.7 Table 5: Validation accuracy on WikiSQL and MultiNLI after applying LoRA to different types of attention weights in GPT-3, given the same number of trainable parameters.
中文: 结果见表5。 o克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克 表5:在GPT-3中将LORA应用到不同类型的注意力分量后,考虑到同样数量的可训练参数,WikiSQL和MultiNLI上的验证精度.
<a id="S0192"></a> Source: p.10 S0192
Original: Adapting both W and q W gives the best performance overall.
中文: 调整W和qW的总体表现最好。
<a id="S0193"></a> Source: p.10 S0193
Original: We find the standard deviation across random seeds to be v consistent for a given dataset, which we report in the first column.
中文: 我们发现随机种子的标准偏差 与给定数据集一致, 我们在第一栏中报告。
<a id="S0194"></a> Source: p.10 S0194
Original: Note that putting all the parameters in ∆W or ∆W results in significantly lower performance, q k while adapting both W and W yields the best result.
中文: 注意将所有参数放入QQW或QQW中会导致性能显著降低,qk同时使W和W两个都适应,结果最好.
<a id="S0195"></a> Source: p.10 S0195
Original: This suggests that even a rank of four q v captures enough information in ∆W such that it is preferable to adapt more weight matrices than adapting a single type of weights with a larger rank. 7.2 WHAT IS THE OPTIMAL RANK r FOR LORA?
中文: 这表明,即使四克v的排名也能在QQW中获取足够的信息,因此比起更能适应分级较大的单一类型的分量,更为可取. 7.2 洛拉的占领区是什么?
<a id="S0196"></a> Source: p.10 S0196
Original: We turn our attention to the effect of rank r on model performance.
中文: 我们把注意力转向r级对模型性能的影响.
<a id="S0197"></a> Source: p.10 S0197
Original: We adapt {W , W }, q v {W , W , W , W }, and just W for a comparison. q k v c q Weight Type r = 1 r = 2 r = 4 r = 8 r = 64 W 68.8 69.6 70.5 70.4 70.0 WikiSQL(±0.5%) q W , W 73.4 73.3 73.7 73.8 73.5 q v W , W , W , W 74.1 73.7 74.0 74.0 73.9 q k v o W 90.7 90.9 91.1 90.7 90.7 q MultiNLI (±0.1%) W , W 91.3 91.4 91.3 91.6 91.4 q v W , W , W , W 91.2 91.7 91.7 91.5 91.4 q k v o Table 6: Validation accuracy on WikiSQL and MultiNLI with different rank r.
中文: 我们适应{W,W},qv{W,W,W,W},只是W比较一下。 q k v c 重量类型 r = 1 r = 2 r = 4 r = 8 r = 64 W 68.8 69.6 70.5 70.4 70.4 70.4 维基语录链接: q W , W 73.4 73.3 73.7 73.8 73.5 q v W, W , W 74.1 73.7 74.0 74.0 73.9 q k v o W 90.7 90.9 91.1 90.7 q MultiNLi (±0.1%) W W 91.3 9.1.3.6 91.4 q v W, W W 9.1.2 91.7 91.7 9.1.5 91.4 q k v o 表 6: WikiSQL 和多NLi的校正准确度,级别不同 r.
<a id="S0198"></a> Source: p.10 S0198
Original: To our surprise, a rank as small as one suffices for adapting both W and W on these datasets while training W alone q v q needs a larger r.
中文: 令人惊讶的是,一个小的军衔足以使W和W在数据集上都适应,而训练W单独q v q则需要更大的r.
<a id="S0199"></a> Source: p.10 S0199
Original: We conduct a similar experiment on GPT-2 in Section H.2.
中文: 我们在H.2节对GPT-2进行了类似的试验。
<a id="S0200"></a> Source: p.10 S0200
Original: Table 6 shows that, surprisingly, LoRA already performs competitively with a very small r (more so for {W , W } than just W ).
中文: 表6显示,令人惊讶的是,LORA已经用一个非常小的r(对{W,W}来说比仅仅W.)进行竞争。
<a id="S0201"></a> Source: p.10 S0201
Original: This suggests the update matrix ∆W could have a very small q v q “intrinsic rank”.6 To further support this finding, we check the overlap of the subspaces learned by different choices of r and by different random seeds.
中文: 这表明更新矩阵QQW可能有一个很小的q v q “intrinsic 排名”。 为了进一步支持这一发现,我们检查通过不同的r选择和不同的随机种子所学到的子空间的重叠.
<a id="S0202"></a> Source: p.10 S0202
Original: We argue that increasing r does not cover a more meaningful subspace, which suggests that a low-rank adaptation matrix is sufficient. 6However, we do not expect a small r to work for every task or dataset.
中文: 我们认为,增加r并不涵盖更有意义的子空间,这表明低级适应矩阵就足够了。 6 然而,我们并不期望一个小r为每一项任务或数据集工作。
<a id="S0203"></a> Source: p.10 S0203
Original: Consider the following thought experiment: if the downstream task were in a different language than the one used for pre-training, retraining the entire model (similar to LoRA with r = d ) could certainly outperform LoRA with a small r. model 10
中文: 考虑以下思想实验:如果下游任务使用的语言与预训中使用的语言不同,那么再培训整个模式(类似于r = d的LORA)肯定能够以一个小r 模式10超越LORA.
<a id="S0204"></a> Source: p.11 S0204
Original: Subspace similarity between different r.
中文: 不同r的子空间相近性.
<a id="S0205"></a> Source: p.11 S0205
Original: Given A and A which are the learned adaptar=8 r=64 tion matrices with rank r = 8 and 64 using the same pre-trained model, we perform singular value decomposition and obtain the right-singular unitary matrices U and U .7 We hope to an- Ar=8 Ar=64 swer: how much of the subspace spanned by the top i singular vectors in U (for 1 ≤ i ≤ 8) is Ar=8 contained in the subspace spanned by top j singular vectors of U (for 1 ≤ j ≤ 64)?
中文: 使用同一预训模式的A和A是学习到的适配器=8 r=64分量矩阵,分级为r=8和64分量矩阵,我们进行单相值分解,并获得右单相单元矩阵U和U.7. 我们希望使用- ar=8 ar=64 swer: U中顶端i单向量所跨越的子空间(对于 1 ≤ i ≤ 8) 中Ar=8 包含在 U上端j单向量所跨越的子空间(对于 1 ≤ j ≤ 64) 中 ?
<a id="S0206"></a> Source: p.11 S0206
Original: We mea- Ar=64 sure this quantity with a normalized subspace similarity based on the Grassmann distance (See Appendix G for a more formal discussion) ||U i(cid:62) U j ||2 φ(A , A , i, j) = Ar=8 Ar=64 F ∈ [0, 1] (4) r=8 r=64 min(i, j) where U i represents the columns of U corresponding to the top-i singular vectors.
中文: 我们mea-Ar=64确定这一数量,并以基于Grassmann距离的正态子空间相似性(见附录G更正式的讨论) QQU i(cid:62) U j ||2 φ (A,A,i,j) = Ar=8 Ar=64 F ∈ [0, 1] (4) r=8 r = 64 min(i,j) U 代表了与上-i单向量对应的U的列.
<a id="S0207"></a> Source: p.11 S0207
Original: Ar=8 Ar=8 φ(·) has a range of [0, 1], where 1 represents a complete overlap of subspaces and 0 a complete separation.
中文: ar=8 ar=8 φ (-)的射程为 [0, 1],其中 1 表示子空间完全重叠,0 为完全分离.
<a id="S0208"></a> Source: p.11 S0208
Original: See Figure 3 for how φ changes as we vary i and j.
中文: 见图3, i和j变化如何。
<a id="S0209"></a> Source: p.11 S0209
Original: We only look at the 48th layer (out of 96) due to space constraint, but the conclusion holds for other layers as well, as shown in Section H.1. 1.0 0.8 0.6 0.4 0.2 0.0 1 6 21 81 32 92 53 04 64 25 85 j i 1 2 3 4 5 6 7 8 Wq 1 6 21 81 32 92 53 04 64 25 85 (Ar =64, Ar =8, i, j) Wv Wq Wv 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 j j j Figure 3: Subspace similarity between column vectors of A and A for both ∆W and ∆W . r=8 r=64 q v The third and the fourth figures zoom in on the lower-left triangle in the first two figures.
中文: 由于空间限制,我们只研究第48层(96层),但结论对其他层也有保留,如H.1节所示:1.0 0.8 0.6 0.4 0.2 0.0 1 6 21 81 32 92 53 04 64 25 85 j i 1 3 4 5 6 7 8 Wq 1 6 81 32 92 53 04 64 25 85 (Ar = 64,Ar = 8 i, j) Wv Wq Wv 1 3 4 5 7 8 1 2 3 4 5 6 7 j j j j j j j j 图3:A和 A列向量在QQW 和 r = 8 r= 64 q v 之间的空间相似性。 第三和第四位数字在前两个数字中放大了左下三角形.
<a id="S0210"></a> Source: p.11 S0210
Original: The top directions in r = 8 are included in r = 64, and vice versa.
中文: r=8的上行方向包含在r=64中,反之亦然.
<a id="S0211"></a> Source: p.11 S0211
Original: We make an important observation from Figure 3.
中文: 我们从图3中得出了重要的意见。
<a id="S0212"></a> Source: p.11 S0212
Original: Directions corresponding to the top singular vector overlap significantly between A and A , while others do not.
中文: 与上单向量相对应的方向在A和A之间明显重叠,而其他方向则不重叠.
<a id="S0213"></a> Source: p.11 S0213
Original: Specifically, ∆W (resp. ∆W ) of A r=8 r=64 v q r=8 and ∆W (resp. ∆W ) of A share a subspace of dimension 1 with normalized v q r=64 similarity > 0.5, providing an explanation of why r = 1 performs quite well in our downstream tasks for GPT-3.
中文: 具体来说,Ar=8 r=64 v q=8的QQW(resp. ∆W)和A的QQW(resp. ∆W)共享了维度1的子空间与正态v q r=64的相似性 > 0.5,解释了r=1为何在我们下游任务中GPT-3的性能相当好.
<a id="S0214"></a> Source: p.11 S0214
Original: Since both A and A are learned using the same pre-trained model, Figure 3 indicates that r=8 r=64 the top singular-vector directions of A and A are the most useful, while other directions r=8 r=64 potentially contain mostly random noises accumulated during training.
中文: 由于A和A都是使用相同的预训模式来学习的,图3显示r=8 r=64,A和A的顶端单数-向导是最有用的,而其他方向r=8 r=64可能包含在训练期间所积累的大部分随机噪声.
<a id="S0215"></a> Source: p.11 S0215
Original: Hence, the adaptation matrix can indeed have a very low rank.
中文: 因此,适应矩阵的排名确实很低。
<a id="S0216"></a> Source: p.11 S0216
Original: Subspace similarity between different random seeds.
中文: 不同随机种子之间的亚空间相似性.
<a id="S0217"></a> Source: p.11 S0217
Original: We further confirm this by plotting the normalized subspace similarity between two randomly seeded runs with r = 64, shown in Figure 4. ∆W appears to have a higher “intrinsic rank” than ∆W , since more common singular value direcq v tions are learned by both runs for ∆W , which is in line with our empirical observation in Table 6. q As a comparison, we also plot two random Gaussian matrices, which do not share any common singular value directions with each other. 7.3 HOW DOES THE ADAPTATION MATRIX ∆W COMPARE TO W ?
中文: 我们进一步证实了这一点,将两个随机播种的跑道与r = 64之间的正态子空间相类似,见图4。 QQW似乎拥有比QQW更高的“内在等级”,因为两个运行都为QQW学习了更常见的单相值 sercq v 等分法,这与表6中我们的经验观察是一致的。 作为比较,我们也绘制出两个随机的高斯矩阵,它们彼此之间没有任何共同的单数值方向. 7.3 " 适应马特利克斯 " 如何协调W?
<a id="S0218"></a> Source: p.11 S0218
Original: We further investigate the relationship between ∆W and W .
中文: 我们进一步调查QQW和W的关系.
<a id="S0219"></a> Source: p.11 S0219
Original: In particular, does ∆W highly correlate with W ? (Or mathematically, is ∆W mostly contained in the top singular directions of W ?) Also, 7Note that a similar analysis can be carried out with B and the left-singular unitary matrices – we stick with A for our experiments. 11
中文: 特别是,QQW与W是否有高度关联? (或者在数学上,QQW主要包含在W的顶端单数方向上?) 另外, 7Note 也可以用 B 和 左 singular 单元矩阵进行类似的分析 — 我们用 A 来进行实验。 第11个
<a id="S0220"></a> Source: p.12 S0220
Original: 0.5 0.4 0.3 0.2 0.1 0.0 1 5 01 51 02 52 03 43 93 44 94 45 95 1 8 16 24 32 40 48 56 j i Wq 1 5 01 51 02 52 03 43 93 44 94 45 95 (Ar=64, A0r=64, i, j) Wv j 1 5 01 51 02 52 03 43 93 44 94 45 95 Random Gaussian j Figure 4: Left and Middle: Normalized subspace similarity between the column vectors of A r=64 from two random seeds, for both ∆W and ∆W in the 48-th layer.
中文: 0: 0: 0: 0: 0: 0: 1 5: 51: 02: 52: 03: 43: 93: 44: 94: 45: 95: 1: 8: 16: 24: 32: 40: 48: 5: 01: 02: 03: 43: 93: 44: 95(Ar=64, A0r=64,i,j) Wv: 1: 5: 01: 02: 52:03: 43: 93: 44: 94: 45: 95 随机高斯克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克
<a id="S0221"></a> Source: p.12 S0221
Original: Right: the same heat-map q v between the column vectors of two random Gaussian matrices.
中文: 右:两个随机高斯矩阵的列向量之间的同热相图 q v.
<a id="S0222"></a> Source: p.12 S0222
Original: See Section H.1 for other layers. how “large” is ∆W comparing to its corresponding directions in W ?
中文: 其他层见H.1节。 与W中的相应方向相比,
<a id="S0223"></a> Source: p.12 S0223
Original: This can shed light on the underlying mechanism for adapting pre-trained language models.
中文: 这可以揭示调整预训语言模型的基本机制.
<a id="S0224"></a> Source: p.12 S0224
Original: To answer these questions, we project W onto the r-dimensional subspace of ∆W by computing U (cid:62)W V (cid:62), with U /V being the left/right singular-vector matrix of ∆W .
中文: 为了回答这些问题,我们通过计算U(cid:62)W V(cid:62)将W投射到QQW的r-维子空间上,其中U/V是QQW的左/右单数-矢量矩阵.
<a id="S0225"></a> Source: p.12 S0225
Original: Then, we compare the Frobenius norm between (cid:107)U (cid:62)W V (cid:62)(cid:107) and (cid:107)W (cid:107) .
中文: 然后,我们把弗罗贝尼乌斯规范比作(Cid:107)U(Cid:62)W V(Cid:62)(Cid:107)和(Cid:107)W(Cid:107).
<a id="S0226"></a> Source: p.12 S0226
Original: As a comparison, we also compute F F (cid:107)U (cid:62)W V (cid:62)(cid:107) by replacing U, V with the top r singular vectors of W or a random matrix. F r = 4 r = 64 ∆W W Random ∆W W Random q q q q ||U (cid:62)W V (cid:62)|| = 0.32 21.67 0.02 1.90 37.71 0.33 q F ||W || = 61.95 ||∆W || = 6.91 ||∆W || = 3.57 q F q F q F Table 7: The Frobenius norm of U (cid:62)W V (cid:62) where U and V are the left/right top r singular vector q directions of either (1) ∆W , (2) W , or (3) a random matrix.
中文: 作为比较,我们还计算出F(Cid:107)U(Cid:62)W V(Cid:62)(Cid:107),取而代之的是以W的顶端r单向量或随机矩阵. F r = 4 r = 64 → W r → W 随机 → W 随机 q q q q q → U (cid:62)W V (cid:62) = 0.32 21.67 0.02 1.90 37.71 0.33 q F → || W = 61.95 → || W = 6.91 → ||∆ W = 3.57 q F q q F q F 表7: U (cid:62)W V (cid:62) 其中U和V是(1) → W,(2) W 或 (3) 随机矩阵的左/右上方正 R 单向量q方向.
<a id="S0227"></a> Source: p.12 S0227
Original: The weight matrices are taken from q q the 48th layer of GPT-3.
中文: 重量矩阵取自q q 第48层GPT-3.
<a id="S0228"></a> Source: p.12 S0228
Original: We draw several conclusions from Table 7.
中文: 我们从表7中得出若干结论。
<a id="S0229"></a> Source: p.12 S0229
Original: First, ∆W has a stronger correlation with W compared to a random matrix, indicating that ∆W amplifies some features that are already in W .
中文: 首先,与随机矩阵相比,QQW与W的关联性更强,表明QQW放大了已经存在于W中的一些特性.
<a id="S0230"></a> Source: p.12 S0230
Original: Second, instead of repeating the top singular directions of W , ∆W only amplifies directions that are not emphasized in W .
中文: 第二,QQW不重复W的顶端单向,只放大了W中未强调的方向.
<a id="S0231"></a> Source: p.12 S0231
Original: Third, the amplification factor is rather huge: 21.5 ≈ 6.91/0.32 for r = 4.
中文: 第三,放大系数相当大:21.5 ≈ 6.91/0.32 r= 4.
<a id="S0232"></a> Source: p.12 S0232
Original: See Section H.4 for why r = 64 has a smaller amplification factor.
中文: 见H.4节,r = 64为什么放大系数较小。
<a id="S0233"></a> Source: p.12 S0233
Original: We also provide a visualization in Section H.3 for how the correlation changes as we include more top singular directions from W . q This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model. 8 CONCLUSION AND FUTURE WORK Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks.
中文: 我们还在H.3节中提供可视化,说明在我们包括来自W.q.的更上等单数方向时,相关性是如何变化的。 这表明,低级适应矩阵可能扩大在一般培训前模式中已经学到但未得到强调的具体下游任务的重要特征。 8 从硬件和为不同任务主办独立实例的存储/交换费用来看,结业和今后的工作精细调整了巨大的语言模型,费用高昂。
<a id="S0234"></a> Source: p.12 S0234
Original: We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality.
中文: 我们建议采用LORA这一高效的适应战略,既不引入推断延迟,也不减少输入序列长度,同时保留高模型质量。
<a id="S0235"></a> Source: p.12 S0235
Original: Importantly, it allows for quick task-switching when deployed as a service by sharing the vast majority of the model parameters.
中文: 重要的是,它允许在作为服务部署时通过共享绝大多数模型参数快速进行任务转换。
<a id="S0236"></a> Source: p.12 S0236
Original: While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.
中文: 虽然我们专注于变形器语言模型,但所拟议的原理一般适用于有密集地层的任何神经网络.
<a id="S0237"></a> Source: p.12 S0237
Original: There are many directions for future works. 1) LoRA can be combined with other efficient adaptation methods, potentially providing orthogonal improvement. 2) The mechanism behind fine-tuning or LoRA is far from clear – how are features learned during pre-training transformed to do well on downstream tasks?
中文: 未来作品有许多方向. (1) LORA可以与其他高效的适应方法相结合,有可能提供正交改进。 2) 微调或LORA背后的机制还远不清楚 — — 在前期训练中学到的特征是如何转化成在下游任务上做得好的?
<a id="S0238"></a> Source: p.12 S0238
Original: We believe that LoRA makes it more tractable to answer this than full fine- 12
中文: 我们认为,LORA 使得它更容易 回答这一点 而不是完全的精细。
<a id="S0239"></a> Source: p.13 S0239
Original: tuning. 3) We mostly depend on heuristics to select the weight matrices to apply LoRA to.
中文: 调音. 3)我们主要依靠休克来选择将LORA应用到的重量矩阵.
<a id="S0240"></a> Source: p.13 S0240
Original: Are there more principled ways to do it? 4) Finally, the rank-deficiency of ∆W suggests that W could be rank-deficient as well, which can also be a source of inspiration for future works.
中文: 有没有更多的原则性方法来做到这一点? 4)最后,QQW的军衔不足也暗示了W也可以是军衔不足,这也可能成为未来作品的灵感来源.
<a id="S0241"></a> Source: p.13 S0241
Original: REFERENCES Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta.
中文: 参考Armen Aghajanyan、Luke Zettlemoyer和索纳尔·古普塔。
<a id="S0242"></a> Source: p.13 S0242
Original: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. arXiv:2012.13255 [cs], December 2020.
中文: Intrinsic 维度解释语言模型精致-通取的效果. arXiv:2012.13255 [cs],2020年12月.
<a id="S0243"></a> Source: p.13 S0243
Original: What Can ResNet Learn Efficiently, Going Beyond Kernels?
中文: ResNet能有效学习什么?
<a id="S0244"></a> Source: p.13 S0244
Original: Full version available at http://arxiv.org/abs/1905.10337.
中文: 全文见http://arxiv.org/abs/1905.10337。
<a id="S0245"></a> Source: p.13 S0245
Original: Backward feature correction: How deep learning performs deep learning. arXiv preprint arXiv:2001.04413, 2020a.
中文: 后向特征校正:深层学习如何进行深层学习. arXiv预印版arXiv:2001.04413,2020a.
<a id="S0246"></a> Source: p.13 S0246
Original: Feature purification: How adversarial training performs robust deep learning. arXiv preprint arXiv:2005.10190, 2020b.
中文: 特性净化:对抗性训练如何进行强有力的深入学习。 arXiv preprint arXiv:2005.10190, 2020b. (中文(简体) ).
<a id="S0247"></a> Source: p.13 S0247
Original: Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overparameterization.
中文: 泽相阿伦-朱相会; 元zh李相会; 赵相会. 通过过度参数化进行深度学习的趋同理论.
<a id="S0248"></a> Source: p.13 S0248
Original: Full version available at http://arxiv.org/abs/1811. 03962.
中文: 全文见http://arxiv.org/abs/1811.03962.
<a id="S0249"></a> Source: p.13 S0249
Original: Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E.
中文: Jimmy Lei Ba, Jamie Ryan Kiros, 和杰弗里·E.
<a id="S0250"></a> Source: p.13 S0250
Original: Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M.
中文: Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafula Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. 维基语录链接 维基语录链接 维基语录链接 维基语录链接 维基文库中相关的条目: 维基语录链接 维基语录链接 维基文库中相关的条目: 维基语录 维基语录 维基语录链接: 维基文库尔语录 维基文库尔语录 维基文库尔语录 维文库尔语录 维文库尔语库尔语库尔语库 维文库尔语库尔语库尔语库尔语
<a id="S0251"></a> Source: p.13 S0251
Original: Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
中文: 齐格勒,杰弗里·吴,克列门斯·温特,克里斯托弗·黑森,马克·陈,埃里克·西格勒,马特乌斯·利特温,斯科特·格雷,本杰明·切斯,杰克·克拉克,克里斯托弗·伯纳,萨姆·麦坎德利希,阿莱克·拉德福德,伊利亚·苏特斯克维尔,和达里奥·阿莫代.
<a id="S0252"></a> Source: p.13 S0252
Original: Language Models are Few-Shot Learners. arXiv:2005.14165 [cs], July 2020.
中文: 语言模型是"少有"的学习者. arXiv:2005.14165 [cs],2020年7月.
<a id="S0253"></a> Source: p.13 S0253
Original: Jian-Feng Cai, Emmanuel J Cande`s, and Zuowei Shen. A singular value thresholding algorithm for matrix completion.
中文: 建丰丙辰,伊曼努埃尔·J·坎德斯,同为沈祖伟. 矩阵完成的单值阈值算法 。
<a id="S0254"></a> Source: p.13 S0254
Original: SIAM Journal on optimization, 20(4):1956–1982, 2010.
中文: SIAM杂志关于优化,20(4):1956-1982,2010.
<a id="S0255"></a> Source: p.13 S0255
Original: Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia.
中文: 丹尼尔·塞尔,莫娜·迪亚布,埃内科·阿吉尔,伊尼戈·洛佩斯-加兹皮奥和露西娅·斯佩克亚.
<a id="S0256"></a> Source: p.13 S0256
Original: Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.
中文: Semeval-2017任务一:语义文字相似多语种和跨语言重点评价.
<a id="S0257"></a> Source: p.13 S0257
Original: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 2017. doi: 10.18653/ v1/s17-2001.
中文: 第11届语义学评价国际研讨会纪要(SemEval-2017),2017. doi:10.18653/v1/s17-2001.
<a id="S0258"></a> Source: p.13 S0258
Original: URL http://dx.doi.org/10.18653/v1/S17-2001.
中文: URL: http://dx.doi.org/10.18653/v1/S17-2001. URL.
<a id="S0259"></a> Source: p.13 S0259
Original: Ronan Collobert and Jason Weston. A unified architecture for natural language processing: deep neural networks with multitask learning.
中文: 罗南·科洛伯特和杰森·韦斯顿. 自然语言处理的统一架构:具有多任务学习的深层神经网络.
<a id="S0260"></a> Source: p.13 S0260
Original: In Proceedings of the 25th international conference on Machine learning, ICML ’08, pp. 160–167, New York, NY, USA, July 2008.
中文: 《第25届机器学习国际会议记录》,ICML ' 08,第160-167页,纽约,美国纽约,2008年7月。
<a id="S0261"></a> Source: p.13 S0261
Original: ISBN 978-1-60558-205-4. doi: 10.1145/1390156.1390177.
中文: ISBN 978-1-60558-205-4. doi:10.1145/1390156.1390177 (中文(简体) ).
<a id="S0262"></a> Source: p.13 S0262
Original: URL https://doi.org/10.1145/1390156.1390177.
中文: URL https://doi.org/10.1145/1390156.1390177 (中文(简体) ).
<a id="S0263"></a> Source: p.13 S0263
Original: Misha Denil, Babak Shakibi, Laurent Dinh, Marc’Aurelio Ranzato, and Nando de Freitas.
中文: 米莎·德尼尔(英语:Misha Denil),巴巴克·沙基比(英语:Babak Shakibi),洛朗·丁(英语:Laurent Dinh),马克·奥雷利奥·兰扎托(英语:Marc-Aurelio Ranzato)和南多·德·弗雷塔斯(英语:Nando de Freitas).
<a id="S0264"></a> Source: p.13 S0264
Original: Predicting parameters in deep learning, 2014.
中文: 深度学习的预测参数,2014年.
<a id="S0265"></a> Source: p.13 S0265
Original: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
中文: 雅各布·德夫林(Jacob Devlin),明-魏相,克通·李(Kenton Lee)和克丽斯蒂娜·图塔诺娃.
<a id="S0266"></a> Source: p.13 S0266
Original: Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.
中文: Bert:深双向变压器前期训练为语言理解,2019a.
<a id="S0267"></a> Source: p.13 S0267
Original: Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
中文: 雅各布·德夫林(Jacob Devlin),明-魏相,克通·李(Kenton Lee)和克丽斯蒂娜·图塔诺娃.
<a id="S0268"></a> Source: p.13 S0268
Original: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019b.
中文: BERT:深双向变形器为语言理解预训. arXiv:181.04805 [cs],2019年5月b.
<a id="S0269"></a> Source: p.13 S0269
Original: URL http://arxiv.org/abs/1810.04805. arXiv: 1810.04805.
中文: URL: http://arxiv.org/abs/1810.04805. arXiv:1810.04805.
<a id="S0270"></a> Source: p.13 S0270
Original: Automatically constructing a corpus of sentential paraphrases.
中文: 自动地构建出一系列的感人语句.
<a id="S0271"></a> Source: p.13 S0271
Original: In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
中文: 2005年第三次国际审校讲习班记录。
<a id="S0272"></a> Source: p.13 S0272
Original: Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini.
中文: 克莱尔·花园特(Claire Gardent),安娜斯塔西亚·希莫里娜(Anastasia Shimorina),沙希·纳拉扬(Shashi Narayan)和劳拉·佩雷斯-贝尔特拉奇尼(Laura Perez-Beltrachini).
<a id="S0273"></a> Source: p.13 S0273
Original: The webnlg challenge: Generating text from rdf data.
中文: webnlg挑战:从rdf数据生成文本.
<a id="S0274"></a> Source: p.13 S0274
Original: In Proceedings of the 10th International Conference on Natural Language Generation, pp. 124–133, 2017. 13
中文: 《第十届自然语言生成国际会议记录》,第124-133页,2017年。
<a id="S0275"></a> Source: p.14 S0275
Original: Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari.
中文: Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, 和安德烈亚·蒙塔纳里.
<a id="S0276"></a> Source: p.14 S0276
Original: When do neural networks outperform kernel methods? arXiv preprint arXiv:2006.13409, 2020.
中文: 神经网络何时超过内核方法? arXiv preprint arXiv:2006.13409, 2020 (英语).
<a id="S0277"></a> Source: p.14 S0277
Original: Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer.
中文: 博格丹·格利瓦,伊花娜·莫克霍尔,梅西杰·比塞克,和阿莱克桑德·沃尔.
<a id="S0278"></a> Source: p.14 S0278
Original: Samsum corpus: A humanannotated dialogue dataset for abstractive summarization.
中文: Samsumecurity:一个人性化附加说明的对话数据集,用于抽象的概括.
<a id="S0279"></a> Source: p.14 S0279
Original: Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques.
中文: 拉尔斯·格拉斯迪克,丹尼尔·克雷斯纳,和克莉丝汀·托布勒. 低档收视率近似技术文献调查.
<a id="S0280"></a> Source: p.14 S0280
Original: Grassmann discriminant analysis: a unifying view on subspace-based learning.
中文: Grassmann 歧见分析:关于亚空间学习的统一观点.
<a id="S0281"></a> Source: p.14 S0281
Original: URL https://doi.org/10.1145/1390156. 1390204.
中文: URL https://doi.org/10.1145/1390156. 1390204 (英语).
<a id="S0282"></a> Source: p.14 S0282
Original: Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May.
中文: 卡伦·哈姆巴祖米扬,克兰特·哈恰特里安和乔纳森·梅.
<a id="S0283"></a> Source: p.14 S0283
Original: WARP: Word-level Adversarial ReProgramming. arXiv:2101.00121 [cs], December 2020.
中文: WARP: Word-level Adversarial Reprogramming. arXiv:2101.00121 [cs],2020年12月.
<a id="S0284"></a> Source: p.14 S0284
Original: URL http://arxiv.org/abs/ 2101.00121. arXiv: 2101.00121.
中文: URL: http://arxiv.org/abs/2101.00121. arXiv:2101.00121.
<a id="S0285"></a> Source: p.14 S0285
Original: Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.
中文: 彭城贺,刘晓东;建丰高;魏祖陈.
<a id="S0286"></a> Source: p.14 S0286
Original: Deberta: Decoding-enhanced bert with disentangled attention, 2021.
中文: Deberta:解码-增强的伯特 有分散的注意力,2021年.
<a id="S0287"></a> Source: p.14 S0287
Original: Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
中文: 尼尔·胡尔斯比,安德烈·朱尔吉乌,斯坦尼斯瓦夫·贾斯特泽布斯基,布鲁纳·莫尔龙,昆廷·德·拉鲁西尔赫,安德烈亚·盖斯蒙多,莫娜·阿塔里扬,和西尔万·盖利.
<a id="S0288"></a> Source: p.14 S0288
Original: Parameter-Efficient Transfer Learning for NLP. arXiv:1902.00751 [cs, stat], June 2019.
中文: NLP. arXiv:1902.00751 [cs, stat],2019年6月. Parameter-Efficial Transfer Language for NLP. arXiv:1902.00751 [cs, stat].
<a id="S0289"></a> Source: p.14 S0289
Original: Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman.
中文: 马克斯·贾德伯格,安德里亚·维达尔迪,和安德鲁·齐瑟曼.
<a id="S0290"></a> Source: p.14 S0290
Original: Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866, 2014.
中文: 以低级扩能加快革命神经网络建设. arXiv preprint arXiv:1405.3866, 2014 (英语).
<a id="S0291"></a> Source: p.14 S0291
Original: Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo` Fusi.
中文: 米哈伊尔·克达克(Mikhail Khodak),尼尔·特宁霍尔茨(Neil Tenenholtz),莱斯特·麦基(Lester Mackey)和尼科洛"富西".
<a id="S0292"></a> Source: p.14 S0292
Original: Initialization and regularization of factorized neural layers, 2021.
中文: 2021年因子化神经地层初始化和规范化.
<a id="S0293"></a> Source: p.14 S0293
Original: Adam: A method for stochastic optimization, 2017.
中文: 亚当:2017年花样优化的一种方法.
<a id="S0294"></a> Source: p.14 S0294
Original: Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
中文: 德米特里·莱皮克欣,许克俊·李克用,许元忠,陈德豪,鄂尔罕·费拉特,黄延平,马克西姆·克里克恩,诺姆·沙泽尔,陈志芬.
<a id="S0295"></a> Source: p.14 S0295
Original: Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
中文: Gshard:用有条件的计算和自动的硬化来放大巨型模型, 2020.
<a id="S0296"></a> Source: p.14 S0296
Original: Brian Lester, Rami Al-Rfou, and Noah Constant.
中文: 布莱恩·莱斯特(Brian Lester),拉米·阿尔-勒福(Rami Al-Rfou),和诺亚·康斯坦特(Noah Constant).
<a id="S0297"></a> Source: p.14 S0297
Original: The Power of Scale for Parameter-Efficient Prompt Tuning. arXiv:2104.08691 [cs], April 2021.
中文: (原始内容存档于2017-10-29) (中文(简体) ). The Political Power for Parameter-Efficient Point Tuning. arXiv:2104.08691 [cs], 2021年4月.
<a id="S0298"></a> Source: p.14 S0298
Original: URL http://arxiv.org/abs/2104.08691. arXiv: 2104.08691.
中文: URL: http://arxiv.org/abs/2104.08691. arXiv: 2104.08691. (中文(简体) ).
<a id="S0299"></a> Source: p.14 S0299
Original: Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski.
中文: 李春由,海拉德·法克豪尔,罗珊·刘和杰森·约辛斯基.
<a id="S0300"></a> Source: p.14 S0300
Original: Measuring the Intrinsic Dimension of Objective Landscapes. arXiv:1804.08838 [cs, stat], April 2018a.
中文: 测量客观地貌的内在维度. arXiv:1804.08838 [cs, stat], 2018年4月a.
<a id="S0301"></a> Source: p.14 S0301
Original: URL http: //arxiv.org/abs/1804.08838. arXiv: 1804.08838.
中文: URL http://arxiv.org/abs/1804.08838. arXiv:1804.08838.
<a id="S0302"></a> Source: p.14 S0302
Original: Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.00190 [cs], January 2021.
中文: 前缀-图宁:优化连续提示以促一代. arXiv:2101.00190 [cs],2021年1月.
<a id="S0303"></a> Source: p.14 S0303
Original: Learning overparameterized neural networks via stochastic gradient descent on structured data.
中文: 通过结构化数据的分层梯度下降来学习超参数神经网络.
<a id="S0304"></a> Source: p.14 S0304
Original: In Advances in Neural Information Processing Systems, 2018.
中文: 2018年神经信息处理系统预科.
<a id="S0305"></a> Source: p.14 S0305
Original: Yuanzhi Li, Yingyu Liang, and Andrej Risteski.
中文: 李元志; 梁英玉; 安德烈·里斯特斯基.
<a id="S0306"></a> Source: p.14 S0306
Original: Recovery guarantee of weighted low-rank approximation via alternating minimization.
中文: 通过交替最小化,保证回收加权低等近似值。
<a id="S0307"></a> Source: p.14 S0307
Original: In International Conference on Machine Learning, pp. 2358–2367.
中文: 机器学习国际会议,第2358至2367页。
<a id="S0308"></a> Source: p.14 S0308
Original: Yuanzhi Li, Tengyu Ma, and Hongyang Zhang.
中文: 元子李从珂;滕玉马;洪阳张从 Zhang.
<a id="S0309"></a> Source: p.14 S0309
Original: Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations.
中文: 超参数化基质感测和神经网络中的算法规范化,有四维活化.
<a id="S0310"></a> Source: p.14 S0310
Original: In Conference On Learning Theory, pp. 2–47.
中文: 《学习理论会议》,第2-47页。
<a id="S0311"></a> Source: p.14 S0311
Original: Zhaojiang Lin, Andrea Madotto, and Pascale Fung.
中文: 赵江林; 安德里亚·马多托; 帕斯克多·冯.
<a id="S0312"></a> Source: p.14 S0312
Original: Exploring versatile generative language model via parameter-efficient transfer learning.
中文: 通过参数高效转移学习,探索多功能基因语言模式.
<a id="S0313"></a> Source: p.14 S0313
Original: In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 441–459, Online, November 2020.
中文: 计算语言学协会的调查结果:EMNLP 2020, 第441-459页, Online, 2020年11月.
<a id="S0314"></a> Source: p.14 S0314
Original: Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.41.
中文: 计算语言学协会. doi:10.18653/v1/2020. finders-emnlp.41.
<a id="S0315"></a> Source: p.14 S0315
Original: URL https://aclanthology. org/2020.findings-emnlp.41. 14
中文: URL https://aclanthology. (原始内容存档于2018-09-29). org/2020. 调查-emnlp.41.
<a id="S0316"></a> Source: p.15 S0316
Original: Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang.
中文: 萧刘,延安郑,郑克道,明定,于杰等入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关出关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关入关
<a id="S0317"></a> Source: p.15 S0317
Original: GPT Understands, Too. arXiv:2103.10385 [cs], March 2021.
中文: GPT understands, Too. arXiv:2103.10385 [cs],2021年3月.
<a id="S0318"></a> Source: p.15 S0318
Original: URL http://arxiv.org/abs/ 2103.10385. arXiv: 2103.10385.
中文: URL http://arxiv.org/abs/21031.0385. arXiv:21031.0385 (中文(简体) ).
<a id="S0319"></a> Source: p.15 S0319
Original: Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
中文: 刘燕汉,弥勒·奥特,纳南·高雅尔,静菲·杜,万达·乔希,丹奇·陈,欧梅尔·李维,迈克·刘易斯,卢克·泽特尔莫耶,和维塞林·斯托亚诺夫.
<a id="S0320"></a> Source: p.15 S0320
Original: Roberta: A robustly optimized bert pretraining approach, 2019.
中文: 罗伯塔:2019年强力优化伯特预训方式.
<a id="S0321"></a> Source: p.15 S0321
Original: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
中文: 脱钩后体重衰减规范化. arXiv预印 arXiv:1711.05101, 2017.
<a id="S0322"></a> Source: p.15 S0322
Original: Decoupled weight decay regularization, 2019.
中文: 2019年脱钩后体重衰减规范化.
<a id="S0323"></a> Source: p.15 S0323
Original: Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder.
中文: 拉贝赫·卡里米·马哈巴迪,詹姆斯·亨德森,和塞巴斯蒂安·鲁德.
<a id="S0324"></a> Source: p.15 S0324
Original: Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
中文: 收缩器:高效低级超复杂适配器层,2021.
<a id="S0325"></a> Source: p.15 S0325
Original: Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al.
中文: 林永南,德拉戈米尔·拉德夫,鲁伊·张,阿姆里特·劳,阿比南德·西瓦普拉萨德,奇亚春·希耶,相克·唐,阿迪特·维亚斯,内哈·韦尔马,普拉纳夫·克里希纳等.
<a id="S0326"></a> Source: p.15 S0326
Original: Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871, 2020.
中文: 达特:开放域结构化数据记录到文本生成. arXiv preprint arXiv: 2007.02871, 2020 (英语).
<a id="S0327"></a> Source: p.15 S0327
Original: Jekaterina Novikova, Ondˇrej Dusˇek, and Verena Rieser.
中文: 杰卡捷琳娜·诺维科娃(英语:Jekaterina Novikova),翁德切雷克·杜斯切克(英语:Ondˇrej Dusˇek)和维莱娜·里瑟(英语:Verena Rieser).
<a id="S0328"></a> Source: p.15 S0328
Original: The e2e dataset: New challenges for endto-end generation. arXiv preprint arXiv:1706.09254, 2017.
中文: e2e数据集:端到端生成的新挑战. arXiv预印 arXiv:1706.09254, 2017.
<a id="S0329"></a> Source: p.15 S0329
Original: Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi.
中文: 萨梅特·奥伊马克(Samet Oymak),扎兰·法比安(Zalan Fabian),明琴·李(Mingchen Li)和马赫迪·索尔塔诺克塔比(Mahdi Soltanolkotabi).
<a id="S0330"></a> Source: p.15 S0330
Original: Generalization guarantees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint arXiv:1906.05392, 2019.
中文: 利用雅可比亚低等结构,为神经网络提供通俗化保障. arXiv预印版 arXiv:1906.05392, 2019.
<a id="S0331"></a> Source: p.15 S0331
Original: Jonas Pfeiffer, Aishwarya Kamath, Andreas Ru¨ckle´, Kyunghyun Cho, and Iryna Gurevych.
中文: 乔纳斯·菲佛,艾斯瓦里亚·卡马特,安德烈亚斯·鲁克勒克,克林亨·乔和伊莉娜·格列维奇.
<a id="S0332"></a> Source: p.15 S0332
Original: Adapterfusion: Non-destructive task composition for transfer learning, 2021.
中文: 适应力化:转移学习的无损任务构成,2021.
<a id="S0333"></a> Source: p.15 S0333
Original: Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and Sanjeev Khudanpur.
中文: 由丹尼尔·波维,高丰成,王一明,克力,海南徐能,马哈萨·亚尔莫哈马迪,桑杰耶夫·胡丹普.
<a id="S0334"></a> Source: p.15 S0334
Original: Semi-orthogonal low-rank matrix factorization for deep neural networks.
中文: 深神经网络的半正弦低等基质分化.
<a id="S0335"></a> Source: p.15 S0335
Original: Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
中文: 阿莱克·拉德福德,卡尔蒂克·纳拉西姆汉,蒂姆·萨利曼斯和伊利亚·苏特斯克韦.
<a id="S0336"></a> Source: p.15 S0336
Original: Improving Language Understanding by Generative Pre-Training. pp. 12, a.
中文: 改进对语言的理解,由Generative Pre-Training编写。
<a id="S0337"></a> Source: p.15 S0337
Original: Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
中文: 阿莱克·拉德福德,杰弗里·吴,雷园童,大卫·卢安,达里奥·阿莫代,和伊利亚·苏特斯克韦.
<a id="S0338"></a> Source: p.15 S0338
Original: Language Models are Unsupervised Multitask Learners. pp. 24, b.
中文: 语言模型是无监督的多任务学习者. pp. 24, b.
<a id="S0339"></a> Source: p.15 S0339
Original: Pranav Rajpurkar, Robin Jia, and Percy Liang.
中文: Pranav Rajpurkar, Robin Jia, 和Percy Liang. (原始内容存档于2019-09-21).
<a id="S0340"></a> Source: p.15 S0340
Original: Know what you don’t know: Unanswerable questions for squad.
中文: 了解你所不知道的:无法回答的问题。
<a id="S0341"></a> Source: p.15 S0341
Original: Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.
中文: 西尔维斯特雷-阿尔维塞·雷布菲,哈坎·比伦和安德烈亚·维达尔迪.
<a id="S0342"></a> Source: p.15 S0342
Original: Learning multiple visual domains with residual adapters. arXiv:1705.08045 [cs, stat], November 2017.
中文: 以剩余适配器学习多视觉域. arXiv:1705.08045 [cs, stat], 2017年11月.
<a id="S0343"></a> Source: p.15 S0343
Original: URL http://arxiv.org/ abs/1705.08045. arXiv: 1705.08045.
中文: URL http://arxiv.org/abs/1705.08045. arXiv:1705.08045.
<a id="S0344"></a> Source: p.15 S0344
Original: Andreas Ru¨ckle´, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych.
中文: 安德烈亚斯·鲁·克勒 格雷高·盖格尔 马克斯·格洛克纳 蒂尔曼·贝克 乔纳斯·菲弗 尼尔斯·里默斯 和伊莉娜·格列维奇
<a id="S0345"></a> Source: p.15 S0345
Original: Adapterdrop: On the efficiency of adapters in transformers, 2020.
中文: 适配器投放 : 关于变压器中适配器的效率, 2020.
<a id="S0346"></a> Source: p.15 S0346
Original: Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran.
中文: 塔拉·恩·赛纳斯(Tara N Sainath),布赖恩·金斯伯里(Brian Kingsbury),维卡斯·辛德瓦尼(Vikas Sindhwani),埃布鲁·阿里索伊(Ebru Arisoy)和布瓦娜·拉马布哈德兰(Bhuvana Ramabhadran).
<a id="S0347"></a> Source: p.15 S0347
Original: Lowrank matrix factorization for deep neural network training with high-dimensional output targets.
中文: 以高维度输出目标进行深神经网络训练的低等矩阵分级.
<a id="S0348"></a> Source: p.15 S0348
Original: In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655– 6659.
中文: 2013年,IEEE关于声学,语音和信号处理的国际会议,第6655–6659页.
<a id="S0349"></a> Source: p.15 S0349
Original: Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.
中文: 穆罕默德·肖伊比,莫斯托法·帕特瓦里,劳尔·普里,帕特里克·勒格雷斯利,贾里德·卡斯珀和布莱恩·卡坦扎罗.
<a id="S0350"></a> Source: p.15 S0350
Original: Megatron-lm: Training multi-billion parameter language models using model parallelism, 2020.
中文: 威震天-lm:使用模型并行论来训练数十亿参数语言模型, 2020.
<a id="S0351"></a> Source: p.15 S0351
Original: Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D.
中文: 理查德·索彻(Richard Socher),亚历克斯·佩雷利金(Alex Perelygin),让·吴(Jean Wu),杰森·楚昂(Jason Chuang),克里斯托弗·德.
<a id="S0352"></a> Source: p.15 S0352
Original: Manning, Andrew Ng, and Christopher Potts.
中文: 曼宁,安德鲁·恩克, 和克里斯多弗·波茨.
<a id="S0353"></a> Source: p.15 S0353
Original: Recursive deep models for semantic compositionality over a sentiment treebank.
中文: 语义成分的回溯性深层模型 比情感树库。
<a id="S0354"></a> Source: p.15 S0354
Original: In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA, October 2013.
中文: 《2013年自然语言处理经验方法会议纪要》,第1631至1642页,美国华盛顿西雅图,2013年10月。
<a id="S0355"></a> Source: p.15 S0355
Original: Association for Computational Linguistics.
中文: 计算语言学协会.
<a id="S0356"></a> Source: p.15 S0356
Original: URL https://aclanthology.org/D13-1170. 15
中文: URL https://aclanthology.org/D13-1170. 15 (中文(简体) ).
<a id="S0357"></a> Source: p.16 S0357
Original: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
中文: 阿希什·瓦斯瓦尼,诺姆·沙泽尔,尼基·帕尔马尔,雅克布·乌斯克克赖特,利翁·琼斯,艾丹·恩·戈麦斯,克萨斯·凯泽和伊利亚·波罗舒欣.
<a id="S0358"></a> Source: p.16 S0358
Original: In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010, 2017.
中文: 第31届神经信息处理系统国际会议记录,第6000-6010页,2017年。
<a id="S0359"></a> Source: p.16 S0359
Original: Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
中文: 亚历克斯·王,阿曼普雷特·辛格,朱利安·迈克尔,费利克斯·希尔,奥梅尔·李维,和塞缪尔·R.
<a id="S0360"></a> Source: p.16 S0360
Original: Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
中文: Glue:自然语言理解的多任务基准和分析平台,2019.
<a id="S0361"></a> Source: p.16 S0361
Original: Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R.
中文: Alex Wang, Yada Pruksachatkun, Nikita Nangia, 阿曼普雷特·辛格, 朱利安·迈克尔, 费利克斯·希尔, 奥梅尔·利维,和塞缪尔·R.
<a id="S0362"></a> Source: p.16 S0362
Original: Superglue: A stickier benchmark for general-purpose language understanding systems, 2020.
中文: Superglue:通用语言理解系统粘接基准,2020年.
<a id="S0363"></a> Source: p.16 S0363
Original: Alex Warstadt, Amanpreet Singh, and Samuel R Bowman.
中文: 亚历克斯·沃斯塔特,阿曼普雷特·辛格和塞缪尔·罗·鲍曼.
<a id="S0364"></a> Source: p.16 S0364
Original: Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018.
中文: 神经网络可接受性判断. arXiv preprint arXiv:1805.12471, 2018 (英语).
<a id="S0365"></a> Source: p.16 S0365
Original: Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference.
中文: 阿迪娜·威廉姆斯,尼基塔·南吉亚,和塞缪尔·鲍曼. 通过推论理解判决的广覆盖质疑书。
<a id="S0366"></a> Source: p.16 S0366
Original: In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112–1122, New Orleans, Louisiana, June 2018.
中文: 计算语言学协会北美分会2018年会议纪要:人文语言技术 第1卷(长篇论文),第1112–1122页,路易斯安那新奥尔良,2018年6月.
<a id="S0367"></a> Source: p.16 S0367
Original: Association for Computational Linguistics. doi: 10.18653/v1/N18-1101.
中文: 计算语言学协会. doi:10.18653/v1/N18-1101.
<a id="S0368"></a> Source: p.16 S0368
Original: URL https://www.aclweb. org/anthology/N18-1101.
中文: URL https://www.aclweb.org/anthology/N18-1101. 页面存档备份,存于互联网档案馆.
<a id="S0369"></a> Source: p.16 S0369
Original: Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Re´mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M.
中文: 托马斯·沃尔夫,莱桑德·德布特,维克多·桑赫,朱利安·乔蒙德,克莱门特·德兰格,安东尼·莫伊,皮尔里克·西斯塔克,蒂姆·劳特,雷米·卢夫,摩根·丰托维茨,乔·戴维森,萨姆·施莱费尔,帕特里克·冯·普拉滕,克拉拉·马,雅辛·杰尼特,朱利安·普鲁,坎文·许,特文·勒·斯考,西尔万·古格,玛丽亚玛·德拉姆,昆廷·莱斯特和亚历山大·米.
<a id="S0370"></a> Source: p.16 S0370
Original: Transformers: State-of-the-art natural language processing.
中文: 变相器:最先进的自然语言处理.
<a id="S0371"></a> Source: p.16 S0371
Original: In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45, Online, October 2020.
中文: 《2020年自然语言处理经验方法会议纪要:系统演示》,第38-45页,Online,2020年10月。
<a id="S0372"></a> Source: p.16 S0372
Original: Association for Computational Linguistics.
中文: 计算语言学协会.
<a id="S0373"></a> Source: p.16 S0373
Original: URL https://www.aclweb.org/anthology/ 2020.emnlp-demos.6.
中文: URL https://www.aclweb.org/anthology/2020.emnlp-demos.6. 互联网档案馆的存檔,存档日期2013-12-21.
<a id="S0374"></a> Source: p.16 S0374
Original: Feature Learning in Infinite-Width Neural Networks. arXiv:2011.14522 [cond-mat], May 2021.
中文: (原始内容存档于2017-10-11). Feature Learning in Infinite-Width Neural Networks. arXiv:2011.14522 [cond-mat],2021年5月.
<a id="S0375"></a> Source: p.16 S0375
Original: URL http://arxiv.org/abs/2011.14522. arXiv: 2011.14522.
中文: URL: http://arxiv.org/abs/2011.14522. arXiv: 2011.14522. 互联网档案馆的存檔,存档日期2011-12-22.
<a id="S0376"></a> Source: p.16 S0376
Original: Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg.
中文: 埃拉德·本·扎肯,肖利·拉夫福格尔和约夫·戈德伯格.
<a id="S0377"></a> Source: p.16 S0377
Original: Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021.
中文: Bitfit:2021年以变压器为主的口罩语言模型的简单参数高效微调.
<a id="S0378"></a> Source: p.16 S0378
Original: Yu Zhang, Ekapol Chuangsuwanich, and James Glass.
中文: 于章,相克波尔·楚昂苏瓦尼奇,和詹姆斯·格拉斯.
<a id="S0379"></a> Source: p.16 S0379
Original: Extracting deep neural network bottleneck features using low-rank matrix factorization.
中文: 利用低等基质分泌法提取出深层神经网络瓶颈特征.
<a id="S0380"></a> Source: p.16 S0380
Original: In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 185–189.
中文: 2014年,IEEE关于声学,语音和信号处理的国际会议(ICASSP),第185–189页.
<a id="S0381"></a> Source: p.16 S0381
Original: Low-rank plus diagonal adaptation for deep neural networks.
中文: 低等加对角适应深层神经网络.
<a id="S0382"></a> Source: p.16 S0382
Original: In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5005–5009.
中文: 2016年,IEEE声学,语音和信号处理国际会议(ICASP),第5005–5009页.
<a id="S0383"></a> Source: p.16 S0383
Original: Victor Zhong, Caiming Xiong, and Richard Socher.
中文: 维克多·钟,凯明·西翁和理查德·索彻.
<a id="S0384"></a> Source: p.16 S0384
Original: Seq2sql: Generating structured queries from natural language using reinforcement learning.
中文: Seq2sql:利用强化学习从自然语言中生成结构化的查询.
<a id="S0385"></a> Source: p.16 S0385
Original: URL http:// arxiv.org/abs/1709.00103. A LARGE LANGUAGE MODELS STILL NEED PARAMETER UPDATES Few-shot learning, or prompt engineering, is very advantageous when we only have a handful of training samples.
中文: URL http://arxiv.org/abs/1709.00103 (中文(简体) ). 大型语言模型仍然需要Parameter更新 很少的学习,或者说即时工程, 是非常有利的, 当我们只有几个训练样本。
<a id="S0386"></a> Source: p.16 S0386
Original: However, in practice, we can often afford to curate a few thousand or more training examples for performance-sensitive applications.
中文: 然而,在实践中,我们往往可以负担为注重业绩的应用提供几千个或更多的培训实例。
<a id="S0387"></a> Source: p.16 S0387
Original: As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.
中文: 如表8所显示,微调大大改进了模型的性能,与大小数据集的几发学习相比。
<a id="S0388"></a> Source: p.16 S0388
Original: We take the GPT-3 few-shot result on RTE from the GPT-3 paper (Brown et al., 2020).
中文: 我们从GPT-3的论文(Brown等,2020年)中取出GPT-3对RTE的几发结果.
<a id="S0389"></a> Source: p.16 S0389
Original: For MNLI-matched, we use two demonstrations per class and six in-context examples in total. 16
中文: 对于MNLI配对,我们使用每班两次演示和总共六个内文实例. 16个
<a id="S0390"></a> Source: p.17 S0390
Original: Acc./%) GPT-3 Few-Shot 40.6 69.0 GPT-3 Fine-Tuned 89.5 85.4 Table 8: Fine-tuning significantly outperforms few-shot learning on GPT-3 (Brown et al., 2020). B INFERENCE LATENCY INTRODUCED BY ADAPTER LAYERS Adapter layers are external modules added to a pre-trained model in a sequential manner, whereas our proposal, LoRA, can be seen as external modules added in a parallel manner.
中文: (acc./%) GPT-3 少 -- -- 40.6 69.0 GPT-3 精细 -- -- 89.5 85.4 表8:微调显著地超过了GPT-3上的几发学习(Brown等,2020年)。 B. 由适应者实验室吸收的资金 适配层是按顺序添加到预训模型中的外部模块,而我们的建议"LORA"则可以看作是平行添加的外部模块.
<a id="S0391"></a> Source: p.17 S0391
Original: Consequently, adapter layers must be computed in addition to the base model, inevitably introducing additional latency.
中文: 因此,除基准模型外,必须计算适配层,从而不可避免地增加延迟。
<a id="S0392"></a> Source: p.17 S0392
Original: While as pointed out in Ru¨ckle´ et al. (2020), the latency introduced by adapter layers can be mitigated when the model batch size and/or sequence length is large enough to full utilize the hardware parallelism.
中文: 虽然如Ru'cle'等(2020年)所指出,但当模型批量大小和/或序列长度足以充分利用硬件平行性时,适配层引入的耐久性可以被减轻.
<a id="S0393"></a> Source: p.17 S0393
Original: We confirm their observation with a similar latency study on GPT-2 medium and point out that there are scenarios, notably online inference where the batch size is small, where the added latency can be significant.
中文: 我们通过对GPT-2介质的类似延缓性研究确认了他们的观察,并指出,有些情况,特别是网上推断,批量规模小,增加的延缓性可能很大。
<a id="S0394"></a> Source: p.17 S0394
Original: We measure the latency of a single forward pass on an NVIDIA Quadro RTX8000 by averaging over 100 trials.
中文: 我们通过平均100多起审判来衡量NVIDIA Quadro RTX8000号卫星的一次前行间隔。
<a id="S0395"></a> Source: p.17 S0395
Original: We vary the input batch size, sequence length, and the adapter bottleneck dimension r.
中文: 我们改变输入批量大小,序列长度,以及适配器瓶颈维度r.
<a id="S0396"></a> Source: p.17 S0396
Original: We test two adapter designs: the original one by Houlsby et al. (2019), which we call AdapterH, and a recent, more efficient variant by Lin et al. (2020), which we call AdapterL.
中文: 我们测试了两种适配器设计:由Houlsby等人(2019年)出品,我们称之为适配器H;由Lin等人(2020年)出品,我们称之为适配器L.
<a id="S0397"></a> Source: p.17 S0397
Original: See Section 5.1 for more details on the designs.
中文: 有关设计的详细情况见第5.1节。
<a id="S0398"></a> Source: p.17 S0398
Original: We plot the slow-down in percentage compared to the no-adapter baseline in Figure 5. 35 30 25 20 15 10 5 0 H r retpadA 0 01 001 052 Seq Len = 128 Seq Len = 256 Seq Len = 512 1 2 4 8 16 32 Batch Size L r retpadA 0 01 001 052 1 2 4 8 16 32 1 2 4 8 16 32 Batch Size Batch Size Figure 5: Percentage slow-down of inference latency compared to the no-adapter (r = 0) baseline.
中文: 与图5.35 30 20 15 10 5 H r retpadA 0 01 001 052 Seq Len = 128 Seq Len = 512 1 2 4 8 16 32 批量大小 L r retpadA 0 01 001 052 1 4 4 16 批量大小 图5: 与不适应分量(r = 0)基线相比,推论纬度慢化的百分比。
<a id="S0399"></a> Source: p.17 S0399
Original: The top row shows the result for AdapterH and the bottom row AdapterL.
中文: 上行显示AdapterH和下行AdapterL的结果.
<a id="S0400"></a> Source: p.17 S0400
Original: Larger batch size and sequence length help to mitigate the latency, but the slow-down can be as high as over 30% in an online, short-sequence-length scenario.
中文: 更大规模的批量尺寸和序列长度有助于减轻延迟,但在在线短序列长的情景中,减速率可以高达30%以上.
<a id="S0401"></a> Source: p.17 S0401
Original: We tweak the colormap for better visibility. C DATASET DETAILS GLUE Benchmark is a wide-ranging collection of natural language understanding tasks.
中文: 我们调整了色彩图,以提高能见度。 C DATASET DETAILS GLUE Bridge是一大批自然语言理解任务.
<a id="S0402"></a> Source: p.17 S0402
Original: It includes MNLI (inference, Williams et al. (2018)), SST-2 (sentiment analysis, Socher et al. (2013)), MRPC (paraphrase detection, Dolan & Brockett (2005)), CoLA (linguistic acceptability, Warstadt et al. (2018)), QNLI (inference, Rajpurkar et al. (2018)), QQP8 (question-answering), RTE (inference), 8https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs 17
中文: 它包括MNLI(推断,Williams等(2018年)),SST-2(sentition analysis, Socher等(2013年)),MRPC(短语检测,Dolan和Brockett(2005年)),CoLA(语言可接受性,Warstadt等(2018年)),QNLI(推断,Rajpurkar等(2018年)),QQP8(问答),RTE(推断),8https://quoradata.quora.com/First-Quora-Dataset-Relelease-Pairs 17
<a id="S0403"></a> Source: p.18 S0403
Original: and STS-B (textual similarity, Cer et al. (2017)).
中文: 和STS-B(文本相似性,Cer等人(2017年))。
<a id="S0404"></a> Source: p.18 S0404
Original: The broad coverage makes GLUE benchmark a standard metric to evaluate NLU models such as RoBERTa and DeBERTa.
中文: 广范围使得GLUE基准成为评价RoBERTa和DeBERTa等NLU模型的标准度量.
<a id="S0405"></a> Source: p.18 S0405
Original: The individual datasets are released under different permissive licenses.
中文: 个人数据集是根据不同的许可发放的。
<a id="S0406"></a> Source: p.18 S0406
Original: WikiSQL is introduced in Zhong et al. (2017) and contains 56, 355/8, 421 training/validation examples.
中文: 维基语录链接:名人名言 维基语录链接:名人名言 维基语录链接:名人名言 维基语录链接:名人名言 - 文学作品 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语 - 谚语
<a id="S0407"></a> Source: p.18 S0407
Original: The task is to generate SQL queries from natural language questions and table schemata.
中文: 任务是从自然语言问题和表 schemata 生成 SQL 查询.
<a id="S0408"></a> Source: p.18 S0408
Original: We encode context as x = {table schema, query} and target as y = {SQL}.
中文: 我们将上下文编码为x={表图,查询}和目标为y={SQL}。
<a id="S0409"></a> Source: p.18 S0409
Original: The dataset is release under the BSD 3-Clause License.
中文: 数据集在 BSD 3- Clause 许可证下发布.
<a id="S0410"></a> Source: p.18 S0410
Original: SAMSum is introduced in Gliwa et al. (2019) and contains 14, 732/819 training/test examples.
中文: SAMSum在Gliwa等人(2019年)中被引入,包含14,732/819个培训/测试实例.
<a id="S0411"></a> Source: p.18 S0411
Original: It consists of staged chat conversations between two people and corresponding abstractive summaries written by linguists.
中文: 它由两个人之间的分阶段聊天对话以及由语言学家所写的相应抽象摘要组成.
<a id="S0412"></a> Source: p.18 S0412
Original: We encode context as ”\n” concatenated utterances followed by a ”\n\n”, and target as y = {summary}.
中文: 我们把上下文编码为“\n” ,然后是“\nn”, 目标为y={摘要}。
<a id="S0413"></a> Source: p.18 S0413
Original: The dataset is released under the non-commercial licence: Creative Commons BY-NC-ND 4.0.
中文: 该数据集以非商业许可证发布: Creative Communitys BY-NC-ND 4.0.
<a id="S0414"></a> Source: p.18 S0414
Original: E2E NLG Challenge was first introduced in Novikova et al. (2017) as a dataset for training end-toend, data-driven natural language generation systems and is commonly used for data-to-text evaluation.
中文: E2E NLG Challenge最初被诺维科娃等(2017年)引入,作为培训端到端,由数据驱动的自然语言生成系统的数据集,并被普遍用于数据到文本的评价.
<a id="S0415"></a> Source: p.18 S0415
Original: The E2E dataset consists of roughly 42, 000 training, 4, 600 validation, and 4, 600 test examples from the restaurant domain.
中文: E2E数据集包括大约42 000个培训、4 600个验证和4 600个来自餐厅域的测试实例。
<a id="S0416"></a> Source: p.18 S0416
Original: Each source table used as input can have multiple references.
中文: 用作输入的每个源表都可以有多个参考.
<a id="S0417"></a> Source: p.18 S0417
Original: Each sample input (x, y) consists of a sequence of slot-value pairs, along with a corresponding natural language reference text.
中文: 每个样本输入(x,y)由槽值对的序列组成,并附有相应的自然语言参考文本.
<a id="S0418"></a> Source: p.18 S0418
Original: The dataset is released under Creative Commons BY-NC-SA 4.0.
中文: 该数据集在Creative Communitys BY-NC-SA 4.0下发布.
<a id="S0419"></a> Source: p.18 S0419
Original: DART is an open-domain data-to-text dataset described in Nan et al. (2020).
中文: DART是一款由Nan等人(2020年)所描述的开放域数据到文本数据集.
<a id="S0420"></a> Source: p.18 S0420
Original: DART inputs are structured as sequences of ENTITY — RELATION — ENTITY triples.
中文: DART 投入结构为ETITY的序列-关系-ETITY 三相.
<a id="S0421"></a> Source: p.18 S0421
Original: With 82K examples in total, DART is a significantly larger and more complex data-to-text task compared to E2E.
中文: 与E2E相比,DART总共有82K个实例,是一个大得多更复杂的数据对文本任务.
<a id="S0422"></a> Source: p.18 S0422
Original: The dataset is released under the MIT license.
中文: 数据集以麻省理工学院许可发布.
<a id="S0423"></a> Source: p.18 S0423
Original: WebNLG is another commonly used dataset for data-to-text evaluation (Gardent et al., 2017).
中文: WebNLG是另一个常用的用于数据到文本评价的数据集(Gardent等,2017年).
<a id="S0424"></a> Source: p.18 S0424
Original: With 22K examples in total WebNLG comprises 14 distinct categories, nine of which are seen during training.
中文: 在WebNLG中共有22K个实例,包括14个不同的类别,其中9个是在培训期间看到的。
<a id="S0425"></a> Source: p.18 S0425
Original: Since five of the 14 total categories are not seen during training, but are represented in the test set, evaluation is typically broken out by “seen” categories (S), “unseen” categories (U) and “all” (A).
中文: 由于14个总类别中有5个在培训期间没有出现,但在测试组中有代表,评价通常按“见”类别(S)、“未见”类别(U)和“所有”类别(A)分类。
<a id="S0426"></a> Source: p.18 S0426
Original: Each input example is represented by a sequence of SUBJECT — PROPERTY — OBJECT triples.
中文: 每个输入例都由SUBJECT-ProPERTY-OBJECT三相的序列来代表.
<a id="S0427"></a> Source: p.18 S0427
Original: The dataset is released under Creative Commons BY-NC-SA 4.0. D HYPERPARAMETERS USED IN EXPERIMENTS D.1 ROBERTA We train using AdamW with a linear learning rate decay schedule.
中文: 该数据集在Creative Communitys BY-NC-SA 4.0下发布. 警告D.1 罗伯特 我们用阿达姆W来训练 并有线性学习速率衰减计划
<a id="S0428"></a> Source: p.18 S0428
Original: We sweep learning rate, number of training epochs, and batch size for LoRA.
中文: 我们扫射学习速度 训练时代的次数 以及LORA的批量规模
<a id="S0429"></a> Source: p.18 S0429
Original: Following Liu et al. (2019), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.
中文: 继刘等(2019年)后,在适应MRPC,RTE,和STS-B时,将LORA模块初始化为我们最好的MNLI检查站,而不是通常的初始化;经过预训的模型会对所有任务保持被冻结.
<a id="S0430"></a> Source: p.18 S0430
Original: We report the median over 5 random seeds; the result for each run is taken from the best epoch.
中文: 我们报告5个随机种子的中位数;每次运行的结果取自最佳时代。
<a id="S0431"></a> Source: p.18 S0431
Original: For a fair comparison with the setup in Houlsby et al. (2019) and Pfeiffer et al. (2021), we restrict the model sequence length to 128 and used a fixed batch size for all tasks.
中文: 为了与Houlsby等人(2019年)和Pfeiffer等人(2021年)的设置进行公平比较,我们把模型序列长度限制为128个并对所有任务使用固定批量尺寸.
<a id="S0432"></a> Source: p.18 S0432
Original: Importantly, we start with the pre-trained RoBERTa large model when adapting to MRPC, RTE, and STS-B, instead of a model already adapted to MNLI.
中文: 重要的是,我们在适应MRPC,RTE和STS-B时,开始使用预先训练的RoBERTa大模型,而不是已经适应了MNLI的模型.
<a id="S0433"></a> Source: p.18 S0433
Original: The runs with this restricted setup are marked with †.
中文: 带有此限制设置的运行标记为 Q.
<a id="S0434"></a> Source: p.18 S0434
Original: See the hyperparameters used in our runs in Table 9. D.2 DEBERTA We again train using AdamW with a linear learning rate decay schedule.
中文: 参见表9中我们运行中使用的超参数。 D.2 贝塔 我们再次使用 AdamW 进行线性学习 速率衰减表。
<a id="S0435"></a> Source: p.18 S0435
Original: Following He et al. (2021), we tune learning rate, dropout probability, warm-up steps, and batch size.
中文: 跟着He等人(2021年),我们调取学习率,退学概率,取暖步骤,分批量大小.
<a id="S0436"></a> Source: p.18 S0436
Original: We use the same model sequence length used by (He et al., 2021) to keep our comparison fair.
中文: 我们使用(He等人,2021年)所用的相同的模型序列长度来保持我们的比较公平.
<a id="S0437"></a> Source: p.18 S0437
Original: Following He et al. (2021), we initialize the LoRA modules to our best MNLI checkpoint when adapting to MRPC, RTE, and STS-B, instead of the usual initialization; the pre-trained model stays frozen for all tasks.
中文: 继He等 (2021年)后,我们在适应MRPC,RTE,和STS-B时,将LORA模块初始化为我们最好的MNLI检查站,而不是通常的初始化;经过预先训练的模型会对所有任务保持冻结.
<a id="S0438"></a> Source: p.18 S0438
Original: We report the median over 5 random seeds; the result for each run is taken from the best epoch.
中文: 我们报告5个随机种子的中位数;每次运行的结果取自最佳时代。
<a id="S0439"></a> Source: p.18 S0439
Original: See the hyperparameters used in our runs in Table 10. 18
中文: 参见表10.18中我们运行中使用的超参数。
<a id="S0440"></a> Source: p.19 S0440
Original: Method Dataset MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Optimizer AdamW Warmup Ratio 0.06 LR Schedule Linear Batch Size 16 16 16 32 32 16 32 16 # Epochs 30 60 30 80 25 25 80 40 RoBERTa base Learning Rate 5E-04 5E-04 4E-04 4E-04 4E-04 5E-04 5E-04 4E-04 LoRA LoRA Config. r = r = 8 q v LoRA α 8 Max Seq.
中文: Method Dataset MNLI SST-2 MRPC CoLA QQP STS-B 优化器 AdamW Warmup Batch 比例 0.06 LR 排程线性批量尺寸 16 16 32 16 # Epochs 30 60 30 80 25 80 40 RoBERTa 基础学习率 5E-04 5E-04 4 E-04 4 E-04 5 E-04 5 E-04 5 E-04 5 E-04 4 E-04 LoRA LoRA Config. r = 8 q v LoRA α 8 Max Seq.
<a id="S0441"></a> Source: p.19 S0441
Original: Len. 512 Batch Size 4 4 4 4 4 4 8 8 # Epochs 10 10 20 20 10 20 20 30 RoBERTa large Learning Rate 3E-04 4E-04 3E-04 2E-04 2E-04 3E-04 4E-04 2E-04 LoRA LoRA Config. r = r = 8 q v LoRA α 16 Max Seq.
中文: 取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取
<a id="S0442"></a> Source: p.19 S0442
Original: Len. 128 128 512 128 512 512 512 512 Batch Size 4 # Epochs 10 10 20 20 10 20 20 10 RoBERTa large Learning Rate 3E-04 4E-04 3E-04 2E-04 2E-04 3E-04 4E-04 2E-04 LoRA† LoRA Config. r = r = 8 q v LoRA α 16 Max Seq.
中文: 取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取
<a id="S0443"></a> Source: p.19 S0443
Original: Len. 128 Batch Size 32 RoBERTa large # Epochs 10 20 20 20 10 20 20 20 AdptP (3M)† Learning Rate 3E-05 3E-05 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 64 Max Seq.
中文: 取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取回取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取
<a id="S0444"></a> Source: p.19 S0444
Original: Len. 128 Batch Size 32 RoBERTa large # Epochs 5 20 20 20 10 20 20 20 AdptP (0.8M)† Learning Rate 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 16 Max Seq.
中文: Len. 128 批量大小 32 RoBERTA 大号 # Epochs 5 20 20 20 20 10 20 20 AdptP (0.8M)† 学习速率 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 16 Max Seq.
<a id="S0445"></a> Source: p.19 S0445
Original: Len. 128 Batch Size 32 RoBERTa large # Epochs 10 5 10 10 5 20 20 10 AdptH (6M)† Learning Rate 3E-05 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 64 Max Seq.
中文: 取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取回取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取取
<a id="S0446"></a> Source: p.19 S0446
Original: Len. 128 Batch Size 32 RoBERTa large # Epochs 10 5 10 10 5 20 20 10 AdptH (0.8M)† Learning Rate 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 8 Max Seq.
中文: Len. 128 批量尺寸 32 RoBERTA 大号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号号 学习率 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 3E-04 Bottleneck r 8 Max Seq.
<a id="S0447"></a> Source: p.19 S0447
Original: Len. 128 Table 9: The hyperparameters we used for RoBERTa on the GLUE benchmark. D.3 GPT-2 We train all of our GPT-2 models using AdamW (Loshchilov & Hutter, 2017) with a linear learning rate schedule for 5 epochs.
中文: Len. 128 表9:我们在GLUE基准上为RoBERTa使用的超参数. D.3 GPT-2 导弹 我们使用AdamW(Loshchilov & Hutter, 2017)来训练我们所有的GPT-2型号,有5个纪元的线性学习速度表.
<a id="S0448"></a> Source: p.19 S0448
Original: We use the batch size, learning rate, and beam search beam size described in Li & Liang (2021).
中文: 我们用李同良(2021年)所描述的批量大小,学习率,和梁搜索分量.
<a id="S0449"></a> Source: p.19 S0449
Original: Accordingly, we also tune the above hyperparameters for LoRA.
中文: 因此,我们还为LORA调制上述超参数。
<a id="S0450"></a> Source: p.19 S0450
Original: We report the mean over 3 random seeds; the result for each run is taken from the best epoch.
中文: 我们报告3个以上随机种子的平均值;每次播种的结果取自最佳时代。
<a id="S0451"></a> Source: p.19 S0451
Original: The hyperparameters used for LoRA in GPT-2 are listed in Table 11.
中文: GPT-2中用于LORA的超参数见表11。
<a id="S0452"></a> Source: p.19 S0452
Original: For those used for other baselines, see Li & Liang (2021). D.4 GPT-3 For all GPT-3 experiments, we train using AdamW (Loshchilov & Hutter, 2017) for 2 epochs with a batch size of 128 samples and a weight decay factor of 0.1.
中文: 用于其他基线的,见李克用(2021年). D.4 GPT-3 对于所有GPT-3实验,我们使用AdamW(Loshchilov & Hutter, 2017)对2个纪元进行训练,批量大小为128个样品,重量衰变系数为0.1.
<a id="S0453"></a> Source: p.20 S0453
Original: Method Dataset MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B Optimizer AdamW Warmup Ratio 0.1 LR Schedule Linear Batch Size 8 8 32 4 6 8 4 4 # Epochs 5 16 30 10 8 11 11 10 DeBERTa XXL Learning Rate 1E-04 6E-05 2E-04 1E-04 1E-04 1E-04 2E-04 2E-04 LoRA Weight Decay 0 0.01 0.01 0 0.01 0.01 0.01 0.1 CLS Dropout 0.15 0 0 0.1 0.1 0.2 0.2 0.2 LoRA Config. r = r = 8 q v LoRA α 8 Max Seq.
中文: Method Dataset MNLI SST-2 MRPC CoLA QQP STS-B 优化器 AdamW Warmup 比率 0.1 LR 排程线性批量尺寸 8 8 32 4 8 4 # Epochs 5 16 30 8 11 10 DeBERTa XXL 学习率 1E-04 6E-05 2E-04 1E-04 2E-04 LoRA 减重 00.01 0.01 0.01 0.01 0.1 CLS 0.1 退学率 0.1 0.2 0.2 0.2 0ra config. r = 8 q v LoRA α 8 Max Seq.
<a id="S0454"></a> Source: p.20 S0454
Original: Len. 256 128 128 64 512 320 320 128 Table 10: The hyperparameters for DeBERTa XXL on tasks included in the GLUE benchmark.
中文: Len. 256 128 64 512 320 128 表10:关于GLUE基准所列任务的DeBERTa XXL超参数。
<a id="S0455"></a> Source: p.20 S0455
Original: Dataset E2E WebNLG DART Training Optimizer AdamW Weight Decay 0.01 0.01 0.0 Dropout Prob 0.1 0.1 0.0 Batch Size 8 # Epoch 5 Warmup Steps 500 Learning Rate Schedule Linear Label Smooth 0.1 0.1 0.0 Learning Rate 0.0002 Adaptation r = r = 4 q v LoRA α 32 Inference Beam Size 10 Length Penalty 0.9 0.8 0.8 no repeat ngram size 4 Table 11: The hyperparameters for GPT-2 LoRA on E2E, WebNLG and DART.
中文: 数据集 E2E WebNLG DART 训练优化 AdamW W Wight Derfect 0.01 0.0 Dropout Prob 0.1 0.0 批量大小 8 # Epoch 5 Warmup Steps 500 学习速率 线性标签平滑 0.1 0.0 学习速率 0.0002 适应r = r = 4 q v LoRA α 32 Beam 10 长刑 0.9 0.8 无重复 ngm 尺寸 4 表11:E2E上GPT-2 LoRA的超参数,WebNLG和DART.
<a id="S0456"></a> Source: p.20 S0456
Original: WikiSQL (Zhong et al., 2017), 768 for MNLI (Williams et al., 2018), and 2048 for SAMSum (Gliwa et al., 2019).
中文: WikisQL (Zhong等,2017年),768代表MNLI (Williams等,2018年),2048代表SAMSum (Gliwa等,2019年).
<a id="S0457"></a> Source: p.20 S0457
Original: We tune learning rate for all method-dataset combinations.
中文: 我们调校所有方法数据集组合的学习率。
<a id="S0458"></a> Source: p.20 S0458
Original: See Section D.4 for more details on the hyperparameters used.
中文: 关于所用超参数的更详细情况,见D.4节。
<a id="S0459"></a> Source: p.20 S0459
Original: For prefix-embedding tuning, we find the optimal l and l p i to be 256 and 8, respectively, totalling 3.2M trainable parameters.
中文: 对于前缀嵌入调子,我们发现最佳的l和l p i分别是256和8,共计3.2M可列车参数.
<a id="S0460"></a> Source: p.20 S0460
Original: We use l = 8 and l = 8 for p i prefix-layer tuning with 20.2M trainable parameters to obtain the overall best performance.
中文: 我们使用l=8和l=8来进行pi前缀层调制,并配有20.2M可列车参数,以获得总体最佳性能.
<a id="S0461"></a> Source: p.20 S0461
Original: We present two parameter budgets for LoRA: 4.7M (r = r = 1 or r = 2) and 37.7M (r = r = 8 q v v q v or r = r = r = r = 2).
中文: 我们为LORA提出两个参数预算:4.7M(r = r = 1或 r = 2)和37.7M(r = r = 8 q v v q v 或 r = r = r = 2.
<a id="S0462"></a> Source: p.20 S0462
Original: We report the best validation performance from each run.
中文: 我们报告每个运行中最好的验证性能。
<a id="S0463"></a> Source: p.20 S0463
Original: The training q k v o hyperparameters used in our GPT-3 experiments are listed in Table 12. E COMBINING LORA WITH PREFIX TUNING LoRA can be naturally combined with existing prefix-based approaches.
中文: 我们的GPT-3实验中使用的训练q k v 超参数表见表12。 E 将LORA与PrefIX TUNING LORA结合,可以自然地与现有的前缀法结合.
<a id="S0464"></a> Source: p.20 S0464
Original: In this section, we evaluate two combinations of LoRA and variants of prefix-tuning on WikiSQL and MNLI.
中文: 在本节中,我们评价了LORA和WikisQL和MNLI上前缀调制的两种组合.
<a id="S0465"></a> Source: p.20 S0465
Original: LoRA+PrefixEmbed (LoRA+PE) combines LoRA with prefix-embedding tuning, where we insert l + l special tokens whose embeddings are treated as trainable parameters.
中文: LoRA+PrefixEmbed (LORA+PE) 将LORA与前缀嵌入调制结合,在其中插入了l+l的特殊符号,其嵌入被作为可被训练参数处理.
<a id="S0466"></a> Source: p.20 S0466
Original: For more on prefixp i embedding tuning, see Section 5.1.
中文: 更多关于前缀 i 嵌入调音,请参见 第 5.1 节.
<a id="S0467"></a> Source: p.20 S0467
Original: LoRA+PrefixLayer (LoRA+PL) combines LoRA with prefix-layer tuning.
中文: LoRA+PrefixLayer (LORA+PL) 将LORA与前缀层调制结合.
<a id="S0468"></a> Source: p.20 S0468
Original: We also insert l + l p i special tokens; however, instead of letting the hidden representations of these tokens evolve natu- 20
中文: 我们还插入了l + l p i 特殊标志;然而,与其让这些标志的隐藏表示演化出nat-20
<a id="S0469"></a> Source: p.21 S0469
Original: Hyperparameters Fine-Tune PreEmbed PreLayer BitFit AdapterH LoRA Optimizer AdamW Batch Size 128 # Epoch 2 Warmup Tokens 250,000 LR Schedule Linear Learning Rate 5.00E-06 5.00E-04 1.00E-04 1.6E-03 1.00E-04 2.00E-04 Table 12: The training hyperparameters used for different GPT-3 adaption methods.
中文: 超参数 Fine-Tune PreEmbed PreLayer BitFit AdapterH LoRA Batchizer AdamW Batch System 128 # Epoch 2 Warmup Tokens 250000 LR 排程线性学习率 5.00E-06 5.00E-04 1.00E-04 1.6E-03 1.00E-04 表12:不同GPT-3 适应方法所使用的训练超参数.
<a id="S0470"></a> Source: p.21 S0470
Original: We use the same hyperparameters for all datasets after tuning learning rate. rally, we replace them after every Transformer block with an input agnostic vector.
中文: 在调试学习率后,我们对所有数据集使用相同的超参数。 在每一个变形器块后,我们用输入不可知向量替换它们。
<a id="S0471"></a> Source: p.21 S0471
Original: Thus, both the embeddings and subsequent Transformer block activations are treated as trainable parameters.
中文: 因此,嵌入式和随后的变形器块活化都被视为可受训练参数.
<a id="S0472"></a> Source: p.21 S0472
Original: For more on prefix-layer tuning, see Section 5.1.
中文: 更多关于前缀层调音,请参见第5.1节.
<a id="S0473"></a> Source: p.21 S0473
Original: In Table 15, we show the evaluation results of LoRA+PE and LoRA+PL on WikiSQL and MultiNLI.
中文: 在表15中,我们在WikisQL和MultiNLI上显示了LORA+PE和LORA+PL的评价结果.
<a id="S0474"></a> Source: p.21 S0474
Original: First of all, LoRA+PE significantly outperforms both LoRA and prefix-embedding tuning on WikiSQL, which indicates that LoRA is somewhat orthogonal to prefix-embedding tuning.
中文: 首先,LORA+PE在WikiSQL上的LORA和前缀-嵌入式调取效果都明显高于前缀-嵌入式调取效果,这表明LORA在前缀-嵌入式调取效果上有些正交.
<a id="S0475"></a> Source: p.21 S0475
Original: On MultiNLI, the combination of LoRA+PE doesn’t perform better than LoRA, possibly because LoRA on its own already achieves performance comparable to the human baseline.
中文: 在MultiNLI上,LORA+PE的组合性能并不比LORA好,可能是因为LORA自己已经实现了与人类基线相仿的性能.
<a id="S0476"></a> Source: p.21 S0476
Original: Secondly, we notice that LoRA+PL performs slightly worse than LoRA even with more trainable parameters.
中文: 第二,我们注意到,LORA+PL的性能比LORA略差,甚至更有可训练参数。
<a id="S0477"></a> Source: p.21 S0477
Original: We attribute this to the fact that prefix-layer tuning is very sensitive to the choice of learning rate and thus makes the optimization of LoRA weights more difficult in LoRA+PL. F ADDITIONAL EMPIRICAL EXPERIMENTS F.1 ADDITIONAL EXPERIMENTS ON GPT-2 We also repeat our experiment on DART (Nan et al., 2020) and WebNLG (Gardent et al., 2017) following the setup of Li & Liang (2021).
中文: 我们将此归因于前缀层调子对学习率的选择非常敏感,因此在LORA+PL中,LORA分量的优化难度更大. F.1 关于GPT-2的补充警告 我们还在李同良(2021年)成立后重复了DART(Nan等,2020年)和WebNLG(Gardent等,2017年)的实验.
<a id="S0478"></a> Source: p.21 S0478
Original: Similar to our result on E2E NLG Challenge, reported in Section 5, LoRA performs better than or at least on-par with prefix-based approaches given the same number of trainable parameters.
中文: 与我们关于E2E NLG Challenge的结果相似,该结果在第5节中报告,由于可训练参数的数量相同,LORA的性能优于或至少是采用基于前缀的方法。
<a id="S0479"></a> Source: p.21 S0479
Original: Method # Trainable DART Parameters BLEU↑ MET↑ TER↓ GPT-2 Medium Fine-Tune 354M 46.2 0.39 0.46 AdapterL 0.37M 42.4 0.36 0.48 AdapterL 11M 45.2 0.38 0.46 FTTop2 24M 41.0 0.34 0.56 PrefLayer 0.35M 46.4 0.38 0.46 LoRA 0.35M 47.1 0.39 0.46 ±.2 GPT-2 Large Fine-Tune 774M 47.0 0.39 0.46 AdapterL 0.88M 45.7 0.38 0.46 ±.1 AdapterL 23M 47.1 0.39 0.45 ±.1 PrefLayer 0.77M 46.7 0.38 0.45 LoRA 0.77M 47.5 0.39 0.45 ±.1 Table 13: GPT-2 with different adaptation methods on DART.
中文: 方法#可训练DART参数 BLEU↑ MET↑ TER↓ GPT-2 中细-Tune 354M 46.2 0.39 0.36 适配器L 037M 42.4 0.36 0.38 适配器L 11M 45.2 0.38 0.36 FTTop2 24M 41.0 0.34 0.56 PrefLayer 03.5M 46.4 0.38 0.36 LoRA 03.5M 47.1 GPT-2 大细-Tune 774M 47.0 0 0.39 0.46 适配器L 0.88M 45.7 0.38 0.46 ±1 适配器L 23M 47.1 0.39 0.45 ±1 PrefLayer 0.77M 46.7 0.38 04.5 0.45 LoRA 0.77M 47.5 0.35 0.45 ±1 表13:DART上具有不同适应方法的GPT-2.
<a id="S0480"></a> Source: p.21 S0480
Original: The variances of MET and TER are less than 0.01 for all adaption approaches. 21
中文: 对所有适应方法而言,MET和TER的差异还不到0.01。 21国
<a id="S0481"></a> Source: p.22 S0481
Original: Method WebNLG BLEU↑ MET↑ TER↓ U S A U S A U S A GPT-2 Medium Fine-Tune (354M) 27.7 64.2 46.5 .30 .45 .38 .76 .33 .53 AdapterL (0.37M) 45.1 54.5 50.2 .36 .39 .38 .46 .40 .43 AdapterL (11M) 48.3 60.4 54.9 .38 .43 .41 .45 .35 .39 FTTop2 (24M) 18.9 53.6 36.0 .23 .38 .31 .99 .49 .72 Prefix (0.35M) 45.6 62.9 55.1 .38 .44 .41 .49 .35 .40 LoRA (0.35M) 46.7 62.1 55.3 .38 .44 .41 .46 .33 .39 ±.4 ±.2 ±.2 GPT-2 Large Fine-Tune (774M) 43.1 65.3 55.5 .38 .46 .42 .53 .33 .42 AdapterL (0.88M) 49.8 61.1 56.0 .38 .43 .41 .44 .35 .39 ±.0 ±.0 ±.0 AdapterL (23M) 49.2 64.7 57.7 .39 .46 .43 .46 .33 .39 ±.1 ±.2 ±.1 Prefix (0.77M) 47.7 63.4 56.3 .39 .45 .42 .48 .34 .40 LoRA (0.77M) 48.4 64.0 57.0 .39 .45 .42 .45 .32 .38 ±.3 ±.3 ±.1 Table 14: GPT-2 with different adaptation methods on WebNLG.
中文: Weather WebNLG BLEUQQ METQ TERX TERX U S A U S A (354M) 27.7 64.2 46.5.30.45.38.76.33.53 适配器L (0.37M) 45.1 54.5 50.2.36.39.38.43 适配器L (11M) 48.3 60.4 54.9.38.43.45.39 FTTOP2 (24M) 18.9 53.6 36.0.23.38.31.99.49.72 适配器 (0.35M) 45.6 62.9 55.1.38.38.38.49.39.39.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.39.40.40.40.39.40.40.40.40.40.39.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40.40. 表14:WebNLG上采用不同适应方法的GPT-2.
<a id="S0482"></a> Source: p.22 S0482
Original: The variances of MET and TER are less than 0.01 for all the experiments we ran. “U” indicates unseen categories, “S” indicates seen categories, and “A” indicates all categories in the test set of WebNLG. F.2 ADDITIONAL EXPERIMENTS ON GPT-3 We present additional runs on GPT-3 with different adaptation methods in Table 15.
中文: 在我们进行的所有实验中, MET 和 TER 的相差小于 0.01 。 “U”表示看不见的类别,“S”表示可见的类别,“A”表示WebNLG测试集中的所有类别。 F.2 关于GPT-3的补充警告 我们对GPT-3进行额外运行,采用不同的适应方法,见表15。
<a id="S0483"></a> Source: p.22 S0483
Original: The focus is on identifying the trade-off between performance and the number of trainable parameters. F.3 LOW-DATA REGIME To evaluate the performance of different adaptation approaches in the low-data regime. we randomly sample 100, 1k and 10k training examples from the full training set of MNLI to form the low-data MNLI-n tasks.
中文: 重点是确定性能与可训练参数数量之间的取舍。 F.3 低度-低度区域 评估低数据制度中不同适应方针的绩效。 我们随机抽取100,1k和10k训练实例 从MNLI的全部训练集中形成低数据MNLI-n任务.
<a id="S0484"></a> Source: p.22 S0484
Original: In Table 16, we show the performance of different adaptation approaches on MNLIn.
中文: 在表16中,我们显示了关于MNLIn的不同适应方针的绩效。
<a id="S0485"></a> Source: p.22 S0485
Original: To our surprise, PrefixEmbed and PrefixLayer performs very poorly on MNLI-100 dataset, with PrefixEmbed performing only slightly better than random chance (37.6% vs. 33.3%).
中文: 令我们惊讶的是,PrefixEmbed和PrefixLayer在MNLI-100数据集上的性能非常差,PrefixEmbed的性能仅略优于随机机会(37.6%对33.3%).
<a id="S0486"></a> Source: p.22 S0486
Original: PrefixLayer performs better than PrefixEmbed but is still significantly worse than Fine-Tune or LoRA on MNLI- 100.
中文: 前缀Layer的性能优于前缀Embed,但仍然比Fine-Tune或LORA在MNLI-100上的更糟糕.
<a id="S0487"></a> Source: p.22 S0487
Original: The gap between prefix-based approaches and LoRA/Fine-tuning becomes smaller as we increase the number of training examples, which might suggest that prefix-based approaches are not suitable for low-data tasks in GPT-3.
中文: 随着培训实例数量的增加,基于前缀的方法和LORA/Fine-distance之间的差距会缩小,这可能表明基于前缀的方法不适合GPT-3中的低数据任务.
<a id="S0488"></a> Source: p.22 S0488
Original: LoRA achieves better performance than fine-tuning on both MNLI-100 and MNLI-Full, and comparable results on MNLI-1k and MNLI-10K considering the (±0.3) variance due to random seeds.
中文: LORA的性能优于MNLI-100和MNLI-Full的微调,考虑到随机种子造成的(±0.3)差异,MNLI-1k和MNLI-10K的可比较结果.
<a id="S0489"></a> Source: p.22 S0489
Original: The training hyperparameters of different adaptation approaches on MNLI-n are reported in Table 17.
中文: 关于MNLI-n的不同适应方法的培训超参数见表17。
<a id="S0490"></a> Source: p.22 S0490
Original: We use a smaller learning rate for PrefixLayer on the MNLI-100 set, as the training loss does not decrease with a larger learning rate. G MEASURING SIMILARITY BETWEEN SUBSPACES In this paper we use the measure φ(A, B, i, j) = ψ(U i , U j ) = (cid:107)U A i(cid:62)UB(cid:107)2 F to measure the subspace A B min{i,j} similarity between two column orthonormal matrices U i ∈ Rd×i and U j ∈ Rd×j, obtained by A B taking columns of the left singular matrices of A and B.
中文: 我们使用MNLI-100系列的PrefixLayer学习率较低,因为培训损失不会随着较大的学习率而减少。 G. 衡量各附属物之间的相似性 在本文中,我们用"(A, B, i, j)"="(U i, U j)"="(cid:107)"U A (cid:62)UB (cid:107)2 F来测量A B min{i, j}两个列正态矩阵U i → Rd×i和U j → Rd×j之间的相似性,由A B取取取A和B左单态矩阵的列.
<a id="S0491"></a> Source: p.22 S0491
Original: We point out that this similarity is simply a reverse of the standard Projection Metric that measures distance between subspaces Ham & Lee (2008). 22
中文: 我们指出,这种相似性只是标准Projection Metric的倒行逆施,它测量了ham & Lee子空间之间的距离(2008年). 22个
<a id="S0492"></a> Source: p.23 S0492
Original: Method Hyperparameters # Trainable Parameters WikiSQL MNLI-m Fine-Tune - 175B 73.8 89.5 l = 32, l = 8 0.4 M 55.9 84.9 p i l = 64, l = 8 0.9 M 58.7 88.1 p i PrefixEmbed l = 128, l = 8 1.7 M 60.6 88.0 p i l = 256, l = 8 3.2 M 63.1 88.6 p i l = 512, l = 8 6.4 M 55.9 85.8 p i l = 2, l = 2 5.1 M 68.5 89.2 p i l = 8, l = 0 10.1 M 69.8 88.2 p i PrefixLayer l = 8, l = 8 20.2 M 70.1 89.5 p i l = 32, l = 4 44.1 M 66.4 89.6 p i l = 64, l = 0 76.1 M 64.9 87.9 p i r = 1 7.1 M 71.9 89.8 r = 4 21.2 M 73.2 91.0 AdapterH r = 8 40.1 M 73.2 91.5 r = 16 77.9 M 73.2 91.5 r = 64 304.4 M 72.6 91.5 r = 2 4.7 M 73.4 91.7 v r = r = 1 4.7 M 73.4 91.3 q v LoRA r = r = 2 9.4 M 73.3 91.4 q v r = r = r = r = 1 9.4 M 74.1 91.2 q k v o r = r = 4 18.8 M 73.7 91.3 q v r = r = r = r = 2 18.8 M 73.7 91.7 q k v o r = r = 8 37.7 M 73.8 91.6 q v r = r = r = r = 4 37.7 M 74.0 91.7 q k v o r = r = 64 301.9 M 73.6 91.4 q v r = r = r = r = 64 603.8 M 73.9 91.4 q k v o r = r = 8, l = 8, l = 4 37.8 M 75.0 91.4 q v p i LoRA+PE r = r = 32, l = 8, l = 4 151.1 M 75.9 91.1 q v p i r = r = 64, l = 8, l = 4 302.1 M 76.2 91.3 q v p i LoRA+PL r = r = 8, l = 8, l = 4 52.8 M 72.9 90.2 q v p i Table 15: Hyperparameter analysis of different adaptation approaches on WikiSQL and MNLI.
中文: 方法超参数#可训练参数WikiSQL MNL-m Fine-Tune - 175B 73.8 89.5 I = 32,l = 8 0.4 M 55.9 84.9 p I = 64,l = 8 0.9 M 58.7 88.1 p i Prefix Embed I = 128,l = 8 1.7 M 60.6 88.0 p i = 256,l = 83.2 M 63.1 88.6 p I = 512,l = 8 6.4 M 55.9 o r r 4 r 910 br br br br br br br r r 73.2 r = 7 7 M7 r 7 r r 7304 r r r = 7 M7 r 13 r r r = = 13 r = 7 r r 13 = r 13 = = r = 13 = = r = 1 = r = 64 603.8 M 73.9 9 9 1 q k v o r = r = 8,l = 8,l = 4 37.8 M 75.0 91.4 q = p i LoRA+PE r = r = 32,l = 8,l = 4 151.1 M 75.9 91.1 q = r = 64,l = 8,l = 4 302.1 M 76.2 91.3 q = p i LoRA+PL r = r = 8,l = 8,l = 4 52.8 M 72.9 90.2 q = p i 表15: 对WikiSQL和 MNLI不同适应方法的超参数分析.
<a id="S0493"></a> Source: p.23 S0493
Original: Both prefix-embedding tuning (PrefixEmbed) and prefix-layer tuning (PrefixLayer) perform worse as we increase the number of trainable parameters, while LoRA’s performance stabilizes.
中文: 前缀嵌入调子(PrefixEmbed)和前缀层调子(PrefixLayer)都表现得更糟,因为我们增加了可受训练参数的数量,而LORA的性能稳定了.
<a id="S0494"></a> Source: p.23 S0494
Original: Performance is measured in validation accuracy.
中文: 性能以验证精度来测量.
<a id="S0495"></a> Source: p.23 S0495
Original: Method MNLI(m)-100 MNLI(m)-1k MNLI(m)-10k MNLI(m)-392K GPT-3 (Fine-Tune) 60.2 85.8 88.9 89.5 GPT-3 (PrefixEmbed) 37.6 75.2 79.5 88.6 GPT-3 (PrefixLayer) 48.3 82.5 85.9 89.6 GPT-3 (LoRA) 63.8 85.6 89.2 91.7 Table 16: Validation accuracy of different methods on subsets of MNLI using GPT-3 175B.
中文: MNLI(m)-100 MNLI(m)-1k MNLI(m)-10k MNLI(m)-392K GPT-3(Fine-Tune) 60.2 85.8 88.9 89.5 GPT-3(前置Embed) 37.6 75.2 79.5 88.6 GPT-3(前置Layer) 48.3 8.2.5 85.9 89.6 GPT-3(LORA) 63.8 85.6 89.2 91.7 表16:使用GPT-3175B对MNLI子集采用不同方法的验证精度.
<a id="S0496"></a> Source: p.23 S0496
Original: MNLIn describes a subset with n training examples.
中文: MNLIn 描述一个包含 n 培训示例的子集.
<a id="S0497"></a> Source: p.23 S0497
Original: We evaluate with the full validation set.
中文: 我们用全套鉴定来评估
<a id="S0498"></a> Source: p.23 S0498
Original: LoRA performs exhibits favorable sample-efficiency compared to other methods, including fine-tuning.
中文: 与包括微调在内的其他方法相比,LORA具有有利的样本效率。
<a id="S0499"></a> Source: p.23 S0499
Original: To be concrete, let the singular values of U i(cid:62)U j to be σ , σ , · · · , σ where p = min{i, j}.
中文: 具体来说,让U i(cid:62)U j的单值为:""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
<a id="S0500"></a> Source: p.23 S0500
Original: We A B 1 2 p know that the Projection Metric Ham & Lee (2008) is defined as: (cid:118) (cid:117) p d(U i , U j ) = (cid:117) (cid:116)p − (cid:88) σ2 ∈ [0, √ p] A B i i=1 23
中文: 我们A B 1 2 p 知道Projection Metric Ham & Lee(2008)的定义是:(cid:118) (cid:117)p d(U i,U j) = (cid:117) (cid:116)p-(cid:88)\\ [0] A B i = 1 23
<a id="S0501"></a> Source: p.24 S0501
Original: Hyperparameters Adaptation MNLI-100 MNLI-1k MNLI-10K MNLI-392K Optimizer - AdamW Warmup Tokens - 250,000 LR Schedule - Linear Batch Size - 20 20 100 128 # Epoch - 40 40 4 2 FineTune 5.00E-6 PrefixEmbed 2.00E-04 2.00E-04 4.00E-04 5.00E-04 Learning Rate PrefixLayer 5.00E-05 5.00E-05 5.00E-05 1.00E-04 LoRA 2.00E-4 PrefixEmbed l 16 32 64 256 p Adaptation- PrefixEmbed l 8 i Specific PrefixTune l = l = 8 p i LoRA r = r = 8 q v Table 17: The hyperparameters used for different GPT-3 adaptation methods on MNLI(m)-n. where our similarity is defined as: (cid:80)p σ2 1 (cid:16) (cid:17) φ(A, B, i, j) = ψ(U i , U j ) = i=1 i = 1 − d(U i , U j )2 A B p p A B This similarity satisfies that if U i and U j share the same column span, then φ(A, B, i, j) = 1.
中文: 超参数 适应 MNLI-100 MNLI-1k MNLI-10K MNLI-392K 优化器 - 亚当W Warmup Tokens - 250000 LR 时刻表 - 线性批量尺寸 - 20 100 128 # Epoch - 40 4 2 FineTune 5.00E-6 PrefixEmbed 2.00E-04 4.00E-04 5.00E-04 学习率 PrefixLayer 5.00E-05 5.00E-05 E-05 1.00E-04 LoRA 2.00E-4 PrefixEmbed l 16 32 64 p 256 p 适应-Prefixed Eune l 8 i i i i i i LoRA r = 8 p q v 表 17:在 MNLI(m (m)上使用不同的GPT-3 适应方法使用的超参数,其相似性被定义为: (cid:80)p/3/2 1 (c:16)(cid:17 i) i i 这种相似性满足于如果U i和U j共用同一列跨度,那么"(A,B,i,j)"= 1.
<a id="S0502"></a> Source: p.24 S0502
Original: If A B they are completely orthogonal, then φ(A, B, i, j) = 0.
中文: 如果A B是完全正交的,则(A、B、i、j)=0。
<a id="S0503"></a> Source: p.24 S0503
Original: Otherwise, φ(A, B, i, j) ∈ (0, 1). H ADDITIONAL EXPERIMENTS ON LOW-RANK MATRICES We present additional results from our investigation into the low-rank update matrices. H.1 CORRELATION BETWEEN LORA MODULES See Figure 6 and Figure 7 for how the results presented in Figure 3 and Figure 4 generalize to other layers. H.2 EFFECT OF r ON GPT-2 We repeat our experiment on the effect of r (Section 7.2) in GPT-2.
中文: 否则, (A、B、一、j)(0、1)。 关于低程材料的补充警告 我们提出我们对低级更新矩阵调查的其他结果。 H.1 洛拉山地之间的腐败 见图6和图7,图3和图4所示结果如何概括到其他层面。 H.2 r对GPT-2的影响 我们重复我们在GPT-2中对r(第7.2节)的影响的实验。
<a id="S0504"></a> Source: p.24 S0504
Original: Using the E2E NLG Challenge dataset as an example, we report the validation loss and test metrics achieved by different choices of r after training for 26,000 steps.
中文: 以E2E NLG挑战数据集为例,我们报告在培训26 000个步骤后,不同选择r实现的验证损失和测试度量度。
<a id="S0505"></a> Source: p.24 S0505
Original: The optimal rank for GPT-2 Medium is between 4 and 16 depending on the metric used, which is similar to that for GPT-3 175B.
中文: GPT-2 Medium的最佳分级根据所使用的度量在4到16之间,这与GPT-3175B相似.
<a id="S0506"></a> Source: p.24 S0506
Original: Note that the relationship between model size and the optimal rank for adaptation is still an open question. H.3 CORRELATION BETWEEN W AND ∆W See Figure 8 for the normalized subspace similarity between W and ∆W with varying r.
中文: 请注意,模型大小与适应的最佳等级之间的关系仍然是一个未决问题。 H.3. W和QQW之间的校正,见图8,W和QQW之间正常的子空间相近性,r.
<a id="S0507"></a> Source: p.24 S0507
Original: Note again that ∆W does not contain the top singular directions of W , since the similarity between the top 4 directions in ∆W and the top-10% of those in W barely exceeds 0.2.
中文: 请注意,QQW并不包含W的顶端单数方向,因为QQW中前4个方向与W中前-10%方向的相似性勉强超过0.2.
<a id="S0508"></a> Source: p.24 S0508
Original: This gives evidence that ∆W contains those “task-specific” directions that are otherwise not emphasized in W .
中文: 这证明,QQW含有W不强调的那些“特定任务”指示。
<a id="S0509"></a> Source: p.24 S0509
Original: An interesting next question to answer, is how “strong” do we need to amplify those task-specific directions, in order for the model adaptation to work well? 24
中文: 下一个令人感兴趣的问题是,我们需要如何“强有力”地扩大这些具体任务的方向,以便模型的适应工作能够顺利进行? 24个
<a id="S0510"></a> Source: p.25 S0510
Original: 1.0 0.8 0.6 0.4 0.2 0.0 1 reyaL i 1 2 3 4 5 6 7 8 Wq Wv Wq Wv 23 reyaL 46 reyaL i i 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 6 21 81 32 92 53 04 64 25 85 j 69 reyaL i 1 2 3 4 5 6 7 8 1 6 21 81 32 92 53 04 64 25 85 (Ar = 8, Ar = 64, i, j) 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 j j j Figure 6: Normalized subspace similarity between the column vectors of A and A for both r=8 r=64 ∆W and ∆W from the 1st, 32nd, 64th, and 96th layers in a 96-layer Transformer. q v H.4 AMPLIFICATION FACTOR One can naturally consider a feature amplification factor as the ratio (cid:107)∆W (cid:107)F , where U and V (cid:107)U(cid:62)W V (cid:62)(cid:107)F are the left- and right-singular matrices of the SVD decomposition of ∆W . (Recall U U (cid:62)W V (cid:62)V gives the “projection” of W onto the subspace spanned by ∆W .) Intuitively, when ∆W mostly contains task-specific directions, this quantity measures how much of them are amplified by ∆W .
中文: 1.0 0.8 0.4 0.2 0.0 1 reyaL i 1 2 3 4 5 6 7 8 Wq Wq Wv 23 reyaL 46 reyaL i 1 2 3 5 6 7 8 1 2 3 5 6 8 21 81 32 92 53 04 64 25 j 69 reyaL i 1 3 5 7 8 (Ar = 8 21 81 92 53 04 64 25 85 (Ar = 8 4 4 5 6 7 1 2 3 3 5 7 7 j j j 图6: 从 第1, 第32, 第64, 第96层变压器中R = 8 r= 64 → 第64 W 和 第96层的A 列向量的正次空间相似性. q诉H.4 拆解装置 人们自然可以将特征放大系数视为比分(cid:107)QQW(cid:107)F,其中U和V(cid:107)U(cid:62)WV(cid:62)(cid:107)F为SVD分解分解的左相和右相相相. (回忆U U (cid:62)W V (cid:62)V将W的"预测"放入由QQW所跨越的子空间. ) 直觉上,当QQW大多包含任务特定方向时,这个数量量度它们中有多少被QQW放大.
<a id="S0511"></a> Source: p.25 S0511
Original: As shown in Section 7.3, for r = 4, this amplification factor is as large as 20.
中文: 如第7.3节所示,r = 4这一放大系数高达20。
<a id="S0512"></a> Source: p.25 S0512
Original: In other words, there are (generally speaking) four feature directions in each layer (out of the entire feature space from the pre-trained model W ), that need to be amplified by a very large factor 20, in order to achieve our reported accuracy for the downstream specific task.
中文: 换句话说,每层(一般)有四个地物取向(出自预先训练过的模型W的整个地物取向),需要用一个非常大的系数20来放大,以达到我们所报告的下游具体任务的准确性.
<a id="S0513"></a> Source: p.25 S0513
Original: And, one should expect a very different set of feature directions to be amplified for each different downstream task.
中文: 并且,我们应当期望为每一个不同的下游任务,扩大一套非常不同的特征方向。
<a id="S0514"></a> Source: p.25 S0514
Original: One may notice, however, for r = 64, this amplification factor is only around 2, meaning that most directions learned in ∆W with r = 64 are not being amplified by much.
中文: 然而,人们可能会注意到,对于r = 64,这个放大系数只有2左右,也就是说,QQW 中学到的大多数方向(r = 64)并没有被很多放大.
<a id="S0515"></a> Source: p.25 S0515
Original: This should not be surprising, and in fact gives evidence (once again) that the intrinsic rank needed to represent the “task-specific directions” (thus for model adaptation) is low.
中文: 这不应令人惊讶,事实上(再次)证明,代表“特定任务方向”所需的内在排名(即模型适应)很低。
<a id="S0516"></a> Source: p.25 S0516
Original: In contrast, those directions in the rank-4 version of ∆W (corresponding to r = 4) are amplified by a much larger factor 20. 25
中文: 相比之下,在QQW的4级版本(对应r=4)中,这些方向被更大的系数20.25所放大.
<a id="S0517"></a> Source: p.26 S0517
Original: 1 7 13 19 0.8 25 31 0.7 37 43 0.6 49 55 0.5 61 0.4 0.3 0.2 0.1 0.0 1 reyaL i Wq Wv 23 reyaL Wq Wv 1 6 11 61 12 62 13 63 14 64 15 65 16 1 7 13 19 25 31 37 43 49 55 61 j 46 reyaL i 1 6 11 61 12 62 13 63 14 64 15 65 16 j 1 6 11 61 12 62 13 63 14 64 15 65 16 j 69 reyaL 1 6 11 61 12 62 13 63 14 64 15 65 16 (Ar=64, A0r=64, i, j) j Figure 7: Normalized subspace similarity between the column vectors of A from two randomly r=64 seeded runs, for both ∆W and ∆W from the 1st, 32nd, 64th, and 96th layers in a 96-layer Transq v former.
中文: 1 7 19 0.8 25 31 0.7 37 0.6 49 0.5 61 0.3 0.2 0.1 0.0 1 reyaL i Wq Wv 23 reyaL Wv 1 6 11 61 12 62 13 14 64 15 65 65 43 43 49 55 61 j 46 reyaL i 1 11 11 61 12 62 16 12 16 6 11 12 13 63 14 64 15 65 16 (Ar=64, A0r=64, i, j) 图7:A纵向量从两个随机的r=64种子跑起的正态分空间相似度,分别为从X-W到96层 Transq v.
<a id="S0518"></a> Source: p.26 S0518
Original: Rank r val loss BLEU NIST METEOR ROUGE L CIDEr 1 1.23 68.72 8.7215 0.4565 0.7052 2.4329 2 1.21 69.17 8.7413 0.4590 0.7052 2.4639 4 1.18 70.38 8.8439 0.4689 0.7186 2.5349 8 1.17 69.57 8.7457 0.4636 0.7196 2.5196 16 1.16 69.61 8.7483 0.4629 0.7177 2.4985 32 1.16 69.33 8.7736 0.4642 0.7105 2.5255 64 1.16 69.24 8.7174 0.4651 0.7180 2.5070 128 1.16 68.73 8.6718 0.4628 0.7127 2.5030 256 1.16 68.92 8.6982 0.4629 0.7128 2.5012 512 1.16 68.78 8.6857 0.4637 0.7128 2.5025 1024 1.17 69.37 8.7495 0.4659 0.7149 2.5090 Table 18: Validation loss and test set metrics on E2E NLG Challenge achieved by LoRA with different rank r using GPT-2 Medium.
中文: BLEU NIST METEOR ROUGE L CIDER 1.23 68.72 8.7215 0.4565 0.7052 2.4329 2. 1.21 69.17 8.7413 0.4590 0.7052 2.4639 4 1.18 70.38 8.8439 04689 0.7186 2.5349 8 1.17 69.57 8.7457 0.4636 0.7196 2.5196 16.16 69.61 8.7483 04629 0.7177 2.4985 32 1.1633 8.7736 0.4642 0.75255 64 1.16248.7174 0.4651 0.7180 2.5070 128 68.738.6718 0.4628 0.7127 2.530 256 1.166892 8.6982 0.4629 0.71282.5012 5.12 1.16 68.78 8.6857 0.4637 0.7 0.7 0.7128 2.5025 1.174.69375. 04659 0.4659 0.7149 2.50909 表18:不同等级r的LORA使用GPT-2 Medium实现的E2E NLG挑战的验证损失和测试成套度量衡.
<a id="S0519"></a> Source: p.26 S0519
Original: Unlike on GPT-3 where r = 1 suffices for many tasks, here the performance peaks at r = 16 for validation loss and r = 4 for BLEU, suggesting the GPT-2 Medium has a similar intrinsic rank for adaptation compared to GPT-3 175B.
中文: 与GPT-3上的r = 1足以完成许多任务不同,这里的性能峰值为r = 16用于验证损失,r = 4用于BLEU,表明与GPT-3 175B相比,GPT-2介质在适应上的内在排名相似.
<a id="S0520"></a> Source: p.26 S0520
Original: Note that some of our hyperparameters are tuned on r = 4, which matches the parameter count of another baseline, and thus might not be optimal for other choices of r. 451 0.200 555 658 0.175 762 0.150 865 969 0.125 1072 0.100 1176 j i Wq Random (Wq, Ar = 4, i, j) (Wq, Ar = 8, i, j) (Wq, Ar = 64, i, j) (Wq, Arand, i, j) j j j Figure 8: Normalized subspace similarity between the singular directions of W and those of ∆W q q with varying r and a random baseline. ∆W amplifies directions that are important but not emphaq sized in W . ∆W with a larger r tends to pick up more directions that are already emphasized in W . 26
中文: 请注意,我们的一些超参数调整在r=4上,与另一个基线的参数数相匹配,因此可能不适于其他选择的r. 451 0.200 555 658 0.1575 7620 865 969 0.125 1072 0100 1176 j i Wq Random(Wq,Ar = 4,i,j)(Wq,Ar = 8,i,j)(Wq,Ar = 64,i,j)(Wq,Arand,i,j) j j j 图8: W和QQW q的单向的正态亚空间相似度,且基线不同。 QQW放大了W中重要但非emphaq大小的取向. 有较大r的QQW倾向于取出更多W26中已经强调的取向.