I covered the basic idea fine: freeze the original weights, inject a low-rank decomposition as a side branch, train only that.
Start by defining LoRA at a high level, then dive into the core mechanics of low-rank decomposition and how it modifies the forward pass. Explain where it is applied in Transformer architectures and discuss practical benefits and key hyperparameters, tying them to real-world trade-offs.
Pro tip: Emphasize that LoRA enables parameter-efficient fine-tuning and allows swapping adapters for different tasks without reloading the base model—a crucial advantage for serving many tasks in production.
Define LoRA as a parameter-efficient fine-tuning method that freezes the pre-trained model weights and injects trainable low-rank matrices into each layer.
Explain that for a weight matrix W, LoRA adds a low-rank decomposition ΔW = BA, where B and A are smaller matrices, and the forward pass becomes h = Wx + BAx. Mention that only A and B are trained.
Specify that LoRA is typically applied to the query and value projection matrices in the self-attention mechanism, and sometimes to other linear layers, depending on the task.
Highlight benefits: drastically reduces trainable parameters and memory, enables efficient task switching by swapping adapters, and maintains performance close to full fine-tuning.
Discuss the rank r (controls capacity), scaling factor alpha (balances the low-rank update), and dropout on LoRA layers for regularization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.