Project 2 - SVD LoRA Adapter

This project was conducted as part of my role as a Research Assistant under Dr. Jie Chen at the MIT AI Watson Lab. My focus was on the efficient implementation of Low-Rank Approximation (LoRA) techniques using adapter methods.

My responsibilities included developing and optimizing algorithms to reduce computational complexity while maintaining accuracy, enabling faster data processing and improving machine learning model efficiency.

1. Original LoRA Method

The original LoRA method reduces the total number of trainable parameters by decomposing the weight matrix into low-rank matrices. This is represented as:

\[ W = A \times B \]

where A and B are low-rank matrices. This transformation allows models to adapt to new tasks with minimal modifications to the original architecture, significantly reducing the number of parameters that need fine-tuning. As a result, LoRA enables efficient transfer learning and task adaptation.

By applying LoRA adapters to models, we can reduce the number of trainable parameters to as little as 1/1000 of the original model (e.g., from 10 billion to 3 million trainable parameters) while simultaneously improving model performance.

2. Our Method: SVD-LoRA Decomposition

In our approach, we propose a novel decomposition of the weight matrix W into three components:

\[ W = P \times S \times \Pi \]

where:

• P ∈ ℝ^{d × r} (low-rank projection matrix)
• S ∈ ℝ^{r × r} (small core matrix)
• Π ∈ ℝ^{r × d} (low-rank reconstruction matrix)

Here is the architecture of our method:

3. Underlying Principles

Our idea is inspired by Kronecker matrix product and multigrid methods. The process involves:

1. Performing balanced clustering on the original weight matrix W.
2. Assigning elements of W to corresponding clusters.
3. Initializing: \[ P = \mathcal{N}(0,1), \quad \Pi = \mathcal{N}(0, 1/k) \] where k is the number of elements in each cluster, and \[ S = 0. \]

4. Training Strategy

Unlike traditional LoRA, we only allow S to be trainable while freezing P and Π. This significantly reduces the number of trainable parameters while maintaining model performance.

For example, our method further reduces the trainable parameters from 3 million to 30 thousand, making it even more efficient for transfer learning and model adaptation tasks.

Reference

Hu, Edward J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." 2021, arXiv, https://arxiv.org/abs/2106.09685.