Intruder Dimensions
Novel high-ranking singular vectors that emerge during LoRA training but are absent in full fine-tuning, potentially interfering with pre-trained model capabilities.
Intruder dimensions are a phenomenon identified in NeurIPS 2024 research where LoRA training creates new dominant directions in the model's weight space that don't correspond to either the original pre-trained knowledge or the intended fine-tuning task. These dimensions can interfere with existing capabilities and accumulate during sequential fine-tuning, causing compounding knowledge degradation. The effect is minimal for single-task adaptation but becomes a significant concern for continual learning pipelines.
Also known as
LoRA intruder dimensions