arxiv:2506.05629

Leveraging Self-Attention for Input-Dependent Soft Prompting in LLMs

Published on Jun 5

· Submitted by

abhi1nandy2 on Jun 9

Upvote

Authors:

Ananth Muppidi ,

Abhilash Nandy ,

Sambaran Bandyopadhyay

Abstract

A new method using input-dependent soft prompting with a self-attention mechanism improves parameter-efficient fine-tuning for large language models, enhancing zero-shot domain transfer.

AI-generated summary

The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.

View arXiv page View PDF Add to collection

Community

abhi1nandy2

Paper author Paper submitter Jun 9

•

edited Jun 9

🎯 ID-SPAM (Input-Dependent Soft Prompting technique with a self-Attention Mechanism) is here! 📚
🧠 Efficiently adapt LLMs with input-aware soft prompts using self-attention
⚡ Minimal parameters, maximum adaptability — say goodbye to heavy fine-tuning!
🌍 Superior zero-shot domain transfer across diverse tasks
🚀 Accepted at ACL 2025 (Main) Conference

🔍 ID-SPAM learns to generate smarter prompts by attending to input tokens with varying importance, outperforming state-of-the-art parameter-efficient tuning methods. Compact, scalable, and ready for real-world domains!