Abduraxim/flant5-paraphraser

📝 Paraphraser Model - Inference Documentation

Overview

This script loads a fine-tuned FLAN-T5 model (based on MT5ForConditionalGeneration) to generate fluent paraphrases of input text. It uses HuggingFace's Transformers library and supports GPU acceleration.

---

🔧 Requirements

Python Packages

pip install torch transformers

Model Checkpoint

Ensure you have a fine-tuned checkpoint directory (e.g., from training with HuggingFace Trainer) available at:

/home/deploy/second_disk/projects/paraphrase_train/flan-t5-paraphrase-v2/checkpoint-7029

---

🚀 Usage

Run the Script

python paraphrase_infer.py

Example Interaction

Enter text to paraphrase (Ctrl+C to exit):
> I want to understand this model better.
→ I would like to gain a deeper understanding of this model.

---

🧠 Functionality Breakdown

1. Load Model and Tokenizer

tokenizer = MT5Tokenizer.from_pretrained(model_path)
model = MT5ForConditionalGeneration.from_pretrained(model_path)

• Loads tokenizer and model weights from the specified checkpoint directory.

2. Device Configuration

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

• Uses GPU if available. Sets model to evaluation mode.

3. Paraphrasing Function

def paraphrase(text, max_length=128, num_beams=4):

Parameters:

• text (str): Input sentence to be paraphrased.
• max_length (int): Max tokens in output (default: 128).
• num_beams (int): Beam width for decoding (default: 4).

Inside:

• Prefixes input with "paraphrase:" (T5-style prompt tuning).
• Applies top-p sampling (top_p=0.95) and slight randomness (temperature=0.8) to encourage variation.

Returns:

• A paraphrased version of the input string.

---

📁 File Structure Suggestion

paraphraser/
├── paraphrase_infer.py  # Your script
├── README.md            # This doc
└── checkpoint/          # Your trained model

--

⚙️ Customization Ideas

• Swap model to T5ForConditionalGeneration if using original T5.
• Deploy as API with FastAPI/Flask.
• Add batch processing support.

---