youssefrekik/qwen3-8b-tunisian-dialect-adapter

Qwen3-8B Tunisian Insurance Assistant

Two-stage fine-tuned model for Tunisian insurance domain with dialect support.

Architecture

1. Dialect Adapter (Stage 1): Trained on Tunisian dialect corpus
2. Task Adapter (Stage 2): Fine-tuned on insurance FAQ data

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "unsloth/Qwen3-8B-unsloth-bnb-4bit",
    load_in_4bit=True,
    device_map="auto"
)

# Load tokenizer with custom tokens
tokenizer = AutoTokenizer.from_pretrained("youssefrekik/qwen3-8b-tunisian-dialect-adapter")
model.resize_token_embeddings(len(tokenizer))

# Load dialect adapter
model = PeftModel.from_pretrained(model, "youssefrekik/qwen3-8b-tunisian-dialect-adapter")

# Load task adapter
model.load_adapter("youssefrekik/qwen3-8b-tunisian-insurance-task-adapter", adapter_name="task")
model.set_adapter(["default", "task"])

# Generate
prompt = "<tunisian>Chnowa el fara9 bin RC w tous risques?</tunisian>"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

• Base Model: Qwen3-8B-4bit
• Stage 1: Dialect adaptation (r=8, 2000 steps)
• Stage 2: Task fine-tuning (r=8, 200 steps)
• Languages: Tunisian Arabic (Arabizi), Modern Standard Arabic, French, English
• Domain: Tunisian insurance

Custom Tokens

Added domain-specific tokens including:

• Language markers: <tunisian>, <arabizi>, <ar>, <fr>
• Insurance terms: <assurance>, <sinistre>, <prime>, etc.
• Tunisian expressions: <khouya>, <chnowa>, <kifach>, etc.

License

Apache 2.0