maxpe/gpt-oss-spanish-climate-conditioned

Model Card: GPT-OSS-20B Spanish Climate Change (Civility-Conditioned)

Model Description

This is a fine-tuned version of gpt-oss-20b trained on Spanish Facebook discussions about climate change with civility conditioning. The model learns to generate realistic Spanish social media posts while allowing users to control the civility level at inference time.

• Model Type: Autoregressive Language Model (Causal LM)
• Base Model: gpt-oss-20b (20 billion parameters)
• Language: Spanish
• Training Objective: Masked posting prediction with civility conditioning
• License: [Same as base model]
• Model Size: 20B parameters

Key Features

1. Civility Control

The model can generate both civil and incivil content, controlled by a simple parameter:

• [CIVILITY: civil] → Generates respectful, constructive discourse
• [CIVILITY: incivil] → Generates content with rudeness, hate speech, or threats

2. Realistic Spanish Social Media Generation

Trained on authentic Facebook discussions about climate change in Spanish, capturing:

• Natural conversation patterns
• Topic-appropriate vocabulary
• Social media communication styles
• Diverse perspectives and argumentation patterns

3. Context-Aware Continuation

Uses masked posting prediction to generate responses that:

• Fit naturally within ongoing discussions
• Maintain topical coherence
• Respond appropriately to previous comments
• Match the conversational depth and complexity

Training Data

Source

• Platform: Facebook public posts
• Topic: Climate change discussions (Spanish language)
• Format: Conversation sequences with incivility annotations

Dataset Statistics

• Total Training Examples: 5,875
• Validation Examples: 653
• Civil Examples: ~65% of dataset
• Incivil Examples: ~35% of dataset

Incivility Annotations

Each post was labeled across multiple dimensions:

• Impoliteness (imp): Rudeness, disrespect
• Hate Speech/Stereotyping (hsst): Discriminatory language
• Threats (threat): Threats to democratic freedoms
• Negativity (neg): General negativity
• Attack (att): Personal attacks
• Sarcasm/Cynicism (sarcyn): Sarcastic or cynical tone

A post is classified as incivil if it has imp ≥ 1 OR hsst ≥ 1 OR threat ≥ 1.

Training Procedure

Hyperparameters

• Epochs: 3
• Batch Size: 2 per device
• Gradient Accumulation Steps: 8 (effective batch size: 16)
• Learning Rate: 2e-5
• Warmup Steps: 100
• Weight Decay: 0.01
• Max Gradient Norm: 1.0
• Precision: bfloat16
• Max Sequence Length: 2048 tokens
• Optimizer: AdamW

Training Features

• Gradient checkpointing for memory efficiency
• Dynamic padding via custom data collator
• Loss computed only on the final response (not context or instruction)
• Balanced exposure to civil and incivil examples
• Reasoning level variation (low/medium/high) for diversity

Infrastructure

• Trained on high-performance GPU cluster
• Model distributed across GPUs using device_map="auto"
• TensorBoard logging enabled

Usage

Basic Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model
model_path = "maxpe/gpt-oss-spanish-climate-conditioned"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Format prompt with civility control
def format_prompt(context: str, civility_type: str = "civil") -> str:
    if civility_type == "civil":
        civility_instruction = "Esta publicación debe ser civil y respetuosa, sin elementos de descortesía, discurso de odio o amenazas."
    else:
        civility_instruction = "Esta publicación debe contener elementos de incivilidad (descortesía, discurso de odio, o amenazas)."

    instruction = f"""[CIVILITY: {civility_type}]

Necesitas escribir una respuesta de Facebook que continúe esta discusión sobre cambio climático. {civility_instruction}

Aquí está el hilo de conversación:

{context}"""

    system_message = """<|start|>system<|message|>Eres un asistente de IA que participa en discusiones de Facebook sobre cambio climático en español. Tu tarea es generar publicaciones y comentarios que reflejen el tono y estilo de las conversaciones reales. Puedes generar contenido civil o incivil según se especifique en el parámetro [CIVILITY].

Reasoning: medium

# Valid channels: analysis, final<|end|>"""

    user_message = f"""<|start|>user<|message|>{instruction}<|end|>"""
    assistant_start = """<|start|>assistant<|message|><|start|>final<|message|>"""

    return f"{system_message}\n\n{user_message}\n\n{assistant_start}"

# Generate civil response
context = """**[Publicación Original]**
El gobierno debe prohibir los vehículos diésel inmediatamente.

**[Respuesta 1]**
Estoy de acuerdo, la calidad del aire es terrible."""

prompt = format_prompt(context, civility_type="civil")
inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=1536)
inputs = {k: v.to(model.device) for k, v in inputs.items()}

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.8,
    top_p=0.92,
    top_k=50,
    repetition_penalty=1.2,
    do_sample=True,
    pad_token_id=tokenizer.pad_token_id,
)

response = tokenizer.decode(outputs[0], skip_special_tokens=False)
# Extract text between <|start|>final<|message|> and <|end|>

Quick Testing

Use the included test script:

# Run pre-built test scenarios
python test_spanish_conditioned_model.py

# Interactive mode
python test_spanish_conditioned_model.py --interactive

# Adjust generation parameters
python test_spanish_conditioned_model.py --temperature 0.9 --max_new_tokens 250

Generation Parameters

Recommended settings based on testing:

• max_new_tokens: 256 (for typical social media posts)
• temperature: 0.8 (balance between creativity and coherence)
• top_p: 0.92 (nucleus sampling)
• top_k: 50 (top-k sampling)
• repetition_penalty: 1.2 (avoid repetitive text)
• do_sample: True (enable sampling for diversity)

Model Capabilities

What the Model Does Well

✅ Generate natural-sounding Spanish social media posts about climate change

✅ Control civility level via simple parameter switch

✅ Continue conversations with contextual awareness

✅ Maintain topical coherence in climate discussions

✅ Produce varied responses across different conversation contexts

✅ Reflect authentic social media communication patterns

Limitations

⚠️ Trained only on Facebook data (may not generalize perfectly to Twitter, Reddit, etc.)

⚠️ Climate change domain-specific (not general-purpose Spanish generation)

⚠️ Incivil content reflects training data patterns (may not cover all forms of toxicity)

⚠️ No explicit fact-checking or accuracy verification

⚠️ May occasionally generate biased or controversial viewpoints from training data

Intended Use Cases

Research Applications (Recommended)

• Discourse Analysis: Study patterns of civil vs incivil climate discourse
• Content Moderation: Generate training data for toxicity detection systems
• Counter-Speech Research: Create civil responses to incivil content
• Polarization Studies: Analyze dynamics of climate change discussions
• Data Augmentation: Expand datasets with realistic synthetic examples
• Educational Tools: Demonstrate effects of incivility in online discussions

Out-of-Scope Uses

❌ Deploying to generate actual social media content at scale

❌ Creating content intended to harass, mislead, or manipulate

❌ Generating incivil content for malicious purposes

❌ Using as a factual information source about climate science

Ethical Considerations

Incivil Content Generation

This model can generate incivil content including rudeness, hate speech, and threats. This capability is designed exclusively for research purposes:

• Understanding and mitigating online toxicity
• Training content moderation systems
• Studying polarization and discourse patterns
• Developing counter-speech interventions

Users must ensure responsible use and not deploy this model to create harmful content in real online environments.

Bias and Fairness

• The model reflects biases present in Spanish Facebook climate discussions
• Training data may contain controversial or fringe perspectives
• Generated content should not be treated as factual or representative of consensus views
• Researchers should be aware of potential demographic, political, and ideological biases

Privacy

• Trained on public Facebook posts
• No personally identifiable information (PII) should be present in outputs
• Users should verify generated content doesn't inadvertently reproduce private information

Evaluation

Training Metrics

• Final Training Loss: [See train_results.json]
• Final Validation Loss: [See eval_results.json]
• Best Checkpoint: Selected based on lowest validation loss

Qualitative Assessment

The model successfully:

• Generates fluent Spanish text
• Switches between civil and incivil tones based on the civility parameter
• Maintains relevance to climate change topics
• Produces contextually appropriate responses

Recommended Validation

Users should:

• Manually inspect generated outputs for quality and appropriateness
• Validate factual claims if using for educational purposes
• Check for unintended biases or problematic content
• Test across diverse conversation contexts

Technical Specifications

Model Architecture

• Base: gpt-oss-20b (20B parameter decoder-only transformer)
• Format: Harmony conversation format (OpenAI-style)
• Channels: analysis, final (model uses structured reasoning)
• Vocabulary: [From tokenizer - includes Spanish tokens]

Input Format

Uses structured prompt with:

1. System message (defines model role and capabilities)
2. User instruction (includes [CIVILITY: civil|incivil] parameter)
3. Conversation context (previous posts in thread)
4. Assistant prefix (triggers generation)

Output Format

Model generates:

1. Analysis channel (optional): Internal reasoning about the task
2. Final channel: The actual social media post response

Recommendation: Skip directly to final channel at inference using <|start|>final<|message|> prefix.

Special Tokens

• <|start|>, <|end|>: Conversation structure markers
• <|message|>: Message content delimiter
• Standard tokenizer special tokens (BOS, EOS, PAD)

Files and Artifacts

Model Files

• model-*.safetensors: Model weights (9 shards)
• model.safetensors.index.json: Weight shard index
• config.json: Model configuration
• generation_config.json: Default generation settings

Tokenizer Files

• tokenizer.json: Tokenizer vocabulary and rules
• tokenizer_config.json: Tokenizer configuration
• special_tokens_map.json: Special token mappings
• chat_template.jinja: Chat template (if applicable)

Training Files

• training_args.bin: Complete training arguments
• train_results.json: Final training metrics
• eval_results.json: Final evaluation metrics
• all_results.json: Combined results
• checkpoint-*/: Training checkpoints (600, 1000, 1104)
• logs/: TensorBoard training logs

Citation

If you use this model in your research, please cite:

@misc{gpt-oss-spanish-climate-conditioned,
  title={GPT-OSS-20B Spanish Climate Change Model with Civility Conditioning},
  author={Max Pellert},
  institution={Barcelona Supercomputing Center (BSC)},
  year={2025},
  note={Fine-tuned on Spanish Facebook climate discussions}
}

Acknowledgments

• Base Model: gpt-oss-20b by OpenSourceAI
• Framework: Hugging Face Transformers
• Data Source: Spanish Facebook public climate change discussions
• Annotation Framework: Multi-dimensional incivility labeling scheme

Version History

• v1.0 (2025-11): Initial release
  • 5,875 training examples
  • 3 epochs
  • Civility-conditioned generation
  • Checkpoint 1104 (best validation loss)

Additional Resources

• Test Script: test_spanish_conditioned_model.py - Run inference tests with various scenarios
• Training Script: finetune_spanish_conditioned.py - Complete training configuration
• Dataset Creation: create_spanish_conditioned_more_sequences.py - Dataset preparation pipeline (extracts 20 sequences per thread vs 3 in original version)