train_run-SmolLM2-135M-Instruct-fadel-full-arabic-diacritization

This model is a fine-tuned version of HuggingFaceTB/SmolLM2-135M-Instruct on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.0470

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.PAGED_ADAMW with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: constant
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 1

Training results

Training Loss	Epoch	Step	Validation Loss
0.134	0.0623	500	0.1328
0.0999	0.1247	1000	0.1022
0.081	0.1870	1500	0.0856
0.0691	0.2493	2000	0.0770
0.0668	0.3117	2500	0.0695
0.0642	0.3740	3000	0.0659
0.0605	0.4364	3500	0.0637
0.0586	0.4987	4000	0.0586
0.0539	0.5610	4500	0.0557
0.0545	0.6234	5000	0.0541
0.0517	0.6857	5500	0.0528
0.0503	0.7480	6000	0.0522
0.0467	0.8104	6500	0.0496
0.0474	0.8727	7000	0.0488
0.0434	0.9350	7500	0.0488
0.0473	0.9974	8000	0.0470

Framework versions

• PEFT 0.14.0
• Transformers 4.51.1
• Pytorch 2.5.1+cu124
• Datasets 3.5.0
• Tokenizers 0.21.0