Ling Mini 2.0 Identity

This model is a fine-tuned version of inclusionAI/Ling-mini-2.0 on the identity dataset (from LLaMA-Factory).

Training procedure

Full fine tuning with DeepSpeed Zero3 offloading and 4 x A100 80GB. For a faster setup, you can use the qingy1337/llamafactory-cu128:latest docker image.

Training hyperparameters

The following hyperparameters were used during training:

model_name_or_path: inclusionAI/Ling-mini-2.0
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z3_config.json

### dataset
dataset: identity
template: bailing_v2
cutoff_len: 8192
max_samples: 10000000000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: ./outputs/
logging_steps: 1
save_steps: 10000000000
save_only_model: true
plot_loss: true
overwrite_output_dir: true
report_to: wandb
run_name: Test-FT

### train
per_device_train_batch_size: 2
gradient_accumulation_steps: 1
learning_rate: 1.0e-6
num_train_epochs: 10.0
lr_scheduler_type: cosine
warmup_ratio: 0.2
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null