cjkasbdkjnlakb/qwen3-4b-agent-0926-quancan

See axolotl config

axolotl version: 0.12.2

# Automatically upload checkpoint and final model to HF
# hub_model_id: username/custom_model_name
# 是否以 8-bit 精度加载模型
load_in_8bit: false
# 是否以 4-bit 精度加载模型（与QLoRA绑定, 强制使用）
load_in_4bit: false
# 是否严格匹配模型结构，关闭表示可加载少部分差异结构（如以适配 adapter）
# strict: false

base_model: Qwen/Qwen3-4B-Instruct-2507

# 数据集设置
chat_template: qwen3
datasets:
- path: /workspace/train_dir_0926-02/goal_data.json # 只使用新增的1500条数据
  type: chat_template # chat_template(自定义格式) alpaca
  roles_to_train: ["assistant"]
  field_messages: messages # 标识的字段
  message_property_mappings: # message_property_mappings={'role':'role', 'content':'content'})
    role: role
    content: content

dataset_prepared_path:
val_set_size: 0.1 # 8卡训练可以适当减少验证集比例，提高训练数据利用率
output_dir: /workspace/train_dir_0926-02/checkpoints
sequence_len: 7000 # 模型所能处理的最大上下文长度(默认2048)
pad_to_sequence_len: true
# context_parallel_size: 2 # 长序列拆分至多个GPU(强制要求 mirco_batch_size: 1)
sample_packing: false # Function call数据通常较短，可以启用packing提高效率
eval_sample_packing: false # 评估时拼接多个样本

# 训练超参数 - 针对function call优化
lora_r: 32 # function call需要更强的表达能力，提高rank
lora_alpha: 64 # 设为 2*r，保持合理的缩放
lora_dropout: 0.1 # 8卡训练数据分布更均匀，可以适当降低dropout

# 批处理设置 - 充分利用8卡H100
micro_batch_size: 8 # H100显存大，可以增加batch size
gradient_accumulation_steps: 2 # 8卡并行，可以减少累积步数
auto_find_batch_size: false # 允许Axolotl不断调整batch_size ⚠️Zero-3不适用
num_epochs: 3 # function call数据质量高，可以多训练几轮

# 优化器设置
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 8e-5 # 在已有模型基础上微调function call，稍微提高学习率

# 精度设置
bf16: auto
tf32: true

# 日志和评估
logging_steps: 30 # 8卡训练速度快，可以适当减少日志频率
flash_attention: true
warmup_ratio: 0.1 # 已有基础模型，减少warmup
evals_per_epoch: 1 # 适中的评估频次
saves_per_epoch: 1 # 适中的保存频次
weight_decay: 0.01 # 8卡训练更稳定，减少正则化

# FSDP配置 - 针对8卡H100优化
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_limit_all_gathers: true
  fsdp_sync_module_states: true
  fsdp_offload_params: false # H100显存充足，无需offload
  fsdp_use_orig_params: false
  fsdp_cpu_ram_efficient_loading: true
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_transformer_layer_cls_to_wrap: Qwen3DecoderLayer
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_activation_checkpointing: true # 使用FSDP的activation checkpointing

workspace/train_dir_0926-02/checkpoints

This model is a fine-tuned version of Qwen/Qwen3-4B-Instruct-2507 on the /workspace/train_dir_0926-02/goal_data.json dataset. It achieves the following results on the evaluation set:

• Loss: 0.0403
• Memory/max Mem Active(gib): 121.88
• Memory/max Mem Allocated(gib): 121.69
• Memory/device Mem Reserved(gib): 136.37

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 64
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 16
• training_steps: 161

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	1.1221	83.21	83.01	84.11
0.2295	1.0	54	0.0622	121.88	121.69	136.37
0.0512	2.0	108	0.0403	121.88	121.69	136.37

Framework versions

• Transformers 4.55.2
• Pytorch 2.6.0+cu126
• Datasets 4.0.0
• Tokenizers 0.21.4