配置文件结构

XTuner 一份 config 通常 200-300 行。本章把"标准模板"按 PART 拆开讲。

5.1整体结构

# PART 1：超参数（最上面）
pretrained_model_name_or_path = "internlm/internlm2-chat-7b"
data_path = "tatsu-lab/alpaca"
batch_size = 1
max_length = 2048
lr = 2e-4
max_epochs = 3
...

# PART 2：Model
model = dict(
    type=SupervisedFinetune,
    ...
    llm=dict(...),
    lora=dict(...),
)

# PART 3：Dataset & Dataloader
alpaca_en = dict(
    type=process_hf_dataset,
    dataset=dict(type=load_dataset, path=data_path),
    tokenizer=tokenizer,
    max_length=max_length,
    dataset_map_fn=alpaca_map_fn,
    template_map_fn=...,
    remove_unused_columns=True,
    shuffle_before_pack=True,
    pack_to_max_length=pack_to_max_length,
)

train_dataloader = dict(
    batch_size=batch_size,
    num_workers=dataloader_num_workers,
    dataset=alpaca_en,
    sampler=dict(type=DefaultSampler, shuffle=True),
    collate_fn=dict(type=default_collate_fn),
)

# PART 4：Scheduler & Optimizer
optim_wrapper = dict(
    type=AmpOptimWrapper,
    optimizer=dict(type=AdamW, lr=lr, weight_decay=weight_decay, betas=betas),
    clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),
    accumulative_counts=accumulative_counts,
    loss_scale='dynamic',
    dtype='bfloat16',
)

param_scheduler = [
    dict(type=LinearLR, start_factor=1e-5, by_epoch=True, begin=0, end=warmup_ratio*max_epochs),
    dict(type=CosineAnnealingLR, ...),
]

# PART 5：Runtime
train_cfg = dict(type=TrainLoop, max_epochs=max_epochs)
default_hooks = dict(...)
custom_hooks = [dict(type=EvaluateChatHook, ...)]
env_cfg = dict(cudnn_benchmark=False, mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0))
visualizer = dict(type=Visualizer, vis_backends=[dict(type=TensorboardVisBackend)])
log_processor = dict(by_epoch=False)
log_level = "INFO"
load_from = None
resume = False
randomness = dict(seed=None, deterministic=False)

5.2PART 1 高频字段

5.3PART 2 model 详解

model = dict(
    type=SupervisedFinetune,
    use_varlen_attn=use_varlen_attn,             # 可变长 attention（packing 配合）
    llm=dict(
        type=AutoModelForCausalLM.from_pretrained,
        pretrained_model_name_or_path=pretrained_model_name_or_path,
        trust_remote_code=True,
        torch_dtype=torch.float16,                # 注意：fp16 还是 bf16
        # QLoRA 时加：
        quantization_config=dict(
            type=BitsAndBytesConfig,
            load_in_4bit=True,
            bnb_4bit_quant_type='nf4',
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
        ),
    ),
    lora=dict(
        type=LoraConfig,
        r=64, lora_alpha=16,
        lora_dropout=0.1, bias='none',
        task_type='CAUSAL_LM',
    ),
)

核心：type=SupervisedFinetune 是 XTuner 自家 wrapper，内部把 base 模型套上 LoRA / 量化 / 长度处理。

5.4PART 3 dataset 详解

alpaca_en = dict(
    type=process_hf_dataset,                # XTuner 的统一入口
    dataset=dict(type=load_dataset, path=data_path),
    tokenizer=tokenizer,
    max_length=max_length,
    dataset_map_fn=alpaca_map_fn,            # ★ 选哪个 map fn 决定格式
    template_map_fn=dict(
        type=template_map_fn_factory,
        template=PROMPT_TEMPLATE.internlm2_chat,   # ★ chat 模板
    ),
    remove_unused_columns=True,
    shuffle_before_pack=True,
    pack_to_max_length=pack_to_max_length,
)

5.5PART 5 Hooks（高频自定义）

custom_hooks = [
    dict(type=DatasetInfoHook, tokenizer=tokenizer),
    dict(
        type=EvaluateChatHook,
        tokenizer=tokenizer,
        every_n_iters=evaluation_freq,
        evaluation_inputs=evaluation_inputs,
        system=SYSTEM,
        prompt_template=prompt_template,
    ),
]

EvaluateChatHook 是 XTuner 招牌 hook：训练每 N 步用预设 prompt 跑一次推理输出到日志，让你实时看效果。

5.6这章你需要带走的

• XTuner config 五大 PART：超参 / Model / Dataset / Optim & Scheduler / Runtime；
• SupervisedFinetune 是顶层 model wrapper；
• process_hf_dataset + map_fn 决定数据格式；
• EvaluateChatHook 让你实时看训练效果。