ms-swift 训练 Qwen3 最佳实践从思考/非思考推理到 SFT、GRPO 与 Megatron 多节点训练【免费下载链接】swiftUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600 LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300 MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).项目地址: https://gitcode.com/GitHub_Trending/swift1/swift本篇基于 ms-swift 仓库中的 Qwen3 最佳实践文档系统梳理 Qwen3 系列模型在 ms-swift 中的完整落地路径如何在推理时切换思考与非思考模式、如何用 LoRA 与全参微调保持模型推理能力、如何开展 GRPO 强化学习以及如何借助 Megatron 并行技术训练 Qwen3-30B-A3B 这类 MoE 模型。读完本文你可以直接复用文档中的命令完成从推理验证、SFT、蒸馏数据生产到 RLHF 与多节点大模型训练的全流程。思考与非思考两种推理模式Qwen3 属于“思考型”thinking模型其默认模板在 ms-swift 中按is_thinkingTrue注册并定义了非思考前缀non_thinking_prefix think\n\n/think\n\n具体见 Qwen3 模板定义。此外仓库中还分别注册了纯思考版qwen3_thinking与纯非思考版qwen3_nothinking模板便于按模型变体精确匹配dataclass class Qwen3MixedTemplateMeta(QwenTemplateMeta): default_system: Optional[str] None non_thinking_prefix: str think\n\n/think\n\n register_template(Qwen3MixedTemplateMeta(LLMTemplateType.qwen3, is_thinkingTrue))从源码结构看模板在生成首 token 时会将response_prefix拼接到输出最前面见 推理前缀拼接逻辑且取值优先级为请求级chat_template_kwargs 模板级response_prefix 由enable_thinking推导的默认前缀见 前缀推导逻辑。这正是下面两种模式切换的实现基础。思考模式直接运行即可模型会先输出think内的推理过程再给出答案CUDA_VISIBLE_DEVICES0 \ swift infer \ --model Qwen/Qwen3-8B \ --infer_backend vllm \ --stream true \ --max_new_tokens 2048 \ --vllm_max_model_len 8192 who are you? Okay, the user is asking who are you? Let me start by introducing myself as Qwen, the large language model developed by Alibaba Cloud. I should mention my capabilities, like answering questions, creating content, and engaging in conversations. But I need to keep it concise. Also, the user might want to know how I can assist them. Maybe I should ask how they can help them today. Let me check if theres anything else important to include. Oh, I should make sure the tone is friendly and approachable. Alright, that should cover it. /think Hello! I am Qwen, a large language model developed by Alibaba Cloud. I can assist with a wide range of tasks, such as answering questions, creating content, writing stories, coding, and more. How can I help you today? clear who are you? /no_think I am Qwen, a large language model developed by Alibaba Cloud. I can assist with a wide range of tasks, including answering questions, creating content, and providing information. How can I help you today?注意交互输入中的/no_think后缀在对话里追加它即可让模型跳过推理过程直接作答。非思考模式非思考模式通过--response_prefix实现该参数表示模型的输出将从给定前缀之后继续等价于将enable_thinking设为 False。对 Qwen3 而言即传入空思考前缀think\n\n/think\n\nCUDA_VISIBLE_DEVICES0 \ swift infer \ --model Qwen/Qwen3-8B \ --infer_backend vllm \ --stream true \ --max_new_tokens 2048 \ --vllm_max_model_len 8192 \ --response_prefix think\n\n/think\n\n who are you? I am Qwen, a large-scale language model developed by Alibaba Cloud. I am designed to assist with a wide range of tasks, including answering questions, creating content, and providing information. How can I assist you today?训练环境准备开始训练前先确保环境依赖安装到位pip install ms-swift -U pip install transformers pip install deepspeed # for multi-GPU training pip install liger-kernel # to save GPU memory resources pip install flash-attn --no-build-isolation # required for packingdeepspeed多卡分布式训练所需示例中使用了--deepspeed zero3liger-kernel节省显存对应--use_liger_kernel trueflash-attn使用--packing true时必须同时设置--attn_impl flash_attn。SFT数据格式与推理能力保持策略自定义数据集格式SFT 自定义数据集采用 messages 结构system字段可选支持 JSON、JSONL、CSV 三种组织形式训练时通过--dataset dataset_path指定。完整的数据集规范见 Custom Dataset Documentation。# General format {messages: [ {role: system, content: system-prompt}, {role: user, content: query1}, {role: assistant, content: response1} ]} # Format with thinking process {messages: [ {role: user, content: Where is the capital of Zhejiang?}, {role: assistant, content: Thought: ...\n\nAnswer:\nThe capital of Zhejiang is Hangzhou.} ]}用无思考链数据训练时如何保住推理能力若希望使用不含思考链的数据训练、同时不破坏模型原有的推理能力文档给出两种方案方案 1推荐--loss_scale ignore_empty_think。训练时忽略think\n\n/think\n\n部分的损失计算从而避免模型学会“放弃思考”。参考训练脚本为 qwen3_demo1.sh同目录还有 qwen3_demo2.sh 与 deepseek_r1.sh后者说明该方法同样适用于 DeepSeek-R1 等模型。其底层实现可以溯源到源码ignore_empty_think对应 IgnoreEmptyThinkLossScale配置规则来自 ignore_empty_think.json其中第一条正则^think\s*/think\s*的权重为0.0即空思考标签序列不参与损失仓库还提供ConcatLossScale用于组合多个 loss scale例如hermesignore_empty_think见 ConcatLossScale 实现。对应的数据集格式为{messages: [ {role: user, content: Where is the capital of Zhejiang?}, {role: assistant, content: think\n\n/think\n\nThe capital of Zhejiang is Hangzhou.} ]}方案 2在 query 中追加/no_think。通过数据集里的/no_think提示让模型走非思考分支参考脚本为 qwen3_demo2.sh。{messages: [ {role: user, content: Where is the capital of Zhejiang? /no_think}, {role: assistant, content: think\n\n/think\n\nThe capital of Zhejiang is Hangzhou.} ]}生产蒸馏推理数据集还可以用大模型批量推理生成带完整思考链的蒸馏数据训练时再与不含 CoT 的数据混合进一步缓解推理能力损失。--val_dataset的选取是任意的推理结果写入result_path后可直接用--dataset distill_dataset.jsonl参与训练。该做法同样适用于 deepseek-r1 等其他推理模型# 4 * 80GiB NPROC_PER_NODE4 \ CUDA_VISIBLE_DEVICES0,1,2,3 \ swift infer \ --model Qwen/Qwen3-32B \ --infer_backend vllm \ --val_dataset AI-ModelScope/alpaca-gpt4-data-en#5000 AI-ModelScope/alpaca-gpt4-data-zh#5000 \ --vllm_gpu_memory_utilization 0.9 \ --vllm_tensor_parallel_size 2 \ --vllm_max_model_len 8192 \ --max_new_tokens 4096 \ --write_batch_size 1000 \ --result_path distill_dataset.jsonl30 分钟自我认知微调本节演示在 30 分钟内对 Qwen3-8B 做自我认知微调一张显存不低于 22GB 的 GPU 即可运行例如 ModelScope 平台提供的 A10 实例。训练完成后模型不再自认为“通义实验室训练的 Qwen”而是“swift 团队训练的 swift-robot”。若需在离线环境训练可手动下载模型与数据集再用--model model-path和--dataset dataset-dir指定本地路径数据集为swift/self-cognition其在仓库中的注册见 self-cognition 数据集定义。# GPU Memory Usage: 22GB CUDA_VISIBLE_DEVICES0 \ swift sft \ --model Qwen/Qwen3-8B \ --tuner_type lora \ --dataset swift/Qwen3-SFT-Mixin#2000 \ swift/self-cognition:qwen3#600 \ --load_from_cache_file true \ --torch_dtype bfloat16 \ --num_train_epochs 1 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-4 \ --lora_rank 8 \ --lora_alpha 32 \ --target_modules all-linear \ --gradient_accumulation_steps 16 \ --eval_steps 50 \ --save_steps 50 \ --save_total_limit 2 \ --logging_steps 5 \ --max_length 2048 \ --output_dir output \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --model_author swift \ --model_name swift-robot参数说明完整参数见 Command Line Arguments Documentation--tuner_type loraLoRA 微调lora_rank 8/lora_alpha 32/target_modules all-linear是 22GB 显存下的典型配置--model_author/--model_name自我认知微调专用分别替换swift/self-cognition数据集中的{{AUTHOR}}与{{NAME}}占位符其语义在 data_args 参数定义 中有明确注释--dataset支持多数据集混训#600、:qwen3#600分别表示按条数采样与按子集/标签采样。微调完成后将--adapters指向最终 checkpoint 目录即可验证效果CUDA_VISIBLE_DEVICES0 \ swift infer \ --adapters output/vx-xxx/checkpoint-xxx \ --stream true \ --temperature 0 \ --max_new_tokens 2048 who are you? I need to introduce myself. Let me start by stating my name, swift-robot. Then, I should mention that Im an AI assistant developed by swift. I should explain my purpose, which is to provide information and assistance. I should also highlight my capabilities, like answering questions, generating text, and engaging in conversation. Its important to keep the tone friendly and approachable. /think Hello! I am swift-robot, an artificial intelligence assistant developed by swift. My purpose is to provide information and assistance to users like you. I can answer questions, generate text, and engage in conversations on a wide range of topics. I am here to help, so feel free to ask me anything you need!默认情况下 ms-swift 从 ModelScope 社区下载模型与数据集如需改用 HuggingFace 社区请追加--use_hf true。合并 LoRA 权重与推送模型swift export \ --adapters output/checkpoint-xxx \ --merge_lora true推送至 ModelScope/HuggingFace推送全量权重时把--adapters换成--model# If pushing full weights, change --adapters to --model. swift export \ --adapters output/checkpoint-xxx \ --push_to_hub true \ --hub_model_id hub-model-id \ --hub_token hub-token \ --use_hf false多卡全参微调以下示例为 4 卡全参训练4 × 60GB 显存--dataset可替换为AI-ModelScope/alpaca-gpt4-data-en等内置数据集直接跑通实验。再次强调指定--packing true时必须同时设置--attn_impl flash_attn。# 4 * 60GB # You can run the experiment by setting --dataset AI-ModelScope/alpaca-gpt4-data-en # Note: If you specify --packing true, you must also set --attn_impl flash_attn NPROC_PER_NODE4 \ CUDA_VISIBLE_DEVICES0,1,2,3 \ swift sft \ --model Qwen/Qwen3-8B \ --tuner_type full \ --dataset your-dataset \ --load_from_cache_file true \ --split_dataset_ratio 0.01 \ --torch_dtype bfloat16 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --learning_rate 1e-5 \ --gradient_accumulation_steps 4 \ --packing true \ --eval_steps 100 \ --save_steps 100 \ --logging_steps 5 \ --max_length 8192 \ --warmup_ratio 0.05 \ --dataloader_num_workers 8 \ --dataset_num_proc 8 \ --save_total_limit 2 \ --save_only_model true \ --output_dir output \ --deepspeed zero3 \ --use_liger_kernel true \ --attn_impl flash_attn强化学习GRPOms-swift 支持 DPO、GRPO、DAPO、PPO、KTO、GKD 等 RLHF 方法本节约定以 Qwen3-8B 的 GRPO 训练为主。GRPO 的完整原理与参数见 GRPO 入门文档 及 自定义奖励函数指南更多 RLHF 训练脚本见 examples/train/rlhf。环境准备在上述 ms-swift 依赖之外还需安装pip install math_verify pip install vllm数据准备GRPO 数据集格式与 SFT 类似但不需要最终的 assistant 回复。若以准确率作为奖励则需额外提供solution列用于计算正确性{messages: [{role: user, content: Tell me tomorrows weather}]} {messages: [{role: user, content: What is 1 1?}, {role: assistant, content: It equals 2}, {role: user, content: What about adding 1?}]} {messages: [{role: user, content: What is your name?}]}其他 RLHF 算法的数据格式见 Custom Dataset Documentation。数据集与奖励函数的匹配规则奖励函数计算数据集格式取决于所用奖励函数。使用内置accuracy或cosine奖励时数据集必须包含solution列自定义奖励数据集中的其余列会以**kwargs形式传入奖励函数便于二次定制实现自定义奖励函数可参考 examples/train/grpo 下的示例脚本与模板。本例使用AI-MO/NuminaMath-TIR数据集计算准确率奖励采样过程由 vLLM 加速# 70G*8 CUDA_VISIBLE_DEVICES0,1,2,3,4,5,6,7 \ NPROC_PER_NODE8 \ swift rlhf \ --rlhf_type grpo \ --model Qwen/Qwen3-8B \ --tuner_type full \ --dataset AI-MO/NuminaMath-TIR#5000 \ --load_from_cache_file true \ --torch_dtype bfloat16 \ --num_train_epochs 1 \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 2 \ --learning_rate 1e-6 \ --save_total_limit 2 \ --logging_steps 5 \ --output_dir output \ --gradient_accumulation_steps 1 \ --warmup_ratio 0.05 \ --dataloader_num_workers 4 \ --max_length 4096 \ --max_completion_length 4096 \ --vllm_max_model_len 8192 \ --reward_funcs accuracy \ --num_generations 16 \ --use_vllm true \ --vllm_gpu_memory_utilization 0.4 \ --sleep_level 1 \ --offload_model true \ --offload_optimizer true \ --deepspeed zero3 \ --vllm_tensor_parallel_size 1 \ --temperature 1.0 \ --top_p 0.85 \ --log_completions true \ --overlong_filter true其中--reward_funcs accuracy在源码中对应MathAccuracy奖励类其注册映射为accuracy: MathAccuracy实现位于 ORM 奖励函数。--num_generations 16表示每条样本采样 16 条回复用于组内优势估计--offload_model/--offload_optimizer配合--vllm_gpu_memory_utilization 0.4实现训练与 vLLM 推理在 8 张 70G 卡上的显存复用。Megatron-SWIFTMoE 大模型多节点训练ms-swift 引入 Megatron 并行技术以加速大模型的 CPT/SFT/DPO/GRPO 训练支持模型列表见 Supported Models and Datasets环境搭建见 Megatron-SWIFT Quick Start。文档以 Qwen3-30B-A3B-Base 为例在阿里云 DLC 上以双节点各 8 卡 80GiB A800 的环境启动训练多机启动方式可参考 examples/train/multi-node。关键并行配置为--pipeline_model_parallel_size 2跨节点流水线与--expert_model_parallel_size 8节点内专家并行并启用 MoE 融合算子与全量激活重计算# 多机环境变量 NNODES/NODE_RANK 由调度平台注入如阿里云 DLC PYTORCH_CUDA_ALLOC_CONFexpandable_segments:True \ NNODES$WORLD_SIZE \ NODE_RANK$RANK \ megatron sft \ --model Qwen/Qwen3-30B-A3B-Base \ --save_safetensors true \ --dataset liucong/Chinese-DeepSeek-R1-Distill-data-110k-SFT \ --load_from_cache_file true \ --split_dataset_ratio 0.01 \ --pipeline_model_parallel_size 2 \ --expert_model_parallel_size 8 \ --moe_permute_fusion true \ --moe_grouped_gemm true \ --moe_shared_expert_overlap true \ --moe_aux_loss_coeff 1e-3 \ --micro_batch_size 1 \ --global_batch_size 16 \ --packing true \ --recompute_granularity full \ --recompute_method uniform \ --recompute_num_layers 1 \ --train_iters 2000 \ --eval_iters 50 \ --finetune true \ --cross_entropy_loss_fusion true \ --lr 1e-5 \ --lr_warmup_fraction 0.05 \ --min_lr 1e-6 \ --output_dir megatron_output/Qwen3-30B-A3B-Base \ --eval_steps 200 \ --save_steps 200 \ --max_length 8192 \ --dataloader_num_workers 8 \ --dataset_num_proc 8 \ --no_save_optim true \ --no_save_rng true \ --sequence_parallel true \ --attention_backend flashMegatron 后端使用与swift sft相同的自定义数据集格式只需指定--dataset dataset_path格式见上文 SFT 一节。原文档还给出了 Qwen3-30B-A3B 全参微调下三种后端的训练速度与显存占用对比数据来自原文档实测Megatron-LMDeepSpeed-ZeRO2DeepSpeed-ZeRO3Training speed9.6s/it-91.2s/itGPU memory16 × 60GiBOOM16 × 80GiB从该对比可以推断对 MoE 大模型Megatron 并行在单步训练速度和显存占用上均显著优于 DeepSpeed 路线且 60GiB 卡型即可运行ZeRO3 需 80GiBZeRO2 则直接 OOM。延伸阅读数据集完整规范Custom Dataset Documentation全量命令行参数Command-line ParametersGRPO 入门与进阶GRPO Get StartedMegatron 训练快速上手Megatron-SWIFT Quick Start思考模型训练脚本examples/train/think_modelRLHF 训练脚本集合examples/train/rlhf【免费下载链接】swiftUse PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600 LLMs (Qwen3.6, DeepSeek-V4, GLM-5.1, InternLM3, Llama4, ...) and 300 MLLMs (Qwen3-VL, Qwen3-Omni, InternVL3.5, Ovis2.5, GLM4.5v, Gemma4, Llava, Phi4, ...) (AAAI 2025).项目地址: https://gitcode.com/GitHub_Trending/swift1/swift创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考