Qwen3高效微调（下）
输入“/”快速插入内容
Qwen3高效微调（下）
Qwen3高效微调（下）
课程说明：
•
此外，公开课全套学习资料，已上传至网盘
需要更系统深入学习大模型可扫码⬆️添加助教咨询喔～
Qwen3高效微调实战（下）
纯代码版
•
导入模型
代码块
from unsloth import FastLanguageModel
import torch
代码块
max_seq_length = 2048
dtype = None
load_in_4bit = True
代码块
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "./Qwen3-32B-unsloth-bnb-4bit",
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
代码块
==((====))== Unsloth 2025.4.7: Fast Qwen3 patching. Transformers: 4.51.3.
\\ /| NVIDIA H800 PCIe. Num GPUs = 1. Max memory: 79.205 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.7.0+cu126. CUDA: 9.0. CUDA Toolkit: 12.6. Triton: 3.3.0
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.30. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Loading checkpoint shards: 100%|██████████| 9/9 [00:14<00:00, 1.63s/it]
代码块
model
代码块
Qwen3ForCausalLM(
(model): Qwen3Model(
(embed_tokens): Embedding(151936, 5120, padding_idx=151654)
(layers): ModuleList(
(0-5): 6 x Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear4bit(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(6): Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(7): Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(8-22): 15 x Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear4bit(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(23-44): 22 x Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(45): Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(46-54): 9 x Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(55-61): 7 x Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear4bit(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(62): Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
(63): Qwen3DecoderLayer(
(self_attn): Qwen3Attention(
(q_proj): Linear4bit(in_features=5120, out_features=8192, bias=False)
(k_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(v_proj): Linear4bit(in_features=5120, out_features=1024, bias=False)
(o_proj): Linear4bit(in_features=8192, out_features=5120, bias=False)
(q_norm): Qwen3RMSNorm((128,), eps=1e-06)
(k_norm): Qwen3RMSNorm((128,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): Qwen3MLP(
(gate_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(up_proj): Linear4bit(in_features=5120, out_features=25600, bias=False)
(down_proj): Linear4bit(in_features=25600, out_features=5120, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
(post_attention_layernorm): Qwen3RMSNorm((5120,), eps=1e-06)
)
)
(norm): Qwen3RMSNorm((5120,), eps=1e-06)
(rotary_emb): LlamaRotaryEmbedding()
)
(lm_head): Linear(in_features=5120, out_features=151936, bias=False)
)