🌠Qwen3: How to Run & Fine-tune

Learn to run & fine-tune Qwen3 locally with Unsloth + our Dynamic 2.0 quants

Qwen's new Qwen3 models deliver state-of-the-art advancements in reasoning, instruction-following, agent capabilities, and multilingual support. All Qwen3 uploads use our new Unsloth Dynamic 2.0 methodology, delivering the best performance on 5-shot MMLU and KL Divergence benchmarks. This means, you can run and fine-tune quantized Qwen3 LLMs with minimal accuracy loss!

We also uploaded Qwen3 with native 128K context length. Qwen achieves this by using YaRN to extend its original 40K window to 128K.

• Running Qwen3 Tutorial	• Fine-tuning Qwen3 Tutorial (TBA)

FIXED UPDATE 04/29/2025: our GGUFs now work on ALL inference engines including llama.cpp, Ollama, LM Studio & Open WebUI.

Qwen3 - Unsloth Dynamic 2.0 with optimal configs:

Dynamic 2.0 GGUF (to run)	128K Context GGUF	Dynamic 4-bit Safetensor (to finetune/deploy)
0.6B 1.7B 4B 8B 14B 30B-A3B 32B 235B-A22B (TBA)	0.6B 1.7B 4B 8B 14B 30B-A3B 32B 235B-A22B (TBA)	0.6B 1.7B 4B 8B 14B 30B-A3B 32B

Qwen3 30B-A3B is now fixed! All uploads are now fixed and will work anywhere with any quant!

🖥️ Running Qwen3

⚙️ Official Recommended Settings

According to Qwen, these are the recommended settings for inference:

Non-Thinking Mode Settings:	Thinking Mode Settings:
Temperature = 0.7	Temperature = 0.6
Min_P = 0.0 (optional, but 0.01 works well, llama.cpp default is 0.1)	Min_P = 0.0
Top_P = 0.8	Top_P = 0.95
TopK = 20	TopK = 20

Chat template/prompt format:

<|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n

For NON thinking mode, we purposely enclose <think> and </think> with nothing:

<|im_start|>user\nWhat is 2+2?<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n

For Thinking-mode, DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions.

Switching Between Thinking and Non-Thinking Mode

Qwen3 models come with built-in "thinking mode" to boost reasoning and improve response quality - similar to how QwQ-32B worked.

Thinking mode

enable_thinking=True

By default, Qwen3 has thinking enabled. When you call tokenizer.apply_chat_template, you don’t need to set anything manually.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # Default is True
)

In thinking mode, the model will generate an extra <think>...</think> block before the final answer — this lets it "plan" and sharpen its responses.

Non-thinking mode

enable_thinking=False

Enabling non-thinking will make Qwen3 will skip all the thinking steps and behave like a normal LLM.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False  # Disables thinking mode
)

This mode will provide final responses directly — no <think> blocks, no chain-of-thought.

🦙 Ollama: Run Qwen3 Tutorial

1. Install ollama if you haven't already! You can only run models up to 32B in size. To run the full 235B-A22B model, see here.

apt-get update
apt-get install pciutils -y
curl -fsSL https://ollama.com/install.sh | sh

1. Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload!

ollama run hf.co/unsloth/Qwen3-32B-GGUF:Q4_K_XL

📖 Llama.cpp: Run Qwen3 Tutorial

1. Obtain the latest llama.cpp on GitHub here. You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

apt-get update
apt-get install pciutils build-essential cmake curl libcurl4-openssl-dev -y
git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
    -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first --target llama-cli llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp

1. Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose Q4_K_M, or other quantized versions.

# !pip install huggingface_hub hf_transfer
import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id = "unsloth/Qwen3-32B-GGUF",
    local_dir = "unsloth/Qwen3-32B-GGUF",
    allow_patterns = ["*UD-Q4_K_XL*"],
)

1. Run the model and try any prompt.

Running Qwen3-235B-A22B (TBA)

For Qwen3-235B-A22B, we will specifically use Llama.cpp for optimized inference and a plethora of options. We're still doing some testing so we'd recommend you wait.

1. We're following similar steps to above however this time we'll also need to perform extra steps because the model is so big.

2. Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD_IQ2_XXS, or other quantized versions..

   # !pip install huggingface_hub hf_transfer
   import os
   os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
   from huggingface_hub import snapshot_download
   snapshot_download(
       repo_id = "unsloth/Qwen3-235B-A22B-GGUF",
       local_dir = "unsloth/Qwen3-235B-A22B-GGUF",
       allow_patterns = ["*UD-IQ2_XXS*"],
   )

3. Run the model and try any prompt.

4. Edit --threads 32 for the number of CPU threads, --ctx-size 16384 for context length, --n-gpu-layers 99 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

Use -ot ".ffn_.*_exps.=CPU" to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

./llama.cpp/llama-cli \
    --model unsloth/Qwen3-235B-A22B-GGUF/Qwen3-235B-A22B-UD-IQ2_XXS.gguf \
    --threads 32 \
    --ctx-size 16384 \
    --n-gpu-layers 99 \
    -ot ".ffn_.*_exps.=CPU" \
    --seed 3407 \
    --prio 3 \
    --temp 0.6 \
    --min-p 0.0 \
    --top-p 0.95 \
    --top-k 20 \
    -no-cnv \
    --prompt "<|im_start|>user\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|im_end|>\n<|im_start|>assistant\n"

🦥 Fine-tuning Qwen3 with Unsloth

Notebooks coming soon!