🐋DeepSeek-R1-0528: How to Run Locally
A guide on how to run DeepSeek-R1-0528 including Qwen3 on your own local device!
DeepSeek-R1-0528 is DeepSeek's new update to their R1 reasoning model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic 1.66-bit version uses 162GB (-80% reduction in size). GGUF: DeepSeek-R1-0528-GGUF
NEW: Huge improvements to tool calling and chat template fixes.
New TQ1_0 dynamic 1.66-bit quant - 162GB in size. Ideal for 192GB RAM (including Mac) and Ollama users. Try: ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0
DeepSeek also released a R1-0528 distilled version by fine-tuning Qwen3 (8B). The distill achieves similar performance to Qwen3 (235B). You can also fine-tune Qwen3 Distill with Unsloth. Qwen3 GGUF: DeepSeek-R1-0528-Qwen3-8B-GGUF
All uploads use Unsloth Dynamic 2.0 for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss.
Tutorials navigation:
⚙️ Recommended Settings
For DeepSeek-R1-0528-Qwen3-8B, the model can pretty much fit in any setup, and even those with as less as 20GB RAM. There is no need for any prep beforehand. However, for the full R1-0528 model which is 715GB in size, you will need extra prep. The 1.78-bit (IQ1_S) quant will fit in a 1x 24GB GPU (with all layers offloaded). Expect around 5 tokens/s with this setup if you have bonus 128GB RAM as well.
It is recommended to have at least 64GB RAM to run this quant (you will get 1 token/s without a GPU). For optimal performance you will need at least 180GB unified memory or 180GB combined RAM+VRAM for 5+ tokens/s.
We suggest using our 2.7bit (Q2_K_XL) or 2.4bit (IQ2_XXS) quant to balance size and accuracy! The 2.4bit one also works well.
Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading.
🐳 Official Recommended Settings:
According to DeepSeek, these are the recommended settings for R1 (R1-0528 and Qwen3 distill should use the same settings) inference:
Set the temperature 0.6 to reduce repetition and incoherence.
Set top_p to 0.95 (recommended)
Run multiple tests and average results for reliable evaluation.
🔢 Chat template/prompt format
R1-0528 uses the same chat template as the original R1 model. You do not need to force <think>\n
, but you can still add it in!
A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call tokenizer.encode(..., add_special_tokens = False)
since the chat template auto adds a BOS token as well.
For llama.cpp / GGUF inference, you should skip the BOS since it’ll auto add it:
The <think>
and </think>
tokens get their own designated tokens.
Model uploads
ALL our uploads - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.
Qwen3 (8B) distill: DeepSeek-R1-0528-Qwen3-8B-GGUF
Full DeepSeek-R1-0528 model uploads below:
We also uploaded IQ4_NL and Q4_1 quants which run specifically faster for ARM and Apple devices respectively.
We've also uploaded versions in BF16 format, and original FP8 (float8) format.
Run DeepSeek-R1-0528 Tutorials:
🦙 Run in Ollama/Open WebUI
Install
ollama
if you haven't already! You can only run models up to 32B in size. To run the full 720GB R1-0528 model, see here.
Run the model! Note you can call
ollama serve
in another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) inparams
in our Hugging Face upload!
(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1_0 (162GB quant):
🦙 Run Full R1-0528 on Ollama/Open WebUI
Open WebUI has made an step-by-step tutorial on how to run R1 here and for R1-0528, you will just need to replace R1 with the new 0528 quant: docs.openwebui.com/tutorials/integrations/deepseekr1-dynamic/
(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1_0 (162GB quant):
If you want to use any of the quants that are larger than TQ1_0 (162GB) on Ollama, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally.
✨ Run Qwen3 distilled R1 in llama.cpp
To run the full 720GB R1-0528 model, see here. Obtain the latest
llama.cpp
on GitHub here. You can follow the build instructions below as well. Change-DGGML_CUDA=ON
to-DGGML_CUDA=OFF
if you don't have a GPU or just want CPU inference.
Then use llama.cpp directly to download the model:
✨ Run Full R1-0528 on llama.cpp
Obtain the latest
llama.cpp
on GitHub here. You can follow the build instructions below as well. Change-DGGML_CUDA=ON
to-DGGML_CUDA=OFF
if you don't have a GPU or just want CPU inference.
If you want to use
llama.cpp
directly to load models, you can do the below: (:IQ1_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar toollama run
. Useexport LLAMA_CACHE="folder"
to forcellama.cpp
to save to a specific location.
Please try out -ot ".ffn_.*_exps.=CPU"
to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.
If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU"
This offloads up and down projection MoE layers.
Try -ot ".ffn_(up)_exps.=CPU"
if you have even more GPU memory. This offloads only up projection MoE layers.
And finally offload all layers via -ot ".ffn_.*_exps.=CPU"
This uses the least VRAM.
You can also customize the regex, for example -ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU"
means to offload gate, up and down MoE layers but only from the 6th layer onwards.
Download the model via (after installing
pip install huggingface_hub hf_transfer
). You can chooseUD-IQ1_S
(dynamic 1.78bit quant) or other quantized versions likeQ4_K_M
. I recommend using our 2.7bit dynamic quantUD-Q2_K_XL
to balance size and accuracy. More versions at: https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF
Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1.
Edit
--threads 32
for the number of CPU threads,--ctx-size 16384
for context length,--n-gpu-layers 2
for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
🎱 Heptagon Test
You can also test our dynamic quants via r/Localllama which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape.

🦥 Fine-tuning DeepSeek-R1-0528 with Unsloth
To fine-tune DeepSeek-R1-0528-Qwen3-8B using Unsloth, we’ve made a new GRPO notebook featuring a custom reward function designed to significantly enhance multilingual output - specifically increasing the rate of desired language responses (in our example we use Indonesian but you can use any) by more than 40%.
While many reasoning LLMs have multilingual capabilities, they often produce mixed-language outputs in its reasoning traces, combining English with the target language. Our reward function effectively mitigates this issue by strongly encouraging outputs in the desired language, leading to a substantial improvement in language consistency.
This reward function is also fully customizable, allowing you to adapt it for other languages or fine-tune for specific domains or use cases.
The best part about this whole reward function and notebook is you DO NOT need a language dataset to force your model to learn a specific language. The notebook has no Indonesian dataset.
Unsloth makes R1-Qwen3 distill fine-tuning 2× faster, uses 70% less VRAM, and support 8× longer context lengths.
Last updated
Was this helpful?