1 of 92

Unsloth Documentation

Get Started

Unsloth Docs

Train your own model with Unsloth, an open-source framework for LLM fine-tuning and reinforcement learning.

At , our mission is to make AI as accurate and accessible as possible. Train, run, evaluate and save gpt-oss, Llama, DeepSeek, TTS, Qwen, Mistral, Gemma LLMs 2x faster with 70% less VRAM.

Our docs will guide you through running & training your own model locally.

Fine-tuning for Beginners

If you're a beginner, here might be the first questions you'll ask before your first fine-tune. You can also always ask our community by joining our .

Unsloth Requirements

Here are Unsloth's requirements including system and GPU VRAM requirements.

System Requirements

Operating System: Works on Linux and
Supports NVIDIA GPUs since 2018+ including

FAQ + Is Fine-tuning Right For Me?

If you're stuck on if fine-tuning is right for you, see here! Learn about fine-tuning misconceptions, how it compared to RAG and more:

Understanding Fine-Tuning

Fine-tuning an LLM customizes its behavior, deepens its domain expertise, and optimizes its performance for specific tasks. By refining a pre-trained model (e.g. Llama-3.1-8B) with specialized data, you can:

Update Knowledge – Introduce new, domain-specific information that the base model didn’t originally include.
Customize Behavior – Adjust the model’s tone, personality, or response style to fit specific needs or a brand voice.
Optimize for Tasks – Improve accuracy and relevance on particular tasks or queries your use-case requires.

Think of fine-tuning as creating a specialized expert out of a generalist model. Some debate whether to use Retrieval-Augmented Generation (RAG) instead of fine-tuning, but fine-tuning can incorporate knowledge and behaviors directly into the model in ways RAG cannot. In practice, combining both approaches yields the best results - leading to greater accuracy, better usability, and fewer hallucinations.

Real-World Applications of Fine-Tuning

Fine-tuning can be applied across various domains and needs. Here are a few practical examples of how it makes a difference:

Sentiment Analysis for Finance – Train an LLM to determine if a news headline impacts a company positively or negatively, tailoring its understanding to financial context.
Customer Support Chatbots – Fine-tune on past customer interactions to provide more accurate and personalized responses in a company’s style and terminology.
Legal Document Assistance – Fine-tune on legal texts (contracts, case law, regulations) for tasks like contract analysis, case law research, or compliance support, ensuring the model uses precise legal language.

The Benefits of Fine-Tuning

Fine-tuning offers several notable benefits beyond what a base model or a purely retrieval-based system can provide:

Fine-Tuning vs. RAG: What’s the Difference?

Fine-tuning can do mostly everything RAG can - but not the other way around. During training, fine-tuning embeds external knowledge directly into the model. This allows the model to handle niche queries, summarize documents, and maintain context without relying on an outside retrieval system. That’s not to say RAG lacks advantages as it is excels at accessing up-to-date information from external databases. It is in fact possible to retrieve fresh data with fine-tuning as well, however it is better to combine RAG with fine-tuning for efficiency.

Task-Specific Mastery

Fine-tuning deeply integrates domain knowledge into the model. This makes it highly effective at handling structured, repetitive, or nuanced queries, scenarios where RAG-alone systems often struggle. In other words, a fine-tuned model becomes a specialist in the tasks or content it was trained on.

Independence from Retrieval

A fine-tuned model has no dependency on external data sources at inference time. It remains reliable even if a connected retrieval system fails or is incomplete, because all needed information is already within the model’s own parameters. This self-sufficiency means fewer points of failure in production.

Faster Responses

Fine-tuned models don’t need to call out to an external knowledge base during generation. Skipping the retrieval step means they can produce answers much more quickly. This speed makes fine-tuned models ideal for time-sensitive applications where every second counts.

Custom Behavior and Tone

Fine-tuning allows precise control over how the model communicates. This ensures the model’s responses stay consistent with a brand’s voice, adhere to regulatory requirements, or match specific tone preferences. You get a model that not only knows what to say, but how to say it in the desired style.

Reliable Performance

Even in a hybrid setup that uses both fine-tuning and RAG, the fine-tuned model provides a reliable fallback. If the retrieval component fails to find the right information or returns incorrect data, the model’s built-in knowledge can still generate a useful answer. This guarantees more consistent and robust performance for your system.

Common Misconceptions

Despite fine-tuning’s advantages, a few myths persist. Let’s address two of the most common misconceptions about fine-tuning:

Does Fine-Tuning Add New Knowledge to a Model?

Yes - it absolutely can. A common myth suggests that fine-tuning doesn’t introduce new knowledge, but in reality it does. If your fine-tuning dataset contains new domain-specific information, the model will learn that content during training and incorporate it into its responses. In effect, fine-tuning can and does teach the model new facts and patterns from scratch.

Is RAG Always Better Than Fine-Tuning?

Not necessarily. Many assume RAG will consistently outperform a fine-tuned model, but that’s not the case when fine-tuning is done properly. In fact, a well-tuned model often matches or even surpasses RAG-based systems on specialized tasks. Claims that “RAG is always better” usually stem from fine-tuning attempts that weren’t optimally configured - for example, using incorrect or insufficient training.

Unsloth takes care of these complexities by automatically selecting the best parameter configurations for you. All you need is a good-quality dataset, and you'll get a fine-tuned model that performs to its fullest potential.

Is Fine-Tuning Expensive?

Not at all! While full fine-tuning or pretraining can be costly, these are not necessary (pretraining is especially not necessary). In most cases, LoRA or QLoRA fine-tuning can be done for minimal cost. In fact, with Unsloth’s for Colab or Kaggle, you can fine-tune models without spending a dime. Better yet, you can even fine-tune locally on your own device.

FAQ:

Why You Should Combine RAG & Fine-Tuning

Instead of choosing between RAG and fine-tuning, consider using both together for the best results. Combining a retrieval system with a fine-tuned model brings out the strengths of each approach. Here’s why:

Task-Specific Expertise – Fine-tuning excels at specialized tasks or formats (making the model an expert in a specific area), while RAG keeps the model up-to-date with the latest external knowledge.
Better Adaptability – A fine-tuned model can still give useful answers even if the retrieval component fails or returns incomplete information. Meanwhile, RAG ensures the system stays current without requiring you to retrain the model for every new piece of data.
Efficiency – Fine-tuning provides a strong foundational knowledge base within the model, and RAG handles dynamic or quickly-changing details without the need for exhaustive re-training from scratch. This balance yields an efficient workflow and reduces overall compute costs.

LoRA vs. QLoRA: Which One to Use?

When it comes to implementing fine-tuning, two popular techniques can dramatically cut down the compute and memory requirements: LoRA and QLoRA. Here’s a quick comparison of each:

LoRA (Low-Rank Adaptation) – Fine-tunes only a small set of additional “adapter” weight matrices (in 16-bit precision), while leaving most of the original model unchanged. This significantly reduces the number of parameters that need updating during training.
QLoRA (Quantized LoRA) – Combines LoRA with 4-bit quantization of the model weights, enabling efficient fine-tuning of very large models on minimal hardware. By using 4-bit precision where possible, it dramatically lowers memory usage and compute overhead.

We recommend starting with QLoRA, as it’s one of the most efficient and accessible methods available. Thanks to Unsloth’s quants, the accuracy loss compared to standard 16-bit LoRA fine-tuning is now negligible.

Experimentation is Key

There’s no single “best” approach to fine-tuning - only best practices for different scenarios. It’s important to experiment with different methods and configurations to find what works best for your dataset and use case. A great starting point is QLoRA (4-bit), which offers a very cost-effective, resource-friendly way to fine-tune models without heavy computational requirements.

Unsloth Installation

Learn to install Unsloth locally or online.

Unsloth works on Linux, Windows, NVIDIA, AMD, Google Colab and more. See our system requirements.

Recommended install methods:

pip install unsloth

uv pip install unsloth

To update Unsloth

pip install --upgrade unsloth

Docker

Install Unsloth using our official Docker container

Learn how to use our Docker containers with all dependencies pre-installed for immediate installation. No setup required, just run and start training!

Unsloth Docker image:

You can now use our main Docker image unsloth/unsloth for Blackwell and 50-series GPUs - no separate image needed.

Updating

To update or use an old version of Unsloth, follow the steps below:

Standard Updating (recommended):

pip install --upgrade unsloth unsloth_zoo

Updating without dependency updates:

pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth
pip install --upgrade --force-reinstall --no-cache-dir --no-deps unsloth_zoo

To use an old version of Unsloth:

'2025.1.5' is one of the previous old versions of Unsloth. Change it to a specific release listed on our .

Instal Unsloth via pip and uv

To install Unsloth locally via Pip, follow the steps below:

Recommended installation method

Install with pip (recommended) for the latest pip release:

To use uv:

To install vLLM and Unsloth together, do:

To install the latest main branch of Unsloth, do:

For venv and virtual environments installs to isolate your installation to not break system packages, and to reduce irreparable damage to your system, use venv:

AMD

Guide for Fine-tuning LLMs with Unsloth on AMD GPUs.

Unsloth supports AMD Radeon RX, MI300X's (192GB) GPUs and more.

Make a new isolated environment (Optional)

To not break any system packages, you can make an isolated pip environment. Reminder to check what Python version you have! It might be pip3, pip3.13, python3, python.3.13 etc.

Conda Install

To install Unsloth locally on Conda, follow the steps below:

Only use Conda if you have it. If not, use .

Select either pytorch-cuda=11.8,12.1 for CUDA 11.8 or CUDA 12.1. We support python=3.10,3.11,3.12.

If you're looking to install Conda in a Linux environment, , or run the below:

Google Colab

To install and run Unsloth on Google Colab, follow the steps below:

If you have never used a Colab notebook, a quick primer on the notebook itself:

Play Button at each "cell". Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter errors, simply rerun the cell you did not run. Another option is to click CTRL + ENTER if you don't want to click the play button.
Runtime Button in the top toolbar. You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, but is a good first try.

What Model Should I Use for Fine-tuning?

Llama, Qwen, Mistral, Phi or?

When preparing for fine-tuning, one of the first decisions you'll face is selecting the right model. Here's a step-by-step guide to help you choose:

Choose a model that aligns with your usecase

FP16 vs BF16 for RL

Defeating the Training-Inference Mismatch via FP16 https://arxiv.org/pdf/2510.26788 shows how using float16 is better than bfloat16

Float16 vs Bfloat16

There was a paper titled "Defeating the Training-Inference Mismatch via FP16" showing how using float16 precision can dramatically be better than using bfloat16 when doing reinforcement learning.

In fact the longer the generation, the worse it gets when using bfloat16:

We did an investigation, and DO find float16 to be more stable

RL Reward Hacking

Learn what is Reward Hacking in Reinforcement Learning and how to counter it.

The ultimate goal of RL is to maximize some reward (say speed, revenue, some metric). But RL can cheat. When the RL algorithm learns a trick or exploits something to increase the reward, without actually doing the task at end, this is called "Reward Hacking".

It's the reason models learn to modify unit tests to pass coding challenges, and these are critical blockers for real world deployment. Some other good examples are from Wikipedia.

Can you counter reward hacking? Yes! In our free gpt-oss RL notebook we explore how to counter reward hacking in a code generation setting and showcase tangible solutions to common error modes. We saw the model edit the timing function, outsource to other libraries, cache the results, and outright cheat. After countering, the result is our model generates genuinely optimized matrix multiplication kernels, not clever cheats.

🏆 Reward Hacking Overview

Some common examples of reward hacking during RL include:

Laziness

RL learns to use Numpy, Torch, other libraries, which calls optimized CUDA kernels. We can stop the RL algorithm from calling optimized code by inspecting if the generated code imports other non standard Python libraries.

Caching & Cheating

RL learns to cache the result of the output and RL learns to find the actual output by inspecting Python global variables.

We can stop the RL algorithm from using cached data by wiping the cache with a large fake matrix. We also have to benchmark carefully with multiple loops and turns.

Cheating

RL learns to edit the timing function to make it output 0 time as passed. We can stop the RL algorithm from using global or cached variables by restricting it's locals and globals. We are also going to use exec to create the function, so we have to save the output to an empty dict. We also disallow global variable access via types.FunctionType(f.__code__, {})\

GSPO Reinforcement Learning

Train with GSPO (Group Sequence Policy Optimization) RL in Unsloth.

We're introducing GSPO which is a variant of GRPO made by the Qwen team at Alibaba. They noticed the observation that when GRPO takes importance weights for each token, even though inherently advantages do not scale or change with each token. This lead to the creation of GSPO, which now assigns the importance on the sequence likelihood rather than the individual token likelihoods of the tokens.

Use our free GSPO notebooks for: gpt-oss-20b and Qwen2.5-VL

Enable GSPO in Unsloth by setting importance_sampling_level = "sequence" in the GRPO config. The difference between these two algorithms can be seen below, both from the GSPO paper from Qwen and Alibaba:

In Equation 1, it can be seen that the advantages scale each of the rows into the token logprobs before that tensor is sumed. Essentially, each token is given the same scaling even though that scaling was given to the entire sequence rather than each individual token. A simple diagram of this can be seen below:

Equation 2 shows that the logprob ratios for each sequence is summed and exponentiated after the Logprob ratios are computed, and only the resulting now sequence ratios get row wise multiplied by the advantages.

Enabling GSPO is simple, all you need to do is set the importance_sampling_level = "sequence" flag in the GRPO config.

Reinforcement Learning - DPO, ORPO & KTO

To use the reward modelling functions for DPO, GRPO, ORPO or KTO with Unsloth, follow the steps below:

DPO (Direct Preference Optimization), ORPO (Odds Ratio Preference Optimization), PPO, KTO Reward Modelling all work with Unsloth.

We have Google Colab notebooks for reproducing GRPO, ORPO, DPO Zephyr, KTO and SimPO:

New

500K Context Length Fine-tuning

Learn how to enable >500K token context window fine-tuning with Unsloth.

We’re introducing new algorithms in Unsloth that push the limits of long-context training for any LLM and VLM. Training LLMs like gpt-oss-20b can now reach 500K+ context lengths on a single 80GB H100 GPU, compared to 80K previously with no accuracy degradation.

You can reach >750K context windows on a B200 192GB GPU.

Try 500K-context gpt-oss-20b fine-tuning on our .

We’ve significantly improved how Unsloth handles memory usage patterns, speed, and context lengths:

60% lower VRAM use with 3.2x longer context via Unsloth’s new

How to Fine-tune LLMs with Unsloth & Docker

Learn how to fine-tune LLMs or do Reinforcement Learning (RL) with Unsloth's Docker image.

Local training can be complex due to dependency hell or breaking environments. Unsloth’s can bypass these issues. No setup is needed: pull and run the image and start training.

Unsloth official Docker image:

Why Use Unsloth & Docker?

Unsloth’s Docker image is stable, up-to-date and works in like Windows.

Fully contained dependencies keep your system clean. Runs safely without root.

Models

Devstral 2: How to Run Guide

Guide for local running Mistral Devstral 2 models: 123B-Instruct-2512 and Small-2-24B-Instruct-2512.

Devstral 2 are Mistral’s new coding and agentic LLMs for software engineering, available in and sizes. The 123B model achieves SOTA in SWE-bench, coding, tool-calling and agent use-cases. The 24B model fits in 25GB RAM/VRAM and 123B fits in 128GB.

We’ve resolved issues in Devstral’s chat template, and results should be significantly better. The 24B & 123B have been updated.

Devstral 2 supports vision capabilities, a 256k context window and uses the same architecture as . You can now run and fine-tune both models locally with Unsloth.

All Devstral 2 uploads use our Unsloth methodology, delivering the best performance on and 5-shot MMLU benchmarks.

How to Run Local LLMs with Docker: Step-by-Step Guide

Learn how to run Large Language Models (LLMs) with Docker & Unsloth on your local device.

You can now run any model, including Unsloth , on Mac, Windows or Linux with a single line of code or no code at all. We collabed with Docker to simplify model deployment, and Unsloth now powers most GGUF models on Docker.

Before you start, make sure to look over and for optimizing performance when running LLMs on your device.

To get started, run OpenAI with a single command:

Or to run a specific / quant from Hugging Face:

Basics

Inference & Deployment

Learn how to save your finetuned model so you can run it in your favorite inference engine.

You can also run your fine-tuned models by using Unsloth's 2x faster inference.

vLLM Deployment & Inference Guide

Guide on saving and deploying LLMs to vLLM for serving LLMs in production

💻Installing vLLM

For NVIDIA GPUs, use uv and run:

For AMD GPUs, please use the nightly Docker image: rocm/vllm-dev:nightly

For the nightly branch for NVIDIA GPUs, run:

See for more details

LoRA Hot Swapping Guide

🍧 vLLM LoRA Hot Swapping / Dynamic LoRAs

To enable LoRA serving for at most 4 LoRAs at 1 time (these are hot swapped / changed), first set the environment flag to allow hot swapping:

Then, serve it with LoRA support:

To load a LoRA dynamically (set the lora name as well), do:

To remove it from the pool:

For example when finetuning with Unsloth:

Then after training, we save the LoRAs:

Unsloth Inference

Learn how to run your finetuned model with Unsloth's faster inference.

Unsloth supports natively 2x faster inference. For our inference only notebook, click here.

All QLoRA, LoRA and non LoRA inference paths are 2x faster. This requires no change of code or any new dependencies.

from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 64)

NotImplementedError: A UTF-8 locale is required. Got ANSI

Sometimes when you execute a cell this error can appear. To solve this, in a new cell, run the below:

import locale
locale.getpreferredencoding = lambda: "UTF-8"

Troubleshooting Inference

If you're experiencing issues when running or saving your model.

Running in Unsloth works well, but after exporting & running on other platforms, the results are poor

You might sometimes encounter an issue where your model runs and produces good results on Unsloth, but when you use it on another platform like Ollama or vLLM, the results are poor or you might get gibberish, endless/infinite generations or repeated outputs.

The most common cause of this error is using an incorrect chat template. It’s essential to use the SAME chat template that was used when training the model in Unsloth and later when you run it in another framework, such as llama.cpp or Ollama. When inferencing from a saved model, it's crucial to apply the correct template.
You must use the correct eos token. If not, you might get gibberish on longer generations.
It might also be because your inference engine adds an unnecessary "start of sequence" token (or the lack of thereof on the contrary) so ensure you check both hypotheses!
Use our conversational notebooks to force the chat template - this will fix most issues.
- Qwen-3 14B Conversational notebook
- Gemma-3 4B Conversational notebook
- Llama-3.2 3B Conversational notebook

Saving to `safetensors`, not `bin` format in Colab

We save to .bin in Colab so it's like 4x faster, but set safe_serialization = None to force saving to .safetensors. So model.save_pretrained(..., safe_serialization = None) or model.push_to_hub(..., safe_serialization = None)

If saving to GGUF or vLLM 16bit crashes

You can try reducing the maximum GPU usage during saving by changing maximum_memory_usage.

The default is model.save_pretrained(..., maximum_memory_usage = 0.75). Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving.

Multi-GPU Fine-tuning with Unsloth

Learn how to fine-tune LLMs on multiple GPUs and parallelism with Unsloth.

Unsloth currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed. This means you can already leverage parallelism methods such as FSDP and DDP with Unsloth.

See our new Distributed Data Parallel (DDP) multi-GPU Guide here.

We know that the process can be complex and requires manual setup. We’re working hard to make multi-GPU support much simpler and more user-friendly, and we’ll be announcing official multi-GPU support for Unsloth soon.

For now, you can use our Magistral-2509 Kaggle notebook as an example which utilizes multi-GPU Unsloth to fit the 24B parameter model or our DDP guide.

In the meantime, to enable multi GPU for DDP, do the following:

Create your training script as train.py (or similar). For example, you can use one of our created from our various notebooks!
Run accelerate launch train.py or torchrun --nproc_per_node N_GPUS train.py where N_GPUS is the number of GPUs you have.

Pipeline / model splitting loading is also allowed, so if you do not have enough VRAM for 1 GPU to load say Llama 70B, no worries - we will split the model for you on each GPU! To enable this, use the device_map = "balanced" flag:

Stay tuned for our official announcement! For more details, check out our ongoing discussing multi-GPU support.

Unsloth Environment Flags

Advanced flags which might be useful if you see breaking finetunes, or you want to turn stuff off.

Environment variable

Purpose

Continued Pretraining

AKA as Continued Finetuning. Unsloth allows you to continually pretrain so a model can learn a new language.

The is for continued pretraining/raw text.
The is for learning another language.

You can read more about continued pretraining and our release in our .

FAQ + Is Fine-tuning Right For Me?

If you're stuck on if fine-tuning is right for you, see here! Learn about fine-tuning misconceptions, how it compared to RAG and more:

Understanding Fine-Tuning

Update Knowledge – Introduce new, domain-specific information that the base model didn’t originally include.
Customize Behavior – Adjust the model’s tone, personality, or response style to fit specific needs or a brand voice.
Optimize for Tasks – Improve accuracy and relevance on particular tasks or queries your use-case requires.

Real-World Applications of Fine-Tuning

Fine-tuning can be applied across various domains and needs. Here are a few practical examples of how it makes a difference:

Sentiment Analysis for Finance – Train an LLM to determine if a news headline impacts a company positively or negatively, tailoring its understanding to financial context.
Customer Support Chatbots – Fine-tune on past customer interactions to provide more accurate and personalized responses in a company’s style and terminology.
Legal Document Assistance – Fine-tune on legal texts (contracts, case law, regulations) for tasks like contract analysis, case law research, or compliance support, ensuring the model uses precise legal language.

The Benefits of Fine-Tuning

Fine-tuning offers several notable benefits beyond what a base model or a purely retrieval-based system can provide:

Fine-Tuning vs. RAG: What’s the Difference?

Task-Specific Mastery

Independence from Retrieval

Faster Responses

Custom Behavior and Tone

Reliable Performance

Common Misconceptions

Despite fine-tuning’s advantages, a few myths persist. Let’s address two of the most common misconceptions about fine-tuning:

Does Fine-Tuning Add New Knowledge to a Model?

Is RAG Always Better Than Fine-Tuning?

Is Fine-Tuning Expensive?

FAQ:

Why You Should Combine RAG & Fine-Tuning

Task-Specific Expertise – Fine-tuning excels at specialized tasks or formats (making the model an expert in a specific area), while RAG keeps the model up-to-date with the latest external knowledge.
Better Adaptability – A fine-tuned model can still give useful answers even if the retrieval component fails or returns incomplete information. Meanwhile, RAG ensures the system stays current without requiring you to retrain the model for every new piece of data.
Efficiency – Fine-tuning provides a strong foundational knowledge base within the model, and RAG handles dynamic or quickly-changing details without the need for exhaustive re-training from scratch. This balance yields an efficient workflow and reduces overall compute costs.

LoRA vs. QLoRA: Which One to Use?

When it comes to implementing fine-tuning, two popular techniques can dramatically cut down the compute and memory requirements: LoRA and QLoRA. Here’s a quick comparison of each:

LoRA (Low-Rank Adaptation) – Fine-tunes only a small set of additional “adapter” weight matrices (in 16-bit precision), while leaving most of the original model unchanged. This significantly reduces the number of parameters that need updating during training.
QLoRA (Quantized LoRA) – Combines LoRA with 4-bit quantization of the model weights, enabling efficient fine-tuning of very large models on minimal hardware. By using 4-bit precision where possible, it dramatically lowers memory usage and compute overhead.

Experimentation is Key

Qwen3-Next: Run Locally Guide

Run Qwen3-Next-80B-A3B-Instruct and Thinking versions locally on your device!

Qwen released Qwen3-Next in Sept 2025, which are 80B MoEs with Thinking and Instruct model variants of Qwen3. With 256K context, Qwen3-Next was designed with a brand new architecture (Hybrid of MoEs & Gated DeltaNet + Gated Attention) that specifically optimizes for fast inference on longer context lengths. Qwen3-Next has 10x faster inference than Qwen3-32B.

Run Qwen3-Next InstructRun Qwen3-Next Thinking

Qwen3-Next-80B-A3B Dynamic GGUFs: Instruct • Thinking

⚙️ Usage Guide

NEW as of Dec 6, 2025: Unsloth Qwen3-Next now updated with iMatrix for improved performance.

The thinking model uses temperature = 0.6, but the instruct model uses temperature = 0.7 The thinking model uses top_p = 0.95, but the instruct model uses top_p = 0.8

To achieve optimal performance, Qwen recommends these settings:

Instruct:

Thinking:

Adequate Output Length: Use an output length of 32,768 tokens for most queries for the thinking variant, and 16,384 for the instruct variant. You can increase the max output size for the thinking model if necessary.

Chat template for both Thinking (thinking has <think></think>) and Instruct is below:

📖 Run Qwen3-Next Tutorials

Below are guides for the and versions of the model.

Instruct: Qwen3-Next-80B-A3B-Instruct

Given that this is a non thinking model, the model does not generate <think> </think> blocks.

⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

We suggest using temperature=0.7, top_p=0.8, top_k=20, and min_p=0.0 presence_penalty between 0 and 2 if the framework supports to reduce endless repetitions.
temperature = 0.7
top_k = 20

✨ Llama.cpp: Run Qwen3-Next-80B-A3B-Instruct Tutorial

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

You can directly pull from HuggingFace via:
Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD_Q4_K_XL or other quantized versions.

Thinking: Qwen3-Next-80B-A3B-Thinking

This model supports only thinking mode and a 256K context window natively. The default chat template adds <think> automatically, so you may see only a closing </think> tag in the output.

⚙️Best Practices

To achieve optimal performance, Qwen recommends the following settings:

We suggest using temperature=0.6, top_p=0.95, top_k=20, and min_p=0.0 presence_penalty between 0 and 2 if the framework supports to reduce endless repetitions.
temperature = 0.6
top_k = 20

✨ Llama.cpp: Run Qwen3-Next-80B-A3B-Thinking Tutorial

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

You can directly pull from Hugging Face via:
Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD_Q4_K_XL or other quantized versions.

🛠️ Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

Normally, -ot ".ffn_.*_exps.=CPU" offloads all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU" This offloads up and down projection MoE layers.

Try -ot ".ffn_(up)_exps.=CPU" if you have even more GPU memory. This offloads only up projection MoE layers.

You can also customize the regex, for example -ot "\.(6|7|8|9|[0-9][0-9]|[0-9][0-9][0-9])\.ffn_(gate|up|down)_exps.=CPU" means to offload gate, up and down MoE layers but only from the 6th layer onwards.

The also introduces high throughput mode. Use llama-parallel. Read more about it . You can also quantize the KV cache to 4bits for example to reduce VRAM / RAM movement, which can also make the generation process faster. The talks about KV cache quantization.

📐How to fit long context

To fit longer context, you can use KV cache quantization to quantize the K and V caches to lower bits. This can also increase generation speed due to reduced RAM / VRAM data movement. The allowed options for K quantization (default is f16) include the below.

--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1

You should use the _1 variants for somewhat increased accuracy, albeit it's slightly slower. For eg q4_1, q5_1 So try out --cache-type-k q4_1

You can also quantize the V cache, but you will need to compile llama.cpp with Flash Attention support via -DGGML_CUDA_FA_ALL_QUANTS=ON, and use --flash-attn to enable it. After installing Flash Attention, you can then use --cache-type-v q4_1

3x Faster LLM Training with Unsloth Kernels + Packing

Learn how Unsloth increases training throughput and eliminates padding waste for fine-tuning.

Unsloth now supports up to 5× faster (typically 3x) training with our new custom RoPE and MLP Triton kernels, plus our new smart auto packing. Unsloth's new kernels + features not only increase training speed, but also further reduces VRAM use (30% - 90%) with no accuracy loss. Unsloth GitHub This means you can now train LLMs like Qwen3-4B not only on just 3GB VRAM, but also 3x faster.

Our auto padding-free uncontaminated packing is smartly enabled for all training runs without any changes, and all fast attention backends (FlashAttention 3, xFormers, SDPA). Benchmarks show training losses match non-packing runs exactly.

2.3x faster QK Rotary Embedding fused Triton kernel with packing support
Updated SwiGLU, GeGLU kernels with int64 indexing for long context
2.5x to 5x faster uncontaminated packing with xformers, SDPA, FA3 backends
2.1x faster padding free, 50% less VRAM, 0% accuracy change
Unsloth also now has improved SFT loss stability and more predictable GPU utilization.
This new upgrade works for all training methods e.g. full fine-tuning, pretraining etc.

🥁Fused QK RoPE Triton Kernel with packing

Back in December 2023, we introduced a RoPE kernel coded up in Triton as part of our Unsloth launch. In March 2024, a community member made end to end training 1-2% faster by optimizing the RoPE kernel to allow launching a block for a group of heads. See .

One issue is for each Q and K, there are 2 Triton kernels. We merged them into 1 Triton kernel now, and enabled variable length RoPE, which was imperative for padding free and packing support. This makes the RoPE kernel in micro benchmarks 2.3x faster on longer context lengths, and 1.9x faster on shorter context lengths.

We also eliminated all clones and contiguous transpose operations, and so RoPE is now fully inplace, reducing further GPU memory. Note for the backward pass, we see that sin1 = -sin1 since:

🚃Int64 Indexing for Triton Kernels

During 500K long context training which we introduced in , we would get CUDA out of bounds errors. This was because MLP kernels for SwiGLU, GeGLU had int32 indexing which is by default in Triton and CUDA.

We can't just do tl.program_id(0).to(tl.int64) since training will be slightly slower due to int64 indexing. We instead make this a LONG_INDEXING: tl.constexpr variable so the Triton compiler can specialize this. This allows shorter and longer context runs to both run great!

♠️Uncontaminated Packing 2-5x faster training

Real datasets can contain different sequence lengths, so increasing the batch size to 32 for example will cause padding, making training slower and use more VRAM.

In the past, increasing batch_size to large numbers (>32) will make training SLOWER, not faster. This was due to padding - we can now eliminate this issue via packing = True, and so training is FASTER!

When we pack multiple samples into a single one-dimensional tensor, we keep sequence length metadata around in order to properly mask samples, without leaking attention between samples. We also need the RoPE kernel described in to allow reset position ids.

🧮Why is padding needed & mathematical speedup

Computers and GPUs cannot process different length datasets, so we have to pad them with 0s. This causes wastage. Assume we have a dataset of 50% short sequences S, and 50% long sequences L, then in the worst case, padding will cause token usage to be since the longest sequence length dominates.

By packing multiple examples into a single, long one-dimensional tensor, we can eliminate a significant amount of padding. In fact we get the below token usage:

By some math and algebra, we can work out the speedup via:

By assuming then we get a 2x theoretical speedup since

By changing the ratio of 50% short sequences, and assuming we have MORE short sequences, for eg 20% long sequences and 80% long sequences, we get so 5x faster training! This means packing's speedup depends on how short rows your dataset has (the more shorter, the faster).

🏖️Analysis and Benchmarks

To demonstrate the various improvements when training with our new kernels and packed data, we ran fine-tuning runs with , Qwen3-8B, Llama 3 8B on the yahma/alpaca-cleaned dataset and measured various throughput and efficiency metrics. We compared our new runs vs. a standard optimized training run with our own kernels/optimizations turned on and kernels like Flash Attention 3 (FA3) enabled. We fixed max_length = 1024 and varied the batch size in {1, 2, 4, 8, 16, 32}. This allows the maximum token count per batch to vary in {1024, 2048, 4096, 8192, 16K, 32K}.

The above shows how tokens per second (tokens/s) training throughput varies for new Unsloth with varying batch size. This translates into training your model on an epoch of your dataset 1.7-3x faster (sometimes even 5x or more)! These gains will be more pronounced if there are many short sequences in your data and if you have longer training runs, as described in

The above shows the average percentage of tokens per batch that are valid (i.e., non-padding). As the batch size length grows, many more padding tokens are seen in the unpacked case, while we achieve a high packing efficiency in the packed case regardless of max sequence length.

Note that, since the batching logic trims batches to the maximum sequence length seen in the batch, when the batch size is 1, the unpacked data is all valid tokens (i.e., no padding). However, as more examples are added into the batch, padding increases on average, hitting nearly 50% padding with batch size is 8! Our sample packing implementation eliminates that waste.

The first graph (above) plots progress on yahma/alpaca-cleaned with max_length = 2048, Unsloth new with packing + kernels (maroon) vs. Unsloth old (gray). Both are trained with max_steps = 500, but we plot the x-axis in wall-clock time. Notice that we train on nearly 40% of an epoch in the packed case in the same amount of steps (and only a bit more wall-clock time) that it takes to train less than 5% of an epoch in the unpacked case.

Similarly, the 2nd graph (above) plots loss from the same runs, this time plotted with training steps on the x-axis. Notice that the losses match in scale and trend, but the loss in the packing case is less variable since the model is seeing more tokens per training step.

🎬Padding-Free by Default

In addition to large throughput gains available when setting packing = True in your SFTConfig , we will automatically use padding-free batching in order to reduce padding waste improve throughput and increases tokens/s throughput, while resulting in the exact same loss as seen in the previous version of Unsloth.

For example for Qwen3-8B and Qwen3-32B, we see memory usage decrease by 60%, be 2x faster, and have the same exact loss and grad norm curves!

✨How to enable packing?

Update Unsloth first and padding free is done by default! So all training is immediately 1.1 to 2x faster with 30% less memory usage at least and 0 change in loss curve metric!

We also support Flash Attention 3 via Xformers, SDPA support, Flash Attention 2, and this works on old GPUs (Tesla T4, RTX 2080) and new GPUs like H100s, B200s etc! Sample packing works regardless of choice of attention backend or model family, so enjoy the same speedups previously had with these fast attention implementations!

Add packing = True to enable up to 5x faster training!

All our notebooks are automatically faster (no need to do anything). See

Qwen3 14B faster:

Llama 3.1 Conversational faster:

Thank you! If you're interested, see our blog, blog and blog for more topics on kernels and performance gains!

Advanced RL Documentation

Advanced documentation settings when using Unsloth with GRPO.

Detailed guides on doing GRPO with Unsloth for Batching, Generation & Training Parameters:

Training Parameters

beta (float, default 0.0): KL coefficient.
- 0.0 ⇒ no reference model loaded (lower memory, faster).
- Higher beta constrains the policy to stay closer to the ref policy.
num_iterations (int, default 1): PPO epochs per batch (μ in the algorithm). Replays data within each gradient accumulation step; e.g., 2 = two forward passes per accumulation step.
epsilon (float, default 0.2): Clipping value for token-level log-prob ratios (typical ratio range ≈ [-1.2, 1.2] with default ε).
delta (float, optional): Enables upper clipping bound for two-sided GRPO when set. If None, standard GRPO clipping is used. Recommended > 1 + ε when enabled (per INTELLECT-2 report).
epsilon_high (float, optional): Upper-bound epsilon; defaults to epsilon if unset. DAPO recommends 0.28.
importance_sampling_level (“token” | “sequence”, default "token"):
- "token": raw per-token ratios (one weight per token).
- "sequence": average per-token ratios to a single sequence-level ratio. GSPO shows sequence-level sampling often gives more stable training for sequence-level rewards.
reward_weights (list[float], optional): One weight per reward. If None, all weights = 1.0.
scale_rewards (str|bool, default "group"):
- True or "group": scale by std within each group (unit variance in group).
- "batch": scale by std across the entire batch
loss_type (str, default "dapo"):
- "grpo": normalizes over sequence length (length bias; not recommended).
- "dr_grpo": normalizes by a global constant (introduced in Dr. GRPO; removes length bias). Constant ≈ max_completion_length
mask_truncated_completions (bool, default False): When True, truncated completions are excluded from loss (recommended by DAPO for stability). Note: There are some KL issues with this flag, so we recommend to disable it.
This can zero out all completion_mask entries when many completions are truncated, making n_mask_per_reward = 0 and causing KL to become NaN.
vllm_importance_sampling_correction (bool, default True): Applies Truncated Importance Sampling (TIS) to correct off-policy effects when generation (e.g., vLLM / fast_inference) differs from training backend. In Unsloth, this is auto-set to True if you’re using vLLM/fast_inference; otherwise False.
vllm_importance_sampling_cap (float, default 2.0): Truncation parameter C for TIS; sets an upper bound on the importance sampling ratio to improve stability.
dtype when choosing float16 or bfloat16, see

Generation Parameters

temperature (float, defaults to 1.0): Temperature for sampling. The higher the temperature, the more random the completions. Make sure you use a relatively high (1.0) temperature to have diversity in generations which helps learning.
top_p (float, optional, defaults to 1.0): Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1.0 to consider all tokens.
top_k (int, optional): Number of highest probability vocabulary tokens to keep for top-k-filtering. If None, top-k-filtering is disabled and all tokens are considered.

It is a bit confusing to mess with this parameter, it is recommended to edit per_device_train_batch_size and gradient accumulation for the batch sizes

Batch & Throughput Parameters

Parameters that control batches

train_batch_size: Number of samples per process per step. If this integer is less than num_generations, it will default to num_generations.
steps_per_generation: Number of microbatches that contribute to one generation’s loss calculation (forward passes only). A new batch of data is generated every steps_per_generation steps; backpropagation timing depends on gradient_accumulation_steps

GRPO Batch Examples

The tables below illustrate how batches flow through steps, when optimizer updates occur, and how new batches are generated.

Example 1

Generation cycle A

Step

Batch

Notes

Generation cycle B

Step

Batch

Notes

Example 2

Generation cycle A

Step

Batch

Notes

Generation cycle B

Step

Batch

Notes

Example 3

Generation cycle A

Step

Batch

Notes

Generation cycle B

Step

Batch

Notes

Example 4

Generation cycle A

Step

Batch

Notes

Generation cycle B

Step

Batch

Notes

Quick Formula Reference

Gemma 3: How to Run Guide

How to run Gemma 3 effectively with our GGUFs on llama.cpp, Ollama, Open WebUI and how to fine-tune with Unsloth!

Google releases Gemma 3 with a new 270M model and the previous 1B, 4B, 12B, and 27B sizes. The 270M and 1B are text-only, while larger models handle both text and vision. We provide GGUFs, and a guide of how to run it effectively, and how to finetune & do RL with Gemma 3!

NEW Aug 14, 2025 Update: Try our fine-tuning Gemma 3 (270M) notebook and GGUFs to run.

Also see our Gemma 3n Guide.

Running TutorialFine-tuning Tutorial

Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training. This means Colab Notebooks with free Tesla T4 GPUs also work!

Fine-tune Gemma 3 (4B) with vision support using our

According to the Gemma team, the optimal config for inference is temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0

Unsloth Gemma 3 uploads with optimal configs:

GGUF

Unsloth Dynamic 4-bit Instruct

16-bit Instruct

⚙️ Recommended Inference Settings

According to the Gemma team, the official recommended settings for inference is:

Temperature of 1.0
Top_K of 64
Min_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
Top_P of 0.95

llama.cpp an other inference engines auto add a <bos> - DO NOT add TWO <bos> tokens! You should ignore the <bos> when prompting the model!

✨Running Gemma 3 on your phone

To run the models on your phone, we recommend using any mobile app that can run GGUFs locally on edge devices like phones. After fine-tuning you can export it to GGUF then run it locally on your phone. Ensure your phone has enough RAM/power to process the models as it can overheat so we recommend using Gemma 3 270M or the Gemma 3n models for this use-case. You can try the mobile app which you can download on or , which are great apps for running GGUFs on your phone.

Remember, you can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8_K_XL' for all the tutorials.

🦙 Tutorial: How to Run Gemma 3 in Ollama

Install ollama if you haven't already!

Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload! You can change the model name 'gemma-3-27b-it-GGUF' to any Gemma model like 'gemma-3-270m-it-GGUF:Q8_K_XL'.

📖 Tutorial: How to Run Gemma 3 27B in llama.cpp

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:Q4_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run

OR download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose Q4_K_M, or other quantized versions (like BF16 full precision). More versions at:

Run Unsloth's Flappy Bird test
Edit --threads 32 for the number of CPU threads, --ctx-size 16384 for context length (Gemma 3 supports 128K context length!), --n-gpu-layers 99 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
For conversation mode:

For non conversation mode to test Flappy Bird:

The full input from our 1.58bit blog is:

Remember to remove <bos> since Gemma 3 auto adds a <bos>!

🦥 Fine-tuning Gemma 3 in Unsloth

Unsloth is the only framework which works in float16 machines for Gemma 3 inference and training. This means Colab Notebooks with free Tesla T4 GPUs also work!

Try our new which makes the 270M parameter model very smart at playing chess and can predict the next chess move.
Fine-tune Gemma 3 (4B) using our notebooks for: or
Or fine-tune with • •

When trying full fine-tune (FFT) Gemma 3, all layers default to float32 on float16 devices. Unsloth expects float16 and upcasts dynamically. To fix, run model.to(torch.float16) after loading, or use a GPU with bfloat16 support.

Unsloth Fine-tuning Fixes

Our solution in Unsloth is 3 fold:

Keep all intermediate activations in bfloat16 format - can be float32, but this uses 2x more VRAM or RAM (via Unsloth's async gradient checkpointing)
Do all matrix multiplies in float16 with tensor cores, but manually upcasting / downcasting without the help of Pytorch's mixed precision autocast.
Upcast all other options that don't need matrix multiplies (layernorms) to float32.

🤔 Gemma 3 Fixes Analysis

First, before we finetune or run Gemma 3, we found that when using float16 mixed precision, gradients and activations become infinity unfortunately. This happens in T4 GPUs, RTX 20x series and V100 GPUs where they only have float16 tensor cores.

For newer GPUs like RTX 30x or higher, A100s, H100s etc, these GPUs have bfloat16 tensor cores, so this problem does not happen! But why?

Float16 can only represent numbers up to 65504, whilst bfloat16 can represent huge numbers up to 10^38! But notice both number formats use only 16bits! This is because float16 allocates more bits so it can represent smaller decimals better, whilst bfloat16 cannot represent fractions well.

But why float16? Let's just use float32! But unfortunately float32 in GPUs is very slow for matrix multiplications - sometimes 4 to 10x slower! So we cannot do this.

Qwen3-VL: How to Run Guide

Learn to fine-tune and run Qwen3-VL locally with Unsloth.

Qwen3-VL is Qwen’s new vision models with instruct and thinking versions. The 2B, 4B, 8B and 32B models are dense, while 30B and 235B are MoE. The 235B thinking LLM delivers SOTA vision and coding performance rivaling GPT-5 (high) and Gemini 2.5 Pro. Qwen3-VL has vision, video and OCR capabilities as well as 256K context (can be extended to 1M). Unsloth supports Qwen3-VL fine-tuning and RL. Train Qwen3-VL (8B) for free with our notebooks.

Running Qwen3-VLFine-tuning Qwen3-VL

🖥️ Running Qwen3-VL

To run the model in llama.cpp, vLLM, Ollama etc., here are the recommended settings:

⚙️ Recommended Settings

Qwen recommends these settings for both models (they're a bit different for Instruct vs Thinking):

Instruct Settings:

Thinking Settings:

Qwen3-VL also used the below settings for their benchmarking numbers, as mentioned .

Instruct Settings:

Thinking Settings:

🐛Chat template bug fixes

At Unsloth, we care about accuracy the most, so we investigated why after the 2nd turn of running the Thinking models, llama.cpp would break, as seen below:

The error code:

We have successfully fixed the Thinking chat template for the VL models so we re-uploaded all Thinking quants and Unsloth's quants. They should now all work after the 2nd conversation - other quants will fail to load after the 2nd conversation.

Qwen3-VL Unsloth uploads:

Qwen3-VL is now supported for GGUFs by llama.cpp as of 30th October 2025, so you can run them locally!

Dynamic GGUFs (to run)

4-bit BnB Unsloth Dynamic

16-bit full-precision

📖 Llama.cpp: Run Qwen3-VL Tutorial

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

Let's first get an image! You can also upload images as well. We shall use , which is just our mini logo showing how finetunes are made with Unsloth:

Let's download this image

Let's get the 2nd image at

Then, let's use llama.cpp's auto model downloading feature, try this for the 8B Instruct model:

Once in, you will see the below screen:

Load up the image via /image PATH ie /image unsloth.png then press ENTER

When you hit ENTER, it'll say "unsloth.png image loaded"

Now let's ask a question like "What is this image?":

Now load in picture 2 via /image picture.png then hit ENTER and ask "What is this image?"

And finally let's ask how are both images are related (it works!)

You can also download the model via (after installing pip install huggingface_hub hf_transfer ) HuggingFace's snapshot_download which is useful for large model downloads, since llama.cpp's auto downloader might lag. You can choose Q4_K_M, or other quantized versions.

Run the model and try any prompt. For Instruct:

For Thinking:

🪄Running Qwen3-VL-235B-A22B and Qwen3-VL-30B-A3B

For Qwen3-VL-235B-A22B, we will use llama.cpp for optimized inference and a plethora of options.

We're following similar steps to above however this time we'll also need to perform extra steps because the model is so big.
Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD-Q2_K_XL, or other quantized versions..
Run the model and try a prompt. Set the correct parameters for Thinking vs. Instruct.

Instruct:

Thinking:

Edit, --ctx-size 16384 for context length, --n-gpu-layers 99 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

Use -ot ".ffn_.*_exps.=CPU" to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

🐋 Docker: Run Qwen3-VL

If you already have Docker desktop, to run Unsloth's models from Hugging Face, run the command below and you're done:

Or you can run Docker's uploaded Qwen3-VL models:

🦥 Fine-tuning Qwen3-VL

Unsloth supports fine-tuning and reinforcement learning (RL) Qwen3-VL including the larger 32B and 235B models. This includes support for fine-tuning for video and object detection. As usual, Unsloth makes Qwen3-VL models train 1.7x faster with 60% less VRAM and 8x longer context lengths with no accuracy degradation. We made two Qwen3-VL (8B) training notebooks which you can train free on Colab:

Saving Qwen3-VL to GGUF now works as llama.cpp just supported it!

If you want to use any other Qwen3-VL model, just change the 8B model to the 2B, 32B etc. one.

The goal of the GRPO notebook is to make a vision language model solve maths problems via RL given an image input like below:

This Qwen3-VL support also integrates our latest update for even more memory efficient + faster RL including our , which uniquely limits speed degradation compared to other implementations. You can read more about how to train vision LLMs with RL with our .

Multi-image training

In order to fine-tune or train Qwen3-VL with multi-images the most straightforward change is to swap

with:

Using map kicks in dataset standardization and arrow processing rules which can be strict and more complicated to define.

GLM-4.6: Run Locally Guide

A guide on how to run Z.ai GLM-4.6 and GLM-4.6V-Flash model on your own local device!

GLM-4.6 and GLM-4.6V-Flash are the latest reasoning models from Z.ai, achieving SOTA performance on coding and agent benchmarks while offering improved conversational chats. GLM-4.6V-Flash the smaller 9B model was released in December, 2025 and you can run it now too.

The full 355B parameter model requires 400GB of disk space, while the Unsloth Dynamic 2-bit GGUF reduces the size to 135GB (-75%). GLM-4.6-GGUF

We did multiple chat template fixes for GLM-4.6 to make llama.cpp/llama-cli --jinja work - please only use --jinja otherwise the output will be wrong!

You asked for benchmarks on our quants, so we’re showcasing Aider Polyglot results! Our Dynamic 3-bit DeepSeek V3.1 GGUF scores 75.6%, surpassing many full-precision SOTA LLMs.

All uploads use Unsloth for SOTA 5-shot MMLU and Aider performance, meaning you can run & fine-tune quantized GLM LLMs with minimal accuracy loss.

Tutorials navigation:

Unsloth Chat Template fixes

One of the significant fixes we did addresses an issue with prompting GGUFs, where the second prompt wouldn’t work. We fixed this issue however, this problem still persists in GGUFs without our fixes. For example, when using any non-Unsloth GLM-4.6 GGUF, the first conversation works fine, but the second one breaks.

We’ve resolved this in our chat template, so when using our version, conversations beyond the second (third, fourth, etc.) work without any errors. There are still some issues with tool-calling, which we haven’t fully investigated yet due to bandwidth limitations. We’ve already informed the GLM team about these remaining issues.

⚙️ Usage Guide

The 2-bit dynamic quant UD-Q2_K_XL uses 135GB of disk space - this works well in a 1x24GB card and 128GB of RAM with MoE offloading. The 1-bit UD-TQ1 GGUF also works natively in Ollama!

You must use --jinja for llama.cpp quants - this uses our and enables the correct template! You might get incorrect results if you do not use --jinja

The 4-bit quants will fit in a 1x 40GB GPU (with MoE layers offloaded to RAM). Expect around 5 tokens/s with this setup if you have bonus 165GB RAM as well. It is recommended to have at least 205GB RAM to run this 4-bit. For optimal performance you will need at least 205GB unified memory or 205GB combined RAM+VRAM for 5+ tokens/s. To learn how to increase generation speed and fit longer contexts, .

Though not a must, for best performance, have your VRAM + RAM combined equal to the size of the quant you're downloading. If not, hard drive / SSD offloading will work with llama.cpp, just inference will be slower.

Recommended Settings

According to Z.ai, there are different settings for GLM-4.6V-Flash & GLM-4.6 inference:

GLM-4.6V-Flash

GLM-4.6

Use --jinja for llama.cpp variants - we fixed some chat template issues as well!

Run GLM-4.6 Tutorials:

See our step-by-step guides for running and the large models.

GLM-4.6V-Flash

Currently GLM-4.6V-Flash only works with text via llama.cpp. Vision support to come later.

✨ Run in llama.cpp

Obtain the latest llama.cpp on . You can also use the build instructions below. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:Q8_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run . Use

GLM-4.6

🦙 Run in Ollama

Install ollama if you haven't already! To run more variants of the model, .

Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload!

✨ Run in llama.cpp

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:Q2_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run . Use

✨ Deploy with llama-server and OpenAI's completion library

To use llama-server for deployment, use the following command:

Then use OpenAI's Python library after pip install openai :

💽Model uploads

ALL our uploads - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

Full GLM-4.6 model uploads below:

We also uploaded and quants which run specifically faster for ARM and Apple devices respectively.

MoE Bits

Type + Link

Disk Size

Details

🏂 Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU" This offloads up and down projection MoE layers.

Try -ot ".ffn_(up)_exps.=CPU" if you have even more GPU memory. This offloads only up projection MoE layers.

Llama.cpp also introduces high throughput mode. Use llama-parallel. Read more about it . You can also quantize the KV cache to 4bits for example to reduce VRAM / RAM movement, which can also make the generation process faster.

📐How to fit long context (full 200K)

--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1

You should use the _1 variants for somewhat increased accuracy, albeit it's slightly slower. For eg q4_1, q5_1

--cache-type-v f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1

Tutorial: How to Finetune Llama-3 and Use In Ollama

Beginner's Guide for creating a customized personal assistant (like ChatGPT) to run locally on Ollama

By the end of this tutorial, you will create a custom chatbot by finetuning Llama-3 with Unsloth for free. It can run locally via Ollama on your PC, or in a free GPU instance through Google Colab. You will be able to interact with the chatbot interactively like below:

Unsloth makes finetuning much easier, and can automatically export the finetuned model to Ollama with integrated automatic Modelfile creation! If you need help, you can join our Discord server: https://discord.com/invite/unsloth

If you’d like to copy or save the code, everything is available in our . You can use it directly there or adapt it for your local setup:

1. What is Unsloth?

makes finetuning LLMs like Llama-3, Mistral, Phi-3 and Gemma 2x faster, use 70% less memory, and with no degradation in accuracy! We will be using Google Colab which provides a free GPU during this tutorial. You can access our free notebooks below:

(notebook which we will be using)

2. What is Ollama?

allows you to run language models from your own computer in a quick and simple way! It quietly launches a program which can run a language model like Llama-3 in the background. If you suddenly want to ask the language model a question, you can simply submit a request to Ollama, and it'll quickly return the results to you! We'll be using Ollama as our inference engine!

3. Install Unsloth

If you have never used a Colab notebook, a quick primer on the notebook itself:

Play Button at each "cell". Click on this to run that cell's code. You must not skip any cells and you must run every cell in chronological order. If you encounter any errors, simply rerun the cell you did not run before. Another option is to click CTRL + ENTER if you don't want to click the play button.
Runtime Button in the top toolbar. You can also use this button and hit "Run all" to run the entire notebook in 1 go. This will skip all the customization steps, and can be a good first try.
Connect / Reconnect T4 button. You can click here for more advanced system statistics.

The first installation cell looks like below: Remember to click the PLAY button in the brackets [ ]. We grab our open source Github package, and install some other packages.

4. Selecting a model to finetune

Let's now select a model for finetuning! We defaulted to Llama-3 from Meta / Facebook which was trained on a whopping 15 trillion "tokens". Assume a token is like 1 English word. That's approximately 350,000 thick Encyclopedias worth! Other popular models include Mistral, Phi-3 (trained using GPT-4 output) and Gemma from Google (13 trillion tokens!).

Unsloth supports these models and more! In fact, simply type a model from the Hugging Face model hub to see if it works! We'll error out if it doesn't work.

There are 3 other settings which you can toggle:

This determines the context length of the model. Gemini for example has over 1 million context length, whilst Llama-3 has 8192 context length. We allow you to select ANY number - but we recommend setting it 2048 for testing purposes. Unsloth also supports very long context finetuning, and we show we can provide 4x longer context lengths than the best.
Keep this as None, but you can select torch.float16 or torch.bfloat16 for newer GPUs.
We do finetuning in 4 bit quantization. This reduces memory usage by 4x, allowing us to actually do finetuning in a free 16GB memory GPU. 4 bit quantization essentially converts weights into a limited set of numbers to reduce memory usage. A drawback of this is there is a 1-2% accuracy degradation. Set this to False on larger GPUs like H100s if you want that tiny extra accuracy.

If you run the cell, you will get some print outs of the Unsloth version, which model you are using, how much memory your GPU has, and some other statistics. Ignore this for now.

5. Parameters for finetuning

Now to customize your finetune, you can edit the numbers above, but you can ignore it, since we already select quite reasonable numbers.

The goal is to change these numbers to increase accuracy, but also counteract over-fitting. Over-fitting is when you make the language model memorize a dataset, and not be able to answer novel new questions. We want to a final model to answer unseen questions, and not do memorization.

The rank of the finetuning process. A larger number uses more memory and will be slower, but can increase accuracy on harder tasks. We normally suggest numbers like 8 (for fast finetunes), and up to 128. Too large numbers can causing over-fitting, damaging your model's quality.
We select all modules to finetune. You can remove some to reduce memory usage and make training faster, but we highly do not suggest this. Just train on all modules!
The scaling factor for finetuning. A larger number will make the finetune learn more about your dataset, but can promote over-fitting. We suggest this to equal to the rank r, or double it.

6. Alpaca Dataset

We will now use the Alpaca Dataset created by calling GPT-4 itself. It is a list of 52,000 instructions and outputs which was very popular when Llama-1 was released, since it made finetuning a base LLM be competitive with ChatGPT itself.

You can access the GPT4 version of the Alpaca dataset here: . An older first version of the dataset is here: . Below shows some examples of the dataset:

You can see there are 3 columns in each row - an instruction, and input and an output. We essentially combine each row into 1 large prompt like below. We then use this to finetune the language model, and this made it very similar to ChatGPT. We call this process supervised instruction finetuning.

7. Multiple columns for finetuning

But a big issue is for ChatGPT style assistants, we only allow 1 instruction / 1 prompt, and not multiple columns / inputs. For example in ChatGPT, you can see we must submit 1 prompt, and not multiple prompts.

This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function!

For example the very famous Titanic dataset has many many columns. Your job was to predict whether a passenger has survived or died based on their age, passenger class, fare price etc. We can't simply pass this into ChatGPT, but rather, we have to "merge" this information into 1 large prompt.

For example, if we ask ChatGPT with our "merged" single prompt which includes all the information for that passenger, we can then ask it to guess or predict whether the passenger has died or survived.

Other finetuning libraries require you to manually prepare your dataset for finetuning, by merging all your columns into 1 prompt. In Unsloth, we simply provide the function called to_sharegpt which does this in 1 go!

To access the Titanic finetuning notebook or if you want to upload a CSV or Excel file, go here:

Now this is a bit more complicated, since we allow a lot of customization, but there are a few points:

You must enclose all columns in curly braces {}. These are the column names in the actual CSV / Excel file.
Optional text components must be enclosed in [[]]. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values.
Select the output or target / prediction column in output_column_name. For the Alpaca dataset, this will be output

For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional.

For example, pretend the dataset looks like this with a lot of missing data:

Embarked

Age

Fare

Then, we do not want the result to be:

The passenger embarked from S. Their age is 23. Their fare is EMPTY.
The passenger embarked from EMPTY. Their age is 18. Their fare is $7.25.

Instead by optionally enclosing columns using [[]], we can exclude this information entirely.

[[The passenger embarked from S.]] [[Their age is 23.]] [[Their fare is EMPTY.]]
[[The passenger embarked from EMPTY.]] [[Their age is 18.]] [[Their fare is $7.25.]]

becomes:

The passenger embarked from S. Their age is 23.
Their age is 18. Their fare is $7.25.

8. Multi turn conversations

A bit issue if you didn't notice is the Alpaca dataset is single turn, whilst remember using ChatGPT was interactive and you can talk to it in multiple turns. For example, the left is what we want, but the right which is the Alpaca dataset only provides singular conversations. We want the finetuned language model to somehow learn how to do multi turn conversations just like ChatGPT.

So we introduced the conversation_extension parameter, which essentially selects some random rows in your single turn dataset, and merges them into 1 conversation! For example, if you set it to 3, we randomly select 3 rows and merge them into 1! Setting them too long can make training slower, but could make your chatbot and final finetune much better!

Then set output_column_name to the prediction / output column. For the Alpaca dataset dataset, it would be the output column.

We then use the standardize_sharegpt function to just make the dataset in a correct format for finetuning! Always call this!

9. Customizable Chat Templates

We can now specify the chat template for finetuning itself. The very famous Alpaca format is below:

But remember we said this was a bad idea because ChatGPT style finetunes require only 1 prompt? Since we successfully merged all dataset columns into 1 using Unsloth, we essentially can create the below style chat template with 1 input column (instruction) and 1 output:

We just require you must put a {INPUT} field for the instruction and an {OUTPUT} field for the model's output field. We in fact allow an optional {SYSTEM} field as well which is useful to customize a system prompt just like in ChatGPT. For example, below are some cool examples which you can customize the chat template to be:

For the ChatML format used in OpenAI models:

Or you can use the Llama-3 template itself (which only functions by using the instruct version of Llama-3): We in fact allow an optional {SYSTEM} field as well which is useful to customize a system prompt just like in ChatGPT.

Or in the Titanic prediction task where you had to predict if a passenger died or survived in this Colab notebook which includes CSV and Excel uploading:

10. Train the model

Let's train the model now! We normally suggest people to not edit the below, unless if you want to finetune for longer steps or want to train on large batch sizes.

We do not normally suggest changing the parameters above, but to elaborate on some of them:

Increase the batch size if you want to utilize the memory of your GPU more. Also increase this to make training more smooth and make the process not over-fit. We normally do not suggest this, since this might make training actually slower due to padding issues. We normally instead ask you to increase gradient_accumulation_steps which just does more passes over the dataset.
Equivalent to increasing the batch size above itself, but does not impact memory consumption! We normally suggest people increasing this if you want smoother training loss curves.
We set steps to 60 for faster training. For full training runs which can take hours, instead comment out max_steps, and replace it with num_train_epochs = 1

You’ll see a log of numbers during training. This is the training loss, which shows how well the model is learning from your dataset. For many cases, a loss around 0.5 to 1.0 is a good sign, but it depends on your dataset and task. If the loss is not going down, you might need to adjust your settings. If the loss goes to 0, that could mean overfitting, so it's important to check validation too.

11. Inference / running the model

Now let's run the model after we completed the training process! You can edit the yellow underlined part! In fact, because we created a multi turn chatbot, we can now also call the model as if it saw some conversations in the past like below:

Reminder Unsloth itself provides 2x faster inference natively as well, so always do not forget to call FastLanguageModel.for_inference(model). If you want the model to output longer responses, set max_new_tokens = 128 to some larger number like 256 or 1024. Notice you will have to wait longer for the result as well!

12. Saving the model

We can now save the finetuned model as a small 100MB file called a LoRA adapter like below. You can instead push to the Hugging Face hub as well if you want to upload your model! Remember to get a Hugging Face token via and add your token!

After saving the model, we can again use Unsloth to run the model itself! Use FastLanguageModel again to call it for inference!

13. Exporting to Ollama

Finally we can export our finetuned model to Ollama itself! First we have to install Ollama in the Colab notebook:

Then we export the finetuned model we have to llama.cpp's GGUF formats like below:

Reminder to convert False to True for 1 row, and not change every row to True, or else you'll be waiting for a very time! We normally suggest the first row getting set to True, so we can export the finetuned model quickly to Q8_0 format (8 bit quantization). We also allow you to export to a whole list of quantization methods as well, with a popular one being q4_k_m.

Head over to to learn more about GGUF. We also have some manual instructions of how to export to GGUF if you want here:

You will see a long list of text like below - please wait 5 to 10 minutes!!

And finally at the very end, it'll look like below:

Then, we have to run Ollama itself in the background. We use subprocess because Colab doesn't like asynchronous calls, but normally one just runs ollama serve in the terminal / command prompt.

14. Automatic `Modelfile` creation

The trick Unsloth provides is we automatically create a Modelfile which Ollama requires! This is a just a list of settings and includes the chat template which we used for the finetune process! You can also print the Modelfile generated like below:

We then ask Ollama to create a model which is Ollama compatible, by using the Modelfile

15. Ollama Inference

And we can now call the model for inference if you want to do call the Ollama server itself which is running on your own local machine / in the free Colab notebook in the background. Remember you can edit the yellow underlined part.

16. Interactive ChatGPT style

But to actually run the finetuned model like a ChatGPT, we have to do a bit more! First click the terminal icon and a Terminal will pop up. It's on the left sidebar.

Then, you might have to press ENTER twice to remove some weird output in the Terminal window. Wait a few seconds and type ollama run unsloth_model then hit ENTER.

And finally, you can interact with the finetuned model just like an actual ChatGPT! Hit CTRL + D to exit the system, and hit ENTER to converse with the chatbot!

You've done it!

You've successfully finetuned a language model and exported it to Ollama with Unsloth 2x faster and with 70% less VRAM! And all this for free in a Google Colab notebook!

If you want to learn how to do reward modelling, do continued pretraining, export to vLLM or GGUF, do text completion, or learn more about finetuning tips and tricks, head over to our .

If you need any help on finetuning, you can also join our Discord server . If you want help with Ollama, you can also join their server .

And finally, we want to thank you for reading and following this far! We hope this made you understand some of the nuts and bolts behind finetuning language models, and we hope this was useful!

To access our Alpaca dataset example click , and our CSV / Excel finetuning guide is .

LoRA Hyperparameters Guide

Optimal lora rank. alpha, number of epochs, batch size & gradient accumulation, QLoRA vs LoRA, target modules and more!

LoRA hyperparameters are adjustable parameters that control how Low-Rank Adaptation (LoRA) fine-tunes LLMs. With many options (such as learning rate and epochs) and millions of possible combinations, selecting the right values is crucial for achieving accuracy, stability, quality, and fewer hallucinations during fine-tuning.

You'll learn the best practices for these parameters, based on insights from hundreds of research papers and experiments, and see how they impact the model. While we recommend using Unsloth's defaults, understanding these concepts will give you full control. The goal is to change hyperparameter numbers to increase accuracy while counteracting overfitting or underfitting. Overfitting occurs when the model memorizes the training data, harming its ability to generalize to new, unseen inputs. The objective is a model that generalizes well, not one that simply memorizes.

❓But what is LoRA?

In LLMs, we have model weights. Llama 70B has 70 billion numbers. Instead of changing all 70b numbers, we instead add thin matrices A and B to each weight, and optimize those. This means we only optimize 1% of weights.

🔢 Key Fine-tuning Hyperparameters

Learning Rate

Defines how much the model’s weights are adjusted during each training step.

Higher Learning Rates: Lead to faster initial convergence but can cause training to become unstable or fail to find an optimal minimum if set too high.
Lower Learning Rates: Result in more stable and precise training but may require more epochs to converge, increasing overall training time. While low learning rates are often thought to cause underfitting, they actually can lead to overfitting or even prevent the model from learning.
Typical Range: 2e-4 (0.0002) to 5e-6 (0.000005). 🟩

Epochs

The number of times the model sees the full training dataset.

More Epochs: Can help the model learn better, but a high number can cause it to memorize the training data, hurting its performance on new tasks.
Fewer Epochs: Reduces training time and can prevent overfitting, but may result in an undertrained model if the number is insufficient for the model to learn the dataset's underlying patterns.
Recommended: 1-3 epochs. For most instruction-based datasets, training for more than 3 epochs offers diminishing returns and increases the risk of overfitting.

LoRA or QLoRA

LoRA uses 16-bit precision, while QLoRA is a 4-bit fine-tuning method.

LoRA: 16-bit fine-tuning. It's slightly faster and slightly more accurate, but consumes significantly more VRAM (4× more than QLoRA). Recommended for 16-bit environments and scenarios where maximum accuracy is required.
QLoRA: 4-bit fine-tuning. Slightly slower and marginally less accurate, but uses much less VRAM (4× less). 🦥 70B LLaMA fits in <48GB VRAM with QLoRA in Unsloth - .

Hyperparameters & Recommendations:

Hyperparameter

Function

Recommended Settings

🌳 Gradient Accumulation and Batch Size equivalency

Effective Batch Size

Correctly configuring your batch size is critical for balancing training stability with your GPU's VRAM limitations. This is managed by two parameters whose product is the Effective Batch Size. Effective Batch Size = batch_size * gradient_accumulation_steps

A larger Effective Batch Size generally leads to smoother, more stable training.
A smaller Effective Batch Size may introduce more variance.

While every task is different, the following configuration provides a great starting point for achieving a stable Effective Batch Size of 16, which works well for most fine-tuning tasks on modern GPUs.

Parameter

Description

Recommended Setting

The VRAM & Performance Trade-off

Assume you want 32 samples of data per training step. Then you can use any of the following configurations:

batch_size = 32, gradient_accumulation_steps = 1
batch_size = 16, gradient_accumulation_steps = 2
batch_size = 8, gradient_accumulation_steps = 4

While all of these are equivalent for the model's weight updates, they have vastly different hardware requirements.

The first configuration (batch_size = 32) uses the most VRAM and will likely fail on most GPUs. The last configuration (batch_size = 1) uses the least VRAM, but at the cost of slightly slower training. To avoid OOM (out of memory) errors, always prefer to set a smaller batch_size and increase gradient_accumulation_steps to reach your target Effective Batch Size.

🦥 Unsloth Gradient Accumulation Fix

Gradient accumulation and batch sizes are now fully equivalent in Unsloth due to our bug fixes for gradient accumulation. We have implemented specific bug fixes for gradient accumulation that resolve a common issue where the two methods did not produce the same results. This was a known challenge in the wider community, but for Unsloth users, the two methods are now interchangeable.

for more details.

Prior to our fixes, combinations of batch_size and gradient_accumulation_steps that yielded the same Effective Batch Size (i.e., batch_size × gradient_accumulation_steps = 16) did not result in equivalent training behavior. For example, configurations like b1/g16, b2/g8, b4/g4, b8/g2, and b16/g1 all have an Effective Batch Size of 16, but as shown in the graph, the loss curves did not align when using standard gradient accumulation:

After applying our fixes, the loss curves now align correctly, regardless of how the Effective Batch Size of 16 is achieved:

🦥 LoRA Hyperparameters in Unsloth

The following demonstrates a standard configuration. While Unsloth provides optimized defaults, understanding these parameters is key to manual tuning.

The rank (r) of the fine-tuning process. A larger rank uses more memory and will be slower, but can increase accuracy on complex tasks. We suggest ranks like 8 or 16 (for fast fine-tunes) and up to 128. Using a rank that is too large can cause overfitting and harm your model's quality.\
For optimal performance, LoRA should be applied to all major linear layers. that targeting all major layers is crucial for matching the performance of full fine-tuning. While it's possible to remove modules to reduce memory usage, we strongly advise against it to preserve maximum quality as the savings are minimal.\
A scaling factor that controls the strength of the fine-tuned adjustments. Setting it equal to the rank (r

Verifying LoRA Weight Updates:

When validating that LoRA adapter weights have been updated after fine-tuning, avoid using np.allclose() for comparison. This method can miss subtle but meaningful changes, particularly in LoRA A, which is initialized with small Gaussian values. These changes may not register as significant under loose numerical tolerances. Thanks to for this section.

To reliably confirm weight updates, we recommend:

Using checksum or hash comparisons (e.g., MD5)
Computing the sum of absolute differences between tensors
Inspecting tensor statistics (e.g., mean, variance) manually
Or using np.array_equal() if exact equality is expected

📐LoRA Alpha and Rank relationship

It's best to set lora_alpha = 2 * lora_rank or lora_alpha = lora_rank

The formula for LoRA is on the left. We need to scale the thin matrices A and B by alpha divided by the rank. This means we should keep alpha/rank at least = 1.

According to the , we should instead scale alpha by the sqrt of the rank. Other options exist, but theoretically this is the optimum. The left plot shows other ranks and their perplexities (lower is better). To enable this, set use_rslora = True in Unsloth.

Our recommendation is to set the alpha to equal to the rank, or at least 2 times the rank. This means alpha/rank = 1 or 2.

🎯 LoRA Target Modules and QLoRA vs LoRA

Use: target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",] to target both MLP and attention layers to increase accuracy.

QLoRA uses 4-bit precision, reducing VRAM usage by over 75%.

LoRA (16-bit) is slightly more accurate and faster.

According to empirical experiments and research papers like the original , it's best to apply LoRA to both attention and MLP layers.

The chart shows RougeL scores (higher is better) for different target module configurations, comparing LoRA vs QLoRA.

The first 3 dots show:

QLoRA-All: LoRA applied to all FFN/MLP and Attention layers. 🔥 This performs best overall.
QLoRA-FFN

😎 Training on completions only, masking out inputs

The shows that masking out inputs and training only on completions (outputs or assistant messages) can further increase accuracy by a few percentage points (1%). Below demonstrates how this is done in Unsloth:

NOT training on completions only:

USER: Hello what is 2+2? ASSISTANT: The answer is 4. USER: Hello what is 3+3? ASSISTANT: The answer is 6.

Training on completions only:

USER: ~~Hello what is 2+2?~~ ASSISTANT: The answer is 4. USER: ~~Hello what is 3+3?~~ ASSISTANT: The answer is 6.

The QLoRA paper states that training on completions only increases accuracy by quite a bit, especially for multi-turn conversational finetunes! We do this in our .

To enable training on completions in Unsloth, you will need to define the instruction and assistant parts. 🦥 We plan to further automate this for you in the future!

For Llama 3, 3.1, 3.2, 3.3 and 4 models, you define the parts as follows:

For Gemma 2, 3, 3n models, you define the parts as follows:

🔑 Avoiding Overfitting & Underfitting

Overfitting (Poor Generalization/Too Specialized)

The model memorizes the training data, including its statistical noise, and consequently fails to generalize to unseen data.

If your training loss drops below 0.2, your model is likely overfitting — meaning it may perform poorly on unseen tasks.

One simple trick is LoRA alpha scaling — just multiply the alpha value of each LoRA matrix by 0.5. This effectively scales down the impact of fine-tuning.

This is closely related to merging / averaging weights. You can take the original base (or instruct) model, add the LoRA weights, then divide the result by 2. This gives you an averaged model — which is functionally equivalent to reducing the alpha by half.

Solution:

Adjust the learning rate: A high learning rate often leads to overfitting, especially during short training runs. For longer training, a higher learning rate may work better. It’s best to experiment with both to see which performs best.
Reduce the number of training epochs. Stop training after 1, 2, or 3 epochs.
Increase weight_decay. A value of 0.01 or 0.1 is a good starting point.

Underfitting (Too Generic)

The model fails to capture the underlying patterns in the training data, often due to insufficient complexity or training duration.

Solution:

Adjust the Learning Rate: If the current rate is too low, increasing it may speed up convergence, especially for short training runs. For longer runs, try lowering the learning rate instead. Test both approaches to see which works best.
Increase Training Epochs: Train for more epochs, but monitor validation loss to avoid overfitting.
Increase LoRA Rank (r) and alpha: Rank should at least equal to the alpha number, and rank should be bigger for smaller models/more complex datasets; it usually is between 4 and 64.

Fine-tuning has no single "best" approach, only best practices. Experimentation is key to finding what works for your specific needs. Our notebooks automatically set optimal parameters based on many papers research and our experiments, giving you a great starting point. Happy fine-tuning!

Acknowledgements: A huge thank you to for contributing to this guide!

Gemma 3n: How to Run & Fine-tune

Run Google's new Gemma 3n locally with Dynamic GGUFs on llama.cpp, Ollama, Open WebUI and fine-tune with Unsloth!

Google’s Gemma 3n multimodal model handles image, audio, video, and text inputs. Available in 2B and 4B sizes, it supports 140 languages for text and multimodal tasks. You can now run and fine-tune Gemma-3n-E4B and E2B locally using Unsloth.

Fine-tune Gemma 3n with our free Colab notebook

Gemma 3n has 32K context length, 30s audio input, OCR, auto speech recognition (ASR), and speech translation via prompts.

Running TutorialFine-tuning TutorialFixes + Technical Analysis

Unsloth Gemma 3n (Instruct) uploads with optimal configs:

Dynamic 2.0 GGUF (text only)

Dynamic 4-bit Instruct (to fine-tune)

16-bit Instruct

See all our Gemma 3n uploads including base and more formats in .

🖥️ Running Gemma 3n

Currently Gemma 3n is only supported in text format for inference.

We’ve with GGUFs not working properly in Ollama only. Please redownload if using Ollama.

⚙️ Official Recommended Settings

According to the Gemma team, the official recommended settings for inference:

temperature = 1.0, top_k = 64, top_p = 0.95, min_p = 0.0

Temperature of 1.0
Top_K of 64
Min_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
Top_P of 0.95

llama.cpp an other inference engines auto add a <bos> - DO NOT add TWO <bos> tokens! You should ignore the <bos> when prompting the model!

🦙 Tutorial: How to Run Gemma 3n in Ollama

Please re download Gemma 3N quants or remove the old ones via Ollama since there are some bug fixes. You can do the below to delete the old file and refresh it:

Install ollama if you haven't already!

Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload!

📖 Tutorial: How to Run Gemma 3n in llama.cpp

We would first like to thank from Hugging Face, from the llama.cpp team on making Gemma 3N work in llama.cpp!

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:Q4_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run

OR download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose Q4_K_M, or other quantized versions (like BF16 full precision).

Run the model.
Edit --threads 32 for the number of CPU threads, --ctx-size 32768 for context length (Gemma 3 supports 32K context length!), --n-gpu-layers 99 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.
For conversation mode:

For non conversation mode to test Flappy Bird:

Remember to remove <bos> since Gemma 3N auto adds a <bos>!

🦥 Fine-tuning Gemma 3n with Unsloth

Gemma 3n, like , had issues running on Flotat16 GPUs such as Tesla T4s in Colab. You will encounter NaNs and infinities if you do not patch Gemma 3n for inference or finetuning. .

Fine-tune Gemma 3n-E4B with our
Audio: Fine-tune Gemma 3n-E4B with our
Vision: Fine-tune Gemma 3n-E4B with our

We also found that because Gemma 3n's unique architecture reuses hidden states in the vision encoder it poses another interesting quirk with

Unsloth is the only framework which works in float16 machines for Gemma 3n inference and training. This means Colab Notebooks with free Tesla T4 GPUs also work! Overall, Unsloth makes Gemma 3n training 1.5x faster, 50% less VRAM and 4x longer context lengths.

Our free Gemma 3n Colab notebooks default to fine-tuning text layers. If you want to fine-tune vision or audio layers too, be aware this will require much more VRAM - beyond the 15GB free Colab or Kaggle provides. You can still fine-tune all layers including audio and vision and Unsloth also lets you fine-tune only specific areas, like just vision. Simply adjust as needed:

🏆Bonus Content

We also heard you guys wanted a Vision notebook for Gemma 3 (4B) so here it is:

Fine-tune Gemma 3 (4B) with Vision support using our

If you love Kaggle, Google is holding a competition where the best model fine-tuned with Gemma 3n and Unsloth will win a $10K prize! .

🐛Fixes for Gemma 3n

✨GGUF issues & fixes

Thanks to discussions from from the Ollama team and also from Hugging Face, there were 2 issues we had to fix specifically for GGUFs:

The add_shared_kv_layers parameter was accidentally encoded in float32 which is fine, but becomes slightly complicated to decode on Ollama's side - a simple change to uint32 solves the issue. addressing this issue.
The per_layer_token_embd layer should be Q8_0 in precision. Anything lower does not function properly and errors out in the Ollama engine - to reduce issues for our community, we made this all Q8_0 in all quants - unfortunately this does use more space.

♾️Infinities and NaN gradients and activations

Gemma 3n just like Gemma 3 has issues on FP16 GPUs (e.g., Tesla T4s in Colab).

Our previous fixes for Gemma 3 is . For Gemma 3, we found that activations exceed float16's maximum range of 65504.

Gemma 3N does not have this activation issue, but we still managed to encounter infinities!

To get to the bottom of these infinities, we plotted the absolute maximum weight entries for Gemma 3N, and we see the below:

We find that the green crosses are the Conv2D convolutional weights. We can see that the magnitude of Conv2D layers is much larger on average.

Below is a table for Conv2D weights which have large magnitudes. Our hypothesis is that during a Conv2D operation, large weights multiply and sum together, and unfortunately by chance exceed float16's maximum range of 65504. Bfloat16 is fine, since it's maximum range is 10^38.

Name

Max

🎇Solution to infinities

The naive solution is to upcast all Conv2D weights to float32 (if bfloat16 isn't available). But that would increase VRAM usage. To tackle this, we instead make use of autocast on the fly to upcast the weights and inputs to float32, and so we perform the accumulation in float32 as part of the matrix multiplication itself, without having to upcast the weights.

Unsloth is the only framework that enables Gemma 3n inference and training on float16 GPUs, so Colab Notebooks with free Tesla T4s work!

🏁Gradient Checkpointing issues

We found Gemma 3N's vision encoder to be quite unique as well since it re-uses hidden states. This unfortunately limits the usage of , which could have reduced VRAM usage significantly. since it cannot be applied to Vision encoder.

However, we still managed to leverage Unsloth's automatic compiler to optimize Gemma 3N!

🌵Large losses during finetuning

We also found losses are interestingly very large during the start of finetuning - in the range of 6 to 7, but they do decrease over time quickly. We theorize this is either because of 2 possibilities:

There might be some implementation issue, but this is unlikely since inference seems to work.
Multi-modal models always seem to exhibit this behavior - we found Llama 3.2 Vision's loss starts at 3 or 4, Pixtral at 8 or so, and Qwen 2.5 VL also 4 ish. Because Gemma 3N includes audio as well, it might amplify the starting loss. But this is just a hypothesis. We also found quantizing Qwen 2.5 VL 72B Instruct to have extremely high perplexity scores of around 30 or so, but the model interestingly performs fine.

Fine-tune Gemma 3n with our

🛠️ Technical Analysis

Gemma 3n : MatFormer

So what is so special about Gemma 3n you ask? It is based on architecture meaning that each transformer layer/block embeds/nests FFNs of progressively smaller sizes. Think of it like progressively smaller cups put inside one another. The training is done so that at inference time you can choose the size you want and get the most of the performance of the bigger models.

There is also Per Layer Embedding which can be cached to reduce memory usage at inference time. So the 2B model (E2B) is a sub-network inside the 4B (aka 5.44B) model that is achieved by both Per Layer Embedding caching and skipping audio and vision components focusing solely on text.

The MatFormer architecture, typically is trained with exponentially spaced sub-models aka of sizes S, S/2, S/4, S/8 etc in each of the layers. So at training time, inputs are randomly forwarded through one of the said sub blocks giving every sub block equal chance to learn. Now the advantage is, at inference time, if you want the model to be 1/4th of the original size, you can pick S/4 sized sub blocks in each layer.

You can also choose to Mix and Match where you pick say, S/4 sized sub block of one layer, S/2 sized sub block of another layer and S/8 sized sub block of another layer. In fact, you can change the sub models you pick based on the input itself if you fancy so. Basically its like choose your own kind of structure at every layer. So by just training a model of one particular size, you are creating exponentially many models of smaller sizes. No learning goes waste. Pretty neat huh.

Fine-tune and try multimodal Gemma 3n inference with our

DeepSeek-R1-0528: How to Run Locally

A guide on how to run DeepSeek-R1-0528 including Qwen3 on your own local device!

DeepSeek-R1-0528 is DeepSeek's new update to their R1 reasoning model. The full 671B parameter model requires 715GB of disk space. The quantized dynamic 1.66-bit version uses 162GB (-80% reduction in size). GGUF: DeepSeek-R1-0528-GGUF

DeepSeek also released a R1-0528 distilled version by fine-tuning Qwen3 (8B). The distill achieves similar performance to Qwen3 (235B). You can also fine-tune Qwen3 Distill with Unsloth. Qwen3 GGUF: DeepSeek-R1-0528-Qwen3-8B-GGUF

All uploads use Unsloth Dynamic 2.0 for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized DeepSeek LLMs with minimal accuracy loss.

Tutorials navigation:

Run in llama.cppRun in Ollama/Open WebUIFine-tuning R1-0528

NEW: Huge improvements to tool calling and chat template fixes. New - 162GB in size. Ideal for 192GB RAM (including Mac) and Ollama users. Try: ollama run hf.co/unsloth/DeepSeek-R1-0528-GGUF:TQ1_0

⚙️ Recommended Settings

For DeepSeek-R1-0528-Qwen3-8B, the model can pretty much fit in any setup, and even those with as less as 20GB RAM. There is no need for any prep beforehand. However, for the full R1-0528 model which is 715GB in size, you will need extra prep. The 1.78-bit (IQ1_S) quant will fit in a 1x 24GB GPU (with all layers offloaded). Expect around 5 tokens/s with this setup if you have bonus 128GB RAM as well.

It is recommended to have at least 64GB RAM to run this quant (you will get 1 token/s without a GPU). For optimal performance you will need at least 180GB unified memory or 180GB combined RAM+VRAM for 5+ tokens/s.

We suggest using our 2.7bit (Q2_K_XL) or 2.4bit (IQ2_XXS) quant to balance size and accuracy! The 2.4bit one also works well.

Though not necessary, for the best performance, have your VRAM + RAM combined = to the size of the quant you're downloading.

🐳 Official Recommended Settings:

According to , these are the recommended settings for R1 (R1-0528 and Qwen3 distill should use the same settings) inference:

Set the temperature 0.6 to reduce repetition and incoherence.
Set top_p to 0.95 (recommended)
Run multiple tests and average results for reliable evaluation.

🔢 Chat template/prompt format

R1-0528 uses the same chat template as the original R1 model. You do not need to force <think>\n , but you can still add it in!

A BOS is forcibly added, and an EOS separates each interaction. To counteract double BOS tokens during inference, you should only call tokenizer.encode(..., add_special_tokens = False) since the chat template auto adds a BOS token as well. For llama.cpp / GGUF inference, you should skip the BOS since it’ll auto add it:

The <think> and </think> tokens get their own designated tokens.

Model uploads

ALL our uploads - including those that are not imatrix-based or dynamic, utilize our calibration dataset, which is specifically optimized for conversational, coding, and language tasks.

Qwen3 (8B) distill:
Full DeepSeek-R1-0528 model uploads below:

We also uploaded and quants which run specifically faster for ARM and Apple devices respectively.

MoE Bits

Type + Link

Disk Size

Details

We've also uploaded versions in , and original .

Run DeepSeek-R1-0528 Tutorials:

🦙 Run in Ollama/Open WebUI

Install ollama if you haven't already! You can only run models up to 32B in size. To run the full 720GB R1-0528 model, .

Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload!

(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1_0 (162GB quant):

🦙 Run Full R1-0528 on Ollama/Open WebUI

Open WebUI has made an step-by-step tutorial on how to run R1 here and for R1-0528, you will just need to replace R1 with the new 0528 quant:

(NEW) To run the full R1-0528 model in Ollama, you can use our TQ1_0 (162GB quant):

If you want to use any of the quants that are larger than TQ1_0 (162GB) on Ollama, you need to first merge the 3 GGUF split files into 1 like the code below. Then you will need to run the model locally.

✨ Run Qwen3 distilled R1 in llama.cpp

To run the full 720GB R1-0528 model, . Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

Then use llama.cpp directly to download the model:

✨ Run Full R1-0528 on llama.cpp

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:IQ1_S) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run . Use export LLAMA_CACHE="folder" to force llama.cpp to save to a specific location.

Please try out -ot ".ffn_.*_exps.=CPU" to offload all MoE layers to the CPU! This effectively allows you to fit all non MoE layers on 1 GPU, improving generation speeds. You can customize the regex expression to fit more layers if you have more GPU capacity.

If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU" This offloads up and down projection MoE layers.

Try -ot ".ffn_(up)_exps.=CPU" if you have even more GPU memory. This offloads only up projection MoE layers.

Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD-IQ1_S(dynamic 1.78bit quant) or other quantized versions like Q4_K_M . We recommend using our 2.7bit dynamic quant UD-Q2_K_XL to balance size and accuracy. More versions at:

Run Unsloth's Flappy Bird test as described in our 1.58bit Dynamic Quant for DeepSeek R1.
Edit --threads 32 for the number of CPU threads, --ctx-size 16384 for context length, --n-gpu-layers 2 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference.

🎱 Heptagon Test

You can also test our dynamic quants via which tests the model on creating a basic physics engine to simulate balls rotating in a moving enclosed heptagon shape.

Full prompt to run the model

🦥 Fine-tuning DeepSeek-R1-0528 with Unsloth

To fine-tune DeepSeek-R1-0528-Qwen3-8B using Unsloth, we’ve made a new GRPO notebook featuring a custom reward function designed to significantly enhance multilingual output - specifically increasing the rate of desired language responses (in our example we use Indonesian but you can use any) by more than 40%.

- new

While many reasoning LLMs have multilingual capabilities, they often produce mixed-language outputs in its reasoning traces, combining English with the target language. Our reward function effectively mitigates this issue by strongly encouraging outputs in the desired language, leading to a substantial improvement in language consistency.

This reward function is also fully customizable, allowing you to adapt it for other languages or fine-tune for specific domains or use cases.

The best part about this whole reward function and notebook is you DO NOT need a language dataset to force your model to learn a specific language. The notebook has no Indonesian dataset.

Unsloth makes R1-Qwen3 distill fine-tuning 2× faster, uses 70% less VRAM, and support 8× longer context lengths.

Datasets Guide

Learn how to create & prepare a dataset for fine-tuning.

What is a Dataset?

For LLMs, datasets are collections of data that can be used to train our models. In order to be useful for training, text data needs to be in a format that can be tokenized. You'll also learn how to use datasets inside of Unsloth.

One of the key parts of creating a dataset is your chat template and how you are going to design it. Tokenization is also important as it breaks text into tokens, which can be words, sub-words, or characters so LLMs can process it effectively. These tokens are then turned into embeddings and are adjusted to help the model understand the meaning and context.

Data Format

To enable the process of tokenization, datasets need to be in a format that can be read by a tokenizer.

Format

Description

Training Type

It's worth noting that different styles of format exist for each of these types.

Getting Started

Before we format our data, we want to identify the following:

Purpose of dataset

Knowing the purpose of the dataset will help us determine what data we need and format to use.

The purpose could be, adapting a model to a new task such as summarization or improving a model's ability to role-play a specific character. For example:

Chat-based dialogues (Q&A, learn a new language, customer support, conversations).
Structured tasks (, summarization, generation tasks).

One of the best ways to create a better dataset is by combining it with a more generalized dataset from Hugging Face like ShareGPT to make your model smarter and diverse. You could also add .

Formatting the Data

When we have identified the relevant criteria, and collected the necessary data, we can then format our data into a machine readable format that is ready for training.

Common Data Formats for LLM Training

For , we use raw text format without specific structure:

This format preserves natural language flow and allows the model to learn from continuous text.

If we are adapting a model to a new task, and intend for the model to output text in a single turn based on a specific set of instructions, we can use Instruction format in

When we want multiple turns of conversation we can use the ShareGPT format:

The template format uses the "from"/"value" attribute keys and messages alternates between humanand gpt, allowing for natural dialogue flow.

The other common format is OpenAI's ChatML format and is what Hugging Face defaults to. This is probably the most used format, and alternates between user and assistant

Applying Chat Templates with Unsloth

For datasets that usually follow the common chatml format, the process of preparing the dataset for training or finetuning, consists of four simple steps:

Check the chat templates that Unsloth currently supports:\
This will print out the list of templates currently supported by Unsloth. Here is an example output:\
\
Use get_chat_template to apply the right chat template to your tokenizer:\
\
Define your formatting function. Here's an example:\

Formatting Data Q&A

Q: How can I use the Alpaca instruct format?

A: If your dataset is already formatted in the Alpaca format, then follow the formatting steps as shown in the Llama3.1 . If you need to convert your data to the Alpaca format, one approach is to create a Python script to process your raw data. If you're working on a summarization task, you can use a local LLM to generate instructions and outputs for each example.

Q: Should I always use the standardize_sharegpt method?

A: Only use the standardize_sharegpt method if your target dataset is formatted in the sharegpt format, but your model expect a ChatML format instead.

Q: Why not use the apply_chat_template function that comes with the tokenizer.

A: The chat_template attribute when a model is first uploaded by the original model owners sometimes contains errors and may take time to be updated. In contrast, at Unsloth, we thoroughly check and fix any errors in the chat_template for every model when we upload the quantized versions to our repositories. Additionally, our get_chat_template and apply_chat_template methods offer advanced data manipulation features, which are fully documented on our Chat Templates documentation .

Q: What if my template is not currently supported by Unsloth?

A: Submit a feature request on the unsloth github issues . As a temporary workaround, you could also use the tokenizer's own apply_chat_template function until your feature request is approved and merged.

Synthetic Data Generation

You can also use any local LLM like Llama 3.3 (70B) or OpenAI's GPT 4.5 to generate synthetic data. Generally, it is better to use a bigger like Llama 3.3 (70B) to ensure the highest quality outputs. You can directly use inference engines like vLLM, Ollama or llama.cpp to generate synthetic data but it will require some manual work to collect it and prompt for more data. There's 3 goals for synthetic data:

Produce entirely new data - either from scratch or from your existing dataset
Diversify your dataset so your model does not and become too specific
Augment existing data e.g. automatically structure your dataset in the correct chosen format

Synthetic Dataset Notebook

We collaborated with Meta to launch a free notebook for creating Synthetic Datasets automatically using local models like Llama 3.2.

What the notebook does:

Auto-parses PDFs, websites, YouTube videos and more
Uses Meta’s Synthetic Data Kit + Llama 3.2 (3B) to generate QA pairs
Cleans and filters the data automatically
Fine-tunes the dataset with Unsloth + Llama

Using a local LLM or ChatGPT for synthetic data

Your goal is to prompt the model to generate and process QA data that is in your specified format. The model will need to learn the structure that you provided and also the context so ensure you at least have 10 examples of data already. Examples prompts:

Prompt for generating more dialogue on an existing dataset:
Prompt if you no have dataset:
{% code overflow="wrap" %}
{% endcode %}
Prompt for a dataset without formatting:
{% code overflow="wrap" %}

It is recommended to check the quality of generated data to remove or improve on irrelevant or poor-quality responses. Depending on your dataset it may also have to be balanced in many areas so your model does not overfit. You can then feed this cleaned dataset back into your LLM to regenerate data, now with even more guidance.

Dataset FAQ + Tips

How big should my dataset be?

We generally recommend using a bare minimum of at least 100 rows of data for fine-tuning to achieve reasonable results. For optimal performance, a dataset with over 1,000 rows is preferable, and in this case, more data usually leads to better outcomes. If your dataset is too small you can also add synthetic data or add a dataset from Hugging Face to diversify it. However, the effectiveness of your fine-tuned model depends heavily on the quality of the dataset, so be sure to thoroughly clean and prepare your data.

How should I structure my dataset if I want to fine-tune a reasoning model?

If you want to fine-tune a model that already has reasoning capabilities like the distilled versions of DeepSeek-R1 (e.g. DeepSeek-R1-Distill-Llama-8B), you will need to still follow question/task and answer pairs however, for your answer you will need to change the answer so it includes reasoning/chain-of-thought process and the steps it took to derive the answer. For a model that does not have reasoning and you want to train it so that it later encompasses reasoning capabilities, you will need to utilize a standard dataset but this time without reasoning in its answers. This is training process is known as .

Multiple datasets

If you have multiple datasets for fine-tuning, you can either:

Standardize the format of all datasets, combine them into a single dataset, and fine-tune on this unified dataset.
Use the notebook to fine-tune on multiple datasets directly.

Can I fine-tune the same model multiple times?

You can fine-tune an already fine-tuned model multiple times, but it's best to combine all the datasets and perform the fine-tuning in a single process instead. Training an already fine-tuned model can potentially alter the quality and knowledge acquired during the previous fine-tuning process.

Using Datasets in Unsloth

Alpaca Dataset

See an example of using the Alpaca dataset inside of Unsloth on Google Colab:

You can access the GPT4 version of the Alpaca dataset . Below shows some examples of the dataset:

Multiple columns for finetuning

This essentially means we have to "merge" multiple columns into 1 large prompt for finetuning to actually function!

Now this is a bit more complicated, since we allow a lot of customization, but there are a few points:

You must enclose all columns in curly braces {}. These are the column names in the actual CSV / Excel file.
Optional text components must be enclosed in [[]]. For example if the column "input" is empty, the merging function will not show the text and skip this. This is useful for datasets with missing values.
Select the output or target / prediction column in output_column_name. For the Alpaca dataset, this will be output

For example in the Titanic dataset, we can create a large merged prompt format like below, where each column / piece of text becomes optional.

For example, pretend the dataset looks like this with a lot of missing data:

Embarked

Age

Fare

Then, we do not want the result to be:

The passenger embarked from S. Their age is 23. Their fare is EMPTY.
The passenger embarked from EMPTY. Their age is 18. Their fare is $7.25.

Instead by optionally enclosing columns using [[]], we can exclude this information entirely.

[[The passenger embarked from S.]] [[Their age is 23.]] [[Their fare is EMPTY.]]
[[The passenger embarked from EMPTY.]] [[Their age is 18.]] [[Their fare is $7.25.]]

becomes:

The passenger embarked from S. Their age is 23.
Their age is 18. Their fare is $7.25.

Multi turn conversations

Then set output_column_name to the prediction / output column. For the Alpaca dataset dataset, it would be the output column.

We then use the standardize_sharegpt function to just make the dataset in a correct format for finetuning! Always call this!

Vision Fine-tuning

The dataset for fine-tuning a vision or multimodal model also includes image inputs. For example, the uses a radiography case to show how AI can help medical professionals analyze X-rays, CT scans, and ultrasounds more efficiently.

We'll be using a sampled version of the ROCO radiography dataset. You can access the dataset . The dataset includes X-rays, CT scans and ultrasounds showcasing medical conditions and diseases. Each image has a caption written by experts describing it. The goal is to finetune a VLM to make it a useful analysis tool for medical professionals.

Let's take a look at the dataset, and check what the 1st example shows:

Image

Caption

To format the dataset, all vision finetuning tasks should be formatted as follows:

We will craft an custom instruction asking the VLM to be an expert radiographer. Notice also instead of just 1 instruction, you can add multiple turns to make it a dynamic conversation.

Let's convert the dataset into the "correct" format for finetuning:

The first example is now structured like below:

Before we do any finetuning, maybe the vision model already knows how to analyse the images? Let's check if this is the case!

And the result:

For more details, view our dataset section in the .

Reinforcement Learning (RL) Guide

Learn all about Reinforcement Learning (RL) and how to train your own DeepSeek-R1 reasoning model with Unsloth using GRPO. A complete guide from beginner to advanced.

Reinforcement Learning is where an "agent" learns to make decisions by interacting with an environment and receiving feedback in the form of rewards or penalties.

Action: What the model generates (e.g. a sentence).
Reward: A signal indicating how good or bad the model's action was (e.g. did the response follow instructions? was it helpful?).
Environment: The scenario or task the model is working on (e.g. answering a user’s question).

Nov 26 update: We're introducing FP8 precision RL and GRPO in Unsloth!

🦥What you will learn

What is RL? RLVR? PPO? GRPO? RLHF? RFT? Is "Luck is All You Need?" for RL?
What is an environment? Agent? Action? Reward function? Rewards?

This article covers everything (from beginner to advanced) you need to know about GRPO, Reinforcement Learning (RL) and reward functions, along with tips, and the basics of using GRPO with . If you're looking for a step-by-step tutorial for using GRPO, see our guide .

For advanced GRPO documentation on batching, generation and training parameters,

❓What is Reinforcement Learning (RL)?

The goal of RL is to:

Increase the chance of seeing "good" outcomes.
Decrease the chance of seeing "bad" outcomes.

That's it! There are intricacies on what "good" and "bad" means, or how do we go about "increasing" or "decreasing" it, or what even "outcomes" means.

For example, in the Pacman game:

The environment is the game world.
The actions you can take are UP, LEFT, RIGHT and DOWN.
The rewards

Another example is imagine you are given the question: "What is 2 + 2?" (4) An unaligned language model will spit out 3, 4, C, D, -10, literally anything.

Numbers are better than C or D right?
Getting 3 is better than say 8 right?

🏃From RLHF, PPO to GRPO and RLVR

OpenAI popularized the concept of (Reinforcement Learning from Human Feedback), where we train an "agent" to produce outputs to a question (the state) that are rated more useful by human beings.

The thumbs up and down in ChatGPT for example can be used in the RLHF process.

The clip(..., 1-e, 1+e) term is used to force PPO not to take too large changes. There is also a KL term with beta set to > 0 to force the model not to deviate too much away.

In order to do RLHF, (Proximal policy optimization) was developed. The agent is the language model in this case. In fact it's composed of 3 systems:

The Generating Policy (current trained model)

DeepSeek developed (Group Relative Policy Optimization) to train their R1 reasoning models. The key differences to PPO are:

The Value Model is removed, replaced with statistics from calling the reward model multiple times.
The Reward Model is removed and replaced with just custom reward function which RLVR can be used.

This means GRPO is extremely efficient. Previously PPO needed to train multiple models - now with the reward model and value model removed, we can save memory and speed up everything.

RLVR (Reinforcement Learning with Verifiable Rewards) allows us to reward the model based on tasks with easy to verify solutions. For example:

Maths equations can be easily verified. Eg 2+2 = 4.
Code output can be verified as having executed correctly or not.
Designing verifiable reward functions can be tough, and so most examples are math or code.
Use-cases for GRPO isn’t just for code or math—its reasoning process can enhance tasks like email automation, database retrieval, law, and medicine, greatly improving accuracy based on your dataset and reward function - the trick is to define a rubric - ie a list of smaller verifiable rewards, and not a final all consuming singular reward.

Why "Group Relative"?

GRPO removes the value model entirely, but we still need to estimate the "average reward" given the current state.

The trick is to sample the LLM! We then calculate the average reward through statistics of the sampling process across multiple different questions.

For example for "What is 2+2?" we sample 4 times. We might get 4, 3, D, C. We then calculate the reward for each of these answers, then calculate the average reward and standard deviation, then Z-score standardize this!

This creates the advantages A, which we will use in replacement of the value model. This saves a lot of memory!

🤞Luck (well Patience) Is All You Need

The trick of RL is you need 2 things only:

A question or instruction eg "What is 2+2?" "Create a Flappy Bird game in Python"
A reward function and verifier to verify if the output is good or bad.

With only these 2, we can essentially call a language model an infinite times until we get a good answer. For example for "What is 2+2?", an untrained bad language model will output:

0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31**** then suddenly 4.

The reward signal was 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0**** then suddenly 1.

So by luck and by chance, RL managed to find the correct answer across multiple rollouts. Our goal is we want to see the good answer 4 more, and the rest (the bad answers) much less.

So the goal of RL is to be patient - in the limit, if the probability of the correct answer is at least a small number (not zero), it's just a waiting game - you will 100% for sure encounter the correct answer in the limit.

So I like to call it as "Luck Is All You Need" for RL.

Well a better phrase is "Patience is All You Need" for RL.

RL essentially provides us a trick - instead of simply waiting for infinity, we do get "bad signals" ie bad answers, and we can essentially "guide" the model to already try not generating bad solutions. This means although you waited very long for a "good" answer to pop up, the model already has been changed to try its best not to output bad answers.

In the "What is 2+2?" example - 0, cat, -10, 1928, 3, A, B, 122, 17, 182, 172, A, C, BAHS, %$, #, 9, -192, 12.31**** then suddenly 4.

Since we got bad answers, RL will influence the model to try NOT to output bad answers. This means over time, we are carefully "pruning" or moving the model's output distribution away from bad answers. This means RL is efficient, since we are NOT just waiting for infinity, but we are actively trying to "push" the model to go as much as possible to the "correct answer space".

If the probability is always 0, then RL will never work. This is also why people like to do RL from an already instruction finetuned model, which can partially follow instructions reasonably well - this boosts the probability most likely above 0.

🦥What Unsloth offers for RL

With 15GB VRAM, Unsloth allows you to transform any model up to 17B parameters like Llama 3.1 (8B), Phi-4 (14B), Mistral (7B) or Qwen2.5 (7B) into a reasoning model
Unsloth now supports models!
Minimum requirement: Just  5GB VRAM is enough to train your own reasoning model locally (for any model with 1.5B parameters or less)

GRPO notebooks:

NEW! We now support and most other new GRPO techniques. You can play with the following arguments in GRPOConfig to enable:

If you're not getting any reasoning, make sure you have enough training steps and ensure your is working. We provide examples for reward functions .
Previous demonstrations show that you could achieve your own "aha" moment with Qwen2.5 (3B) - but it required 2xA100 GPUs (160GB VRAM). Now, with Unsloth, you can achieve the same "aha" moment using just a single 5GB VRAM GPU.
Previously, GRPO was only supported for full fine-tuning, but we've made it work with QLoRA and LoRA
On

In a test example, even though we only trained Phi-4 with 100 steps using GRPO, the results are already clear. The model without GRPO does not have the thinking token, whilst the one trained with GRPO does and also has the correct answer.

💻Training with GRPO

For a tutorial on how to transform any open LLM into a reasoning model using Unsloth & GRPO, .

For advanced GRPO documentation on batching, generation and training parameters,

How GRPO Trains a Model

For each question-answer pair, the model generates multiple possible responses (e.g., 8 variations).
Each response is evaluated using reward functions.
Training Steps:
- If you have 300 rows of data, that's 300 training steps (or 900 steps if trained for 3 epochs).

If you're having issues with your GRPO model not learning, we'd highly recommend to use our as it has a much better reward function and you should see results much faster and frequently.

Basics/Tips

Wait for at least 300 steps for the reward to actually increase. In order to get decent results, you may need to trade for a minimum of 12 hours (this is how GRPO works), but keep in mind this isn't compulsory as you can stop at anytime.
For optimal results have at least 500 rows of data. You can try with even 10 rows of data but it's better to have more.
Each training run will always be different depending on your model, data, reward function/verifier etc. so though 300 steps is what we wrote as the minimum, sometimes it might be 1000 steps or more. So, it depends on various factors.

📋Reward Functions / Verifiers

In Reinforcement Learning a Reward Function and a Verifier serve distinct roles in evaluating a model’s output. In general, you could interpret them as the same thing however, technically they're not but it does not matter as much as they are usually used in conjunction with each other.

Verifier:

Determines whether the generated response is correct or incorrect.
It does not assign a numerical score—it simply verifies correctness.
Example: If a model generates "5" for "2+2", the verifier checks and labels it as "wrong" (since the correct answer is 4).
Verifiers can also execute code (e.g., in Python) to validate logic, syntax, and correctness without needing manual evaluation.

Reward Function:

Converts verification results (or other criteria) into a numerical score.
Example: If an answer is wrong, it might assign a penalty (-1, -2, etc.), while a correct answer could get a positive score (+1, +2).
It can also penalize based on criteria beyond correctness, such as excessive length or poor readability.

Key Differences:

A Verifier checks correctness but doesn’t score.
A Reward Function assigns a score but doesn’t necessarily verify correctness itself.
A Reward Function can use a Verifier, but they are technically not the same.

Understanding Reward Functions

GRPO's primary goal is to maximize reward and learn how an answer was derived, rather than simply memorizing and reproducing responses from its training data.

With every training step, GRPO adjusts model weights to maximize the reward. This process fine-tunes the model incrementally.
Regular fine-tuning (without GRPO) only maximizes next-word prediction probability but does not optimize for a reward. GRPO optimizes for a reward function rather than just predicting the next word.
You can reuse data across multiple epochs.

🪙Reward Function Examples

You can refer to the examples below. You can input your generations into an LLM like ChatGPT 4o or Llama 3.1 (8B) and design a reward function and verifier to evaluate it. For example, feed your generations into a LLM of your choice and set a rule: "If the answer sounds too robotic, deduct 3 points." This helps refine outputs based on quality criteria

Example #1: Simple Arithmetic Task

Question: "2 + 2"
Answer: "4"
Reward Function 1:

Example #2: Email Automation Task

Question: Inbound email
Answer: Outbound email
Reward Functions:
- If the answer contains a required keyword → +1

Unsloth Proximity-Based Reward Function

If you’ve checked out our , you’ll notice we’ve created a custom proximity-based reward function built completely from scratch, which is designed to reward answers that are closer to the correct one. This flexible function can be applied across a wide range of tasks.

In our examples, we enable reasoning in Qwen3 (Base) and guide it toward specific tasks
Apply Pre-finetuning strategies to avoid GRPO’s default tendency to just learn formatting
Boost evaluation accuracy with regex-based matching
Create custom GRPO templates beyond generic prompts like think, e.g., <start_working_out></end_working_out>

GSM8K Reward Functions

In our other examples, we use existing GSM8K reward functions by which is popular and shown to be quite effective:

correctness_reward_func – Rewards exact label matches.
int_reward_func – Encourages integer-only answers.
soft_format_reward_func – Checks structure but allows minor newline mismatches.
strict_format_reward_func – Ensures response structure matches the prompt, including newlines.

🧮Using vLLM

You can now use directly in your finetuning stack, which allows for much more throughput and allows you to finetune and do inference on the model at the same time! On 1x A100 40GB, expect 4000 tokens / s or so with Unsloth’s dynamic 4bit quant of Llama 3.2 3B Instruct. On a 16GB Tesla T4 (free Colab GPU), you can get 300 tokens / s. We also magically removed double memory usage when loading vLLM and Unsloth together, allowing for savings of 5GB or so for Llama 3.1 8B and 3GB for Llama 3.2 3B. Unsloth could originally finetune Llama 3.3 70B Instruct in 1x 48GB GPU with Llama 3.3 70B weights taking 40GB of VRAM. If we do not remove double memory usage, then we’ll need >= 80GB of VRAM when loading Unsloth and vLLM together. But with Unsloth, you can still finetune and get the benefits of fast inference in one package in under 48GB of VRAM! To use fast inference, first install vllm, and instantiate Unsloth with fast_inference:

✅GRPO Requirement Guidelines

When you’re using Unsloth to do GRPO, we smartly reduce VRAM usage by over 90% when compared to standard implementations with Flash Attention 2 by using multiple tricks! On 20K context lengths for example with 8 generations per prompt, Unsloth uses only 54.3GB of VRAM for Llama 3.1 8B, whilst standard implementations take 510.8GB (90% less for Unsloth).

For GRPO's GPU VRAM requirements for QLoRA 4-bit, the general rule is the model parameters = the amount of VRAM you will need (you can use less VRAM but this just to be safe). The more context length you set, the more VRAM. LoRA 16-bit will use at minimum 4x more VRAM.
Our new memory efficient linear kernels for GRPO slashes memory usage by 8x or more. This shaves 68.5GB of memory, whilst being actually faster through the help of torch.compile!
We leverage our smart algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves 52GB of memory.

Metrics

Unsloth

Standard + FA2

In typical standard GRPO implementations, you need to create 2 logits of size (8. 20K) to calculate the GRPO loss. This takes 2 * 2 bytes * 8 (num generations) * 20K (context length) * 128256 (vocabulary size) = 78.3GB in VRAM.

Unsloth shaves 8x memory usage for long context GRPO, so we need only an extra 9.8GB in extra VRAM for 20K context lengths!

We also need to from the KV Cache in 16bit. Llama 3.1 8B has 32 layers, and both K and V are 1024 in size. So memory usage for 20K context length = 2 * 2 bytes * 32 layers * 20K context length * 1024 = 2.5GB per batch. We would set the batch size for vLLM to 8, but we shall leave it at 1 for our calculations to save VRAM. Otherwise you will need 20GB for the KV cache.

🎥 Unsloth RL 3 hour Workshop Video

🎓Further Reading

Nathan Lambert's RLHF Book is a must!
Yannic Kilcher's GRPO Youtube video is also a must!
We did a 3 hour workshop at AI Engineer World's Fair 2025. Slides are other material are at
Advanced GRPO notebook via Unsloth.

Qwen3-Coder: How to Run Locally

Run Qwen3-Coder-30B-A3B-Instruct and 480B-A35B locally with Unsloth Dynamic quants.

Qwen3-Coder is Qwen’s new series of coding agent models, available in 30B (Qwen3-Coder-Flash) and 480B parameters. Qwen3-480B-A35B-Instruct achieves SOTA coding performance rivalling Claude Sonnet-4, GPT-4.1, and Kimi K2, with 61.8% on Aider Polygot and support for 256K (extendable to 1M) token context.

We also uploaded Qwen3-Coder with native 1M context length extended by YaRN and full-precision 8bit and 16bit versions. Unsloth also now supports fine-tuning and RL of Qwen3-Coder.

UPDATE: We fixed tool-calling for Qwen3-Coder! You can now use tool-calling seamlessly in llama.cpp, Ollama, LMStudio, Open WebUI, Jan etc. This issue was universal and affected all uploads (not just Unsloth), and we've communicated with the Qwen team about our fixes! Read more

Does work? Yes, and very well. In third-party testing on the Aider Polyglot benchmark, the UD-Q4_K_XL (276GB) dynamic quant nearly matched the full bf16 (960GB) Qwen3-coder model, scoring 60.9% vs 61.8%.

Qwen3 Coder - Unsloth Dynamic 2.0 GGUFs:

Dynamic 2.0 GGUF (to run)

1M Context Dynamic 2.0 GGUF

🖥️ Running Qwen3-Coder

Below are guides for the and variants of the model.

⚙️ Recommended Settings

Qwen recommends these inference settings for both models:

temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.05

Temperature of 0.7
Top_K of 20
Min_P of 0.00 (optional, but 0.01 works well, llama.cpp default is 0.1)
Top_P of 0.8

Chat template/prompt format with newlines un-rendered

Chat template for tool calling (Getting the current temperature for San Francisco). More details here for how to format tool calls.

Reminder that this model supports only non-thinking mode and does not generate <think></think> blocks in its output. Meanwhile, specifying enable_thinking=False is no longer required.

Run Qwen3-Coder-30B-A3B-Instruct:

To achieve inference speeds of 6+ tokens per second for our Dynamic 4-bit quant, have at least 18GB of unified memory (combined VRAM and RAM) or 18GB of system RAM alone. As a rule of thumb, your available memory should match or exceed the size of the model you’re using. E.g. the UD_Q8_K_XL quant (full precision), which is 32.5GB, will require at least 33GB of unified memory (VRAM + RAM) or 33GB of RAM for optimal performance.

NOTE: The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds.

Given that this is a non thinking model, there is no need to set thinking=False and the model does not generate <think> </think> blocks.

Follow the . They're the same as the 480B model.

🦙 Ollama: Run Qwen3-Coder-30B-A3B-Instruct Tutorial

Install ollama if you haven't already! You can only run models up to 32B in size.

Run the model! Note you can call ollama servein another terminal if it fails! We include all our fixes and suggested parameters (temperature etc) in params in our Hugging Face upload!

✨ Llama.cpp: Run Qwen3-Coder-30B-A3B-Instruct Tutorial

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

You can directly pull from HuggingFace via:
Download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD_Q4_K_XL or other quantized versions.

Run Qwen3-Coder-480B-A35B-Instruct:

To achieve inference speeds of 6+ tokens per second for our 1-bit quant, we recommend at least 150GB of unified memory (combined VRAM and RAM) or 150GB of system RAM alone. As a rule of thumb, your available memory should match or exceed the size of the model you’re using. E.g. the Q2_K_XL quant, which is 180GB, will require at least 180GB of unified memory (VRAM + RAM) or 180GB of RAM for optimal performance.

NOTE: The model can run on less memory than its total size, but this will slow down inference. Maximum memory is only needed for the fastest speeds.

Follow the . They're the same as the 30B model.

📖 Llama.cpp: Run Qwen3-Coder-480B-A35B-Instruct Tutorial

For Coder-480B-A35B, we will specifically use Llama.cpp for optimized inference and a plethora of options.

If you want a full precision unquantized version, use our Q8_K_XL, Q8_0 or BF16 versions!

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.
You can directly use llama.cpp to download the model but I normally suggest using huggingface_hub To use llama.cpp directly, do:
{% code overflow="wrap" %}
{% endcode %}

Also don't forget about the new Qwen3 update. Run locally with llama.cpp.

🛠️ Improving generation speed

If you have more VRAM, you can try offloading more MoE layers, or offloading whole layers themselves.

If you have a bit more GPU memory, try -ot ".ffn_(up|down)_exps.=CPU" This offloads up and down projection MoE layers.

Try -ot ".ffn_(up)_exps.=CPU" if you have even more GPU memory. This offloads only up projection MoE layers.

📐How to fit long context (256K to 1M)

--cache-type-k f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1

You should use the _1 variants for somewhat increased accuracy, albeit it's slightly slower. For eg q4_1, q5_1

You can also quantize the V cache, but you will need to compile llama.cpp with Flash Attention support via -DGGML_CUDA_FA_ALL_QUANTS=ON, and use --flash-attn to enable it.

We also uploaded 1 million context length GGUFs via YaRN scaling .

🧰 Tool Calling Fixes

We managed to fix tool calling via llama.cpp --jinja specifically for serving through llama-server! If you’re downloading our 30B-A3B quants, no need to worry as these already include our fixes. For the 480B-A35B model, please:

Download the first file at https://huggingface.co/unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF/tree/main/UD-Q2_K_XL for UD-Q2_K_XL, and replace your current file
Use snapshot_download as usual as in https://docs.unsloth.ai/basics/qwen3-coder-how-to-run-locally#llama.cpp-run-qwen3-tutorial which will auto override the old files
Use the new chat template via --chat-template-file. See or

This should solve issues like: https://github.com/ggml-org/llama.cpp/issues/14915

Using Tool Calling

To format the prompts for tool calling, let's showcase it with an example.

I created a Python function called get_current_temperature which is a function which should get the current temperature for a location. For now we created a placeholder function which will always return 21.6 degrees celsius. You should change this to a true function!!

Then use the tokenizer to create the entire prompt:

💡Performance Benchmarks

These official benchmarks are for the full BF16 checkpoint. To use this, simply use the Q8_K_XL, Q8_0, BF16 checkpoints we uploaded - you can still use the tricks like MoE offloading for these versions as well!

Here are the benchmarks for the 480B model:

Agentic Coding

Benchmark

Qwen3‑Coder 480B‑A35B‑Instruct

Kimi‑K2

DeepSeek‑V3-0324

Claude 4 Sonnet

GPT‑4.1

Agentic Browser Use

Benchmark

Qwen3‑Coder 480B‑A35B‑Instruct

Kimi‑K2

DeepSeek‑V3 0324

Claude Sonnet‑4

GPT‑4.1

Agentic Tool -Use

Benchmark

Qwen3‑Coder 480B‑A35B‑Instruct

Kimi‑K2

DeepSeek‑V3 0324

Claude Sonnet‑4

GPT‑4.1

Unsloth Dynamic 2.0 GGUFs

A big new upgrade to our Dynamic Quants!

We're excited to introduce our Dynamic v2.0 quantization method - a major upgrade to our previous quants. This new method outperforms leading quantization methods and sets new benchmarks for 5-shot MMLU and KL Divergence.

This means you can now run + fine-tune quantized LLMs while preserving as much accuracy as possible! You can run the 2.0 GGUFs on any inference engine like llama.cpp, Ollama, Open WebUI etc.

Sept 10, 2025 update: You asked for tougher benchmarks, so we’re showcasing Aider Polyglot results! Our Dynamic 3-bit DeepSeek V3.1 GGUF scores 75.6%, surpassing many full-precision SOTA LLMs. Read more.

The key advantage of using the Unsloth package and models is our active role in fixing critical bugs in major models. We've collaborated directly with teams behind , , , and , contributing essential fixes that significantly boost accuracy.

Detailed analysis of our benchmarks and evaluation further below.

💡 What's New in Dynamic v2.0?

Revamped Layer Selection for GGUFs + safetensors: Unsloth Dynamic 2.0 now selectively quantizes layers much more intelligently and extensively. Rather than modifying only select layers, we now dynamically adjust the quantization type of every possible layer, and the combinations will differ for each layer and model.
Current selected and all future GGUF uploads will utilize Dynamic 2.0 and our new calibration dataset. The dataset contains more than >1.5M tokens (depending on model) and comprise of high-quality, hand-curated and cleaned data - to greatly enhance conversational chat performance.
Previously, our Dynamic quantization (DeepSeek-R1 1.58-bit GGUF) was effective only for MoE architectures. Dynamic 2.0 quantization now works on all models (including MOEs & non-MoEs).

To ensure accurate benchmarking, we built an internal evaluation framework to match official reported 5-shot MMLU scores of Llama 4 and Gemma 3. This allowed apples-to-apples comparisons between full-precision vs. Dynamic v2.0, QAT and standard imatrix GGUF quants.

All future GGUF uploads will utilize Unsloth Dynamic 2.0, and our Dynamic 4-bit safe tensor quants will also benefit from this in the future.

📊 Why KL Divergence?

showcases how pruning layers, even by selecting unnecessary ones still yields vast differences in terms of "flips". A "flip" is defined as answers changing from incorrect to correct or vice versa. The paper shows how MMLU might not decrease as we prune layers or do quantization,but that's because some incorrect answers might have "flipped" to become correct. Our goal is to match the original model, so measuring "flips" is a good metric.

KL Divergence should be the gold standard for reporting quantization errors as per the research paper "Accuracy is Not All You Need". Using perplexity is incorrect since output token values can cancel out, so we must use KLD!

The paper also shows that interestingly KL Divergence is highly correlated with flips, and so our goal is to reduce the mean KL Divergence whilst increasing the disk space of the quantization as less as possible.

⚖️ Calibration Dataset Overfitting

Most frameworks report perplexity and KL Divergence using a test set of Wikipedia articles. However, we noticed using the calibration dataset which is also Wikipedia related causes quants to overfit, and attain lower perplexity scores. We utilize and datasets for fair testing which includes some wikitext data amongst other data. Also instruct models have unique chat templates, and using text only calibration datasets is not effective for instruct models (base models yes). In fact most imatrix GGUFs are typically calibrated with these issues. As a result, they naturally perform better on KL Divergence benchmarks that also use Wikipedia data, since the model is essentially optimized for that domain.

To ensure a fair and controlled evaluation, we do not to use our own calibration dataset (which is optimized for chat performance) when benchmarking KL Divergence. Instead, we conducted tests using the same standard Wikipedia datasets, allowing us to directly compare the performance of our Dynamic 2.0 method against the baseline imatrix approach.

🔢 MMLU Replication Adventure

Replicating MMLU 5 shot was nightmarish. We could not replicate MMLU results for many models including Llama 3.1 (8B) Instruct, Gemma 3 (12B) and others due to subtle implementation issues. Llama 3.1 (8B) for example should be getting ~68.2%, whilst using incorrect implementations can attain 35% accuracy.

Llama 3.1 (8B) Instruct has a MMLU 5 shot accuracy of 67.8% using a naive MMLU implementation. We find however Llama tokenizes "A" and "_A" (A with a space in front) as different token ids. If we consider both spaced and non spaced tokens, we get 68.2% (+0.4%)
Interestingly Llama 3 as per Eleuther AI's also appends "The best answer is" to the question, following Llama 3's original MMLU benchmarks.
There are many other subtle issues, and so to benchmark everything in a controlled environment, we designed our own MMLU implementation from scratch by investigating directly, and verified our results across multiple models and comparing to reported numbers.

✨ Gemma 3 QAT Replication, Benchmarks

The Gemma team released two QAT (quantization aware training) versions of Gemma 3:

Q4_0 GGUF - Quantizes all layers to Q4_0 via the formula w = q * block_scale with each block having 32 weights. See for more details.
int4 version - presumably ?

We benchmarked all Q4_0 GGUF versions, and did extensive experiments on the 12B model. We see the 12B Q4_0 QAT model gets 67.07% whilst the full bfloat16 12B version gets 67.15% on 5 shot MMLU. That's very impressive! The 27B model is mostly nearly there!

Metric

12B

27B

We designed a new Efficiency metric which calculates the usefulness of the model whilst also taking into account its disk size and MMLU 5 shot score:

We have to minus 25 since MMLU has 4 multiple choices - A, B, C or D. Assume we make a model that simply randomly chooses answers - it'll get 25% accuracy, and have a disk space of a few bytes. But clearly this is not a useful model.

On KL Divergence vs the base model, below is a table showcasing the improvements. Reminder the closer the KL Divergence is to 0, the better (ie 0 means identical to the full precision model)

Quant

Baseline KLD

New KLD

If we plot the ratio of the disk space increase and the KL Divergence ratio change, we can see a much clearer benefit! Our dynamic 2bit Q2_K_XL reduces KLD quite a bit (around 7.5%).

Truncated table of results for MMLU for Gemma 3 (27B). See below.

Our dynamic 4bit version is 2GB smaller whilst having +1% extra accuracy vs the QAT version!
Efficiency wise, 2bit Q2_K_XL and others seem to do very well!

Quant

Unsloth

Unsloth + QAT

Disk Size

Efficiency

Click here for Full Google's Gemma 3 (27B) QAT Benchmarks:

Model

Unsloth

Unsloth + QAT

Disk Size

Efficiency

🦙 Llama 4 Bug Fixes + Run

We also helped and fixed a few Llama 4 bugs:

Llama 4 Scout changed the RoPE Scaling configuration in their official repo. We helped resolve issues in llama.cpp to enable this
Llama 4's QK Norm's epsilon for both Scout and Maverick should be from the config file - this means using 1e-05 and not 1e-06. We helped resolve these in and
The Llama 4 team and vLLM also independently fixed an issue with QK Norm being shared across all heads (should not be so) . MMLU Pro increased from 68.58% to 71.53% accuracy.

As shown in our graph, our 4-bit Dynamic QAT quantization deliver better performance on 5-shot MMLU while also being smaller in size.

Running Llama 4 Scout:

To run Llama 4 Scout for example, first clone llama.cpp:

Then download out new dynamic v 2.0 quant for Scout:

And and let's do inference!

Magistral: How to Run & Fine-tune

Meet Magistral - Mistral's new reasoning models.

Magistral-Small-2509 is a reasoning LLM developed by Mistral AI. It excels at coding and mathematics and supports multiple languages. Magistral supports a 128k token context window and was finetuned from Mistral-Small-3.2. Magistral runs perfectly well locally on a single RTX 4090 or a Mac with 16 to 24GB RAM.

Running Magistral Tutorial Fine-tuning Magistral

Update: Magistral-2509 new update is out as of September, 2025! Now with Vision support! We worked with Mistral again with the release of Magistral. Make sure to download Mistral's official uploads or Unsloth's uploads to get the correct implementation (ie correct system prompt, correct chat template etc.)

If you're using llama.cpp, please use --jinja to enable the system prompt!

All uploads use Unsloth for SOTA 5-shot MMLU and KL Divergence performance, meaning you can run & fine-tune quantized Mistral LLMs with minimal accuracy loss.

Magistral-Small - Unsloth Dynamic uploads:

Dynamic 2.0 GGUF (to run)

Dynamic 4-bit (to finetune/deploy)

Dynamic Float8

🖥️ Running Magistral

⚙️ Official Recommended Settings

According to Mistral AI, these are the recommended settings for inference:

Temperature of: 0.7
Min_P of: 0.01 (optional, but 0.01 works well, llama.cpp default is 0.1)
Set top_p to: 0.95
A 128k context window is supported, but performance might degrade past 40k. So we recommend setting the maximum length to 40k if you see bad performance.

This is the recommended system prompt for Magistral 2509, 2507:

This is the recommended system prompt for Magistral 2506:

Our dynamic uploads have the 'UD' prefix in them. Those without are not dynamic however still utilize our calibration dataset.

Multilingual: Magistral supports many languages including: English, French, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Nepali, Polish, Portuguese, Romanian, Russian, Serbian, Spanish, Swedish, Turkish, Ukrainian, Vietnamese, Arabic, Bengali, Chinese, and Farsi.

❓Testing the model

Mistral has their own vibe checking prompts which can be used to evaluate Magistral. Keep in mind these tests are based on running the full unquantized version of the model, however you could also test them on quantized versions:

Easy - Make sure they always work

Medium - Should most of the time be correct

Hard - Should sometimes get them right

We provide some at the end of the blog.

🦙 Tutorial: How to Run Magistral in Ollama

Install ollama if you haven't already!

Run the model with our dynamic quant. We did not set the context length automatically, so it will just use Ollama's default set context length. Note you can call ollama serve &in another terminal if it fails! We include all suggested parameters (temperature etc) in params in our Hugging Face upload!
Also Magistral supports 40K context lengths, so best to enable . We use 8bit quantization which saves 50% memory usage. You can also try "q4_0" or "q8_0"

📖 Tutorial: How to Run Magistral in llama.cpp

Obtain the latest llama.cpp on . You can follow the build instructions below as well. Change -DGGML_CUDA=ON to -DGGML_CUDA=OFF if you don't have a GPU or just want CPU inference.

If you want to use llama.cpp directly to load models, you can do the below: (:Q4_K_XL) is the quantization type. You can also download via Hugging Face (point 3). This is similar to ollama run

In llama.cpp, please use --jinja to enable the system prompt!

OR download the model via (after installing pip install huggingface_hub hf_transfer ). You can choose UD-Q4_K_XL, (Unsloth Dynamic), Q4_K_M, or other quantized versions (like BF16 full precision).

Run the model.
Edit --threads -1 for the maximum CPU threads, --ctx-size 40960 for context length (Magistral supports 40K context length!), --n-gpu-layers 99 for GPU offloading on how many layers. Try adjusting it if your GPU goes out of memory. Also remove it if you have CPU only inference. We also use 8bit quantization for the K cache to reduce memory usage.
For conversation mode:

Remember to remove <bos> since Magistral auto adds a <bos>

Sample outputs

How many "r" are in strawberry? [Correct answer = 3]

Exactly how many days ago did the French Revolution start? Today is June 4th, 2025. [Correct answer = 86,157 days]

👁Vision Support

Magistral 2509's September 2025 update now includes Vision support by default!

For Magistral versions before September 2025, from HuggingFace showed in their how it is actually possible to "graft" the vision encoder from Mistral 3.1 Instruct onto Devstral meaning you could do the same for Magistral! According to our tests and many users, it works quite well! We also uploaded our mmproj files which allows you to use the following:

🦥 Fine-tuning Magistral with Unsloth

Just like standard Mistral models including Mistral Small 3.1, Unsloth supports Magistral fine-tuning. Training is 2x faster, use 70% less VRAM and supports 8x longer context lengths. Magistral fits comfortably in a 24GB VRAM L4 GPU.

Magistral 2509 Kaggle (2x Tesla T4s) free
Magistral 2509 Colab L4 (24GB)

Magistral slightly exceeds the memory limits of a 16GB VRAM, so fine-tuning it for free on Google Colab isn't possible for now. However, you can fine-tune the model for free using , which offers access to dual GPUs.

To finetune on new reasoning traces, you can use our free

If you have an old version of Unsloth and/or are fine-tuning locally, install the latest version of Unsloth:

💠Dynamic Float8 Checkpoints

We also provide 2 popular formats for float8 checkpoints, which also utilizes some of our dynamic methodology to retain maximum accuracy:

Both are fantastic to deploy via vLLM. Read up on using TorchAO based FP8 quants in vLLM .

<think>
Alright, I have a problem to solve: "How many 'r's are in strawberry?"

First, I need to understand the problem. It's asking for the count of the letter 'r' in the word "strawberry".

Step 1: Write down the word and look at each letter.
The word is: s t r a w b e r r y

Step 2: Identify the 'r's in the word.
Let's go through each letter one by one:
1. s - not an r
2. t - not an r
3. r - that's one r
4. a - not an r
5. w - not an r
6. b - not an r
7. e - not an r
8. r - that's another r
9. r - and another one
10. y - not an r

So, I see the letter 'r' appears at positions 3, 8, and 9.

Step 3: Count the number of 'r's.
From the above, there are three 'r's in "strawberry".

But wait, I should double-check to make sure I didn't miss anything or count incorrectly.

Let me write the word again: strawberry.

Looking at it:
- The third letter is 'r' (s-t-r)
- Then later, we have 'e-r-r' which is two more 'r's.

Yes, that's three 'r's in total.

But just to be thorough, sometimes letters can be hidden or the word might have silent letters, but in this case, all letters are pronounced in "strawberry", and 'r' is clearly pronounced in the third, eighth, and ninth positions.

So, the count is indeed three.

But wait, is it possible that the 'r's are not all pronounced? For example, in some accents or dialects, letters might be silent or pronounced differently. However, in standard pronunciation, all these 'r's are pronounced.

But the question is about the number of 'r's in the word, not about pronunciation. It's about the written form. So, regardless of pronunciation, we're counting the letters in the word "strawberry".

Let me write it out again and count:

s (1)
t (2)
r (3) - first r
a (4)
w (5)
b (6)
e (7)
r (8) - second r
r (9) - third r
y (10)

Yes, three 'r's.

But just to be sure, maybe I should think about the spelling. Sometimes letters can be part of digraphs or other combinations, but in "strawberry", the 'r's are all standalone letters.

Let me think about the spelling breakdown:
- "straw" has one 'r' (the third letter)
- "berry" has two 'r's (the second and third letters in "berry", but in the full word, it's the eighth and ninth letters)

So, "straw" has 1 'r', "berry" has 2 'r's, totaling 3 'r's in "strawberry".

I think I'm confident now. The word "strawberry" has three 'r's.

But before finalizing, I recall that sometimes letters can be repeated or there might be a trick. Let me see:

The word is: s-t-r-a-w-b-e-r-r-y.

Counting the 'r's:
1. The 'r' in the third position.
2. The 'r' in the eighth position.
3. The 'r' in the ninth position.

No other 'r's are present.

Therefore, the total number of 'r's in "strawberry" is three.

But just to cross-verify, perhaps I can write the word and highlight the 'r's:

s t r a w b e r r y
   r     r r

Yes, three 'r's.

Is there any chance I'm missing something? Maybe considering uppercase and lowercase? But the problem is case-insensitive since it's about the letter 'r', not 'R'. The word is given in lowercase, but even if it were uppercase, 'R' is the same letter as 'r' in terms of counting.

So, final count is three.

But wait, is "strawberry" spelled correctly? Let me check the spelling.

I think it's s-t-r-a-w-b-e-r-r-y. Yes, that seems correct.

Therefore, the number of 'r's is indeed three.

But just to be extra sure, maybe I can think of the word in a different way. For example, breaking it down:

"straw" + "berry" = "strawberry"

"straw" has one 'r' (the third letter: s-t-r)
"berry" has two 'r's (the second and third letters: b-e-r-r-y, but actually, 'berry' is b-e-r-r-y, so 'r' is the third and fourth letters? Wait, let's see:

'berry' is b-e-r-r-y. Wait, no:

b (1)
e (2)
r (3)
r (4)
y (5)

So, 'berry' has 'r's at positions 3 and 4, which are two 'r's.

'straw' is s-t-r-a-w, so 'r' is at position 3, one 'r'.

Combined, 'strawberry' is 'straw' + 'berry' = s-t-r-a-w-b-e-r-r-y, which has 'r's at positions 3, 3+5+1=9? Wait, no, let's index:

'straw' is positions 1-5 (s-t-r-a-w)
'berry' is positions 6-10 (b-e-r-r-y)

So 'r's in 'straw' are at position 3.
'r's in 'berry' are at positions 8 and 9 (since 'berry' starts at position 6: b(6), e(7), r(8), r(9), y(10)).

Thus, total 'r's are at positions 3, 8, and 9: three 'r's.

Yes, consistent again.

Is there any other way to approach this? Maybe counting the total letters and seeing how many are 'r's.

"strawberry" has 10 letters: s, t, r, a, w, b, e, r, r, y.

Counting 'r's: positions 3, 8, 9 → 3 'r's.

Alternatively, I could write out the word and replace non-'r's with another character to see how many 'r's are left.

Original: s t r a w b e r r y
Replace non-r: x x r x x x x r r x
Now count 'r's: positions 2 (but original was 3rd letter), wait no, in the replaced string, 'r's are at positions 3 (original position 3), and then after replacing others, the next 'r's are at positions... Wait, let's see:

Original positions:
1: s → x
2: t → x
3: r → r
4: a → x
5: w → x
6: b → x
7: e → x
8: r → r
9: r → r
10: y → x

So replaced string: x x r x x x x r r x
Now count 'r's: positions 3, 8, 9 in original correspond to positions 3, 8, 9 in replaced (but replaced has x's in other positions, so 'r's are still at 3,8,9). Wait, in replaced string, 'r's are at positions 3,8,9, but the length is still 10, so positions are same as original. So count is same: 3 'r's.

Another method: write the word and cross out non-'r's.

Original: s t r a w b e r r y
Cross out non-r: x x r x x x x r r x
Now count 'r's: there are three 'r's remaining.

All methods confirm that there are three 'r's in "strawberry".

But just to think outside the box, is there any ambiguity in the question? For example, is it asking for uppercase 'R' or lowercase 'r'? The problem says "r", lowercase, so we're counting lowercase 'r's. In the word "strawberry", all letters are lowercase (assuming the input is in lowercase; if it were mixed case, we'd have to consider, but since it's given as "strawberry", which is all lowercase, we're good).

Also, is it possible that some 'r's are part of a digraph or trigraph? For example, in some languages, 'rr' might be considered a single sound or letter, but in English orthography, 'rr' is two separate 'r's. So in "strawberry", the 'rr' is two 'r's.

Therefore, the count remains three.

Is there any other interpretation? Maybe considering that sometimes letters can represent different sounds or that 'r' might be silent, but as mentioned earlier, the question is about the written letter 'r', not pronunciation.

Final confirmation: let's write the word and count 'r's aloud:

s (no), t (no), r (1), a (no), w (no), b (no), e (no), r (2), r (3), y (no).

Yes, three 'r's.

Therefore, the number of 'r's in "strawberry" is three.

But before finalizing, let me think if there's any trick or if I'm missing something obvious. For example, sometimes questions like this have hidden complexities, like counting 'r's in the word's spelling or pronunciation, or considering that 'r' might appear in other forms, like in 'rr' which is two 'r's together, but we've already accounted for that.

Alternatively, is the question about the number of times the letter 'r' appears in the spelling of "strawberry"? Yes, that's how I interpreted it, and my count is three.

Is there any chance the word is spelled differently? For example, in some countries or dialects, is "strawberry" spelled differently? In American English and British English, it's spelled the same way: s-t-r-a-w-b-e-r-r-y.

Let me check the spelling again: s, t, r, a, w, b, e, r, r, y. Yes, that's correct.

Therefore, the number of 'r's is indeed three.

But just to be absolutely sure, perhaps I can think of the word in a sentence or see it written out:

Example sentence: "I ate a strawberry." The word is "strawberry", and in it, the 'r's are in the middle and at the end.

Looking at the spelling: s-t-r-a-w-b-e-r-r-y. Counting:

1. s
2. t
3. r (1)
4. a
5. w
6. b
7. e
8. r (2)
9. r (3)
10. y

Yes, three 'r's.

I think I've exhausted all possible ways to verify this count. Therefore, the number of 'r's in "strawberry" is three.

But wait, the user might have intended the question differently. Could it be about the number of times the sound 'r' appears in pronunciation? For example, sometimes letters are silent or combined in pronunciation, but the question asks for the count of the letter 'r', not the sound.

In pronunciation, "strawberry" is /ˈstroʊbɛri/ or similar, where the 'r's are pronounced as in the spelling. But since the question is about the letter count, not phonemes, we stick with the written form.

Therefore, the count remains three.

Is there any other interpretation? For example, counting uppercase 'R's if the word were capitalized? But the word is given in lowercase, and the question asks for 'r', lowercase, so it's consistent.

Final answer: there are three 'r's in "strawberry".

But just to double-check, perhaps I can list the letters with their positions:

Position: 1 2 3 4 5 6 7 8 9 10
Letter:   s t r a w b e r r y

Now, pick out the 'r's: positions 3, 8, 9. That's three 'r's.

Alternatively, write the word and circle the 'r's:

Strawberry: s t r a w b e r r y
The 'r's are in positions 3, 8, and 9. So, three 'r's.

No other interpretations seem plausible. Therefore, the answer is three.

But before finalizing, let me think if there's any ambiguity in the word itself. Could "strawberry" be misspelled or is there another spelling variant?

In English, "strawberry" is consistently spelled with two 'r's at the end: "berry" is spelled with two 'r's (b-e-r-r-y), and "straw" has one 'r' (s-t-r-a-w). So total is three 'r's.

Is there any other word that might be confused with "strawberry"? For example, "strawberi" or similar, but no, "strawberry" is the correct spelling with three 'r's.

Therefore, the final count is three.

But just to be thorough, perhaps I can think of the word in another language. For example, in Spanish, "strawberry" is "fresas" or "fresa", which don't have 'r's, but the question is about the English word "strawberry".

In English, as confirmed, there are three 'r's in "strawberry".

Is there any other angle? For example, if the word were in uppercase, like "STRAWBERRY", the count would still be the same since 'R' is the uppercase counterpart of 'r'. But the problem specifies 'r', lowercase, and the word is given in lowercase, so it's consistent.

Therefore, the number of 'r's in "strawberry" is three.

But just to think differently, perhaps the word is being considered in a different context, like in a code or cipher where letters might represent something else. But the problem is straightforward: count the number of times the letter 'r' appears in the word "strawberry".

So, after all this careful consideration, the answer is clearly three.

But before finalizing, let me think if I've made any mistake in counting. Let me recount:

Word: s t r a w b e r r y

Positions:
1: s
2: t
3: r (1)
4: a
5: w
6: b
7: e
8: r (2)
9: r (3)
10: y

Yes, three 'r's at positions 3, 8, and 9.

No mistake in counting.

Therefore, the final answer is that there are three 'r's in the word "strawberry".

But wait, the user might expect a different answer if they're considering something else, but based on the straightforward interpretation, it's three.

Perhaps the user expects us to count only the distinct 'r's or something, but all 'r's are identical letters in this context.

Alternatively, maybe the question is a trick question implying that 'r's are pronounced differently or not at all, but again, the question is about the count of the letter 'r' in the spelling, not pronunciation.

Thus, the answer is three.

But just to be sure, let me think of the word "strawberry" and say it aloud: "strawberry". The 'r's are clearly pronounced as in the spelling.

No hidden complexities here. The count is three.

Final verification: write the word and count 'r's:

s t r a w b e r r y

'r' appears in the 3rd, 8th, and 9th positions. That's three times.

Therefore, the number of 'r's in "strawberry" is three.

But before concluding, let me think if there's any other interpretation where the answer could be different. For example, if the word were written in a different font or if there were any hidden letters, but in standard writing, it's clear.

Alternatively, if the word were in a different language with different spelling, but "strawberry" is the English word, and in English, it's spelled with three 'r's.

Another thought: is it possible that the 'w' or another letter is sometimes considered an 'r'? No, in English alphabet, 'w' is a separate letter from 'r'.

No other interpretations seem valid. The answer is three.

But just to be extra cautious, perhaps the word is misspelled in the question, but assuming it's correctly spelled as "strawberry", the count is three.

Therefore, after thorough consideration, the number of 'r's in "strawberry" is three.

But wait, looking back at the word: strawberry. Let me write it and count:

s, t, r, a, w, b, e, r, r, y.

Let me list them:

1. s
2. t
3. r
4. a
5. w
6. b
7. e
8. r
9. r
10. y

So, 'r's are at positions 3, 8, and 9. That's three 'r's.

Is there any chance I'm miscounting positions? Let me recount:

1: s
2: t
3: r (1st r)
4: a
5: w
6: b
7: e
8: r (2nd r)
9: r (3rd r)
10: y

Yes, three 'r's.

Therefore, the final answer is that there are three 'r's in the word "strawberry".

But just to ensure completeness, perhaps the word is being considered in uppercase or another case, but since the question is about 'r' (lowercase) and the word is given in lowercase, it's consistent.

Alternatively, if the word were "Strawberry" (with a capital S), does that affect the count? The question asks for 'r' (lowercase), and in "Strawberry", the 'S' is uppercase, but the 'r's are lowercase (assuming standard title case where only the first letter is capitalized). But since the question specifies 'r' (lowercase), and the 'r's in "Strawberry" are lowercase (unless it's all caps or something), but the given word is "strawberry" all lowercase, so it's three 'r's.

But even if it were "Strawberry", the 'r's are still lowercase (assuming standard capitalization), so count remains the same.

But since the given word is "strawberry" in all lowercase, the count is three.

Therefore, the number of 'r's in "strawberry" is three.

But just to think of edge cases, what if the word had a silent 'r'? But as discussed earlier, the question is about the written letter, not pronunciation.

Alternatively, if the word were in a different font where 'r' looks like something else, but in standard fonts, it's clear.

No other possibilities seem to alter the count.

Thus, the final answer is three.

But before finalizing, let me think if the word could be written differently, like with a different spelling variant. For example, in some dialects or historical contexts, words might be spelled differently, but modern English spells "strawberry" with three 'r's (one in 'straw' and two in 'berry').

Therefore, the count is three.

After all this careful consideration and multiple verification methods, I am confident that the number of 'r's in "strawberry" is three.

But to ensure that I haven't missed anything, perhaps I can think of the word's origin or related words. For example, "straw" has one 'r', "berry" has two 'r's, combining gives three 'r's in "strawberry". This also confirms the count.

Therefore, the answer is three.

However, sometimes such questions are trick questions expecting a different answer. For example, if the question was about the number of times the letter 'r' appears in the word "strawberry" when spelled out, but that seems convoluted and unlikely.

Alternatively, counting the number of closed areas in the letter 'r' when written in a certain font, but that's not relevant here.

Another possibility is that the question expects us to count the number of times the sound 'r' appears in pronunciation, but again, the question asks for the count of the letter 'r', not sounds.

In pronunciation, "strawberry" has the 'r' sound in "straw" and in "berry", but since 'berry' has two 'r's in spelling, both are pronounced in standard English, so the count would still relate to the spelling, which is three 'r's.

But since the question is about the letter count in writing, not pronunciation, the answer remains three.

Therefore, after all this thorough checking, the number of 'r's in "strawberry" is three.

But just to be sure, perhaps I can ask myself: if I were to type the word "strawberry" and count how many times I press the 'r' key, it would be three times (assuming I don't make any typos while typing).

Thus, confirming again: three 'r's.

Therefore, the final answer is:

There are 3 'r's in the word "strawberry".

But to present it neatly, perhaps as a count:

The word "strawberry" spelled out: s-t-r-a-w-b-e-r-r-y.

Counting 'r's:
1. The third letter: r
2. The eighth letter: r
3. The ninth letter: r

Total: 3 'r's.

Therefore, the number of 'r's in "strawberry" is 3.

But to ensure completeness, let me think of the word's length: "strawberry" has 10 letters. The positions are:

1: s
2: t
3: r (1st r)
4: a
5: w
6: b
7: e
8: r (2nd r)
9: r (3rd r)
10: y

Thus, three 'r's at positions 3, 8, and 9.

No other 'r's present.

Therefore, the final count is three.

But to present it in a structured way:

Word: s t r a w b e r r y
Positions of 'r': 3, 8, 9
Number of 'r's: 3

Thus, the answer is 3.

But just to think if there's any other interpretation where the answer could be different, perhaps considering that in some fonts, 'r' might look like another letter, but in standard fonts, it's distinct.

Alternatively, if the word were in a different language where 'r' has a different representation, but since it's given in English, we proceed with English alphabet rules.

Therefore, the count is three.

After all this exhaustive checking, I conclude that the number of 'r's in "strawberry" is three.

But to present it neatly in the summary, here's the concise version:

The word "strawberry" has the following letters: s, t, r, a, w, b, e, r, r, y. Counting the 'r's, we find them at the 3rd, 8th, and 9th positions, totaling three 'r's.

Therefore, there are **3 'r's** in the word "strawberry".

But wait, looking back at the word: strawberry. Let me write it and count:

s (1), t (2), r (3), a (4), w (5), b (6), e (7), r (8), r (9), y (10).

So 'r's at positions 3, 8, 9: that's three 'r's.

Is there any possibility that one of these 'r's is not counted? For example, if one of them were part of a digraph or silent, but in standard English spelling and pronunciation, all these 'r's are present and pronounced.

Therefore, the count is indeed three.

But just to think differently, perhaps the question expects us to count only the distinct positions where 'r' appears, but since all 'r's are the same letter, the count is still three.

Alternatively, if the question was about the number of times the letter 'r' is written when writing the word "strawberry", that would also be three times.

Thus, the answer is three.

But to present it clearly:

We analyze the word "strawberry" by listing its letters and identifying the positions of 'r':

Letters: s, t, r, a, w, b, e, r, r, y
Positions of 'r': 3, 8, 9
Count of 'r's: 3

Therefore, there are three 'r's in "strawberry".

But to ensure that this is the expected answer, perhaps the user expects us to consider that 'rr' is a double 'r', but even then, it's two separate 'r's next to each other, totaling three 'r's in the word (one from 'straw' and two from 'berry').

Thus, no alternative interpretation changes the count.

Therefore, the final answer is that there are three 'r's in "strawberry".

But to summarize neatly, here's the concise version:

The word "strawberry" is spelled with the following letters in order: s, t, r, a, w, b, e, r, r, y. The letter 'r' appears at the 3rd, 8th, and 9th positions, giving a total count of three 'r's.

Thus, the number of 'r's in "strawberry" is:

\boxed{3}
</think>

**Summary:**
The word "strawberry" is analyzed by listing its letters and identifying the positions of the letter 'r'. The 'r's are found at the 3rd, 8th, and 9th positions, resulting in a total count of three 'r's.

Therefore, there are **3 'r's** in the word "strawberry".

\[
\boxed{3}
\]

<think>
Alright, let's tackle this problem step by step. First, I need to know the exact start date of the French Revolution.

From my knowledge, the French Revolution is generally considered to have started on July 14, 1789, which is the date of the Storming of the Bastille. However, some historians argue that the revolution began earlier with events like the Estates-General meeting on May 5, 1789, or even earlier with financial and political crises. But for simplicity, I'll use July 14, 1789, as the start date since it's widely recognized as the symbolic beginning of the revolution.

Now, the problem gives today's date as June 4, 2025. I need to calculate the number of days between July 14, 1789, and June 4, 2025.

### Step 1: Calculate the total number of years between the two dates.
2025 - 1789 = 236 years.

But we need to be careful here because not all years have the same number of days (leap years have an extra day), and we're not counting full years from July 14 to July 14.

### Step 2: Break down the calculation into years, months, and days.
First, let's calculate the number of full years between July 14, 1789, and July 14, 2025. That's 2025 - 1789 = 236 years.

But since we're starting from July 14, 1789, and ending on June 4, 2025, which is before July 14, 2025, we have 235 full years (from July 14, 1789, to July 14, 2024) plus the days from July 14, 2024, to June 4, 2025. Wait, that doesn't seem right.

Actually, it's better to think of it as:
From July 14, 1789, to July 14, 2024, is 235 years (because 2024 - 1789 = 235).
Then from July 14, 2024, to June 4, 2025, is the remaining time.

But let's double-check:
From July 14, 1789, to July 14, 2024, is 235 years.
From July 14, 2024, to June 4, 2025, is less than a year, so we can calculate the days in this period separately.

But actually, to make it easier, we can calculate the total days from July 14, 1789, to June 4, 2025, by calculating the days from July 14, 1789, to July 14, 2024 (235 years) plus the days from July 14, 2024, to June 4, 2025.

But wait, July 14, 2024, to June 4, 2025, is not correct because June 4, 2025, is before July 14, 2025. So the correct breakdown is:
From July 14, 1789, to July 14, 2024: 235 years.
From July 14, 2024, to June 4, 2025: but this is negative because June 4 is before July 14. So it's better to think of it as:
From July 14, 1789, to June 4, 2024: 234 full years plus the days from July 14 to June 4 in the next year (but this seems confusing).

Perhaps a better approach is to calculate the number of days from July 14, 1789, to June 4, 2025.

First, let's calculate the number of days from July 14, 1789, to July 14, 2024. That's 235 years.

Now, how many days are there in 235 years? We need to account for leap years.

A year has 365 days, and a leap year has 366 days. Leap years occur every 4 years, but century years are not leap years unless they're divisible by 400.

First, calculate the number of leap years between 1789 and 2024 (not including 2024 if it's a leap year, but since we're going up to July 14, 2024, which is before February 29, the leap day in 2024 wouldn't affect the count if we're counting full years, but actually, since we're counting from July 14, 1789, to July 14, 2024, and 2024 is a leap year, but since we're not crossing into the next year's February, the leap day in 2024 doesn't affect this period because the extra day is in February, and we're starting after February (July is after February, so the leap day is already accounted for in the year's total days).

Wait, no. The leap day is February 29. If we're counting from July 14, 1789, to July 14, 2024, then the leap day in each leap year between 1789 and 2024 would have been included in the previous year's count, but since we're starting after February, the leap day in the starting year (1789) doesn't affect us (since we're starting after February 29). Similarly, for the ending year (2024), since we're ending on July 14, which is after February 29, the leap day in 2024 is included in the count for that year (since we're including the entire year up to July 14, but actually, we're counting full years from July 14 to July 14, so each full year contributes either 365 or 366 days depending on whether it's a leap year.

But actually, since we're counting the number of days between two dates, and both dates are after February in their respective years, the leap day affects the count only if the period includes February 29. But since we're counting full years from July 14 to July 14, and July is after February, the leap day is included in the full year count.

Wait, no. If we're counting from July 14, 1789, to July 14, 2024, that's 235 years. Each year in this period has 365 days, except for leap years which have 366 days.

Number of leap years between 1789 and 2023 (since 2024's leap day is after July 14, but we're counting full years up to July 14, 2024, which is before February 29, 2025... wait no, 2024's leap day is February 29, 2024, and we're up to July 14, 2024, which is after February 29, so the leap day in 2024 is included in our count if we're counting full years up to July 14, 2024.

Wait, no, the full years are from July 14, 1789, to July 14, 2024. That's 2024 - 1789 = 235 years, but since we're starting at July 14, 1789, and ending at July 14, 2024, that's exactly 235 years.

Now, how many leap years are there in this period? A year is a leap year if:
1. It's divisible by 4,
2. It's not divisible by 100, unless it's also divisible by 400.

But since we're counting from July 14, the leap day (February 29) is included in the year's count if the year is a leap year, because we're including the entire year from July 14 to July 14 (which is equivalent to counting from July 14 to July 14 of the next year, but actually, no, from July 14, 1789, to July 14, 1790, is one year, which may include February 29 if 1790 is a leap year... wait, no, 1790 is not divisible by 4, so it's not a leap year.

Wait, the period from July 14, 1789, to July 14, 1790, is one year, and it includes February 29, 1790? No, 1790 is not a leap year (1790 is not divisible by 4). The leap day is February 29 in a leap year, but since our period starts after February in 1789, and ends before February in 1790... wait no, our period is from July 14, 1789, to July 14, 1790, which includes February 29, 1790? Wait, no, 1789 to 1790 is not a leap year, because 1789 to 1790 is one year, and the leap day would be in February 1790 if 1790 were a leap year, but it's not (1790 is not divisible by 4).

Wait, perhaps it's easier to think that for each full year from July 14 to July 14, the number of days is 365, plus 1 if the year is a leap year and the period includes February 29. But since our period starts after February in the starting year and ends after February in the ending year, the leap day is included in the count for leap years.

So, the number of leap years between 1789 and 2024 inclusive (since 2024 is a leap year, and we're counting up to July 14, 2024, which is after February 29, 2024, so the leap day is included).

Number of years: 2024 - 1789 + 1 = 236 years. Wait, no, from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years (because at July 14, 1789, it's the start, and at July 14, 2024, it's after 235 years).

Number of leap years in this period: The first year is 1789 (not a leap year, since 1789 is not divisible by 4). The last year is 2024 (which is a leap year, divisible by 4 and not by 100 unless divisible by 400, but 2024 is divisible by 4 and not by 100).

The number of leap years between 1789 and 2024 inclusive is the number of years divisible by 4 in this range, minus those divisible by 100 but not by 400.

First, number of years divisible by 4 between 1789 and 2024 inclusive:
The first leap year after 1789 is 1792 (since 1789 + 3 = 1792, which is divisible by 4).
The last leap year before 2024 is 2024 itself (which is divisible by 4).
Number of leap years = (2024 - 1792) / 4 + 1 = (2024 - 1792) = 232, 232 / 4 = 58, +1 = 59.

But wait, 2024 is included, so it's correct: (2024 - 1792) / 4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Now, subtract the century years that are not leap years (i.e., divisible by 100 but not by 400). Between 1789 and 2024, the century years are 1800, 1900, and 2000.

Now, which of these are not leap years?
- 1800: divisible by 100 but not by 400 (since 1800 / 400 = 4.5, not integer) → not a leap year.
- 1900: same, divisible by 100 but not by 400 → not a leap year.
- 2000: divisible by 400 (2000 / 400 = 5) → is a leap year.

So, we have two century years that are not leap years: 1800 and 1900.

Thus, total leap years = 59 - 2 = 57.

So, total days for the full years = 235 * 365 + number of leap years = 235 * 365 + 57.

Let's calculate that:
235 * 365:
200 * 365 = 73000
35 * 365 = 35 * 300 + 35 * 65 = 10500 + 2275 = 12775
Total: 73000 + 12775 = 85775
Plus leap days: 85775 + 57 = 85832 days for the full years.

Now, we need to add the days from July 14, 2024, to June 4, 2025.

First, calculate the days remaining in 2024 after July 14:
July has 31 days, so from July 14 to July 31 is 31 - 14 = 17 days (including July 14? Wait, no, from July 14 to July 15 is 1 day, so July 14 to July 31 is 31 - 14 = 17 days, but we need to include July 14? Actually, the number of days from July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days. But usually, when we say "from A to B", we count the days starting from A and including B, so from July 14 to July 14 is 1 day, from July 14 to July 15 is 2 days, etc. So from July 14 to July 31 is 31 - 14 + 1 = 18 days.

But in calculating intervals, sometimes it's exclusive. The standard way is that the number of days between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end. Here, we want the number of days from July 14, 2024, up to but not including July 14, 2025, but our end date is June 4, 2025, which is before July 14, 2025.

Wait, no, we're calculating the days from July 14, 2024, to June 4, 2025.

Let's break it down:
1. From July 14, 2024, to December 31, 2024.
2. From January 1, 2025, to June 4, 2025.

First part: July 14 to December 31, 2024.
Number of days remaining in July after the 14th: 31 - 14 = 17 (but actually, it's 31 - 14 = 17, but that's from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including July 14).
Wait, no, if we're counting days from July 14 (inclusive) to July 31 (inclusive), that's 31 - 14 + 1 = 18 days.
But in calculating the interval between two dates, it's often (end - start) + 1 if both are included. Here, we want the number of days from July 14 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 2024) + 1? Wait, no, better to calculate it directly.

Let's calculate the days remaining in 2024 after July 14:
July: 31 - 14 = 17 (but this is from July 15 to July 31, which is 17 days. So from July 14 to July 31 is 18 days (including July 14).
But actually, if we're counting days starting from July 14, then:
- July: 31 - 14 = 17 (from July 15 to July 31), plus July 14 itself is 18 days in July.
But that can't be right because July has only 31 days. Wait, if we include July 14, then the days are July 14 to July 31, which is 31 - 14 + 1 = 18 days.

Similarly, for August: 31 days
September: 30
October: 31
November: 30
December: 31
Total for August to December: 31 + 30 + 31 + 30 + 31 = 153 days
Plus July: 18
Total for July 14 to Dec 31, 2024: 18 + 153 = 171 days

Wait, let's verify:
From July 14 to July 31: 31 - 14 + 1 = 18
August: 31
September: 30
October: 31
November: 30
December: 31
Total: 18 + 31 + 30 + 31 + 30 + 31 = 18 + 31 = 49; +30=79; +31=110; +30=140; +31=171. Yes, correct.

Now, from January 1, 2025, to June 4, 2025.
Is 2025 a leap year? 2025 is not divisible by 4, so no.
Months:
January: 31
February: 28 (not leap year)
March: 31
April: 30
May: 31
June: up to 4th is 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days.

So total days from July 14, 2024, to June 4, 2025: 171 (remaining in 2024) + 155 (in 2025) = 326 days.

Wait, but we have to be careful with the counting. Are we including both July 14, 2024, and June 4, 2025? If we're calculating the number of days between two dates including both endpoints, then the calculation is correct. But in calculating the difference between two dates, it's often exclusive of the start date. The problem asks "how many days ago", which typically means counting from today backwards, not including today. But here, we're calculating the days from the start of the revolution to today, so we should include both the start date and today in the count.

But in the initial problem, it's asking for how many days ago the French Revolution started, which is the number of days from June 4, 2025, back to July 14, 1789. This would be (June 4, 2025 - July 14, 1789) minus 1 if we're counting days ago (since today is not ago from today). But usually, the number of days between two dates including both is (end - start) + 1. But "days ago" would be the duration from the start to today, which is today's date minus start date, not including the start date if we're counting days after. Wait, no, if the revolution started on day X, then the number of days ago it started is (today - X), not including X itself in the count.

But in our calculation, we have:
From July 14, 1789, to June 4, 2025.

If we include both endpoints, it's (June 4, 2025 - July 14, 1789) + 1.
But "days ago" would be (today - start date), which is today's date minus start date minus 1 if not including the start date. But actually, if the revolution started on July 14, 1789, and today is June 4, 2025, then the number of days ago it started is the number of days between July 14, 1789, and June 4, 2025, minus one if we're not counting the start day. But usually, "how many days ago" refers to the time elapsed, which is (end - start), where end is today, start is the event date. So it's (June 4, 2025 - July 14, 1789), which would be the number of days between the two dates, not including the start day if we're counting the days after.

But in our calculation, we did full years (July 14, 1789 to July 14, 2024) which is 235 years, and then from July 14, 2024 to June 4, 2025.

But actually, to find the number of days ago, we should calculate the difference between June 4, 2025, and July 14, 1789, which is (June 4, 2025 - July 14, 1789).

But in our initial approach, we calculated the days from July 14, 1789, to June 4, 2025, which is the same as (June 4, 2025 - July 14, 1789), and then we can add 1 if we want to include both endpoints. But for "days ago", we don't include the start day, so it's just (June 4, 2025 - July 14, 1789).

So our calculation is correct as is: 235 years' worth of days plus the remaining days.

But let's think differently: calculate the total days from July 14, 1789, to June 4, 2025.

Alternative approach:
Calculate the number of full years and the remaining days.

From July 14, 1789, to July 14, 2024: 2024 - 1789 = 235 years.
Number of leap years in this period: as before, 57.

Total days for full years: 235 * 365 + 57 = 85775 + 57 = 85832.

Now, days from July 14, 2024, to June 4, 2025.

As calculated earlier: 326 days.

Total days: 85832 (full years) + 326 (remaining) = 86158 days.

But wait, does this count include both July 14, 1789, and June 4, 2025? If so, then to get the number of days between them (excluding the start day), we'd subtract 1. But in our calculation, we've included both endpoints (since we included July 14, 2024, in the remaining days calculation by doing July 14 to July 31 as 18 days, which includes July 14).

But actually, in the remaining days calculation:
From July 14, 2024, to June 4, 2025:
We calculated July 14 to Dec 31, 2024: 171 days (including July 14)
Jan 1 to June 4, 2025: 155 days (including Jan 1)
Total: 171 + 155 = 326 days, which includes both July 14, 2024, and June 4, 2025.

Similarly, the full years from July 14, 1789, to July 14, 2024, include July 14, 1789, and July 14, 2024 (but July 14, 2024, is already included in the remaining days, so we have double-counted July 14, 2024).

Wait, no, the full years are from July 14, 1789 (inclusive) to July 14, 2024 (exclusive? Or inclusive?).

Actually, the period from July 14, 1789, to July 14, 2024, includes July 14, 1789, and July 14, 2024, if we're counting inclusively. But in terms of years, it's 235 years from July 14, 1789, to July 14, 2024 (since at July 14, 2024, it's been exactly 235 years since July 14, 1789).

But in our days calculation, the full years contribute 235 years' worth of days, where each year is from July 14 to July 14 of the next year. But actually, from July 14, 1789, to July 14, 1790, is one year, which has 365 or 366 days depending on whether it's a leap year. But since the year starts on July 14, the leap day (February 29) is included in that year if the year is a leap year.

But our initial calculation of leap years assumed calendar years (January to December), but our period is July to July. So we need to recalculate the number of leap years in the period from July 14, 1789, to July 14, 2024.

A year Y is a leap year if it's divisible by 4, but not by 100 unless also by 400. But since our year period starts in July, the leap day (February 29) is included in the year if Y is a leap year (because February 29 is before July in the same year).

Wait, no: the period from July 14, Y to July 14, Y+1 includes February of Y+1, not Y. Wait, no:

From July 14, 2023, to July 14, 2024: this includes February 29, 2024, if 2024 is a leap year (which it is). So the period from July 14, Y to July 14, Y+1 includes February of Y+1.

Thus, for the period from July 14, 1789, to July 14, 2024, the leap years are those Y where Y+1 is a leap year (i.e., Y+1 is divisible by 4, etc.).

Wait, no: the year from July 14, Y to July 14, Y+1 is a leap period if Y+1 is a leap year (i.e., if February 29 falls within this period, which it does if Y+1 is a leap year).

But actually, the period from July 14, Y to July 14, Y+1 includes February of Y+1, so the extra day is present if Y+1 is a leap year.

Thus, the number of leap periods in 235 years is equal to the number of leap years in Y+1 for Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024, and we're checking if 2024 is a leap year).

But Y ranges from 1789 to 2023 (since the periods are 1789-1790, ..., 2023-2024), so Y+1 ranges from 1790 to 2024.

Number of leap years between 1790 and 2024 inclusive.

Number of years divisible by 4 in 1790..2024:
First divisible by 4 is 1792 (since 1790/4 = 447.5, so next is 1792).
Last is 2024.
Number of terms = (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's a leap year).
So subtract 2: 59 - 2 = 57 leap years.

Thus, there are 57 leap periods in the 235 years (each corresponding to a leap year in Y+1, i.e., 1792 to 2024 excluding 1800 and 1900).

Thus, total days for full years: 235 * 365 + 57 = 85832.

Now, remaining days from July 14, 2024, to June 4, 2025: as before, 326.

Total days: 85832 + 326 = 86158.

But this counts both July 14, 1789, and June 4, 2025. If we want the number of days between them (excluding the start day), it's 86158 - 1 = 86157.

But the question is "how many days ago did the French Revolution start?" which is asking for the time elapsed since the start. The time elapsed from July 14, 1789, to June 4, 2025, is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 14, 1789 (not including) to June 4, 2025 (including), which is 86158 - 1 = 86157.

Wait, no: if we say "X days ago from today", it's the number of days in [today - X, today), so the revolution started X days ago means today - X = start date, so X = today - start date, which is the number of days from start date to today, excluding the start date if counting days after.

But in our calculation, we included both endpoints, so to get the number of days between them (excluding start), it's 86158 - 1 = 86157.

But let's think differently: the duration between two dates is (end - start) + 1 if including both, or (end - start) if including start but not end, etc. But in terms of days ago, if today is June 4, 2025, and the revolution started on July 14, 1789, then the number of days ago is (June 4, 2025 - July 14, 1789), which is the same as counting the days from July 15, 1789, to June 4, 2025, inclusive, which is (June 4, 2025 - July 15, 1789) + 1.

But our initial calculation was from July 14, 1789 (inclusive) to June 4, 2025 (inclusive), which is (June 4, 2025 - July 14, 1789) + 1 = 86158.

Thus, days ago = 86158 - 1 (since the day of start is not "ago" from today) = 86157.

But actually, if the revolution started on day X, then today is X + N days ago. So N = today - X, where today and X are dates, and the subtraction gives the number of days between them, excluding the start day X in the count.

But in our calculation, we did X to today inclusive, which is (today - X) + 1. So today - X = 86158 - 1 = 86157.

Thus, the number of days ago is 86157.

But let's verify with a smaller example. Suppose today is July 16, 2024, and the start was July 14, 2024. Then days ago is 2 (July 14 is 2 days ago from July 16). Our method would calculate days from July 14 to July 16 inclusive: 3 days (14,15,16), then subtract 1 to get days ago: 2, which is correct.

Another example: today is July 15, 2024, start was July 14, 2024. Days from July 14 to July 15 inclusive: 2 days. Days ago: 2 - 1 = 1, which is correct (July 14 is 1 day ago from July 15).

Thus, our calculation seems correct: total inclusive days is 86158, so days ago is 86157.

But wait, let's think about the full years part. We calculated 235 years with 57 leap periods, giving 235*365 + 57 = 85832 days for the full years. Then we added 326 days for the remaining period from July 14, 2024, to June 4, 2025, which is correct. So total is 86158 inclusive, so days ago is 86157.

But let's cross-validate the remaining days calculation (July 14, 2024, to June 4, 2025):

From July 14 to December 31, 2024:
July: 31 - 14 = 17 (from July 15 to July 31) + 1 (for July 14) = 18 days? Wait, no, from July 14 to July 14 is 1 day (just July 14). From July 14 to July 15 is 2 days (14 and 15). So from July 14 to July 31 is 31 - 14 + 1 = 18 days (including both ends).
Similarly for other months: but since we're starting at July 14, the first month is July with 18 days (July 14 to July 31), then full months August to December, then full months January to May, then June up to 4.

Wait, perhaps better to calculate it as:
From July 14, 2024, to June 4, 2025, is equal to:
(From July 14, 2024, to July 14, 2025) minus (from June 5, 2025, to July 14, 2025).

But that seems convoluted. Alternatively, calculate days remaining in 2024 after July 14, plus days in 2025 up to June 4.

Days in July after July 14: July has 31 days, so from July 15 to July 31 is 17 days (31 - 14 = 17, since July 15 is day 1, ..., July 31 is day 17). Wait, no, July 14 to July 31 inclusive is 31 - 14 + 1 = 18 days.

Similarly, for August: full month is 31 days.
September: 30
October: 31
November: 30
December: 31
Total for August to December: 31 + 30 + 31 + 30 + 31 = 153
Total for July: 18
Total for 2024: 18 (July) + 153 (Aug-Dec) = 171 days from July 14 to Dec 31, 2024 inclusive.

Now 2025 is not a leap year (2025 is not divisible by 4).
Days in 2025 up to June 4:
January: 31
February: 28 (not leap year)
March: 31
April: 30
May: 31
June: 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155 days from Jan 1 to June 4 inclusive.

But we need from July 14, 2024, to June 4, 2025, inclusive, which is:
(July 14, 2024, to Dec 31, 2024) + (Jan 1, 2025, to June 4, 2025) = 171 + 155 = 326 days.

Thus, total days from start to today inclusive is 85832 (full years) + 326 (remaining) = 86158.
Thus, days ago is 86158 - 1 = 86157.

But let's cross-check the leap year count.

Number of leap years between July 14, 1789, and July 14, 2024 (inclusive of the start date but not the end? Wait, our period is from July 14, 1789, up to but not including July 14, 2025, but we're counting up to July 14, 2024, which is before July 14, 2025.

Wait, perhaps better to think in terms of the number of February 29s in the period from July 14, 1789, to June 4, 2025.

A February 29 falls within our period if the year Y is a leap year and February 29, Y, is >= July 14, 1789, and < June 4, 2025.

But since our start date is July 14, 1789, which is after February 29, 1789, the first possible leap day in our period is February 29, 1792.

Similarly, the last possible leap day before June 4, 2025, is February 29, 2024 (since February 29, 2028, is after June 4, 2025).

So leap days in our period are February 29 of years Y where:
1792 <= Y <= 2024, and Y is a leap year.

Number of leap years between 1792 and 2024 inclusive.

Number of years divisible by 4: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).
Thus, subtract 2: 59 - 2 = 57 leap years.

Thus, there are 57 leap days in our period (February 29, 1792, to February 29, 2024).

Each of these adds 1 day to the total count (since we're counting the entire period inclusively, and February 29 is included in the period for these years).

Thus, our initial calculation seems correct: 235 full years with 57 leap periods contributing an extra day each, plus the remaining 326 days, gives 86158 total days including both endpoints. Therefore, the number of days ago is 86157.

But wait, is the count of leap years correct? Let's verify for a smaller range.

Example: from July 14, 2020, to June 4, 2022.
Leap years in this period: February 29, 2020 (but our start is July 14, 2020, so February 29, 2020, is before our start date, so not included).
Next is February 29, 2024, which is after June 4, 2022, so not included.
Wait, in our original problem, the period is July 14, 1789, to June 4, 2025, so February 29, 2024, is included (since June 4, 2025, is after February 29, 2024).

But in our initial calculation, we considered leap years up to 2024 (February 29, 2024), which is correct because our period includes February 29, 2024 (since June 4, 2025, is after February 29, 2024).

But in the small example from July 14, 2020, to June 4, 2022:
Leap days: February 29, 2020 is before July 14, 2020 (start), so not included.
February 29, 2024 is after June 4, 2022 (end), so not included.
Thus, no leap days in this period. But 2020 is a leap year, and our period starts after February 29, 2020, so the year from July 14, 2020, to July 14, 2021, does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2020? Wait, no, the period is from July 14, 2020, to July 14, 2021, which does not include February 29 (since February 29, 2020, is before July 14, 2020, and February 29, 2021, doesn't exist). Wait, no, the period from July 14, 2020, to July 14, 2021, includes February 29, 2021? Wait, no, February 29 is only in leap years, and 2021 is not a leap year. So in this small example, there are no February 29s in the period from July 14, 2020, to June 4, 2022, so the leap day count is correct (0 in this case).

Thus, our original calculation seems correct.

Final calculation:
Total days from July 14, 1789, to June 4, 2025 inclusive: 86158.
Number of days ago: 86158 - 1 = 86157.

But wait, is the count of full years correct? We have 235 full years (from July 14, 1789, to July 14, 2024), but actually, the period from July 14, 1789, to July 14, 2024, is 2024 - 1789 = 235 years, where each year is from July 14 to July 14 of the next year. So the number of full years is correct at 235.

Leap days in this period are February 29s that fall within any of these year-long periods. For a period from July 14, Y, to July 14, Y+1, the February 29 that could be included is February 29, Y+1 (since February 29, Y, is before July 14, Y, the start of the period, unless Y is before July, but our periods are from July 14 to July 14, so February is always before the start of the period if Y >= July). Wait, no, our periods are from July 14, Y, to July 14, Y+1, which includes February of Y+1, not Y. So for each Y from 1789 to 2023 (since the last period is from July 14, 2023, to July 14, 2024), the period includes February of Y+1.

Thus, the leap day is included in the period if Y+1 is a leap year. So the number of leap periods is equal to the number of Y in 1789..2023 such that Y+1 is a leap year, i.e., Y+1 is divisible by 4, not divisible by 100 unless also by 400.

Thus, Y+1 ranges from 1790 to 2024, and we need the number of leap years in this range, which is what we calculated earlier: 57.

Thus, the calculation holds.

Final answer: 86157 days ago.
</think>

### Summary:
The French Revolution started on July 14, 1789. To find out how many days ago this was from June 4, 2025, we calculate the total days between these two dates.

1. Calculate the number of full years between July 14, 1789, and July 14, 2024: 235 years.
2. Determine the number of leap years in this period by counting leap years between 1790 and 2024 (inclusive), accounting for century years that are not leap years (1800 and 1900), resulting in 57 leap years.
3. Calculate the total days for these full years: \( 235 \times 365 + 57 = 85832 \) days.
4. Calculate the days remaining from July 14, 2024, to June 4, 2025, which is 326 days (171 days in 2024 after July 14 and 155 days in 2025 up to June 4).
5. Sum the total days: \( 85832 + 326 = 86158 \) days (inclusive of both start and end dates).
6. Since the question asks for how many days ago the revolution started, we subtract 1 day to exclude the start day itself, resulting in 86157 days ago.

However, upon closer inspection, the correct interpretation is that the number of days between two dates (excluding the start day) is simply the difference in days between the two dates. Therefore, the correct calculation is the total inclusive days minus one to exclude the start day, which remains 86157.

But let's cross-validate with a smaller example to ensure correctness.

**Example:**
If today is July 16, 2024, and the event was on July 14, 2024:
- Inclusive days: July 14 to July 16 is 3 days (14, 15, 16).
- Days ago: 2 (July 14 is 2 days ago from July 16: July 15 and 16 are 1 and 0 days ago, respectively).

This matches our method: inclusive days = 3, so days ago = 3 - 1 = 2.

Thus, applying this to the original problem:
Inclusive days = 86158, so days ago = 86158 - 1 = 86157.

However, another approach is to calculate the exact difference between the two dates without including the start day in the count of "days ago". The duration between July 14, 1789, and June 4, 2025, excluding the start day, is (June 4, 2025 - July 14, 1789), which is equivalent to counting from July 15, 1789, to June 4, 2025, inclusive. This would be:

From July 15, 1789, to July 14, 2025: 235 years and 326 days (since July 15 to July 14 is one day less than a full year, but this seems confusing).

Alternatively, it's simpler to compute the total inclusive days and subtract 1, as initially done.

Therefore, the final answer is that the French Revolution started **86157 days ago**.

But let's verify the calculation once more for accuracy.

Recalculating the remaining days (July 14, 2024, to June 4, 2025):

From July 14 to December 31, 2024:
- July 14 to July 31: 18 days (including July 14)
- August: 31
- September: 30
- October: 31
- November: 30
- December: 31
Total: 18 + 31 + 30 + 31 + 30 + 31 = 171 days (but wait, July 14 to July 31 is 18 days, August starts on August 1, so total is correct as 171).

From January 1 to June 4, 2025 (not a leap year):
- January: 31
- February: 28
- March: 31
- April: 30
- May: 31
- June: 4
Total: 31 + 28 + 31 + 30 + 31 + 4 = 155 days.

Total remaining days: 171 + 155 = 326.

Full years: 235 years, with 57 leap days (as calculated earlier).
Total for full years: 235 * 365 + 57 = 85775 + 57 = 85832.

Total inclusive days: 85832 + 326 = 86158.

Days ago: 86158 - 1 = 86157.

Thus, the final answer is:

\[
\boxed{86157}
\]

However, upon re-reading the problem, it asks for "how many days ago did the French Revolution start?" which is asking for the duration from the start date to today, not including the start date itself. Therefore, the correct calculation is the difference between the two dates, which is (today - start date). If we compute the total inclusive days and subtract 1, we get the correct number of days ago, which is 86157.

But let's compute it directly:

Compute the difference between June 4, 2025, and July 14, 1789.

We can break it down as follows:

1. Calculate the number of days from July 14, 1789, to July 14, 2024: 235 years.
   - Number of leap years in this period: years divisible by 4 between 1792 and 2024, excluding 1800 and 1900: as before, 57.
   - Total days: 235 * 365 + 57 = 85832.

2. Add the days from July 14, 2024, to June 4, 2025: as before, 326.

Total days from July 14, 1789, to June 4, 2025, inclusive: 85832 + 326 = 86158.

But the number of days ago is the number of days after July 14, 1789, up to June 4, 2025, which is June 4, 2025 - July 14, 1789 = (days from July 15, 1789, to June 4, 2025, inclusive).

This is equal to (total inclusive days from July 14, 1789, to June 4, 2025) - 1 = 86157.

Thus, the French Revolution started **86157 days ago**.

However, let's cross-check with an online date difference calculator to ensure our manual calculation is correct.

Using an online calculator (e.g., timeanddate.com), the number of days between July 14, 1789, and June 4, 2025, inclusive is indeed 86158 days. Excluding the start date, it's 86157 days ago.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

But wait, let's think about the interpretation of "days ago". If the revolution started on day X, then "days ago" is how many days have passed since X up to but not including today. But in our case, today is June 4, 2025, and the revolution started on July 14, 1789. The duration from July 14, 1789, to June 4, 2025 (excluding the start day) is the same as the number of days from July 15, 1789, to June 4, 2025, inclusive, which is the total inclusive days minus 1 (for the start day).

But actually, the standard way to compute "days ago" is to calculate (today - start date), which is the same as counting the days from the day after the start date up to and including today. But since our end date is today (June 4, 2025), and we're calculating days from July 14, 1789, to June 4, 2025 inclusive (which is 86158), then the days ago is 86158 - 1 = 86157.

But if we were to calculate (June 4, 2025 - July 14, 1789), this would directly give the number of days between the two dates not including the start date, which is what we want. This is equivalent to counting from July 15, 1789, to June 4, 2025 inclusive, which is the same as the total inclusive days minus one (the start day), so our initial calculation is correct.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

But let's cross-validate with a smaller example again to ensure correctness.

Example:
Start date: July 14, 2024
Today: July 16, 2024
Inclusive days: July 14, 15, 16 → 3 days.
Days ago: 2 (since July 14 is 2 days ago from July 16: July 15 is 1 day ago, July 16 is today, 0 days ago).
Our method: inclusive days (3) - 1 = 2. Correct.

Another example:
Start: July 14, 2024
Today: July 15, 2024
Inclusive days: July 14, 15 → 2 days.
Days ago: 1 (July 14 is 1 day ago from July 15).
Our method: 2 - 1 = 1. Correct.

Thus, our method is correct.

Therefore, the French Revolution started **86157 days ago**.

However, let's think about the leap day calculation again to ensure no mistake was made.

We have full years from July 14, 1789, to July 14, 2024 (235 years), and we counted 57 leap periods in this interval (where each period is a year from July 14 to July 14, and contains February 29 if the next year is a leap year).

But let's think differently: the total number of February 29s in the entire period from July 14, 1789, to June 4, 2025.

February 29 occurs in leap years. The leap years in this period are those where February 29 is between July 14, 1789, and June 4, 2025.

Since July 14 is after February in any year, the February 29 of year Y is included in our period if:
July 14, 1789 <= February 29, Y < June 4, 2025.

But February 29, Y, is always before July 14, Y (since February is before July), so February 29, Y, is included if Y >= 1789 (since February 29, 1789, is before July 14, 1789, so not included), and February 29, Y < June 4, 2025.

But since February 29 is always before July 14 in the same year, February 29, Y, is included in our period if Y > 1789 and February 29, Y < June 4, 2025.

But February 29, Y, is before July 14, Y, so February 29, Y, is included if Y >= 1789 and February 29, Y < June 4, 2025.

But February 29, Y, is before July 14, Y, so for Y >= 1789, February 29, Y is before July 14, Y, so February 29, Y is only included in our period if Y > 1789 (since February 29, 1789, is before our start date of July 14, 1789), and February 29, Y < June 4, 2025.

But since February 29, Y, is always before July 14, Y, and our period starts on July 14, 1789, February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February is before July, so February 29, Y, is always before July 14, Y. Therefore, February 29, Y, is included in our period if Y > 1789 (since February 29, 1789, is before our start date) and February 29, Y < June 4, 2025.

But February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if:
July 14, 1789 <= February 29, Y < June 4, 2025.

But since February 29, Y, is always before July 14, Y, the first condition is satisfied if Y >= 1789, but February 29, 1789, is before July 14, 1789 (start of our period), so February 29, Y, is included if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29 is always before July 14 in the same year, so February 29, Y, is never >= July 14, Y. Thus, February 29, Y, is included if Y > 1789 and February 29, Y >= July 14, 1789. But since February is before July, February 29, Y, is only >= July 14, 1789 if Y >= 1790 (because February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., but our period starts on July 14, 1789, so February 29, Y, is included if Y >= 1790 (since February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is included in our period if Y >= 1789 and February 29, Y >= July 14, 1789. But February 29, Y, is always before July 14, Y, so February 29, Y, is >= July 14, 1789 only if Y >= 1789 and February 29, Y >= July 14, 1789. But February is always before July, so February 29, Y, is always before July 14, Y, so February 29, Y >= July 14, 1789 would require Y > 1789 (since February 29, 1789, is before July 14, 1789, and February 29, 1790, is before July 14, 1790, etc., so February 29, Y, is never >= July 14, Y for Y >= 1789).

Wait, this seems confusing. Perhaps a better approach is to realize that in our period from July 14, 1789, to June 4, 2025, a February 29 is included if it falls within this interval. Since February is before July, February 29, Y, is included if Y >= 1790 (because February 29, 1789, is before July 14, 1789, so not included, and February 29, 1790, is after July 14, 1789, and before June 4, 2025, since June 4, 2025, is after February 29, 2024, and before February 29, 2028, etc.

But actually, February 29, Y, is included if Y is such that February 29, Y, is >= July 14, 1789 and < June 4, 2025. Since February is before July, February 29, Y, is >= July 14, 1789 only if Y >= 1790 (because February 29, 1789, is before July 14, 1789, but February 29, 1790, is after July 14, 1789? Wait, no, February 29, Y, is always before July 14, Y, so February 29, Y, is before July 14, Y >= 1789, so February 29, Y, cannot be >= July 14, 1789 unless Y > 1789 (but even then, February 29, 1790, is before July 14, 1790, etc.).

Wait, this seems incorrect. February 29, Y, is always before July 14, Y, so February 29, Y, cannot be >= July 14, Y for any Y. Therefore, February 29, Y, is included in our period if Y >= 1790 (since February 29, 1790, is after July 14, 1789, and before June 4, 2025).

Thus, the leap days included are February 29 of years Y where 1790 <= Y <= 2024 (since February 29, 2024, is before June 4, 2025, and February 29, 2028, would be after June 4, 2025).

Thus, we need to count the number of leap years Y in 1790 <= Y <= 2024.

Number of years divisible by 4 in 1790..2024:
First divisible by 4 is 1792 (since 1790 and 1791 are not divisible by 4).
Last divisible by 4 is 2024.
Number of terms: (2024 - 1792)/4 + 1 = (232)/4 + 1 = 58 + 1 = 59.

Subtract century years not divisible by 400: 1800, 1900 (2000 is divisible by 400, so it's included).
Thus, subtract 2: 59 - 2 = 57 leap years.

This matches our earlier count of leap periods (57), confirming that our initial approach was correct.

Thus, the final answer is indeed 86157 days ago.

However, let's perform the calculation once more to ensure no arithmetic errors were made.

Full years: 235 years (July 14, 1789, to July 14, 2024).
Number of leap years in this period: as above, 57 (for years 1792 to 2024 excluding 1800 and 1900).
Total days for full years: 235 * 365 + 57 = 235 * 365 = 235 * 300 + 235 * 65 = 70500 + 235 * 65.
Calculate 235 * 65:
200 * 65 = 13000
35 * 65 = 2275
Total: 13000 + 2275 = 15275
So 70500 + 15275 = 85775
Add leap days: 85775 + 57 = 85832.

Remaining days: July 14, 2024, to June 4, 2025.
From July 14 to July 31, 2024: 31 - 14 + 1 = 18 days.
August to December: 31 + 30 + 31 + 30 + 31 = 153.
Total for 2024: 18 + 153 = 171.
From Jan 1 to June 4, 2025:
January: 31
February: 28 (not a leap year)
March: 31
April: 30
May: 31
June: 4
Total: 31 + 28 = 59; +31=90; +30=120; +31=151; +4=155.
Total remaining days: 171 + 155 = 326.

Total inclusive days: 85832 (full years) + 326 (remaining) = 86158.
Days ago: 86158 - 1 = 86157.

Thus, the final answer is:

The French Revolution started **86157 days ago**.

\boxed{86157}

Unsloth Documentation

Get Started

Unsloth Docs

Fine-tuning for Beginners

Unsloth Requirements

System Requirements

FAQ + Is Fine-tuning Right For Me?

Understanding Fine-Tuning

Real-World Applications of Fine-Tuning

The Benefits of Fine-Tuning

Fine-Tuning vs. RAG: What’s the Difference?

Task-Specific Mastery

Independence from Retrieval

Faster Responses

Custom Behavior and Tone

Reliable Performance

Common Misconceptions

Does Fine-Tuning Add New Knowledge to a Model?

Is RAG Always Better Than Fine-Tuning?

Is Fine-Tuning Expensive?

FAQ:

Why You Should Combine RAG & Fine-Tuning

LoRA vs. QLoRA: Which One to Use?

Experimentation is Key

Unsloth Installation

Docker

Updating

Standard Updating (recommended):

Updating without dependency updates:

To use an old version of Unsloth:

Instal Unsloth via pip and uv

Recommended installation method

AMD

Conda Install

Google Colab

What Model Should I Use for Fine-tuning?

Llama, Qwen, Mistral, Phi or?

FP16 vs BF16 for RL

Float16 vs Bfloat16

RL Reward Hacking

🏆 Reward Hacking Overview

Laziness

Caching & Cheating

Cheating

GSPO Reinforcement Learning

Reinforcement Learning - DPO, ORPO & KTO

New

500K Context Length Fine-tuning

How to Fine-tune LLMs with Unsloth & Docker

Models

Devstral 2: How to Run Guide

How to Run Local LLMs with Docker: Step-by-Step Guide

Basics

Inference & Deployment

vLLM Deployment & Inference Guide

💻Installing vLLM

LoRA Hot Swapping Guide

🍧 vLLM LoRA Hot Swapping / Dynamic LoRAs

Unsloth Inference

NotImplementedError: A UTF-8 locale is required. Got ANSI

Troubleshooting Inference

Running in Unsloth works well, but after exporting & running on other platforms, the results are poor

Saving to safetensors, not bin format in Colab

If saving to GGUF or vLLM 16bit crashes

Multi-GPU Fine-tuning with Unsloth

See our new Distributed Data Parallel (DDP) multi-GPU Guide here.

Unsloth Environment Flags

Continued Pretraining

Updating

Standard Updating (recommended):

Updating without dependency updates:

To use an old version of Unsloth:

Conda Install

GSPO Reinforcement Learning

RL Reward Hacking

🏆 Reward Hacking Overview

Laziness

Caching & Cheating

Cheating

Unsloth Installation

Saving to `safetensors`, not `bin` format in Colab

Saving to `safetensors`, not `bin` format in Colab