👁️Vision Fine-tuning

Details on vision/multimodal fine-tuning with Unsloth

Fine-tuning vision models has numerous use cases across various industries, enabling models to adapt to specific tasks and datasets. We provided 3 example notebooks for vision finetuning.

Note: Gemma 3 also works, just change Qwen or Pixtral's notebook to the Gemma 3 model.

1. Llama 3.2 Vision finetuning for radiography: Notebook How can we assist medical professionals in analyzing Xrays, CT Scans & ultrasounds faster.

2. Qwen2.5 VL finetuning for converting handwriting to LaTeX: Notebook This allows complex math formulas to be easily transcribed as LaTeX without manually writing it.

3. Pixtral 12B 2409 vision finetuning for general Q&A: Notebook One can concatenate general Q&A datasets with more niche datasets to make the finetune not forget base model skills.

It is best to ensure your dataset has images of all the same size/dimensions. Use dimensions of 300-1000px to ensure your training does not take too long or use too many resources.

To finetune vision models, we now allow you to select which parts of the mode to finetune. You can select to only finetune the vision layers, or the language layers, or the attention / MLP layers! We set them all on by default!