Unsloth Multi-GPU support

Learn how to fine-tune LLMs on multiple GPUs and parallelism with Unsloth.

Unsloth currently supports multi-GPU setups through libraries like Accelerate and DeepSpeed. This means you can already leverage parallelism methods such as FSDP and DDP with Unsloth.

However, we know that the process can be complex and requires manual setup. We’re working hard to make multi-GPU support much simpler and more user-friendly, and we’ll be announcing official multi-GPU support for Unsloth soon.

In the meantime, several contributors have created repos to enable or improve multi-GPU support with Unsloth, including:

• unsloth-5090-multiple: A fork enabling Unsloth to run efficiently on multi-GPU systems, particularly for the NVIDIA RTX 5090 and similar setups.

• opensloth: Unsloth with support for multi-GPU training including experimental features.

Stay tuned for our official announcement! For more details, check out our ongoing Pull Request discussing multi-GPU support.