๐Text-to-Speech (TTS) Fine-tuning
Learn how to to fine-tune TTS & STT voice models with Unsloth.
Fine-tuning TTS models allows them to adapt to your specific dataset, use case, or desired style and tone. The goal is to customize these models to clone voices, adapt speaking styles and tones, support new languages, handle specific tasks and more. We also support Speech-to-Text (STT) models like OpenAI's Whisper.
With Unsloth, you can fine-tune TTS models 1.5x faster with 50% less memory than other implementations with Flash Attention 2. This support includes Sesame CSM, Orpheus, and models supported by transformers (e.g. CrisperWhisper, Spark and more).
We've uploaded TTS models (original and quantized variants) to our Hugging Face page.
Fine-tuning Notebooks:
Whisper Large V3 - Speech-to-Text (STT)
Choosing and Loading a TTS Model
For TTS, smaller models are often preferred due to lower latency and faster inference for end users. While fine-tuning a model under 3B parameters is often ideal, our primary example uses Orpheus-TTS (3B), a Llama-based speech model.
This is because Orpheus is pre-trained on a large speech corpus and excels at generating realistic speech with built-in support for emotional cues like laughs and sighs. Its architecture makes it one of the easiest TTS models to utilize and train as it can be exported via llama.cpp meaning it has great compatibility across all inference engines. For unsupported models, you'll only be able to save the LoRA adapter safetensors.
Because voice models are usually small in size, you can train the models using LoRA 16-bit or full fine-tuning FFT which may provide higher quality results. To load it in LoRA 16-bit:
When this runs, Unsloth will download the model weights if you prefer 8-bit, you could use load_in_8bit = True
, or for full fine-tuning set full_finetuning = True
(ensure you have enough VRAM). You can also replace the model name with other TTS models.
Preparing Your Dataset
At minimum, a TTS fine-tuning dataset consists of audio clips and their corresponding transcripts (text). Letโs use the Elise dataset which is ~3 hour single-speaker English speech corpus. There are two variants:
MrDragonFox/Elise
โ an augmented version with emotion tags (e.g. <sigh>, <laughs>) embedded in the transcripts. These tags in angle brackets indicate expressions (laughter, sighs, etc.) and are treated as special tokens by Orpheusโs tokenizerJinsaryko/Elise
โ base version with transcripts without special tags.
The dataset is organized with one audio and transcript per entry. On Hugging Face, these datasets have fields such as audio
(the waveform), text
(the transcription), and some metadata (speaker name, pitch stats, etc.). We need to feed Unsloth a dataset of audio-text pairs.
Instead of solely focusing on tone, cadence, and pitch, the priority should be ensuring your dataset is fully annotated and properly normalized.
Option 1: Using Hugging Face Datasets library โ We can load the Elise dataset using Hugging Faceโs datasets
library:
This will download the dataset (~328 MB for ~1.2k samples). Each item in dataset
is a dictionary with at least:
"audio"
: the audio clip (waveform array and metadata like sampling rate), and"text"
: the transcript string
Orpheus supports tags like <laugh>
, <chuckle>
, <sigh>
, <cough>
, <sniffle>
, <groan>
, <yawn>
, <gasp>
, etc. For example: "I missed you <laugh> so much!"
. These tags are enclosed in angle brackets and will be treated as special tokens by the model (they match Orpheusโs expected tags like <laugh>
and <sigh>
. During training, the model will learn to associate these tags with the corresponding audio patterns. The Elise dataset with tags already has many of these (e.g., 336 occurrences of โlaughsโ, 156 of โsighsโ, etc. as listed in its card). If your dataset lacks such tags but you want to incorporate them, you can manually annotate the transcripts where the audio contains those expressions.
Option 2: Preparing a custom dataset โ If you have your own audio files and transcripts:
Organize audio clips (WAV/FLAC files) in a folder.
Create a CSV or TSV file with columns for file path and transcript. For example:
Use
load_dataset("csv", data_files="mydata.csv", split="train")
to load it. You might need to tell the dataset loader how to handle audio paths. An alternative is using thedatasets.Audio
feature to load audio data on the fly:Then
dataset[i]["audio"]
will contain the audio array.Ensure transcripts are normalized (no unusual characters that the tokenizer might not know, except the emotion tags if used). Also ensure all audio have a consistent sampling rate (resample them if necessary to the target rate the model expects, e.g. 24kHz for Orpheus).
In summary, for dataset preparation:
You need a list of (audio, text) pairs.
Use the HF
datasets
library to handle loading and optional preprocessing (like resampling).Include any special tags in the text that you want the model to learn (ensure they are in
<angle_brackets>
format so the model treats them as distinct tokens).(Optional) If multi-speaker, you could include a speaker ID token in the text or use a separate speaker embedding approach, but thatโs beyond this basic guide (Elise is single-speaker).
Fine-Tuning TTS with Unsloth
Now, letโs start fine-tuning! Weโll illustrate using Python code (which you can run in a Jupyter notebook, Colab, etc.).
Step 1: Load the Model and Dataset
In all our TTS notebooks, we enable LoRA (16-bit) training and disable QLoRA (4-bit) training with: load_in_4bit = False
. This is so the model can usually learn your dataset better and have higher accuracy.
Step 2: Advanced - Preprocess the data for training (Optional)
We need to prepare inputs for the Trainer. For text-to-speech, one approach is to train the model in a causal manner: concatenate text and audio token IDs as the target sequence. However, since Orpheus is a decoder-only LLM that outputs audio, we can feed the text as input (context) and have the audio token ids as labels. In practice, Unslothโs integration might do this automatically if the modelโs config identifies it as text-to-speech. If not, we can do something like:
However, Unsloth may abstract this away: if the model is a FastModel with an associated processor that knows how to handle audio, it might automatically encode the audio in the dataset to tokens. If not, youโd have to manually encode each audio clip to token IDs (using Orpheusโs codebook). This is an advanced step beyond this guide, but keep in mind that simply using text tokens wonโt teach the model the actual audio โ it needs to match the audio patterns.
Let's assume Unsloth provides a way to feed audio directly (for example, by setting processor
and passing the audio array). If Unsloth does not yet support automatic audio tokenization, you might need to use the Orpheus repositoryโs encode_audio
function to get token sequences for the audio, then use those as labels. (The dataset entries do have phonemes
and some acoustic features which suggests a pipeline.)
Step 3: Set up training arguments and Trainer
We do 60 steps to speed things up, but you can set num_train_epochs=1
for a full run, and turn off max_steps=None
. Using a per_device_train_batch_size >1 may lead to errors if multi-GPU setup to avoid issues, ensure CUDA_VISIBLE_DEVICES is set to a single GPU (e.g., CUDA_VISIBLE_DEVICES=0). Adjust as needed.
Step 4: Begin fine-tuning
This will start the training loop. You should see logs of loss every 50 steps (as set by logging_steps
). The training might take some time depending on GPU โ for example, on a Colab T4 GPU, a few epochs on 3h of data may take 1-2 hours. Unslothโs optimizations will make it faster than standard HF training.
Step 5: Save the fine-tuned model
After training completes (or if you stop it mid-way when you feel itโs sufficient), save the model. This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!
This saves the model weights (for LoRA, it might save only adapter weights if the base is not fully fine-tuned). If you used --push_model
in CLI or trainer.push_to_hub()
, you could upload it to Hugging Face Hub directly.
Now you should have a fine-tuned TTS model in the directory. The next step is to test it out and if supported, you can use llama.cpp to convert it into a GGUF file.
Last updated
Was this helpful?