⚠️Errors

To fix any errors with your setup, see below:

Saving to GGUF / vLLM 16bit crashes

You can try reducing the maximum GPU usage during saving by changing maximum_memory_usage.

The default is model.save_pretrained(..., maximum_memory_usage = 0.75). Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving.

Evaluation Loop - also OOM or crashing.

First split your training dataset into a train and test split. Set the trainer settings for evaluation to:

new_dataset = dataset.train_test_split(test_size = 0.01)
SFTTrainer(
    args = TrainingArguments(
        fp16_full_eval = True,
        per_device_eval_batch_size = 2,
        eval_accumulation_steps = 4,
        eval_strategy = "steps",
        eval_steps = 1,
    ),
    train_dataset = new_dataset["train"],
    eval_dataset = new_dataset["test"],

This will cause no OOMs and make it somewhat faster with no upcasting to float32.

NotImplementedError: A UTF-8 locale is required. Got ANSI

See https://github.com/googlecolab/colabtools/issues/3409

In a new cell, run the below:

import locale
locale.getpreferredencoding = lambda: "UTF-8"