⚠️Errors

To fix any errors with your setup, see below:

Saving to GGUF / vLLM 16bit crashes

You can try reducing the maximum GPU usage during saving by changing maximum_memory_usage.

The default is model.save_pretrained(..., maximum_memory_usage = 0.75). Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving.

Evaluation Loop - also OOM or crashing.

Set the trainer settings for evaluation to:

SFTTrainer(
    args = TrainingArguments(
        fp16_full_eval = True,
        per_device_eval_batch_size = 2,
        eval_accumulation_steps = 4,
        evaluation_strategy = "steps",
        eval_steps = 1,
    ),
    train_dataset = train_dataset,
    eval_dataset = eval_dataset,

This will cause no OOMs and make it somewhat faster with no upcasting to float32.

NotImplementedError: A UTF-8 locale is required. Got ANSI

See https://github.com/googlecolab/colabtools/issues/3409

In a new cell, run the below:

import locale
locale.getpreferredencoding = lambda: "UTF-8"