How to Fine-Tune an LLM on Your Own Data: Step-by-Step Guide + Code

Fine-tuning has a reputation for being hard and expensive. For a lot of use cases it is neither, as long as you know when it is the right tool and how to prepare your data. This guide walks through the whole process: what fine-tuning is, when to reach for it (and when not to), how to prepare data, how to run a job with real code, and how to evaluate the result without fooling yourself.

What fine-tuning is, and why you might need it

Fine-tuning means taking a pre-trained model and training it further on your own examples so it performs better on one specific task. You are not building a model from scratch, you are nudging an existing one towards your data, your format, and your style.

Use it when:

• You have domain-specific data (medical, legal, or company-specific knowledge and phrasing).
• You want output in your own voice, template, or house style, consistently.
• You need a reliable output format every time (always valid JSON, always the same structure).
• You want to reduce errors on a narrow, well-defined task.

Do not use it when:

• You just need a general chatbot: use the API as-is, it is cheaper.
• You have fewer than about 100 examples: too little for the model to learn from.
• You need the very latest capabilities: fine-tuned models are frozen and lag new releases.

Fine-tuning versus RAG

The most common mistake is fine-tuning when retrieval-augmented generation (RAG) would do. If your problem is "the model does not know my facts", that is a knowledge problem, and RAG solves it by fetching the right documents at query time. If your problem is "the model does not respond in my style or format", that is a behaviour problem, which is where fine-tuning shines.

Before you start: prepare your data

This is 80 per cent of the work, and where most projects succeed or fail.

Step 1: gather training data

You want at least 100 examples, ideally 500 or more, of input-output pairs. For customer support that is (question, ideal reply). For code generation it is (problem description, clean code). For classification it is (text, label).

Step 2: format it correctly

Providers expect a specific structure. The OpenAI chat format looks like this, one JSON object per line in a .jsonl file:

{
  "messages": [
    {"role": "system", "content": "You are a support agent for SaaS Company X."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Go to Settings, then Security, then Change Password. Enter your current password, then the new one twice."}
  ]
}

Anthropic's format is similar but typically omits a separate system turn in the example itself:

{
  "messages": [
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": "Go to Settings, then Security, then Change Password..."}
  ]
}

Step 3: validate quality

Check for typos, inconsistencies, and bias in the responses. Remove vague questions and unhelpful answers. Aim for maybe 10 to 20 per cent of your set to cover diverse and edge cases, so the model learns the boundaries, not just the easy middle.

Step 4: split your data

• Training set: about 80 per cent, the examples the model learns from.
• Validation set: about 20 per cent, held back to check it is not just memorising.
• Never let the same example sit in both.
• Keep a small eval set of 10 to 20 hard cases you check by hand after training.

Choosing a base model

Your choice comes down to budget, data privacy, and how much control you want.

• New to this? Start with a managed fine-tuning API. It is the simplest path from data to working model.
• Need privacy or high volume? Choose an open-source model you can host yourself.
• Smaller is often smarter. A fine-tuned small model frequently beats a giant general model on your narrow task, at a fraction of the cost.

Running the fine-tune: three paths

Path A: managed API (easiest)

Upload your .jsonl file, start a job, wait, then call the resulting model by its ID. The code sketch below uses the OpenAI client, but every managed provider follows the same shape.

from openai import OpenAI
client = OpenAI()

# 1. Upload data
file = client.files.create(
    file=open("train.jsonl", "rb"),
    purpose="fine-tune"
)

# 2. Start the job
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18"
)

# 3. Use the resulting model once training completes
response = client.chat.completions.create(
    model=job.fine_tuned_model,  # e.g. "ft:gpt-4o-mini:my-org::abc123"
    messages=[{"role": "user", "content": "How do I reset my password?"}]
)

Path B: open-source with LoRA (most control)

LoRA (Low-Rank Adaptation) trains a small set of adapter weights instead of the whole model, which makes it far cheaper and faster. It runs on a single consumer or rented GPU, even a 24GB card for smaller models. The rough flow: load the base model in 4-bit, attach LoRA adapters, train on your dataset, then merge or serve the adapters. Tools like Hugging Face transformers with peft and bitsandbytes, or wrappers such as Axolotl and Unsloth, handle the plumbing. Choose this when you want the model on your own hardware and to avoid ongoing API fees.

Path C: managed platforms (middle ground)

Services like Together AI, Replicate, or Predibase run the GPUs for you while still giving you open-model flexibility. A good fit if you want open-source models without managing infrastructure yourself.

Evaluating your fine-tuned model

Do not trust "it looks good". Measure it.

• Hold-out test: run your 10 to 20 hand-picked eval examples through both the base and the fine-tuned model, and compare side by side.
• Automatic metrics: for structured tasks, measure exact-match or valid-JSON rate. For free text, score against a rubric using a strong model as a judge.
• Watch for overfitting: if it nails training examples but stumbles on slightly different ones, you trained too long or had too little data.
• A/B in production: route a small percentage of real traffic to the new model and compare quality and cost before a full rollout.

What fine-tuning actually costs

• Managed API: training is billed per token in your dataset, typically a few dollars to low tens for small sets, plus a modest premium on inference versus the base model.
• Open-source (LoRA): GPU rental only, roughly a few dollars per hour for a capable card, and small jobs often finish in under an hour.
• Hidden costs: data preparation time is the real expense, along with evaluation and re-training when your task or data drifts.

Deploying your model

• Managed API: nothing to host. Call the returned model ID like any other model.
• Open-source: serve with vLLM, TGI, or Ollama for local use.
• Keep the base updated: fine-tuned models are frozen, so plan to re-fine-tune when a better base ships or your data changes.
• Version everything: tag each model with its training-data version so you can roll back cleanly.

Common pitfalls to avoid

1. Too little data. Under 100 examples rarely teaches anything useful. Aim for 500 or more.
2. Low-quality data. Garbage in, garbage out. The model copies your examples, mistakes and all.
3. Fine-tuning when RAG would do. If the problem is missing facts, use RAG.
4. No evaluation set. Without a hold-out set you cannot tell improvement from overfitting.
5. Over-training. Too many epochs make the model rigid and worse at anything novel.
6. Ignoring cost drift. Fine-tuned inference can cost more per token, so check the maths at your volume.

Frequently asked questions