Part I · the basics

Fine-tuning a language model

What fine-tuning changes, when to use it, and how to go from a folder of examples to a private model running on your own machine.

Fine-tuning a small open model to do one task well is cheaper and simpler than its reputation suggests. This part covers how it works and the full workflow. Part II then uses it on a real production system and shows what that takes.

01 · First, do you even need it?

Three ways to make a model better at your task

Try these in order. Each one is more work than the last, and most tasks are solved before the third.

Cheapest

Prompting

Put instructions and a few examples in the prompt. If that gets you there, stop. No training, no infrastructure.

For knowledge

Retrieval (RAG)

Keep facts in a search index and let the model read them when it needs them. Use this when the problem is what the model knows: documents, catalogs, data that changes.

For behavior

Fine-tuning

Adjust the weights so the model behaves a certain way by default: a fixed format, a tone, a tool-calling pattern. You don't have to spell it out in the prompt every time.

The rule of thumbFine-tune for behavior, retrieve for knowledge. If you're fine-tuning to teach the model facts, you probably want RAG instead. Part II relies on this split.

02 · What actually changes

You don’t retrain the whole model

A model is a big stack of weight matrices. Full fine-tuning rewrites all of them, which works but is expensive and easy to get wrong. LoRA leaves the original weights frozen and trains a small add-on layer (the “adapter”) instead. You update a few million parameters rather than several billion.

QLoRA goes one step further and loads the frozen base in 4-bit precision, so it fits in much less memory. That is what lets you fine-tune an 8B model on a single GPU, or a laptop.

Frozen base

The original billions of weights, left untouched (and in QLoRA, stored in 4-bit). All the general ability stays here.

Trained adapter

A small extra matrix per layer. It's the only part that trains, and it's small enough to email. Attach or merge it when you run the model.

03 · The workflow

Five steps, start to finish

The whole loop, in the order you do it. Most of the work is in step one. The training itself is quick.

Collect and format your examples

Turn your reports, Q&A, and labels into chat-style JSONL. This is the slow part, and it matters most: the model learns whatever is in your examples.

Pick a small instruct model

A 1B–8B instruction-tuned base. Smaller is cheaper, faster, and easier to run privately. Start with the smallest size that might do the job.

Train an adapter (LoRA / QLoRA)

Freeze the 4-bit base and train just the adapter on your data. This takes minutes to a couple of hours, not days.

Evaluate on held-out data

Check that it answers the way you want on examples it never saw. Read the outputs, not the loss number. Fix what it gets wrong and retrain.

Fold in and run locally

Merge or attach the adapter, export to a portable format like GGUF, and run it offline. No per-token bill.

04 · Choosing a base

Pick the smallest model that works

Start with the smallest instruction-tuned model that might handle the task, and only move up if your evaluation says you need to. Smaller models are cheaper to train and easier to run yourself.

If you need…	Reach for
Simple tagging & extraction	A ~1–2B instruct model. Fastest, runs anywhere
A solid all-round starter	A ~3–4B instruct model. The usual sweet spot
Clean structured / JSON output	A model with strong instruction-following at 3–8B
More reasoning headroom	An ~8B model, when smaller sizes plateau on your task

Whatever you pick, train with the exact chat template that model expects. Getting the template wrong is the most common reason a run quietly produces nonsense.

05 · Hyperparameters

The three settings that matter

There's a long list of settings, but for LoRA these three do most of the work. Leave the rest at their defaults.

Rank (r)

How much room the adapter has to learn. 8 to 16 is a good default. Raise it only if the model can't capture the behavior; higher rank also means more chance of overfitting.

Learning rate

How big each training step is. Too high and the model breaks or forgets the base; too low and nothing changes. For LoRA, 1e-4 to 2e-4 is a good place to start.

Epochs

How many times the model sees your data. One to three is usually enough. More than that and it starts memorizing. Judge by held-out behavior, not training loss.

06 · Running it & the bill

Test locally, train on a rented GPU

Run the pipeline on your own machine first, for free, to catch bugs. Then rent a GPU just for the full run. A small QLoRA job costs a few dollars, as long as you remember to shut the instance down.

# 1. Test the loop locally (Apple MLX), on a small slice — free
mlx_lm.lora --model <base-model-4bit> --train \\
    --data ./data --iters 200 --batch-size 1

# 2. Full QLoRA run on a rented GPU (e.g. Unsloth + a spot instance)
python train.py --model <base-model> --data ./data.jsonl \\
    --rank 16 --lr 2e-4 --epochs 2

# 3. Export to a portable format and run it offline, locally
#    (merge adapter -> GGUF -> a local runtime such as Ollama)

Test locally first

A quick run on your laptop catches template and data bugs before you pay for a GPU.

The real cost

A GPU you forgot to turn off. Export, copy it down, then terminate.

07 · The checklist

Do it in this order

One pass through the whole loop. Three mistakes cause most bad runs: too few or too similar examples, watching the loss number instead of the actual outputs, and leaving the GPU running.

Gather 200+ real examples

Collect input-and-ideal-output pairs for your task. With under about 50, or examples that all look alike, the model won't learn much. Variety matters more than volume.

Convert to chat JSONL, hold back ~10%

Format them with the exact chat template the model expects, or quality drops. Keep about 10% aside for evaluation.

Prove the loop locally first

Run a small slice on your own machine (Apple MLX works well) to confirm the pipeline runs end to end before you pay for a GPU.

Run the full train on a rented GPU

Launch a spot GPU and run the full QLoRA job. A short run on clean data beats a long run on messy data.

Evaluate against the untuned base

Compare the tuned model to the base on held-out data. If it isn't clearly better, fix the data before touching the settings.

Export, pull down, and shut the GPU off

Export to a portable format, copy it down, and shut the instance off. A forgotten GPU is the only thing here that really costs money.

End of Part I

Part II: a real example

Part I covered the technique. Part II uses it on a real production agent, where the training is the easy step and the hard part is working out whether the small model can actually replace the big one.

The tool

See it run, side by side

Every measurement in Part II came from a small comparison tool: send one message to two engines at once and watch the tool calls, the full trace, the tokens, and the cost. Below, the fine-tuned model (left, on a laptop) is set against GPT-5.2 (right). Both are wired to the same live data services, so the only thing that differs is the model.

$Side-by-side comparison: the fine-tuned model and GPT-5.2 both return the same five methane datasets from the live RAG service; the fine-tuned side costs $0, GPT-5.2 costs a fraction of a cent.$

One query, two models. Both call the live RAG service and get the same five datasets. The only real difference is $0 on the left (local) versus a paid call on the right.

An expanded trace showing the user message, the model writing a collections_rag_tool call, and the raw tool result.

Every turn expands into a full trace: the user message, the model’s tool call, and the raw tool result it gets back.

Part II · a real case study

From toy to production: replacing a production LLM agent with a model on your laptop

How I fine-tuned Qwen3-8B to stand in for GPT-5.2 behind a geospatial dataset agent: what I measured, what worked, and the gaps that still need closing before it could replace the original.

Part I covered how to fine-tune. Here the goal is harder — actually replacing a paid frontier model in a live product, and being honest about whether it worked. The techniques (LoRA, QLoRA, MLX) are the same. What changes is that training is no longer the hard part; measuring whether the small model is good enough is. The numbers below are from one run on one system, so treat them as an example of how to think about the problem, not as benchmarks to copy.

01 · Why this can be fine-tuned at all

A narrow job, not an open-ended assistant

You can't replace most chat assistants with a small model, because there's no limit to what “good” means for them. This agent is the opposite. It has seven fixed tools that run in a set order, confirmation steps where the user has to approve, fixed response templates, a finite list of datasets behind a search tool, and a clear refusal for anything off-topic. Every step is either right or wrong, which means every step can be measured.

The ideaBehavior in the weights, knowledge in the tools. The model is trained to pull out the inputs, pick the right tool, read results back, and refuse off-topic requests. The dataset facts live in the tool outputs, so when the catalog changes you don't retrain anything. The RAG service does the ranking; the model only writes the search query and passes along whatever the user picks. I checked this directly: the fine-tuned model and GPT-5.2 write queries that retrieve the same datasets about 95% of the time.

Who does whatIt's easy to assume the model picks the datasets. It doesn't. For the query "i want methane datasets": the model writes the search string and calls the RAG tool; the RAG service embeds that string, scores every collection by cosine similarity, and returns the top results already ranked; the model just reads that list back. Wire both the fine-tuned model and GPT-5.2 to the same live RAG service and they return the same five collections, in the order the service's similarity scores set (0.773, 0.772, 0.766, …) — numbers the model never sees. The model writes the query and reads back the answer; the service does the ranking.

The cost

GPT-5.2 was projected at about $30K a month. One conversation makes 11 model calls and uses around 80K input tokens.

The alternative

About $890 a month on a hosted fallback, or close to nothing run locally, with a prompt 24× smaller.

02 · The methodology

Six steps, in order

Training is the easy part. The discipline is in the order: build the benchmark before training, check the spec against production, split the data so you can catch memorization, and measure whether it works in a real conversation, not just one turn at a time.

Step 01

Build the benchmark before any training

For each assistant turn, give the model the system prompt and the conversation so far from a known-good recording, then check its decision against what the recording actually did. Before trusting the benchmark at all, confirm the production model can reproduce its own recording at close to 100%. Mine couldn't at first — two scoring bugs had to be fixed before it did.

If the model that produced the data can't pass the benchmark, the benchmark is broken.

Step 02

Generate the data from the policy, then check it against production

The policy is deterministic, so I can generate the training transcripts mechanically instead of writing them by hand. Then I check those transcripts against the live production model. The first pass agreed 90% of the time, which turned up four real bugs in my generator. After fixing them, agreement was 98.8% on tool choice and 92.5% on the echoed text, with no disagreements at the confirmation gates.

What you assume is correct is just a guess until the recording backs it up.

Step 03

Split the data so you can catch memorization

If your test examples come from the same generator as your training ones, they leak into each other. So I hold back whole categories of input — certain topics, places, and ways of phrasing dates — and use those only for evaluation. A hash check on the training set warns me if any test item also showed up in training.

The response templates are shared on purpose; they are the policy. What's held back is the inputs.

Step 04

Train small and local, with the policy baked into the weights

LoRA on a 4-bit Qwen3-8B with MLX. I train against a short stub prompt instead of the full one, so the policy ends up in the weights and each request carries far fewer input tokens. The loss only applies to the turn being trained.

Behavior in the weights, knowledge in the tools.

Step 05

Evaluate in layers

Replay to compare turn by turn. The untrained base model as a control, to see what training actually added. Free-running rollout to check whole conversations. Repeated runs to measure variance. And an outside review of the method itself.

Replay tells you whether each turn is right. Rollout tells you whether the whole thing works.

Step 06

Move from synthetic data to real captures

The production backend runs locally and hands back its full state every turn, so I can record real conversations without any logging setup. Once I have those, they become the main training data, and the generator is only there to fill in cases the real ones miss.

Twenty real conversations turned up a tool-error case my generator never produced.

03 · Results, and how each was checked

Every claim says how it was checked

No result is stated without the method behind it. The colored dot is the real verdict, not the one I was hoping for.

Claim	Result & how verified
The benchmark is calibrated	GPT-5.2 reproduces its own recording 100/100/100, three times running replay self-test, repeated
The generated data matches production	98.8% tool / 92.5% echo / 0 gates on 164 held-out decisions live GPT-5.2 replay, two rounds
Fine-tuning clearly beats the base model	base with stub prompt 65.5% / 10% / 9 gates → fine-tuned 98.8% / 93.8% / 0 gates same runtime and prompt, only the weights differ
The fine-tuned model matches GPT-5.2 turn for turn	tools 98.8 = 98.8 · echo 93.8 vs 92.5 · gates 0 = 0 same decisions, same scorer (replay)
Corrected: the earlier completion failure	The “0–25% completion” number was a bug in the evaluator, not the model. The same adapter scores 100% once the evaluator is fixed, and it really does call every tool. Exposure bias is still a real risk; this number wasn't evidence of it. re-run on a fixed evaluator
Date conversion didn't generalize	a held-out “spring 2021” gave the wrong date range (84.5% args vs 98.8%); later fixed by handing date parsing to a plain code tool held-out replay, grouped by failure
It follows the full prompt too, not just the stub	given GPT-5.2’s full 14,571-char prompt instead of the 507-char stub, the fine-tuned model still routes tools and hits the gate correctly — at 3.7× the input tokens for the same result, which is why the stub is worth it live replay, stub vs full prompt
Same datasets from the live RAG service	wired to the real RAG/Geodini/STAC services, the fine-tuned model and GPT-5.2 return the same five methane collections for the same query. The difference is $0 (local) versus a paid call, not the datasets. side-by-side on live services

04 · The main lesson

Replay versus rollout, and a correction

In replay, the model gets the correct history at every turn, so one bad turn never affects the next. In rollout, the model feeds on its own previous answers, and a small slip early on can snowball into a state the training data never covered. So replay tells you whether each turn is right; only rollout tells you whether the whole conversation holds together.

The correctionI first wrote this section around a 0–25% rollout completion rate, which I took to be the model's main weakness. Later, while fixing something unrelated in the rollout evaluator, the same adapter scored 100% and was calling every tool. The failure was in my evaluation loop, not the model. Exposure bias is real and worth worrying about, but that particular number was a measurement bug. The lesson worth keeping: test your evaluator as hard as you test your model.

05 · Failure modes

The five that are easy to miss

Each of these went unnoticed for a while. They're the ones a beginner's guide rarely warns you about.

Check the measurement, not just the model

My headline example of exposure bias turned out to be a bug in the evaluator, not a failure in the model. The same adapter scored 0% completion under the old rollout loop and 100% under a fixed one, and it really was calling every tool. Replay versus rollout is still the right way to think about it, but you have to check the evaluator as carefully as you check the model.

An 8B model won't generalize phrase-to-date conversion

I trained on “summer” and “spring” still failed. The model had memorized specific date phrases, not learned the pattern. Either cover every case in training, or — better — move the conversion into plain code, which fixes it for every model at once.

The serving stack can fool you

mlx_lm.server 0.31.3 quietly ignored the adapter, so I was benchmarking the base model while thinking it was the fine-tune. mlx_lm.fuse quietly drops the LoRA on a 4-bit base. Always confirm the served model can reproduce a training example before you trust any number from it.

Scoring rules quietly inflate results

Prefix-matching on the echoed text accepted anything tacked on the end. Fuzzy argument matching let the wrong place through. One gate metric lumped all text turns together. Fixing each one changed which model looked better — the score had been measuring the scorer.

Reading the list back is its own risk

The RAG service picks and ranks the datasets, but the model still has to repeat that list to the user, and a model can reorder, drop, or invent an entry while doing it. GPT-5.2 spells the whole list out in prose; the fine-tuned model just says “Found 5” and leaves the list to the interface. The safer design builds the user-facing list in code from the tool output instead of trusting the model to repeat it — which is the whole reason for the gated controller.

06 · Hardening the method

What an outside review still found

The common thread: copying what production does isn't the same as being able to replace it. A public-facing agent needs a stricter bar than “matches GPT-5.2 on replay.”

Matching production isn't the same as being right

Three things get mixed up here: what the agent should do, what GPT-5.2 actually does, and what the fine-tuned model does. If you only ever match the middle one, the student quietly inherits whatever GPT-5.2 gets wrong. You need a small set of human-checked examples to anchor against.

Decide what tool output the model is allowed to trust

The model reads and acts on whatever the tools and RAG service hand back, so those outputs need to be validated against a schema. And you have to draw a boundary: can a dataset's metadata contain text that reads like an instruction?

Ambiguous queries need a rule for who decides

Something like “vegetation loss last spring” shouldn't be settled by the production model or the generator on its own. Decide up front who labels these cases and how.

Covering every case isn't the same as matching real traffic

Held-out synthetic examples hit every branch of the policy, but they don't reflect how often real users phrase things each way. You need both: one test for coverage, one for real traffic. A model can pass one and fail the other.

07 · The open question

Should the model be the agent, or just fill in the slots?

The results lean toward the second. The fine-tuned model is good at pulling out inputs and picking options, which suggests that asking an 8B model to carry the whole workflow in its weights is the wrong split. The fragile, exact parts belong in code; let the model do the language work it's good at.

Leave to the model

The language work: handling reworded requests, working out intent, pulling out values, and spotting off-topic questions.

Leave to code

Anything that has to be exact: tracking workflow state, dates, IDs, and the fixed response templates.

A stronger model can stay in the loop for the cases the small one is unsure about, so replacement doesn't have to be all-or-nothing. What the data can't tell me yet is whether an 8B model with plain fine-tuning is enough to stay consistent over long conversations, or whether closing that gap needs preference training or a bigger base model. Honestly, that's not settled either way.

08 · The acceptance contract

Define “done,” then grade against it

Replacing the model is a yes-or-no call, not a gut feeling. Each row has to clear a stated bar, and a failure counts for more or less depending on how bad it is.

Dimension	Acceptance bar	Severity if failed
Full pipeline completion	≥ production − X%, or ≥ 99% on critical paths	Blocker
Dataset / place / date accuracy	≥ 98–99%, or delegated to a deterministic tool	Critical
Confirmation-gate violations	0 tolerated	Blocker
Unsafe or off-policy completions	0 in the adversarial suite	Blocker
Tool-error recovery	≥ 95–99%	Critical
Regression vs GPT-5.2 on high-severity scenarios	no statistically significant drop	Blocker

09 · Reproduce it

The commands, end to end

Check the benchmark, generate and split the data, train locally, then evaluate with both replay and rollout.

# 1. Calibrate the benchmark (the recorded teacher must score ~100%)
python3 -m agent_bench run --models recorded

# 2. Generate data and the disjoint train/eval split
python3 -m agent_bench generate --per-scenario 8 --sft-system stub

# 3. Verify the synthetic spec against live production
python3 -m agent_bench run --models gpt-5.2 \\
    --narratives-dir datagen_out/eval_narratives

# 4. Fine-tune locally (LoRA on Qwen3-8B 4-bit via MLX)
bash mlx_kit/train.sh

# 5. Serve the adapter, then evaluate per-turn (replay) and free-running (rollout)
python3 mlx_kit/serve_patched.py \\
    --model mlx-community/Qwen3-8B-4bit \\
    --adapter-path mlx_kit/adapters \\
    --port 8091

python3 -m agent_bench run     --models qwen3-8b-ft-clean-local --system stub \\
    --narratives-dir datagen_out/eval_narratives   # is each turn right?
python3 -m agent_bench rollout --models qwen3-8b-ft-clean-local --system stub \\
    --narratives-dir datagen_out/eval_narratives   # does the whole conversation hold?

Bottom line

Don't trust your own good numbers

The hard part here wasn't training an 8B model. It was not believing any result, good or bad, until it held up under a test built to break it. The run earned a real win: fine-tuning clearly beats the base model and matches GPT-5.2 turn for turn. It also earned an honest caveat: replacing the original still needs more data behind the numbers, a test on real traffic, and a safety bar that holds up under repeated runs. Some of the gaps flagged here were later closed, and one alarming result turned out to be a bug in the measurement. Catching and fixing those is the whole point.

Every number here comes with how it was checked. That matters more than any single result.

Source, benchmark, and the comparison tool: github.com/ajinkyakulkarni/finetuning-field-guide