Fine-Tuning Local LLMs

The Big Picture

We have a phone agent that answers calls for a plumbing company. It uses a local AI model (running on your laptop, not in the cloud) to generate what the receptionist says. The problem: off-the-shelf models are generalists. They can write essays, answer trivia, and generate code — but they don't naturally sound like a warm, efficient plumbing receptionist.

Fine-tuning is the process of taking a general-purpose model and teaching it a specific skill by showing it hundreds of examples of how to do that skill well. It's like the difference between hiring someone who "speaks English" versus someone who's been trained as a receptionist — same underlying ability, but focused through practice.

Key Terms Glossary

Read this section first. Every term used in the rest of the guide is defined here.

Why Fine-Tune Instead of Prompting?

You might ask: "Why not just write a really good system prompt?" That's a fair question. Here's why fine-tuning is better for this use case:

The System Prompt Problem

Right now, every time the phone agent generates a response, it sends a ~500-word system prompt to the model:

[System prompt: 500 words explaining persona, rules, examples]
[Conversation history: 200-400 words]
[Model generates: 20 words]

That's 700+ input tokens processed for every single 20-word reply. On a 3.8B model running locally, processing those input tokens takes real time — about 800-1200ms just for the prompt, before the model even starts generating.

What Fine-Tuning Changes

After fine-tuning, the persona knowledge is baked into the model's weights. The system prompt shrinks from 500 words to ~50:

[System prompt: 50 words -- just "You are Smith Plumbing receptionist"]
[Conversation history: 200-400 words]
[Model generates: 20 words]

The Quality Difference

When Prompting IS Enough

Fine-tuning isn't always the answer. For our large judge model (Gemma4 26B), prompting is sufficient — it's big enough to follow complex instructions without fine-tuning. We only fine-tune the small, fast models (3.8B-7B) that struggle with prompt-following.

Our Hardware: Apple M3 Pro

Why This Matters

Fine-tuning an LLM typically requires expensive NVIDIA GPUs (A100, H100) that cost $10,000+ each. Apple Silicon changes this equation because of unified memory.

What We're Working With

Memory Budget During Fine-Tuning

Model weights (quantized):     ~4 GB  (for a 7B model in 4-bit)
LoRA adapter weights:          ~0.1 GB
Optimizer state:               ~0.3 GB
Gradient computation:          ~2 GB
Training batch:                ~1 GB
-----------------------------------------------
Total:                         ~7.5 GB per model

Our available memory:          36 GB
Headroom for OS + apps:        ~8 GB
Available for training:        ~28 GB  (comfortable margin)

Speed Expectations

The Software Stack

Here's every piece of software involved and how they connect:

 +-------------------------------------------------------------+
 |                OUR FINE-TUNING PIPELINE                      |
 |                                                              |
 |  +--------------+    +--------------+    +--------------+    |
 |  |  Training    |    |   MLX +      |    |   Ollama     |    |
 |  |  Data        |--->|   mlx-lm     |--->|              |    |
 |  |  (JSONL)     |    |  (fine-tune) |    |  (deploy)    |    |
 |  +--------------+    +--------------+    +--------------+    |
 |        |                    |                    |           |
 |   922 turn-level      LoRA adapters        GGUF model      |
 |   examples             (~60 MB)           ready to serve    |
 |                             |                    |           |
 |                    +--------+--------+           |           |
 |                    |  Convert to     |           |           |
 |                    |  GGUF format    |---------->|           |
 |                    +-----------------+           |           |
 |                                                  |           |
 |                                          +-------+------+   |
 |                                          | Phone Agent   |   |
 |                                          | (uses Ollama  |   |
 |                                          |  for inference)|  |
 |                                          +--------------+   |
 +-------------------------------------------------------------+

Software Requirements

What is LoRA?

LoRA (Low-Rank Adaptation) is the technique that makes fine-tuning possible on consumer hardware. This section explains how it works at an intuitive level.

The Problem LoRA Solves

A 7B model has ~7 billion parameters organized into large matrices (tables of numbers). Full fine-tuning means updating all 7 billion numbers, which requires:

• Loading all 7B weights into memory (~28 GB at full precision)
• Storing a copy of the gradients (~28 GB)
• Storing optimizer states (~28 GB)
• Total: ~84 GB — way more than our 36 GB

The LoRA Insight

Researchers discovered that when you fine-tune a model on a specific task, the changes to the weight matrices are usually "low-rank" — meaning they can be approximated by much smaller matrices.

Imagine a weight matrix W that's 4096 x 4096 (about 16 million numbers). Instead of modifying W directly, LoRA adds two small matrices:

Original:  W                    (4096 x 4096 = 16,777,216 parameters)

LoRA:      W + A x B            where:
           A is (4096 x 8)      = 32,768 parameters
           B is (8 x 4096)      = 32,768 parameters
                                -------------------------
           Total LoRA params:   = 65,536 parameters  (0.4% of original!)

The "8" in the middle is the LoRA rank. A rank of 8 means each adapter matrix has 8 columns/rows. Higher rank = more expressive but more parameters.

What This Means Practically

Intuitive Analogy

The Full Pipeline

Here's every step from training data to deployed model, in order:

Step-by-Step Walkthrough

Step 1: Prepare Data (Already Done)

Our training data is already in the right format. The key file is data/splits/train_mlx.jsonl, where each line is a JSON object with a messages array:

{
  "messages": [
    {"role": "system", "content": "You are a friendly, warm phone receptionist..."},
    {"role": "user", "content": "[phone rings]"},
    {"role": "assistant", "content": "Hi, Smith Plumbing, how can I help?"}
  ]
}

Each example trains the model to generate one specific assistant reply given the conversation history. We have 922 of these in the training set.

See Training Methodology for full details on how this data was generated and structured.

Step 2: Install Dependencies

pip install mlx-lm huggingface_hub

mlx-lm brings in the fine-tuning tools. huggingface_hub handles downloading base model weights.

Step 3: Download Base Weights

# These download the full-precision HuggingFace weights (~8-14 GB each)
huggingface-cli download microsoft/phi-4-mini-instruct
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3
huggingface-cli download google/gemma-2-2b-it

These are cached in ~/.cache/huggingface/ and reused across fine-tuning runs.

Step 4: Fine-Tune with LoRA

The actual fine-tuning command for each model:

# Phi4-mini (~15 min)
python -m mlx_lm.lora \
  --model microsoft/phi-4-mini-instruct \
  --data data/splits/ \
  --train \
  --batch-size 2 \
  --lora-rank 8 \
  --num-layers 16 \
  --iters 600 \
  --learning-rate 1e-5 \
  --adapter-path adapters/phi4-mini

# Mistral 7B (~25 min)
python -m mlx_lm.lora \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --data data/splits/ \
  --train \
  --batch-size 2 \
  --lora-rank 8 \
  --num-layers 16 \
  --iters 600 \
  --learning-rate 1e-5 \
  --adapter-path adapters/mistral-7b

# Gemma E2B (~25 min)
python -m mlx_lm.lora \
  --model google/gemma-2-2b-it \
  --data data/splits/ \
  --train \
  --batch-size 2 \
  --lora-rank 8 \
  --num-layers 16 \
  --iters 600 \
  --learning-rate 1e-5 \
  --adapter-path adapters/gemma-e2b

What happens during training

MLX loads the base model, attaches LoRA adapter matrices to each transformer layer, then runs through the training examples. For each batch, it:

1. Feeds the conversation history through the model
2. Compares the model's predicted next tokens against the actual assistant reply
3. Computes how wrong the predictions were (the "loss")
4. Updates only the LoRA adapter weights to reduce the loss

The loss should steadily decrease. If it plateaus, the model has learned what it can from this data.

Step 5: Test the Adapter

Before converting, do a quick sanity check:

python -m mlx_lm.generate \
  --model microsoft/phi-4-mini-instruct \
  --adapter-path adapters/phi4-mini \
  --prompt "You are a receptionist at Smith Plumbing.\n\nUser: Hi, my kitchen sink is leaking pretty bad.\n\nAssistant:"

If it responds like a warm receptionist ("Oh no, a leaking sink — let's get that taken care of!"), the fine-tuning worked.

Step 6: Fuse and Convert

# Merge adapter into base model
python -m mlx_lm.fuse \
  --model microsoft/phi-4-mini-instruct \
  --adapter-path adapters/phi4-mini \
  --save-path models/phi4-mini-fused

# Convert to GGUF for Ollama (4-bit quantization)
python -m mlx_lm.convert \
  --model models/phi4-mini-fused \
  --quantize q4_K_M \
  --to-gguf \
  --output models/smith-plumbing-phi4.gguf

Fusing permanently merges the LoRA adapter into the base model weights. After fusing, you have a standalone model that doesn't need the adapter file.

Converting to GGUF takes the fused model and packages it in the format Ollama expects, with 4-bit quantization to keep the file size manageable.

Step 7: Import into Ollama

Create a Modelfile:

# models/Modelfile.phi4
FROM ./smith-plumbing-phi4.gguf

PARAMETER temperature 0.7
PARAMETER top_p 0.9

SYSTEM """You are a friendly phone receptionist at Smith Plumbing."""

Then import:

ollama create smith-plumbing-phi4 -f models/Modelfile.phi4

Now ollama run smith-plumbing-phi4 runs your fine-tuned model, and the phone agent can use it by setting LLM_FAST=smith-plumbing-phi4 in the .env file.

Steps 8-9: Evaluation and Deployment

See Training Methodology for the tournament evaluation design and deployment process.

How to Know If It Worked

During Training: Watch the Loss

Healthy training: Both train and val loss decrease together.

Overfitting: Train loss keeps dropping but val loss plateaus or increases. The gap between them is the "generalization gap." If it exceeds ~0.5, stop training or reduce the LoRA rank.

After Training: Tournament Evaluation

Run the 3 base models and 3 fine-tuned models through the same 21 held-out test conversations. Score each with the LLM judge on 6 phases x 3 dimensions = 18 quality metrics.

Success Criteria

1. At least one fine-tuned model scores >4.0/5 average
2. Fine-tuned model beats its base by >0.3 average
3. Post-tuning latency stays within 4-second budget
4. Test scores within 0.5 of training scores (no overfitting)

Common Pitfalls

Further Reading

Our Project Docs

External Resources

Key Concepts to Study Further

• Transformer architecture — the neural network design all modern LLMs use
• Attention mechanism — how transformers decide which parts of the input to focus on
• Tokenization — how text becomes numbers (BPE, SentencePiece)
• Gradient descent — the optimization algorithm that updates weights during training
• Quantization — compressing model weights for faster inference