The previous post was the story: three rounds of fine-tuning, $35 of compute, what worked and what didn’t. This two-part series is the recipe. What we do, in order, to turn a stock 4B base model into a specific character.

Part 1 (this post) is the data pipeline. Writing the persona prompt, generating a synthetic corpus with Gemini, cleaning it and evaluating it with Claude Haiku. By the end we’ll have train.jsonl and val.jsonl ready to train on and a passes_all score that tells us whether training is worth doing.

Part 2 is everything after that: the LoRA SFT run on Qwen3-4B, eval against the same Haiku judge and merging to GGUF for local inference.

Code is at github.com/arunma/learn-you-an-sft. I’ve kept this roughly in the order I built it. Each stage writes a file and the next one picks up where the previous one stopped.

Here’s the whole pipeline at a glance:

flowchart LR
    A[persona_prompt.md
~350 lines] --> B
    Q[15k question seeds
Gemini Flash] --> B[Generate answers
Gemini 2.5 Pro]
    B --> C[Clean
ftfy + fasttext + MinHash]
    C --> D[Quality-gate
Claude Haiku judge]
    D --> E[train.jsonl
+ val.jsonl]

    classDef input fill:#fef3c7,stroke:#d97706,color:#78350f
    classDef synth fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef stage fill:#ecfccb,stroke:#65a30d,color:#365314
    classDef output fill:#fce7f3,stroke:#be185d,color:#831843

    class A,Q input
    class B synth
    class C,D stage
    class E output

What we need

For Part 1 we don’t need a GPU. It’s all API calls to Gemini and Anthropic plus local pandas. The GPU work starts in Part 2.

Three API keys, all loaded from .env:

• GEMINI_API_KEY: synthesis. Cheap.
• ANTHROPIC_API_KEY: the judge. Most of the cost lives here.
• HF_TOKEN: pushing the adapter (Part 2 only).

1
2
3

git clone https://github.com/arunma/learn-you-an-sft
cd learn-you-an-sft && uv sync
cp .env.example .env && vi .env

For Monty, one full round of Part 1 (synth + filter + judge) cost around $8. The judge accounts for most of it. Worth paying for, since it prevents us from training on bad data.

The persona prompt is the whole project

persona_prompt.md is the most important file in the repo. It sounds like I am being over-dramatic given we have the training script, the LoRA config and the eval rubric sitting next to it, but it’s true. What we write here decides everything downstream. This is what Gemini reads when it impersonates our character and generates the training corpus. Our model will be exactly as opinionated, as funny and as specific as this document tells Gemini to be.

The Monty version is ~350 lines: identity, what he hates, when to ask questions back and when to drop the voice entirely (crisis prompts). A small slice from the middle:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

## The three-axis target

Every response should hit three axes:

1. **Useful** — the asker walks away with the actual answer.
2. **Witty** — a real observation. Not setup-punchline. Not a quip glued to a Wikipedia entry.
3. **Edged** — a point of view. No hedging, no both-sidesing, no "great question," no "it depends" as a cop-out.

Only useful = Wikipedia. Only witty = joke account. Only edged =
contrarian on Twitter. Land all three.

I realised early on that the examples do more work than the rules. If we find ourselves writing “the character should be playful but firm,” we may as well delete that sentence and write two examples that show playful + firm. Show, don’t tell. Gemini reads examples and copies them. It reads adjectives and ignores them.

Iterate on the prompt in a chat window first. Drop it into Gemini AI Studio, ask it ten test questions and read the answers. If half of them sound generic, the prompt needs work. Tighten until ten random questions get ten satisfactory in-character responses. Only then are we ready for the pipeline. The pipeline is the cheap part and the persona prompt is the real work.

Step 1 — Generate the question pool

1

uv run python -m prep questions

Gemini Flash, ~100 questions per category × 10 categories of things people text their friends: tech opinions, life decisions, banter, ask-back triggers, beginner technical and so on. Dedup, shuffle, trim to ~1k. About $1 in API calls.

The structured-output trick is a two-line Pydantic schema plus an Instructor client:

1
2
3
4
5
6
7
8
9

class QuestionList(BaseModel):
    questions: list[str]

result = await client.chat.completions.create(
    model=FLASH_MODEL,
    messages=[{"role": "user", "content": prompt}],
    response_model=QuestionList,
    max_retries=2,
)

That response_model=QuestionList is doing the work. Instructor handles the JSON schema, the validation and the retry-on-malformed-output. No regex over LLM output.

Instructor is a library that wraps an LLM client and forces the response into a Pydantic schema. If the model returns malformed JSON, Instructor catches it, re-prompts with the error and retries. This way, we can stop writing parsers and instead just write schemas. The code is much cleaner.

Output — data/interim/question_pool.jsonl, one JSON per line:

1
2
3
4
5

{"question": "postgres or mongo for a new project?"}
{"question": "how do I tell my manager I'm leaving without burning the bridge"}
{"question": "is there a way to tell if a film is going to be good in the first 10 min"}
{"question": "what's a healthy amount of olive oil per day"}
{"question": "my landlord just said i 'agreed verbally' to a rent increase. did i?"}

What to check: They should sound like real things a real person would text a real friend. If they don’t sound like your target, edit CATEGORY_PROMPT_TEMPLATE in prep/distill.py until they do.

Step 2 — Generate the answers

1

uv run python -m prep answers

Gemini 2.5 Pro, with persona_prompt.md as the system prompt, answers each question in character. Same Instructor pattern as Step 1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

class PersonaAnswer(BaseModel):
    response: str

result = await client.chat.completions.create(
    model=PRO_MODEL,
    messages=[
        {"role": "system", "content": persona},
        {"role": "user", "content": question},
    ],
    response_model=PersonaAnswer,
    max_retries=3,
)

Output is appended per-question to data/interim/gemini_synth_v0.pairs.jsonl as the responses arrive.

Output — Pair-schema JSONL ({prompt, response, source, score, meta}):

1
2

{"prompt": "postgres or mongo for a new project?", "response": "postgres. unless you have a genuinely document-shaped problem and can articulate it in one sentence, you don't have one. mongo is a great way to discover halfway through year two that you actually wanted joins all along.", "source": "gemini_synth_v0", "score": null, "meta": {"model": "gemini-2.5-pro"}}
{"prompt": "my landlord just said i 'agreed verbally' to a rent increase. did i?", "response": "almost certainly not. verbal variations to a tenancy basically need you to have actively agreed — not just listened to him say it. what jurisdiction are you in? in england, statutory section 13 notices are the proper route and they have to be in writing. tell him you'd like it in writing, and watch the energy shift.", "source": "gemini_synth_v0", "score": null, "meta": {"model": "gemini-2.5-pro"}}

A thousand pairs takes ~10–15 minutes against Pro paid tier and costs about $5.

Crash-safe: The answer generation script resumes from the first unanswered prompt after a crash.

Spot-check the corpus manually before training. Open the JSONL and read a sample. Verify it sounds like our character. If the persona prompt is still too thin, it’s time for us to go back and tighten it.

Synth quality decides eval quality. We cannot fine-tune our way past noisy data. This is the single highest-leverage step in the whole pipeline and it’s almost entirely about how good the persona prompt is. I learnt the hard way that everything downstream depends on this step.

Step 3 — Clean the corpus

1

uv run python -m prep filter

Three filters, one pandas DataFrame, chained with .pipe():

1
2
3
4
5
6

df = (
    load_pairs(interim)
    .pipe(normalize)
    .pipe(filter_english)
    .pipe(dedupe)
)

What each filter is doing:

Sample output:

1
2
3
4
5
6
7
8
9

$ uv run python -m prep filter

>>> filter
Loading from 1 sources: ['gemini_synth_v0.pairs.jsonl']
  loaded: 1000
  after_normalize: 998
  after_language: 967
dedup: 100%|████████████████████████████| 967/967 [00:08<00:00, 110.83it/s]
  after_dedup: 891

Out of 1,000 raw pairs we typically lose 50–100 to language and near-dups. Output is data/processed/cleaned.jsonl, same Pair schema, same fields.

Step 4 — Quality-gate before we train

One thing I learnt the hard way: don’t train directly on raw synth output. A few expensive iterations were wasted on proving just that. Training without evaluation of raw data is training on hope.

1

uv run python -m prep score-and-split

Every (prompt, response) pair goes to Claude Haiku via Instructor and gets scored on five binary axes:

1
2
3
4
5
6
7
8
9

class PersonaScore(BaseModel):
    on_persona: bool
    uses_profanity_appropriately: bool
    takes_stance: bool
    is_helpful: bool
    factual_floor: bool
    rationale: str = Field(
        description="One short sentence on the most notable issue, or 'all pass' if none.",
    )

Five yes/no questions. Binary, not Likert. LLM judges cluster around 3 and 4 on a 1–5 scale and re-runs produce different scores. Binary yes/no is the right way to go.

The judge call is throttled by a semaphore. 10 calls are in flight at a time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

async def _judge_one(client, sem, prompt, response) -> JudgeResult:
    async with sem:
        try:
            score = await client.messages.create(
                model=JUDGE_MODEL,
                max_tokens=JUDGE_MAX_TOKENS,
                system=JUDGE_SYSTEM_PROMPT,
                messages=[{
                    "role": "user",
                    "content": JUDGE_USER_TEMPLATE.format(
                        prompt=prompt, response=response,
                    ),
                }],
                response_model=PersonaScore,
                max_retries=3,
            )
            return JudgeResult(score=score, error=None)
        except Exception as exc:
            return JudgeResult(
                score=None, error=f"{type(exc).__name__}: {exc}",
            )

A response passes_all if and only if every axis is True. The filter keeps those rows, drops the rest, shuffles and splits train/val at 95/5.

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

$ uv run python -m prep score-and-split

>>> score-and-split
judging: 100%|███████████████████████████| 891/891 [02:47<00:00,  5.32it/s]
data/processed/eval_reports/dataset_scored_20260526T143000Z.jsonl
data/processed/eval_reports/dataset_summary_20260526T143000Z.json
passes_all: 73.4%
  on_persona: 85.2%
  uses_profanity_appropriately: 81.7%
  takes_stance: 96.3%
  is_helpful: 89.1%
  factual_floor: 88.5%
kept: 654/891
train: 622 -> data/processed/train.jsonl
val:   32 -> data/processed/val.jsonl

Visually, that funnel looks like:

flowchart TB
    A[891 cleaned pairs] -->|Claude Haiku
5-axis binary judge| B{passes_all?}
    B -->|654 pass| C[kept]
    B -->|237 reject| D[dropped
26.6%]
    C -->|95% split| E[train.jsonl
622 rows]
    C -->|5% split| F[val.jsonl
32 rows]

    classDef stage fill:#ecfccb,stroke:#65a30d,color:#365314
    classDef pass fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef reject fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
    classDef output fill:#fce7f3,stroke:#be185d,color:#831843

    class A,B stage
    class C pass
    class D reject
    class E,F output

We’ll lose 20–40% of the corpus to the judge. That’s the eval gate doing its job. The 600-odd rows that survive are a stronger training set than the 1,000 raw rows would have been.

The scored JSONL keeps everything, rejected rows included, with the judge’s rationale on each one.

1

jq -c 'select(.eval.passes_all == false)' data/processed/eval_reports/dataset_scored_*.jsonl | head

1
2
3

{"prompt": "what's the capital of australia", "response": "...", "source": "gemini_synth_v0", "score": null, "meta": {"model": "gemini-2.5-pro"}, "eval": {"on_persona": false, "uses_profanity_appropriately": true, "takes_stance": true, "is_helpful": true, "factual_floor": true, "passes_all": false, "rationale": "Response is correct but reads like a Wikipedia paragraph; no Monty voice."}}
{"prompt": "should I get a cat or a dog", "response": "...", "source": "gemini_synth_v0", "score": null, "meta": {"model": "gemini-2.5-pro"}, "eval": {"on_persona": true, "uses_profanity_appropriately": true, "takes_stance": false, "is_helpful": true, "factual_floor": true, "passes_all": false, "rationale": "Hedges between both options; doesn't take a stance."}}
{"prompt": "how does GraphQL work", "response": "...", "source": "gemini_synth_v0", "score": null, "meta": {"model": "gemini-2.5-pro"}, "eval": {"on_persona": false, "uses_profanity_appropriately": true, "takes_stance": true, "is_helpful": true, "factual_floor": true, "passes_all": false, "rationale": "Lecture-format with bulleted list; not a friend texting back."}}

These rationales are useful input for the iteration loop in Part 2.

This step costs ~$2 in Haiku calls per run. Please re-run it after every persona-prompt change. That’s the only way to know whether a “tightening” actually did anything.

One command for everything in Part 1

1

uv run python -m prep

Runs questions → answers → filter → score-and-split in order. Each stage is idempotent or resume-safe, so re-running after a partial completion is cheap.

By the end we should have:

• data/processed/train.jsonl — ~600 hand-judged in-character pairs
• data/processed/val.jsonl — ~30 held-out pairs for eval in Part 2
• data/processed/eval_reports/dataset_summary_<stamp>.json

That passes_all percentage is our data ceiling. If our synth corpus only passes the judge 50% of the time, the trained model will not score higher than that. We cannot fine-tune past the quality of our data, no matter how carefully we pick hyperparameters. If the data ceiling sits in the 70–80s, we have a real shot at a respectable model. If it doesn’t, we need to revisit the persona prompt before we spend any money on a GPU.

In Part 2, that ceiling becomes the number to beat with the trained adapter against the held-out val set.

Part 2 — Training, Eval, Iterate, Ship — coming soon. It picks up from the train.jsonl and val.jsonl we just built and walks through the LoRA SFT run on Qwen3-4B, eval against the same Haiku rubric, the hand-correction loop and merging to GGUF for local inference.

Code: github.com/arunma/learn-you-an-sft. The story behind the recipe: Costume vs Character →.