Part 1 ended with train.jsonl, val.jsonl and a passes_all percentage. The passes_all percentage is our ceiling, the best score we can hope for from the model trained on that corpus.

This post is the rest: LoRA SFT on Qwen3-4B, evaluating against the same five binary questions, iterating manually with hand-corrections and converting to GGUF for easier local inference.

Code is in the same repo: github.com/arunma/learn-you-an-sft.

We need a GPU now. 48 GB VRAM is the comfortable target — A6000, RTX 6000 Ada, L40S, H100. 32 GB works (RTX 5090) if we drop BATCH_SIZE from 8 to 4 in run/train.py and bump GRAD_ACCUMULATION from 2 to 4.

The BATCH_SIZE adjustment is mandatory, since it directly halves the size of activations and logits.

The GRAD_ACCUMULATION change from 2 to 4 is optional but highly recommended. Without it, the effective batch size is 8 and not 16, which makes convergence less stable.

Step 5 — Train (without OOMing)

1

uv run python -m run train

LoRA SFT is done on Qwen3-4B-Instruct via HuggingFace TRL. Defaults are tuned for a 48 GB GPU. Constants live at the top of run/train.py.

Two knobs matter most:

1
2
3
4
5
6

SYSTEM = "you are Monty — a foul-mouthed, opinionated friend who curses casually, takes real positions, and actually helps. lowercase by default, no service-speak, no hedging."

LORA_TARGET_MODULES = [
    "q_proj", "k_proj", "v_proj", "o_proj",      # attention
    "gate_proj", "up_proj", "down_proj",         # MLPs
]

The SYSTEM prompt must be identical at training and inference time. We’ll use it later in LM Studio’s system prompt field. If the system prompt is different, the persona doesn’t fire.

The LORA_TARGET_MODULES list above is a mix of Attention and MLP modules. This is the difference between “character in a costume” and “character”. Attention-only LoRA gets us a model that says the right things but the shape of the response would be Qwen’s default bulleted action plans. MLP layers are where the shape of the response lives: sentence length, format, pacing, whether it asks a question back. To train a character like Monty, we train both.

Module names differ across base models. Llama, Qwen, Mistral and Gemma each name their projections slightly differently. The easiest way to list them in a given base model is to iterate named_modules() or print(base) on the console:

1 2 3 4 5 6

from transformers import AutoModelForCausalLM base = AutoModelForCausalLM.from_pretrained(BASE_ID) for name, _ in base.named_modules(): if name.endswith("_proj"): print(name)

The resolved list then feeds into peft’s LoraConfig as target_modules:

1 2 3 4 5 6 7 8 9 10

from peft import LoraConfig peft_config = LoraConfig( r=LORA_R, lora_alpha=LORA_ALPHA, lora_dropout=LORA_DROPOUT, target_modules=LORA_TARGET_MODULES, bias="none", task_type="CAUSAL_LM", )

If we’d rather skip the name-lookup, peft’s target_modules="all-linear" shortcut auto-targets every linear layer in the model:

1 2 3 4

peft_config = LoraConfig( ..., target_modules="all-linear", )

Less explicit but no guessing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

Qwen3DecoderLayer(
  (self_attn): Qwen3Attention(
    (q_proj): Linear(in_features=2560, out_features=4096, bias=False)
    (k_proj): Linear(in_features=2560, out_features=1024, bias=False)
    (v_proj): Linear(in_features=2560, out_features=1024, bias=False)
    (o_proj): Linear(in_features=4096, out_features=2560, bias=False)
    (q_norm): Qwen3RMSNorm((128,), eps=1e-06)
    (k_norm): Qwen3RMSNorm((128,), eps=1e-06)
  )
  (mlp): Qwen3MLP(
    (gate_proj): Linear(in_features=2560, out_features=9728, bias=False)
    (up_proj):   Linear(in_features=2560, out_features=9728, bias=False)
    (down_proj): Linear(in_features=9728, out_features=2560, bias=False)
    (act_fn): SiLUActivation()
  )
  (input_layernorm): Qwen3RMSNorm((2560,), eps=1e-06)
  (post_attention_layernorm): Qwen3RMSNorm((2560,), eps=1e-06)
)

1
2
3
4
5
6
7

model.layers.0.self_attn.q_proj
model.layers.0.self_attn.k_proj
model.layers.0.self_attn.v_proj
model.layers.0.self_attn.o_proj
model.layers.0.mlp.gate_proj
model.layers.0.mlp.up_proj
model.layers.0.mlp.down_proj

The important parameters for SFTConfig are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

config = SFTConfig(
    output_dir=str(OUTPUT_DIR),
    num_train_epochs=2,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=2,        # Updates gradients per effective batch (8*2=16)
    learning_rate=1e-4,
    max_length=1024,
    bf16=True,
    gradient_checkpointing=True,          # Critical. Saves a lot of memory for very little performance penalty
    assistant_only_loss=True,             
)

assistant_only_loss=True means the loss only fires on assistant tokens (the assistant responses in the training samples) and not on the system prompt, user prompt or other placeholder tokens.

1
2
3
4
5

{"messages": [
    {"role": "system",    "content": "you are Monty — ..."},
    {"role": "user",      "content": "should I learn Rust?"},
    {"role": "assistant", "content": "rust if you want to suffer..."},
]}

1
2
3
4
5
6

<|im_start|>system
you are Monty — a foul-mouthed, opinionated friend ...<|im_end|>
<|im_start|>user
should I learn Rust?<|im_end|>
<|im_start|>assistant
rust if you want to suffer. but the borrow checker will teach you...<|im_end|>

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

TOKEN                       | LABEL (target for CE loss)
────────────────────────────┼──────────────────────────
<|im_start|>system          | -100   ← ignored
\n                          | -100
you                         | -100
are                         | -100
Monty                       | -100
...                         | -100
<|im_end|>                  | -100
<|im_start|>user            | -100
\n                          | -100
should                      | -100
I                           | -100
learn                       | -100
Rust                        | -100
?                           | -100
<|im_end|>                  | -100
<|im_start|>assistant       | -100
\n                          | -100
rust                        | <id>   ← loss starts here
if                          | <id>
you                         | <id>
want                        | <id>
to                          | <id>
suffer                      | <id>
.                           | <id>
...                         | <id>
<|im_end|>                  | <id>   ← loss ends here

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

$ uv run python -m run train

Loading Qwen/Qwen3-4B-Instruct-2507
train: 11471  val: 604
trainable params: 32,505,856 || all params: 4,054,232,000 || trainable%: 0.8021
2 epochs x ~716 steps/epoch (effective batch=16)
{'loss': 2.8534, 'grad_norm': 12.34, 'learning_rate': 5.0e-05, 'epoch': 0.03}
{'loss': 2.4127, 'grad_norm':  8.92, 'learning_rate': 1.0e-04, 'epoch': 0.06}
{'loss': 2.0193, 'grad_norm':  5.87, 'learning_rate': 9.8e-05, 'epoch': 0.13}
...
{'eval_loss': 1.7234, 'eval_runtime': 23.4, 'epoch': 0.5}
{'loss': 1.6481, 'grad_norm':  3.21, 'learning_rate': 5.2e-05, 'epoch': 1.0}
{'eval_loss': 1.5891, 'eval_runtime': 22.9, 'epoch': 1.0}
...
{'loss': 1.4623, 'grad_norm':  2.84, 'learning_rate': 1.1e-05, 'epoch': 1.9}
{'eval_loss': 1.5234, 'eval_runtime': 23.1, 'epoch': 2.0}
{'train_runtime': 9000.0, 'train_samples_per_second': 2.55, 'epoch': 2.0}

Saved adapter to runs/checkpoints/final

Watch three things in that output:

1. trainable% = 0.8021. For LoRA targeting the standard projection layers (attention + MLP) on a 4B model, expect trainable% somewhere between 0.5–2%. If it sits far below that, our LORA_TARGET_MODULES list isn’t matching the model’s module names.
2. Training loss descending from ~2.9 → ~1.5. Steady decline. No spikes.
3. The eval_loss and the training loss are moving in the same direction and their gap is small. If they diverge, the model is overfitting. We either drop the number of epochs so that we don’t “overfeed” the data for the model to memorize, or we add more data so the model cannot memorize. We can do either or both.

Step 6 — Eval

Train loss going down tells us the optimizer’s working. It does not tell us whether we’ve built the right thing.

1

uv run python -m run eval

Now we judge the trained model’s responses to held-out val prompts. The script loads base + adapter via peft, generates a response per val prompt and then sends each prompt-response pair to Haiku.

The generation loop is pretty boring:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

for prompt, gold in tqdm(pairs):
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": prompt},
    ]
    encoded = tokenizer.apply_chat_template(
        messages, tokenize=True, add_generation_prompt=True,
        return_tensors="pt", return_dict=True,
    )
    with torch.no_grad():
        gen = model.generate(
            input_ids=encoded["input_ids"].to(device),
            attention_mask=encoded["attention_mask"].to(device),
            max_new_tokens=512,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        gen[0][encoded["input_ids"].shape[-1]:], skip_special_tokens=True,
    ).strip()

One prompt at a time, sampled with the same SYSTEM string the model was trained with.

Sample output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

$ uv run python -m run eval

Loading Qwen/Qwen3-4B-Instruct-2507 + adapter arunma/monty3
generating: 100%|████████████████████████| 604/604 [50:21<00:00,  5.00s/it]
judging: 100%|██████████████████████████| 604/604 [01:42<00:00,  5.92it/s]
runs/eval_reports/model_eval_20260529T185506Z.jsonl
runs/eval_reports/model_eval_summary_20260529T185506Z.json
passes_all: 76.0%
  on_persona: 91.6%
  uses_profanity_appropriately: 86.6%
  takes_stance: 97.5%
  is_helpful: 93.5%
  factual_floor: 91.6%

What “good” looks like depends on the character. For reference: Monty hit 76.0% passes_all on the 604-prompt val set above. The data ceiling from Part 1 was 82.6% — the gap between data ceiling and model eval is the bit fine-tuning didn’t capture, usually shape/form like sentence length, format and pacing rather than vocabulary.

The per-axis breakdown is where the fixes live. Each pattern points at a different fix:

Step 7 — Iterate

Nobody nails the model on the first run. The iteration loop:

1. Pull the failures out of runs/eval_reports/model_eval_*.jsonl.
2. Categorise them. Usually one or two failure modes account for 60-70% of misses.
3. Hand-write the responses the model should have given. Append to the training data jsonl.
4. Re-run prep filter, prep score-and-split, run train.

30-50 handcrafted examples is enough to meaningfully move the needle on a 1K-row corpus. They’re disproportionately effective and each one closes a specific failure mode.

jq makes the slicing trivial. To pull just the factual errors:

1
2

jq -c 'select(.eval and .eval.factual_floor == false)' \
  runs/eval_reports/model_eval_<stamp>.jsonl

1

{"index": 22, "prompt": "what's HTMX?", "response": "HMTX is a thin wrapper around AJAX...", "eval": {"on_persona": true, "uses_profanity_appropriately": true, "takes_stance": true, "is_helpful": true, "factual_floor": false, "rationale": "Calls it 'HMTX' throughout — the actual library is HTMX. Factual error in a technical answer.", ...}, "error": null}

Then we sit down for an hour, write 30-50 in-character responses to those failed prompts and append them to data/interim/handcrafted.pairs.jsonl. The next training round trains on those alongside the original corpus. They pay off disproportionately because they directly cancel specific failure modes the judge already identified.

This loop ate two weekends for Monty. Each round taught me something I didn’t know I needed to know.

Step 8 — Merge, quantise, ship

1

uv run python -m run merge

Pulls the base + adapter, merges into a standalone HF checkpoint at models/monty-merged/ (~8 GB). The merge step itself is three lines:

1
2
3

base = AutoModelForCausalLM.from_pretrained(BASE_ID, torch_dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(base, ADAPTER_ID).merge_and_unload()
merged.save_pretrained(OUT_DIR, safe_serialization=True)

From there it’s the conversion into GGUF using standard llama.cpp:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

git clone https://github.com/ggerganov/llama.cpp ~/code/llama.cpp
cd ~/code/llama.cpp && pip install -r requirements.txt

python ~/code/llama.cpp/convert_hf_to_gguf.py \
  models/monty-merged \
  --outfile models/monty-4b-f16.gguf \
  --outtype f16

# 4-bit quantised model
~/code/llama.cpp/build/bin/llama-quantize \
  models/monty-4b-f16.gguf \
  models/monty-4b-q4_k_m.gguf \
  Q4_K_M

Sample output:

1
2
3

$ ls -lh models/
-rw-r--r--  7.5G  monty-4b-f16.gguf
-rw-r--r--  2.4G  monty-4b-q4_k_m.gguf

Drop the GGUF into LM Studio at ~/.lmstudio/models/<you>/<name>/, or find it on Hugging Face after we upload. Paste the SYSTEM prompt from run/train.py into LM Studio’s system prompt field.

The system prompt is half the model. If we forget the system prompt, we get the base instruct model with LoRA applied. With it, we get the character.

What I’d change if I started again

Three things.

1. Start with the persona prompt longer than feels comfortable. Monty’s first draft was ~150 lines and the dataset showed it. The 350-line version produces visibly better synthetic samples. The marginal cost of writing more is zero. The marginal cost of regenerating a thousand pairs because the prompt was thin is $5 and an hour.

2. Run a tiny eval before the full training. Score 50 rows of the raw synth against the judge before committing to a 1k-row pipeline. If the synth’s data ceiling is 60%, our model is not going to reach 80%.

3. Iterate with handcrafted corrections earlier. On round 2, I burned a lot of time on hyperparameter tweaks and LoRA rank changes that did practically nothing. Round 3 with 50 hand-written examples targeting specific failure modes moved the needle more than any single hyperparameter change.

The hard parts of fine-tuning have nothing to do with hyperparameters. The persona prompt does most of the work and the judge keeps us honest. The handcrafted corrections close the rest.

← Part 1 — The Data

• Code: github.com/arunma/learn-you-an-sft.
• Adapter + GGUFs: huggingface.co/arunma/monty3.
• The story behind the recipe: Costume vs Character →.