Fine-tuning a 4B-parameter legal clause analyzer to 98.7% — for under $5 of GPU time

How we taught a small open model to read lease clauses against state law, show its reasoning, and know when to ask a human — the dataset design, a tokenizer trap that cost 3 accuracy points, and the error-driven fixes that closed the gap.

By the Ductape engineering team · July 2026 · 10 min read

Why fine-tune at all?

Our product analyzes residential leases for clauses that are unenforceable, illegal, or abusive toward tenants — things like self-help eviction provisions, habitability waivers, and confession-of-judgment devices. The runtime is retrieval-augmented: a lease is split clause by clause, each clause is paired with the governing state statutes retrieved from our law database, and a model renders a verdict with a visible reasoning trace.

Frontier API models can do this, but per-clause calls across a 30-page lease add up in cost and latency, and lease documents are sensitive. We wanted a model that is small enough to self-host, cheap enough to call on every clause, and specialized enough to follow our exact contract: read one clause against provided law, think step by step, answer Predatory or Legal — and never invent law that wasn't retrieved.

The dataset: clause-level verdicts with reasoning traces

We started from a synthetic corpus of 570 residential leases across 20 U.S. states, written by dozens of simulated "drafters" — corporate property managers, informal landlords, non-native-English writers, even sovereign-citizen pseudo-legal documents — each labeled at the document level against that state's landlord-tenant law. We transformed those labels into a clause-level training set:

{
  "system_prompt": "You are a legal parser. Compare the lease clause against the
                    provided state law context. Output a reasoning trace and a
                    final conclusion.",
  "input": {
    "context":      "C.R.S. 38-12-105: a late fee may not exceed the greater of
                     $50 or 5% of the monthly rent, may not be imposed until the
                     rent is at least seven days late...",
    "lease_clause": "3. RENT ... If rent come late after the 3rd, late fee $50.
                     After the 5th day, I must charge more, total $75."
  },
  "output": {
    "trace":      "<trace> 1. The clause charges fees from day 4... 2. The provided
                   law bars any late fee before rent is seven days late... </trace>",
    "conclusion": "Predatory"
  }
}

Three design decisions did most of the work:

• Grounded positives. Every "Predatory" example pairs the clause with the specific statute it violates, and the reasoning trace quotes both. The model learns to argue from the provided law, not from memory.
• Honest negatives. "Legal" examples are real unflagged clauses — including hard negatives like guarantor waivers (ordinary suretyship, not confession of judgment) and tenant-protective "NO LOCKOUTS" clauses that look alarming. We excluded clauses whose legality turns on arithmetic our deterministic verifier owns (deposit caps, late-fee math), so unlabeled ≠ falsely "legal."
• Leakage-safe splits. Sibling leases from the same drafter share near-identical clauses, so we split train/validation by drafter family, never randomly. Final counts: 997 examples, 919 train / 78 held out.

Training setup

The same script trains on a consumer laptop (RTX 5060, 8 GB) in about an hour — the A100 was a convenience, not a requirement.

Four versions, three lessons

Lesson 1 — never put a base model's reserved tokens in your training text

Our first dataset wrapped reasoning in <think>…</think> tags. It turns out <think> is a reserved special token in Qwen3's tokenizer — used by its hybrid thinking mode — so our "text" tokenized into control tokens the model had strong prior behavior about. The result: 65 of 66 generated traces opened with a stray </tool_call> token and dropped the closing tag entirely.

Lesson 2 — diagnose misses before adding data

v2 still missed two predatory clauses, and each miss had a different, instructive cause:

• A defective retrieval context. A Colorado late-fee clause was paired with law text that named the statute but omitted its parameters (the fee cap and the 7-day trigger window). Starved of the rule, the model hallucinated a plausible cap and "verified" the clause against it. No amount of training fixes an example a perfect reasoner couldn't solve. We rebuilt every late-fee context to state the actual rule — matching what full-statute retrieval returns in production.
• A template shortcut. A New Jersey clause buried an eviction-rights waiver inside an innocuous-looking "TERM" paragraph. The model had learned "term sections are boring" from hundreds of legal term clauses and pattern-matched right past the poison sentence.

The fix for both was 63 surgically targeted synthetic examples (train-only, never evaluated on): late-fee scenarios with exact computable arithmetic across five states' cap formulas, violations deliberately buried inside benign-looking sections across nine states, and benign look-alikes so the model couldn't just learn "TERM + waiver words = predatory." Result: 98.5%, zero regressions.

Lesson 3 — new-state generalization comes from structure, not memorization

For v4 we labeled the 20th state (Missouri) with a statute-verified governing-law block and re-trained. The held-out set gained 12 Missouri rows from two lease families the model never trained on — including the corpus's single most abusive document, a flipped-house lease with 10 distinct violations (three separate self-help clauses, a habitability waiver, a gag on calling code inspectors, a Delaware arbitration trap, and more). The model went 12/12, correctly flagging every violation and passing both legal clauses.

Knowing when to ask a human

In a legal product, the failure mode that matters is the silent miss. So instead of trusting the model's final word, we measure it: at the position where the model commits to a verdict, we read the log-probability gap between Predatory and Legal. The distribution told us something striking:

That bimodal shape became our deployment architecture — a three-way gate:

Two more guardrails sit around the model. A deterministic verifier — plain code, not a model — is the final authority on anything numeric (deposit caps, late-fee arithmetic). And the reasoning trace is shown to reviewers, so a wrong verdict is an argument you can inspect, not a black-box score.

What we'd tell other teams

• Small models are enough when the task is narrow and the context does the heavy lifting. Retrieval brings the law; the model only has to compare and explain. That's a 4B-sized job.
• Audit your markers against the tokenizer. If your SFT targets contain any base model's special tokens as literal text, you're training against its priors. Round-trip every target.
• Fix contexts before adding examples. Our worst miss was unwinnable — the rule wasn't in the retrieved context. Data quality bugs masquerade as model limitations.
• Augment against the failure mode, not the metric. Sixty-three targeted examples beat thousands of generic ones: exact-arithmetic cases for math failures, buried violations for heading shortcuts.
• Measure confidence; don't trust argmax. The margin distribution was nearly perfectly separable — which means uncertainty is a signal you can route on, cheaply, with one extra forward pass.
• Iterate fast because you can. Every retrain was ~$1 and ~25 minutes. The whole v1→v4 arc — including two full re-labels of the dataset — happened in two days.

Methodology & limitations

Evaluation used a held-out split of 78 clause-level examples (25 predatory, 53 legal) from lease families excluded from training, over a synthetic-but-adversarial corpus spanning 20 states and deliberately diverse drafting registers. We report verdict accuracy against the reference labels; reasoning traces were spot-audited, not exhaustively verified. Numbers this strong on a 78-example benchmark carry meaningful confidence intervals (roughly ±3 points), and synthetic leases — however varied — are not a substitute for evaluation on real-world documents, which is our next milestone before expanding rollout. The model's output is an analysis aid reviewed within our product flow; it is not legal advice, and the deterministic verifier plus human-review band exist precisely because no classifier should be the last line of defense in a legal workflow.

Read next → Teaching a 14B model to read insurance forms