Lesson 3 · ~12 minutes

Tuning your bands on real data

Lessons 1 and 2 used 0.8 and 0.5 as placeholders. This lesson shows how to choose your own lines from evidence.

Your win for today: given a target accuracy, find the confidence line that automates as much as possible without missing the target. Then write the script that finds it on your own data.

Why 0.8 is just a guess

Choosing a line isn't something the model does for you. A model gives you a prediction with a confidence, and turning that into an action is a separate decision that depends on what a mistake costs you.³ A wrong FAQ tag is cheap; a wrong refund isn't. That's why TypeSafe's own pattern gives each action its own line, set by "the consequences of acting on a wrong classification."²

Mental model: a dial with two readings. Move the line right and Jev acts on fewer cases (lower coverage), but the cases it does act on are more often right (higher accuracy). Move it left and you get the opposite. Tuning means picking the point on that dial where accuracy just meets what your error cost allows.

Where the model breaks down: the dial is only valid for the model version, the question wording and the kind of traffic you measured it on. Change any of those and you have to measure again.¹

🏷️labeled setPast examples where you know the right answer, such as the team that actually handled each ticket.

📈coverageThe share of traffic at or above the line, which Jev handles automatically.

🎯accuracyHow often Jev is right within that automatic slice, not across everything.

⚖️targetThe minimum accuracy your error cost allows, for example 95%.

Play with the dial

Each dot is one labeled ticket that went through Jev, placed by its confidence. Green means Jev was right and red means it was wrong. Drag the slider and watch the two numbers trade off. This data is synthetic, made up for practice.

Notice three things:

Red dots are denser on the left. Low confidence predicts mistakes, which is why the line works at all.
The bucket bars are noisy. With only 80 examples, some buckets hold just a handful of dots, so their percentages jump around. That's exactly why Copes suggests labeling "a few hundred."¹ Make decisions with the cumulative numbers in the readout, not single buckets.
There's no free lunch. Every step right removes some red dots, and some green ones too.

Challenge (counts toward your practice): your target is 95% accuracy. Find the lowest line that meets it, then press Lock in.

Doing it on real data

This process has real stages, so step through it:

Label real past examples with the right answer, such as the team that actually handled each one. Copes suggests "a few hundred if you can."¹

Collect: run every example through Jev once, using the exact production question, and save the choice and confidence next to the label.¹

Measure: for each candidate line, work out coverage and the accuracy of the automatic slice. No new API calls are needed.¹

Pick the lowest line whose accuracy meets each action's target. Riskier actions get higher targets and therefore higher lines.²

Ship, then repeat when things change. When the model version or the question wording changes, re-run and compare the tables before switching. "Wording moves confidence too."¹

The minimal reference model

Two short scripts in the same shape as Copes' collect.js and thresholds.js.¹ Run the first one once (it costs API calls) and the second as often as you like (it runs locally).

// collect.ts: run each labeled example through Jev ONCE
import { choice, TypeSafeClient } from '@typesafe-ai/sdk'
import { readFileSync, writeFileSync } from 'node:fs'

const MODEL = 'jev-1.13.0' 1
const labeled = JSON.parse(readFileSync('labeled.json', 'utf8'))
//    → [{ message, team }]
const client = new TypeSafeClient()

const rows = []
for (const ex of labeled) {
  const { answers } = await client.systemOne({
    model: MODEL,
    state: { message: ex.message },
    questions: {
      team: choice('Which team should handle `message`?', TEAMS), 2
    },
  })
  const { choice: picked, confidence } = answers.team
  rows.push({ label: ex.team, choice: picked, confidence })
}
writeFileSync(`answers-${MODEL}.json`, JSON.stringify(rows))3

// thresholds.ts: per line, how much is automatic? how right?
import { readFileSync } from 'node:fs'
const file = 'answers-jev-1.13.0.json'
const rows = JSON.parse(readFileSync(file, 'utf8'))
const pct = (a: number, b: number) =>
  b ? (100 * a / b).toFixed(1) + '%' : '–'

for (const t of [0.5, 0.6, 0.7, 0.8, 0.9]) {
  const auto  = rows.filter(r => r.confidence >= t) 4
  const right = auto.filter(r => r.choice === r.label).length
  console.log(t, 'coverage', pct(auto.length, rows.length),
                 'accuracy', pct(right, auto.length))
}
// Pick the LOWEST t whose accuracy meets this action's target. 5

Pin the model version. The dial belongs to one version. The SDK's systemOne({ model }) overrides the client default, and that default is jev-latest, which moves when a new version ships (SystemOneRequest, TypeSafeClientConfig).
Use the exact production question. Any wording change shifts confidence, so measure what you'll actually ship.
Name the file after the model. When a new version comes out, you diff two tables before switching.
Measure accuracy only over the automatic slice. That's the set your code will act on without a human.
Each action gets its own target. Tagging might accept 85%, while refunds might need 95% or more. Run the same table and choose different lines.

Jev's rate limit is 80 requests per second.⁵ A sequential loop over a few hundred examples stays well under it.

Practice: from memory

1. You raise the act line from 0.7 to 0.9. What happens?

2. You reworded the question to make it more atomic. What now?

3. Which action needs the higher accuracy target?

4. Accuracy for choosing a line should be measured over…

5. (Lesson 2) noul('Was the order shipped more than 5 days ago?') breaks which rule?

6. (Lesson 1) With lines at act: 0.8, confirm: 0.5, confidence comes back as 0.55. What happens?

🏆 Milestone challenge: Tune thresholds

Collect at least 50 labeled examples from a real feature. A few hundred is better.
Run collect.ts and thresholds.ts. If direct sign-up is closed, use the playground or the Vercel AI Gateway (see the cheat sheet).
Paste me the table, the line you chose for each action, and the error-cost reasoning behind each target.

If you don't have data yet, write down which actions your feature would take, the target for each, and why. That earns a partial review. I'll mark Tune thresholds mastered when real numbers come in.

Go deeper

📄 Article: How to use Jev confidence scores (Flavio Copes)
The full version of this lesson's scripts, with output tables. Read the "collect" and "thresholds" sections.
📄 Docs: Confidence-gated routing (TypeSafe)
Official pattern for giving each action its own line, with a banking example.
🛠 Interactive: Thresholds and the confusion matrix (Google ML Crash Course)
The general, model-agnostic version of today's dial, with a slider over three toy datasets. Try the "Imbalanced" one.
📄 Docs: Tuning the decision threshold (scikit-learn)
A clear explanation of why choosing a line is a decision problem separate from the model, with cost-sensitive examples.
▶ Video · 7:40: "TypeSafe Jev: Precision AI via Confidence-Based Decision Architecture" (Some Hot Tech Things)
A community overview of confidence-based design. It isn't official and I haven't watched it, so check its claims against the docs.
▶ Video · 5:19: "TypeSafe Jev For Beginners" (GenAI Geek)
A community walkthrough in the playground. Useful if you want to see calls run before writing code. Same caveat as above.

Stuck or curious? Ask me follow-up questions in the terminal. I'm your teacher for this, and nothing is too basic to ask.