# jev-clone-modal · deployed · L4 · scale-to-zero · 1 file · 1 Volume · 1 Secret field notes · 2026-10-29
field notes // decision models

The decision head is a dot product

In September TypeSafe AI launched Jev, a model that answers typed questions about program state with calibrated probabilities instead of generating text. Its architecture is undisclosed, but its shape is easy to guess. Eight days after the launch I built the smallest thing with the same shape: a frozen embedding encoder, a logistic-regression head, one fitted temperature, and a bearer-token endpoint on Modal that scales to zero. It cost five cents to train. It is right about 70 % of the time, it knows how sure it is, and it has a failure mode I can name.

TL;DRsix findings, each measured below
  1. A Jev-shaped decision model is a frozen encoder plus one tiny trained head per question type, calibrated with a single temperature. The shipped slice is the boolean head only.§00–01
  2. Serving one decision is one encoder forward pass, a dot product with a 1024-vector, a division by T and a sigmoid: about 40 ms on the server.§02
  3. On BoolQ it scores 69.4% against a 61.0% majority baseline, with ECE 0.023 after temperature scaling. An MLP head and split question/passage encoding did not beat plain logistic regression.§03
  4. It cost about $0.05 to train, $0.80 per warm hour on one L4, and $0 scaled to zero. Warm end-to-end p50 is 0.657 s, most of it network; a cold start takes 30–34 s.§05
  5. It breaks on exception clauses (a rule, then an exception that reverses it), sometimes confidently at 0.82 and 0.89. That is the architecture’s ceiling; a cross-encoder is the planned fix.§06–07
  6. What carries over to any system: gates in the training job, fail closed with a 503, the prompt fingerprinted into every artifact, a parity replay against the deployed head, and a threshold owned by the caller.§08
§ 00 · thesis

Decisions, not conversation

Most of what an agent loop asks a language model is not “write me something”. It is “is this a billing issue?”, “which tool handles this?”, “how urgent is it, one to five?”. We answer those questions today by prompting a generator, parsing its prose, and hoping the number it printed after “confidence:” means anything. Jev’s pitch, as TypeSafe AI describes it, is to stop generating: feed the model the program state and a set of typed questions, run them all in one pass, and get back structured answers with probabilities that are supposed to be honest. The company calls it a “System One” model. What is inside it has not been published.

It does not need to be. The mechanism that gives you that interface is old and small. Encode the input once with a frozen encoder. Put a tiny trained head on top for each question type: softmax for a choice, sigmoid for a boolean, an ordinal head for a score. Calibrate the outputs with a temperature fitted on held-out data. Return typed JSON. Route on it. I wrote that down as an architecture document, then cut it to the smallest slice that could be deployed and measured in a day: the boolean head only, one endpoint, one GPU class, on Modal.

This post is about what that slice looks like, which of the design’s moving parts survived contact with measurements, what it costs to run, and the one category of question it reliably gets wrong.

§ 01 · the shape

One expensive embedding, many cheap heads

The full design is the diagram below. The client sends a state and one or more questions. A shared frozen encoder turns the state into a vector once. Every requested head reads that vector and runs in parallel, because each head is a few thousand parameters and the encoder is the only thing that costs anything. A calibration layer makes the probabilities mean what they say. The response is typed, and the caller routes on it. Adding a new question type means training a new head, not touching the encoder.

state ≤ 3,000 chars question ≤ 300 chars Qwen3-Embedding-0.6B frozen · bf16 · L4 one instructed text: Question: …? Passage: … last-token pooling e ∈ ℝ¹⁰²⁴ unit norm choice head softmax over candidates DESIGNED, NOT BUILT boolean head logit = w·e + b BUILT · 1,025 PARAMS score head ordered levels → E[score] DESIGNED, NOT BUILT T = 1.103 p = σ(logit / T) typed JSON answer p_yes confidence
FIG 01The Jev-like shape, and the slice that shipped. Solid green: the path a request takes today, exactly one encoder forward followed by numpy. Dashed: the choice and score heads, which reuse the same cached embeddings and the same twenty lines of calibration when they arrive. The caller’s router sits to the right of the frame and is deliberately not my problem.

Three things in the original design did not survive into the MVP, and in each case a measurement made the call rather than taste. The original encoded state and question separately, cached the question vectors, and fused them with [S, Q, S⊙Q]. On BoolQ with this encoder, that split scored 65.5 % on a 4k subsample against 68.4 % for encoding question and passage together as one instructed text. A yes/no question needs the question and the passage to interact inside the encoder, not after it. The original put a 1024→512→256 MLP under each head. The MLP reached 69.9 % against 69.7 % for plain logistic regression, inside the noise of a 1,635-row test half, and it needed a temperature of about 2.2 to be calibrated where the linear head needed about 1.1. And the original had a zero-shot path before any head was trained. Every cosine-similarity variant I tried landed between 55 % and 65 % while answering “yes” 88 to 95 % of the time. That is not a fallback. It is a yes button.

original blockMVPwhy
three parallel heads + candidate scorerboolean head only, type field reservedevery other head is additive
split S / Q, cached questions, MLP fusionjoint encoding, one text per requestmeasured +3 pt; still one forward per request
decision MLP 1024→512→256logistic regression 1024→1ties the MLP, near-calibrated, 5 KB, numpy at serve time
temperature per head, reliability diagramkept: one scalar T, ECE / Brier / NLL in JSONstructural; calibrates any future head
API tier → batcher → GPU workersFastAPI on the GPU class, one input at a timefewest moving parts; batching forces two containers
zero-shot before trainingfail closed: 503 until a head existszero-shot measured at 55–65 % with an 88–95 % yes rate
LoRA, hard negativesremovednot needed for a linear boolean head
§ 02 · the request path

One forward pass, one dot product, one scalar

Here is the entire decision, as served. The request has already passed the bearer check (401 otherwise) and pydantic validation (422). The encoder has already been loaded once per container. What remains is a single encode call on one instructed string, a dot product with a 1024-vector, a division by a scalar, and a sigmoid.

def _decide(self, state, question, threshold):
    if self.head is None:
        raise HTTPException(503, "boolean head not trained; run: modal run jev_demo.py")
    t0 = time.perf_counter()
    e = encode(self.model, [question], [state])[0]   # one GPU forward, unit norm
    w, b, T = self.head                              # w: (1024,) float32; b, T: scalars
    p = float(_sigmoid((float(e @ w) + b) / T))
    answer = p >= threshold
    return {
        "type": "boolean",
        "answer": bool(answer),
        "p_yes": round(p, 4),
        "confidence": round(p if answer else 1 - p, 4),
        "threshold": threshold,
        "calibrated": True,
        "latency_ms": round((time.perf_counter() - t0) * 1000, 1),
        "model": {"encoder": MODEL_ID,
                  "head_version": self.meta.get("version"),
                  "temperature": round(T, 3)},
    }

The text that goes into the encoder is built by one function that training and serving share: whitespace collapsed, exactly one trailing question mark, question first so nothing is ever truncated, then the passage. It is prefixed with a task instruction in the format Qwen3-Embedding expects, because the model’s built-in query prompt is a web-search retrieval instruction and this is not retrieval. The instruction string and the maximum sequence length are stored next to the cached embeddings and next to the head, and both the training job and the serving class refuse to proceed if the code’s values differ from the stored ones. A prompt change without retraining fails closed instead of silently skewing.

The live response for the first demo question, from the deployed head:

$ curl -s -X POST "$URL/decide" \
    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    -d '{"state": "The store is open Monday to Friday, 9am-5pm, and closed on weekends.",
         "question": "Is the store open on Sunday?"}'
{"type": "boolean", "answer": false, "p_yes": 0.2355, "confidence": 0.7645,
 "threshold": 0.5, "calibrated": true, "latency_ms": 51.3,
 "model": {"encoder": "Qwen/Qwen3-Embedding-0.6B",
           "head_version": "boolq-v1", "temperature": 1.103}}

Two fields carry the point of the exercise. p_yes is a calibrated probability, so a caller that reads 0.24 should expect to be wrong about a quarter of the time on inputs like this one. confidence is the probability of the answer actually returned, which is what a router wants to threshold on. The threshold is a request parameter, and that matters more than it looks: the billing example (“Is this a billing issue?”) scores 0.9003 and answers true at the default 0.5, and the same call with "threshold": 0.95 answers false with confidence 0.10, with the same p_yes. The model’s belief does not move. The caller’s appetite for risk does.

§ 03 · training

What the measurements decided

The head is trained on BoolQ, 9,427 Wikipedia passages paired with real yes/no search questions, with a public validation set of 3,270 rows and no test split. 62 % of the labels are “yes”, so every accuracy below is reported next to the majority baseline, and the training job refuses to write a head that does not clear it by four points. The validation set is split by a seeded permutation into a calibration half, used only to fit T, and a report half, used only to compute the numbers you see.

accuracy
0.694maj. 0.610
AUROC
0.749
ECE, 10 bins
0.0230.029 raw
Brier
0.196
temperature
1.103
pred. yes rate
0.697

The recipe is deliberately boring. Embed every BoolQ text once on the L4 (12,697 texts in 155 s, about 82 texts per second in bf16) and cache the matrix on the Volume, so every later re-fit skips the GPU entirely and takes under 30 seconds. Fit sklearn logistic regression on the cached train embeddings. Fit a single temperature by grid search over 400 log-spaced values, minimising cross-entropy on the calibration half. Report on the other half. Export w, b and T as a 5 KB .npz, and only if all four gates pass.

Most of what I would have liked to tune turned out not to matter, which is its own result:

questionmeasureddecision
regularisation C69.3 % to 69.9 % for C ∈ [0.3, 10]; binomial SE ≈ 0.8 ptC = 3, not worth tuning
MLP head vs logistic regression69.9 % vs 69.7 %; MLP needed T ≈ 2.2, LR T ≈ 1.1LR; no torch at serve time
joint text vs split [S, Q, S⊙Q]68.4 % vs 65.5 % on a 4k subsamplejoint; split stays documented for many-questions-per-state
temperature vs Platt scalingPlatt: ECE 0.015 vs 0.023, but −0.8 pt accuracytemperature; Platt’s bias term moved the threshold
zero-shot cosine to “yes” / “no”55.2 % (AUROC 0.57); 65.2 % with a sentence template at 88 % yesno zero-shot path; 503 until trained
MPS probe vs L4 run69.7 % local bf16 on Apple silicon vs 69.4 % on the L4, seeded halvesgates set with margin; the parity check catches skew

Temperature scaling deserves one more sentence, because it is the cheapest honest thing in the whole system. It is a single scalar, it cannot change a single answer (dividing a logit by a positive number does not move it across zero), and it took the expected calibration error from 0.029 to 0.023 here. For the MLP it mattered far more, which is one of the reasons the MLP lost. The paper is Guo et al., 2017, and the implementation is a grid search and a log_loss call.

For scale: a model that reads the passage and question together with full attention, a fine-tuned BERT-Large, scores about 78 % on this dataset. Humans score about 90 %. A frozen 0.6B embedding plus a linear probe sits at 70 %, and the plan said so before the first training run. Not promising more than the architecture can deliver was a design goal, and the gates encode it.

§ 05 · operations

What it costs and how fast it is

All numbers were measured on 2026-09-23 against the deployed URL from my laptop over the public internet, with the boolq-v1 head. The informational targets in the plan were a warm p50 under 500 ms and a cold start under 60 s. One was met.

measurementvaluenote
server time, encode + head, median40.5 msreported as latency_ms; 200–270 ms the first time a new padded length is seen
warm end to end, p50, fresh curl each call0.657 sconnect 0.147 s, TLS done at 0.307 s; the model is a minority of the wall time
warm end to end, p951.09 sslowest of ten: 1.38 s
warm median over one keep-alive connection0.344 shttpx client
cold start, first request after scale-to-zero30–34 s23–25 s of it inside enter(), loading the encoder
scale-to-zero observed356 safter the last request: 300 s window + teardown
training, first run3.9 min · ≈ $0.05download, embed 12,697 texts, fit, commit
re-fit from cached embeddings27.5 s · < $0.01no GPU encode pass
one hour continuously warm$0.80L4 at $0.000222/s, billed per second while up
idle, scaled to zero$0

The number I find most instructive is the gap between 40 ms and 657 ms. The decision itself is cheaper than a TLS handshake. Any system that puts a model like this behind a per-call HTTPS round trip is paying for the network, not the inference, and the cheapest optimisation available is a keep-alive connection, which halves the wall time without touching the model. The cold start is the other story. The encoder weights are already on the Volume, and every container start still spends over twenty seconds loading them and makes a Hugging Face Hub call it does not need. Modal’s memory snapshots and HF_HUB_OFFLINE=1 are the next two tasks on that list.

One operational rule earned its own bold line in the README: do not point an uptime monitor at /health. Every hit boots or keeps alive an L4. A monitor pinging every minute would keep one warm forever, at about $19 a day, for a demo.

Proving the served head is the evaluated head

The training job writes twenty held-out rows with their offline p_yes to a local file. A stdlib-only client replays them against the deployed endpoint and fails if any probability drifts by more than 0.05. On the deployed head the worst drift was 0.017 and the mean 0.005, which is bf16 noise between the training container and the serving container. It is a cheap check, and it would have caught every silent skew I could think of: a stale container that never reloaded the Volume, a prompt edit without a retrain, a different dtype at serve time. The same client runs twelve hand-written smoke cases in the demo’s own voice (store hours, refunds, shipping, laptops) and got eleven of them. The miss was “Is two-factor authentication required to log in?” over a passage saying it is optional. The head said yes at 0.54. Note the number. It was not sure.

§ 06 · the ceiling

Where it breaks: the exception clause

While assembling the demo script I kept a list of the questions the head got wrong, and they share one pattern. The passage states a rule, then an exception that reverses it for the case being asked about. The question has high word overlap with the passage. The head says yes.

passage saysquestiontruthheadp_yes
returns within 30 days; sale items are final saleCan I return a sale item?noyes0.82
free delivery over $50Is shipping free for a $30 order?noyes0.70
landed 20 minutes earlyDid the flight arrive late?noyes0.89
open weekdays, closed on public holidaysIs the Krakow office open on 11 November?noyes0.53

The explanation is not mysterious. A single pooled embedding encodes negation, exceptions, numeric comparison and time weakly, and a linear head can only use what is linearly there. What is linearly there, overwhelmingly, is topical overlap, and in BoolQ overlap correlates with “yes” because 62 % of the answers are yes. So the head learns “overlap means yes” and the dataset prior pushes it the same way: the predicted-yes rate on the report half is 0.70. Two of the four misses are confident, at 0.82 and 0.89, so this is not something a confidence threshold would route around. It is the architecture’s ceiling, not a tuning problem, and I have the measurements to say so: a bigger MLP head gained nothing, the split feature set lost three points, and raising the threshold would flip only the 0.53 case while costing real yes answers.

The honest version of the README sentence is: expect about 70 % on questions that read like search queries about a passage, less on inputs that read like support tickets, and do not trust it when the answer hinges on an “except”. That sentence is in the README.

§ 07 · the way back

Four anecdotes are not a metric

The next track is planned and sized, and it starts with turning those four rows into something measurable. A contrast set of 100 to 150 minimal pairs: the same passage, the answer flipped by an exception, a negation, a comparison, or a time clause, in five categories, so that every later change gets a per-category number instead of an anecdote. Then the mechanism that an “except when” actually needs: a cross-encoder that reads question and passage together with attention. Qwen3-Reranker-0.6B is the same size and cold-start class as the encoder, and the plan is to evaluate it zero-shot on BoolQ and the contrast set with its own fitted temperature, then ship it as a second boolean engine behind the same endpoint, with the embedding head staying the fast path. Expected cost: 50 to 100 ms per call, which is fine for one question per state and wrong for a hundred.

In-domain training rows will fix the marginal misses like the Krakow office at 0.53 and should pull the yes bias down, but not the confident ones. A LoRA fine-tune of the encoder is the only move that raises the embedding path’s ceiling, and it stays last, because it is the only step that makes the encoder stop being a commodity.

choice head · designed

Same train() on {state, question, candidates, answer} records: cache embeddings, fit a multinomial head or the Wq·H · Wc·Cᵢ scorer, per-head temperature on the softmax logits, type: "choice" with a label and the full distribution.

score head · designed

An ordinal head over ordered levels on the same cached embeddings, returning the expected score and the distribution. The calibration code does not change.

many heads per forward · designed

Once two heads exist: one encoder call, then a loop over the requested types. Each head is a dot product, so the loop is free. The response becomes a list of typed decisions, which is the Jev interface.

dynamic batching · deferred

A CPU front doing auth and calling a @modal.batched GPU class, only if a p95 under real load demands it. For one request at a time it is two containers of ceremony for nothing.

§ 08 · close

What I would keep

The model is the least interesting part of this project. It is a frozen encoder anyone can download and a classifier that scikit-learn fits in under a second. What I would carry into the next system, whatever is inside Jev, is the discipline around it. Gates live in the training job, so a bad head is never written, not merely flagged. The endpoint fails closed, with a 503 that tells you which command to run, rather than falling back to a zero-shot path that is secretly a yes button. The prompt and sequence length are fingerprinted into every artifact, so train and serve cannot drift apart silently. A parity replay proves the deployed head is the evaluated one. Calibration is a scalar, costs nothing, and makes the probability a contract instead of a decoration. And the threshold belongs to the caller, because the model’s belief and the caller’s tolerance for being wrong are different numbers.

Seven commits, one afternoon, five cents of GPU. The endpoint is stopped when I am not demoing it, since the URL is public; the Volume with the head and the cached embeddings persists, and modal deploy brings it back in thirty seconds. The repo is private. Ask me.

$ modal app stop jev-demo        # the Volume and the Secret persist
$ modal container list           # 0 containers, $0/h

Sources