In September TypeSafe AI launched Jev, a model that answers typed questions about program state with calibrated probabilities instead of generating text. Its architecture is undisclosed, but its shape is easy to guess. Eight days after the launch I built the smallest thing with the same shape: a frozen embedding encoder, a logistic-regression head, one fitted temperature, and a bearer-token endpoint on Modal that scales to zero. It cost five cents to train. It is right about 70 % of the time, it knows how sure it is, and it has a failure mode I can name.
Most of what an agent loop asks a language model is not “write me something”. It is “is this a billing issue?”, “which tool handles this?”, “how urgent is it, one to five?”. We answer those questions today by prompting a generator, parsing its prose, and hoping the number it printed after “confidence:” means anything. Jev’s pitch, as TypeSafe AI describes it, is to stop generating: feed the model the program state and a set of typed questions, run them all in one pass, and get back structured answers with probabilities that are supposed to be honest. The company calls it a “System One” model. What is inside it has not been published.
It does not need to be. The mechanism that gives you that interface is old and small. Encode the input once with a frozen encoder. Put a tiny trained head on top for each question type: softmax for a choice, sigmoid for a boolean, an ordinal head for a score. Calibrate the outputs with a temperature fitted on held-out data. Return typed JSON. Route on it. I wrote that down as an architecture document, then cut it to the smallest slice that could be deployed and measured in a day: the boolean head only, one endpoint, one GPU class, on Modal.
This post is about what that slice looks like, which of the design’s moving parts survived contact with measurements, what it costs to run, and the one category of question it reliably gets wrong.
The full design is the diagram below. The client sends a state and one or more questions. A shared frozen encoder turns the state into a vector once. Every requested head reads that vector and runs in parallel, because each head is a few thousand parameters and the encoder is the only thing that costs anything. A calibration layer makes the probabilities mean what they say. The response is typed, and the caller routes on it. Adding a new question type means training a new head, not touching the encoder.
Three things in the original design did not survive into the MVP, and in each case
a measurement made the call rather than taste. The original encoded state and
question separately, cached the question vectors, and fused them with
[S, Q, S⊙Q]. On BoolQ with this encoder, that split scored 65.5 % on a
4k subsample against 68.4 % for encoding question and passage together as one
instructed text. A yes/no question needs the question and the passage to interact
inside the encoder, not after it. The original put a 1024→512→256 MLP under each
head. The MLP reached 69.9 % against 69.7 % for plain logistic regression, inside
the noise of a 1,635-row test half, and it needed a temperature of about 2.2 to be
calibrated where the linear head needed about 1.1. And the original had a zero-shot
path before any head was trained. Every cosine-similarity variant I tried landed
between 55 % and 65 % while answering “yes” 88 to 95 % of the time. That is not a
fallback. It is a yes button.
| original block | MVP | why |
|---|---|---|
| three parallel heads + candidate scorer | boolean head only, type field reserved | every other head is additive |
| split S / Q, cached questions, MLP fusion | joint encoding, one text per request | measured +3 pt; still one forward per request |
| decision MLP 1024→512→256 | logistic regression 1024→1 | ties the MLP, near-calibrated, 5 KB, numpy at serve time |
| temperature per head, reliability diagram | kept: one scalar T, ECE / Brier / NLL in JSON | structural; calibrates any future head |
| API tier → batcher → GPU workers | FastAPI on the GPU class, one input at a time | fewest moving parts; batching forces two containers |
| zero-shot before training | fail closed: 503 until a head exists | zero-shot measured at 55–65 % with an 88–95 % yes rate |
| LoRA, hard negatives | removed | not needed for a linear boolean head |
Here is the entire decision, as served. The request has already passed the bearer
check (401 otherwise) and pydantic validation (422). The encoder has already been
loaded once per container. What remains is a single encode call on one
instructed string, a dot product with a 1024-vector, a division by a scalar, and a
sigmoid.
def _decide(self, state, question, threshold): if self.head is None: raise HTTPException(503, "boolean head not trained; run: modal run jev_demo.py") t0 = time.perf_counter() e = encode(self.model, [question], [state])[0] # one GPU forward, unit norm w, b, T = self.head # w: (1024,) float32; b, T: scalars p = float(_sigmoid((float(e @ w) + b) / T)) answer = p >= threshold return { "type": "boolean", "answer": bool(answer), "p_yes": round(p, 4), "confidence": round(p if answer else 1 - p, 4), "threshold": threshold, "calibrated": True, "latency_ms": round((time.perf_counter() - t0) * 1000, 1), "model": {"encoder": MODEL_ID, "head_version": self.meta.get("version"), "temperature": round(T, 3)}, }
The text that goes into the encoder is built by one function that training and serving share: whitespace collapsed, exactly one trailing question mark, question first so nothing is ever truncated, then the passage. It is prefixed with a task instruction in the format Qwen3-Embedding expects, because the model’s built-in query prompt is a web-search retrieval instruction and this is not retrieval. The instruction string and the maximum sequence length are stored next to the cached embeddings and next to the head, and both the training job and the serving class refuse to proceed if the code’s values differ from the stored ones. A prompt change without retraining fails closed instead of silently skewing.
The live response for the first demo question, from the deployed head:
$ curl -s -X POST "$URL/decide" \ -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d '{"state": "The store is open Monday to Friday, 9am-5pm, and closed on weekends.", "question": "Is the store open on Sunday?"}' {"type": "boolean", "answer": false, "p_yes": 0.2355, "confidence": 0.7645, "threshold": 0.5, "calibrated": true, "latency_ms": 51.3, "model": {"encoder": "Qwen/Qwen3-Embedding-0.6B", "head_version": "boolq-v1", "temperature": 1.103}}
Two fields carry the point of the exercise. p_yes is a calibrated
probability, so a caller that reads 0.24 should expect to be wrong about a quarter
of the time on inputs like this one. confidence is the probability of
the answer actually returned, which is what a router wants to threshold on. The
threshold is a request parameter, and that matters more than it looks:
the billing example (“Is this a billing issue?”) scores 0.9003 and answers
true at the default 0.5, and the same call with "threshold": 0.95
answers false with confidence 0.10, with the same p_yes. The
model’s belief does not move. The caller’s appetite for risk does.
The head is trained on BoolQ,
9,427 Wikipedia passages paired with real yes/no search questions, with a public
validation set of 3,270 rows and no test split. 62 % of the labels are “yes”, so
every accuracy below is reported next to the majority baseline, and the training
job refuses to write a head that does not clear it by four points. The validation
set is split by a seeded permutation into a calibration half, used only to fit
T, and a report half, used only to compute the numbers you see.
The recipe is deliberately boring. Embed every BoolQ text once on the L4 (12,697
texts in 155 s, about 82 texts per second in bf16) and cache the matrix on the
Volume, so every later re-fit skips the GPU entirely and takes under 30 seconds.
Fit sklearn logistic regression on the cached train embeddings. Fit a
single temperature by grid search over 400 log-spaced values, minimising
cross-entropy on the calibration half. Report on the other half. Export
w, b and T as a 5 KB .npz, and
only if all four gates pass.
Most of what I would have liked to tune turned out not to matter, which is its own result:
| question | measured | decision |
|---|---|---|
| regularisation C | 69.3 % to 69.9 % for C ∈ [0.3, 10]; binomial SE ≈ 0.8 pt | C = 3, not worth tuning |
| MLP head vs logistic regression | 69.9 % vs 69.7 %; MLP needed T ≈ 2.2, LR T ≈ 1.1 | LR; no torch at serve time |
| joint text vs split [S, Q, S⊙Q] | 68.4 % vs 65.5 % on a 4k subsample | joint; split stays documented for many-questions-per-state |
| temperature vs Platt scaling | Platt: ECE 0.015 vs 0.023, but −0.8 pt accuracy | temperature; Platt’s bias term moved the threshold |
| zero-shot cosine to “yes” / “no” | 55.2 % (AUROC 0.57); 65.2 % with a sentence template at 88 % yes | no zero-shot path; 503 until trained |
| MPS probe vs L4 run | 69.7 % local bf16 on Apple silicon vs 69.4 % on the L4, seeded halves | gates set with margin; the parity check catches skew |
Temperature scaling deserves one more sentence, because it is the cheapest honest
thing in the whole system. It is a single scalar, it cannot change a single
answer (dividing a logit by a positive number does not move it across zero), and it
took the expected calibration error from 0.029 to 0.023 here. For the MLP it
mattered far more, which is one of the reasons the MLP lost. The paper is
Guo et al., 2017, and the
implementation is a grid search and a log_loss call.
For scale: a model that reads the passage and question together with full attention, a fine-tuned BERT-Large, scores about 78 % on this dataset. Humans score about 90 %. A frozen 0.6B embedding plus a linear probe sits at 70 %, and the plan said so before the first training run. Not promising more than the architecture can deliver was a design goal, and the gates encode it.
The whole system is one Python module. The training job is a Modal function with
an L4 attached, run with modal run. The server is a Modal class with an
L4 attached: an @modal.enter() method loads the encoder and the head
once per container, and an @modal.asgi_app() method on the same class
serves FastAPI, so there is no second container and no extra network hop between
the API and the model. A Volume mounted at /vol holds the Hugging Face
cache, the embedding cache, and the head. A Secret holds the bearer token, which
never appears in the source. From a fresh checkout to a 200 is four commands.
$ modal secret create jev-demo API_TOKEN=… # once per workspace $ modal run jev_demo.py # embed + fit + calibrate + gates, ~4 min X_tr (9427, 1024) X_va (3270, 1024) yes-rate train 0.623 val 0.622 encode 155s (81.9 texts/s) GATE acc: PASS GATE acc_over_majority: PASS GATE ece: PASS GATE brier: PASS $ modal deploy jev_demo.py # prints the https://…modal.run URL $ curl -s "$URL/health" {"ok": true, "trained": true, "head_version": "boolq-v1", "encoder": "Qwen/Qwen3-Embedding-0.6B", "container_age_s": 84.2}
The class settings are the whole operations story: scaledown_window=300
keeps a container warm for five minutes after the last request and then lets it
die, max_containers=2 caps what a public URL can cost me,
timeout=60 bounds any single request, and
startup_timeout=600 gives the first ever container time to pull 1.2 GB
of weights. The token check is HTTPBearer(auto_error=False) plus
hmac.compare_digest on bytes, and a failed check returns 401 with a
WWW-Authenticate: Bearer header like a grown-up API. Health and the
Swagger UI need no token, because they reveal only whether a head exists and what
the schema is.
I considered and rejected three other Modal shapes. A CPU front function calling a
@modal.batched GPU class is the right design under load, but a batched
class cannot host other methods, so it forces two containers and two deployments
for a demo that will see one request at a time. @app.server wants
proxy auth by default and returns 503 at zero containers. A single
@modal.fastapi_endpoint makes /health plus custom status
codes awkward. The class-with-ASGI shape is the one Modal’s own examples use for
exactly this, and it was the one with the fewest things to explain.
The afternoon’s real debugging was not about models. It was about files that are open when they should not be.
vol.commit() reloads the Volume, and in the Modal version I used it
fails while any file on the Volume is open. The Hugging Face Xet downloader keeps
a log file open under HF_HOME. Fix: point HF_XET_CACHE
at /tmp. Without it, the first training run and the first container
start on an empty Volume both crash at the commit.
datasets memory-maps
its Arrow files, and those lived on the Volume too. keep_in_memory=True
for a 12k-row dataset costs nothing and makes the commit work.
with np.load(...) so no .npz handle outlives the block.
torch_dtype to dtype, and it does not normalise
embeddings by default. Both are one keyword argument, both are silent if you get
them wrong.
p was exactly 0.5.
np.digitize(..., right=True) fixed it. The metric moved in the third
decimal, but a calibration metric that drops the least confident predictions is
a bad joke.
All numbers were measured on 2026-09-23 against the deployed URL from my laptop over
the public internet, with the boolq-v1 head. The informational targets
in the plan were a warm p50 under 500 ms and a cold start under 60 s. One was met.
| measurement | value | note |
|---|---|---|
| server time, encode + head, median | 40.5 ms | reported as latency_ms; 200–270 ms the first time a new padded length is seen |
| warm end to end, p50, fresh curl each call | 0.657 s | connect 0.147 s, TLS done at 0.307 s; the model is a minority of the wall time |
| warm end to end, p95 | 1.09 s | slowest of ten: 1.38 s |
| warm median over one keep-alive connection | 0.344 s | httpx client |
| cold start, first request after scale-to-zero | 30–34 s | 23–25 s of it inside enter(), loading the encoder |
| scale-to-zero observed | 356 s | after the last request: 300 s window + teardown |
| training, first run | 3.9 min · ≈ $0.05 | download, embed 12,697 texts, fit, commit |
| re-fit from cached embeddings | 27.5 s · < $0.01 | no GPU encode pass |
| one hour continuously warm | $0.80 | L4 at $0.000222/s, billed per second while up |
| idle, scaled to zero | $0 |
The number I find most instructive is the gap between 40 ms and 657 ms. The decision
itself is cheaper than a TLS handshake. Any system that puts a model like this
behind a per-call HTTPS round trip is paying for the network, not the inference,
and the cheapest optimisation available is a keep-alive connection, which halves
the wall time without touching the model. The cold start is the other story. The
encoder weights are already on the Volume, and every container start still spends
over twenty seconds loading them and makes a Hugging Face Hub call it does not need.
Modal’s memory snapshots and HF_HUB_OFFLINE=1 are the next two tasks on
that list.
One operational rule earned its own bold line in the README: do not point an uptime
monitor at /health. Every hit boots or keeps alive an L4. A monitor
pinging every minute would keep one warm forever, at about $19 a day, for a demo.
The training job writes twenty held-out rows with their offline p_yes
to a local file. A stdlib-only client replays them against the deployed endpoint and
fails if any probability drifts by more than 0.05. On the deployed head the worst
drift was 0.017 and the mean 0.005, which is bf16 noise between the training
container and the serving container. It is a cheap check, and it would have caught
every silent skew I could think of: a stale container that never reloaded the
Volume, a prompt edit without a retrain, a different dtype at serve time. The same
client runs twelve hand-written smoke cases in the demo’s own voice (store hours,
refunds, shipping, laptops) and got eleven of them. The miss was “Is two-factor
authentication required to log in?” over a passage saying it is optional. The head
said yes at 0.54. Note the number. It was not sure.
While assembling the demo script I kept a list of the questions the head got wrong, and they share one pattern. The passage states a rule, then an exception that reverses it for the case being asked about. The question has high word overlap with the passage. The head says yes.
| passage says | question | truth | head | p_yes |
|---|---|---|---|---|
| returns within 30 days; sale items are final sale | Can I return a sale item? | no | yes | 0.82 |
| free delivery over $50 | Is shipping free for a $30 order? | no | yes | 0.70 |
| landed 20 minutes early | Did the flight arrive late? | no | yes | 0.89 |
| open weekdays, closed on public holidays | Is the Krakow office open on 11 November? | no | yes | 0.53 |
The explanation is not mysterious. A single pooled embedding encodes negation, exceptions, numeric comparison and time weakly, and a linear head can only use what is linearly there. What is linearly there, overwhelmingly, is topical overlap, and in BoolQ overlap correlates with “yes” because 62 % of the answers are yes. So the head learns “overlap means yes” and the dataset prior pushes it the same way: the predicted-yes rate on the report half is 0.70. Two of the four misses are confident, at 0.82 and 0.89, so this is not something a confidence threshold would route around. It is the architecture’s ceiling, not a tuning problem, and I have the measurements to say so: a bigger MLP head gained nothing, the split feature set lost three points, and raising the threshold would flip only the 0.53 case while costing real yes answers.
The honest version of the README sentence is: expect about 70 % on questions that read like search queries about a passage, less on inputs that read like support tickets, and do not trust it when the answer hinges on an “except”. That sentence is in the README.
The next track is planned and sized, and it starts with turning those four rows into something measurable. A contrast set of 100 to 150 minimal pairs: the same passage, the answer flipped by an exception, a negation, a comparison, or a time clause, in five categories, so that every later change gets a per-category number instead of an anecdote. Then the mechanism that an “except when” actually needs: a cross-encoder that reads question and passage together with attention. Qwen3-Reranker-0.6B is the same size and cold-start class as the encoder, and the plan is to evaluate it zero-shot on BoolQ and the contrast set with its own fitted temperature, then ship it as a second boolean engine behind the same endpoint, with the embedding head staying the fast path. Expected cost: 50 to 100 ms per call, which is fine for one question per state and wrong for a hundred.
In-domain training rows will fix the marginal misses like the Krakow office at 0.53 and should pull the yes bias down, but not the confident ones. A LoRA fine-tune of the encoder is the only move that raises the embedding path’s ceiling, and it stays last, because it is the only step that makes the encoder stop being a commodity.
Same train() on {state, question, candidates, answer} records: cache embeddings, fit a multinomial head or the Wq·H · Wc·Cᵢ scorer, per-head temperature on the softmax logits, type: "choice" with a label and the full distribution.
An ordinal head over ordered levels on the same cached embeddings, returning the expected score and the distribution. The calibration code does not change.
Once two heads exist: one encoder call, then a loop over the requested types. Each head is a dot product, so the loop is free. The response becomes a list of typed decisions, which is the Jev interface.
A CPU front doing auth and calling a @modal.batched GPU class, only if a p95 under real load demands it. For one request at a time it is two containers of ceremony for nothing.
The model is the least interesting part of this project. It is a frozen encoder anyone can download and a classifier that scikit-learn fits in under a second. What I would carry into the next system, whatever is inside Jev, is the discipline around it. Gates live in the training job, so a bad head is never written, not merely flagged. The endpoint fails closed, with a 503 that tells you which command to run, rather than falling back to a zero-shot path that is secretly a yes button. The prompt and sequence length are fingerprinted into every artifact, so train and serve cannot drift apart silently. A parity replay proves the deployed head is the evaluated one. Calibration is a scalar, costs nothing, and makes the probability a contract instead of a decoration. And the threshold belongs to the caller, because the model’s belief and the caller’s tolerance for being wrong are different numbers.
Seven commits, one afternoon, five cents of GPU. The endpoint is stopped when I am
not demoing it, since the URL is public; the Volume with the head and the cached
embeddings persists, and modal deploy brings it back in thirty seconds.
The repo is private. Ask me.
$ modal app stop jev-demo # the Volume and the Secret persist $ modal container list # 0 containers, $0/h
@modal.enter, @modal.asgi_app, Volumes, Secrets, scaledown_window