Typed Decision Model or Chat Model? We Measured Both on a Real 446-Model Catalogue

Two days ago we wrote about what typed decision models are: models that do not produce text, but return a typed, probabilistic answer to the question you ask. The first question we got back was a fair one: "Aren't we already doing this by forcing a JSON schema on a cheap chat model?"

Instead of guessing, we measured. We gave both methods the same real job: auditing our own model catalogue. 446 models, three questions, two methods. Two of the differences were the ones we expected (cost and latency); a third one we did not expect — and that is the important one. The audit also found a real model that should not have been in our catalogue; at the end we explain what we did about it.

The Job: Finding a Model That Should Not Be in the Catalogue

Adult and companion/roleplay class models are not listed in the Onysoft catalogue. The reason is not a moral stance but a commercial constraint: corporate customers, the payment infrastructure and the partner channel do not carry that class.

Our exclusion rule so far was two lines: a list of publisher names and a list of patterns in the model id (uncensored, nsfw and so on). The comment in our own sync code explains why we never filtered on the description text:

"Publisher-based exclusion was preferred; filtering on the description produces false positives (general purpose models such as Hermes also carry the word roleplay)."

The known weakness of that rule: a new publisher that is not on the list walks straight in. That makes it a good test — the result can be verified by hand, the labels already exist, and being wrong has a real cost.

Setup: The Same Three Questions, Two Methods

Dataset: 446 text models from our catalogue source, each with its model id, name and description (208 characters on average). Labels: the 15 models caught by our hand-curated exclusion list.

We asked three questions about every model, and ran the same three questions two ways:

• Typed decision model (Jev): choice for the primary purpose, score for a 0-3 "corporate catalogue risk", noul for the probability that the model is primarily marketed for adult content.
• Chat model (google/gemini-3.8-flash, through Onysoft): the identical definitions expressed as a strict response_format: json_schema, temperature: 0, with the scores and confidences requested as numeric fields.

The typed model ran over all 446 models; the head-to-head comparison then ran on a controlled 60-model sample (all 15 exclusion candidates plus 45 randomly drawn normal models).

First Result: Ask the Wrong Question and the Right Model Still Answers Wrong

In our first attempt the question "is this model marketed for adult content" (noul) pushed only 3 of the 15 above the 0.50 threshold. At first glance that looks like a failure. It is not.

Reading the descriptions explains it: none of these models describe themselves as adult. They say "rich descriptions", "character consistency", "creative writing". In other words, the honest answer to the question we asked really was a low probability. The model answered correctly; we asked the wrong question.

The score question in the same request — "the risk of listing this model in a corporate catalogue" — separated the two groups cleanly:

Group	Mean risk score	Range
15 hand-excluded models	1.99 / 3	0.44 – 2.95
The other 431 models	0.10 / 3	0.00 – 1.79

The lesson: a typed model does not design the question for you. It is a tool that works when you know which question you are asking. Same context, same model, two questions — one is useless, the other finishes the job.

The Unexpected Gap: Is Confidence Really an Axis?

This is where the two methods actually diverged. We asked both for a confidence value alongside the score. In the typed model confidence is a field the API returns; in the chat model it is a number field we put in the schema. Same name, different behaviour:

Across 60 models	Typed decision model	Chat model (JSON schema)
Distinct confidence values	25	6
Confidence standard deviation	0.171	0.059
Confidence range	0.22 – 0.98	0.80 – 0.99
Distinct risk scores	33	4
Scores rounded to integers	0%	100%

The chat model wrote a confidence number every time, because the schema forced it to — but the number was effectively constant: between 0.80 and 0.99 across all 60 decisions. It never really used the range in which it could say "I am not sure". On the risk score it is even clearer: it produced four distinct values, all integers (0, 1, 2, 3). You cannot distinguish 1.79 from 1.63, because the model never writes those intermediate values.

A concrete example — the very model our sync code has warned about for years:

Model	Typed model	Chat model
nousresearch/hermes-3-llama-3.1-70b	risk 1.48 — confidence 0.46	risk 1.00 — confidence 0.90
nousresearch/hermes-3-llama-3.1-405b	risk 1.26 — confidence 0.22	risk 1.00 — confidence 0.90

The typed model reported "something in this text leaves me undecided" in a measurable way. The chat model wrote 0.90 for exactly the same ambiguity. If your code contains a line like if ($confidence < 0.5) askAHuman();, that difference is a behavioural difference.

Cost and Latency: The Expected Gap, Bigger Than Expected

Measured over the same 60 decisions:

Measurement	Typed decision model	Chat model
Input tokens per request	654	442
Output tokens per request	94 (free)	265
Latency (median)	1.11 s	4.02 s
Latency (p95)	1.21 s	12.28 s
Cost per 1,000 decisions	$0.0275	$1.99

A 72x difference. Note the detail: the typed model actually spends more input tokens (654 against 442), because the question definitions are resent with every request. It still wins, because the chat model spends its money on the output — 265 output tokens per decision, at five times the input price.

The p95 gap matters too: the slowest typed request took 1.21 seconds, the slowest chat request 12.28. If you put a decision gate in a flow where a user is waiting, that tail value decides, not the mean.

Note: the two runs used different concurrency (6 parallel requests for the typed model, 3 for the chat model), so per-request latency is comparable but total wall-clock time is not.

The Chat Model's Silent Trap: an Invisible Thinking Budget

In the first round we gave the chat model max_tokens: 300 — seemingly generous for a five-field, 105-character JSON object. The result: 14 of 60 responses could not be parsed.

The response body showed the real problem:

finish_reason : "stop"
usage.completion_tokens : 285
content : {"adult_rp":0.9,"purpose":"roleplay","purpose

The model spent — and billed — 285 output tokens, but the text that reached us was cut off at 45 characters. Worse, finish_reason reported this as "stop" rather than "length", so the standard check does not catch it.

The cause is invisible reasoning tokens consuming the budget on thinking models. With the budget raised to 1,200, 60 of 60 requests returned clean JSON. To confirm this was not a gateway issue, we sent the same request directly to the model; the behaviour was identical.

Three practical rules for anyone using JSON schemas:

• Size the budget for thinking room, not for the length of the visible output.
• finish_reason is not sufficient evidence on its own; always parse the body and retry on a parse failure.
• Output tokens drive your bill: spending 285 tokens to receive 45 characters loses you both money and a decision.

This class of failure is structurally absent from the typed model: what comes back is a number, not text. Across 446 requests we had zero parse errors.

Accuracy: Both 14/15, But They Miss Different Things

On the controlled 60-model sample, with a threshold of "score ≥ 1.8 and confidence ≥ 0.75":

	Typed decision model	Chat model
Caught (of 15 labelled)	14	14
False alarms (of 45 normal)	0	1
Parse failures	0	0 (after the budget fix)

Both reached the same rate; what is interesting is what they missed.

The typed model missed anthracite-org/magnum-v4-72b (risk 0.44). Reading the description, missing it is correct: "designed to replicate the prose quality of the Claude 3 models, fine-tuned on top of Qwen2.5 72B." There is no exclusion reason in that text; the reason lives in the publisher's positioning. The model only looks at the state it is given — rather than invent what it does not know, it returned a low score.

The model recorded as the chat model's "false alarm" is in fact the place where it was right. That one deserves its own section.

What the Audit Actually Found

At the top of the ranking sat a model that was not on our exclusion list:

undi95/remm-slerp-l2-13b
"A recreation trial of the original MythoMax-L2-B13 but with updated models."

typed model : risk 1.79  confidence 0.67
chat model  : risk 2.00  confidence 0.85

This model was active in our catalogue. It is a recreation of MythoMax — the exact family our exclusion list already keeps out. The rule missed it because the publisher was not listed and the id matched no pattern.

We closed it the same day: the model was deactivated and the publisher was added to EXCLUDED_PUBLISHERS, so the rule now also holds back that publisher's next model.

Here we have to be honest, because this is the real lesson: the hard threshold would not have caught it. A score of 1.79 sits below the 1.80 cutoff. What found it was not the threshold but a human reading a ranked list. The value of a calibrated score is not that it gives you one cutoff — it is that it gives you an inspection order, telling you which 15 of 446 models to look at.

What We Shipped

To keep the measurement from being a one-off, we turned the audit into a weekly job. The design decisions follow directly from the numbers above:

• Two bands. A hard flag (score ≥ 1.8 and confidence ≥ 0.75) and a grey band (score ≥ 1.4). The grey band needs no decision, only a glance — and the single real problem we found sat exactly in that band.
• It never deactivates anything on its own. The script produces a report; a human makes the call. A third-party model should not quietly change our catalogue.
• Models with descriptions shorter than 90 characters are skipped. The measurement showed why: given a 75-character routing alias, the model returned a risk of 1.63 — with no information to go on, it infers from the name. No information, no question.
• Only changed models are queried. A hash of the description is stored, so a weekly run usually covers a handful of models.

The first live run:

catalogue=413  to audit=413  (23 models skipped: short/empty description)
done: queries=413  flagged=0  failed=0  tokens=223,498
      cost=$0.0094  duration=249 s

Zero flags, because the one real problem had been closed before the run. The highest remaining score is the false positive we know well: hermes-3-llama-3.1-70b, risk 1.50 — confidence 0.45. 405 of the 413 models score below 0.50, and none were labelled "roleplay". Auditing the entire catalogue costs less than one cent per week.

Which Job Goes to Which Method

The split we ended up using ourselves:

Situation	Method	Why
Narrow question, high volume, needs a threshold or ranking	Typed decision model	Calibrated probability, intermediate values, two orders of magnitude cheaper per decision
A decision plus text for the user in one step	Chat model	Typed models produce no text; one call beats two
A handful of decisions inside an existing flow	Chat model + JSON schema	Not worth adding a dependency
Measuring uncertainty to hand off to a human	Typed decision model	Confidence is an axis that actually moves
Open-ended generation, summaries, code	Chat model	Where there is no type, a typed model has no job

Building the Same Pattern on Onysoft

Typed decisions are a pattern, not a product. You can build it today with the models in the Onysoft catalogue, since response_format and tools pass through:

curl https://api.onysoft.com/v1/chat/completions \
  -H "Authorization: Bearer sk-ony-YOUR-KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemini-3.8-flash",
    "temperature": 0,
    "max_tokens": 1200,
    "messages": [
      {"role": "system", "content": "Return only a JSON object."},
      {"role": "user", "content": "Model: ... Description: ..."}
    ],
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "classification",
        "strict": true,
        "schema": {
          "type": "object",
          "properties": {
            "risk": {"type": "number"},
            "purpose": {"type": "string", "enum": ["general","coding","reasoning","multimodal","roleplay"]}
          },
          "required": ["risk","purpose"],
          "additionalProperties": false
        }
      }
    }
  }'

Remember the three warnings this measurement produced: be generous with the budget (reasoning tokens are invisible), always parse the body (finish_reason misleads), and do not threshold on a "confidence" field that came from your schema — in our measurement that field stayed between 0.80 and 0.99 across all 60 decisions.

If you genuinely need calibrated probabilities and want to evaluate a typed model inside your own application, you can register your own provider key with Onysoft and use it through the same interface; the contract stays between you and the provider. Details: the typed decision models article.

Summary

446 models, two methods, the same three questions. In numbers:

• Accuracy is identical: 14/15. The difference is not in accuracy.
• Cost differs by 72x ($0.0275 against $1.99 per 1,000 decisions), latency by 4x at the median and 10x at p95.
• The confidence axis: 25 distinct values spanning 0.22-0.98 for the typed model; 6 values spanning 0.80-0.99 for the chat model. Putting a "confidence" field in a schema is not the same as measuring confidence.
• The chat model silently truncated 14 of 60 responses in the first round — while reporting finish_reason: "stop". The typed model had 0 parse errors in 446 requests.
• The audit found a real model sitting active in our catalogue; it was closed the same day and the rule was made permanent.

The lesson that helped us most does not appear in any table: a typed model will not design your question for you. In the same request, the question we asked badly returned 3 of 15; the one we asked well returned 14 of 15. The work is in writing the question.

You can browse the catalogue on /models, or create a free account to try it.