Jev vs OpenAI Decisions API: Accuracy, Cost and Speed on 266 Cantonese and Thai Forum Posts
TL;DR: On the same posts and the same label definitions, TypeSafe’s Jev and OpenAI’s Decisions API are statistically close. Sentiment is a tie (78.0% vs 78.0% on 150 held-out Cantonese posts; 82.8% vs 83.6% on 116 Thai posts). Jev is ahead on Cantonese relevance (92.0% vs 88.7%). Intent splits by language: OpenAI on Cantonese (86.7% vs 85.3%), Jev on Thai (93.1% vs 89.7%). Jev costs about 44% as much per post ($4.49 vs $10.11 per 100,000 posts), latency is similar (median 0.36 s vs 0.39 s), and OpenAI is more overconfident on relevance. Claude Haiku 5.5, a generative model, is still 3 to 5 points ahead of both on most labels. All gaps between Jev and OpenAI are within noise.
OpenAI has added a decisions endpoint that looks a lot like what TypeSafe calls a System One model: you send text and typed questions, you get probabilities back. We had just benchmarked Jev against Claude Haiku, so we ran OpenAI’s version through the same harness. This page is the comparison. For the earlier Jev vs Claude results, see the Jev vs Claude Haiku benchmark; for what Jev is, see the explainer.
What is the OpenAI Decisions API, and how does it map to Jev?
OpenAI documents a POST https://api.openai.com/v1/decisions endpoint. Per the OpenAI decisions guide (read 2026-10-12), only gpt-6-luna is supported, and it is in public beta, with general availability expected “in the coming weeks”. OpenAI does not call it “System One”; that is TypeSafe’s term, so on this page we say “decision endpoint” or “decision model” for both. A third-party news report put the preview at DevDay on 2026-09-29 and the public beta opening to all developers on 2026-10-06; treat those dates as reported.
| Jev (TypeSafe) | OpenAI Decisions | |
|---|---|---|
| Model tested | jev-1.13.0 (pinned) | gpt-6-luna (only model supported) |
| Input | state: a string, object or array | input: a text string, or user messages with input_text / input_image parts |
| Questions | A map of named questions | An array; each has a unique name |
| Yes/no | Noul, returns a yes-probability | predicate, returns probability |
| Multi-class | Choice, returns the choice plus confidence | choice with choices, returns choice, probabilities, confidence |
| Graded | Score, 5 levels in our setup | score with ordered levels, returns a probability-weighted level index from 0, plus probabilities and confidence |
| Images | No, text only | Yes, inline base64 data URLs |
| Input price | $0.042 per 1M tokens, output free | $0.10 per 1M tokens, no output, cache-read or cache-write charges (regional and long-context multipliers may apply) |
| Status | Early access at launch; today’s docs say to log in and get a key | Public beta |
| Rate limits, language support | Rate limit stated in docs; English primary, CJK “handled but not equally well” | Not stated on the page we read |
Both let you ask several independent questions in one request. OpenAI’s answers can also come back as type: "refusal". A difference that mattered in practice: OpenAI’s predicate has no criteria field, so we folded the true/false descriptions into its instructions.
How did we test them fairly?
- Same definitions. The three questions (relevance, sentiment, intent) use the same “v3” wording we used for Jev and Claude Haiku. Our structured state was sent to OpenAI as a JSON string in
input. - Same posts. The 150 held-out LIHKG posts from the main benchmark, plus a new 116-post Pantip set.
- Same threshold search. Thresholds were tuned on the even-numbered LIHKG rows with the same grids for both (sentiment 0.25 to 1.5, relevance 0.20 to 0.80) and reported on the held-out odd 150 rows. The tuned cut-offs differ: Jev sentiment at or below −1.25 is negative and at or above +0.75 is positive, relevance at or above 0.55; OpenAI sentiment at or below −0.25 and at or above +1.0, relevance at or above 0.65. Claude Haiku uses integer ±1 and a boolean.
- No Thai tuning. The LIHKG thresholds were carried over to Thai unchanged.
- One request per post, three questions. For both decision models.
The answer key is the one from the main benchmark: AI-labelled before any model ran, with a native Cantonese reader agreeing with 20 of 20 spot-checked LIHKG labels. The Thai key was not reviewed by a native Thai reader (see the caveats).
Which is more accurate on Cantonese forum posts?
150 held-out LIHKG posts, final definitions:
| Model | Sentiment | Relevance | Intent |
|---|---|---|---|
Jev (jev-1.13.0) | 78.0% | 92.0% | 85.3% |
OpenAI Decisions (gpt-6-luna) | 78.0% | 88.7% | 86.7% |
| Claude Haiku 5.5 (generative LLM, for reference) | 82.7% | 95.3% | 90.0% |
Sentiment is identical. Jev is 3.3 points ahead on relevance (derived), OpenAI 1.4 points ahead on intent (derived). With 150 rows the noise is roughly ±6 points, so none of these gaps separates the two.
Which is more accurate on Thai forum posts?
116 Pantip posts, with the LIHKG thresholds carried over (no tuning on Thai):
| Model | Sentiment | Relevance* | Intent |
|---|---|---|---|
| Jev | 82.8% | 89.7% | 93.1% |
| OpenAI Decisions | 83.6% | 91.4% | 89.7% |
| Claude Haiku 5.5 | 86.2% | 94.0% | 94.8% |
*Thai relevance is not informative. 94% of the Thai posts are relevant, because they come from brand threads, so answering “relevant” every time would score 94.0%. No model beat that. Ignore the relevance column here.
Two baselines make the other columns meaningful: “always neutral” scores 72.4% on Thai sentiment and “always other” scores 66.4% on intent. All three models beat both. Restricted to rows the labeller marked unambiguous (103 for sentiment, 99 for intent), sentiment was 85.4% (Jev), 88.3% (OpenAI) and 86.4% (Haiku), and intent 93.9%, 93.9% and 97.0%. For how the Thai set was built and what it shows, see our Thai sentiment analysis guide.
Does either model give usable confidence scores?
Combining the 300 LIHKG and 116 Thai posts:
| Model | Sentiment confidence ≥ 0.7: share / accuracy | Relevance “sure” (≤ 0.2 or ≥ 0.8): share / accuracy |
|---|---|---|
| Jev | 62% / 90.0% | 82% / 95.9% |
| OpenAI Decisions | 58% / 92.1% | 95% / 90.7% |
Sentiment confidence is usable on both. Relevance is where they differ: OpenAI marks 95% of relevance answers as “sure”, but those are right only 90.7% of the time, while Jev marks 82% as sure and is right 95.9% of the time. OpenAI is the more overconfident of the two on relevance.
What that means for routing: if you plan to auto-accept confident answers and send the rest to a human or a second model, Jev’s relevance “sure” band is a better filter. With OpenAI, a probability near 0 or 1 on relevance still lets through more wrong answers, so you would need a stricter cut-off that you tune yourself. Sentiment confidence works as a routing signal on either.
How do cost and speed compare?
One request per post, three questions, measured on our runs:
| Model | Avg input tokens per post | Cost per post | Per 100,000 posts | Median latency (p90) |
|---|---|---|---|---|
| Jev | 1,070 | $0.0000449 | about $4.49 | 0.36 s (0.41 s) |
| OpenAI Decisions | 1,011 | $0.0001011 | about $10.11 | 0.39 s (0.50 s) |
| Claude Haiku 5.5 (batched 15 per request) | 307 in + 151 out | $0.000106 | about $10.60 | about 15 s per 15-post batch |
Jev costs about 44% as much as OpenAI Decisions per post for the same task ($4.49 vs $10.11 per 100,000). That is derived from vendor list prices and token counts, not from our own pricing. Jev’s price per token is 0.42 times OpenAI’s, and OpenAI used slightly fewer tokens on our prompts, which is why the total ratio is 44% rather than 42% (derived). At 1 million posts a month that is about $44.90 vs $101.10 (derived). Both are small next to the value of the labels. The first OpenAI call in the session took 4.2 s, likely a cold start; the medians cover all calls. For deeper cost modelling see Jev pricing per post.
Which should you use: Jev, OpenAI Decisions or an LLM?
Jev is the better fit when:
- cost per label at high volume matters (about 44% of OpenAI in our setup);
- you want calibrated confidence on relevance, to auto-accept most rows safely;
- you are sending structured state (title, opening post, quoted post, judged text) and want typed answers back.
OpenAI Decisions is the better fit when:
- you need image input, which Jev does not support;
- the rest of your stack is already on OpenAI and one more vendor account is the real cost;
- you accept public-beta terms, a single supported model, and unstated rate limits and language support.
A generative LLM such as Claude Haiku 5.5 is the better fit when:
- you want the highest accuracy and are willing to pay about 2.4 times Jev per post (derived: $10.60 vs $4.49);
- you need text out (explanations, summaries, translation), which neither decision model produces.
On this task the two decision models are interchangeable on accuracy. Pick on price, calibration, and whether you need images. To set Jev up, see how to use Jev for sentiment and relevance.
What are the limits of this comparison?
- AI-labelled answer keys. The LIHKG key was spot-checked 20 of 20 by a native Cantonese reader; the Thai key was not reviewed by a native Thai reader. An AI-written key could favour the model family it came from.
- Small samples. 150 held-out Cantonese rows and 116 Thai rows. Differences of a few points are noise.
- Two forums. LIHKG (Cantonese) and Pantip (Thai), not general text. Thai relevance is uninformative because 94% of the posts are relevant.
- Thresholds. Tuned on LIHKG and carried to Thai. Different cut-offs would change both models.
- Versions and date.
jev-1.13.0,gpt-6-lunaandclaude-haiku-5-5, as of 2026-10-12. OpenAI’s endpoint is in public beta and may change before general availability. Launch dates are from a news report. - Text only. We did not test OpenAI’s image input.
Where this runs in production
Our optional “AI sentiment labels” switch on the LIHKG actor on Apify uses Jev, chosen for cost, speed and calibrated confidence given the small accuracy gaps. It costs $0.002 per labelled row, with the first 20 rows free per run on the free plan, effective 2026-10-27. This article is independent analysis: TypeSafe, OpenAI and Anthropic do not endorse or sponsor it.
From least effort to most freedom
FAQ
Is Jev better than OpenAI’s Decisions API? Not clearly. On the same posts and definitions, sentiment was a tie (78.0% vs 78.0% on 150 Cantonese posts; 82.8% vs 83.6% on 116 Thai posts). Jev was ahead on Cantonese relevance (92.0% vs 88.7%), OpenAI on Cantonese intent (86.7% vs 85.3%) and Jev on Thai intent (93.1% vs 89.7%). All gaps are within noise. The clearer differences are cost (Jev about 44% of OpenAI per post) and relevance calibration.
How much does the OpenAI Decisions API cost? OpenAI’s docs list $0.10 per million input tokens, with no output, cache-read or cache-write charges; regional and long-context multipliers may apply. In our setup (one request per post, three questions) that came to about 1,011 input tokens, $0.0001011 per post, or about $10.11 per 100,000 posts. Jev (jev-1.13.0) cost about $4.49 per 100,000 posts for the same task.
What model does the OpenAI Decisions API use? Only gpt-6-luna is supported, according to the docs we read on 2026-10-12. OpenAI describes the endpoint as in public beta, with general availability expected in the coming weeks.
Does the OpenAI Decisions API support images? Yes. The input can be user messages with input_text and input_image parts, with images sent inline as base64 data URLs. Jev is text only. We tested text only, so our numbers say nothing about image accuracy.
Which is faster, Jev or OpenAI Decisions? They were close. Median latency per post was 0.36 s for Jev (p90 0.41 s) and 0.39 s for OpenAI Decisions (p90 0.50 s). The first OpenAI call in our session took 4.2 s, probably a cold start, and is included in the measurements.
Is Claude more accurate than both? On most labels in our tests, yes. Claude Haiku 5.5, a generative LLM, scored 82.7% sentiment, 95.3% relevance and 90.0% intent on the Cantonese posts, 3 to 5 points above both decision models on most labels. It also costs more per post than Jev and is slower, and it is not a like-for-like comparison with a decision API.
Further reading
- Jev vs Claude Haiku: the 405-post benchmark: the main test behind the Cantonese numbers.
- Thai sentiment analysis with LLMs: the Pantip test in detail.
- Cantonese sentiment analysis with LLMs: what is hard about Cantonese text.
Chad runs 30+ published Apify actors, including the LIHKG scraper whose posts were used here. Numbers are from a run on 2026-10-12; raw posts are not published.