Jev vs Claude Haiku: An Independent Sentiment Benchmark on 405 Cantonese, Thai and Singlish Forum Posts

TL;DR: On 150 held-out Cantonese forum posts, Claude Haiku 5.5 was ahead of Jev (jev-1.13.0) by 3 to 5 points: sentiment 82.7% vs 78.0%, relevance 95.3% vs 92.0%, intent 90.0% vs 85.3%. Jev costs about $4.50 per 100,000 posts against about $10.60 for Haiku, answers in a median 0.36 s, and its confidence is well calibrated: when it is confident on sentiment (60% of posts) it is right 92.2% of the time. A hybrid that uses Jev when confident and Haiku otherwise matched or beat Haiku on all three labels. With 150 rows the noise is roughly 6 points, so treat Haiku’s lead as likely, not proven.

Update (2026-10-12): we ran OpenAI’s new Decisions API (gpt-6-luna) on the same posts with the same definitions. It tied Jev on sentiment (78.0% vs 78.0% on Cantonese), trailed it on relevance (88.7% vs 92.0%) and cost about 2.25× more per post. Full three-way results: Jev vs OpenAI Decisions API.

TypeSafe publishes its own benchmarks for Jev, and there are explainers, but we could not find an independent, multilingual comparison on real forum text. This is ours: same questions, same label definitions, same posts, two models. For the hands-on setup, see the practical guide to using Jev.

What did we test, and when?

Both models received the same label definitions, for three labels used in social listening:

1. Relevance: does the post concern the tracked brand or keyword?
2. Sentiment: negative, neutral or positive (a −2 to +2 scale underneath).
3. Intent: complaint, praise, question or other.

Newer model versions may give different numbers.

Which posts did we use?

All posts are real, public forum posts collected with our own Apify actors. We do not publish the raw posts (forum terms), only short quotes as examples.

How was the answer key made, and why does it matter?

Accuracy is only as meaningful as the answer key. Ours was built in three steps.

• Blind AI labelling. An AI labeller (Claude) wrote the gold labels before any model under test was called on those posts, so the key was not anchored on model outputs.
• Native spot check. A native reader (the site owner) checked 20 Round B labels, including the 10 where both models disagreed with the key, and agreed with all 20.
• Two explicit definitions. The owner set these for the final round: (1) a brand that is only the scene of an incident (a Tesla in a crash, a crime inside an MTR station, fraud at an HSBC branch) counts as relevant; (2) reports of bad events count as negative even when written neutrally, so a headline about a death after eating a McDonald’s burger is negative, and good news is positive.

The caveat is real: an AI-written key could favour the model family it came from, and Haiku is a Claude model. The 20/20 spot check reduces that risk but does not remove it.

How did we keep the thresholds fair?

Jev returns scores and probabilities, and you choose where to cut them. Tuning cut-offs on the same rows you report would flatter Jev. In Round B we tuned thresholds on the even-numbered rows only and report accuracy on the odd-numbered 150 rows, which the tuning never saw. The final cut-offs: sentiment at or below −1.25 is negative, at or above +0.75 is positive, and relevance yes-probability at or above 0.55 is relevant.

Is Jev or Claude Haiku more accurate? Main results

Round B, 150 held-out LIHKG posts, final definitions (“v3” questions). The hybrid column uses Jev when it is confident and Haiku otherwise; the percentage is the share of rows Jev handled.

Restricting to rows the labeller marked unambiguous narrows the gap slightly:

Haiku is ahead by 3 to 5 points on every label. The direction is consistent, but with 150 rows the noise is roughly ±6 points, so no single gap is conclusive.

Does it matter if the post is a title or a reply?

Yes. Replies are harder for both models, and Jev’s sentiment is actually ahead on replies.

Fifty replies is a very small sample. Do not read the reply sentiment result as Jev winning.

Can you trust Jev’s confidence scores?

This is where Jev earns its place. Over all 300 Round B rows with the final questions:

With the earlier “v2” questions on the same 300 posts, relevance “sure” answers covered 62% of posts at 93.5%, and the rest were close to a coin flip (54%). Better definitions improved the calibration, not just the accuracy. In practice you can auto-accept confident answers and route the rest to a human or a second model, which is what the hybrid column does.

Why do thresholds matter so much for Jev sentiment?

On the same 300 LIHKG posts with the earlier v2 questions, mapping Jev’s score at a naive ±0.5 gave 64.0% sentiment accuracy. Tuned thresholds gave 78.7% on held-out rows. Jev shades neutral text slightly negative, so the negative threshold has to be stricter than the positive one. If you benchmark Jev with default cut-offs you will badly underrate it.

How did Jev do on Cantonese, Thai and Singlish?

Round A used the earlier v2 questions and earlier definitions, with 105 posts. “Strict” counts “slightly negative” as negative; “lenient” accepts either neutral or the polar class for “slightly” answers. Treat this as directional, not a per-language ranking.

In the pilot, Jev did better than Haiku on Cantonese and Haiku did better on Thai. That sits oddly with TypeSafe’s note that English is Jev’s primary training language and CJK is “handled but not equally well”, and it is a reminder that small samples and definitions move results. Note also that Haiku led on Cantonese in the larger, later Round B.

Do definitions matter more than the model?

In our data, yes. Under the earlier strict relevance definition (a brand that is merely the scene of an incident is not relevant), Jev beat Haiku on relevance, 88.0% versus 80.7%. Under the final definition where the scene counts, Haiku is ahead. The same pair of models swapped places because we changed one sentence. Decide your definitions before you compare models, and write them into the prompt or criteria.

Jev also reads definitions literally. With the earlier intent definition, it labelled criticism of a government minister as “complaint” (a product complaint). Adding “criticism of individual people, politicians or the government is other” lifted Jev’s intent accuracy from 82.7% to 85.3% (titles 87% to 91%).

What do sarcasm and literal readings look like?

Context helps. Adding the thread’s opening post fixed one sarcastic Cantonese reply (「食唔盡ai 俾人笑到最後就有份」), which the first question version read as positive and the second read correctly as negative.

Both models still miss some:

• 「加價900不跌反升」 praises the iPhone 17 for holding its value, but reads like a complaint about a price rise.
• A Singlish reply telling someone to hum a PRC song “起来起来” is mocking, but both models called it positive.

Two more measured behaviours: Jev’s docs warn that it can favour the first Choice option, but with the sentiment options reversed 98.1% of answers were unchanged (105 posts). And on HardwareZone, where replies embed “X said: … Click to expand…”, sending the quote as a separate context field raised Jev sentiment accuracy from 62.5% to 70.0%. The practical guide covers both.

How do cost and speed compare?

Both costs are tiny next to what the labels are worth. Speed matters for real-time agents; for batch scraping it rarely does. TypeSafe’s own materials make larger speed claims; ours are measured on this specific task.

Should you use Jev or Claude Haiku?

Jev is the better fit when:

• cost per label at very high volume matters;
• latency matters (an agent deciding in real time, a stream filter);
• you want calibrated confidence so that you can auto-accept most rows and review the rest;
• you want typed outputs and no JSON parsing or text generation.

Haiku is the better fit when:

• you want the highest accuracy on mild tone and intent, where it led by 3 to 5 points in our test;
• the language is Thai (small pilot, directional);
• you are labelling a few thousand rows once and cost is irrelevant.

Use both when you want near-Haiku accuracy at lower cost: let Jev answer the confident rows (51% of sentiment, 83% of relevance, 77% of intent in our test) and send the rest to Haiku. That matched Haiku on sentiment and relevance and was 1.3 points higher on intent.

What are the limits of this benchmark?

• The answer key was AI-labelled. A native reader spot-checked 20 of 20 labels as correct, which reduces but does not eliminate bias towards the Claude family.
• Samples are small: 150 held-out rows in the main test, 105 in the multilingual pilot.
• The main test covers one forum (LIHKG). The multilingual results are a pilot with earlier definitions.
• Versions are jev-1.13.0 and claude-haiku-5-5 as of 2026-10-12.
• Haiku was called in batches of 15 posts, Jev one post per request. A different prompt design could shift either result.
• Vendor claims elsewhere (such as “193.6x faster”) come from TypeSafe’s own benchmark. Ours is independent and task-specific.

Where this runs in production

We use the same three labels as an optional “AI sentiment labels” switch on our LIHKG actor on Apify. The engine is Jev, chosen for cost, speed, calibrated confidence and no extra vendor account, given the small accuracy gap. It costs $0.002 per labelled row, with the first 20 rows free per run on the free plan, effective 2026-10-27. This article is independent analysis: neither TypeSafe nor Anthropic endorses or sponsors it. For the forum data itself, see our Hong Kong forums hub.

From least effort to most freedom

I run it for you Managed, daily to your sheet You do nothing. From US$299/mo. Use it here Run the free tool Instant, no setup, on this site. Run my actor On Apify, your account Same data, you run it and pay per result. Build your own Sign up for Apify Free account — build the pipeline yourself.

This post contains Apify affiliate links (no extra cost to you). Costs and comparisons are stated honestly, including where Apify is not the best fit.

FAQ

Is Jev better than Claude? Not on raw accuracy in our test. On 150 held-out Cantonese forum posts, Claude Haiku 5.5 scored 82.7% on sentiment, 95.3% on relevance and 90.0% on intent, against 78.0%, 92.0% and 85.3% for Jev. Jev was cheaper (about $4.50 versus about $10.60 per 100,000 posts), faster per post (median 0.36 s) and returned calibrated confidence. With 150 rows the noise is roughly 6 points, so no single gap is conclusive.

How much does Jev cost per post compared with Claude Haiku? In our setup, Jev (jev-1.13.0) cost about $0.000045 per post (1,063 input tokens for three questions, output tokens free) and Haiku 5.5 about $0.000106 per post (307 input plus 151 output tokens, batched 15 per request). That is roughly $4.50 versus $10.60 per 100,000 posts at list prices.

Does Jev work in Chinese or Cantonese? It worked usefully in our tests, but it is not tuned for it. TypeSafe’s docs say English is the primary training language and that CJK is handled but not equally well. In our pilot Jev beat Haiku on Cantonese sentiment and relevance, while on the 300-post main test Haiku was ahead by 3 to 5 points.

Does Jev work for Thai? In our 25-post Pantip pilot, Haiku did better than Jev on Thai (sentiment 88.0% strict versus 68.0%, relevance 100% versus 96.0%). The sample is small and used earlier definitions, so treat it as directional.

Can Jev write text like an LLM? No. Jev is a System One decision model: it returns typed answers with probabilities and does not generate text. That is why it is a better fit for classification-style labels than for summaries or explanations.

What is a System One model? TypeSafe uses the term for models like Jev that turn natural language and application state into typed judgments and probabilities that code can combine, instead of producing free-form text.

Further reading

• How to use Jev for sentiment and relevance: request shape, 7-step recipe, v3 questions: the setup behind every number above, with copyable question definitions.
• Build a no-code forum sentiment pipeline with Apify + n8n + GPT: the scheduled alternative if you would rather wire the labels into a workflow.
• Hong Kong forums topic hub: LIHKG and HKGolden data guides.

Chad runs 30+ published Apify actors, including the LIHKG scraper whose posts were used here. Numbers are from a run on 2026-10-12; raw posts are not published.