Cantonese Sentiment Analysis with LLMs: What Is Hard and What Accuracy to Expect
TL;DR: On 150 held-out posts from LIHKG (Hong Kong’s biggest forum), Claude Haiku 5.5 reached 82.7% three-way sentiment accuracy and TypeSafe’s Jev (
jev-1.13.0) 78.0%. The hard parts were not vocabulary alone: sarcasm, replies that need thread context, bad news written neutrally, and brands that are only the scene of an incident. Your definitions move the score more than the model does, so settle them before you measure. One forum, small sample, AI-labelled answer key with a 20 of 20 native spot check.
This page is for analysts, researchers and brand teams who want to read the mood of Hong Kong forums at scale. It is a problem-first guide; the model comparison in depth is in the Jev vs Claude Haiku benchmark, and the build is in the how-to guide.
Why is Cantonese forum sentiment hard?
Five patterns showed up in our data, each with a real short example.
| Trap | What happens | Example from our test |
|---|---|---|
| Sarcasm | Positive words, negative meaning | 「食唔盡ai 俾人笑到最後就有份」 was read as positive by an earlier question version; adding the thread’s opening post as context fixed it |
| Literal vs intended tone | Praise phrased like a complaint | 「加價900不跌反升」 praises the iPhone 17 for holding its value but reads like a complaint about a price rise; both models missed it |
| Quoted or thread-dependent replies | A short reply means nothing alone | Sending the quote as separate context raised Jev sentiment accuracy from 62.5% to 70.0% on a Singlish forum with the same quote pattern |
| Bad news written neutrally | Neutral headline, negative event | A headline about a death after eating a McDonald’s burger is negative under our definition |
| Brand as scene | The brand is only where it happened | A Tesla in a crash, a crime in an MTR station, fraud at an HSBC branch |
Mocking in mixed scripts also trips models: a Singlish reply telling someone to hum a PRC song “起来起来” was mocking, but both models called it positive.
How accurate are AI models on LIHKG posts?
We compared Claude Haiku 5.5 and Jev on 300 LIHKG posts (keywords 港鐵, 麥當勞, Tesla, 滙豐; 200 thread titles and 100 replies). Thresholds were tuned on the even-numbered rows and accuracy is reported on the odd-numbered 150 held-out rows.
| Three-way sentiment | Jev | Claude Haiku 5.5 |
|---|---|---|
| All 150 held-out posts | 78.0% | 82.7% |
| Titles (n=100) | 77% | 85% |
| Replies (n=50) | 80% | 78% |
Haiku led by about 5 points overall. With 150 rows the noise is roughly ±6 points, so no single gap is conclusive. Relevance and intent results, calibration and cost are in the benchmark. TypeSafe says Jev’s primary training language is English and CJK is handled but not equally well, which is a useful reminder not to extrapolate to other languages from this one forum.
Which definitions must you decide before measuring?
In our data, definitions drove scores more than models did. Under an earlier strict relevance definition (a brand that is merely the scene of an incident is not relevant), Jev beat Haiku on relevance, 88.0% versus 80.7%. Under the final definition where the scene counts, Haiku was ahead. Settle at least these:
- Is bad news negative even when written neutrally? We said yes.
- Does a brand that is only the scene of an incident count as relevant? We said yes.
- Who or what is a complaint about? Criticism of politicians or the government is not a product complaint. Adding that sentence lifted Jev intent accuracy from 82.7% to 85.3%.
- What is neutral? Facts, questions, and posts with no feeling.
Write these into the model’s instructions. Jev reads literally, so unwritten assumptions turn into errors.
How should you set it up in practice?
- Use structured fields:
keyword,thread_title,opening_post(up to 600 characters as context),quoted_post(context) andpost(the judged text). - Use a 5-level score for sentiment and set your own neutral band. Our tuned thresholds were at most −1.25 for negative and at least +0.75 for positive; a naive ±0.5 cut-off gave 64.0% versus 78.7% tuned on an earlier question version.
- Filter by confidence. Jev was confident on 60% of sentiment rows at 92.2% accuracy; auto-accept those and review the rest.
- Label a sample yourself and keep a held-out half.
The full request and 7-step recipe are in how to use Jev. For what each setup costs, see Jev pricing; for background on the model, what is Jev.
How do you get LIHKG data?
Our LIHKG actor on Apify collects public threads and replies. It also has an optional “AI sentiment labels” switch that adds the relevance, sentiment and intent labels described here, using Jev, at $0.002 per labelled row, with the first 20 rows free per run on the free plan, effective 2026-10-27. To monitor topics over time, see the LIHKG crisis timeline guide and the early-warning keyword scan. More sources are in the Hong Kong forums hub.
What are the limits of these numbers?
- The answer key was written by an AI labeller (Claude) before any model was called; a native reader spot-checked 20 labels, including the 10 where both models disagreed with the key, and agreed with all 20. That reduces but does not remove bias towards the Claude family.
- 150 held-out rows, one forum, four brand keywords. Not a general claim about Cantonese or about Hong Kong forums beyond LIHKG.
- Versions are
jev-1.13.0andclaude-haiku-5-5as of 2026-10-12. - This is independent analysis; neither TypeSafe nor Anthropic endorses or sponsors it.
From least effort to most freedom
FAQ
How accurate is LLM sentiment analysis on Cantonese forum posts? On 150 held-out LIHKG posts with our final definitions, Claude Haiku 5.5 scored 82.7% on three-way sentiment and Jev (jev-1.13.0) 78.0%. The answer key was AI-labelled with a 20 of 20 native spot check, the sample is small (roughly 6 points of noise), and it covers one forum, so treat it as a guide, not a guarantee.
Why is Cantonese sentiment analysis hard? In our data the main traps were sarcasm, literal readings that miss the intended tone, replies that depend on the thread or a quoted message, news that is neutral in wording but negative in meaning, and brands that appear only as the scene of an incident.
Is LIHKG sentiment easier on titles or replies? Titles in our test. Claude Haiku scored 85% sentiment on titles and 78% on replies; Jev scored 77% and 80%. Fifty replies is a very small sample, so do not read the Jev reply figure as a win.
How do I get LIHKG data for sentiment analysis? Our LIHKG actor on Apify collects public thread and reply data. It has an optional AI sentiment labels switch at $0.002 per labelled row, with the first 20 rows free per run on the free plan, effective 2026-10-27.
Do I need to define what counts as negative before measuring? Yes. In our data, changing one relevance definition swapped the ranking of the two models (Jev 88.0% versus Haiku 80.7% under the strict definition, with Haiku ahead under the final one). Decide your definitions first.
Further reading
- Jev vs Claude Haiku: independent benchmark
- How to use Jev for sentiment and relevance
- What is Jev? The System One model explained
- Jev pricing: cost per post
- Hong Kong forums topic hub
Chad runs 30+ published Apify actors, including the LIHKG scraper whose posts were used here. Numbers are from a run on 2026-10-12; raw posts are not published.