Thai Sentiment Analysis with LLMs: How Accurate Are Three Models on 116 Pantip Posts?
TL;DR: On 116 real Pantip posts about six Thai brands, all three models we tested beat the trivial baselines on the two labels that are informative. Sentiment: Jev 82.8%, OpenAI Decisions 83.6%, Claude Haiku 5.5 86.2%, against 72.4% for always answering “neutral”. Intent: 93.1%, 89.7% and 94.8%, against 66.4% for always answering “other”. Haiku was best on intent, Jev was cheapest (about $4.49 vs $10.11 for OpenAI and $10.60 for Haiku per 100,000 posts), and Thai relevance scores are meaningless because 94% of the posts were relevant. The sample is small and the key was not reviewed by a native Thai reader.
If you are choosing a model to label Thai forum text for a brand-monitoring or insights project, published accuracy numbers for Thai are thin. This is a small, honest measurement on one forum, with the baselines you need to read it. For the practical side of getting Pantip data in the first place, see Social Listening Thailand: Search Pantip in Thai Too; this page is about labelling accuracy.
What makes Thai forum sentiment tricky?
We will not claim linguistic reasons we did not measure. What we saw in the data was more mundane, and it shaped how we built the test:
- Customer-support chasing threads. Many comments are customers asking a brand to fix an order, a package or a bill. They are complaints in intent, even when the wording is polite.
- Repeated duplicate complaints. One member posted the same complaint 11 times in an AIS thread. Left in, it would count the same sentence 11 times.
- Bare-brand titles. 22 search-result titles were only the brand name, such as “Shopee”. They are trivially relevant and neutral, and would inflate every score.
- Official brand accounts replying. Customer-service accounts answer inside threads, so some “comments” are the brand’s own voice.
How did we build the 116-post set?
We searched Pantip for six brand keywords with our own scraper: AIS, Shopee, Grab, เซเว่น (7-Eleven), iPhone and ทรู (True). That gave 60 search-result titles plus 11 topics (opening posts and comments), 165 raw posts.
We removed 49 before labelling:
| Removed | Count | Why |
|---|---|---|
| Titles that are only the brand name | 22 | Trivially relevant and neutral |
| Trivial or empty posts | 8 | Nothing to judge |
| Duplicates | 19 | One member posted the same complaint 11 times in an AIS thread |
The final set is 116 posts: 36 titles, 9 opening posts and 71 comments. They were labelled blind by Claude (before any model under test ran), using the same final definitions as our Cantonese benchmark. A native Thai reader has not reviewed this key.
How did the three models do on Thai?
Models: Jev (jev-1.13.0), OpenAI’s Decisions API (gpt-6-luna) and Claude Haiku 5.5, with the same label definitions. The thresholds for the two decision models were carried over from LIHKG, with no tuning on Thai.
| Model | Sentiment | Relevance* | Intent |
|---|---|---|---|
| Jev | 82.8% | 89.7% | 93.1% |
| OpenAI Decisions | 83.6% | 91.4% | 89.7% |
| Claude Haiku 5.5 | 86.2% | 94.0% | 94.8% |
*Not informative. 94% of the Thai posts are relevant, so always answering “relevant” scores 94.0%. No model beat that, so this column tells you nothing about relevance detection. Our brand-thread sample simply has almost no irrelevant posts.
What are the baselines?
| Baseline | Sentiment | Intent |
|---|---|---|
| Always “neutral” / always “other” | 72.4% | 66.4% |
All three models beat both baselines. On sentiment the margin is 10 to 14 points (derived), on intent 23 to 28 points (derived).
What if you keep only the clear-cut rows?
Restricting to rows the labeller marked unambiguous:
| Label | Jev | OpenAI | Haiku | n |
|---|---|---|---|---|
| Sentiment | 85.4% | 88.3% | 86.4% | 103 |
| Intent | 93.9% | 93.9% | 97.0% | 99 |
On sentiment the ordering changes (OpenAI is first here), which is a good sign that the differences between the three models are within noise.
What do the numbers mean in practice?
- Any of the three is usable for coarse Thai sentiment and intent triage; all clear the baselines by a wide margin.
- Haiku is best on intent (94.8%) and leads sentiment on the full set, but only by a few points on 116 posts.
- Jev is the cheapest. Our measured cost per 100,000 posts, one request per post and three questions: Jev about $4.49, OpenAI Decisions about $10.11, Haiku about $10.60 (batched 15 per request). Details in Jev pricing per post and Jev vs OpenAI Decisions.
- Use confidence to route. The two decision models return probabilities. Auto-accept confident answers and send the rest to review or a second model. In a combined set of 300 Cantonese and 116 Thai posts, Jev’s sentiment answers at confidence 0.7 or above covered 62% of posts at 90.0% accuracy; OpenAI’s covered 58% at 92.1%. Jev’s relevance “sure” answers covered 82% of posts at 95.9% accuracy, against 95% at 90.7% for OpenAI, so OpenAI’s relevance confidence is the less reliable gate.
- Check your own mix. If your data is not brand-thread posts, relevance becomes a real task and these relevance numbers do not apply.
For how Thai compares with Cantonese, which was the main benchmark, see Cantonese sentiment analysis with LLMs and the Jev vs Claude Haiku benchmark. In the earlier 25-post Pantip pilot (older definitions), Haiku also led Jev on strict sentiment, 88.0% vs 68.0%; that pilot is superseded by this 116-post test.
What do the posts look like?
Two short examples with all three models in agreement:
- 「เติมเน็ตแพ็คเกจเสริมตัดเงินธนาคารไป ไม่ได้รับเน็ต … 😡」: “paid for an add-on data package via the bank, never got the data.” All three models: negative.
- 「Grab ส่งไว」: “Grab delivers fast.” A praise title.
The first is typical of the support-chasing pattern: a complaint that is really a request for help.
What did we notice for social listening?
Official brand customer-service accounts reply directly inside Pantip threads. In our sample we saw AIS agents posting from a call-centre address, 7-Eleven’s mascot account “น้องเปา” (Nong Pao), True, dtac and CardX. Many comments are customers chasing support.
For a brand team, that points to a use case beyond mention counting: track complaints and check whether, and how fast, an official account replied. Sentiment and intent labels give you the complaint filter; the reply is a second row in the same thread. We did not measure response times in this test.
How do you get Pantip data to analyse?
Our pantip-scraper-en actor on Apify returns Pantip topics and comments, including nested replies. Pulling the rows is covered in the Pantip social listening guide, including why you should search both Latin and Thai spellings of a brand. Regional context is in the Southeast Asia forums hub.
The AI sentiment labels are not a Pantip feature. They launch on our LIHKG actor only, with pricing effective 2026-10-27. For Pantip, run the model of your choice on the rows you export.
From least effort to most freedom
What are the limits of this test?
- The Thai key was labelled by an AI and not reviewed by a native Thai reader. (The Cantonese key was spot-checked 20 of 20 by a native reader.) An AI-written key could favour Claude.
- Small sample. 116 posts; a gap of a few points is noise.
- One forum, six brands. Pantip brand threads only, not all Thai text such as Facebook or X.
- Relevance is uninformative because 94% of the posts are relevant.
- Thresholds were not tuned on Thai. They were carried over from LIHKG.
- Versions and date.
jev-1.13.0,gpt-6-luna,claude-haiku-5-5, as of 2026-10-12. OpenAI’s endpoint is in public beta. - The earlier 25-post pilot is superseded and used older definitions.
- Raw posts are not published (forum terms); only short quotes appear.
This article is independent analysis. TypeSafe, OpenAI and Anthropic do not endorse or sponsor it.
FAQ
How accurate is LLM sentiment analysis on Thai forum text? On 116 Pantip posts, sentiment accuracy was 82.8% for Jev, 83.6% for OpenAI’s Decisions API and 86.2% for Claude Haiku 5.5, against 72.4% for always answering neutral. Intent accuracy was 93.1%, 89.7% and 94.8%, against 66.4% for always answering other. Sample size is small and the answer key was not reviewed by a native Thai reader, so treat these as indicative.
Which model is best for Thai sentiment analysis? Claude Haiku 5.5 scored highest on sentiment (86.2%) and intent (94.8%) in our test, but the gaps to Jev and OpenAI Decisions were a few points on 116 posts, within noise. Jev was the cheapest at about $4.49 per 100,000 posts, against about $10.11 for OpenAI Decisions and about $10.60 for Haiku. The right choice depends on whether you value the last few points of accuracy or the cost.
Can I trust the relevance scores in your Thai test? No. 94% of the Thai posts were relevant because they came from brand threads, so answering relevant every time scores 94.0%. The models scored 89.7% to 94.0%, which means none of them beat that trivial baseline. Use the sentiment and intent columns instead.
Can I get Pantip posts with sentiment labels from your actor? Not today. Our pantip-scraper-en actor returns the posts and comments. The optional AI sentiment labels launch on our LIHKG actor only, with pricing effective 2026-10-27. You can run the same models on Pantip rows yourself.
Do you have to tune thresholds for Thai? We did not. We carried over the thresholds tuned on Cantonese LIHKG posts and still beat the baselines with all three models. Tuning on a labelled Thai sample would likely help the two decision models, but we have not measured that.
Is Pantip good for monitoring customer support? It looked that way in our sample. Official brand accounts, including AIS agents, 7-Eleven’s mascot account Nong Pao, True, dtac and CardX, reply directly inside threads, and many comments are customers chasing support. That makes complaint and support-response tracking a natural use case.
Chad runs 30+ published Apify actors, including the pantip-scraper-en whose posts were used here. Numbers are from a run on 2026-10-12; raw posts are not published.