Using an LLM to Spot Coordinated Word-of-Mouth Posts: What a Keyword Dictionary Misses, and How to Fill the Gap
In one line: Paid word-of-mouth posting is a real industry. A keyword dictionary can tell you whether a single post reads positive or negative, but it can’t tell you whether a batch of posts, taken together, looks too coordinated — that takes an LLM reading the timing, wording and cross-account pattern across the batch, and even then the result is a signal, not proof.
Coordinated word-of-mouth posting isn’t a hypothetical
Search “口碑” (“word-of-mouth”) on Threads and you land directly on recruiting posts. One from @wawajump_jump reads: “Looking for long-term collaborators ✨ word-of-mouth reply posting ✨ mainly social media posting-on-behalf and word-of-mouth comments. Platforms: Threads, Dcard, FB, Bahamut, Mobile01, FashionGuide, etc.” — the same operator running “word-of-mouth commenting” across six communities at once (Threads, Dcard, Facebook, Bahamut, Mobile01 and FashionGuide), not a one-platform, one-off case.
People who’ve worked in marketing and PR are also sharing, from the reader’s side, how they learn to spot these posts. One post from @th6li9 describes the “occupational habit” the job leaves you with: “1. Quickly spot word-of-mouth posts. 2. Don’t trust KOLs — plenty of them have blown up in scandal anyway.” In other words, people inside the industry know how to spot this, but that judgment isn’t something every social-listening operator has, and it’s hard to replicate by having a human read every single post.
If you’re scraping forum volume to run sentiment analysis, this directly affects your conclusions: a batch of arranged positive comments inflates “volume” and pushes “sentiment” up, making you misread the mood; a concentrated burst of negative reviews could just as easily be a competitor’s operation rather than genuine user dissatisfaction. Either way, you need another layer of judgment beyond a sentiment score.
What a keyword dictionary can and can’t do
Our own asia-social-listening actor’s built-in sentiment analysis uses a lightweight, rule-based keyword dictionary (about ten positive/negative words each for Traditional Chinese and English), scoring (positive hits − negative hits) / (positive + negative + 1) and thresholding at ±0.15 into positive/neutral/negative — that’s documented plainly on its Apify Store listing, and it is not an AI/LLM model.
That dictionary approach has its place: it’s cheap, instant, and good enough for a rough read on general sentiment. But it has two structural blind spots:
- It can’t read sarcasm. A sarcastic or mocking post can be full of “positive” words on the surface while meaning the opposite. A dictionary only counts words; it has no notion of context.
- It can’t see that a batch of posts looks too coordinated. The dictionary scores each post independently — it never compares wording, structure or tone across posts from different accounts posted around the same time, which is exactly where arranged word-of-mouth campaigns tend to show their seams.
Closing those two gaps means changing the unit of analysis: instead of asking “is this one sentence positive or negative,” you feed a batch of posts together and have an LLM read the pattern across them.
What an LLM can see that a dictionary can’t — and what it still can’t see
Academic work on detecting “coordinated” or astroturfing campaigns typically groups signals into three buckets: timing, content, and network structure. A paper on LLM-based coordinated-campaign detection (arXiv:2501.11849) points to a timing signal — “In coordinated campaigns, retweets often occur in rapid bursts, indicating synchronized activity” — alongside content-level signals like consistent tone and text similarity.
The interesting part is that even that paper can’t get the third bucket, network structure. The same paper states it plainly: “Since 𝕏 does not provide information about the original re-tweeting actions, we predict the propagation trees” — the platform doesn’t expose the actual retweet relationships, so the authors reconstruct a likely propagation tree from mentions, follower relationships and weighted probabilities, rather than working from a real interaction graph.
That’s actually good news for anyone doing this outside a research lab: nobody has a clean relationship graph for this problem — everyone is piecing it together from partial signals. What asia-social-listening gives you is a deduplicated mention list (mention_id / author / text / created_at / platform / sentiment / is_duplicate / duplicate_of, among other fields), with no follower graph or posting history — so we don’t even attempt the “reconstruct a propagation tree” step. What an LLM can add on top is the two signal types that don’t require a relationship graph: text content and timing distribution. The difference isn’t that academia has magic data we lack; it’s that they spend an extra layer of inference approximating the structural signal, and we choose not to guess at it.
Concretely, here’s what each method can and can’t catch:
| Signal | Keyword dictionary | LLM reading a batch |
|---|---|---|
| Positive/negative wording in a single post | ✅ | ✅ |
| Sarcasm (positive wording, mocking intent) | ❌ | Partially (reads context) |
| Exact duplicate posts | ❌ (but the actor’s is_duplicate text-hash check catches this) | ✅ |
| Same script, reworded across different accounts | ❌ | Partially (reads cross-post structural similarity) |
| Same brand term surging across accounts/platforms in a short window | ❌ | ✅ (created_at + platform are comparable) |
| Account history, follower graph, brand-new accounts | ❌ | ❌ (the actor has none of these fields — needs separate investigation) |
How to design the classifier
Once you’ve pulled a batch of mentions with asia-social-listening, the actual flow looks like this:
Step 1 — Let the built-in dedup do what it’s good at. With deduplication: true, exact duplicates and cross-platform reposts of the same text get flagged is_duplicate: true. That’s the cheapest “copy-paste” signal available, and it costs nothing to compute — don’t waste an LLM call on it.
Step 2 — Bucket the remaining, non-duplicate mentions by time. For example, group everything from the same day and the same brand_matched into time windows based on created_at, so the LLM reads “the batch of posts inside this window” rather than one post at a time.
Step 3 — Use a structured prompt that has the LLM grade a signal, not deliver a verdict:
// Example prompt (illustrative — wire it into whatever LLM API you use)
{
"system": "You will receive multiple social posts from the same time window and the same brand keyword (with author, platform, timestamp and text). Your task is to flag which posts show a coordination signal worth human review — not to determine whether they are genuine or fake. For each post, output coordination_signal (low/medium/high) and reasons (an array listing only the specific signals you observed, e.g. wording/structure closely matches another post, or many different accounts used near-identical phrasing to praise the same selling point in a short window). If there isn't enough signal, mark it low — do not speculate about an account's identity or motive.",
"input": "<the mention list JSON for this time window>"
}Use the returned coordination_signal to rank posts and pull the highest-scoring batch for human review — not to auto-remove content or publicly name anyone. That distinction matters; see the FAQ below.
The two approaches, side by side
Keyword dictionary (sentimentAnalysis) | LLM batch classification | |
|---|---|---|
| Cost | Built in, free (bundled into mention billing) | Extra LLM API cost (depends on volume and model) |
| Speed | Instant | Needs an extra API call (can be batched to save cost) |
| Catches sarcasm | No | Partially |
| Catches the same script across accounts | No | Partially (reads text similarity) |
| Catches account history / network structure | No | No (needs a separate data source) |
| Output certainty | High (fixed rules, reproducible) | Probabilistic signal, needs human review |
| Best for | A quick read on general sentiment | Screening suspicious volume when you have review capacity |
Three ways to start
From least effort to most freedom
FAQ
What can’t a keyword dictionary catch? It can’t catch sarcasm, and it can’t catch the same script rewritten in different words across different accounts — it only counts positive/negative word hits inside a single post, and never compares whether a batch of posts looks suspiciously coordinated.
Can an LLM classifier catch paid word-of-mouth posts with certainty? No — it’s a probabilistic signal, not proof. Even academic work on coordinated-campaign detection stresses combining timing, content similarity and other signals rather than judging from a single post (see arXiv:2501.11849). Treat it as a ranking tool that surfaces posts for human review, not something you use to publicly accuse a specific business or person.
Isn’t that what asia-social-listening’s built-in sentiment analysis already does? No. Its sentiment analysis is a lightweight rule-based keyword dictionary (about ten positive and ten negative words each for Traditional Chinese and English), not an AI model — it only returns positive/neutral/negative and has no notion of whether a post was arranged. To do the kind of classification in this article, you need to wire in an LLM separately.
Does this catch coordinated word-of-mouth activity on Threads or Dcard? asia-social-listening v1 monitors LIHKG, PTT, HardwareZone and Telegram — it does not cover Threads, Dcard, Facebook, Bahamut or Mobile01. To cover bahamut-scraper or mobile01-scraper volume, run those actors alongside it.
Is this legal? We only scrape public content, never login-walled data, and we follow each site’s terms of service and local law.
Further reading
- Build a no-code forum sentiment pipeline with Apify + n8n + GPT: if you want the sentiment side of this automated end-to-end with Slack alerts, this pipeline wires the same kind of LLM scoring into a scheduled run.
- Singapore’s EDMW turns over every 6 minutes: a look at the coverage side of the same problem — which boards and platforms a listening setup actually reaches, and which ones it structurally misses.
Chad runs 40+ published Apify actors, including asia-social-listening used here. Cost and field figures come from pipelines that actually run; academic research and competitor pricing are linked to their sources, and unverified claims are left out.