LIHKG Scraper Has No Official Docs: Why Thread List, Search and Full-Thread Are 3 Different Builds, Not 1
In one line: LIHKG has no official API, so before you write a line of a lihkg scraper, you need to know that its three read modes — board thread list, site-wide search, and full-thread-with-replies — have different inputs, different output shapes and different pagination rules. One pagination loop can’t drive all three endpoints; decide whether that’s worth building yourself or worth paying a maintained actor to have already solved.
What you’re actually up against: no docs, an endpoint nobody officially owns
LIHKG (連登), one of Hong Kong’s largest discussion forums, has never published formal developer API documentation. Anyone who wants its data is reading the same undocumented api_v2 JSON endpoint that LIHKG’s own app uses — reverse-engineered by the community, not handed over by LIHKG. A GitHub issue tracking how to compute the endpoint’s x-li-digest request signature shows contributors working out the hash formula by trial and error, and the calculation method has already changed once mid-discussion. Nothing here comes with an uptime or stability promise.
Right now, reading the thread-list, search and full-thread endpoints doesn’t require a login or an auth header — but that’s an observed current state, not a documented guarantee. LIHKG could add a header requirement (something like X-LI-DEVICE) or start blocking datacenter egress IPs at any time. If you’re building a lihkg scraper, the first decision isn’t “how do I parse the JSON” — it’s “which of three genuinely different data shapes am I actually after,” because the pagination logic for one does not transfer to the other two.
Three modes, three different shapes
| Thread list | Keyword search | Full thread | |
|---|---|---|---|
| Input | Category IDs (an all value expands to every postable board) | Keywords, sent to LIHKG’s site-wide search | Thread IDs or thread URLs |
| Output | Topic list; content is null — titles and metadata only | Search results, same shape as thread list but not limited to one board | Full original post + every reply, replies numbered by floor |
| Sort / filter | Optional title filter by keyword (doesn’t widen scope) | Sort by latest-reply, latest-post, or relevance | Optional reply cap per thread |
| Cost shape | Billed per item | Billed per item | Original post billed per thread; replies billed per reply — can run into the thousands on a popular thread |
| Use case | Scan what boards have posted lately | Monitor a brand or product keyword across the whole site | Pull a complete conversation on one specific thread |
The part most likely to trip up a DIY build: keywords means two different things in two different modes. In search mode, it’s sent to LIHKG’s real site-wide search and returns results from anywhere on the site. In thread-list mode, the same field name only filters titles within the board list you already fetched — it never expands your scope beyond that board. Same name, different behavior; easy to wire up wrong if you’re writing both paths yourself.
Build vs buy: where the engineering work actually goes
If you build this yourself, four problems don’t have a shortcut:
Knowing when to stop paginating. Both thread-list and search page through results, but paging all the way to the end every run burns requests on data you’ve already seen — or trips rate limiting. A scheduled monitor should stop early once it hits content older than its last successful run, not re-walk the whole history every time.
Floor numbers aren’t contiguous. Full-thread mode returns replies numbered by floor, but deleted replies vanish from the output entirely — they don’t leave a gap marker. That means “the highest floor number I saw” is not the same number as “how many replies I actually got back.” A DIY build that assumes floor count equals reply count will misjudge both volume and cost.
Raw HTML needs cleanup. Reply content comes back with HTML tags (<br>, <img>, <span>) and HTML entities (&, <) still in it — turning that into clean plain text is work you have to write yourself, not something the endpoint hands you pre-cleaned.
The anti-bot risk has no advance fix. LIHKG’s public read endpoints don’t currently require an auth header, but the same community reverse-engineering effort around the x-li-digest signature shows LIHKG’s app already has that capability built in — it just isn’t being enforced on these particular endpoints today. If that changes, a self-built scraper has to be re-reverse-engineered from scratch, with no warning beforehand. A maintained actor’s advantage here isn’t a technical trick — it’s that someone is watching for the break and fixes it once, instead of every user fixing it independently.
Decide using three questions:
- Scale — a one-off pull from a single board is a few hours of work; not worth maintaining a scraper for.
- Frequency — scheduled, recurring monitoring across multiple boards or keywords means you own all four problems above indefinitely, not just once.
- Tolerance for anti-bot risk — if LIHKG tightens its endpoint, a DIY build means you re-do the reverse-engineering; a maintained actor means someone else is watching for it.
What each mode actually costs
| Event | Price | Applies to |
|---|---|---|
| Listing item | $0.002 each | Thread list, keyword search |
| Thread detail | $0.008 each | The original post in full-thread mode |
| Reply item | $0.003 each | Every reply returned in full-thread mode |
A popular, long thread can have thousands of replies, so full-thread cost scales with reply count — not with thread count. If you just need a sense of what’s active, run thread-list or search first and only pull full threads for the ones actually worth the deeper read, rather than fetching every reply on every thread by default.
Three ways to start
From least effort to most freedom
FAQ
Does LIHKG have an official public API? No. LIHKG has never published developer API documentation. Its own app talks to an undocumented api_v2 JSON endpoint, and the developer community has had to reverse-engineer that endpoint’s request signing — a GitHub issue tracking the x-li-digest signature shows the calculation has already changed once mid-discussion, so a method that works today is not guaranteed to keep working.
What’s actually different between the three read modes? Thread list takes category IDs and returns a board’s topic list with content set to null — titles and metadata only. Search takes keywords and runs them against LIHKG’s site-wide search, returning the same shape as thread list but not limited to one board. Full-thread takes thread IDs or URLs and returns the entire original post plus every reply, with replies numbered by floor — a completely different data shape from the other two modes.
If I only need the first screen of a board, is thread list enough? Yes. Thread list is the cheapest mode and the right default for scanning what’s new in one or more boards. Only reach for full-thread when you need the actual conversation — the original post plus every reply — because that mode is billed per reply and a popular thread can have thousands.
What breaks most often in a DIY LIHKG scraper? Four things: knowing when to stop paginating so you don’t burn requests on data you already have; floor numbers in full threads that aren’t contiguous because deleted replies disappear from the output, so you can’t assume floor count equals reply count; raw reply HTML that needs manual stripping before it’s plain text; and the standing risk that LIHKG adds an auth header or starts blocking datacenter IPs, which you’d have to re-reverse-engineer with no advance warning.
How is each mode priced? Thread list and search are both billed per item returned. Full-thread billing has two parts: a per-thread charge for the original post, plus a per-reply charge — so a popular, long thread costs more than a quiet one. Apify also auto-charges a flat per-run platform fee on top; that’s billed by Apify itself, not by the actor’s code.
Further reading
- Free GitHub proxy lists have thousands of stars — can you actually scrape with one?: once you’ve decided to build your own, the next problem is where your requests come from — this one breaks down free lists versus a paid residential proxy.
- Why does a crisis thread on LIHKG keep “cooling down” in the headlines but not in the replies?: a worked example of scheduling LIHKG-scraper’s search mode to track whether a thread is actually still active, not just reported as resolved.
- LIHKG’s early-warning keyword scan: if you land on “buy,” this is the search-mode setup for catching a thread before it spreads from a small board to a high-traffic one.
Chad runs 30+ published Apify actors, including LIHKG-scraper used in the worked example above. The claim that LIHKG has no official API docs and that its signature scheme has been reverse-engineered by the community is checked against the public GitHub record; the actor’s modes, fields and prices are checked against the actor’s own input schema and Store listing — anything that can’t be checked against a source isn’t in this article.