Blocked by Cloudflare? First Figure Out If It's a Block or a JS Challenge — Proxies Don't Fix Both

In one line: When a Cloudflare-protected site starts blocking your scraper, look at what the response actually contains before you touch your proxy settings — a clean 403 error page and a “Just a moment…” verification screen are two different Cloudflare actions, and only one of them is fixed by a better IP.

“My scraper stopped working against a Cloudflare site” is not one problem

Cloudflare sits in front of a huge share of the sites anyone might want to scrape — forums, e-commerce, ticketing, you name it — so “it worked yesterday, now I’m getting blocked” is one of the most common messages in any scraping community. The instinctive fix is to swap tools or swap proxies. But whatever you’re running your requests through — Apify, a plain script, a paid scraping API — is just an execution environment. You can point it at a target with a bare datacenter IP and no browser at all, or with a full headless browser carrying a consistent fingerprint, and Cloudflare treats those two setups completely differently. The question that actually matters isn’t which tool is “better” — it’s which Cloudflare mechanism is stopping you, because the two main ones don’t share a fix.

Two different walls: a clean block vs. a challenge

Cloudflare’s own documentation splits WAF rule actions into different tiers by confidence. At the high-confidence end, a rule can just Block: traffic that scores as “definitively automated” gets rejected outright, and Cloudflare states plainly that “blocking it carries minimal false positive risk” — no verification step, just a rejection.

Challenge is a different action entirely. Cloudflare’s challenge-page documentation describes it as a gate that “acts as a gate between the visitor and your website or application while Cloudflare verifies the authenticity of the visitor.” Concretely, the page “intercepts the visitor from getting to the destination URL by holding the request and evaluating the browser environment for automated signals, and serving a challenge” — that’s the “Just a moment…” screen. Only after a visitor passes does “the original request continue to the destination URL.” That evaluation runs on signals from Cloudflare’s Bot Management, which scores every request from 1–99, where the documentation notes “scores below 30 are commonly associated with bot traffic.”

Here’s the part that trips people up: a Challenge response can also carry a 403 status code. Cloudflare’s own support documentation confirms this directly: a 403 with Cloudflare branding “may be triggered by… WAF Custom or Managed Rules with the challenge or block action” — not the block action alone. Apify’s anti-scraping documentation makes the same point more specifically: “the Cloudflare challenge screen may return a 403 status code even if it is evaluating the fingerprint and the request is not blocked.” So the status code by itself isn’t reliable evidence of which wall you hit; you have to look at the response body — a clean error page versus a page carrying JS verification code and a “Just a moment…” title.

Why that difference decides whether a proxy helps

A real Block-tier case: LIHKG-scraper’s default proxy

LIHKG-scraper, which scrapes Hong Kong’s LIHKG forum, is a clean example of Case 1. Its proxyConfiguration input field ships with Apify’s (datacenter) proxy turned on by default, and the actor’s own description explains why in plain terms: without a proxy, a run shares its exit IP with every other proxy-less run on the platform, and “LIHKG will rate-limit you because of someone else’s traffic (429s, missing replies).”

That’s a textbook Case 1 problem — a plain IP-reputation and volume issue (HTTP 429, not a JS verification page), with no browser fingerprint involved. The fix maps to the same layer, but it isn’t a jump to residential: turning the proxy on at all — Apify’s own datacenter tier, not residential — is what gets a run off the shared, platform-wide egress pool, and the rate limiting clears up. No headless browser required.

Datacenter proxy isn’t permanent immunity, though. A separate actor sharing the same connection layer measured, on 2026-09-05, that LIHKG’s Cloudflare still rate-limits a portion of datacenter-proxy sessions — 15 fresh sessions in a row came back 6 clean, 9 hit with a 429 challenge page. The fix there wasn’t upgrading to residential either; it was retrying with a freshly-rotated session on the same datacenter tier. Same lever — a different exit IP — not the fingerprint layer Dcard’s challenge cares about.

A Cloudflare JS challenge is a different order of problem. If what comes back is a “Just a moment…” page rather than a clean error or a 429, you’ve moved into Case 2 — switching to residential proxy, whether from no proxy or from datacenter, might help a little (IP reputation is a real input into the score), but it won’t be the decisive fix, because what’s holding you back is the fingerprint layer, not the IP.

Once you’ve confirmed you’re in Case 2, what next

If the diagnosis comes back as a JS verification page, Apify’s anti-scraping documentation shifts the focus from proxy to browser fingerprint:

• Use Crawlee’s built-in fingerprint generation, and keep a session’s fingerprint stable rather than rotating it on every request — Apify’s own guidance is blunt about this: “the default browser fingerprint may actually be more effective than an inconsistently generated fingerprint,” so constant rotation reads as more automated, not less.
• Stop treating every “blocked” status code as a hard stop — Apify’s docs do this by setting sessionPoolOptions.blockedStatusCodes to [] — since a 403 during a challenge doesn’t necessarily mean you were actually rejected. Once you remove that default handling, add your own logic to wait for the challenge to resolve and the page to redirect, and retire the session for a fresh one when a page turns out to be genuinely blocked.
• If a consistent fingerprint still lands on a CAPTCHA, Apify’s docs treat that as a sign your IP has been “graylisted” — with a large proxy pool, retiring the session for a new IP is usually the faster fix; with a small pool, that’s when a captcha-solving service becomes the next, more expensive, lever.

The one-line takeaway: “my scraper can’t get past this site” doesn’t tell you anything about which fix applies. What matters is whether the response is a clean rejection or a JS verification page — and only the first one gets solved by a better IP address.

Pick one and start

From least effort to most freedom

I run it for you Managed, daily to your sheet You do nothing. From US$299/mo. Use it here Run the free tool Instant, no setup, on this site. Run my actor On Apify, your account Same data, you run it and pay per result. Build your own Sign up for Apify Free account — build the pipeline yourself.

This post contains Apify affiliate links (no extra cost to you). Costs and comparisons are stated honestly, including where Apify is not the best fit.

FAQ

My scraper can’t get past a Cloudflare-protected site — does that mean the platform or tool I’m using is broken? Not necessarily. Whatever you’re running it on is just an execution environment — a bare HTTP request and a full headless browser with a real fingerprint behave completely differently against Cloudflare, even from the same platform. The question worth answering first isn’t which tool is better, it’s which Cloudflare mechanism is actually stopping you, because the two mechanisms need different fixes.

What’s the actual difference between an HTTP 403 and a Just a moment page? It’s the mechanism behind them. Cloudflare’s Block action returns a 403 with no verification step at all. Its Challenge action can also carry a 403 status code, but the response body is a JS verification page that evaluates your browser environment before deciding whether to let you through. So the status code alone is not enough to tell them apart — you have to look at whether the response body contains that verification page.

Does switching to a residential proxy fix a Cloudflare JS challenge? Not reliably, though it can lower how often you trigger one. Cloudflare’s own support docs confirm a challenge action can carry a 403 just like a block action can, and Apify’s documentation is direct about the practical effect: Cloudflare’s challenge screen can return a 403 while it’s evaluating your browser fingerprint, even when the request isn’t actually blocked. Proxy is only one input; a clean residential IP with a fingerprint that isn’t generated well, or that changes on every request, can still get flagged as automated traffic.

LIHKG-scraper ships with a proxy on by default — is that solving the same problem? Same broad category (anti-bot), different layer entirely. LIHKG-scraper’s own listing explains that without a proxy it shares an exit IP with other runs on the platform, and gets rate-limited by LIHKG with 429 responses because of that shared traffic — a pure IP-reputation and volume problem that Apify’s own datacenter-tier proxy solves directly (no need to jump to residential), with no browser fingerprint involved. A Cloudflare JS challenge is a level up from that, and the two can’t be treated with the same fix.

Once I’ve confirmed I’m stuck on a JS challenge, what’s the next step? Proxy stops being the lever, and browser fingerprint consistency becomes the lever. Apify’s documentation for this recommends using Crawlee’s built-in fingerprint generation, keeping a session’s fingerprint stable rather than rotating it constantly, removing the default blocked-status-code handling, and adding your own retry and session-retirement logic instead of trusting the raw status code — since a 403 doesn’t always mean you were actually blocked. If a consistent fingerprint still runs into a CAPTCHA, Apify’s docs treat that as a sign the IP is graylisted: with a large proxy pool, switch sessions; with a small one, that’s when a captcha-solving service becomes the next, more expensive, option.

Further reading

• Free GitHub Proxy Lists vs Apify Residential Proxy: once you’ve confirmed you’re in the Block/IP-reputation case, this piece breaks down free lists vs datacenter vs residential proxy by price and survival rate.
• Is That Hong Kong Listing Actually For Sale?: a concrete case on the other end of this spectrum — Midland’s site returns 403 to datacenter IPs specifically, so that actor needs residential proxy just to load at all.

Chad runs 40+ published Apify actors, including LIHKG-scraper used here. Every claim about Cloudflare’s Block/Challenge mechanics links back to Cloudflare’s own documentation or Apify’s anti-scraping academy; anything that couldn’t be independently verified was left out.