Website Scraping : Crawler vs Rendering vs WebUnlocker

Piloterr offers three complementary website products. This guide explains how they differ and when to use each.

Quick introduction to Web Scraping

Web scraping is the programmatic retrieval of web content (HTML/JSON) to extract structured information. There are two practical approaches:

• Request-mode : issue HTTP requests with realistic headers, TLS, and network fingerprints to fetch server responses directly.
• Browser-mode : drive headless browsers that execute JavaScript, load resources, and render the final DOM.

Common challenges include dynamic client-side rendering, redirects, pagination, rate limits, geo/locale variance, and enterprise anti-bot systems. Use scraping responsibly and in compliance with applicable laws and target site terms.

Piloterr Products

• Website Crawler : HTTP request mode with advanced fingerprinting. Fastest and lowest cost (1 credit), ideal for static HTML and API/JSON endpoints. No JavaScript execution.
• Website Rendering : Realistic headless browsers that fully execute JavaScript. Supports waits and selectors for reliable DOM readiness. Higher cost (2 credits). May occasionally fail on heavy/slow pages or strict anti-bot setups.
• Website WebUnlocker : HTTP request mode with enterprise anti-bot bypass (Cloudflare, DataDome, PerimeterX, Akamai, etc.). Allowlist required. 3 credits. 100% success rate on approved domains. No JavaScript execution.

How they work ?

Crawler (Request-mode)

• Performs direct HTTP(S) requests with smart header and TLS fingerprinting.
• Does not run JavaScript; returns raw HTML or body payload quickly.
• Optional flags like allow_redirects and return_page_source control behavior.

Rendering (Browser-mode)

• Launches realistic browsers to fetch and render pages client-side.
• Executes Javascript, loads resources, and can wait for DOM stability with wait_in_seconds or wait_for selectors; supports timeout, block_ads, and browser instructions.
• More resource intensive but essential for JS-heavy apps.

WebUnlocker (Anti-bot)

• Performs direct HTTP(S) requests with advanced anti-bot bypass, similar to Crawler but tuned for hardened targets.
• Does not run JavaScript; returns raw HTML or body payload after passing vendor challenges.
• Options like allow_redirects and return_page_source work the same way as on Crawler.
• Tuned to pass advanced anti-bot challenges on allowlisted domains with near-instant success and stability.

When to use which ?

• Choose Crawler : when pages are mostly static, you’re hitting API endpoints, you need maximum throughput/lowest latency, or you want the most economical option.
• Choose Rendering : when content is rendered client-side, you need precise DOM readiness, or you require interaction-like behavior (JS execution).
• Choose WebUnlocker : when you face enterprise-grade bot defenses (e.g., Cloudflare, DataDome, PerimeterX, Akamai) on static or server-rendered pages and require a 100% success rate on approved domains. For client-side JavaScript, use Rendering instead.

When to pick WebUnlocker?

Choose WebUnlocker when the use case is critical: you need real-time data (pricing, inventory, listings, regulatory compliance…) and cannot accept 10–20 second latency per request, which Rendering often imposes on protected targets. WebUnlocker stays in HTTP request mode (no JS execution) with tuned anti-bot bypass to return responses in seconds, with 100% success on approved domains.

Key differences at a glance

Explore the documentation

• Website Crawler
• Website Rendering
• Website WebUnlocker

Conclusion

Use the Crawler when you need speed, scale, and the lowest cost for static or server-rendered pages and APIs. Choose Rendering when the site relies on client-side JavaScript and you need DOM-aware waits. Pick WebUnlocker for static or server-rendered properties protected by enterprise anti-bot systems on approved domains: it delivers very low latency with a 100% success rate, without running JavaScript.