Skip to main content
Network status
Log in
Piloterr
Back to library

Website Crawler API

A robust solution for efficiently extracting a wide range of data from web pages via high-performance request-mode crawling.

Active1 credit = 1 requestGET/v2/website/crawler
Try endpoint for freePiloterr

Endpoint Overview

Detailed documentation, pricing, and usage examples.

OverviewLink to Overview

The Website Crawler API provides high-performance, request-mode scraping of any public webpage, returning the full HTML content. Unlike browser-based rendering, it operates without JavaScript execution, making it extremely fast and cost-effective for static or server-rendered pages.

QuickstartLink to Quickstart

GET https://api.piloterr.com/v2/website/crawler?query=https://example.com

ParametersLink to Parameters

Parameter Type Required Description
query string yes A website URL with either the http or https protocol
allow_redirects boolean no If set to false, do not follow redirects. true by default
return_page_source boolean no If set to true, the response will be the raw HTML source of the page. false by default

Response fieldsLink to Response fields

Field Type Description
(response body) string Full HTML content of the scraped webpage

NotesLink to Notes

  • For JavaScript-heavy pages, consider using the Website Rendering or Website WebUnlocker endpoint instead
  • Only successful HTTP responses count toward your credit balance
  • Supports both http and https protocols

Main use casesLink to Main use cases

  • Scrape product pages from e-commerce websites to extract pricing, descriptions, and availability at scale
  • Monitor competitor websites for content changes and updates in near real-time
  • Extract structured content from blogs, news sites, and directories for data aggregation pipelines
  • Build lead generation tools by crawling company pages for contact and firmographic information

Related APIs

Expand your data capabilities with these complementary tools.

View All
Website Email Phone Extractor/v2/website/email_phone_extractor

Extract emails and phone numbers from websites for comprehensive contact details, including social media profiles across 12+ platforms.

GET1 credit = 1 requestActive
Website Rendering Instructions (POST)/v2/website/rendering

Execute browser automation instructions (scroll, scroll_to_bottom) during headless rendering to trigger lazy-loaded content, bypass anti-bot detection, and scrape dynamically loaded pages.

POST2 credits = 1 requestActive
Website Screenshot/v2/website/screenshot

Browser-like Screenshot API that captures webpages as PNG/JPEG/WebP or PDF for previews, QA, reporting, monitoring, and archiving no headless setup.

GET2 credits = 1 requestActive
Website Technology/v2/website/technology

Identify the technologies behind any website CMS, frameworks, analytics, CDN, hosting, and more — for competitive market analysis and technological insight.

GET1 credit = 1 requestActive
Website WebUnlocker/v2/website/webunlocker

WebUnlocker is a hybrid of realistic JS rendering and request-mode crawling, engineered to bypass advanced anti-bot systems on whitelisted websites.

GET3 credits = 1 requestActive

Ready to get started?

Your web scraping API is one click away. Start with +500 credits, no infrastructure to set up, no proxies to manage, and no credit card required.

  • +500 credits
  • No credit card required
  • All endpoints included
Start free (+500 credits)PiloterrTalk to a data expert