Saltar al contenido principal
Estado de la red
Iniciar sesión
Piloterr
Volver a la biblioteca

API de Website Crawler

Fetch public pages with fast HTTP crawling (no JavaScript). Best first choice for static or server-rendered HTML; use Rendering or WebUnlocker when JS or anti-bot blocks you.

Activo1 crédito = 1 solicitudGET/v2/website/crawler
Prueba el endpoint gratisPiloterr

Resumen del Endpoint

Documentación detallada, precios y ejemplos de uso.

Choosing the right endpointLink to Choosing the right endpoint

Piloterr exposes three complementary ways to fetch a public web page. Pick the lightest engine that works for your target site.

Endpoint Engine Credits Use when
Website Crawler HTTP request (no JavaScript) 1 The HTML you need is already in the first response: blogs, docs, server-rendered catalog pages, sitemaps.
Website Rendering Headless browser (JavaScript executed) 2 Content appears only after JS runs: React/Vue/Angular SPAs, lazy-loaded listings, pages that need wait_for or POST browser_instructions.
Website WebUnlocker Browser + unlock stack 3 Website Rendering or Website Crawler hit bot walls (Cloudflare, DataDome, Akamai, PerimeterX) and the domain is on Piloterr's approved whitelist.

Decision flow

  1. Try Crawler first if you do not need client-side rendering. It is the fastest and cheapest option.
  2. Switch to Rendering when the DOM is empty, prices load asynchronously, or you must wait for a selector / run browser steps.
  3. Escalate to WebUnlocker only for hardened retail or marketplace domains that block normal browser traffic. Whitelist approval is required before production use.

See also: Website Scraping guide.

OverviewLink to Overview

Website Crawler sends a direct HTTP GET to a public URL and returns the page HTML. No headless browser runs: there is no JavaScript execution, no DOM rendering, and no anti-bot unlock layer.

That makes Crawler ideal for volume and speed on sites that already ship meaningful HTML in the first response. When the page is a SPA, loads data after scroll, or sits behind advanced bot detection, move up to Website Rendering or Website WebUnlocker (see table above).

Costs 1 credit per successful call.

QuickstartLink to Quickstart

GET https://api.piloterr.com/v2/website/crawler?query=https://example.com

ParametersLink to Parameters

Parameter Type Required Description
query string yes Public URL with http:// or https://
allow_redirects boolean no Follow redirects when true (default). Set false to inspect redirect chains
return_page_source boolean no Return raw HTML source when true. Default false

ResponseLink to Response

Field Description
Response body HTML string of the fetched page

Related endpointsLink to Related endpoints

Need Endpoint
JavaScript-rendered content Website Rendering
Hard anti-bot on a whitelisted domain Website WebUnlocker
Detect protection before scraping Website Antibot

NotesLink to Notes

  • Only successful HTTP responses are billed.
  • Authenticated or paywalled pages are out of scope.
  • If you get empty or skeleton HTML, the site likely needs Website Rendering instead.

Main use casesLink to Main use cases

  • Bulk-fetch product or listing pages that render on the server
  • Monitor static competitor pages for content changes
  • Harvest blog posts, directories, and documentation at low latency
  • Capture redirect chains with allow_redirects=false

APIs relacionadas

Amplía tus capacidades de datos con estas herramientas complementarias.

Ver todo
Website Antibot/v2/website/antibot

Detecte qué protección anti-bot protege un sitio antes de scrapear. Vendor, confianza e indicios en una sola consulta.

GET1 crédito = 1 solicitudActivo
Website Crawler (POST)/v2/website/crawler

Rastree sitios web y endpoints API por URL y obtenga respuestas JSON estructuradas via POST.

POST1 crédito = 1 solicitudActivo
Website Email Phone Extractor/v2/website/email_phone_extractor

Extrae correos electrónicos y números de teléfono de sitios web para obtener información de contacto completa, incluyendo perfiles en más de 12 plataformas sociales.

GET1 crédito = 1 solicitudActivo
Website Rendering/v2/website/rendering

Render JavaScript-heavy pages in a headless browser and return post-render HTML. Use when Crawler returns empty DOM; escalate to WebUnlocker if bot protection blocks the session.

GET2 créditos = 1 solicitudActivo
Website Rendering Instructions (POST)/v2/website/rendering

Ejecuta instrucciones de automatización del navegador (scroll, scroll_to_bottom) durante el renderizado headless para activar contenido en lazy loading, eludir la detección de bots y extraer páginas cargadas dinámicamente.

POST2 créditos = 1 solicitudActivo
Website Screenshot/v2/website/screenshot

Screenshot API con renderizado tipo navegador: captura páginas web en PNG/JPEG/WebP o PDF para previews, QA, reporting, monitoreo y archivado, sin infraestructura headless.

GET2 créditos = 1 solicitudActivo
Website Technology/v2/website/technology

Identifica las tecnologías detrás de cualquier sitio web CMS, frameworks, analytics, CDN, hosting: para análisis de mercado competitivo e insights tecnológicos.

GET1 crédito = 1 solicitudActivo
Website WebUnlocker/v2/website/webunlocker

Bypass advanced anti-bot systems (Cloudflare, DataDome, Akamai, PerimeterX) on whitelisted domains. Combines browser rendering and unlock tooling; use after Crawler and Rendering fail.

GET3 créditos = 1 solicitudActivo
Website WebUnlocker (POST)/v2/website/webunlocker

Envíe peticiones POST a sitios web o APIs en lista blanca a través de WebUnlocker para eludir la protección anti-bot avanzada y obtener la respuesta upstream.

POST3 créditos = 1 solicitudActivo

¿Listo para empezar?

Tu API de web scraping está a un clic. Comienza con +500 créditos, sin infraestructura que configurar, sin proxies que gestionar y sin necesidad de tarjeta de crédito.

  • +500 créditos
  • Sin tarjeta de crédito
  • Todos los endpoints incluidos
Empieza gratis (+500 créditos)PiloterrHabla con un experto en datos