Website Crawler
Website Crawler

Website Crawler

A robust solution for efficiently extracting a wide range of data from web pages.
Public access
Private access
July 25, 2024
2500
ms
798
k requests
354
votes
▶ Try for freeGet it customized
50 free requests, no credit card required
Request accessGet it customized
Access to this API endpoint is limited, submit a request to gain access

Header

x-api-key required

string

This parameter defines the Piloterr private key to use for authentification.

Parameters

query

string

A website URL with either the http or https protocol

impersonate_version

string

Impersonate a browser version (e.g chrome124)

allow_redirects

boolean

If set to false, do not follow redirects. false by default.

Sign up now and enjoy 50 free API requests + full access to our premium features and tools.
Request access to this private endpoint and unlock 50 free API requests, along with unrestricted use of our public API endpoints and in-house proxy pool.

Our advanced HTML scraping API is a robust solution designed for efficiently extracting data from web pages. Leveraging cutting-edge technology, our API ensures stable requests, even on the most intricate websites. Ideal for businesses and developers needing access to real-time, dependable information.

Please always use this API in compliance with prevailing laws and the terms of use of the targeted websites. Emulates Chrome TLS fingerprint, backed by rotating proxies and smart retries. TLS Fingerprinting is a technique that allows you to bypass TLS certificate pinning.

Key Features

Cutting-Edge Technology

Leveraging state-of-the-art technology, our API ensures stable requests. It's tailored to work seamlessly even with the most intricate websites, providing you with accurate data every time.

Built for Businesses and Developers

This is an essential tool for businesses and developers who require access to real-time, dependable information. Whether you're into market research, competitor analysis, or web development, our API will transform the way you gather data.

Safety and Compliance

Always ensure to use this API in compliance with prevailing laws and the terms of use of the targeted websites. We advocate for responsible and ethical scraping practices.

Advanced TLS Fingerprinting

Our API emulates the Chrome TLS fingerprint. What's TLS Fingerprinting? It's a technique that allows you to bypass TLS certificate pinning, enhancing the flexibility and efficiency of your scraping operations. And with the backing of rotating proxies and smart retries, you're ensured uninterrupted service.

Effortless Bypassing Capabilities

Step into the future of web scraping with our advanced HTML scraping API. Your data extraction tasks just got a lot easier.

  • Bypass Cloudflare Bot Management: Seamlessly navigate through Cloudflare's security.
  • Bypass Akamai: Overcome Akamai's protective measures effortlessly.
  • Bypass PerimeterX: Easily bypass PerimeterX defenses.
  • Bypass DataDome: DataDome's security measures will no longer impede your progress.

Supported impersonate versions

  • Chrome: 99, 100, 101, 104, 107, 110, 116, 119, 120, 123, 124
  • Chrome (Android): 99
  • Edge: 99, 101
  • Safari: 15.3, 15.5, 17.0
  • Safari (iOS): 17.2
  • GET
    POST
    /api/v2/website/crawler
    Example Request
    curl --location --request GET 'https://piloterr.com/api/v2/website/crawler?query=https://example.com' \
    --header 'Content-Type: application/json' \
    --header 'x-api-key: <token>' \
    Response
    <html>
    <head> <title>Example Domain</title> <meta charset="utf-8" /> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /> <meta name="viewport" content="width=device-width, initial-scale=1" /> <style type="text/css"> body { background-color: #f0f0f2; margin: 0; padding: 0; font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif; } div { width: 600px; margin: 5em auto; padding: 2em; background-color: #fdfdff; border-radius: 0.5em; box-shadow: 2px 3px 7px 2px rgba(0, 0, 0, 0.02); } a:link, a:visited { color: #38488f; text-decoration: none; } @media (max-width: 700px) { div { margin: 0 auto; width: auto; } } </style>
    </head> <body>
    <div> <h1>Example Domain</h1> <p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p> <p><a href="https://www.iana.org/domains/example">More information...</a></p>
    </div>
    </body>
    </html>
    Premium
    $49/mo
    2,50€ = 1,000 requests
    18,000 requests
    7 requests per second
    Clean & parsed data
    Email support
    Premium+
    $99/mo
    2,23€ = 1,000 requests
    40,000 requests
    10 requests per second
    Clean & parsed data
    Email support
    Startup
    $249/mo
    2,08€ = 1,000 requests
    110,000 requests
    15 requests per second
    Clean & parsed data
    Email & chat support
    Startup+
    $499/mo
    $2.17 = 1,000 requests
    230,000 requests
    20 requests per second
    Clean & parsed data
    Chat & Email support
    Enterprise+
    $999/mo
    $1.88 = 1,000 requests
    530,000 requests
    30 requests per second
    Clean & parsed data
    Account manager
    Custom
    +$2,000/mo
    Custom CPM price
    Custom requests count
    Custom requests rate
    Clean & parsed data
    Account manager
    Premium
    45€/mo
    2,50€ = 1,000 requests
    18,000 requests
    7 requests per second
    Clean & parsed data
    Email support
    Premium+
    89€/mo
    2,23€ = 1,000 requests
    40,000 requests
    10 requests per second
    Clean & parsed data
    Email support
    Startup
    229€/mo
    2,08€ = 1,000 requests
    110,000 requests
    15 requests per second
    Clean & parsed data
    Email & chat support
    Startup+
    459€/mo
    2,00€ = 1,000 requests
    230,000 requests
    20 requests per second
    Clean & parsed data
    Chat & Email support
    Enterprise+
    919€/mo
    1,73€ = 1,000 requests
    530,000 requests
    30 requests per second
    Clean & parsed data
    Account manager
    Custom
    $2,000/mo
    Custom CPM price
    Unlimited requests
    Custom requests rate
    Clean & parsed data
    Account manager

    Get 50 free requests when signing-up

    Use Piloterr's private proxy pool

    Get 10% off when buying annual subscriptions, contact us to get started.
    Accepted payment methods:

    Get 50 free requests now!

    Sign up now and enjoy 50 free API requests + full access to our premium endpoints!

    "Piloterr offers a great set of APIs to retrieve companies and profiles data. Endpoints are well documented, responses are fast and there is great support if needed (way better than rapidapi by the way)! A really good job from the Piloterr team! Continue the good work guys!"

    Mantiks

    Alexandre, CEO Mantiks

    "Thanks to Piloterr for handling our use-cases professionally"

    Datagma

    Raphael, CEO

    "Piloterr raises the game of automation in Growth Hacking with innovative integrations such as LinkedIn Profile info and Crunchbase, opening up new perspectives."

    Kévin Zimmermann

    Growth

    Frequently asked questions

    Everything you need to know about our products & services.

    Are Piloterr solutions secured and GDPR compliant?

    We have implemented industry-standard security measures to ensure the safety of your data. Our platform uses SSL encryption to protect your API requests. We're also fully aligned with GDPR and CCPA compliance standards.

    How frequently are new APIs added, can I suggest new ones?

    We are always adding new APIs to our platform based on customer demand and emerging technologies. If you have a specific API you would like us to add, please contact our support team and let us know.

    Can I upgrade, downgrade and cancel anytime?

    Active plans can be upgraded, downgraded and cancelled anytime. Changes are immediate. When changing plan, the remaining unused requests of your current plan are added from your next invoice amount.

    What is your cancellation policy?

    We understand that things change. You can cancel your plan at any time and we’ll refund you the difference that you already paid (annual subscriptions only and under the terms described here).

    Do my unused credits rollover to the next month?

    Your monthly subscription restarts with your monthly allowance of successful search credits on the first day of your billing cycle's subscription.

    Do you offer technical support for users?

    Yes! We are always happy to help users to be able to use our API! You can chat us and we'll try and get back to you within a single business day.

    How requests API calls are calculated?

    Only successful requests are counted towards your monthly searches. Cached, errored and failed searches are not.

    Can I request custom API endpoints?

    Of course! We are always expanding our code and building new APIs for different search engines. You can create a feature request on our Roadmap. We aren't able to build APIs for every websites for a variety of reasons but we do always make additions where they are needed and able to be done.