AI stacks
Multi-agent web scraping with CrewAI and Piloterr
Give CrewAI agents custom tools that call Piloterr REST APIs. Research crews scrape, enrich, and synthesize live web data, without browser farms or fragile scrapers.
- Custom tools for any Piloterr endpoint
- Role-based agents: researcher, analyst, writer
- Structured JSON for reliable agent outputs
- Anti-bot bypass for production agent crews
At a glance
Tools
agent capabilities
Crews
multi-agent teams
400+
data sources
REST
HTTP API
Why connect CrewAI
Research crews
A researcher agent scrapes via Piloterr, an analyst agent processes JSON, and a writer agent produces the final report.
Custom tools
Define @tool functions that wrap Piloterr HTTP calls, agents invoke them autonomously during tasks.
Reliable data layer
Agents receive structured JSON instead of parsing HTML, fewer hallucinations, better citations.
Scale with credits
Pay per successful scrape. Failed requests aren't charged, important when agents retry.
CrewAI + Piloterr workflows
Multi-agent patterns for market research, lead gen, and competitive intelligence.
Market research crew
Scrape SERP, news, and competitor pages, synthesize into a briefing document.
Lead enrichment crew
Domain lookup → company data → LinkedIn profile → scored lead output.
Monitoring crew
Scheduled crew checks pricing pages and alerts when changes are detected.
Due diligence crew
Multi-source scrape for investment or partnership research.
Why crews need Piloterr tools
| Approach | DIY | Piloterr |
|---|---|---|
| Agent web browse | Unreliable on protected sites | Managed scraping API |
| DIY scraper tools | Break on layout changes | Stable JSON schemas |
| Headless browser tools | Slow crew execution | Fast structured responses |
| Token efficiency | Raw HTML in context | Compact JSON fields |
Connect CrewAI in four steps
Step 1
Install CrewAI
pip install crewai requests
Step 2
Get your API key
Set PILOTERR_API_KEY in your environment.
Get your API keyStep 3
Define scraping tools
Create @tool functions that call Piloterr endpoints and return JSON.
Step 4
Assign tools to agents
Add tools to agent definitions and compose a Crew with tasks and process.
Workflow recipes
Competitive intel crew
Researcher scrapes competitor sites, analyst compares pricing, writer produces weekly report.
Lead scoring crew
Enrichment tools gather company + LinkedIn data, scorer agent ranks leads.
Due diligence crew
Multi-source scrape for M&A target research with cited sources.
Weekly market intel crew
Scheduler kicks off a crew that scrapes news, competitors, and pricing changes into a Monday briefing.
When to pick CrewAI vs LangChain
Scenario
Multi-agent role-based workflows
Recommendation: CrewAI
Scenario
Single agent with tools
Recommendation: LangChain
Scenario
RAG indexing pipelines
Recommendation: LlamaIndex
Scenario
Production ETL at scale
Recommendation: Python SDK
CrewAI tool and crew example
A research crew that searches Google News via Piloterr.
import os
import requests
from crewai.tools import tool
PILOTERR_KEY = os.environ["PILOTERR_API_KEY"]
BASE = "https://api.piloterr.com/v2"
@tool("Search Google News")
def search_news(query: str) -> str:
"""Search Google News and return article titles, sources, and snippets."""
response = requests.post(
f"{BASE}/google/news",
headers={"x-api-key": PILOTERR_KEY, "Content-Type": "application/json"},
json={"query": query, "location": "Paris, FR", "page": 1},
timeout=60,
)
response.raise_for_status()
results = response.json().get("organic_results", [])
return "\n".join(
f"- {r.get('title')} ({r.get('source')}): {r.get('snippet')}"
for r in results[:10]
)See also
LangChain
LlamaIndex
n8n
MakeTransparent credit pricing
Pay only for successful requests. Start with +500 credits, then scale with plans from $49/mo.
Premium
$49/mo
18,000 credits
Premium+
$99/mo
40,000 credits
Startup
$249/mo
110,000 credits