Scraping salary data helps businesses stay competitive by understanding compensation benchmarks across industries. It also empowers job seekers to make informed career decisions based on real-world salary insights. Comparably is a valuable resource for salary transparency, providing insights into compensation across different companies and departments. However, extracting this data programmatically can be challenging due to dynamic content loading and anti-bot measures.
Render dynamic salary pages with the Website Rendering API: see also headless browser.
In this tutorial, we'll show you how to scrape Comparably's salary data using Python and Piloterr's powerful Website Rendering API.
Why use Piloterr for Comparably Scraping?
Comparably uses Angular and loads content dynamically, making traditional scraping methods ineffective. Piloterr's browser rendering API solves this by:
- Rendering JavaScript : Fully executes the Angular application
- Bypassing protection : Handles Cloudflare and other anti-bot measures
- Browser instructions : Allows scrolling to trigger lazy-loaded content**
- Wait conditions : Ensures content is fully loaded before extraction
Prerequisites
Before starting, you'll need:
And a Piloterr API key - sign up at Piloterr
Step 1 : Fetch the rendered HTML
First, let's use Piloterr to get the fully rendered HTML of a company page:
This tells the browser to scroll_to_bottom of the page over a duration of 15 seconds, with a 2 seconds pause afterward. It ensures that all dynamic content - especially sections loaded on scroll, like salary data on Comparably - is fully rendered before capturing the page.
Step 2 : Complete scraping script
Here's the complete script that ties everything together :
Example Output
When you run this script, you'll get structured data like:
This approach gives you reliable access to Comparably's salary data, making it perfect for compensation research, market analysis, or building salary comparison tools.
