Extract structured data with ChatGPT after you fetch the page

Paste a product URL into ChatGPT and it will often return a price. Open the live page: the number is wrong, outdated, or invented. That is not a failing scraper. The model never opened the URL. It inferred the value from training data and stated it with confidence.

Treat ChatGPT as a parser, not a browser. Piloterr loads the page. Try JSON-LD or a CSS selector first. Only the remaining text is sent to an OpenAI model. Pydantic discards any object that does not match the schema.

The demonstration uses books.toscrape.com, a public catalog intended for practice. Stay on public pages, respect robots.txt, and read Is web scraping legal? before targeting a commercial site.

In this tutorial you will:

1. Fetch a product page with Website Crawler (1 credit).
2. Read title, price, stock, and UPC from CSS when the markup is stable.
3. Fall back to gpt-5.6-luna structured outputs when it is not.
4. Split a category page on article.product_pod instead of slicing mid-word.

Two tasks, two tools

A language model reads text. It does not render JavaScript, rotate IP addresses, or pass a Cloudflare challenge. Ask it to retrieve a URL and you receive a blocked page, an empty SPA, or an invented payload.

Keep the split explicit:

1. Fetch a complete page (Piloterr, or a library endpoint if one exists).
2. Parse fields (CSS, JSON-LD, or a model).

ChatGPT is appropriate when the markup is irregular, or changes often enough that maintaining a large set of selectors costs more than a schema. Amazon, LinkedIn, and similar targets already have maintained APIs. Use those.

Parse the DOM before you call the model

Start with a CSS extractor. On this catalog the title is h1, the price is p.price_color, and the UPC sits in a table row. That path consumes fewer tokens than gpt-5.6-luna and returns the same string the next day.

from bs4 import BeautifulSoup
from pydantic import BaseModel


class Book(BaseModel):
    title: str
    price: str
    availability: str
    upc: str | None = None
    isbn: str | None = None


def extract_book_css(html: str) -> Book | None:
    soup = BeautifulSoup(html, "html.parser")
    title = soup.select_one("article.product_page h1")
    price = soup.select_one("article.product_page p.price_color")
    stock = soup.select_one("article.product_page p.instock.availability")
    if not (title and price and stock):
        return None

    upc = None
    for row in soup.select("article.product_page table tr"):
        label = row.select_one("th")
        value = row.select_one("td")
        if label and value and label.get_text(strip=True) == "UPC":
            upc = value.get_text(strip=True)
            break

    return Book(
        title=title.get_text(strip=True),
        price=price.get_text(strip=True),
        availability=" ".join(stock.get_text().split()),
        upc=upc,
        isbn=None,
    )

isbn stays None because this page has no ISBN. Do not ask a model to invent one.

Look for, in this order:

• A library API (Amazon Product, and so on)
• application/ld+json on the page
• A stable CSS or XPath node
• A model, for remaining text: unstructured specifications, FAQ blocks, descriptions that mix units and caveats

Plain price strings can go through Price-Parser without an LLM.

import json


def json_ld_blocks(html: str) -> list[dict]:
    soup = BeautifulSoup(html, "html.parser")
    blocks = []
    for node in soup.select('script[type="application/ld+json"]'):
        if not node.string:
            continue
        try:
            parsed = json.loads(node.string)
        except json.JSONDecodeError:
            continue
        blocks.extend(parsed if isinstance(parsed, list) else [parsed])
    return blocks

Choose a fetch mode before you write prompts

An unsuitable mode returns empty HTML. The model then "extracts" from a login page.

Full comparison: Crawler vs Rendering vs WebUnlocker. This tutorial uses Crawler because books.toscrape.com is server-rendered.

Fetch the HTML

Python 3.10+, a Piloterr API key, and an OpenAI key. Both in the environment. Never in the script.

python --version
mkdir fetch-then-extract && cd fetch-then-extract
pip install requests beautifulsoup4 "openai>=2" pydantic
export PILOTERR_API_KEY="your-x-api-key"
export OPENAI_API_KEY="your-openai-key"

Crawler is a GET. query is the target URL. The JSON body is the HTML string.

import os
import requests

API = "https://api.piloterr.com/v2/website/crawler"
KEY = os.environ["PILOTERR_API_KEY"]
PAGE = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
CATALOG = "https://books.toscrape.com/"


def fetch_html(url: str, endpoint: str = API, **extra: object) -> str:
    response = requests.get(
        endpoint,
        headers={"x-api-key": KEY},
        params={"query": url, "allow_redirects": "true", **extra},
        timeout=60,
    )
    response.raise_for_status()
    html = response.json()
    if not isinstance(html, str) or len(html) < 200:
        raise RuntimeError("Fetch returned too little HTML. Try Rendering or WebUnlocker.")
    return html

A few hundred kilobytes is expected. A few hundred characters is usually a challenge page. Switch mode. Do not ask the model to compensate for incomplete HTML.

For a JavaScript-heavy product page, the same request pattern applies to Rendering (2 credits):

RENDER = "https://api.piloterr.com/v2/website/rendering"

def fetch_rendered(url: str) -> str:
    return fetch_html(url, endpoint=RENDER, wait_in_seconds=4)

Documentation: Website Crawler and Website Rendering. Successful calls bill 1 or 2 credits. Client 4xx and most platform 5xx do not. See pricing and the Markdown version /pricing.md.

Reduce the HTML before you call the model

Scripts, SVG paths, and cookie banners consume tokens and hide the fields you need.

def readable_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "noscript", "svg", "iframe"]):
        tag.decompose()
    main = soup.select_one("article.product_page") or soup.body or soup
    return main.get_text("\n", strip=True)

Structured outputs, then validation

When CSS fails, the prompt becomes the extractor. Name every field. Use null when the text has no value. client.responses.parse binds the response to the Pydantic class, so a malformed object is never treated as a trusted dict.

from openai import OpenAI

client = OpenAI()

SYSTEM = (
    "Read the page text and fill the schema. "
    "Copy strings as they appear. "
    "Use null when a key is absent. "
    "Do not invent a title, price, UPC, or ISBN."
)


def extract_book_llm(text: str) -> Book:
    response = client.responses.parse(
        model="gpt-5.6-luna",
        input=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": f"Fill the Book schema from this page text.\n\n{text}"},
        ],
        text_format=Book,
        reasoning={"effort": "none"},
    )
    book = response.output_parsed
    if book is None:
        raise ValueError("Model returned no parsed object")
    return book

gpt-5.6-luna is enough for field copy. Step up to gpt-5.6 only when two prices disagree or a table is misread. Run a sample of 20 pages, measure failures, then choose the model. client.chat.completions.parse still works if you stay on Chat Completions. Prefer responses.parse on openai 2.x.

One script

if __name__ == "__main__":
    html = fetch_html(PAGE)
    book = extract_book_css(html)
    if book is None:
        book = extract_book_llm(readable_text(html))
    print(book.model_dump_json(indent=2))

Expected shape on the "A Light in the Attic" product page (UPC is on the page; ISBN is not):

{
  "title": "A Light in the Attic",
  "price": "£51.77",
  "availability": "In stock (22 available)",
  "upc": "a897fe39b1053632",
  "isbn": null
}

A hallucinated ISBN can sit in a spreadsheet without raising an alert, until an order is placed against it.

Long pages: split on product cards

Sending an entire category in one prompt exceeds the context window. Do not slice at 12,000 characters mid-word. Parse the list, then extract one card.

def product_cards(html: str) -> list[str]:
    soup = BeautifulSoup(html, "html.parser")
    return [str(card) for card in soup.select("article.product_pod")]


def extract_catalog(html: str) -> list[Book]:
    books: list[Book] = []
    seen: set[str] = set()
    for card_html in product_cards(html):
        book = extract_book_css(card_html) or extract_book_llm(readable_text(card_html))
        key = book.title.casefold()
        if key in seen:
            continue
        seen.add(key)
        books.append(book)
    return books

Listing cards have no UPC table. CSS returns None and the model fills title and price from the card text. Merge on title, or keep the href if you stored it.

print(extract_catalog(fetch_html(CATALOG)))

Let the agent fetch, keep the same split

Pasting HTML into the ChatGPT interface is suitable for a demonstration. Daily processing belongs in Python, or in a tool the agent calls.

Piloterr MCP exposes Crawler and Rendering at https://mcp.piloterr.com/ with the same x-api-key. Claude and Cursor can fetch a page. You still validate JSON in your code. Specification notes: Piloterr MCP on 2026-07-28.

Practices to avoid

• Ask ChatGPT for a live URL and trust the answer.
• Send raw HTML with advertisements and scripts still attached.
• Store model output without a Pydantic (or equivalent) check.
• Use Rendering on a static catalog and bill 2 credits without need.
• Scrape behind authentication, or collect personal data, on the grounds that the model "needed context".

FAQ

Can ChatGPT scrape a website by itself?

No. It has no browser you control and no reliable way to load a URL you just typed. It can rewrite text you already fetched. Fetch with Website Crawler or Website Rendering, then extract.

Why not call requests.get directly?

You can, on books.toscrape.com. On a JavaScript storefront or a Cloudflare-protected site you receive an empty root or a challenge page. The model will still emit fields from that response. Piloterr returns the page a browser would display, and you are billed only on a successful fetch.

When is Rendering the appropriate fetch?

When the price or list is missing from "View source" but visible in DevTools after the application hydrates. Wait with wait_in_seconds or wait_for. If the HTML is already in the first response, stay on Crawler.

How can you prevent invented prices?

CSS or JSON-LD first. Then copy-verbatim instructions, reasoning={"effort": "none"}, structured outputs, and Pydantic. Compare the price string with Price-Parser. If validation fails, retry once or fall back to a selector. Do not fill in a missing price.

Should you use ChatGPT or a library API?

Library API when that site is already covered. ChatGPT for an ad hoc page or an unstable template. Combining both is appropriate: fetch with Crawler, parse with CSS, call the model only on the fields that failed.

What if the listing is too long for one model call?

Extract per article or per row. Deduplicate on title or URL. Do not chunk by character index unless the page has no list markup.

Is this legal?

Public pages, reasonable rates, no authentication, no personal data, and the site's terms still apply. Start with Is web scraping legal?. For commercial reuse, obtain a license or an official feed when one exists.