AI data extraction from web pages
Extraction should begin with a decision, not a crawler: what record do you need at the end?
"Extract useful information" is too vague to validate. A workable request names the fields, the permitted sources, how missing values should be represented, and which source URL supports each record.
Define the output before collecting pages
For a public supplier directory, the record might be:
| Field | Type | Rule |
|---|---|---|
| supplierName | Text | Use the displayed legal or trading name |
| category | Controlled text | Map to the agreed category list |
| location | Text | Preserve the locality as published |
| certification | List | Include only explicitly stated certificates |
| sourceUrl | URL | Keep the page used for the record |
| capturedAt | Date and time | Record when the source was captured |
Do not ask a model to infer a certification or location that the page does not state. Use an empty value and a review flag instead.
Capture a controlled source set
In Seekdown, a website data capture job starts from selected URLs and follows links within its hostname, path, depth, and page limits. Use included paths for the relevant directory and exclude navigation that produces duplicates or unrelated records.
Run a small capture first. Ten representative pages will reveal template variations and missing fields faster than a full-domain run.
Inspect the captured records
Open the memory dataset and search for several known product names, headings, or values. Check that the body contains the text needed for extraction and that each record retains its source URL.
If the important value appears only after a login, inside an image, or in a script the capture does not retrieve, change the source method rather than filling the gap by guessing.
Validate structured results
Build a review sample that includes common, sparse, and unusual pages.
| Check | Pass condition |
|---|---|
| Completeness | Required fields are present or explicitly marked missing |
| Accuracy | Values match the visible source |
| Consistency | Equivalent values use the same format |
| Provenance | Every record retains its source URL |
| Freshness | The capture timestamp fits the business need |
| Exceptions | Pages with a different template are flagged |
Calculate accuracy only from a labelled sample and record its size. For example, "47 of 50 reviewed records matched the source" is auditable; "highly accurate extraction" is not.
Export without losing provenance
Seekdown datasets can be exported as CSV or JSON. Enterprise datasets can also be exposed through an OData feed for Excel Power Query or Power BI. CSV exports include fields such as key, title, status, URL, body, summary, tags, source, and discovery timestamp.
Keep the source URL and capture time in downstream records. Without them, a reviewer cannot tell whether a surprising value came from the page, the extraction rule, or an old capture.