Customize Your Website Scraper
If you built from a website, a data capture job decides which pages become knowledge. Left alone it does a reasonable job, but a few minutes of tuning keeps it on the content that matters and off the pages that don't (logins, checkouts, tag archives).
How the Data Capture Job Works
A data capture job starts from the pages you give it and works page by page: it opens a page, extracts and processes its content into your dataset, reads the links on that page, and then follows those discovered links to the next pages — repeating outward until it hits the limits you set. Everything below is about drawing those limits, so it visits the right pages and skips the rest.
The job's settings are grouped into a few sections. These three are the ones worth knowing early.
Navigation — Where It Goes
This section controls the job's reach:
- Initial URLs — the entry points it starts from. Each one is a seed it discovers links from.
- Hostnames — the domains it's allowed to visit. When set, it only follows links whose host matches, so it can't wander off to external sites.
- Included paths — glob patterns; when set, only URLs matching them are crawled (e.g.
*/docs/*). - Excluded paths — glob patterns that are skipped even if included above (e.g.
*/tag/*). - Depth — how many link hops from the initial URLs it follows.
0= just the pages you gave it. - Sitemap URL — an optional
sitemap.xmlused as an extra source of pages, to reach ones not linked from navigation. - Max number of contents to crawl — the page cap for a run: Preview 100, 500, 1500, or 5000.
Targeting a partial section of a site
Combine hostnames with include/exclude paths to capture exactly one area:
- Only the docs. Initial URL https://www.acme.com/docs/, hostname https://www.acme.com, included path */docs/*. Everything outside /docs is ignored. - The whole site, minus the noise. Leave included paths empty (all pages qualify) and exclude the clutter: */tag/*, */login/*, */cart/*. - One subdomain only. Initial URL https://blog.acme.com/, hostname https://blog.acme.com, included path */blog/* — the job stays on the blog and never crosses to the main site.
Excluded paths win over included ones, so you can include a broad section and carve exceptions out of it.
Trial workspaces use Preview
On the free trial, crawls are capped at Preview (up to 100 pages) — enough for a small site or a focused section. Upgrade to unlock the larger caps.
Contents — What It Stores
This section shapes the content the job produces:
- Content type — a label applied to every item this job creates. It's more than housekeeping: your assistant can use the content type to discriminate between results (for example, answer only from a given type) and to drive custom visualizations, rendering that type differently in the chat.
- Content lifespan days — how often a piece of content is forced to be re-processed. Once an item is older than its lifespan, the next run refreshes it instead of skipping it — so pages that change often stay current.
- Tags — labels attached to everything the job captures. Use them to filter content, and to personalize what the assistant answers with (for example, scope answers to a tagged subset).
Others
A single option: Analyze and skip cookie consent dialogs (on by default), so consent banners don't get captured as page content.