Adjust Capture Scope

The scope is how you tell a data capture job where it may go and how much to take. Get it right and it captures exactly the content you want — nothing more, nothing less.

Before You Start

  • Open the website job.
  • Know which pages belong in, and which don't.

Steps

  1. Open the job's details page from Data capture.
  2. Choose Edit properties to open the configuration form.
  3. Open the Navigation settings — this is where all the scope controls live.
  4. Review the initial URLs (where the job starts) and the hostnames list. The job will only follow links that match one of the allowed hostnames, so if your site uses subdomains (e.g. docs.acme.com and blog.acme.com), add each one.
  5. Add included paths to restrict the job to specific sections. For example, */docs/* keeps only pages whose URL contains /docs/. If you leave this empty, the job visits everything on the allowed hostnames.
  6. Add excluded paths for areas you want to skip — login pages, admin panels, tag archives, and so on. Excluded paths take priority over included paths, so you can include a broad section and carve out exceptions within it.
  7. Set the crawl depth (how many link-hops from the starting URLs the job will travel) and the max content count (an absolute cap on how many pages it processes in a single run).
  8. Save the updated configuration.
The job's Navigation settings — initial URLs with site tags, and the hostnames list that restricts the job to specific domains (sample data).
The job's Navigation settings — initial URLs with site tags, and the hostnames list that restricts the job to specific domains (sample data).
A site's URL tree with path rules applied — the docs section kept (green) and admin, login and tag paths skipped (red).
A site's URL tree with path rules applied — the docs section kept (green) and admin, login and tag paths skipped (red).

Path rule examples

Included and excluded paths are glob patterns matched against a URL's path:

- Keep one section: included path */docs/* captures only the documentation and ignores everything else. - Skip the noise: excluded paths */login/*, */cart/*, */tag/* drop sign-in, checkout, and tag-archive pages. - Carve an exception: include a broad section and exclude a sub-path within it — excluded paths win over included ones.

Verify the Result

Re-run the job, then check the discovered links and refreshed contents to see the new scope in action.

Troubleshooting

  • Too little content usually means the scope is too narrow.
  • Too much means it's too broad.
  • Remember an excluded path can quietly cancel out a page you thought you'd included.
Contact us

Still need help?

Tell us what you want your website assistant to answer. We will help you map the right content, controls, and launch path.