What is a recurring website crawler?
A recurring website crawler revisits a defined set of web pages on a schedule. In Seekdown, a website data capture job retrieves the allowed pages and processes their content into a memory dataset.
The recurrence matters because an assistant answers from the dataset, not directly from the live page at the instant a visitor asks. If the dataset contains an older capture, the answer may reflect that older text.
Recurring does not mean unrestricted
A useful crawler needs boundaries:
| Control | Purpose |
|---|---|
| Initial URLs | Define where discovery begins |
| Allowed hostnames | Stop the job from following links onto unrelated domains |
| Included paths | Limit capture to relevant sections |
| Excluded paths | Remove login, archive, checkout, or other unwanted areas |
| Link depth | Limit how far discovery follows links |
| Page cap | Limit how many pages a run can retrieve |
| Schedule | Decide when the job runs again |
A crawler configured only with a homepage can discover far more than the assistant should use. Scope is therefore part of answer quality and data governance, not just a crawler setting.
What a recurring run can and cannot do
A run can retrieve changed website text, update indexed records, and make the processed content available to connected assistants.
It cannot:
- decide which of two conflicting public pages is authoritative;
- recover important facts that exist only in an unreadable image;
- access private material without an appropriate capture method;
- guarantee that every discovered page processed successfully; or
- decide whether a changed answer is acceptable to your team.
Those tasks still require source ownership and review.
Choose recurrence from publishing behavior
Start with the slowest cadence that meets the content need.
- Use an hourly or multi-hour cadence for sources that genuinely change during the day.
- Use a daily cadence for active documentation, news, or a frequently edited catalog.
- Use a weekly cadence for stable manuals and policies.
- Use manual runs when changes follow a controlled release and should not publish before approval.
Splitting a site into several capture jobs can be clearer than giving the whole domain one aggressive schedule.
When you may not need one
A recurring website crawler adds little value when the source is a static file updated once a year, the dataset is intentionally frozen for audit reasons, or each change must pass a manual approval step before it can affect answers.
In those cases, a documented manual refresh and verification process may be more appropriate. The goal is current, controlled source material, not recurrence for its own sake.