How website crawling keeps answers fresh
A website crawler does not answer visitor questions. It supplies the current source material that an assistant can search.
In Seekdown, a website data capture job moves through New, Running, Analysis, and Finished states. Error can occur during the workflow. The captured pages are processed and indexed in the connected memory dataset.
What happens during a run
- Discover: The job starts from the configured URLs and follows allowed links.
- Retrieve: It opens pages within the hostname, path, depth, and page limits.
- Process: It extracts useful page content and prepares it for the dataset.
- Index: The dataset makes the content searchable.
- Inspect: The team reviews discovered links, captured records, and errors.
Calling this an "AI website crawler" can hide the operational controls that matter. Freshness depends less on the label and more on whether the right URLs were reached and processed.
Controls that affect the answer
| Control | Freshness risk when misconfigured |
|---|---|
| Hostnames | A documentation subdomain is never visited |
| Included paths | A new section falls outside the allowed pattern |
| Excluded paths | A broad rule removes a needed page |
| Link depth | Deep detail pages are never discovered |
| Page cap | The run stops before reaching the full source set |
| Schedule | The dataset is older than the source changes require |
After the first run, compare discovered links with a sitemap or known page list. After later runs, check recent errors and spot-check changed sources.
Diagnose a stale answer in order
- Open the live source and confirm it contains the new fact.
- Confirm the URL is within the job scope.
- Check that the latest run reached Finished.
- Find the page in the captured records.
- Confirm the dataset body contains the changed text.
- Ask the question again and open its citation.
- Search for another indexed page that still states the old fact.
This sequence distinguishes a crawl problem from a source conflict or answer problem.
A schedule can reduce manual work, but it should be paired with a regression question for each important source area. That is the evidence that the crawler is keeping answers aligned with the website.