Content Freshness & Recrawling

How an AI assistant stays updated when your website changes

A website change moving through capture, processing, and an updated answer
A website change moving through capture, processing, and an updated answer

Changing a page does not instantly change an assistant's answer. The new text must be captured, processed, and indexed in the dataset the assistant uses.

A scheduled website data capture job can repeat that workflow. The schedule removes the need to upload every changed page by hand, but it still needs a sensible scope, a cadence, and checks for failed or incomplete captures.

What happens after a page changes

StageWhat happensWhat can go wrong
Source editYour team publishes a new price, policy, or instructionConflicting old pages remain public
CaptureThe job revisits pages within its hostname and path rulesThe changed page falls outside the configured scope
ProcessingCaptured content is cleaned and indexed in the datasetImportant text is hidden in an image or fails to process
AnswerThe assistant retrieves the refreshed passageAnother source contradicts the new page
ReviewA known question is asked againNobody checks whether the cited source changed

This is why "automatic updates" should not be treated as a promise that every change will produce a correct answer. Scheduling initiates a repeatable capture; verification confirms the result.

Choose a cadence based on the source

Seekdown website capture jobs support intervals from every hour to every seven days, plus Never for manual runs.

SourceStarting cadenceReason
Inventory or frequently changing listingsEvery 1 to 4 hoursAvailability may change during the day
Active documentation or release notesEvery 24 hoursDaily capture usually follows normal publishing
Policies and manualsEvery 7 days or manual after approvalThese sources change less often and may require review
Mixed websiteSeparate jobs by sectionOne schedule rarely fits every source

The shortest interval is not automatically the best. More frequent captures consume crawl resources and can repeatedly process pages that did not change.

Seekdown schedule selector showing supported capture intervals with sample data
Seekdown schedule selector showing supported capture intervals with sample data

Verify a refresh with a controlled question

Use one page and one question whose answer you know.

  1. Record the current answer and cited source.
  2. Make a visible, factual change to the source page.
  3. Run the capture job or wait for its next scheduled run.
  4. Confirm the page was captured and processed.
  5. Ask the same question again.
  6. Check both the answer and its cited source.

If the answer remains old, inspect the job scope, discovered links, processing state, and duplicate sources before changing the assistant's response instructions.

A recrawl preserves source conflicts

If an old FAQ and a new policy page disagree, capturing both again does not decide which one is authoritative. Fix or exclude the conflicting source.

Assign ownership to the source

The assistant is the last step in a content workflow. Product owners still need to update product pages, support teams still need to maintain procedures, and someone must review capture failures.

A practical ownership record contains the source section, its owner, expected update frequency, capture job, and one regression question. That small record makes stale answers easier to diagnose than a general instruction to "keep the chatbot current."