If your chatbot still quotes the old policy after a help-center update, check the content it actually retrieved, any competing copies and the conversation history. Publishing the correct page does not prove that the bot imported it or stopped using an older source. Start with a fresh test chat and save the answer’s source reference before changing the model.
Why the chatbot still quotes the old policy
Treat the wrong answer as an evidence problem first. The bot may be reading an outdated imported passage, another enabled document or an instruction that repeats the retired rule. A missing or inaccessible policy can also leave the answer unsupported, which needs a different investigation from stale content.
Work with one changed sentence rather than the entire knowledge base. Record the approved old wording, the approved new wording and any effective-date or customer-group exception. Without that boundary, a valid historical rule can look like a stale answer when it actually applies to an earlier purchase.
The examples below use a change from a 30-day return window to a 14-day window to illustrate the checks. They are not a recommended returns policy, a real customer transcript or a test of a particular vendor. Use your own approved wording and preserve exceptions when comparing answers.
The edit reached the page. It may not have reached the bot.
A support system can hold a published article and a separate representation used for retrieval. Check the text visible in the bot’s content manager, its enablement and its ingestion status, rather than repeatedly reopening the public page. A current date on the website is useful evidence about publication, but it says little about an imported copy.
In-product articles and crawled pages do not move together
Refresh behavior depends on the connection type. Intercom’s Knowledge FAQs distinguish native content, synced sources and PDFs; the same refresh assumption should not be applied to all three. Identify how your particular policy enters the bot before deciding that waiting will fix it.
Look for the exact sentence in the ingested content after the update. Check whether the correct page is excluded, failed, unavailable to the intended audience or disabled for the agent. A successful source-level refresh is a checkpoint, not proof that every relevant passage was captured.
- Publication: the approved sentence is visible on the intended page.
- Ingestion: the connected content contains that sentence.
- Availability: the bot can use it for the tested audience, language and channel.
- Answer: a fresh conversation returns the rule with its relevant exception.
For crawled websites, inspect the page-level result and preview an answer that requires the edited passage. Intercom’s website-sync documentation warns that successful syncing does not always capture all page content. Its scheduling also has source-specific conditions, so this guide does not prescribe a universal refresh delay.
Find the second copy before you rewrite the prompt
Search for the retired wording wherever the bot has permission to read. Include public pages, uploaded documents, translated articles, snippets and guidance. A saved reply matters if your system explicitly feeds it into answers; its mere presence in the help desk does not prove the bot uses it.
PDF uploads, translations and saved replies
A new file name is not evidence that the old file was replaced. Inspect the actual connected-file list and follow the vendor’s update process, including disabling or removing the older retrieval copy when appropriate. Keep any record required for audit purposes outside the active answer source if it no longer belongs there.
Translations deserve the same policy check as the main-language article. Test the affected language and customer group instead of assuming that one English answer validates every version. If a historical rule remains valid for some customers, state that scope explicitly rather than deleting it indiscriminately.
The example still sitting in the instructions
Instructions can quietly preserve the rule you changed on the page. Search examples, guidance and any prompt text your team can inspect for the old deadline, fee or eligibility sentence. Update an explicit contradiction instead of layering a second instruction over it and hoping the agent chooses correctly.

A conflict does not require an obviously broken source. Gorgias documents knowledge conflicts across guidance, help articles, documents and website content. Compare the competing passages and their scope; do not assume that the newest publication date determines which one the agent will use.
Ask in a new chat, then open the old one
Start a genuinely new test conversation and ask the policy question without supplying the old answer in the question itself. Save the question, answer, source reference and time, using the same audience, language and channel as the affected case. Then repeat the question in the existing conversation and compare what changed.
If the fresh chat gives the current rule and the existing chat repeats the old one, conversation history is a working hypothesis. Differences in audience, permissions, channel or settings can produce the same pattern. Keep those conditions aligned before attributing the mismatch to context carryover.
If both chats still return the old wording, continue the source investigation. If the answers alternate, inspect competing passages and repeat the test with representative paraphrases. One correct response is a sample; it cannot establish that all routes through the policy now work.
Use the observations to choose the next check, rather than turning them into a confidence score. The experience below keeps unfinished checks visible and can print a short handoff for the support owner. It does not connect to your help desk or test a live chatbot.
| Observation | Next check |
|---|---|
| Fresh chat current; existing chat old | Compare history and settings, then review the affected conversation. |
| Both chats old | Inspect imported text, connected copies and instructions. |
| Answers change between tests | Compare competing sources, audience and question wording. |
| Answer has no supporting passage | Inspect source coverage and retrieval evidence; use human handling if unresolved. |
| Both sampled answers current | Record the test scope and check outstanding sources; do not declare every channel fixed. |
Record your observations with Policy Source Check
Policy Source Check
Record the mismatch. Find your next check.
Evidence first · No account neededThis check suggests an investigation step from observations you enter. It cannot inspect your bot, confirm a cause or clear an answer for production.
Interactive controls are not ready. Manual method: compare a fresh chat, inspect the imported source, search connected copies and preserve any customer-facing quote.
Your next check
Choose what you know. Untested and unknown observations stay visible in the result.
Your entries stay in this page’s memory and are sent only to the local report window. Reloading clears them. No tracking is included.

What to remove from active retrieval, and what to keep
Fix a confirmed conflict at its source. Update or disable the retired answer copy using your system’s documented controls, then inspect the ingested replacement and repeat the tests. Preserve a required historical record and any still-valid exception instead of erasing the policy history indiscriminately.
When a current answer depends on an order date, region or customer tier, make that dependency explicit in the authoritative content. Removing a historical rule without representing its remaining scope can create a new gap. A shorter knowledge base helps only when it still contains the information the support task needs.
- Capture the conflicting passage and the source that contains it.
- Confirm which rule applies with the policy owner.
- Update or disable the obsolete retrieval copy and follow the supported refresh process.
- Retest the edited rule, an exception and the affected existing conversation.
When the bot invented the rule instead
An answer that differs from the current policy is not automatically an invented answer. It may come from an inaccessible source, a different customer scope or connected data that your first check missed. Compare the answer with the cited passage or retrieval trace before deciding there is no supporting source.
If no approved source supports the rule, check coverage, permissions and fallback behavior. The agent should route an unresolved policy question to a human rather than supply a plausible deadline. A citation can also be misleading if the linked page does not support the answer; our explainable AI examples provide useful context for checking an explanation against evidence.
A bot that reads an external service has another place to investigate. Check the returned value and the scope of that request, not just the help article. If your integration uses MCP, the Model Context Protocol guide explains that connection layer; editing static text will not change a conflicting external response.
What this check cannot fix
This workflow cannot resolve a vendor outage, determine what your business must honor or correct a customer’s account. Keep affected policy questions with a human while the answer remains unresolved. If the bot already communicated a material refund, price or eligibility promise, preserve the message and have the appropriate policy owner review the response; seek qualified legal advice if the obligation is disputed.

| Record | Why it helps |
|---|---|
| Approved wording and scope | Distinguishes a retired rule from a valid historical exception. |
| Imported passage and source reference | Shows what the agent could use, rather than only what the website displays. |
| Fresh and existing chat results | Separates observations across sessions without claiming a cause. |
| Time, language, audience and channel | Makes the comparison repeatable. |
| Customer exposure and responsible owner | Keeps investigation separate from follow-up on an already sent quote. |
Frequently asked questions
Why does the chatbot give the old answer right after I click publish?
Publication and ingestion can be separate steps. Check the imported sentence and whether the bot can use that source for the tested audience. Also search competing content before assuming the only problem is a refresh delay.
Does uploading a new file replace the old file?
Do not assume it does. Inspect the connected-file list and follow your vendor’s replacement process. A new file can coexist with an older enabled copy, so retest after updating the active sources.
Why does a new chat pass and the customer’s open chat still fail?
The existing conversation may retain earlier context, but that is only one explanation. Compare audience, channel, language and settings as well as the history. Use a human handoff for an affected customer while investigating.
Can two articles make the bot switch between 14 days and 30 days?
Conflicting enabled sources can lead to inconsistent answers. Check whether the different windows are obsolete copies or valid rules for different customer groups. Clarify the scope and remove the contradiction before retesting.
How do I tell a stale source from an invented policy?
Inspect the cited or retrieved passage. Old wording in that passage supports a stale-source explanation, while an answer with no approved supporting passage needs a coverage and grounding investigation. Absence of a visible citation alone does not prove invention.
Should I keep answering that question by hand while I check?
Use human handling for the affected policy question while its automated answer remains unresolved. Preserve any quote already sent and give the owner the source and test evidence. Resume automation through your normal review process, rather than relying on one correct sample.
Next steps: close the loop on the changed sentence
Choose one rule, inspect its imported copy and compare fresh and existing conversations under the same conditions. Resolve any enabled contradiction, record what remains unknown and give the affected question an owner. If you are still designing the broader support workflow, use how to use AI in customer service for the setup decisions this troubleshooting check assumes are already in place.



