The 2019 Org Chart Strikes Back: A Copilot Hallucination Story
An anonymised account of what happens when Copilot grounds in a stale tenant — and what the post-mortem revealed about the cleanup we should have done first.
We’ve scanned a lot of Microsoft 365 tenants.
Every single one has at least three of these five patterns running.
The fifth, almost universally. When you stack five on top of each other and give Copilot the run of the tenant, the results stop being “AI assistance” and start being “AI-mediated misinformation.”
Almost every governable policy in a mature organisation exists in two or more versions inside SharePoint.
The old PDF from 2020 that someone uploaded to a department site for reference.
The updated PDF from 2023 that lives in the Policies hub. Both are searchable. Copilot will surface whichever scores higher on relevance, which is usually the one in the more-trafficked site — which is often the old one. The fix isn’t policy authoring; it’s identifying and removing the duplicates that shouldn’t be searchable.
Org charts get updated annually if you’re lucky, never if you’re typical.
Copilot will happily tell new hires that Sarah reports to Daniel, even though Sarah moved teams 14 months ago. Anything that asks “who handles X” or “who should I escalate Y to” gets answered with a snapshot from whenever the last org chart was uploaded.
We’ve seen Copilot recommend escalation paths to people who left the company two years prior.
If your company has ever acquired or been acquired by another, you have content from that company sitting in your tenant.
It came in as a migration, it never got integrated, and nobody owns it now.
Copilot can read it. Copilot will quote from it.
Your support team’s answers will reference products that haven’t existed for years, processes from the other side of an integration that was completed in 2022, contact information for departments that don’t exist on your org chart.
Teams recordings have transcripts. Transcripts are indexed.
When someone asks Copilot “what did we decide about X,” the answer often comes from a meeting recording — sometimes a recent one, sometimes a 2021 one.
Decisions that were reversed, opinions that were not the final position, and brainstorm content that was never adopted all read as “what the company decided” if Copilot finds them and presents them as a quote.
A single business document — a proposal template, a contract, a deck — might exist in 30 versions across the tenant. v1, v2, v3-FINAL, v3-FINAL-2, v3-FINAL-FINAL, draft-mark-edits, draft-after-legal, and on. Copilot doesn’t know which is the live one.
It looks at the metadata, the filename, the last-modified date, the link count — and makes a guess. Sometimes it’s right. Often it’s not.
When it’s wrong, Copilot tells a colleague to use the third-draft pricing in a deal that closed at the seventh draft.
None of these are Copilot bugs. Copilot is doing exactly what it’s supposed to do: find the best content match in the tenant for the question being asked.
The bug is in the tenant — it has too many sources of truth, too few definitive ones, and nobody maintaining the difference.
Each of the five patterns is detectable.
Duplicate policies show up in content hashing. Stale org charts are timestamped.
Acquired-company wikis are usually whole sites with a recognisable URL pattern and zero recent activity.
Meeting recordings have a retention policy you’ve never set. Versioning hell shows up as files with “FINAL” in the name modified after the file with “FINAL-FINAL” in the name.
A storage scan that surfaces these categories is the same scan that improves Copilot’s accuracy. Two birds, one bill.
An anonymised account of what happens when Copilot grounds in a stale tenant — and what the post-mortem revealed about the cleanup we should have done first.
Microsoft 365 Copilot grounds every answer in your tenant’s own content. When that content is stale, contradictory, or duplicated, Copilot doesn’t shrug — it answers with confidence. Here’s why that’s a problem and what to do about it.
Most Copilot governance programs focus on access controls, sensitivity labels, and prompt filters. They overlook the most consequential variable: the quality of the underlying content. Here’s why content hygiene is the foundation, and what to do this quarter.
opilot grounds its answers in your tenant’s content — and the content is full of contradictions, duplicates, and zombie data. Here are the five patterns we see in every scan that turn Copilot into a confident misinformer.
That extra TB Microsoft is about to sell you is probably already in your tenant. You just can’t see it.
A 143-user enterprise tenant cut $18,000 off its annual Microsoft bill in 90 days — without touching live content. Here’s the playbook.