The 2019 Org Chart Strikes Back: A Copilot Hallucination Story
A real account of what happens when Copilot hallucination grounds in a stale tenant. See what the post-mortem revealed about the cleanup that should have…
An anonymised account of what happens when Copilot grounds in a stale tenant — and what the post-mortem revealed about the cleanup we should have done first.
This story is real. Names changed; the bug is exactly as it happened.
A regional sales lead at a mid-market software company asks Microsoft 365 Copilot a routine question: “Who’s the new VP of Customer Success?” The VP started a month ago, replaced someone who’d been in the role for four years.
The internal announcement went out, the new VP was added to the executive team page on the intranet, and the company-wide all-hands had been recorded with introductions.
Copilot answers with the name of the previous VP.
Confidently. Cites a 2019 org chart PDF that lived in the HR site.
The sales lead, trusting Copilot, references the previous VP by name in an email to a major customer about an escalation. The customer — who reads tech news — replies: “I thought [previous VP] left to take a role at [competitor]?” Embarrassing but contained.
Two weeks later, a procurement coordinator asks Copilot which approver they need to route a vendor request to. Copilot tells them.
The named approver hasn’t worked at the company for 18 months.
The procurement coordinator emails a stranger asking for approval; the email bounces; the procurement coordinator wastes two days finding the actual approver; the vendor’s invoice gets paid late; the relationship sours.
Two more weeks: a new hire’s manager asks Copilot for the company expense policy.
Copilot returns the 2020 policy, which capped meal reimbursements at $25. The new hire submits a $33 dinner and gets denied.
They escalate. The escalation reaches HR.
HR confirms the policy is now $40. HR also discovers — to their visible surprise — that Copilot is still surfacing the 2020 policy.
The CIO calls a meeting.
The investigation
What they found wasn’t a bug. It was a tenant that had been allowed to drift for six years.
- The 2019 org chart was still in HR’s SharePoint site, in a “Historical” folder. The folder name didn’t matter; Copilot indexes folder contents.
- The 2020 expense policy was in the Policies hub, untouched, alongside the 2024 update. Both were searchable.
- The Customer Success VP transition had been announced via email and an intranet update — but the intranet’s “Leadership” page had a hardcoded image showing the old org chart. The image had alt text. Copilot read the alt text.
- The former approver was still listed as the approval contact in 14 different documents across the procurement site. Each of those documents had been versioned over the years, but the approval contact field had never been updated.
Six issues.
Zero Copilot bugs. All of them content hygiene problems that predated Copilot by years.
What the company did
Within a week of the meeting, they ran a comprehensive storage and content scan.
Output: 1,847 recommendations across stale sites, duplicate documents, files with obsolete references, and orphaned content. They didn’t action all of them — the team did the top 200, which covered the categories most likely to be cited by Copilot.
Three months later: no new Copilot incidents. Storage bill down 19% as a side effect.
The retrospective takeaway
The customer success VP incident, the procurement incident, the expense policy incident — the company had been running for years without Copilot and these issues never surfaced.
People who needed an answer asked another person. They got the right answer.
The wrong answer was sitting in SharePoint the whole time, but nobody read it because nobody had a reason to.
Copilot is the reason. Every question is now an opportunity for the tenant to lie to an employee who has no way to know it’s lying.
This isn’t an argument against Copilot. It’s an argument for treating Copilot rollout as a content-quality milestone, not just a license-purchasing decision.
The tenant has to be inventoried, the obvious offenders archived, the unowned content claimed or deleted.
If that work doesn’t happen before Copilot is enabled, the work happens after, in incidents.
The order doesn’t change the work. It changes whether you do the work calmly or in a meeting called by your CIO.
