The Assistant Had to Be Right

The question was whether an answer could be trusted

During a member call, the question was usually ordinary: what changed in the plan this year, does the plan cover a particular benefit, which document explains the cost, or where does a staff member find the exact language? The answer was rarely ordinary to retrieve. It could be buried in a long annual PDF, in a different plan's version of the same document, or in a year-over-year change that looked small until someone gave the wrong explanation.

By the time I had worked through roughly 10,000 Medicare Stars outreach calls and several thousand pharmacy-formulary update calls, the pattern was familiar: people needed plan information while the work was live, and the documents already existed. The effort lay in finding the right passage for the right plan and year, then being able to defend it when the conversation moved on.

Those calls are discovery evidence, not assistant-user totals or model-training data. They showed me which questions kept returning and where lookup friction entered the conversation.

The assistant mattered only when its answers could be trusted, and that trust came from keeping every answer tied to plan information we had already checked.

A PDF is a member deliverable, not a knowledge model

Evidence of Coverage (EOC), Annual Notice of Change (ANOC), Medicare Summary of Benefits (SB), and the applicable ACA benefit-summary documents have an important public job. They explain coverage, costs, rights, and changes to members and prospective members. CMS's Medicare Communications and Marketing Guidelines describes the standardized and model-material context around the Medicare documents.

Internally, those PDFs were being asked to do something else as well: act like a searchable data model. They were large, dense, and spread across annual document sets. A file could have a plan name, a product variation, a market, a document type, and several places where a benefit was described. The document was necessary, but it was a poor operating home for the question a staff member had to answer in the next minute.

Retrieval therefore became an accuracy problem before it became a speed problem, because a fluent answer from the wrong year or product would be more dangerous than a slower answer that sent someone back to the source.

The source set had a boundary

The work sat inside Jefferson Health Plans' 2024–2025 plan cycle during my tenure from January 2024 through approximately May or June 2025, although the project did not take that whole period to build. I worked under a skunkworks-style remit with broad technical autonomy, reported to the Vice President of Medicare Stars and the Chief Product Officer, and carried the work as a Green Belt project supported through Jefferson Health Plans' Project Management Office and Villanova training. Those were the conditions under which I worked; the analysis here is my own.

The prepared corpus contained more than 5,000 pages of EOC, ANOC, Medicare Summary of Benefits, and applicable ACA benefit-summary material. The Medicare documents carried useful formulary context—such as which formulary applied, drug-benefit classes, deductibles, and summarized changes—but the pipeline did not ingest or represent a complete drug-level formulary listing thousands of medications.

The plan-material boundary included Jefferson Health Plans Medicare Advantage products, including D-SNP variants, and Jefferson Health Plans ACA Individual and Family Plans. The standalone Health Partners Plans Medicaid and CHIP lines were outside this system boundary. Those were the contemporaneous 2024–2025 product terms; current rebranding should not be projected backward into this account.

The source documents arrived downstream of the regulated annual plan cycle. That upstream process mattered, but it did not certify the searchable pages, the conversion, or GPT-4o. My work began after final plan materials were available and concerned whether the information survived its move into a form people and software could inspect.

Keeping the source intact after conversion

I owned the source structure and the validation path. The PDFs were converted into structured SharePoint and Excel knowledge so the information could be searched by product, plan, year, document, section, and benefit context. SharePoint became a place where a staff member could inspect a section rather than rely on a loose file name; Excel made comparison and review work explicit.

The conversion had to earn its place, so Python compared character content, counts, and positional structure for every converted item, with each discrepancy receiving manual review. I then visually compared every page across every included line of business with its source PDF, using text parity to find missing or displaced material and page review to see what the counts could not.

That was separate from the actuarial and regulatory work that produced the final plan documents. The final plan material supplied the source authority; a downstream knowledge page could carry that information forward but could not inherit a certification it did not have. My validation work was an internal release check layered on top of that authority.

The design also kept member information out of the assistant's job. No PHI was needed. Staff established the relevant plan and benefit context and asked general plan-information questions; the workflow did not need member names, IDs, dates of birth, claims, medications, or clinical details. That is a property of this plan-material workflow, not a compliance badge for every surrounding call or enterprise system.

Route the question before asking the model

The contemporary GPT-4o model could not reliably infer every branch from a question alone. I built the Copilot Studio worktree to narrow product, plan, eligibility or business-line, year, market, document, and benefit context before retrieval. The routing choices were not decorative prompts. They were how the system reduced the chance that similar plan names or familiar benefit language would be treated as interchangeable.

Microsoft's SharePoint knowledge-source guidance for Copilot Studio explains the platform context; it does not describe this private implementation. Here, citations were mandatory. An answer in Copilot Studio or Teams linked to the exact supporting SharePoint section, and that structured section linked onward to the original PDF in a separate JHP SharePoint source system.

When the assistant could not support an answer, staff moved to the searchable SharePoint layer. With no automatic live-agent handoff, the fallback gave them a direct path back to inspectable source material instead of relying on the model to recognize its own error.

Checked-answer path

Following an answer back to the source

The useful unit was not a fluent response by itself. It was a response with a short, inspectable route back through the context and evidence that supported it.

1. Resolve the context

   The worktree narrowed product, plan, year, market, document type, and benefit topic before retrieval.

2. Route before GPT-4o

   Copilot Studio selected the relevant knowledge path before the model searched the prepared material.

3. Cite the exact section

   The answer linked to the supporting SharePoint section rather than asking confidence to stand in for evidence.

4. Open the original PDF

   The structured section carried an onward link to the final source document for ordinary verification.

5. Fall back to search

   When the answer was unsupported, staff returned to searchable SharePoint instead of an automatic handoff or invented response.

The model shortened retrieval. It did not replace the source.

A shared layer for two operating groups

Teams was the primary interface because that was where the work happened most often. SharePoint and PowerApps provided other access paths into the same operating model. Medicare Stars Outreach used the system during inbound and outbound member conversations about annual changes and current plan information. Product used it to verify, compare, and analyze plan details and to prepare leaders for meetings and decisions.

Product and Medicare Stars staff tested the assistant against questions and edge cases from their work, reviewed what it returned, and fed corrections back into the source and routing work.

The value was shared plan literacy at the speed of live work. A staff member could enter a call or a meeting with a source they could inspect, rather than carrying a personal note or asking someone to remember which PDF had the answer, making the checked source easier to reach and discuss while its authority stayed where it belonged.

What the pilot actually showed

During the pilot, I observed roughly 30–45 seconds saved on a comparable lookup. I did not record a denominator or sample size for a formal time-and-motion study, so I cannot extrapolate that observation to staff hours, financial savings, member outcomes, quality scores, or Star Ratings.

More important than the seconds saved, the project replaced a habit of navigating dense PDFs and personal memory with a collective year-over-year way to check what the plan said, what changed, and where the language came from. Outreach and Product could use the same source structure while asking different questions of it.

SharePoint grounding, citations, and retrieval were already available patterns, so the defensible contribution lay in the work before generation: learning the recurring lookup problem from calls, preparing the source set, checking the conversion, routing the question, and preserving the path back to the original document.

Supporting documents

Inspect the system, then use the method

The architecture shows how the historical system worked. The planning workbook helps teams make their own decisions about sources, routing, review, fallback, and release.

Historical system · 2024–2025

Knowledge-pipeline architecture

Follow final plan materials through structured knowledge, conversion review, worktree routing, citations, source fallback, and actual Outreach and Product use.

Read the architecture →

Planning workbook

Plan-material validation workbook

Settle vocabulary, source authority, decision rights, permissions, routing keys, review ownership, fallback behavior, and release gates before treating retrieval code as the hard part.

Use the workbook →

The part I would carry forward

I still think the useful move was making the background information good enough that every GPT-4o answer had somewhere honest to point. In healthcare work, speed and fluency attract attention, while the quieter work of accuracy lives in the source register, the conversion check, the routing choice, the citation, and the moment when a person searches the source instead of filling a gap with confidence.

The assistant was usable because the plan information underneath it had earned trust first.