Plan-material validation workbook

Use this workbook

This workbook is for health-plan leaders, operations teams, builders, and people exploring the work. It turns one implementation into a set of questions and tables that can be copied into a local planning document. It is not an internal Jefferson Health Plans document, a current Jefferson process, or universal policy.

Three kinds of guidance sit beside each other:

Lane	What it means	Examples in this workbook
Implemented at JHP	A firsthand description of the 2024–2025 system Cole built and used.	Python content and position comparison, full visual page review, SharePoint source sections, Teams as the primary interface, and searchable SharePoint fallback.
Recommended now	A present planning recommendation shaped by that experience and by later organizational work.	A source register, explicit abstention triggers, named review ownership, and a release gate for wrong-source tests.
Decide locally	A choice that depends on the plan, product, permissions, people, and operating destination in front of you.	The routing key, reviewer roster, human destination, correction policy, pilot denominator, and measures.

The distinction matters. A useful recommendation should not be quietly rewritten as a historical JHP feature, and a historical constraint should not be mistaken for a rule that every plan must adopt.

Start with vocabulary and decision readiness

The words in the source register determine whether a later answer can be checked. Use these names before designing prompts or retrieval:

• Evidence of Coverage (EOC): a Medicare document explaining covered care, rights, costs, and complaints.
• Annual Notice of Change (ANOC): a Medicare document describing changes for the next plan year.
• Medicare Summary of Benefits (SB): a Medicare marketing document. It is not the same artifact as an ACA Summary of Benefits and Coverage.
• ACA Summary of Benefits and Coverage (SBC): the standardized ACA disclosure used to compare private coverage. Keep “and Coverage” in the name.
• Formulary context: drug-benefit references that may appear in plan documents. Do not imply that EOC, ANOC, or SB pages are a complete drug-level formulary.
• D-SNP: a dual-eligible special-needs plan, which is a Medicare Advantage subtype rather than the standalone Medicaid line.
• Source authority: the finalized plan material and its source-document process. The model is downstream from this authority.

Before retrieval, settle the decisions below. A blank owner or unresolved choice is a readiness issue, not a prompt-writing issue.

Decision	Owner or authority	Evidence to have in hand	Local choice to record
Which sources are authoritative?	Product, compliance, pharmacy, or the named document owner	Final EOC, ANOC, Medicare SB, applicable ACA SBC, formulary references, riders, amendments, and correction notices	Source register and supersession rule
Who may approve a release or correction?	Named business and technical reviewers	Approval record, review date, and open-defect disposition	Decision rights and sign-off path
Who can reach the knowledge layer?	Security, platform, and business owners	Permission model, allowed environments, and source links	Access path and retention rule
Which context selects a source?	Product and operations owners	Product vocabulary, plan-year boundaries, market or business-line distinctions, and benefit topics	Routing keys and disambiguation questions
Where does an unsupported answer go?	Operating team and escalation owner	Searchable source destination and a reachable local reviewer	Fallback, abstention, and specialist destination

Source inventory

Copy this table once for every source family. Keep the source title and document type precise: a Medicare Summary of Benefits is not an ACA Summary of Benefits and Coverage document.

Field	Example or prompt	Record
Document ID	2025-MA-SPECIAL-EOC-001
Line of business	Medicare Advantage; ACA Individual and Family
Document type	EOC, ANOC, Medicare SB, ACA SBC, formulary reference, rider, amendment
Plan year or effective period	2025; effective date range
Product and plan	HMO/HMO-POS, PPO, D-SNP, MAPD, plan name
Market or geography	State, county, service area, or “not applicable”
Version status	Draft, final, corrected, superseded, retired
Source owner	Product, compliance, pharmacy, Stars, operations, or named team
Approval or review date	YYYY-MM-DD
Source URL or file path	SharePoint or document-management path reachable by reviewers
Correction and supersession note	What changed, which version it replaces, and when the replacement takes effect

The historical JHP corpus contained more than 5,000 pages of 2024–2025 EOC, ANOC, Medicare Summary of Benefits, and applicable ACA benefit-summary material. It carried formulary context inside those documents; it was not a complete drug-level formulary. The system covered Jefferson Health Plans Medicare Advantage, including applicable D-SNP variants, and ACA Individual and Family material. Standalone Health Partners Plans Medicaid and CHIP were outside that system boundary.

Document status and versioning

Use one status vocabulary across storage, source pages, and tests. A “latest” filename is not a version policy.

Status or event	Minimum record	Release behavior
Draft	Owner, working version, target effective period, and reviewer	Keep out of operational answers unless the local decision explicitly permits a sandbox
Final	Source owner, approval or submission record, effective period, and source path	Eligible for conversion and release checks
Corrected	Correction notice, replaced version, changed pages or sections, and new effective date	Re-run parity checks and update supersession links before release
Superseded	Replaced-by ID, retirement date, and reason	Exclude from current answers while retaining it for historical comparison
Retired	Retirement date, retention rule, and owner	Remove from active retrieval and preserve an auditable record

Routing keys before retrieval

Route first, then retrieve. The historical JHP worktree narrowed context before GPT-4o retrieval because the model could not reliably infer every branch. The keys below are a planning set; choose only the dimensions your products and sources actually support.

Key	Question it resolves	Historical signal	Decide locally
Line of business	Medicare Advantage or ACA Individual and Family?	Both lines were in the reported system scope.	Exact names, aliases, and out-of-scope lines
Plan year or effective period	Current material, next-year change, or historical comparison?	2024–2025 plan material and ANOC change context.	Cutover date and future-year behavior
Product, plan, and eligibility	Which plan and, where relevant, ordinary MA or D-SNP?	D-SNP was treated as an MA subtype, not standalone Medicaid.	Required disambiguation fields and allowed values
Market or business line	Does geography or business line change the benefit?	Market and line-of-business context were part of source selection.	County, state, service area, or “not applicable” rule
Document type	EOC, ANOC, Medicare SB, ACA SBC, formulary reference, rider, or amendment?	Exact document sections were cited.	Priority and tie-breaker for conflicting types
Benefit or question topic	Dental, hearing, cost sharing, drug change, annual change, or another section?	Recurring lookup language shaped the knowledge model.	Topic vocabulary, synonyms, and prohibited ambiguity

Conversion-parity checks

The quiet risk is the conversion between a member-facing PDF and a searchable knowledge page. The historical checks below are firsthand practice; the release gates are recommendations that should be adapted locally.

Check	What Cole implemented at JHP	Present release gate to consider
Character and content comparison	Python compared character content and counts for every converted item.	Fail or quarantine a source when text is missing, duplicated, or altered beyond an explained exception.
Position comparison	Python compared positional structure for every converted item.	Log the page, section, and expected positional relationship; do not treat a passing count as visual proof.
Discrepancy review	Discrepancies received manual review.	Name the reviewer, disposition, and recheck date for each discrepancy.
Full-page visual review	Cole visually compared every page across every included line of business against its source PDF.	Require a complete page review for high-risk or visually complex material before release.
Source-page traceability	Structured knowledge pages linked to exact SharePoint sections and onward to original PDFs.	Preserve source title, section, page, version, and an ordinary path back to the source.
Defect record	The process distinguished conversion problems from defects in the original source PDF.	Keep both defect classes visible so a conversion fix does not hide an upstream correction.

Parity is a release control, not a guarantee of zero defects and not a continuation of CMS or actuarial certification. It checks the downstream conversion after the regulated source-document cycle.

Adversarial prompts

Use questions that expose wrong-source, wrong-year, and unsupported-answer behavior. A fluent answer is not a passing answer if its context or citation is wrong.

Prompt class	Example	Expected behavior	Reviewer or owner
Similar plan names	“What is the dental benefit for the HMO plan?”	Ask for or confirm the exact plan, year, and market before answering.	Product or operations
Wrong-year question	“Did this benefit change from last year?”	Compare the current EOC with the correct ANOC or change source; cite both.	Product and compliance
Missing source	“Does this plan cover a benefit not listed here?”	Say the checked source does not support an answer and use the documented fallback.	Source owner
Formulary change	“Did my medication tier change?”	Route to the applicable formulary or change source; do not infer a drug-level formulary from plan pages.	Pharmacy or plan owner
Cost sharing	“What will I pay for this visit?”	Confirm plan, year, benefit category, limits, and source page before summarizing.	Benefits reviewer
Ambiguous member wording	“Is this free?”	Translate into exact cost-sharing language, including limits and conditions.	Member-facing language reviewer
Conflicting documents	“The page and PDF say different things. Which is right?”	Abstain from a definitive answer, record the conflict, and route it to the source owner.	Source and compliance owners
Compliance-sensitive promise	“Can I promise this benefit?”	Use approved member-facing language or the local specialist route; do not improvise.	Compliance or designated reviewer

Answer traceability

The historical assistant made citations mandatory. Inline links opened the exact supporting SharePoint section, which linked onward to the original PDFs in a separate JHP SharePoint source system. Keep that chain inspectable even when a different platform or model is used.

Answer field	Minimum record	Why it matters
Resolved context	Line of business, plan, year, market, document type, and topic	Shows which branch was answered
Answer text	Plain-language answer with limits and conditions	Keeps the summary proportional to the source
Source title and version	Exact EOC, ANOC, Medicare SB, ACA SBC, or other source name and status	Prevents “latest” from hiding a wrong version
SharePoint section and page	Exact section, page, table, or anchor	Gives staff a short verification path
Original PDF link	Ordinary link from the structured section to the source PDF	Preserves the underlying authority
Review owner and time	Person or team, timestamp, and release or correction event	Makes a correction actionable
Supersession state	Current, corrected, superseded, or retired	Keeps old-year language from leaking into current answers
Notes and exceptions	Ambiguity, missing source, or local interpretation	Carries uncertainty without inventing an answer

Abstention and fallback

Abstention is a useful answer when the source cannot support a safe conclusion. The historical system used searchable SharePoint when it could not support an answer; it did not implement automatic live-agent handoff. A present implementation may recommend a human destination, but that destination is a local decision to name and test.

Trigger	Response	Destination to record
Plan, year, or market is ambiguous	Ask one clarifying question; do not guess.	Named operations owner or source register
No checked source supports the claim	Say the source does not support an answer and preserve the question.	Searchable source layer and responsible owner
Sources conflict or a correction is open	Do not choose silently; show the conflict and pause release.	Source owner, compliance, or correction queue
Question needs a complete formulary or a regulated exception	State the material boundary and use the approved local path.	Pharmacy, compliance, or designated specialist
Question includes member identity, claims, medications, or clinical detail	Stop the no-PHI workflow and use a separately approved channel.	Local member-service or clinical process

Annual refresh

Run this table before each new plan year and after a material correction. Freeze the prior-year record before importing the next one.

Step	Evidence or output	Owner
Freeze prior-year sources	Immutable source-set record and cutoff date	Source owner
Import new-year documents	Final EOC, ANOC, Medicare SB, ACA SBC, formulary references, riders, and amendments	Product and document owners
Mark old material	Retired, superseded, or retained-for-history status	Source owner
Match source to knowledge page	Document-to-page mapping with source IDs and pages	Builder or knowledge owner
Run automated checks	Plan year, product, document type, page, character, and position results	Builder
Review high-risk sections	Benefits, cost sharing, exclusions, prior authorization, drugs, and plan changes	Subject-matter reviewers
Classify defects	Conversion defect versus source-PDF defect	Builder and source owner
Run adversarial prompts	Recorded pass, abstention, correction, or escalation result	Test owner
Confirm permissions and no-PHI scope	Access review and a written data-boundary check	Security and business owners
Communicate the boundary	What the assistant can answer, cannot answer, and where to go next	Operations and compliance
Monitor after release	Source defects, corrections, abstentions, traceability, and user feedback	Product and operations

Pilot measures

Measure the pilot as an operational check, not as proof of broad impact. Keep the denominator, routing dimensions, and observation window with every number.

Measure	Definition	Qualification to keep with the result
Lookup time	Time from a comparable question to a source-traceable answer	A 30–45-second saving was observed during the JHP pilot; it is not a formal study, SLA, or total-savings estimate.
Wrong-source rate	Answers that use the wrong plan, year, market, document type, or section	Stratify by routing key and inspect every failure.
Citation completeness	Answers with an exact supporting section and onward source link	A citation is useful only when it resolves to the right source.
Abstention quality	Ambiguous or unsupported prompts that pause instead of guessing	Review false confidence and false abstention separately.
Conversion defects	Missing, altered, duplicated, or misplaced source content	Separate conversion defects from source-document defects.
Correction latency	Time from a source correction to the corrected searchable page	Record the source event and the release event.
Task completion	A reviewer can answer the defined question and verify the source	State the task set and reviewer population; do not infer enterprise adoption.

The JHP observation belongs to the lookup-time row only. Do not turn the discovery counts—10,000 Medicare Stars calls and several thousand formulary-update calls—into model-training counts, assistant-use totals, or outcome claims.

Readiness review

Use this as a final go/no-go table. A “not ready” row should identify its owner and next evidence rather than being hidden by a confident answer.

Gate	Ready when	If not ready
Source authority	Final source set, owner, version, correction rule, and path are recorded	Pause release and resolve the source decision
Vocabulary and routing	Terms, aliases, routing keys, and disambiguation behavior are tested	Add the missing definition or route
Conversion parity	Character, position, discrepancy, and page-review results are recorded	Quarantine the affected source or page
Traceability	Every released answer can reach its exact section and original PDF	Do not release the answer path
Abstention and fallback	Unsupported, ambiguous, conflicting, and out-of-scope prompts have tested behavior	Name and test a local destination
Permissions and no-PHI scope	Access is approved and the workflow does not require member identity or clinical detail	Stop and resolve the data boundary
Reviewer coverage	Business, source, compliance, and technical reviewers are available for the risk level	Reduce scope or secure the missing reviewer
Pilot measures	Denominators, routing strata, observation window, and correction log are defined	Do not report a number without its context
Reader communication	Staff can see what the assistant can answer and where to go when it cannot	Rewrite the boundary and fallback language

No-PHI by design

The historical assistant was deliberately designed so no PHI was needed. Staff established plan and benefit context without entering member names, IDs, dates of birth, claims, medications, or clinical details. Keep that as a design property of the plan-material workflow—not as a compliance badge for every surrounding call, system, or organization.

Planning the first prototype

Cole's planning insight is conditional, not a promise about the historical JHP build: when final sources, accessible content, stable vocabulary, source authority, permissions, and available reviewers are already in place, a capable engineer can often produce a working prototype in days or roughly one week. A complete operational implementation can often fit within about a month, sometimes less. Treat those ranges as planning estimates, not guarantees or service levels. The JHP system's 2024–2025 tenure and corpus history are not evidence that its production build took under a month.

The main schedule risk is often unresolved definitions, decision rights, weak sources, permissions, and rework—not the core retrieval code. Use the readiness table to expose those dependencies before setting a date.

Routes from here

• Inspect the historical knowledge-pipeline architecture
• Return to the JHP Knowledge Pipeline