# Medicare Advantage Plan-Material Assistant Validation Workbook

Use this workbook before launching a Medicare Advantage EOC, ANOC, Summary of Benefits, formulary, or plan-material assistant. The goal is to keep the assistant close to approved source documents and prevent the model from becoming the source of truth.

## 1. Source Inventory

| Field | Example | Notes |
| --- | --- | --- |
| Document ID | 2025-HMO-EOC-001 | Stable internal identifier. |
| Document type | EOC, ANOC, Summary of Benefits, formulary, rider, amendment | Use consistent document names. |
| Plan year | 2025 | Required for every source. |
| Product / plan | HMO, PPO, D-SNP, MAPD, plan name | Match internal product naming. |
| Market / geography | PA, NJ, county, service area | Capture if benefits vary by market. |
| Version status | Draft, final, superseded, expired | Only final/approved documents should be operational. |
| Source owner | Product, compliance, Stars, pharmacy, operations | Name the accountable team. |
| Approval date | YYYY-MM-DD | Useful for audit trails. |
| Source URL / file path | SharePoint or document-management link | Must be reachable by reviewers. |

## 2. Metadata Schema

| Field | Required? | Purpose |
| --- | --- | --- |
| plan_year | Yes | Prevents prior-year answers from leaking into current answers. |
| product | Yes | Separates HMO, PPO, D-SNP, MAPD, and related variations. |
| plan_name | Yes | Prevents similar-plan-name retrieval errors. |
| market | If applicable | Needed when benefits vary by geography. |
| document_type | Yes | Distinguishes EOC, ANOC, Summary of Benefits, formulary, riders, and amendments. |
| section_name | Yes | Supports source verification and navigation. |
| source_page | Yes | Lets staff verify answers against the approved document. |
| effective_period | Yes | Clarifies current versus future language. |
| review_owner | Yes | Names the team accountable for correction. |
| review_status | Yes | Draft, validated, approved, retired. |

## 3. Adversarial Test Questions

| Test type | Example question | Expected behavior |
| --- | --- | --- |
| Similar plan names | "What is the dental benefit for the HMO plan?" | Ask for or confirm the exact plan before answering. |
| Prior-year confusion | "Did this benefit change from last year?" | Cite ANOC or change document, not only current EOC language. |
| Missing source | "Does this plan cover a benefit not listed in the documents?" | Say the source does not support an answer and escalate. |
| Drug-change question | "Did my medication tier change?" | Route to the right formulary or medication-change source. |
| Cost-sharing question | "What will I pay for this visit?" | Cite the correct plan, benefit category, and page. |
| Ambiguous member wording | "Is this free?" | Translate into exact benefit/cost-sharing language with caveats. |
| Current versus future | "What changes next year?" | Use ANOC/future-year source, not current-year EOC only. |
| Compliance-sensitive language | "Can I promise this benefit?" | Use approved member-facing language or escalate. |

## 4. Annual Refresh Checklist

1. Freeze the prior-year source set before adding new materials.
2. Import new-year EOC, ANOC, Summary of Benefits, formulary, riders, and amendments.
3. Mark every old source as retired, superseded, or retained.
4. Run document-to-knowledge-page matching.
5. Run automated validation checks for plan year, plan name, document type, and source page.
6. Run human review on high-risk sections: benefits, cost sharing, exclusions, prior authorization, drugs, and plan changes.
7. Test adversarial questions before release.
8. Confirm permissions and no-PHI boundaries.
9. Communicate what the assistant can and cannot answer.
10. Monitor source defects, escalation volume, answer corrections, and user feedback after launch.