Penn AQM Customer Service Workshop Evaluation

The question was whether training changed an observable habit

In a live access center, leadership has to hold two ideas at once: a call should move efficiently, and the patient should hear clear acknowledgement while it does. Penn Medicine's Patient Engagement Center had Automated Quality Monitoring (AQM) measures that made a customer-service opportunity visible, so the operating question asked whether a workshop changed the measured language beyond showing that people attended it.

The project and evaluation ran from October 2025 through April 2026. It became Evaluating a Customer Service Workshop: A Difference-in-Differences Analysis of Contact Center Agent Performance, presented during Patient Access Collaborative (PAC) Annual Symposium poster programming in Columbus on May 14, 2026.

From a measured opportunity to a fair comparison

Training and operations partners selected Courtesy Phrases and Positive Language as the primary AQM behaviors. The workshop used examples, practice, and reinforcement around the small words that make acknowledgement audible in a call: “Can I please have your name and date of birth?” and “Thank you” after a patient answers. This is a representative acknowledgement scene, not a quoted transcript.

The evaluation included workshop attendees with sufficient pre/post Verint AQM observations and contemporaneous comparison agents with usable data matched on available baseline and operating context. The poster gives no numeric observation threshold. The design was a matched, quasi-experimental difference-in-differences (DiD) estimate: it compares change across the same period; it is not randomized evidence and does not prove that training alone caused every observed change.

Evidence folio

The symposium poster

The Penn Medicine poster describes the workshop, matched quasi-difference-in-differences design, language estimates, balancing result, and limitations. A text transcript accompanies the original raster PDF.

Inspect full artifact →

Evidence from the field

The symposium poster

Close

Download the historical raster poster (PDF, 0.4 MB).

The second group photograph shows the wider Penn team at the symposium in Columbus.

What the estimates say

The two primary language measures moved upward relative to the matched comparison agents. The estimates below are reported exactly in points per 100, with their confidence intervals and p-values.

Difference-in-differences estimates

Measured service language changed after the workshop

Data table

Difference-in-differences estimates, points per 100. Source: Patient Access Collaborative poster.

Measure	Reported estimate	95% CI	p-value
Courtesy phrases	+3.65	+2.97 to +4.33	< 0.001
Positive language	+1.35	+0.88 to +1.81	< 0.001
Average call duration	not significant	−0.25 to +5.79	0.072

Courtesy Phrases rose +3.65 points per 100 (95% CI +2.97 to +4.33, p < 0.001). Positive Language rose +1.35 points per 100 (95% CI +0.88 to +1.81, p < 0.001). These are workshop-associated estimates from a quasi-DiD comparison, not randomized proof.

Average Call Duration (ACD) was the balancing metric, with a 95% CI of −0.25 to +5.79 seconds and p = 0.072; the poster does not provide an ACD point estimate. Because the interval crosses zero, this nonsignificant result leaves the time cost unresolved and cannot support a zero-cost claim.

The opportunity gradient is a coaching signal

Courtesy Phrases showed a descending baseline gradient: +6.47 points per 100 for low-baseline agents, +2.98 for mid-baseline agents, and +1.42 for high-baseline agents.

Opportunity gradient

The lift was largest where the baseline was lowest

Data table

Courtesy-phrase improvement, points per 100, by baseline tercile.

Baseline tercile	Courtesy-phrase gain
Low baseline	+6.47
Mid baseline	+2.98
High baseline	+1.42

The gradient helps a training team decide where coaching may be most useful. It should not be read as a precisely larger causal effect for one group: regression to the mean may contribute, even with a contemporaneous matched comparison.

Scale, completion, and credit

The poster reports 576 agents across 29 departments, 96 workshops, and six weeks of delivery. The 100% completion figure refers only to the included training cohort; it is not participation by every agent, every invitation, or the entire Patient Engagement Center.

Cole A. Lyons was the analyst and researcher responsible for the analysis and research/poster work, while William R. Stewart, Manager of Quality, and Kristi M. Madden, Manager of Training, are full coauthors whose substantial contributions shaped the overall design, training, messaging, and leadership communication. Training and operations partners created and delivered the workshop and enabled the evaluation, and Penn Medicine owns the public historical poster.

The boundary of the evidence

Under a matched, quasi-DiD design, the workshop was associated with higher measured Courtesy Phrases and Positive Language. Those language indicators support a connection between observable behavior and fair training, coaching, and operational learning, with the denominator, comparison, and uncertainty kept in view; they leave patient trust, satisfaction, adherence, clinical outcomes, and any general effect of automated monitoring unmeasured.

Notes

The AHRQ CAHPS measures PDF and AHRQ patient-experience overview describe helpful, courteous, and respectful staff as dimensions of patient experience. This evaluation measured language, not patient-reported experience.

Hewitt and colleagues' 2009 naturalistic study, Front desk talk, distinguishes task-centred interaction from polite language and rapport. It is context for the acknowledgement scene, not evidence that these phrases alone cause trust or satisfaction.

The 2024 JAMA research synthesis on word choice supports a broader role for wording in empathy, competence, trust, and encounter outcomes without isolating “please” or “thank you” as sufficient causes.

The official 2026 PAC symposium agenda records the May 14 poster programming. O'Neill and colleagues' controlled pre/post methods comparison explains the assumptions and comparison choices required for difference-in-differences.