Evaluating a Customer Service Workshop

Background and aim

The poster asks whether a customer-service workshop could improve measurable agent language in a live access-center setting. Its aim was to use Automated Quality Monitoring (AQM) results to select a center-wide training opportunity, teach the prioritized behaviors, and evaluate whether those behaviors changed after training.

The intervention and evaluation ran from October 2025 through April 2026. The work was presented during Patient Access Collaborative Annual Symposium poster programming in Columbus on May 14, 2026.

Four-step intervention

The poster describes four linked steps from measurement to practice and reinforcement:

1. Leadership workgroup: Senior leaders reviewed AQM results and selected the key performance indicators (KPIs) with the greatest center-wide opportunity for improvement.
2. Training workgroup: Trainers and high-performing agents translated the selected KPIs into workshop content.
3. Module one, live workshop: A one-hour instructor-led workshop used call clips, live evaluations, and group activities to reinforce the AQM standard.
4. Module two, eLearning: A thirty-minute self-paced eLearning module used exercises and an assessment in which trainees scored call recordings against the AQM standard.

Evaluation design and cohort boundary

This was a matched, quasi-experimental difference-in-differences (DiD) evaluation. Workshop attendees were compared with contemporaneous matched agents who did not receive the workshop during the same pre/post period. The design estimates workshop-associated change; it is not randomized evidence or proof that training alone caused every observed change.

The included analysis required workshop attendees with sufficient pre/post Verint AQM observations and comparison agents with usable matched data. The poster gives no numeric cutoff. Matching used available baseline performance, queue or workgroup context, and evaluation volume. Verint AQM evaluations, workshop attendance, and queue metadata were aggregated to the agent-month for analysis.

Findings and baseline gradient

The primary findings are two-way DiD estimates for Courtesy Phrases and Positive Language. The same analysis tracked ACD as a balancing metric. The poster reports the following values exactly:

Courtesy Phrases showed the largest workshop-associated change, while Positive Language showed a smaller detectable change. The Courtesy Phrases gradient (+6.47 low, +2.98 mid, +1.42 high) is a coaching signal, not a precise heterogeneous causal effect; regression to the mean may contribute.

ACD was not statistically significant. The reported interval crosses zero, so the result is not proof of zero cost and does not rule out a possible duration increase. No ACD point estimate is supplied here.

Scale and completion

The poster reports 576 agents across 29 departments and 96 workshops delivered over six weeks. It reports 100% completion only for the included training cohort; that completion figure is not a claim about every agent or every workshop invitation.

Limitations

• The evaluation was not randomized. The estimates describe workshop-associated change relative to contemporaneous matched agents, not randomized causation.
• Inclusion required sufficient observations, but the poster gives no numeric cutoff.
• The baseline gradient is subject to regression to the mean and should guide coaching rather than be read as proof of a larger causal effect for one group.
• ACD was not statistically significant, but its 95% confidence interval permits a possible duration increase; it is not proof of zero cost.
• AQM behaviors are measured language indicators. They do not establish patient trust, satisfaction, adherence, or clinical outcomes.

Lessons

The operating lesson is to connect measurement to fair training and coaching. AQM is most useful when an explicit behavior measure helps training and operations partners decide what to practice, how to reinforce it, and how to inspect change without hiding uncertainty. Keeping the denominator, comparison, and balancing metric visible makes the result more useful to the people acting on it.

Next steps

More pre-intervention months would strengthen comparison baselines, and longer-term tracking would test whether KPI improvement is sustained. The poster also points toward targeted coaching informed by AQM data and continued review of Positive Language criteria when a performance gap could reflect either a coaching need or an evaluation-method question.

Credit and ownership

Cole, Stewart, and Madden are the poster's three full coauthors. Stewart's and Madden's substantial contributions shaped the overall design, training, messaging, and leadership communication.

• Cole A. Lyons — analyst and researcher: Led the analysis and research/poster work.
• William R. Stewart — Manager of Quality and full coauthor: Contributed to design, quality framing, messaging, and leadership communication.
• Kristi M. Madden — Manager of Training and full coauthor: Contributed to design, training, messaging, and leadership communication.

Training and operations partners created and delivered the workshop and enabled the evaluation. Penn Medicine owns the public historical poster; the analysis and interpretation on this page are mine.

Download the historical raster poster (PDF · raster image, 0.4 MB) or return to the Penn AQM evaluation.