Delivered case study · Conversation quality
Making conversation quality visible at scale.
Exploring how customer-centre supervisors could move from manually reviewing a small sample of calls to finding the conversations that need attention.
and review coverage
Situation
Supervisors were responsible for conversation quality, compliance and coaching.
Reviewing customer conversations was part of the supervisor and customer-centre manager role. They needed to check whether customer experience professionals followed the expected script, complied with regulations and protocols, and received support or training when needed.
The existing workflow supported structured assessments, but every review depended on a person opening a call, listening to it, and recording the result manually.
Problem statement
The process made quality review and improvement difficult to scale.
Because conversations were assessed one at a time, supervisors could review less than 5% of calls in an average week. This limited their view of the wider operation and made issues difficult to track and address through ongoing monitoring.
- Missed compliance issuesRegulatory risk and financial loss could go undetected.
- Limited monitoring coverageThe small sample made recurring issues difficult to track and address.
- Coaching gapsSupport depended on which conversations happened to be selected.
- Hidden patternsTrends across customer experience professionals, teams and conversation types were hard to see.
- Manual classificationTagging added time to an already demanding role.
- Missed commercial opportunitiesRelevant cross-sell or upsell moments could remain unseen.
Role & team
Discovery and structure preparation within a wider initiative.
I worked with the same core team as the wider customer-centre experience, focusing this phase on discovery, framing and structural foundations.
Scope note: later development expanded to external integrations and moved to another team.
Current scenario
How it used to work.
A supervisor searched historical reporting for one conversation, opened the call, selected a questionnaire and rated each question. Comments could be added before the assessment was saved.
UI structure
Keep the conversation visible while the assessment stays focused.
The screen kept the call context and assessment task together. Select a view to explore how the structure supported the review.
A focused review surface.
The timeline, conversation context and questionnaire stay together so the supervisor can assess the call without losing orientation.
Script
Extend the review pattern across products.
The same assessment structure can support historical review, live coaching and product-specific integrations. Coverage can move from less than 5% of calls reviewed manually to up to 100%* through connected criteria and automation.
*Coverage is configurable by an administrator or developer, depending on the product setup. Research indicated that admins may set specific queues to 100% coverage and others to 0%, based on contract scope, legal requirements and cost.
Adapt the conversation context.
Voice, chat and digital channels can use the same review structure.
Reuse the language.
Supervisors, managers and quality teams share a consistent pattern.
Keep the model extensible.
External providers can supply signals without changing the assessment frame.
Small, manual sample.
Open one call → listen → tag manually → rate each question.
Connected review at scale.
Keep context visible → apply connected criteria → record structured evidence → coach with context.
Keep the call visible.
Caller, queue and timeline anchor the supervisor in the selected conversation.
Make the criteria explicit.
The questionnaire selector sets the script or criteria being applied.
Record consistent evidence.
Shared rating controls support quick review, with comments available when context is needed.
Open one call → listen → tag manually → rate each question.
Keep context visible → apply a questionnaire → record structured evidence → coach with context.
Move from manual setup to connected criteria.
Previously, questionnaires were uploaded manually, one by one or in batches. The new direction supports prompting or API-based access to existing questionnaires, depending on the product.
Previous
Questions were created and maintained manually, making small changes slow to repeat across questionnaires.
New experience
Prompt for a language operator or connect an existing one through an API, depending on the product.
Add language operators
Select reusable language operators to build the criteria for this assessment.
Conversation details
Analysed on Aug 7, 2024 at 17:14 BST Voice
Assistant
00:00:00IntroductionThank you for calling the enrolment centre. I’m a virtual assistant. How can I help today?
Alex Morgan
00:01:32IntroductionI’m looking for help with my account.
Assistant
00:02:00I’m happy to help with that. I’ll ask a few questions first.
Alex Morgan
00:03:10PlanI’d like to understand the available options.
Support live coaching as well as historical review.
Supervisors can observe an active conversation while a customer experience professional is working, or return to historical reporting for a completed assessment. The same signals and user context carry across both views.
Adapt the conversation context.
Voice, chat and digital channels can use the same review structure.
Reuse the language.
Supervisors, managers and quality teams share a consistent pattern.
Keep the model extensible.
External providers can supply signals without changing the assessment frame.
+44 20 7946 0958Alex MorganChristina SmithConcept explorations & prototype validation
Make the change testable before it becomes expensive.
I used lightweight concepts and working prototypes to explore how questionnaire setup, question reuse and conversation evidence could fit together. Each iteration focused on a decision the team needed to make before moving into development.
Keep the review anchored.
Tested how the conversation, questionnaire and assessment actions could share one focused surface.
What to test → orientation while reviewingMake criteria reusable.
Explored a prompt or API-led path for adding language operators and assembling product-specific questionnaires.
What to test → setup effort and flexibilityShow evidence beside the call.
Rendered operator results next to the transcript so a supervisor could connect a signal with the conversation context.
What to test → confidence in the evidenceProtect question history.
Used create, duplicate, rename and edit states to explore how teams could evolve questions without losing meaning.
What to test → safe maintenance over timeDesign opportunity
Move from isolated assessments to scalable quality insight.
How might we help supervisors understand which conversations need attention, identify patterns across a larger sample, and focus human review where it has the greatest impact?
Open one call → choose a questionnaire → listen → rate every question → add comments.
Surface meaningful signals → prioritise review outliers → understand patterns → identify root causes and coach with context.
Next questions
The next draft needs the solution story.
This first version establishes the current workflow and the scale problem. The next iteration will add the design direction, research evidence, collaboration model and outcome once those details are confirmed.
- What signals or classifications should be automated or suggested?
- Prioritisation was explored elsewhere.A separate initiative addressed how supervisors could prioritise conversations for review. Details are omitted here to protect privacy.
- What views would help teams compare patterns across customer experience professionals or queues?
- What changed after the new experience was introduced?
Takeaway
Make quality visible before it becomes a problem.
The opportunity is to move from a small, manually selected sample towards structured signals that help teams see patterns, focus review and coach with context.
Surface meaningful signals→Prioritise review outliers→Understand patterns→Coach with context