Most contact centers can tell you exactly how often they evaluate their agents. Far fewer can tell you how often they evaluate the evaluation.
A QA scorecard gets created. Teams debate the categories, assign weights, define what constitutes a passing score, train evaluators, and build reporting around it. Eventually, the scorecard becomes part of the infrastructure of the contact center.
And then something interesting happens: the scorecard stops being questioned.
Agents continue being evaluated on the same greeting, verification steps, call-control behaviors, empathy statements, closing language, and compliance requirements because those are the things the organization decided mattered.
Maybe they still do. But when was the last time you checked?
Customer Experience Doesn't Stand Still
Think about how much can change while a QA scorecard stays exactly the same.
Products change. Pricing changes. Regulations change. Competitors introduce new offerings. Companies add self-service options and AI agents. Customers become more comfortable solving simple problems themselves, which means the conversations reaching human agents become more complicated.
Even customer expectations around communication evolve. Language that sounded perfectly natural five years ago may sound scripted today. A process that once felt efficient may now feel unnecessarily cumbersome. Customers who have already interacted with a chatbot may expect the human agent to understand what happened before the handoff.
Yet an agent can still be evaluated against a definition of quality created for a very different customer experience.
That's the problem with institutionalizing a scorecard. Eventually, the measurement can become more permanent than the thing it's supposed to measure.
A Scorecard Is a Hypothesis
We tend to treat QA criteria as rules. It may be more useful to think of them as hypotheses.
When you decide that a particular behavior belongs on your scorecard, you're effectively saying: we believe this behavior contributes to a successful customer interaction. That's a testable idea.
Does using the customer's name actually correlate with better outcomes? Does following the prescribed opening make a meaningful difference? Do agents who score highly on empathy behaviors also have customers who escalate less frequently? Which behaviors are associated with successful resolutions, higher conversions, lower customer effort, stronger compliance, or fewer repeat contacts?
Some criteria will absolutely prove their value. Others may turn out to matter less than expected.
And there may be behaviors you aren't measuring at all that matter considerably more.
The Highest QA Score Isn't Necessarily the Best Conversation
This is where QA can become disconnected from the customer.
Imagine Agent A scores 98% because they follow every required step perfectly. They use the approved greeting, follow the script, verify the account correctly, demonstrate the required empathy behavior, and close the call exactly as instructed.
Agent B scores 90%. They occasionally deviate from the prescribed language, but they solve problems quickly, explain complicated issues exceptionally well, and rarely generate repeat contacts.
Which agent delivered the better customer experience?
The answer isn't necessarily Agent A or Agent B. The point is that the score alone can't tell you unless the things you're scoring are actually connected to the outcomes you care about.
A perfectly executed scorecard can still measure the wrong things.
100% Conversation Analysis Changes the Question
Historically, revisiting a scorecard was difficult because organizations had limited information.
QA teams listened to a small sample of calls. Leaders reviewed metrics and escalations. Customer feedback might come from surveys representing another small subset of interactions. Updating the scorecard often required workshops, calibration sessions, and a significant amount of judgment.
That isn't bad methodology. It was often the best methodology available.
But analyzing conversations at scale gives organizations another source of evidence.
Instead of asking only, "What do we think agents should be doing?" you can also ask, "What actually happens in conversations that produce the outcomes we want?"
Patterns across thousands of interactions can reveal which behaviors appear consistently in successful conversations, where customers struggle, which objections are emerging, what drives escalations, where compliance risk occurs, and how those patterns change over time.
Your QA program can start learning from the conversations it evaluates.
Some Behaviors Become More Important. Others Become Less Important.
Consider what happens when the nature of customer contacts changes.
If self-service successfully handles more simple transactions, human agents may increasingly receive complicated, emotionally charged, or unusual interactions. In that environment, rigid script adherence might become less valuable while problem solving, judgment, and the ability to explain complexity become more important.
If a new regulation introduces significant risk, a behavior that barely mattered six months ago may suddenly deserve substantial weight.
If customers repeatedly express confusion about a new product feature, the ability to explain that feature clearly may become an important quality indicator.
If an organization introduces an AI agent before the human handoff, context retention may become critical to the experience.
Quality isn't fixed because the environment in which quality is delivered isn't fixed.
Your Scorecard Should Be Able to Learn
This doesn't mean rebuilding your QA program every month.
It means creating a mechanism for questioning it.
Conversation intelligence can help organizations identify emerging patterns that deserve investigation. Maybe agents with stronger outcomes consistently do something that isn't currently scored. Maybe a heavily weighted behavior shows little relationship to the outcomes the organization cares about. Maybe a new customer issue is appearing frequently enough that the scorecard should account for it.
Those findings shouldn't automatically rewrite the scorecard. Human judgment still matters. Compliance requirements aren't optional simply because they don't correlate with customer satisfaction, and correlation alone doesn't establish that a behavior caused an outcome.
But the data can tell you where to look.
Instead of waiting for an annual QA redesign, teams can continuously examine whether their definition of quality still reflects what is happening in the real world.
Compliance and Customer Experience Aren't the Same Thing
There's also an important distinction between behaviors that must happen and behaviors that improve the experience.
Certain authentication steps, disclosures, consent requirements, or regulatory language may be mandatory regardless of how customers respond to them. Those belong in QA because the organization has an obligation to ensure they happen.
But not every quality criterion falls into that category.
When mandatory requirements and experiential behaviors all live on the same scorecard, it can become easy to treat them as equally permanent. They aren't.
One group asks, "Did we do what we are required to do?"
The other asks, "Did we do what creates a good outcome for this customer?"
The first may change when policies or regulations change. The second should evolve as customers do.
The Scorecard Can Become a Feedback Loop
At MosaicVoice, we believe automated QA becomes much more powerful when it isn't simply automating an old manual process.
Evaluating 100% of conversations is valuable. But the larger opportunity is using that information to understand what quality actually looks like across the operation.
You identify behaviors that appear to drive better outcomes. You incorporate them into QA, coaching, knowledge, or real-time guidance. You observe what happens. You refine the behavior. And as customers, products, and risks change, you continue looking for the next signal.
The scorecard becomes part of a learning system rather than a static checklist.
That's a very different role for QA.
Ask a Different Question at Your Next Calibration
Most QA calibration sessions focus on consistency. E.g. Would two evaluators score this interaction the same way? Are we interpreting the criterion correctly? Did the agent satisfy the requirement?
Those are important questions. But there's another one worth adding:
Why are we measuring this?
If the answer is simply, "because it's on the scorecard," that's worth investigating.
What outcome is the behavior intended to influence? Do we have evidence that it does? Has anything changed since we decided it mattered? Is there something happening in today's conversations that deserves more attention?
Calibration shouldn't only make sure everyone is applying the scorecard consistently.
It should occasionally make sure the scorecard is still right.
The Bottom Line
Your QA scorecard is a definition of quality.
And definitions need to evolve.
The behaviors that mattered three years ago may still matter today. Some may matter even more. Others may have become less relevant, while entirely new behaviors have emerged as products, technology, regulations, and customer expectations changed.
The goal isn't to change your scorecard for the sake of changing it. It's to make sure you're not confusing consistency of measurement with accuracy of measurement.
Conversation intelligence gives contact centers an opportunity they haven't historically had: to continuously compare what they believe creates a great interaction with what is actually happening across thousands of real customer conversations.
Your agents shouldn't be measured against yesterday's customer experience.
Your definition of quality should be learning as quickly as your customers are changing.