In episode five of the Talent Risk Assessor series on Defensible Decisions, Scott Kelly (Birmingham/Washington, D.C.) and Cam Ellis (Atlanta) are joined by Pete Bell of the firm’s data analytics team to examine performance ratings—one of the foundational pillars of employment decisions from merit to promotion to termination. The speakers walk through how the talent risk assessor goes beyond a company-wide view to flag gender or race-based differences in ratings within a specific manager’s team or job family, and discuss the impact of calibration gaps, unclear criteria, and documentation. The speakers then trace how performance ratings cascade into pay, promotion, reduction-in-force selections, and discipline and termination decisions, underscoring the advantage of investigating a statistical flag under privilege.
Home > Insights & Resources > Podcasts > Talent Risk Assessor, 5: Performance Ratings as the Foundation
Talent Risk Assessor, 5: Performance Ratings as the Foundation
By T. Scott Kelly and Cameron W. Ellis
0:00:00
0:00:00
Transcript
Announcer: Welcome to the Ogletree Deakins podcast, where we provide listeners with brief discussions about important workplace legal issues. Our podcasts are for informational purposes only and should not be construed as legal advice. You can subscribe through your favorite podcast service. Please consider rating this podcast so we can get your feedback and improve our programs. Please enjoy the podcast.
Scott Kelly: Welcome back to Defensible Decisions, the podcast from Ogletree Deakins, where we explore how workforce analytics and employment law expertise combine to help organizations spot workforce trends, manage legal risk, and support EEO. I’m your host, Scott Kelly.
And a quick note before we start, everything in this episode is informational. We’re going to be talking about our Talent Risk Assessor, which is a service and a product that we have that has a dashboard that’s got outputs from different analysis that we’re conducting under privilege. Any trends that we’re seeing or statistical flags on their own shouldn’t establish a legal violation, it’s just a starting point for the kind of inquiry we’re going to walk through today. Over the past four episodes, we’ve followed a multi-state employer as it used the Talent Risk Assessor to examine hiring dispositions and hiring documentations, separations, and talent progression and retention.
Today, we’re going to turn to something that underpins nearly all of those decisions and that’s performance ratings.
Joining me today are Pete Bell from our Data Analytics team. You would have heard from Pete in our very first episode where Pete, Chrissy Blantz, and I really talked about the Talent Risk Assessor, why we decided to come up with it for our clients. And also today, we’re joined by Cam Ellis from our Employment Law team. We’re going to be drawing on Ogletree’s combined bench of a number of data analytic professionals and employment lawyers that have been involved in coming up with the Talent Risk Assessor. And Cam and Pete are two of those that were instrumental in getting us to where we are today working with clients. Welcome, Pete. Welcome, Cam.
Pete Bell: Thanks, Scott. And this is a topic that’s really going to tie together a lot of the organizational patterns we can see by looking at the data.
Cam Ellis: I agree, Scott. I’m excited to be on the podcast with you today. Performance management is one of those areas where a lot of the things we’ve talked about already, the documentation and what have you, will be important here too. But it’s also an area where legal exposure can quietly accumulate if no one is paying attention to distribution patterns, calibration, and that documentation.
Scott Kelly: All right. Before we dive in, let’s connect this to where we’ve been. Most recently in episode four of the Talent Risk Assessor series, we looked at time to promotion and tenure and how progression barriers can drive retention risk. Performance ratings are certainly often part of that story, they’re going to factor into who gets promoted and who doesn’t. Today, we’re looking at that foundation that those ratings are built upon.
Cam Ellis: That’s the right framing, Scott. Performance ratings are an outcome. If they lack consistency or aren’t tied to job related criteria, every downstream decision that relies on those ratings inherents the vulnerability. Systemically, that’s often pay or variable comp. In terms of what might be the most immediate risk, a termination defended on the basis of poor performance only holds up if that rating is itself defensible.
Pete Bell: And what’s great here, Scott, is performance ratings are some of the richest data sets that we have. They should be assigned regularly, often on an annual basis. They should be standardized, which is a point we’ll hit on later in the podcast as well. And they should touch effectively every employee. This makes them pretty ideal for a statistical analysis and gives us the opportunity to check if they’re being applied differently by demographic class such as race or gender. We can look at the manager level or business unit level and see if we look for any patterns and ensure we’re doing the right comparisons.
Scott Kelly: The Talent Risk Assessor really can actually flag when those distributions look anomalous, is that right?
Pete Bell: Yeah, that’s right. And I think where the assessor really sticks out is in the ability to drill down. We don’t necessarily need to see the organization-wide view, we can take a deeper dive and look more at a specific manager’s team, business unit, or job group. This will tell us whether the ratings are distributed differently by race or gender, and that’s the comparison that we really care about legally.
Scott Kelly: Very good. Let’s get into our hypothetical for this episode. It’s the same employer, the same platform. The multi-jurisdiction employer conducts annual performance reviews, we’re going to say on a four-level rating scale, viewed at a high level, company-wide, nothing looks too unusual. But using the Talent Risk Assessor, we’re looking within one business unit and then within a particular manager’s teams inside that unit. And in doing so, we see that it surfaces a flag. Women are receiving the lowest rating tier at a materially higher rate than men in the same job family under the same managers. What are we looking for?
Pete Bell: Yeah, it’s a great question. And so, we can use statistical methods to evaluate how performance ratings are being applied across managers or across business units but the comparison that we’re most interested in is a little narrower than that. And that’s an important distinction. It’s not in itself a legal risk if we have one manager who is maybe a harsher grader and another manager who’s more lenient. That is still important. It’s a calibration question. It’s not something we want to overlook as it could impact other elements of the employee life cycle. But the issue we’re really focused on isolating is a little more specific. Within a specific business unit or within a particular manager’s team, are women receiving lower ratings than men do in that same group, those that are being evaluated by the same manager against the same criteria?
In that hypothetical where women were receiving lower ratings than men, at that level of granularity, that’s something we do need to investigate. And what the stats are going to tell us is, is that different large enough and is it consistent enough that it warrants a closer look?
Scott Kelly: And what does that tell us?
Cam Ellis: It tells us we need to dig deeper, ask more questions. A flag is not a conclusion, it’s an anomaly that warrants legally informed inquiry. There might be several possible explanations. There could be a calibration issue, that’s where managers in the unit are applying standards differently from one another. It could be that the criteria themselves aren’t clearly defined leading to inconsistent application. It could just be a documentation gap where managers aren’t adequately recording the basis for their ratings. It might be budget driven where managers are allocating variable comp and then backfilling appropriate ratings. Before that, there might be an even simpler reason. Right?
Pete Bell: Sure. Let’s go with it this way, the ratings could be in themselves perfectly accurate. Right? They may be genuine performance differences that are being reflected within the job family. And analytics don’t necessarily tell us that. What the analytics do is identify hotspot areas where we want to drill down so that our review of performance is more targeted rather than just guesswork.
Scott Kelly: The system is essentially saying, here’s an area where the pattern is different enough from what you’d expect that you should understand why.
Cam Ellis: That’s right, Scott. And from a legal risk perspective, that proactive identification is valuable. If those ratings later form the basis for an adverse employment decision or decisions, terminations, denial of promotions, comp differences, and no one looked at that distribution pattern or tried to explain it, that could become a significant vulnerability in litigation. Bias might have seeped in somewhere.
Scott Kelly: All right. Let’s break down the key elements of defensible performance management. What does the organization need to get it right?
Cam Ellis: There are four things I like to focus on and hopefully these will be reviewed because a lot of them are best practices across a variety of decision making. First, we need some consistent standards. Are the same performance expectations being applied to everyone in comparable roles? Second, job related criteria, that’s the factors being evaluated, need to be actually tied to the requirements of the position. Third, and this is where things often go wrong, calibration or consistency within the decision-making environment. In other words, is a given manager, business unit, or job group applying standards evenly across employees regardless of race, gender, or another protected characteristic? And fourth, documentation quality. Not to sound like a broken record but we’d like to know if there is sufficient contemporaneous evidence supporting each rating.
Pete Bell: That’s right. And so, there’s two main points I want to hit on what Cam said. To be clear, the analytics can speak to each of those, but what I want to focus on are two main points there. One of the points Cam raised is that manager A may simply be a tougher grader than manager B. That can still be an issue, we want to make sure we have calibration and training for our managers. But it’s not the legal risk signal that we’re focused on. The one we’re more interested on and the one that the statistics will really get to is whether managers A’s own team, their business unit, their job group are rating folks differently by protected characteristic. And it’s the second question is the one that we’re built to isolate. On documentation, we can’t necessarily assess that through the statistics alone but we can identify areas where if the documentation is sparse or absent, and that’s in the area where we’re seeing differences in ratings and what’s driving the flag, well, if we don’t have the documentation there, that’s going to be correlated with areas that are harder to defend.
Scott Kelly: Walk me through that, Pete. In our hypothetical, what does the drill-down really look like once you’re inside a specific manager’s team?
Pete Bell: Sure. Let’s say we identify a statistical flag at the job family level, what we can do is hold the job family constant and look within the manager’s teams that make up that job family. We’re asking whether employees that the manager rates, say women in a given job family, are they consistently landing at the bottom of the tier compared to men in the same job family and under the same manager? When that pattern holds within a manager’s team, even if it’s not across the entire job family or business unit, it points towards something worth investigating at that manager level to ensure that manager is applying the criteria appropriately and isn’t making a gender-based decision. Even though that’s not a company-wide conclusion, it gives us the hotspot area. It tells us where we need to investigate further.
Cam Ellis: Pete, picking up on what you just said with the gender-specific decision, enforcement agencies are today often focused on theories of intentional discrimination. And so how the organization responds matters a great deal from a legal perspective. If an employer’s HR comp team is taking it upon themselves to review legal risks without the protection of attorney-client privilege, that could be problematic. For example, if evidence exists at a manager’s team after reviewing the underlying documentation, talking to the manager, checking whether the criteria were applied in the same way to everyone, and finding out there are anomalies, perhaps not unlawful ones even, that could all be used against an employer. That’s why we’re alerting our clients that without a careful focus on keeping everything privileged, an employer could easily find itself in the territory of no good deed going unpunished.
Scott Kelly: All right. If you work to set up the privilege correctly, and ideally, you’re going to run the analysis we have set up in the performance module of the Talent Risk Assessor, the statistical flag creates a responsibility to act. Is that right, Cam, or do you have a different approach?
Cam Ellis: I’d say I have a very similar approach. I would characterize it as it creates a responsibility to investigate and respond appropriately. The response might be training, it might be documentation showing why the organization is okay with the anomaly. It might be revised criteria. It might be enhanced oversight for a particular manager. The key is that the organization engaged with the issue and is ready to show through documentation that it has engaged with the issue rather than ignoring it and then to also ensure that the organization takes steps to mitigate risk.
Scott Kelly: Okay, thanks. Why don’t we talk about why this matters beyond just the performance review itself? How do ratings cascade through other employment decisions, Cam?
Cam Ellis: How many of our clients say they are pay for performance, that they tie terminations to performance? Performance ratings are upstream of nearly every significant employment action. Pay increases are often tied to performance ratings. Variable compensation or total compensation is often tied to performance. Promotion decisions frequently rely on performance history. When an organization conducts a reduction in force, performance ratings are commonly used as a selection criteria and discipline, and termination decisions are almost always framed in terms of performance deficiencies. If the ratings themselves are flawed, every downstream decision inherits that flaw.
Pete Bell: Right. And this is where we can draw connections to prior episodes. Episode four, I believe focused on time to promotion and tenure. If our promotion decisions are leaning on performance ratings and those ratings aren’t being applied consistently across managers, then our progression patterns should be examined, and we should trace it back to that calibration issue. Earlier I said maybe the legal risk is not focused on if managers are being consistently across, but we do have to think about if our promotional practices are going to be impacted by that calibration. The same is true of separations like we heard in episode three. If our performance ratings are being factored into who is selected and we have calibration gaps in those performance differences, then our selections and the corresponding criteria may be compromised.
Scott Kelly: All right. Well, what about the discipline and termination decisions?
Cam Ellis: There too, Scott. Progressive discipline typically begins with a performance rating or performance conversation. If the initial rating is inconsistent with how others are being rated for similar work, the entire disciplinary chain that follows is built on a questionable foundation. The Talent Risk Assessor’s ability to flag these upstream issues is what makes it valuable, you’re identifying that risk at the source rather than discovering it only when a downstream decision is challenged.
Pete Bell: And exactly as Cam said, we can, using the assessor, trace that pathway. If we see a rating pattern for a specific manager’s team or a specific unit, that could be reflected in the compensation data and promotion rates and separation rates. This gives your organization effectively a complete picture of whether performance management is creating cascading effects.
Scott Kelly: Let’s try to wrap up this episode. I’m going to ask each of you to tell me what you would want listeners to take away from our conversation today.
Pete Bell: Sure, Scott. I would say that the distribution analysis that the talent assessor completes can help us to identify race or gender-based differences in performance at a given level or family or even within a given manager’s team. It gives us the ability to identify hotspots and best identify potential root causes such as a particular manager’s biases that could be driving the issue. These managerial differences have impacts on the entire employee life cycle because it can lead to differences in promotional rates, separation rates, or other aspects of the employee life cycle. It’s important that we’re calibrating those scores across managers as well.
Cam Ellis: And from a legal perspective, performance management is where defensibility either starts or breaks down. If your ratings are consistent, job related, calibrated and documented, everything built on top of them is stronger. If they’re not, everything downstream is exposed. The flag prompts inquiry, maybe it’s training, maybe criteria need to be clarified, whatever the case is, the organization that investigates is in a fundamentally different legal position than the one that doesn’t. As we were talking about earlier, this is informational, not legal advice and dashboard pattern on its own is not a legal violation, but it tells you where to look.
Scott Kelly: Thanks for that reminder. Cam and Pete, thanks for those insights. In our next episode, episode six of this Talent Risk Assessor series, we’re going to take the natural next step and look at comp and pay. Performance ratings certainly drive pay decisions in a lot of clients or employer situations, so we’ll examine what happens when the Talent Risk Assessor analyzes some compensation distributions and identifies potential disparities. That conversation is going to bring everything together that we’ve discussed across this series so far.
Thank you, Pete. And thank you, Cam. I think you’re going to be joining me for this next episode so I’m looking forward to that. And for our listeners, thanks for tuning in and listening to Defensible Decisions. Until next time.
Announcer: Thank you for joining us on the Ogletree Deakins podcast. You can subscribe to our podcast on Apple Podcasts or through your favorite podcast service. Please consider rating and reviewing so that we may continue to provide the content that covers your needs. And remember, the information in this podcast is for informational purposes only and is not to be construed as legal advice.
Speakers
Share Podcast
Practice Group
Workforce Analytics and Compliance
Ogletree Deakins’ Workforce Analytics and Compliance Practice Group provides tailored guidance and legal recommendations for a myriad of workforce issues, informed by data-driven, state-of-the-art compliance and risk assessment services. Our services encompass all stages of the employment life cycle, such as selections, career advancement, compensation and benefits, and retention, which enables employers to make informed decisions […]