What Are Performance Reviews Really About? 'Enough about you. Let's talk about me.'

Ever get the feeling that, judging by what came up in your performance review, your boss must have been talking about someone else? If so, you're not crazy.

It's entirely possible that how you're being evaluated has less to do with you than with the person on the other side of the desk. At least, that's the conclusion of an intriguing study led by Steven Scullen, a professor of management at Drake University. A team of researchers spent months analyzing performance ratings of about 5,000 managers from a wide range of industries. Each was evaluated by six colleagues of different ranks, including bosses, peers, and direct reports.

Not only did the six raters each give different ratings for almost every manager, the study found, but "65% of the variance in ratings was attributable to personal biases and idiosyncrasies of individual raters," while only 25% could be traced back to how well each manager actually did his or her job.

"For example, one person might give you low marks on strategic planning," says Scullen. "But that doesn't mean a different one would share the same view. Much of the variation comes from raters' different opinions of the importance of strategic planning, or their own skills in strategic planning, or other personal factors that have nothing to do with the work you performed."

Bosses' ratings were somewhat more consistent than either peers' or subordinates', the researchers found, and thus probably more reliable. Still, even bosses' ratings of the same person's performance varied by as much as 51%, versus anywhere from 62% to 71% in the ratings that came from peers and subordinates.

Companies could make reviews more accurate, Scullen says, by having more than one person do them, and then averaging the results. "It also helps to get independent views, rather than a panel of evaluators where there is one dominant voice and everyone else just goes along," he adds. Even better, he suggests trying "as much as possible to come up with a truly objective, quantitative measure" — easier for some jobs (sales, for instance) than for others — so that individual raters' biases carry minimal weight.

"But employers need to spend more time clarifying our definitions of what we want from each role, what success looks like, and then communicating that," Scullen says. "It should be an ongoing discussion with the person who's being rated."

Deloitte apparently agrees. Based partly on Scullen's study, the global consulting and audit giant recently announced it is revamping its entire performance-appraisal system, scrapping annual reviews and 360-degree feedback altogether.