A calibration meeting, third Tuesday of the quarter. Two shift managers have rated the same team leader on “communicates effectively”. One gave a 4, one gave a 2. Neither is being lazy. Both are certain, and both can point at the framework to defend the number.
They are both right, which is the problem. The anchor under that competency does not describe anything either manager could have watched happen. It describes a verdict. Each reached it from different evidence and neither can be shown to be wrong, because the standard was never a standard. It was an opinion with a rating scale attached.
This is where competency work fails. Not in the list of competencies, which is usually fine, and not in the number of levels. It fails in the sentence under each level, the only part a manager touches on a Thursday afternoon, written last, in a hurry, by someone who will never use it.
Why anchors drift into attitudes
Anchor sets are usually drafted from the competency name outward. Someone takes “collaboration”, thinks about what a good collaborator is like, and writes adjectives: works well across teams, is trusted by peers, handles conflict maturely. These read fluently and are unusable, because each one asks the rater to run a second judgement first. What counts as “well”? Trusted by whom?
There is a second cause. Anchors are written to be defensible in an audit sense rather than usable in a live sense. Language that cannot be pinned down cannot be challenged, so it survives review. “Demonstrates strong ownership” will never be proved false. It will also never be proved true, which means it cannot support a promotion, a development plan or a hiring call.
The consequence shows up in the data. When researchers tested a behaviourally anchored scale on medical students in simulated emergencies, agreement between trained raters on the same recorded performance ranged from 0.26 to 0.53 depending on the component, with communication the weakest. The same raters re-rating their own earlier scores sat between 0.47 and 0.70. Two experts watched the same video and landed on different numbers, and the anchors were the standard tool for the job. Published figures like that should make anyone cautious about treating a single manager’s rating as evidence.
The test that separates an anchor from an opinion
An anchor is only useful if a second person, watching the same ten minutes of work, marks the same box. That is the whole test.
Run four questions against every line. Fail any and it is not an anchor yet.
Can it be seen or heard? “Gives the customer the next step before ending the call” is visible. “Is customer-centric” is not. If the sentence names a state of mind, a value or a reputation, it is a conclusion and belongs in the competency definition, not in the anchor.
Does it name the situation? Behaviour is conditional. A person who escalates every issue is not demonstrating judgement, and one who escalates nothing is not demonstrating ownership. The anchor has to say where: in a disagreement with a peer, at handover, when a deadline slips, in the first two minutes of a call. Without a setting the same action can be read as strength or weakness.
Is there one action, not three? Anchors that bundle three behaviours into one line cannot be rated at the edge, because the ratee will do two of the three. Split them or cut two.
Does the level above differ in scope, not in adjectives? The common failure is a ladder built from adverbs: does it sometimes, usually, consistently, always. That is frequency, not capability, and it rewards tenure. The level difference should be a change in what the person is responsible for: does the task, then makes the task work for others, then decides whether the task should exist.
A field check. Read the anchor to a manager from another function and ask what they expect to see. A clarifying question means a hole. If they describe a scene back, it is close.
Writing the set in the order the work happens
Most teams write the top level first because it is the aspirational language. Reverse it.
Start with the meets-expectations level and write only that. Describe a competent person doing the job on an ordinary Tuesday. This level carries the most weight in practice, because it is where most ratings land and where the argument about “is this good enough” actually happens. Public sector anchor sets tend to define this level as consistently meeting the agreed standard, and then treat anything above it as requiring documented evidence. That is the right shape: the middle is the default, and exceptions need proof.
Collect incidents before writing words. Ask six managers for two real examples each, done well and done badly, in their own words, from the last quarter. Twelve incidents show which situations recur, and the anchor is written from those.
Write the level below as a specific omission. Not “requires support” but “hands the issue to the supervisor without proposing an option”. Omission is easier to observe than weakness, and it gives the manager something to coach rather than something to judge.
Write the level above as a change of scope. Not “does it more consistently” but “sets the pattern other people follow, and covers for it when the person is on leave”. Two components: influence and durability. Both observable.
Keep four levels at most, ideally three. Every level you add multiplies the number of boundaries two raters can disagree on. A four point scale with clear descriptors produces better agreement than a seven point scale with vague ones, and it takes a tenth of the time to write.
Three failures, rewritten
The pattern in the left column is what most draft anchors look like. The right column is the same intent made observable.
| Draft anchor | Why it fails | Rewritten anchor (meets expectations) |
|---|---|---|
| Demonstrates strong ownership of client issues | Names a disposition, not an action; “strong” is a judgement | Logs the client issue the same day, names an owner, and updates the client before the promised date even when there is no progress |
| Communicates effectively with stakeholders | No situation, no action, no evidence standard | In a project review, states the decision needed and the date it is needed by, in the first two minutes, before presenting status |
| Is a team player who supports colleagues | Reputation language; cannot be rated at the edges | Covers a colleague’s on-call slot when asked, and tells the affected client in advance rather than after |
| Shows leadership potential | A prediction, not an observation; belongs in a talent review, not an anchor | When a deadline slips, proposes the revised sequence, names what will move, and tells the affected team before the client hears it |
Each rewrite has a trigger (a client issue, a review, a request, a slip), one action, and a completion checkable without asking the person. None uses an adjective of degree.
A pilot that costs one afternoon
Illustrative example. A 300-person services firm had five competencies and a calibration problem: two managers disagreed on 40 per cent of ratings, and HR could not defend either number. They picked one competency and rewrote only its meets-expectations anchor, from twelve manager incidents, into four observable lines. Then they ran three raters through six recorded customer calls, thirty minutes of briefing, no discussion between raters.
On the first pass, all three raters agreed on two of the six calls. On the four disagreements, the reason was the same: the anchor did not say what counted when the customer interrupted. They added one clause, “returns to the original question after the interruption”, and re-ran the same six calls with a fresh set of three raters. Agreement rose to five of six. The set took two hours to write and one afternoon to test. Nothing was deployed until the second pass.
The number that matters is not the agreement rate. It is that four disagreements produced one sentence of specification, which will now end every future calibration argument on that competency. Rewrite the second competency the same way before touching the third. Five at once produces a document nobody has watched working, which is how the original problem started.
The limit of good anchors
Better wording is necessary and not sufficient. In a randomised trial with 47 surgeons, raters who received frame-of-reference training scored higher agreement than untrained raters on all three instruments, but the confidence intervals overlapped and the difference was not statistically significant. Even the trained group sat below the 0.8 threshold normally wanted for high-stakes decisions. Training helped; it did not solve it.
Three consequences. Do not use a single rater for a decision that changes someone’s pay or job; use two, or an exercise with a fixed scoring key. Report agreement as a percentage rather than presenting a rating as a measurement. Treat anchors as one input into a conversation: a manager who has watched the work and can cite an incident does not need a perfect scale.
There is a cost the vendor case rarely mentions. Well-written anchors constrain managers, making it harder to reward someone you like and harder to punish someone you do not. Some will resist, politely, saying the anchors are too rigid for their team. That objection is usually accurate about the work and honest about the motive.
One limit about this article. The rewrites are for a services context, where behaviour happens in conversations. On a shopfloor the principles hold but the evidence is physical: a handover sheet, a start-up sequence, a near miss. Do not copy these lines into a plant.
Monday
Take one competency. Ask four managers for two real incidents each from the last quarter, one good and one bad, written as what they saw rather than what they concluded. From those eight incidents, rewrite only the meets-expectations anchor into no more than four lines, each with a trigger, one action and a checkable finish. Then find one recording, one meeting or one call and have two managers rate it separately, without discussing it first.
Count the disagreements and read what caused them. That is your specification. If you want the anchor set assessed against a proper framework, or a scoring design that two raters can hold to under pressure, our competency framework work and assessment design both start from exactly this test. The wider argument about how many competencies a manager can realistically carry is in a competency framework nobody uses, and the rater-side version of the same problem is in assessment centres without a psychometrician on site.
Sources
https://pmc.ncbi.nlm.nih.gov/articles/PMC9090385 https://pmc.ncbi.nlm.nih.gov/articles/PMC6281450 https://www.vic.gov.au/place-based-approaches-capability-framework/implementation-tool-behaviourally-anchored-rating https://www.purdue.edu/hr/lod/media/docs/annualevaluationcompetencybehavioranchors.pdf
