The wash-up is ninety minutes old and the room has settled into a trade. A junior assessor scored a candidate 2 on communication in the group exercise, and she has the evidence: the candidate withdrew for most of the meeting. A senior manager scored the same candidate 4 after a strong interview. Neither number is wrong on its face, because they come from different exercises. Then the facilitator asks the senior manager whether he would be happy to go down to a 3. He is not. The junior assessor offers to move her score up instead.
That exchange is on record, from a study of four wash-ups across three organisations reported by Dewberry and Jordan at a British Psychological Society conference. Notice what did not happen: nobody asked for the evidence behind either number. The conversation was about who would move.
This is scoring drift, the cheapest failure in an assessment centre to fix and the most expensive to ignore. The client never sees the exercises. They see a shortlist reordered after a discussion they cannot audit.
Where the drift comes from
Drift is a design problem with three sources, and they stack. The first is an anchor written as an adjective. Most centres define a dimension as “communicates clearly and persuasively”. That describes a person, not a behaviour, so there is nothing to tick. Two assessors read it, each retrieves their own idea of a good communicator, and both believe they apply the same standard. They apply two standards with one label.
The second is unmeasured severity. Some assessors are consistently harder than others across every candidate and dimension. That cancels out only when the same candidates meet the same mix of raters. Change who observes exercise three and the ranking moves.
The third is the integration session. The BPS guidelines expect assessors to finish their own evaluations, including the report form, before the wash-up. When the day runs late that sequence compresses and the discussion becomes the rating. The loudest voice sets the score, and the paperwork is written to match.
A fourth factor is structural. The 2015 Taskforce guidelines ask for assessor performance to be monitored periodically, with retraining or decertification when ratings show a lack of reliability. Few centres keep the data that would allow it, so drift repeats every cohort.
What the standards actually require
Keep two documents open while you design the next centre: the British Psychological Society’s best practice guidelines and the International Taskforce guidelines published in the Journal of Management in 2015. Both are free, and more specific than most internal centre manuals.
The BPS guidelines say assessor training “will typically last at least two days and be largely interactive”, that assessors should rate within two months of training or take refresher training, and that anyone who has not assessed for a year should be refreshed before rating again. They also require training in the Observe, Record, Classify and Evaluate process as applied to your simulations, and independent evaluation before integration.
The Taskforce is more careful about hours. Length and quality of training are not synonymous, it notes, and one day may be sufficient for a well-structured centre with few exercises, a qualified trainer and selected assessors. What it insists on is competence: stated performance guidelines, and evidence an assessor can apply them.
Both point the same way. Calibration cannot be bought with a training day. It has to be demonstrated in your centre, on your exercises, with your people, and checked next cohort.
Step one: rewrite the anchors so they can be counted
Write three anchors per dimension: below the bar, at the bar, above it. Each is one behaviour with a condition and a threshold, phrased so two people watching the same eight minutes would both tick it without speaking.
The test is simple. Hand the anchor to an assessor who has not seen the exercise and ask what they would look for. If they name a different behaviour from the one you meant, the anchor is unfinished. If they ask how often, put the frequency into the anchor.
| Weak anchor | Why it fails | Rewritten anchor |
|---|---|---|
| Communicates clearly and persuasively | An adjective about a person, not an act | States the decision and the reason in the first 30 seconds of a two-minute update, unasked |
| Handles conflict well | Unmeasurable, and every rater pictures something different | Names the disagreement, asks the other person what they want, restates the agreed next step before the exercise ends |
Two rules keep this honest. Anchors describe what a person did, not what they are like, so verbs beat adjectives. And anchors belong to the exercise: “at the bar” in a written analysis is not “at the bar” in a role play.
If nobody can hold your competency framework in their head, this is the same subtraction problem: a dimension you cannot write three countable anchors for is one your centre cannot rate. How to cut a framework down to what managers actually use makes that case.
Step two: pair raters by rotation, not by seniority
Once the anchors are countable, control who sees what. Build a matrix before the day: exercises down the side, assessors across the top. Then apply four constraints.
- Every candidate is seen by at least three assessors during the day.
- Every dimension is rated at least twice, in two different exercises, by two different assessors.
- No pair works together on more than two exercises, so a shared blind spot cannot repeat.
- The most senior person is not automatically the chair, or the pair lead.
Constraint two is the one people skip, and it is what makes drift visible. If one person rates a dimension, you cannot tell whether their standard is the centre’s standard.
The last constraint needs arguing for. The A&DC review describes assessors using seniority to persuade others to accept or reject candidates, in a centre the organisation believed was well run. Seniority is not evidence. Give the chair role to whoever can hold the evidence order.
Step three: the calibration hour
This is the part that changes results, and it costs one hour before the first candidate arrives.
Take one recorded exercise from a previous cohort, six to eight minutes long, with clear behaviour in it. Every assessor scores it alone on the real rating sheet. No discussion. Then compare dimension by dimension, and talk only about gaps of two points or more on a five-point scale. Small differences are noise. Large ones are a broken anchor or a different internal standard.
Every gap ends with one written ruling of 25 words or fewer, added to an anchor log. “Decision stated with no reason is a 2, not a 3.” Or: “One interruption is not a 1; sustained interruption is.” By the third cohort that log is the most-read page in the folder, because it is specific and written by the people who use it.
Two rules keep the hour from becoming a debate. The exercise designer speaks last, so nobody anchors on intent. If a dimension still splits after twenty minutes, those anchors are unfinished, and you have found out before the centre.
For external assessors, this is also the handover that makes their training transferable to your exercises. Same principle as a trainer handover internal facilitators actually read: one page, written for the specific room.
Step four: a wash-up that follows evidence order
The wash-up is where drift gets corrected or signed off. Evidence first, score last. Each assessor reads the behaviour they observed, in the order the exercises happened, before anyone says a number. The chair asks “what did you see” whenever a rating arrives without an observation attached.
No trading. Scores move when new evidence is presented, not because a senior person is uncomfortable, and every move is written in a drift log: candidate, dimension, from, to, evidence cited. That log is your monitoring data.
Rotate the speaking order. The junior assessor goes first on at least one candidate per session, so the senior view is not the frame everyone adjusts to.
Review the log before the next centre. Three patterns matter: one assessor whose ratings move more than anyone else’s, one dimension producing most of the movements, one exercise where they cluster.
A worked example
The numbers below are illustrative, built to show the shape of the arithmetic, not to report a client result.
A manufacturer runs a one-day centre for 22 shift supervisor candidates: four exercises, six assessors, one HR coordinator. The first shortlist is rejected by the plant head as “not the people I see on the floor”. The coordinator pulls the sheets and computes, per dimension, the largest gap between two raters who scored the same candidate.
Handling conflict shows an average gap of 1.4 points on a five-point scale across the 18 candidates rated on it twice. Worse, six of the 22 cross the shortlist cut-off depending on which rater saw them. The anchor reads: “keeps the conversation calm and constructive.”
The fix takes ten hours: four to rewrite three dimensions as countable anchors with the designer and two assessors, six assessor-hours for a calibration session on one recorded role play, producing eleven ruling lines. The pairing matrix is rebuilt so every dimension is rated twice, in different exercises, by different people.
At the next sitting the average gap falls to 0.5 points and one candidate crosses the cut-off instead of six. No exercise and no candidate pool changed. Ten hours against a day that consumes about 48 assessor-hours is a fifth of delivery cost. You are not buying accuracy, you are buying the right to defend a decision.
What this does not fix
Calibration makes ratings consistent, not valid. If the exercise does not force the behaviour you care about, you get reliable ratings of nothing, and a tighter process makes the wrong answer look better evidenced.
It does not survive assessor turnover alone. The BPS guidance is explicit about rating within two months of training and refreshing after a year away, and the Taskforce asks for monitoring with recertification. An anchor log helps a newcomer; it does not replace the two days.
There is a measurement limit too. With three or four raters per dimension you cannot compute meaningful reliability statistics per centre, and should not pretend otherwise in a report. The spread on the calibration exercise is a proxy, not a coefficient.
Drift returns when a new exercise is written, an assessor joins, or a centre runs late. The calibration hour is a recurring cost, not a project.
Do this on Monday
Pull the scoring sheets from your last centre and compute one number per dimension: the largest gap between two raters who scored the same candidate. Take the widest dimension, rewrite its three anchors as countable behaviours, and book 45 minutes with two assessors to score one recorded exercise separately. If the gap is still two points after that, you have found the real problem before it reached a shortlist.
Our assessment practice starts with that arithmetic rather than the exercise library, and the same discipline runs through the competency work behind the dimensions and the coaching that follows a centre.
Sources
https://www.klug-md.de/downloads/BPS_PTC_AC_Guidelines.pdf https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=6688&context=lkcsb_research https://y2cp.com/pdf/publications/adc/are_wash_ups.pdf https://journals.sagepub.com/doi/10.1177/009102600503400102