Every quarter a training deck lands on someone’s desk with a 4.6 out of 5 satisfaction score and a line that says “return on investment: 340 per cent”. The number is usually produced by dividing a notional value of improved performance by the programme fee. Nobody in the room believes it, including the person who built the slide. It survives because everyone has agreed not to ask.
Kirkpatrick’s four levels are still the default language of training evaluation, and the model is fine. The problem is that level three and level four are routinely asserted rather than measured, and asserting them has become the industry norm.
What each level actually costs
The levels are not equally hard. They are not even close. The reason most programmes stop at level two is that level one and two can be collected inside the room, while level three requires access to the participant’s working life and level four requires a number from finance.
| Level | What it proves | Real effort | Who has to cooperate |
|---|---|---|---|
| 1. Reaction | The room worked | Three questions at the end | Nobody |
| 2. Learning | People can recall and apply it in a test | Pre and post assessment, same items | Facilitator |
| 3. Behaviour | The behaviour shows up at work, 60 to 90 days later | Observer ratings or a short 360, chased | Reporting manager, sometimes peers |
| 4. Results | A business metric moved in the expected direction | Baseline, a control comparison, finance sign-off | Business owner, finance |
Level one is cheap and almost useless alone. Keep it to three questions and do not put it in the executive summary as if it were an outcome.
Level two proves people can answer questions. It is worth having, because a programme where nobody learned anything cannot possibly change behaviour, but it is a gate, not a result.
Level three is the one that matters and the one most often skipped. Where a client has no internal capacity to run it, we run the observer pulse as part of the assessment work. It needs a measurement at 60 to 90 days against the same behaviours that were assessed before the programme. If a full 360 is too heavy, three observer ratings per participant is enough to see a shift, provided the observers are the same people before and after.
Level four is not a training metric. It is a business metric the programme claims to influence: attrition in a cohort, internal mobility, sales cycle length, first-pass yield, customer scores. Pick one the business already tracks, agree the expected direction before the programme starts, and accept that other things will also move it.
Level four is not a training metric. It is a business metric the programme claims to influence.
The mechanism that produces fake numbers
The faking is not usually dishonest. It is structural. Three things push an evaluation team towards theatre.
The measures are chosen after the result is known. If you decide what to measure once the programme is over, you will unconsciously pick the metric that moved. Fixing the measures before the proposal is signed is the single highest-value discipline in this whole article.
The follow-up depends on people who were never asked. Level three needs managers to spend fifteen minutes filling a form. If that was never agreed, the data will not arrive, and the honest response is to report “not measured”. Most teams instead extrapolate from level two, which is where the fiction enters.
Nobody is rewarded for a null result. A programme that did not move the metric is a useful finding. In most organisations it is a career risk, so it gets smoothed into a positive story. The result is that the next budget decision is made on numbers nobody trusts.
What we actually run
For a typical six-week behavioural programme, four data points:
- Baseline assessment before the first session, on the behaviours the programme claims to change.
- End-of-programme assessment on the same items, so the delta is comparable.
- A 60-day observer pulse: three ratings per participant from people who work with them, on the same anchors.
- One business metric pulled from the client’s own reporting, with the expected direction written down in advance.
That is it. No dashboard, no quarterly review committee. The report is a single page: what moved, what did not, what we would change next cycle, and what we could not measure and why.
Clients renew on that page, not on the certificate. It is also the reason our coaching engagements start with a baseline rather than a goal sheet. It is also the only artefact from a programme that is still useful a year later, because it tells you which parts of the design were doing the work.
The transfer side of this, meaning what has to happen in the weeks after the session for level three to be worth measuring, is in why behavioural training fades in six weeks.
A worked example
A services client ran a programme for sixty first-time managers, focused on giving feedback on poor performance. The first cycle was evaluated at levels one and two only, and reported as a success on a 4.4 satisfaction score.
For the second cycle we added two things. Each participant nominated three observers before the programme started, and we agreed with the HR head that the business metric would be the number of documented performance conversations closed within thirty days of a missed target.
At day sixty, observer ratings on the “gives specific, timely feedback” anchor had moved from a mean of 2.4 to 3.1 on a five-point scale. The business metric moved from 41 per cent to 58 per cent closure within thirty days.
Both numbers are modest and both are believable, which is why the HR head took them to the leadership review. A claim of 340 per cent return would not have survived the first question.
The honest limit
Level four is often not computable, and pretending otherwise wastes everyone’s time. If the programme is one of nine things that could have moved a customer score, you cannot attribute the movement to the training. The honest version is to say the metric moved in the expected direction, name the other plausible causes, and stop there.
For most behavioural programmes the defensible claim is level three: the behaviour changed, measured by people who work with the participant, against anchors agreed in advance. That is a real result. It is also a much smaller claim than “340 per cent return”, and it is the one that survives scrutiny.
Explaining a null result without losing the budget
Sooner or later a programme will not move its metric. The way that is handled decides whether the evaluation survives as a practice.
Four sentences do most of the work: the measure was fixed in advance and it did not move; the behaviour measure did or did not move, here is the number; the likely cause is X, based on the observer comments; the next cycle changes Y and will be measured the same way.
What kills the practice is not a null result. It is a null result presented as a soft positive, because the sponsor eventually discovers that the reporting cannot be trusted, and stops funding evaluation rather than funding programmes.
What to do on Monday
Take one programme per quarter. Not all of them. Fix the four measures in writing before the proposal is signed, name the person who will supply each one, and put the day-sixty pulse in the calendar at kickoff.
Then publish the null results as well as the wins. The first honest report is uncomfortable. The second one changes how the business buys training.