The review meeting has a spreadsheet in it. Forty-eight supervisors were assessed before the programme and again four months later, and the second column is higher. The business head asks whether the training did that. Nobody can answer, because no group of supervisors was left alone.
So the room splits. One side claims the gain, the other retreats to feedback scores. Both are wrong, and the second costs more, because it teaches the business that L&D cannot answer a question about its own work.
You do not need a randomised trial. You need to be able to name, out loud, which rival explanations you ruled out and how. The four levels are not the problem, as the honest version of that model argues. The design underneath level three is.
What a control group actually buys you
A control group removes named alternative explanations. If the untrained group improved by the same amount, the training probably did not cause it. If they did not move and the trained group did, it is a plausible cause, not a proven one. An IES review of evaluation practice made the gap plain: everyone knows they should do this properly, few admit to having got it right.
In practice these are the rivals worth naming in a training review:
- Regression to the mean. The cohort was chosen because their numbers were poor, and poor numbers drift back towards average on their own. Barnett, van der Pols and Dobson describe the effect plainly: it makes natural variation in repeated data look like real change, and it grows with measurement error and with selecting on a baseline score. That is precisely what a struggling cohort is, and it is invisible unless someone looks for it.
- Season and other changes. Quarter-end pushes, monsoon logistics, festival demand, audit cycles, a new incentive, a new supervisor, a price change, a system rollout, an attrition wave. A three-month programme that starts in April and ends in July measured both months as much as it measured the training, and most plants change two or three other things in a quarter.
- Being watched. People who know they are in a programme behave differently for a while.
Now the business reality. You cannot withhold training from a group that was already promised it, and no sponsor accepts a lottery for a programme they have committed to. That is a constraint to design around, not a failure of nerve.
Most of them can be handled without a control group. None completely.
Substitute one: the next cohort, measured while it waits
Most Indian rollouts are staggered anyway. Twenty sites, four trainers, three quarters. The second batch exists because delivery capacity is finite. Use it.
This is close to what researchers call a stepped-wedge design: every group starts untrained, and the intervention reaches groups at set intervals until everyone has it. The London School of Hygiene and Tropical Medicine notes that such rollouts are usually chosen for practical or ethical reasons, and can then be turned into a rigorous evaluation.
Four rules make it honest:
- The order must come from the roster, not from need. If you train the worst-performing sites first and compare them with better sites, you have built selection bias into the design and the comparison is worthless. Order by shift pattern, geography, trainer availability, anything that is not correlated with performance.
- Both groups get measured at the same calendar points. Not “before and after” for each group on their own timetable. Same weeks, same instrument, same person administering it.
- The waiting group must actually be trained on the promised date. If the programme is cancelled or pulled forward because of an escalation, you lose the comparison and you must say so rather than quietly dropping it.
- Say what the comparison is. “Sites 5 to 8 had not yet started when the first measurement was taken” is a sentence a finance reviewer can follow. “Statistically significant” is not.
The waiting cohort answers season and the other-change problem better than anything else you can run, because both groups lived through the same quarter.
Substitute two: a number the business already counts
Self-report is the weakest link in most evaluations and the one people trust most. A facilitator asks whether supervisors now give feedback more often, and the ones who liked the workshop say yes. That is a survey about the workshop, not the floor.
Pick one number the operation already records weekly, and define it before the programme starts. Examples that exist in most Indian plants and sales organisations:
- first-time-right or rework percentage from quality
- credit notes raised per hundred invoices
- escalations that reached a second level, or tickets reopened after closure
- safety observations raised and closed by supervisors
- average days from customer complaint to first response
The rules are stricter than they sound: the same recorder, the same definition, the same frequency. Eight to twelve weeks of history before the programme, so you can see the normal weekly swing. A named owner in the business, not in L&D, who confirms the definition held.
The reason this works is not that operational data is perfect. It is that it was not collected to please you. Nobody filling a quality sheet knows it will end up in a training review six months later.
Substitute three: measure four times, not twice
Two points make a line, and a line always looks like a trend. Four points show you the natural variation, which is what tells you whether a movement is worth reporting.
Take three readings before the programme, spaced by the operating cycle rather than by convenience, then one at the agreed review point. If the three pre-readings already wander by six points, a four-point improvement after training is noise, and reporting it as impact is the kind of claim that costs a function its credibility in year two.
There is a second reason to measure early. A Solomon four-group study of post-training transfer interventions tested for pretest sensitisation because earlier transfer studies rarely did. The technical name hides a simple fact: asking people about a behaviour makes them notice it.
So a gain from pre-test to post-test holds at least three things: the training, the drift of extreme scores back towards average, and the effect of having been asked.
Write the rival explanations down before you see the data
Before the programme ends, sit with the business owner and write two lists. What we expect to move, by how much, in which direction. Then what else will change in this period: the incentive change, the new machine, the manager who resigned, the price rise, the audit. Date it and have the business owner sign it.
When the results arrive, read the second list first. If the metric also moved in an untouched department, say so before anyone else finds it. This is scoping work, and it belongs in the discovery stage of an engagement, because by report time the decisions are already made.
The point is not procedural neatness. It is that you cannot afterwards choose the story that flatters the programme. L&D people are human beings with a renewal to protect.
Claiming what you can actually defend
| The claim | What it needs |
|---|---|
| “Participants rated the programme 4.4 out of 5” | Reaction sheets. Honest, and says nothing about work. |
| “Knowledge scores rose from 61 to 78” | A test before and after. Say the cohort was selected and name the drift risk. |
| “The trained group improved more than the waiting group” | Staggered rollout, identical measurement, roster-based order. |
| “Rework fell 4 points after the programme and held for nine weeks” | The operational metric, enough history, the rival-explanation log, and no claim of causation. |
| “The programme caused a 4-point fall in rework” | A design most corporates cannot run. Say so, or do not say it. |
The fourth row is where most well-run programmes should land: specific, checkable, and it survives a finance review because it does not overreach. It sits beside the number clients ask for first, attendance plus one leading indicator.
A worked example, illustrative
Numbers below are constructed to show the arithmetic, not drawn from a client.
A cement products company runs a supervisor coaching programme across four plants, 96 supervisors, starting with plants 1 and 2 and moving to plants 3 and 4 a quarter later. The order came from trainer availability, not performance. The metric is rework percentage, recorded weekly by quality. Nobody in L&D chose it.
Eight weeks before the start, plants 1 and 2 sit between 7.9 and 9.1 per cent weekly, plants 3 and 4 between 8.2 and 9.4. Over the programme quarter, plants 1 and 2 fall to 5.6 to 6.8. Plants 3 and 4, still untrained, stay at 7.8 to 9.3. The bands have stopped overlapping.
The report makes four statements. The trained plants moved by about two and a half points. The untrained plants did not. The window included a price rise and a new shift pattern at plant 2, both logged before the data was read. The claim stops there: consistent with the programme working, not proof of it.
Where this fails
If the programme is rolled out everywhere at once, there is no waiting group and no cleverness replaces one. Report the pattern and say what you cannot claim.
If the metric is owned by the training team, the independence argument collapses. A number the trainer reports about their own programme is a survey with extra steps. And if the metric comes from a rater filling a form, the defence rests on rater training and pairing, which is the discipline described in the assessment centre design work.
If the cohort is small, three supervisors at one site, weekly counts will bounce enough to hide any real change. Use a longer window or a coarser measure, and accept that the honest answer is “not measurable at this size”.
And if both groups improve equally, that is a finding, not a failure. Something else is working or the metric is drifting, and the business case needs rebuilding. The instinct to suppress that is the instinct that makes an evaluation function worthless.
Monday
Pick the metric this week, before the next cohort starts. One number, already recorded weekly, with a named owner in the business. Ask that owner for the last twelve weeks in writing. Put three dates in the calendar: the pre-readings and the review point. If the rollout is staggered, write the order down and check it is not correlated with performance.
An afternoon, and it decides what you can say in March.
Sources
- Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology, 2005. https://pubmed.ncbi.nlm.nih.gov/15333621/
- London School of Hygiene and Tropical Medicine. Stepped-wedge trials. https://www.lshtm.ac.uk/research/centres/centre-evaluation/stepped-wedge-trials
- Testing the Effectiveness of Transfer Interventions Using Solomon Four-Group Designs. Education Sciences, 2020. https://www.mdpi.com/2227-7102/10/4/92
- Tamkin P, Yarnall J, Kerrin M. Kirkpatrick and Beyond: a review of models of training evaluation. Institute for Employment Studies Report 392, 2002. https://www.employment-studies.co.uk/system/files/resources/files/392.pdf
