Chapter 19 in two minutes

Chapter 19 looks at what research still needs to do. The first generation of studies set out to show that systems function, learners engage and short-term results improve. The next generation has to explain why aligned delivery helps: through cognitive, motivational or social mechanisms, for which learners, and in which contexts.

That means longer studies. Immediate test scores are easy to collect and tend to overstate long-term transfer, so the chapter calls for tracking capability and performance over months or years, and for measures of application in practice rather than recall. In Kirkpatrick's terms, the gap is in behaviour and results.

It also means testing with trainers who did not develop the approach, to rule out expectancy effects and show the results generalise. And it raises open questions about learner data: who owns it, how long it is kept, how it is protected from misuse, and what recourse learners have.

Showing that something works and explaining why it works are different jobs.

Watch or listen

Prefer to take it in by eye or by ear? Both cover the core ideas of Chapter 19 in a few minutes.

Film, 2 min

The next era of AI workplace learning

Where research on AI-supported training goes next, and why proving it works is only the first half of the job. Captions are built in.

Conversation, 6 min

How to prove AI training works

A conversation about longitudinal studies, independent trainers, measuring behaviour rather than recall, and the ethics of learner data.

0:000:00

Chapter 19 at a glance

A quick overview, then the full argument with the evidence. Select either one to see it full size.

Four-panel infographic: move from demonstration to explanation; prioritise longitudinal capability over scores; test using independent, neutral trainers; and measure behaviour change, not recall. — The Future of AI Learning Research

Detailed infographic: generation one, the demonstration phase, and generation two, the explanation phase; defining boundary conditions for who benefits most and in which contexts; longitudinal studies at six months and two years; measuring application in practice rather than knowledge recall; the independent trainer test; and the evaluation focus shifting from reaction and learning, Kirkpatrick levels 1 and 2, to behaviour and results, levels 3 and 4. — The Next Generation of Learning Intelligence

How the ideas connect

Every idea in Chapter 19 on one page. Look at it before you read to see the shape of the argument, or afterwards as a recap.

Open the mind map

Put it into practice

Short activities to run with a group or work through on your own. Each one has a printable card.

With a group (3)

Time
15 to 20 minutes
Who
6 or more people
You will need
Sticky notes and pens
Download printable card
Icebreaker
The Future Research Wall
What we do not yet know is the more interesting half of the chapter.
Take a sticky note and write one open question or concern from the chapter that stayed with you.
Add a symbol or short phrase for why it matters to you.
Post it, read the wall, and group yourselves by theme.
In your group, introduce yourself, explain your question, and name one thing you want to learn from the others.
Working on your own? Write three open questions, then rank them by which would change your practice most if it were answered. That is where to read next.
Time
10 to 15 minutes
Who
6 or more people
You will need
Space to move around
Download printable card
Icebreaker
The Evidence Line-Up
Where you stand on evidence says more about you than a round of introductions.
Mark a line across the room, strongly agree at one end, strongly disagree at the other.
Read a statement out: short-term test gains are strong evidence that training worked.
Stand where you actually sit, then tell two or three people near you what put you there.
Hear from a few people at different points, then run a second statement on ethics and governance.
Working on your own? Write your position on each statement, then write the strongest case against it. If that case is thin, you have not tested your own view.
Time
20 to 25 minutes
Who
6 or more people
You will need
Interview grids and pens
Download printable card
Icebreaker
Research Partner Interview Grid
A short grid, three conversations, and the room’s priorities become visible.
Four prompts in a grid: most urgent research area, limitation that worries you, data that shouldn’t be misused, what strong long-term evidence looks like.
Rotate through three short conversations, introducing yourself and taking notes each time.
Read your notes back. Find one pattern and one answer that surprised you.
Name one person whose view you want to hear more of, and say why.
Working on your own? Answer the four prompts, then put the same questions to two colleagues. The disagreement about urgency is usually the telling one.

On your own (3)

Time
20 to 30 minutes
Who
On your own
You will need
Notebook or journal and pen
Download printable card
Journaling
From Findings to Future Research
Showing that something works and explaining why it works are different jobs.
Write what the first generation of research set out to do, and what the next one should take on.
Say why longitudinal studies matter, and why the trainer may be a confounding variable.
Answer this: what evidence would make you trust a training method more, and why?
Sketch a study of your own. What you would test, who takes part, how long you would track them, and the ethical issue you would have to handle.
Working with a group? Write first, then underline one sentence you are confident about and one you are not. Collect the second kind and work through those.
Time
25 to 35 minutes
Who
On your own
You will need
Paper and pen
Download printable card
Problem solving
Design a Better Study
One trainer, strong test scores, warm feedback, and a report written the same week.
List five weaknesses in this evidence and say what each one hides.
Redesign the study: the question, who takes part, how long you follow them, and what you measure beyond test scores.
Answer plainly: can the company call this proven? Then name the one change you would make first.
Working with a group? Each person names the single change they would make first, then defends the order. The ranking argument is where the thinking shows.
Time
30 to 40 minutes
Who
On your own
You will need
Paper and pen
Download printable card
Worksheet
Showing or Explaining
Proving something works is the easier half. The field has mostly done that half.
Read Approach A. One trainer delivers adaptive training to a single cohort. Learners complete a satisfaction survey and a quiz straight afterwards. Scores improve, feedback is warm, and the result is reported as evidence that the method works.
Read Approach B. Several independent trainers run the same approach across multiple cohorts. Learners are assessed immediately and again months later, on practice, adaptability and behaviour change, with data governance defined in advance.
Compare them: what each one can honestly claim; what immediate gains do and do not show; how each handles the trainer as a variable; what each counts as an outcome worth measuring.
Critique both. What is the smaller study still good for? Why might it look effective and not generalise? What does the longer design let you say? What limitation survives even here?
Make a judgement. Of these five, which improvement matters most: explaining mechanisms, longitudinal study, independent trainers, ethics and governance, or measuring meaningful outcomes? Defend one against another. Finish the sentence: What researchers should stop relying on is…

Go deeper

The research and writing behind Chapter 19.