Pharmaceutical and medical education budgets move through thousands of dinner meetings every year in the United States, and the majority of them are judged by two numbers: how many healthcare providers showed up, and what the demographic split looked like. That is the entire scorecard in most programs. Richard “Rick” Davis, who has spent his career inside medical education, calls that a very poor return for the time, effort, and money spent. His argument is uncomfortable for anyone running these initiatives, because it does not blame budgets or bad speakers. It blames measurement. Companies are spending real money to change clinical behavior, while not collecting the data that can tell them the likelihood of creating behavior change.
The Satisfaction Trap
Davis points to Kirkpatrick’s four levels of evaluation, developed in the early 1950s, as the simplest way to see the problem:
- Level one asks whether the audience liked the program.
- Level two asks whether they learned anything.
- Level three asks whether their behavior changed.
- Level four asks whether that change was sustained.
“Unfortunately the greatest number of programs measure only to level one, that’s it. Decades after the framework was published, most programs still stop at “did they enjoy it.” says Davis.
He traces his own awareness of the gap to a conversation early in his medical career. He asked a sponsor of a meeting what made a meeting successful and was told they got people to attend. Pressed on whether learning mattered, the answer was “yes,” people learned important things. Pressed on what, specifically, they learned. “We really didn’t get into that detail,” Davis recalls being told. That detail is the whole point. Attendance and demographics are useful opening information, he says, but they are not the end goal, and treating them as the final verdict means an organization never discovers what its own program truly accomplished.
What A Real Evaluation Process Measures
Davis frames effective evaluation around two deceptively plain questions:
- Did the audience learn what they needed to learn?
- Did the audience receive educational value for time spent?
Answering either one requires knowing things most programs never capture. You have to know what the audience knew walking in. You have to know what they knew walking out. Beyond that, you have to know where gaps in their understanding still exist after the program ends. And you need a defined follow-up mechanism to reiterate key messages and clarify what remains unclear.
Gaps are inevitable, and Davis treats them as a design problem rather than a failure. When feedback shows something did not stick, his first move is to examine how it was originally presented and why it failed to land, honestly. The fix is often far less exotic than teams expect. “Sometimes it’s as simple as order,” he says, noting that changing the sequence of the story can produce entirely different assimilation. Sometimes learners simply need more opportunity to apply the material. Medical science rarely lands on first exposure. Reiteration is not a patch on a weak program; it is part of how learning works, and looking at the material from the learner’s point of view is the discipline most teams skip.
Why Most Programs Have No Plan Past Step Two
The single most overlooked step, in Davis’s view, is upstream of the feedback entirely. Without an overarching strategic educational plan, organizations have no structure into which feedback can be plugged. The data arrives and nobody knows what it means for the next move. A real plan builds in checkpoints where feedback gets collected and converted into defined next steps. Absent that, evaluation becomes an exercise in collecting opinions nobody can act on.
The underlying misconception, he argues, is that an educational initiative is itself the learning event. “Learning is not an event. It’s a process,” Davis says, which means the plan needs reiteration built in and timed deliberately to increase retention. Asked how many organizations operate this way, his estimate is blunt. If the process had five steps, most teams have planned step one, many would have thought about step two, and very seldom does anyone reach step three. What follows is reactive rather than proactive, a collection of one-offs instead of a sequence. A properly designed plan leverages every element against the one before and the one coming after, producing continuity, consistent reinforcement of key messages, and what Davis calls greater return on education alongside return on investment.
Design For The Result, Not The Agenda
The question Davis would put to every program leader cuts at design rather than delivery: did you teach the audience what they needed to learn, or what you wanted to teach them? It sounds like a distinction without a difference. It is not. “Many times they’re not the same,” he says, and that mismatch is the first disconnect in a chain that ends with a program nobody can prove worked. If the goal is behavior change, and it usually is, teams have to understand the sequence of events that leads a healthcare provider to try something outside their current portfolio of behaviors or implement something new in practice.
That means measuring beyond attendance, beyond demographics, beyond level one. Davis wants to know whether the intended key messages were communicated quantifiably rather than speculatively, whether the information is likely to be incorporated into daily practice and to what degree, and what gaps remain. When asked about AI, his read on artificial intelligence is pragmatic rather than evangelical. It does not fix a program measuring the wrong things, though it may surface the fact that the approach was wrong in the first place. Where it helps is speed and accuracy, sorting information faster and delivering deeper insight, which matters because in learning, timing is everything. Hit the learner with the next step at the opportune moment and the whole process gets more effective, a rifle shot rather than a shotgun blast. Given the daily demands on practicing healthcare providers, Davis argues they should be the best-served demographic from an education standpoint. Today he believes they may be the most underserved.
Follow Richard “Rick” Davis on LinkedIn for more insights on medical education strategy, learning design, and program evaluation.
