Training evaluation is an extremely important part of running any successful course or learning journey. Without proper evaluation, how would you know whether a course is having an important impact or just sending the learners to sleep? If you’ve been following our series, you’ll be up to speed on a variety of evaluation models including Kirkpatricks Model, the CIRO Model, the Phillips ROI Model, and Brinkerhoff’s Success Case Method (SCM). In this post, we’ll be exploring Kaufman's Levels of Learning Evaluation, outlining its key concepts and telling you everything you need to know about this fascinating learning evaluation model.
In this post, we’ll be discussing things like:
- At a glance: Kaufman’s Five Levels of Evaluation
- What is Kaufman’s Model of Learning Evaluation?
- How does Kaufman’s model differ from Kirkpatrick’s?
- Key concepts in Kaufman’s Five Levels of Evaluation
- Conclusion
Sound fair?
Before we begin, we’ll offer a summary of Kaufman’s Model for readers who are already familiar with the Kirkpatrick Model on which it is based:
1. At a glance: Kaufman’s Five Levels of Evaluation
-
Kaufman’s model mirrors the four levels of Kirkpatrick’s model.
-
Kaufman divides Kirkpatrick’s Level 1 (Reaction) into two sections: “Input” and “Process.”
-
Kaufman’s fifth level evaluates results for both the customer and society in general.
-
Kaufman’s model is positioned as “more practical” than Kirkpatrick by the author.
2. What is Kaufman’s Model of Learning Evaluation?
Roger Kaufman and John M. Keller published Levels of evaluation: Beyond Kirkpatrick in the winter 1994 edition of Human Resource Development Quarterly. This work became known as Kaufman’s Five Levels of Evaluation and is commonly referred to as Kaufman’s Model of Learning Evaluation.
Kaufman’s model is one of a number of learning evaluation models that build on the Kirkpatrick Model, one of the most popular and widely-used training evaluation models of all time. As you may know, Don Kirkpatrick introduced his four-level methodology in 1959 through a series of articles published in the Journal of the ASTD. He published his seminal work, Evaluating Training Programs: The Four Levels, in 1993, finally making his ideas available in an easy-to-use reference guide. This work underpins a number of subsequent training evaluation models, including Kaufman’s.
As Kaufman’s Five Levels of Evaluation is closely based on Kirkpatrick’s four levels, it’s worth recapping the Kirkpatrick levels first. The Kirkpatrick Model is a four-level approach to evaluating training effectiveness that can be applied to any course or training program.
The levels are as follows:
-
Level 1: Reaction
This level tells you what the participants thought about the training. What were their overall impressions?
-
Level 2: Learning
This level shows you what, if any, learning took place. Did the participants learn something from the training? If so, what?
-
Level 3: Behavior
This level shows you whether the training produced on-the-job changes. Did the participants use the knowledge and skills they gained from the training when they went back to work?
-
Level 4: Results
The final level, Results, looks at whether the expectations of the organization's stakeholders were met. In other words, did the training accomplish what they expected it to?
Kaufman’s Five Levels of Evaluation is a response, or reaction to, Kirkpatrick’s model and aims to improve upon it in various ways. With Levels of evaluation: Beyond Kirkpatrick, Kaufman and Keller were aiming to develop a “more effective approach to evaluation” by using an “expanded concept of evaluation”. Their aim was to develop Kirkpatrick’s Model to include result-related questions. They believed that this approach would “contribute to continuous improvement by comparing intentions with results”. (Kauffman, Keller, 1994).
Kaufman’s Model of Learning Evaluation (Kaufman’s Five Levels of Evaluation)
Based on the Kirkpatrick Model, Kaufman’s Five Levels of Evaluation are as follows:
Level 1a: Input
This covers the training materials such as digital resources that are used to support the training or coaching.
Level 1b: Process
The second part of the first level measures process acceptability and efficiency. In other words, the actual delivery of the learning experience.
Level 2: Acquisition
Kaufman’s second level studies the payoffs for both individuals and small groups. The ‘micro-level client’ would usually be the learner, so you would be studying whether they acquired the learning and whether they used it on the job.
Level 3: Application
The third level evaluates how well participants utilize what they learned in their on-the-job performance.
Level 4: Organisational payoffs
Kaufman’s fourth level measures payoffs for the organization as a whole. The ‘macro-level client’ would typically be the business or organization undertaking the evaluation. This level includes performance improvement evaluations and a cost-benefit and/or cost-consequence analysis.
Level 5: Societal Outcomes
Kaufman’s fifth level focused on what he termed ‘mega-level clients’. This could refer to a business’ clientele and/or to society as a whole.
3. How does Kaufman’s model differ from Kirkpatrick’s?
To better understand Kaufman’s model, it’s helpful to start by noting the differences between the Kirkpatrick Model and Kaufman’s work.
There are three main differences:
-
Kaufman divided Kirkpatrick’s Level 1 into “Input” and “Process.”
-
He grouped Kirkpatrick’s Levels 2 and 3 as ‘Micro’ levels.
-
He added a fifth level that evaluates results for both the customer and society.
We’ll take a look at each of these changes separately.
Change 1. Kaufman divided Kirkpatrick’s Level 1 into “Input” and “Process”
Kaufman’s decision to split Kirkpatrick’s Level 1 into ‘Input’ and ‘Process’ is incredibly helpful. What this does is separate the evaluation of resource quality and availability from evaluation of its delivery.
Why?
It is important to assess the quality of resources, whether they are sourced from outside an organization or within the workplace. An otherwise well-run course could fall down due to low quality or insufficient resources. For example, the course material could be the wrong level for the learners.
On the other hand, a training course with perfectly robust materials could be poorly delivered. By separating resources from delivery, Kaufman’s model makes it far easier to see which factor was responsible for the success or failure of a course.
Change 2. Grouping Kirkpatrick’s Levels 2 and 3 as ‘Micro’ levels
Kaufman’s Model groups Kirkpatrick’s Levels 2 and 3 as ‘micro’ levels. Overall, this is less helpful as there are different reasons why course participants may or may not obtain new skills and knowledge from a course compared with the extent to which they apply what they learned on the job.
For example, using Kirkpatrick’s model, a Level 2 assessment would measure the learning that took place on a course while a Level 3 assessment records whether the learning had an impact on the participant’s on-the-job performance. This approach provides useful data that will help resolve any issues. By grouping these as ‘micro’ levels, Kauffman’s model places less emphasis on the reasons why learning isn’t making its way to the workplace.
Change 3. Adding a fifth level that evaluates results for both the customer and society
Arguably, Kaufman’s addition of a fifth level where societal consequences and customer benefits are evaluated makes his model less practical. Obtaining robust data at this level would be too expensive for most organizations. This would most likely result in the collection of lower quality, anecdotal evidence, with questionable results.
Most organizations would simply include the societal/customer impact in their business goal as a part of what they hope the training to achieve. For instance, a supermarket running a customer service training course may state their intention that they expect the training to have a positive impact on society and consumers.
4. Key concepts in Kaufman’s Five Levels of Evaluation
Having outlined Kaufman’s Model and shown how it differs from Kirkpatrick’s, it’s worth exploring the three key concepts that underpin this model.
-
Mirroring Kirkpatrick’s levels
-
Use of micro/macro/mega terminology
-
Sources of data
Let’s look at these key concepts in greater detail.
1. Mirroring Kirkpatrick’s levels
Kaufman’s decision to present his training evaluation model in five levels was clearly an attempt to mirror Kirkpatrick’s four levels. This undoubtedly helped his model gain acceptance from those who were familiar with Kirkpatrick’s and perhaps this makes the model more approachable. However, it doesn’t help our understanding of the model. In fact, it may be more helpful to think of Kaufman’s model as a six-level taxonomy, as follows:
-
Level 1: Input
-
Level 2: Process
-
Level 3: Acquisition
-
Level 4: Application
-
Level 5: Organizational Results
-
Level 6: Societal/customer consequences
This presentation of Kaufman’s model is helpful in a number of ways and can be less confusing than the original way in which it was presented. Firstly, it separates ‘input’ from ‘process’ and places equal importance on both aspects. Secondly, making ‘Acquisition’ and ‘Application’ separate levels (instead of ‘micro-levels’) underscores the need to separate learning from behavior.
2. Use of micro/macro/mega terminology
Kaufman used micro/macro/mega terminology when presenting his work, as follows:
-
Level 3: Acquisition (Micro-level)
-
Level 4: Application (Micro-level)
-
Level 5: Organizational Results (Macro-level)
-
Level 6: Societal/customer consequences (Mega-level)
The decision to present ‘Acquisition’ and ‘Application’ as ‘micro-levels’, can lead to confusion over how best to interpret Kaufman’s model and utilize the data. This decision was likely driven by a desire to mirror Kirkpatrick’s level, yet doesn’t add much to the usefulness of the training evaluation model.
3. Sources of data
One helpful aspect of Kaufman’s model is the way that it replaces the ‘measure of learner satisfaction’ found in Kirkpatrick’s model with Level 1a: Input. This addresses one of the central criticisms of Kirkpatrick’s Level 1 (Reaction). Participant reaction surveys are highly subjective and seldom produce useful actionable data.
For example, data from a Level 1 Reaction survey given to a group of trainees who just completed a data handling training course could garner favorable responses, indicating that the course was successful and well-received. Yet these results could have been affected by a number of unrelated factors such as:
-
The personality of the instructor
-
The quality of the venue
-
The food or refreshments provided
In other words, a positive Level 1 survey may not have any relation to the results of the level 2 survey (learning) despite the levels being presented as a taxonomy, which seemingly indicates that they should. In contrast, Kaufman’s model is looking directly at the learning resources and delivery, separating them from delivery and identifying the strengths and weaknesses of both. This shows that Kaufman realizes that input from learners isn’t the only valuable source of data.
5. Conclusion
While Kaufman’s Five Levels of Evaluation may not be your go-to option for training evaluation, it does offer some useful ideas and concepts that you can apply to your training evaluation practices. For example, evaluating your learning materials separately from the delivery of your training or coaching course can be a valuable approach. This gives you a better understanding of what went right and what went wrong.
By separating these two factors, you can identify problems with your materials and resources earlier in the process and address them before progressing with your evaluation. Kaufman’s model clearly illustrates the value of separating quality standards for your materials from standards for your delivery method.
The other useful lesson to draw from Kaufman’s model is the value of data besides the input you receive from learners. Whichever learning evaluation model you use, be aware of the myriad of factors that can affect learning outcomes. If the evaluation model you choose doesn’t factor in this data, you could find ways to incorporate it and make your evaluation more complete.
Kaufman’s model also highlights a danger common to almost all training evaluation models; know your limits. Just as we wouldn’t recommend conducting a complete five-level Phillips ROI study for more than five to ten percent of your courses, so too would we caution against trying to implement Kaufman’s model in its entirety. Kaufman’s fifth level about societal impact is simply too far removed from your organization’s goals to be helpful or useful. We don’t recommend trying to capture data about the extent to which your training program made an impact on society.
Our recommendation is to include customer impact and benefits as part of the business metrics you evaluate at level 4. You can always include Kaufman’s Level 5 impact as a goal in your business plan. If you're ready to find out how Kodo Survey can help you evaluate the effectiveness of your training programs and learning journeys, request a demo with one of our experts today.
Related Posts