One of the most popular and widely-used methods of evaluating training is the Kirkpatrick Model. In 1959, Don Kirkpatrick introduced his revolutionary four-level method of evaluating the effectiveness of training through a series of articles published in the Journal of the ASTD. Kirkpatrick later expanded his system to become the 1993 book Four Levels of Training Evaluation. This model was quickly adopted by a wide range of enterprises that were looking to evaluate the effectiveness of their training. However, when it comes to the Kirkpatrick model, there is a big difference between expectations and reality. In this post, we’ll explore the differences between what people expect from the Kirkpatrick Model and how the actual system works.
Expectations vs Reality: Why is it important?
People’s assumptions of how a method works or how it should be implemented can have a vast impact on the effectiveness of the model.
In other words: doing it wrong doesn’t work.
The Kirkpatrick model still has the same four levels that Kirkpatrick himself introduced in 1959:
-
Reaction
-
Learning
-
Behavior
-
Results
However, over the years a number of faulty practices have crept into the ways that people are applying the system. People were using the model incorrectly and were making certain assumptions or carrying certain expectations that were causing them to conduct Kirkpatrick training evaluations incorrectly.
Kirkpatrick’s son James and his wife Wendy wanted to set the record straight and address the incorrect expectations that people had about the Kirkpatrick model. This led them to co-author a revised edition of the Kirkpatrick model, called Kirkpatrick's Four Levels of Training Evaluation.
This book addresses some of the flawed expectations that people have about the system and shows people how to carry out training evaluation correctly. The current definitive Kirkpatrick Model is known as the New World Kirkpatrick Model.
In this post, we’ll use this model to show how people’s expectation of the system differs from reality.
Expectation #1. Participant satisfaction correlates with learning success
As we’ve noted, the Kirkpatrick Model contains four levels; reaction, learning, behavior, and results. Most people expect that the first level, reaction, or participant satisfaction, correlates in some way to the other levels; learning (learning success) and behavior (learning transfer). However, empirical research by Michael Gessler shows that no such correlation exists.
Reality: Participant satisfaction has no proven link to learning success
The first level of the Kirkpatrick Model involves asking participants for their reaction to the training. It’s a fairly common expectation that the results from level one will have some correlation with the results seen in level two and three. However, in 2009, Michale Gessler of the University Bremen, Institute Technology and Education, published a paper in the International Journal of Management in Education that challenged this commonly held expectation.
According to Gessler’s study, when the Kirkpatrick method was applied to 43 training courses, the data found “no correlation between reaction and learning and also no correlation between reaction and behavior” (Gessler, 2009). In other words, the way that trainees react to training has no bearing on whether the training produced results.
Here’s the kicker:
Trainees will respond positively to training depending on a wide range of factors, many of which have no relationship with the quality of learning.
For example, trainees will respond positively to:
-
A charismatic or engaging instructor
-
Favorable training conditions
-
Training materials that are engaging and not overly challenging
These factors have no correlation with whether the training was meaningful or delivered results in the workplace.
Likewise, trainees may respond negatively to:
-
Dry or dull instructors
-
Challenging materials
Yet training under these circumstances could result in workplace changes and generate a return on stakeholder’s expectations.
In Gessler’s own words;
“the practice of evaluating professional training based on participant satisfaction requires further development.”
There’s simply no empirical evidence to suggest a link between whether participants enjoyed training and whether the training was effective. This runs contrary to what many people expect this when they start learning about the Kirkpatrick method.
Expectation #2: Some Kirkpatrick levels are more important than others
One of the most common expectations is that one of the four levels in the Kirkpatrick model – Reaction, Learning, Behavior or Results – are more important than others.
Reality: All four levels are equally important
Don Kirkpatrick himself emphasized that if the purpose of the evaluation is to change behavior and improve results, all four levels are of equal importance. Kirkpatrick and his son James addressed this in their 2007 book Implementing the Four Levels where they wrote about building a “compelling chain of evidence as to the value of learning to the bottom line.” (p. 123). They made it clear that the four levels must be worked through in the same order and sequence for the program to work. This will “maximize the meaningfulness” of the results.
Expectation #3: The levels have remained unchanged since the 1950s
The Kirkpatrick mode was first introduced in 1959 and made people assume that the system has remained the same ever since.
Reality: The Kirkpatrick model has evolved over time
When learning about the Kirkpatrick model, you must be aware that Don Kirkpatrick’s original model has evolved over time. Don’s son James and James’ wife Wendy noticed that many people were misapplying the model and failing to implement it correctly. This led them to create the Kirkpatrick Business Partnership Model and the New World Four Levels. These more recent works are the current definitive guide and return the focus to the need to give stakeholders a return on their expectations.
Expectation #4: ROI or ROE are the ultimate indicators of value in the Kirkpatrick Model
Many people believe that the principal aim of the Kirkpatrick model is to help enterprises improve their ROI (Return on Investment) or ROE (Return on Equity).
Reality: Return on Expectations (ROE) is the ultimate indicator of value in the Kirkpatrick Model
In their 2009 book, The Kirkpatrick Four Levels: A Fresh Look after 50 Years 1959-2009, James and Wendy Kirkpatrick wanted to set the record straight and address the many incorrect ways that people were using the Kirkpatrick model. They noted that ROE (Return on Expectations) was frequently confused with the common meaning of ROE, ‘Return on Equity’, which means a comparison of net income with the shareholders’ equity. In fact, the Kirkpatrick Model is designed to measure the Return on Expectations of any training.
ROE is a measurement of the benefits of training delivered through a number of interventions. The cornerstone of this approach is formal training. But the model also requires whoever is carrying out the evaluation to find out the needs of the stakeholders and then decide how to convert their expectations into actual outcomes. This requires a strong partnership between managers, supervisors and the people conducting the training. The Kirkpatrick Model isn’t designed to measure the Return on Equity or the Return on Investment, contrary to what many people believe.
Expectation #5: Learning takes place during the training
The Kirkpatrick Model aims to evaluate the effectiveness of training and is used to suggest ways that the training can be improved. Many people, therefore, expect that the majority of learning will occur during the training itself.
Reality: The majority of learning occurs in the workplace
Many people assume that the bulk of the learning takes place during training. This misconception was addressed by James and Wendy Kirkpatrick in their 2009 book during which they emphasized that formal training by itself will not deliver significant bottom-line outcomes.
Here’s the deal:
The Kirkpatrick Model was designed around the understanding that the majority of learning takes place on the job. This is addressed in Level 3 – Behavior – and the model emphasizes the need for supervisors and managers to play a key role after the training has finished.
The key to learning transfer is with your managers. Managers need to prepare the trainees for the training and the Kirkpatrick model encourages learning professionals, such as the course instructor, to partner with management. Managers also need to help reinforce newly learned knowledge and skills after the training has finished. So, the managers should be involved both before, during and after a training program.
It’s surprisingly common for Level 3 evaluations to suggest that the training has not translated into workplace practices or results. There may be other factors that explain by the training hasn’t had the desired effect.
For example:
-
Was there a lack of partnership between the learning professional and management?
-
Did the management fail to prepare the trainees for the training?
-
Were the trainees supported after the training?
-
Did the managers provide accountability?
Expectation #6. The Kirkpatrick model emphasizes estimates, assumptions, and empirical data
As the fourth Kirkpatrick level is ‘Results’, many people assume that the Kirkpatrick method emphasizes data. This is a fairly common expectation that people have before they learn about the Kirkpatrick Model in greater detail.
Reality: The Kirkpatrick model emphasizes the transfer of learning to workplace behavior
The purpose of the Kirkpatrick model is to analyze whether a training course or program offers good value for money and to what extent it delivers bottom-line outcomes. However, this doesn’t mean that the method emphasizes empirical data over other factors.
As Don Kirkpatrick and his son James proclaimed in their 2007 book, “We are not isolationists”. What they meant by that was that they acknowledged that the results from any particular training course can come from a variety of factors. Their model does not attempt to isolate outcomes (such as estimates and financial data) related to training. Instead, they argue that evaluation must take a holistic approach.
So, how do you measure and know which factors were important?
Following the Kirkpatrick model in the sequence is crucial if you want to apply it effectively. Many factors contribute to the transfer of learning to workplace behavior.
These may include:
-
The skill and quality of the instructor
-
The quality of the trainees
-
The level of support offered in the workplace
-
The preparedness of the management
-
The relevance of the training
The Kirkpatrick Level 4 evaluation does emphasize the measurement of results based on key data such as assumptions and financial data. Instead, the main emphasis is on validating that the learning has translated into workplace behavioral changes. The Kirkpatrick model also emphasizes the need to focus on collective efforts to accomplish a return on stakeholder expectations.
Expectation #7. The Kirkpatrick Model is a type of summative assessment
As the Kirkpatrick Model’s four levels are designed to be applied post-training, many people assume that it is a form of summative assessment that can only assess what has gone before.
Reality: The Kirkpatrick Model can be used as formative, summative and ipsative assessment
The main focus of the Kirkpatrick model is to analyze whether a particular training was effective and to what extent stakeholder expectations were met. It is commonly regarded as a summative assessment that takes place once the assessment has finished. In reality, it offers many different approaches to assessing training and it also provides a way to improve training.
The Kirkpatrick Model may be used in the following ways:
Formative
Formative assessment takes place before any learning has taken place. The first level of the Kirkpatrick Model, Reaction, can be used to gauge trainee’s reactions after training, making it a type of summative assessment. However, Level 2 – Learning – calls for the use of pre-training assessment known as formative assessment. Results from this can be used to tailor the type of training offers, and to improve and refine it.
Ipsative
Ipsative assessment measures people’s performance against themselves at some point in the past. For instance, an annual or biannual assessment of an employee could be used as a type of ipsative assessment to gauge whether they had maintained, improved or declined in their performance over the past year.
The Kirkpatrick model can be applied as a form of ipsative assessment. The second level, learning, can be spaced out over several weeks, months or even longer. This gives an insight into whether trainees have improved, relative to whether they were prior to starting the course.
The third Kirkpatrick level can also be used as a type of ipsative assessment. This level aims to evaluate whether the learning that took place during the training has translated into on-the-job changes. One way to apply this level is to conduct a short workplace assessment shortly after the training. This gives you a baseline as to where the trainees are at a set point in time. You can then conduct another workplace assessment a couple of months after the training.
Conclusion
Addressing these seven key expectations about the Kirkpatrick Model should help you get to grips with the system in record time. If you'd like to know how to maximize the business impact of your training or development, download our free white paper.
Related posts:
The Noob Guide to Understanding Kirkpatrick model evaluation
How to Master Kirkpatrick model of training evaluation in 6 Simple Steps
Kirkpatrick level 3: Free evaluation examples
Kirkpatrick level 4: learn How to measure your ROI training