Research shows that when it comes to allocating resources to training, the highest priority among U.S. companies in 2019 will be increasing the effectiveness of their training programs. Here’s how to evaluate any training course or program in four simple steps.
Why evaluate training effectiveness?
According to Training magazine’s 2018 Training Industry Report, total U.S. training expenditures last year were 6.4 percent lower compared with 2017. Companies spent an average of $986 per learner, compared with $1,075 the previous year, yet they provided their trainees with almost the same number of hours of training.
How?
The answer lies in the way that companies worked to increase the effectiveness of their training programs. According to the report from Training magazine, 34 percent of companies plan to prioritize the allocation of resources towards this area in 2019.
One of the best routes to increasing the effectiveness of any training program is through evaluation. With so much money and effort devoted to training (U.S. employees received an average of 46.7 hours of training last year) the real issue at stake becomes clear: Did they really learn anything?
The cost of ineffective training
Training magazine’s report highlights the potential dangers facing firms that run ineffective training programs. While government and military training programs were the costliest at $1,433 per learner, the data shows that small companies spent more ($1,096) than midsized ($858) or large firms ($1.046). In other words, if learners aren’t receiving benefits or putting what they learn into practice, companies stand to realize a poor return on their investment in training.
That’s why learning evaluation models, such as the Kirkpatrick Model, have found favor among companies and organizations of all sizes. Evaluation models give companies a way of measuring whether a training program has met its goals and the goals of the stakeholders. Evaluation models can help improve the quality of the training provided and enhance the outcomes for both the learners and stakeholders alike.
Introducing the Kirkpatrick Learning Evaluation Model
By far the most widely recognized method for evaluating training effectiveness is the Kirkpatrick Training Evaluation Model. It was created by Don Kirkpatrick in the 1950s and introduced through a series of articles published in the Journal of the ASTD. Over time, Kirkpatrick’s approach to evaluating training effectiveness was developed into the 1993 book Four Levels of Training Evaluation.
Kirkpatrick’s model offers a four-level method for evaluating any training program or course. It provides a useful blueprint for assessing any training program. The Kirkpatrick model has undergone significant improvements and changes over the years. In 2007, Don’s son James and his daughter-in-law Wendy co-authored the book, Kirkpatrick's Four Levels of Training Evaluation. The model is currently known as the New World Kirkpatrick Model.
Here’s how it works:
The Kirkpatrick model features four levels:
-
Level 1: Reaction
-
Level 2: Learning
-
Level 3: Behaviour
-
Level 4: Impact
Here’s a brief explanation of each level:
Level 1: Reaction
The first level of the Kirkpatrick Model is known as the ‘Reaction’ level. This is the most basic way to evaluate any type of training as it simply establishes how the learners responded to the training. The most common approach is to ask participants to answer a series of training effectiveness survey questions.
Level 2: Learning
The second level of evaluation in the Kirkpatrick Model attempts to establish whether learning took place. Can the participants demonstrate a greater knowledge or understanding of the course materials after training?
Level 3: Behavior
Level 3 of the Kirkpatrick learning evaluation model looks at whether the participants made use of their training when they returned to work. Did they apply their knowledge, understanding or skills to their jobs? This help you take simple steps to measure training effectiveness.
Level 4: Results
The Kirkpatrick model focuses on ROE – Return on Expectations. In level 4, the model looks that the expectations held by stakeholders and examines whether the training helped to meet them, and if so, to what extent.
As you can see, the Kirkpatrick model provides an extremely comprehensive way of evaluating the effectiveness of any training. But it clearly cannot all be accomplished in just four simple steps.
Therefore, we’ll have to focus on the first level of the Kirkpatrick Model and ask:
Did the training have the right conditions for learning to take place?
How to evaluate training in four steps
Conveniently, level one evaluations can be broken down into four clear steps:
-
Step #1. Deciding your key questions and data collection methods
-
Step #2. Creating your evaluation
-
Step #3. Deploying your evaluation
-
Step #4. Data analysis and reporting
Let’s see how this looks in practice.
Step #1. Deciding your key questions and data collection methods
The goal of Level 1 evaluations is to gather reactions to the training and see how the learners responded to the training methods, the course instructors, the facilities and the administration of the training.
The specific questions you ask will depend on three main factors:
1. What are the business stakeholders’ expectations of the training?
2. What are the objectives of the training?
3. What are the overall objectives of the evaluation?
It is vital that you align the evaluation objects closely to the key business stakeholders’ expectations of the training and original training objectives. It should flow from one to the next – expectations, training objectives, evaluation objectives.
To illustrate this, let’s consider a scenario where a call center’s stakeholders have decided to deploy a training course that helps their employees improve levels of customer satisfaction.
The stakeholders’ expectations may be that the training will result in a positive improvement in customer satisfaction.
The training’s objectives would be to help operators answer customer questions faster and more accurately.
Therefore, the objectives of the evaluation should be to find out whether the learners (the call center operators) felt that the training was conducive to learning.
To obtain the necessary feedback, the sequential approach outlined below will help you decide which questions to ask. These may include:
Did the learners enjoy the training?
Did they feel it was a useful way to spend their time?
Did they find the training relevant? Why/why not?
Was the training practical? Could they use it in the workplace?
Would the learners recommend the training? Why/why not?
Did the training motivate them to continue with their learning?
For more sample questions that will help you evaluate participants' reaction to training, download our free training effectiveness evaluation form. This is one of the best methods to evaluate training effectiveness.
How will you collect your data?
Once you have chosen the most appropriate questions to ask, you need to decide how you will collect your data. These two tasks (the questions and the data collection methods) are interrelated so must be considered as one step. You can only formulate your questions when you know how you’ll collect it: a good question for a survey or questionnaire may not work well in the focus group, for example.
There are many good reasons to evaluate training effectiveness The most common approach is to design a simple survey or questionnaire that learners can complete during or after the training.
Alternatively, you can use:
-
meetings
-
written reports
-
interviews
-
focus groups
To a large extent, your data collection method will depend on the size of the organization you are working within, and your budget. Interviews and focus groups will necessitate more time and money than simple surveys, for instance.
Step #2. Creating your evaluation
Your second step is to create your evaluation, according to the questions and data collection methods you chose in Step 1. You can produce a handwritten or typed evaluation yourself or use a platform for measuring training effectiveness and learning impact such as KodoSurvey.
How to create a survey
Begin by choosing an appropriate format. It could be a simple printed paper survey, an emailed survey, or an online survey via a platform. You’ll need to spend time creating effective questions and possibly conduct a pilot test to iron out any issues before you deploy your survey.
Longer evaluations tend to have lower completion rates but shorter tend to reduce the quality of the results. So you’ll need to find a balance in your number of questions – so that it drives high quality without reducing completion rates. KodoSurvey will help you automate many stages of creating and deploying a both survey’s that evaluate your training on Kirkpatrick’s first level and tests that evaluate training effectiveness and learning impact on job performance.
Meetings
If you chose to hold a meeting, you’ll need to draft an agenda with the points you want to discuss. Take account of the number of attendees and give time for them to offer feedback about their training experience.
Written reports
If you’d prefer to have learners complete written reports, you’ll need to create written guidance that helps participants understand your expectations. Your writing should have clear instructions about the areas that you want the learners to cover.
Interviews
Interviews can be an excellent way of obtaining high-quality feedback about learners’ reactions to training. Interview questions will be roughly similar to survey questions, except the interviewer will be on-hand to explain any terms or concepts that the interviewees may need explaining.
Focus groups
Focus groups lend themselves well to large organizations with high numbers to learners. You can use sampling techniques to understand typical learner reactions to training without having to gather data from every participant.
Step #3. Deploying your evaluation
Some companies are spending millions on training without knowing its full effect. That's why using an evaluation is essential. Your approach will depend on the type of evaluation you chose, the size of your company and your time and budget restraints. Ideally, you should deploy your evaluation to all learners, although, as we’ve noted, you may use sampling techniques if there are large numbers of participants.
Your evaluation should take place at the end of training, or as soon as possible after the end of the training so that it is still fresh in peoples’ minds.
A survey or questionnaire can easily be sent to all participants, while written reports, interviews, and focus groups lend themselves well to sampling methods.
Confidentiality
Learners are unlikely to provide honest answers if they believe that their responses may be collected and used against them. Reaction evaluations that offer anonymity and confidentiality have a much higher completion rate and provide much more useful, honest data than evaluations where the identity of the trainee is known.
For surveys or questionnaires, you should avoid using or requesting the user’s personal information when asking for their responses. You should not ask learners to enter any information that could be used to identify them such as their name, or email address.
You can also offer both written and verbal assurances in other scenarios such as meetings, reports, interviews and focus groups that any information they provide will be held in the strictest confidence and will not be used against them in any way. It is important to make the learners aware of any steps you have taken to ensure confidentiality.
Step #4. Data analysis and reporting
The fourth and final step of evaluating at Level 1 is to collect the responses and create a report to share with key stakeholders. While few companies currently complete this step, we believe that measuring training effectiveness will soon become standard. You may also wish to discuss the results of the evaluation with the participants, the course trainers, and the participants' managers.
If you used KodoSurvey to help create and deploy a Level 1 survey, you will automatically get a custom reports using the data. If you used another method, the onus is on your or your staff to collate and interpret the results of the data and draw your own conclusions.
The goal of evaluating participant reaction to training is to identify potential barriers to learning and help learners feel that their needs and preferences are being met. It is highly important to act upon the outcomes and take the necessary action.
For instance, if the learners felt that the training didn’t make especially good use of their time, find out which areas could be improved to make it more worthwhile and relevant in the future.
If the participants reported that they didn’t feel the training was very practical, take this data into account and make future training more effective.
Evaluating training effectiveness to improve training
This simple four-step approach to evaluating training can help any company or training provider understand the strengths and weaknesses of their training and learn how it can be modified in the future. All training programs and courses can be refined and improved so that they meet stakeholders’ expectations. However, keep in mind that research by Michael Gessler shows now correlation between satisfied participants and the use of learning on the job. If the training is of strategic character or for a high volume of participants (i.e. high cost) we always recommend measuring on at least level 2 and 3 as well. Our learning analytics tool KodoSurvey automates this for you.
At the heart of the Kirkpatrick model is the goal of accomplishing a return on stakeholders’ expectations, or ROE. If training failed to meet these expectations, you can find out what can be done to improve it in the future.
To learn more about how to evaluate training effectiveness, our white paper Maximizing business impact from training and development outlines everything you need to consider when evaluating training. Download it today and take your training evaluation to the next level!
Related posts:
- 10 Training Effectiveness Survey Questions to Ask
- How to Measure Training Effectiveness in 4 Simple Steps