Training courses and seminars are designed to help participants gain new knowledge and skills, but how effectively they do this requires evaluating, or measuring training effectiveness continuously.
Training is big business for companies and organizations or all sizes. Research shows that large U.S. companies spent an average of $19.7 billion on training in 2018 while midsize firms spent $2.1 million and small companies spent $355,731 on average. Enterprises routinely measure the effectiveness of the training they offer to their employees. This helps them determine their return on investment (ROI) and discover to what impact corporate training & development is having on employee performance.
What is training effectiveness and how do you measure it?
Training effectiveness refers to the quality of the training provided and measuring whether the training met its goals and objectives. One of the most widely used ways to evaluate training is the Kirkpatrick Model. This approach, developed by Don Kirkpatrick in the 1950s, offers a four-level approach to evaluating any course or training programs.
The four levels are:
Level 1: Reaction
How did the participants react or respond to the training?
Level 2: Learning
What did participants learn from the training?
Level 3: Behavior
Did the trainees take what they learned and put it into practice on-the-job?
Level 4: Results
Did the training meet the stakeholders’ expectations? What was the return on these expectations (ROE)?
In the Kirkpatrick Model of Training Evaluation, training is judged against two main factors.
1. The stakeholders’ expectations
The stakeholders are usually the management of a company or organization.
2. The objectives of the training
What was the training designed to accomplish?
Training evaluations are an attempt to get gather data about the effectiveness of the training and the value it offered the organization. If you'd like to learn more about the Kirkpatrick Model, check out our Noob guide to the Kirkpatrick Learning Evaluation Model; you'll be up to speed in no time!
In this post, we’ll show you how to measure training effectiveness and help you evaluate any training course or program in four simple steps.
Step #1. Identify what you need to measure
If you decide to follow an established training evaluation program such as the Kirkpatrick Model then you will have a pre-determined path. You’ll start by measuring the participants’ reactions to the training, then measure what learning took place and what behavioral changes made their way into the workplace environment. Lastly, you’ll measure the overall results of the training and its impact on the business.
However, you don’t need to follow this model. Therefore your first step is to make your evaluation fit for purpose and decided what you need to measure. This is a critical part of choosing the right things to evaluate.
You have five main options, as follows:
1. Identify the expected outcomes and the goals of the training
You must ensure that the training meets the stakeholders’ expectations and/or the business needs and that the evaluation measures against these expectations.
2. Measure participant reaction
This helps you determine whether the training created the conditions necessary for learning to take place. This is the first level (Level 1) in the Kirkpatrick Model.
3. Measure learning
This helps you find out what learning took place. The Kirkpatrick Model’s level 2 (Learning) addresses this area.
4. Measure job impact
This helps you determine whether the training impacted the learners’ on-the-job performance. Kirkpatrick’s Level 3 – Behavior – focuses on this.
5. Measure business impact, and ROI
At this level, you’d examine the impact that the training had on the business. The Kirkpatrick model focuses on Return on Expectations (ROE), not Return on Investment (ROI). However, Jack Phillips expanded upon the Kirkpatrick model in 1980 and added a fifth level that calculated the financial return of any training program.
Which evaluation is right for you?
For a comprehensive measurement of training effectiveness, you may wish to complete all stages. In this case, you’d simply follow through the stages sequentially.
However, some organizations place less importance on some areas and omit them entirely from the evaluation process.
For instance, there is little evidence to suggest that participant reaction has any correlation with learning or job impact. This leads some organizations to skip this step entirely.
Likewise, the fifth option – measuring business impact – may not be necessary for every training effectiveness evaluation. A simple one-day training program is unlikely to require an evaluation of this depth or complexity, for instance.
A wide variety of factors such as the goals, costs, and visibility of the training will determine which type of things you need to measure. ROI evaluations are usually reserved only for training programs and events that are expected to change business performance in some way.
Step #2. Create an evaluation schedule
Once you’ve decided what you want to measure, you need to create a schedule. Each type of training evaluation must be conducted at a specific point in the process.
1. Identify the expected outcomes and the goals of the training
Ideally, you should start this step before the training is developed. You need a clear understanding of the following two areas:
- What are the stakeholders’ expectations?
Everything you measure and assess will be judged against these expectations. Therefore, it is vitally important that you clarify what these expectations are before the training is developed.
- What are the objectives of the training?
Next, examine the goals for the training. Do these match the stakeholders’ expectations? If not, address this during the training development phase.
These two areas will help you define the training needs and clarify how you’ll measure success.
2. Measure participant reaction
Participant reaction is usually measured quite soon after the training is completed. Some courses ask participants to complete a pre-training survey, but this isn’t mandatory.
3. Measure learning
At a bare minimum, you should ask participants to complete a post-training learning test or quiz. This could be a paper-based test, a verbal test such as an interview, or a meeting or focus group. You could also use a practical test where the learners perform a task related to their jobs.
A more comprehensive way to measure learning is to conduct a pre-training test and a post-training test and compare the results from each. This is a form of ipsative assessment where you are measure participants’ current performance against their previous performance. This offers a more objective way of assessing the effectiveness of training.
4 & 5. Measuring job impact, business impact, and ROI
To measure behavioral changes, you should wait two or three months after the training has been completed. This gives the learners time to apply their learning. The same applies to the measurement of business impact and financial benefits such as calculating the ROI of training.
Step #3. Design and deploy your evaluations
Your third stage is to design and deploy your evaluations in the areas you want to measure.
Identifying stakeholder expectations
The first stage of identifying the stakeholders’ expectations, the training needs and the measures of success can be accomplished through one or a series of meetings with the stakeholders and the training course providers. You need to analyze the needs of business managers and those of the training participants to ensure there’s a good fit between expectations and reality.
Evaluating participant reactions
To measure participant reaction, the most straightforward approach is to design a simple survey. Dedicated platforms such as KodoSurvey help automate much of the process of designing and deploying evaluations. They automate many of a processes such as sending the surveys, reminders and everything else you need to make your evaluation a success.
Measuring learning
To measure learning, you need to develop learning evaluations to measure the knowledge and skills that the training is intended to impact. Regardless of whether you design these evaluations yourself or use a platform, you’ll need to liaise with the course trainers and/or managers to help facilitate these evaluations. One pre-training and one post-training evaluation should be sufficient to measure whether learning has taken place.
Evaluating behavioral changes
To measure behavioral changes such as job impact, you’ll need to evaluate training participants in the workplace. There are multiple approaches to this aspect of measuring training effectiveness including:
Some examples of level three evaluation strategies include:
- Workplace observations
- Peer observations
- Self-reflections
- Pre- and post-training assessments
- Pre- and post-training self-assessments
If you’d like more tips about designing an assessment, check out our Kirkpatrick Level 3 Free evaluation examples. These examples offer ways to implement a behavioral evaluation.
One of the easiest approaches is to use a post-test survey two or three months after the training took place. You would ask the participants to judge their own on-the-job performance against KSAs (Knowledge, Skills, and Attributes).
As surveys are self-reported and a form of self-reflection, this wouldn’t give you the objectiveness or a professional assessment. This makes it simple and cost-effective, but it relies on trust and responses would require further work to independently verify.
Some sample questions you may ask include:
- Briefly describe the training you completed
- Did the training provide any lessons related to your on-the-job roles?
- If yes, please describe them.
- Did the training improve your ability to perform your job?
- If so, please write how
Measuring business impact and ROI
You’ll need to collect financial data relevant to the stakeholders’ expectations that help you calculate the ROE (Return on Expectations) and/or the ROI (Return on Investment). This usually entails meeting with business managers.
Step #4. Data analysis and reporting
The last step of measuring training effectiveness is to analyze the data you collected and generate reports for relevant parties. Your approach to this step will depend on the size of your organization, your available time and budget, the number of participants and the stakeholders’ requirements.
As we mentioned, not all training requires all levels of evaluation and, similarly, not all training data needs to be analyzed and reported in the same way.
Some example, some organization stakeholders may simply request that the evaluation data is processed and passed on as raw statistics. This information could be presented during a meeting as a printed document with the findings clearly presented. For example, “27 percent of participants increased their levels of knowledge and understanding as a result of the training”.
Other stakeholders may request a more detailed report where the findings are analyzed in more details. They may wish to see what impact a training course had on business metrics, for instance.
If you are struggling with simple spreadsheets, limited LMS reporting (Learning Management System) and need help analyzing and interpreting the data, consider using a dedicated platform such as KodoSurvey. This will help you collect and analyze everything need to make data-driven decisions and become a strategic learning partner.
Participant reaction
If you used a platform like KodoSurvey to design and deploy your participant reaction surveys, the platform will help crunch the data and generate reports that you can share with the stakeholders, management and the participants themselves.
Measuring learning
Many course trainers are able to administer both the pre-training and post-training evaluations. They can analyze the results themselves to identify areas where learning took place. This data can be used to generate a report that expresses the effectiveness of the training in terms of facilitating learning.
Measuring job impact
Assessing the participants’ on-the-job performance can be accomplished in a number of ways. Responsibility for gathering and analyzing data is routinely handed to the participants’ supervisors or managers. They can generate reports that help identify what impact the training had on workplace performance.
An alternative and costlier approach is to hire a trained evaluator to conduct workplace observations and assessments. Arguably, this approach may produce more accurate but the additional expense may be a barrier for some organizations.
Measuring business impact, and ROI
If you are following the Kirkpatrick model and simply want to evaluate the training according to stakeholders’ expectations, you or the business managers can gather financial data and draw links between the benefits of the training and the evidence.
For example, following a training course aimed to improve customer satisfaction, you may expect customers to report a more favorable impression on the company and staff. You could gather this data through surveys and argue that the training met stakeholders’ expectations.
If you want to extend this approach to level 5 and look at ROI, you may need an LMS or training evaluation platform to help process training costs and business data to illuminate whether a training program or event changed overall business performance in some way.
Over to you!
If you'd like to know how to maximize the business impact of your training or development, download our free white paper.
Related posts: