If you’re responsible for delivering high-quality training, you’ll understand the importance of measuring the effectiveness of the training you provide. Enterprises of all sizes understand that without formally assessing training effectiveness, they cannot calculate whether it was worth the investment. This is known as Return on Investment (ROI) and in today’s post, you’ll learn how to measure your ROI with the Kirkpatrick model.
What is Kirkpatrick’s training evaluation model?
In 1959, Don Kirkpatrick introduced his learning evaluation model through a series of articles published in the Journal of the ASTD. He later expanded the model and, in 1993, published the Four Levels of Training Evaluation. This book revolutionized how businesses evaluate their training programs.
The Kirkpatrick model features four distinct levels:
-
Level 1: Reaction
-
Level 2: Learning
-
Level 3: Behaviour
-
Level 4: Impact
Here’s a brief overview of each level:
Level 1: Reaction
The starting point for the Kirkpatrick Model is to find out how the course participants reacted to the training. Were they satisfied or dissatisfied? Why?
Level 2: Learning
The second Kirkpatrick level investigates what learning took place. The most common approach is to ask the trainees to take two quizzes or tests, one before and one after the training.
Level 3: Behavior
The third level looks at whether any of the skills or learning from the training makes it into the workplace. It studies the on-the-job performance of the trainees and sees whether they are applying what they’ve learned while at work.
Level 4: Results
The last Kirkpatrick level examines whether the stakeholder’s expectations were met and if so, to what extent.
So, what do you notice?
The Kirkpatrick model doesn’t emphasize the measurement of Return on Investment (ROI).
What does the Kirkpatrick Model actually measure?
The Kirkpatrick Model model primarily measures training effectiveness and impact based on shareholders' expectations. Don Kirkpatrick termed this metric ROE - Return on Expectations.
Here’s the deal:
The stakeholders in any organization have certain expectations of what any given training should deliver. The Kirkpatrick model emphasizes the need to accomplish a return on these expectations.
Over time, Don Kirkpatrick’s original model has evolved. Don’s son James and Wendy (Don’s daughter in law) have expanded the model to create the New World Four Levels and the Kirkpatrick Business Partnership Model. Interestingly, the latest versions of the Kirkpatrick Model still focus squarely on ROE – Return on Expectations.
But doesn’t 'ROE' mean 'Return on Equity'?
Yes. This is where confusion about the Kirkpatrick Model comes from. ROE commonly means ‘return on equity’ which is calculated by comparing shareholders’ equity to the net income of a company. The Kirkpatrick Model uses ‘ROE’ to mean ‘return on expectations’ and describe these expectations – those of the shareholders – as the ultimate indicator of value.
So, what’s ROE in the Kirkpatrick Model?
ROE is a holistic measurement of all quantitative and qualitative benefits that a training course or program provides and to what extent they meet shareholders’ expectations. Formal training is usually at the core of the Kirkpatrick approach but it can also include any initiative or program that comes about through intervention.
To measure ROE in the Kirkpatrick Model, those responsible for the learning must clarify the following areas:
-
What are the expectations of the stakeholders?
-
How can these expectations be converted into business outcomes?
-
Are these outcomes measurable or observable?
-
Can the learning process be refined or improved to increase ROE?
If you notice, the Kirkpatrick Model focuses on determining whether training produces workplace changes and whether these changes meet stakeholders’ expectations. There is no emphasis on financial formulas, assumptions or estimates. Crucially, ROI isn’t a staple of the Kirkpatrick approach.
How does ROI fit in with the Kirkpatrick Model?
The Kirkpatrick Model was already well established by the time that Jack Phillips published his own work on training evaluation in 1980. Phillips wanted to expand the Kirkpatrick Model beyond Level 4 and offer enterprises a way of calculating the financial return of a training program. This is known as ‘return on investment’ (ROI) and Phillips labeled his methodology ‘Level 5’.
What is Level 5?
Level 5 isn’t featured in the original Kirkpatrick Model but is an additional level of evaluation created by Phillips that uses cost-benefit analysis to determine the value of training programs. In other words, it helps companies calculate whether the money they spent on training produced measurable business results.
Look:
Let’s imagine that a company pays for an expensive, high-profile training program. The original Kirkpatrick Model would only evaluate the effectiveness of the training, whether it translated into workplace changes and whether these changes met the expectations of the company’s stakeholders.
Phillips’ Level 5 training evaluation goes one step further:
Level 5 evaluation helps a company determine whether the money they spent on the training translated into real-world benefits such as increased revenue or decreased operating costs.
What other changes did Jack Phillips make?
In addition to adding a fifth level, Phillips also make substantial changes to the original Kirkpatrick Model.
Level 3
While the original Kirkpatrick Model looked at behavior in the workplace, Phillips expanded this level to cover Application and Implementation. This subtle difference made it easier for companies to see why training wasn’t being converted to on-the-job changes. Was it being applied incorrectly? Or implemented ineffectively?
Level 4
Phillips also broadened Level 4 and renamed it ‘Impact’. This expanded version of Level 4 helped identify whether processes other than training were responsible for bringing out learning and outcomes.
Introducing the Phillips ROI Methodology
To understand how to measure ROI using the Kirkpatrick model, you need to understand what’s known as the Phillips ROI Methodology. This methodology helps to isolate the effects of any given training program.
This differs substantially from the original Kirkpatrick Model:
Don and his son James wrote in their 2007 revised book that “We are not isolationists”. What they meant was that they took a holistic approach to training evaluation and didn’t try to pinpoint specific data that led to results.
Phillips’ ROI Methodology aims to isolate the effects of a program so that the cause of improvements identified in Level 3 and Level 4 can be correctly attributed.
How to calculate ROI using Phillips’ Methodology
Phillips’ methodology uses a variety of techniques to isolate the effects of a training program or course.
These include:
-
Estimates that have been adjusted for error
-
Control groups
-
Trend line analysis
-
Forecasting models
To measure ROI, additional information is required.
ROI compares the financial benefits of training to the costs of the training itself. The same scales or metrics can be used to measure both benefits and costs as they are both measured with money. To implement Phillips’ Level 5 ROI calculation, businesses must essentially conduct a type of cost-benefit analysis. This is known as ROI Determination or Level 5 for short.
How to measure the ROI of your training
ROI determination helps calculate whether training has impacted business revenues or profit and how this compares to the cost incurred. Demonstrating the ROI of training is difficult as you are trying to link the training to a quantifiable factor, such as a process improvement or productivity improvement. This gives you a monetary figure that reflects the value of the training to the company’s bottom line.
Step #1. Decide how you will measure ROI
There are various ways to measure ROI and how you measure it will impact on which factors you measure, before and after training.
The standard formula for measuring the ROI Of training is as follows:
ROI (percentage) = ((Monetary benefits – Training costs)/Training Costs) x 100.
Alternatively, you can measure ROI in terms of the decreased cost or time to produce a product, either on a per-item basis or in bulk.
Step #2. Choose which factors you’ll measure
This step depends on what training is offered and the expectations of the stakeholders. Which areas do they want the training to impact?
For example, let’s say that you want to calculate the ROI of a training course for a pottery business. The company employs five potters and they make an average of 40 pots each over a typical 40-hour, 5-day week, or an average of one pot per hour.
The training course aims to teach the potters how to use a new, faster glazing technique to glaze the inside of the pots. The stakeholders’ expectation is that this new technique should increase the rate at which potters can make their pots.
Step #3. Calculate your company’s pre-training production cost analysis
Collect detailed financial information about your company’s costs including overhead, distribution, facilities, equipment, materials, and wages. Calculate the company’s gross profits prior to the training taking place.
Let’s say that the pots cost the company $5 to make and that each pot can be sold for a profit of $5 after all other costs (distribution, facilities, and overheads) have been accounted for. Over a 50-work week year, each potter generates of profit of $10,000 for the business.
Step #4. Calculate the cost of the training and then provide the training
Your next step is to calculate the cost of training the potters to use the new glazing technique. If it costs $1,000 to run the course for all five potters, the cost of training is $200 per employee.
Once you’d established this cost, you’d then run the training course as planned.
Step #5. Conduct post-training production cost analysis
At some point following the training, you’d run the same analysis of the company’s costs as you did before the training. In our example about the glazing training course, you should be able to observe that the potters are able to complete more pots in the same amount of time, using the new technique they have learned.
You can compare pre- and post-production rates and determine whether an improvement was made.
Step #6. Calculate the new benefit to your company
You can now compare employee productivity since receiving the training with their pre-training performance. You should see a clear increase in both employee productivity and the company’s profits. This helps you calculate the net benefit to your company as a result of the training.
For instance, following the introduction of the new technique, the potters are able to complete an average of 45 pots per week. Over a 50-work week year, each potter will now be generating an average annual profit of $11,250 for the business. This is £1,250 more profit compared with before the training.
To calculate the increased profit that the employee now makes for the company, you should use the net benefit – the increased profit of $1,250 per potter – and subtract the training costs ($200 per employee).
Apply the ROI formula noted above and calculate the percentage return on your investment.
ROI (percentage) = ((Monetary benefits: $1,250 – Training costs: $200)/Training Costs: 200) x 100.
This gives an ROI of 5.25%. ROI is usually reported as a percentage and represents the annual net benefit of the training beyond the initial investment.
This demonstrates that the training was worthwhile and resulted in increased profits for the company.
You will see a positive ROI when the impact on the company’s profits is higher than the cost incurred from offering the training.
How does the ROI statistic help?
Now that we’ve explored how to measure your ROI, you can use it to compare the effectiveness of the training to other programs either within your company or from other companies. ROI can also be used to make investment decisions and forecast benefits.
Do all training programs need an ROI study?
It’s worth noting that according to Phillips, his ROI Methodology isn't required for all training courses. As the highest possible evaluation level, a wide variety of factors such as the goals, costs, and visibility of the training will determine whether an ROI study makes sense.
Listen:
A simple in-house professional development course may not require an ROI study. Whereas a costly course with far-reaching implications may warrant one. It depends on what Phillips termed the ‘Chain of impact’.
Phillips’ ‘Chain of Impact’ vs Kirkpatrick’s ‘Chain of Evidence’
Phillips’s guiding principle for deciding which training courses require a comprehensive ROI study is known as the ‘Chain of Impact’. This is quite similar to Kirkpatrick’s ‘Chain of Evidence’. It can help you decide which training course requires which level of evaluation.
In her book, The Bottomline on ROI, Patti Phillips’ suggested that between 90 and 100 percent of training courses should be evaluated at Level 1 (Reaction), yet this number drops as you climb the levels of evaluation. According to Phillips, only five to ten percent of training courses require a Level 5 ROI evaluation.
Which approach should you use?
The choice between using the four-level Kirkpatrick Model or the five-level Phillips’ methodology depends on your stakeholders’ expectations.
Do they want to focus on simply getting a return on their expectations (ROE)?
Or, do they want to see a return on their investment (ROI)?
If it’s the former, then Kirkpatrick’s makes sense. If the latter, Phillips’ expanded version of Kirkpatrick is your best option.
Summary
Hopefully, this post has given you a good understanding of how to measure your ROI using an expanded form of the Kirkpatrick model. As we’ve shown, ROI calculations are unnecessary for the vast majority of training courses but can be a helpful way to helping an organization reach its program and business goals.
Need More?
For more tips on how to use the Kirkpatrick model to evaluate training and maximize business impact, download our free white paper.
Related posts:
The Noob Guide to Understanding Kirkpatrick model evaluation
How to Master Kirkpatrick model of training evaluation in 6 Simple Steps
Kirkpatrick levels of evaluation: Expectation vs reality
Kirkpatrick level 3: Free evaluation examples