If you’ve been following our blog series, you’ll know that we’ve covered a range of learning evaluation models that represent a wide range of approaches, in particular, the Kirkpatrick Model. While there are many other learning models that we’ve yet to write about, one of the ones we’re often asked about is the Phillips Learning Evaluation Model.
In past posts, we identified the Phillips ROI model as one of the best ways of evaluating training effectiveness, alongside the Kirkpatrick Model and the CIPP Evaluation Model. That’s why we thought we’d present a complete Philips ROI Model tutorial for beginners. We’ll show you exactly what it is and explain how you should apply it.
This tutorial will cover the following areas:
- What is the Phillips Model?
- What are the levels of the Phillips ROI Methodology?
- How does the Phillips ROI Model work?
- Example of the Phillips ROI Model in action
- Common criticisms of the Phillips ROI Model
- Conclusion
1. What is the Phillips Model?
In past articles, we discussed how Don Kirkpatrick’s training evaluation model became well established as the dominant form of training evaluation during the 1970s and 1980s. When Jack Phillips published his own book about training evaluation in 1980, he wanted to build up Kirkpatrick’s work and address several of its shortcomings. His approach became known as the Phillips ROI Model, and is best thought of as an expanded version of the Kirkpatrick Model.
To understand the Phillips ROI Model, it’s worth recapping the Kirkpatrick Model. Don Kirkpatrick’s learning evaluation model offers organizations an effective four-step strategy for evaluating any course or training program.
The four levels of the Kirkpatrick model are:
Level 1: Reaction
Level 2: Learning
Level 3: Behavior
Level 4: Results
Here’s how each level works:
Level 1: Reaction
At this level, you gauge how the participants reacted or responded to the training. This helps you identify whether the conditions for learning were present.
Level 2: Learning
The second stage is to gauge what, if anything, the participants learned from the training. You may use short quizzes or practical tests to assess this.
Level 3: Behavior
The third stage takes place a while after the training. Using various assessment methods, you try to assess whether the course participants put what they learned into practice on-the-job.
Level 4: Results
Lastly, you need to evaluate whether the training met the stakeholders’ expectations. The goal of the Kirkpatrick model is to determine the return on these expectations, known as ROE (Return on Expectations).
But Jack Phillips felt that the Kirkpatrick's Learning Evaluation Model was incomplete. One of the biggest changes he made was adding a fifth level of evaluation, return on investment (ROI). He made other tweaks and changes to other levels too, as we’ll see in the following sections.
2. What are the levels of the Phillips ROI Methodology?
Jack Phillip’s decision to add a fifth level to the Kirkpatrick Model is by far the most commonly quoted aspects of the Phillips ROI Model. The Phillips model has five levels that broadly follow the scope and sequence of the Kirkpatrick model.
The five levels of the Phillips ROI Methodology are as follows:
Level 1: Reaction
In common with the Kirkpatrick taxonomy, the Phillips ROI Models begins by evaluating the participants’ reaction to the training they received. The most common approach is to use short questionaries or surveys to collect data about what people thought about their training. This helps an organization assess whether the conditions necessary for learning were in place.
Level 2: Learning
The second level of the Philips ROI Model evaluates whether learning took place. This is exactly the same as the Kirkpatrick Model and usually entails giving the participants multiple-choice tests or quizzes before and/or after the training.
Level 3: Application and Implementation
Level 3 of the Kirkpatrick Model looks at workplace behavior and assesses whether participants are using what they learned on-the-job. One of the main criticisms of the Kirkpatrick Model has always been that it doesn’t gather enough data to help improve training; it simply tells you whether the training was put into practice or not. Jack Phillips felt that this level could be improved, so he expanded it to cover both application and implementation.
The Phillips ROI Model makes it far easier to see why training does or doesn’t translate into workplace changes. If there is a problem, the Phillips ROI Model helps you determine whether the issue lay with the application of the learning or with its implementation? This subtle but crucial difference can prove extremely helpful for organizations.
For example, let’s imagine that employees in a factory receive training to operate a new type of equipment. The Phillips Model level 2 evaluation confirms that the training was successful, but the level 3 evaluation shows that the employees are not using their new skills and knowledge. A Kirkpatrick level 3 evaluation would simply tell you that the training wasn’t being implemented. But a Phillips ROI Model level 3 evaluation would show you more about where the problem lay. For instance, the employees could be able to operate the new machines but are faced with a problem that prevents them from implementing it, such as the machines being faulty or inoperable.
Level 4: Impact
While the fourth level of the Kirkpatrick taxonomy focuses purely on results, the Phillips ROI model is much broader and looks at the impact of the training on the organization. This helps identify whether factors other than training were responsible for delivering the outcomes. This represents a substantial improvement over the Kirkpatrick model as it allows you to see whether factors such as sales data, weather or supply issues were responsible for changes in performance metrics.
Level 5: Return on investment (ROI
Unlike the Kirkpatrick taxonomy that simply measures training results against stakeholder expectations (ROE), the Phillips ROI model contains a fifth level. This is designed to measure ROI – return on investment. This level uses cost-benefit analysis to determine the value of training programs. It helps companies measure whether the money they invested in the training has produced measurable returns, and if so, what they are.
3. How does the Phillips ROI Model work?
In a nutshell, the Phillips ROI model works by focusing on the following areas:
- Data collection
- Isolating the impact of training versus the impact of other factors
- Accounting for intangible benefits
- Calculating the return on investment (ROI)
To meet the end goal of calculating ROI, the Phillips model calls for business data to be gathered before, during and after the training. This data is analyzed for quantifiable factors such as:
- Process improvements
- Productivity improvements
- Increased profits
Using this data, you can then compare the cost of the training with the monetary benefit that is provided. This gives you an indication of the value of the training, and the impact it had on the organization's bottom line.
To understand more about how this works, let’s look back at the fourth level of the Phillips ROI Model. Level 4 – Impact – is all about determining the business impact that a learning program had on a company or organization. Phillips’ ROI Model states that this impact can be translated into monetary terms and compared to the total cost of running the training.
These costs could include:
- Program development
- Program delivery
- Labor costs
- Time for participants to complete the training
Using the Phillips’ ROI model, you can then use the standard ROI formula to calculate the training’s ROI.
4. Example of the Phillips Model in action
Let’s do the math and see how the Phillips ROI Model works in practice.
Consider the following real-world example: A vacuum cleaner manufacturing company wants to improve sales of a particular product line. They need a training course that improves their sales team member’s ability to recall specific details about these products. The assumption is that if the sales team are better informed about the product, sales will increase, and so will the earnings.
Training development costs: The company pays a third-party vendor $10,000 to develop a training program for its sales team members. Let’s assume that the training won’t require updating and all internal costs that support the training are included in the fee.
Implementation Costs: For simplicity’s sake, let’s say that all implementation costs, such as receiving help for running the resource, amount to $2,000 over the course of one year.
Time Costs: If the company has 20 sales staff across its organization each receiving an average hourly rate of $20, then with each staff member taking the ten-hour training once, the total time cost for training all 200 staff members would be 20 people x $20 x 10 hours = $4,000. This time cost is an important factor as you must still pay your operators their hourly wage while they are completing their training.
However, to maintain the smooth functioning of its business, the company will need to hire replacement workers to cover their employees while they take the training. This can double the costs, so 20 people x $20 x 10 hours = $4,000. Therefore the organization’s total time costs would be $8,000.
Cost-benefit: Let’s say that the organization runs the training course for two years and sees a 10 percent increase in sales during that time. This amounts to an extra $50,000 gross profit.
To calculate your ROI, you would use the following forumula:
ROI % = ($50,000 - $20,000)/$20,000
This gives you an ROI of 150 percent.
5. Common criticisms of the Phillips ROI Model
Although the Phillips ROI Model is often accepted as an enhanced version of the Kirkpatrick Model, it is not without its detractors and there are several common criticisms of the model, and of calculating training ROI in general.
Here, we’ll look at two of the most common criticisms. They are:
1. Timing
2. Usefulness
Let’s look at each of these separately.
1. Timing
One of the most common criticisms of the Phillips ROI Model have been that it calculates ROI only after the program has been delivered. If you calculate that a program cost more to deliver than the value it delivered, it is too late to make any changes.
Critics argue that it would be possible to work out rough estimates of the ROI of different training courses before selecting a training program. However, Jack and Patti Phillips, acually wroteThe Consultants Guide to Results-Driven Business Proposals: How to Write Proposals That Forecast Impact and ROI, that focused exclusively on forecasting, in 2010.
These ROI estimates are of course extremely useful when planning a learning program. You could use them to determine business goals and budgets and decide whether to go ahead or revise your plans before making a financial commitment.
2. Usefulness
Another common criticism of the Phillips ROI Model is that knowing the ROI of a training course offers little practical benefit to a business or organization. It’s nice to know, but it’s not essential. After all, if you have evaluated the training course on Phillips first four levels and already know the program was a success from the Level 4 metrics (Impact), there’s little need to spend the time and costs involved in calculating level 4, the ROI.
According to Jack Philips himself, ROI calculations aren’t required for all training courses. He says that they recommend all programs to be evaluated at Level 1, about 90 percent of programs at Level 2, 30 percent of programs at Level 3, 10-20 percent of programs at Level 4, and even, 5-10 percent of programs at Level 5. As the highest possible evaluation level, a wide variety of factors such as the goals, costs, and visibility of the training will determine whether an ROI study makes sense. Author Patti Phillips describe this further in The Bottomline on ROI). "The good news is that 5-10 percent of programs usually cover about half of the learning budget because these are the most expensive programs, the programs with the largest audiences, and the programs that consume a tremendous amount of time and resources." says Jack Philips.
6. Conclusion
Now that you know the basics of the Phillips ROI model, and know how to apply it to your organization, it should help you decide upon your next steps. Using the Phillips ROI Model, you can determine the effectiveness of your training and decide whether it is helping your organization meet its business goals. This can, in turn, help you plan the right training to offer in the future as determine whether you’re on the right lines.
If you want to learn more about other training evaluations models and receive the latest news and updates delivered straight to your inbox, be sure to check out our blog. You’ll learn plenty about measuring learning, evaluating its effectiveness and exploring your next steps.If you're ready to find out how Kodo Survey can help you evaluate the effectiveness of your training programs and learning journeys, request a demo with one of our experts today.
Related Posts