The complete story of a training program success or failure
Patricia and Jack Phillips, from ROI Institute, included return on investment (ROI) as the fifth level of the traditional Kirkpatrick evaluation levels. According to this new ROI methodology for training and education practitioners, the new evaluation framework includes five levels of data:
- Level 1 Reaction, Satisfaction, and Planned Action—Data representing participants’ reactions to the program and their planned actions is collected and analyzed. Reactions may include participants’ views of the course content, facilitation, and learning environment. This category of data also includes data often used to predict application of acquired knowledge and skills, including relevance, importance, amount of new information, and participants’ willingness to recommend the program to others.
- Level 2 Learning—Data representing the extent to which participants acquired new knowledge and skills is collected and analyzed. This category of data also includes the level of confidence participants have in their ability to apply what they have learned.
- Level 3 Application and Implementation—Data is collected and analyzed to determine the extent to which participants effectively apply their newly acquired knowledge and skills. This category also includes data that describes the barriers that prevent application and any supporting elements (enablers) in the knowledge transfer process.
- Level 4 Business Impact—Data is collected and analyzed to determine the extent to which participants’ applications of acquired knowledge and skills positively influenced key measures that were intended to improve as a result of the program. When reporting data at Level 4, a step to isolate the program’s effects on these measures from other influences is always taken.
- Level 5 Return on Investment—Impact measures are converted to monetary values and compared with the fully loaded program costs. You can have improvement in productivity, for example, but you must determine the monetary value of that improvement and what that improvement cost you in order to calculate ROI. If the monetary value of productivity’s improvement exceeds the cost, your calculation results in a positive ROI.
Each level of evaluation answers basic questions regarding the program success.
Level of Evaluation | Key Questions |
Level 1: Reaction, Satisfaction, and Planned Action |
|
Level 2: Learning |
|
Level 3: Application and Implementation |
|
Level 4: Business Impact |
|
Level 5: ROI |
|
The reason for referring to evaluation data as levels is that it facilitates managing and reporting the data. More important, however, these five levels present data in a way that makes it easy for the audience to understand the results reported. Each level of evaluation provides important, stand-alone data. Reported together, the five-level ROI framework represents data that tells the complete story of program success or failure.
Reference: Return of Investment Basics, Patricia Pulliam Phillips, Jack J. Phillips, ASTD Press, 2005 ROI Institute
