Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive

Piramide ROIMeasuring and evaluating learning has earned a place among the critical issues in the learning and development and performance improvement fields. For decades, this topic has been on conference agendas and discussed at professional meeting. Journals and newsletters regularly embrace the concept, dedicating increased print space to it. Professional organisations have been created to exchange information on measurement and evaluation, and more than twenty-five books provide significant coverage of the topic. Even top executives have an increased appetite for evaluation data.

Although interest in the topic has heightened and much progress has been made, it is still an issue that challenges even the most sophisticated and progressive learning and development departments. While some professionals argue that having a successful evaluation process is difficult, others are quietly and deliberately implementing effective evaluation systems. The latter group has gained tremendous support from the senior management team and has made much progress. Regardless of the position taken on the issue, the reasons for measurement and evaluation are intensifying. Almost all learning and performance improvement professionals share a concern that they must show the results of learning investments. Otherwise, funds may be reduced or the department may not be able to maintain or enhance its present status and influence within the organisation.

The dilemma surrounding the evaluation of learning is a source of frustration with many senior executives-even within the field itself. Most executive realised that learning is a basic necessity when organisation experience significant growth or increased competition. They need intuitively feel that providing learning opportunities is valuable, logically anticipating a payoff in important bottom-line measures, such as productivity improvements, quality enhancements, cost reductions, time savings, and improved customer service. Yet the frustration comes from the lack of evidence to show that programs really work. While results are assumed to exist and learning programs appear to be necessary, more evidence is needed, or executives may feel forced to adjust funding in the future. A comprehensive measurement and evaluation process represents the most promising, logical, and rational approach to show this accountability. Insources is delivering a two-day workshop around how to measure the contributions of learning and development and performance improvement programs.

Key Questions
When individuals pursue a comprehensive process, they often have anxiety, issues, and concerns. They have important questions that they want resolved. Exhibit 1.1 shows a list if typical questions that individual's face, regardless of the type of organisation or the organisation's stage of growth and development each of these issues, as well as many others. Each question is covered with responses that can help resolved many measurement and evaluation system challenges.

  • How can I move up in the evaluation chain?
  • How can I collect data efficiently?
  • What data should be collected at each level?
  • How can I design a practical evaluation strategy that has credibility with stakeholders?
  • What support do I need for evaluation?
  • How can I integrate data in a management scorecard?
  • How should evaluations data be used?
  • How can I get the internal support to design and implement my evaluation strategy?
  • How can I proceed if the evaluation reveals an unacceptable result?
  • How can I develop practical and credible tests?
  • How can I use the evaluation process to implement a result-based philosophy?
  • How can I make cost-effective decisions at each evaluation level?
  • How can I convince clients that my program is linked to business performance measures?

Global Evaluation Trends
Measurement and evaluations have been changing and evolving- in both the private and public sectors-across organisation and cultures, not only in the Unites States, but across all developed countries. The following trends have been identified:

  • Organisations are increasing their investments in measurement and evaluation with the best practice groups spending 3 to 5 percent of the learning and development budget on measurement and evaluation.
  • Organisation are moving up the value chain, away from measuring reaction and learning to measuring application, impact, and occasionally ROI.
  • The increased focus on measurement and evaluation is largely driven by the needs of the clients and sponsors of learning projects, programs, initiatives, and solutions.
  • Evaluation is an integral part of the design, development, delivery, and implementation of programs.
  • A shift from a reactive approach to a proactive approach is occurring, with evaluation being addressed early in the cycle.
  • Measurement and evaluation processes are systematic and methodical, often designed into the delivery process.
  • Technology is significantly enhancing the measurement and evaluation process, enabling large amounts of data to be collected, processed, analysed, and integrated across programs.
  • Evaluation planning is becoming a critical part of the measurement and evaluation cycle.
  • The implementation of comprehensive measurement and evaluation processes usually leads to increased emphasis on initial needs analyses.
  • Organisations with comprehensive measurement and evaluation systems in place have enhanced their program budgets.
  • Organisations without comprehensive measurements and evaluation systems have reduced or eliminated their program budgets.
  • The use of ROI is emerging as an essential part of many measurement and evaluation systems. It is a fast-growing metric- 70 to 80 percent of organisations has it on their wish lists.
  • Many successful examples of comprehensive measurement and evaluation applications are available in all types of organisations and cultures.

These trends are creating a never-ending appetite for more information, resources, knowledge, and skills in the measurement and evaluation process.

Reference: The Value of Learning, Jack and Patti Phillips, 2014

Write comment (0 Comments)

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive

A Review of Best Practices

By Jack and Patricia Phillips
The demand for HR's accountability through measurement continues to increase. Here, Jack and Patti Phillips provide an overview of best practices in the return on investment (ROI) process and describe how application of these measurement practices can positively impact the HR function in the public sector.

Special Issues in the Public Sector
Public sector activities are complex and public sector organizations implementing ROI confront several unique issues that can be impediments to measurement at the ROI level. The following issues lead to common misconceptions that must be addressed before public sector units can move forward with comprehensive evaluation practices:

  • Absence of revenues and profits. Government programs are typically not profit-driven. The perception that an ROI value can only be developed when there are profits and revenues is far from true. The vast majority of case studies, even in the private sector, develop monetary benefits based on cost saving or cost avoidance. When productivity is improved, quality is enhanced and cycle times are reduced, the result is a cost saving—a direct, monetary contribution.
  • Absence of hard data. The perception that only intangible data—not hard data—are available in government agencies is not necessarily the case. Even in the simplest government unit has output, quality, cost and time—the four major categories of hard data.
  • Whose ROI? The ROI is a comparison of monetary benefits with the costs from the viewpoint of certain constituencies, perhaps more for public sector organizations than in private sector.
  • Government services are essential and, therefore, should not have this level of evaluation. Many government services are essential and must be provided, regardless of the accountability. Programs in the private sector can be altered if they are not contributing value.
  • Restricted range of options to correct problems. When a private sector program is determined to have major problems, management has a range of corrective options, including discontinuing the program. Such may not be the case in the public sector.

Yet, despite these unique challenges, many Australian's public sector entities are considering incorporating the ROI Methodology into their evaluation systems. The Australian Department of Defense has already adopted the ROI methodology to measure the impact of a new human resources information system.

Measuring the return on investment in people, programs, and processes is now a commonplace global practice. With origins in the private sector, the fiscal responsibility trend is increasing in the public realm as public policies, government regulations, executive-level agendas, taxpayer demands, and privatization of government organizations and agencies drive the need to measure the ROI of people, programs and services in the public sector. In addition, an emerging breed of government managers with a business mindset is pressing to replace ineffective evaluation methods, especially in light of increased costs for many programs and initiatives. Each of these driving forces brings a renewed focus on measuring ROI in the public sector.

The ROI methodology has proven to be a feasible, credible and sound approach to meeting the accountability requirements of the public sector network, its internal clients and the research community. With the increased acceptance and public sector utilization of ROI, much of the focus has now turned to best practices for ROI implementation. The following 11 best practices represent the state-of-the-art with those organizations that have successfully implemented ROI.

Best Practice #1:The ROI methodology is implemented as a process improvement tool and not a performance evaluation tool for the HR staff.
HR staff acceptance is critical for the implementation of this process. No individual or group is willing to create a tool that will ultimately be used to evaluate his or her performance. Consequently, many organizations recognize that ROI is a process improvement tool and communicate this posture early. The process shows not only the success of a particular project, program, or solution, but also provides detail into how the project can be revised to add additional value. Barriers and enablers to success are always identified.

Best Practice #2:The ROI methodology generates a micro-level scorecard with six types of data.figure-1
As Figure 1 shows, the data represents a balanced, micro level scorecard of performance for individual programs. In all, it represents both qualitative and quantitative data, often taken at different time frames and from various sources.

This approach expands upon Donald Kirkpatrick's four-level framework for categorizing evaluation data (i.e. reaction, learning, behavior, and results, as described in Evaluating Training Programs: The Four Levels,© 1998, Berrett-Koehler Publishers) to incorporate a fifth level of evaluation, which serves to capture the financial impact or return-on-investment (ROI) of HRD programs.

The ROI model includes techniques for isolating the impact of an HR or training solution and also provides for a sixth data measure: Intangible Benefits... which are those benefits that have not been converted to monetary value. These entail such measures as increased morale, improved teamwork, or increased job satisfaction.

The first type of data—participants' reaction—is measured on almost all HR functions and programs, usually with questionnaires and surveys. Learning measurements check to ensure that participants gain new skills, absorb knowledge, and grasp how to make the HR program successful. Measuring application and implementation determines if participants are executing desired program behaviors successfully. Measuring impact focuses on the actual results achieved directly from the program involvement. Typical measures include output, quality, costs, time, and employee satisfaction. ROI is the ultimate level of evaluation, where the impact measures are converted to monetary benefits and compared with the costs. ROI is usually expressed as a percentage or benefit/cost ratio. In addition to tangible, monetary benefits, most HR programs will produce intangible, non-monetary benefits—benefits from implementation and impact that are not converted to monetary value.

Best Practice #3:ROImethodology data are being integrated to create a macro scorecard for the HR function.Figure-2
As more and more studies are conducted, data are rolled up to create a macro level scorecard, showing the value of the function.
As shown in Figure 2, the individual micro scorecard evaluation data are integrated into the overall macro level scorecard. This approach requires a few similar questions to be asked each time. These are then integrated, typically by using technology, to create the HR macro level scorecard.

Best Practice #4:ROI impact studies are conducted very selectively, usually involving five to 10 percent of all programs and solutions.
Programs that are usually targeted for levels four and five are those that are strategically focused, expensive, high profile, controversial and reflective of management's interest. This does not mean that other programs are not evaluated. It is recommended that all programs be evaluated at level one and the vast majority at level two, but only a few select programs are taken to levels three, four and five. Most importantly, those involving the actual ROI calculation— taken to the fifth level—are evaluated at all five levels.

Best Practice #5:ROI evaluation targets are developed, showing the percent of programs evaluated at each level.
Figure-3Organizations are targeting the desired number of programs to evaluate at each level, expressed as a percent. Target levels are developed reflecting the resources available and the feasibility of evaluation at each level. Targets usually begin at 100 percent of programs at level one and conclude with five to 10 percent of programs at level five. Figure 3 shows a typical profile of targets developed by a best practice organization.

Best Practice #6:A variety of data collection methods are used in ROI analysis.
ROI evaluation is not restricted to a particular type of data collection method such as monitoring of business data. To develop a complete profile of the six types of results data generated from the ROI methodology, data can be collected at different times and gathered from different sources using a variety of methods, including:

  • Follow-up surveys and questionnaires,
  • On-the-job observation to capture application and use,
  • Tests and assessments measuring the extent of learning,
  • Interviews to measure reaction and implementation,
  • Focus groups,
  • Action plans, and
  • Performance records monitoring to show improvement in various performance records and operational data.

The challenge in data collection is selecting appropriate methods for given settings and specific programs, within time and budget constraints.

Best Practice #7: For a specific ROI evaluation, the effects of HR are isolated from other factors.
HR professionals often overlook the step of isolating the effects of the particular HR program or solution. Although a difficult issue, best practice organizations realize there must be some method in place to show the direct contribution of the HR program to make a business linkage to a specific HR effort. Many best practice organizations are currently using a variety of techniques to tackle this issue with each impact study. These techniques include:

  • Comparison groups—a pilot group of participants in an HR program is compared with a group not participating in the program,
  • Trend lines—projected program values are compared with the actual data during program implementation and at the program's conclusion, and
  • Estimations by participants, stakeholders, and independent experts.

While many factors influence performance data after a program is implemented, these specific techniques can pinpoint the degree of improvement directly related to the program. In reality, this step is critical to helping executives understand the relative contribution of HR. Otherwise, there's a temptation to slash the budgets of major programs because there's no clear connection between the program and the business value.

Best Practice #8:Business impact data are converted to monetary values.
For some programs, intangible, non-monetary benefits have important value, such as improved public image, increased job satisfaction, increased organizational commitment, reduced stress, and improved teamwork. These days, it may not be enough to report HR program value simply in terms of intangible benefits or in terms of outcomes expressed as quality improvement, cycle time reduction, turnover reduction, or enhancement in customer loyalty.

Defining the actual value of the improvement in monetary terms is essential for ROI analysis because an ROI calculation compares the monetary value with the cost of the program. Best practice organizations are using a full array of approaches to develop monetary values.

Best Practice #9:The ROI methodology is being implemented for about three to five percent of the HR budget.
One of the common fears of ROI implementation is the excessive cost in both time and direct funds. Best practice agencies report that they can implement the
ROI methodology for roughly three to five percent of the total budget, using appropriate evaluation targets discussed in number five. Available cost savings approaches include the following:

  • Plan for evaluation early in the process
  • Build evaluation into the training process
  • Share the responsibilities for evaluation
  • Require participants to conduct major steps
  • Use shortcut methods for major steps
  • Use sampling to select the most appropriate programs for ROI analysis
  • Use estimates in the collection and analysis of data
  • Develop internal capability to implement the ROI process
  • UtilizeWeb-based software to reduce time
  • Streamline the reporting process

When implementing ROI, many public sector organizations have migrated from a very low level of investment (around one percent or less) to the three to five percent level by a process of gradual budget enhancements. These enhancements sometimes come directly from the cost savings generated from the use of the ROI methodology.

Best Practice #10:ROI forecasting is being implemented routinely.
Senior executives are sometimes asking for a forecast of ROI before a project begins. Consequently, best practice organizations are routinely using ROI forecasting approaches to enhance the decision making process. The credibility of the process is greatly increased by the use of conservative adjustments and built-in steps to secure input from the best experts.

Best Practice #11:The ROI methodology is used as a tool to strengthen/ improve the HR process.
A significant payoff for using the ROI process over time is that it transforms the role of HR in the organization. Application of the process increases HR alignment with business needs, improves the efficiency of design, development, and delivery, and enhances the value of the HR function in the organization.
Furthermore, it builds respect, support and commitment from internal groups, including senior executives and major program sponsors.

Collectively, these best practices are evolving as hundreds of organizations are using ROI each year. The best practices underscore the progress in the evolution of ROI application and use in both the public and private sector. Yet the best process, tool, technique, or model will only be successful if it is properly used and becomes a routine part of the HR function. Implementation issues include assigning evaluation roles and responsibilities, building the necessary skills, developing transition plans and goals around the process, and preparing the environment, individuals and support teams for this type of comprehensive analysis.

The ROI Methodology, developed by Dr. Jack Phillips, is a comprehensive step-by-step process used to assess, measure, and evaluate training, performance improvement, human resources, quality, and technology. The ROI Institute is a research, benchmarking, and consulting organization that provides workshops, publications, and consulting services on the ROI Methodology™. It is the only organization that supports, updates, refines, and distributes information and provides services on this Methodology. Dr. Jack Phillips serves as chairman of the ROI Institute. Dr. Patti Phillips serves as president and CEO.

Insources has signed an International Partnership Agreement with ROI Institute to offer learning opportunities, networking, publications and consulting services around the ROI Methodology™ in Australia.

Write comment (0 Comments)

Star InactiveStar InactiveStar InactiveStar InactiveStar Inactive

By Jack and Patti Phillips

ROI, return on investment, is a metric fundamental to business and government alike. Executives and chief administrators recognize it, and business and operations managers appreciate it. It is calculated consistently and recognized across sectors around the world by those stakeholders with fiduciary responsibility for investments in people, projects, and processes.

Unfortunately, misinformation and misuse cloud the value of this simple, yet powerful, metric. While its existence in performance improvement, learning and development, and human resources is not new, when reading much of the literature and comments made in conference sessions, one would think ROI is a new phenomenon, and one from which people should run.

ROI's use in learning and HR began in the 1970s when my husband and business partner, Jack Phillips, conducted the first ROI study on a cooperative education program. It has been used for years in quality and productivity training. In more recent years, it has grown to be a standard metric for many leadership development and coaching programs, as well as other "soft" solutions. Yet the confusion around ROI remains.

The crux of the confusion lies in how and when to use ROI, and how to report it so stakeholders recognize the complete success of a program or project. Fear and angst around ROI exist, because like most investments, a negative ROI is inevitable for poorly implemented and misaligned programs. On the flip side, if the ROI is extraordinarily high, fear exists that the results will not be perceived as credible. This fear is unwarranted if you use a credible approach to develop it, follow fundamentally sound standards, and apply it consistently across all types of programs.

Yet, too many people would rather listen to the naysayers than figure it out for themselves. Jack and Patti Phillips have published over 40 books with ASTD and even more with other publishers, such as SHRM, Berrett-Koehler, McGraw Hill, and Wiley. These publications describe ROI's use and importance in showing the contribution of programs and projects. They, along with many conference presentations and workshops, offer learning and development professionals opportunities to understand what ROI is, what it is not, and how to use it. Yet, many professionals still don't get the point—they miss the bottom line.

Reported alone, ROI describes the economic impact of programs, projects, and processes. Reported in the context of other measures, it contributes to the complete story of program success and informs decisions about resource allocation.

To learn how to show the ROI of your programs, projects, and initiatives, register now for an upcoming offering of Insources' and ROI Institute Measuring Return on Investment program.

Write comment (0 Comments)

Privacy Policy
Terms of Sale
Terms of Use

  • Email:
  • Phone: 1300 208 774
  • ABN 74 625 075 041 


© 2019 - 2020 by Insources Group Pty Ltd. All rights reserved.