A Review of Best Practices
By Jack and Patricia Phillips
The demand for HR’s accountability through measurement continues to increase. Here, Jack and Patti Phillips provide an overview of best practices in the return on investment (ROI) process and describe how application of these measurement practices can positively impact the HR function in the public sector.
Special Issues in the Public Sector
Public sector activities are complex and public sector organizations implementing ROI confront several unique issues that can be impediments to measurement at the ROI level. The following issues lead to common misconceptions that must be addressed before public sector units can move forward with comprehensive evaluation practices:
- Absence of revenues and profits. Government programs are typically not profit-driven. The perception that an ROI value can only be developed when there are profits and revenues is far from true. The vast majority of case studies, even in the private sector, develop monetary benefits based on cost saving or cost avoidance. When productivity is improved, quality is enhanced and cycle times are reduced, the result is a cost saving—a direct, monetary contribution.
- Absence of hard data. The perception that only intangible data—not hard data—are available in government agencies is not necessarily the case. Even in the simplest government unit has output, quality, cost and time—the four major categories of hard data.
- Whose ROI? The ROI is a comparison of monetary benefits with the costs from the viewpoint of certain constituencies, perhaps more for public sector organizations than in private sector.
- Government services are essential and, therefore, should not have this level of evaluation. Many government services are essential and must be provided, regardless of the accountability. Programs in the private sector can be altered if they are not contributing value.
- Restricted range of options to correct problems. When a private sector program is determined to have major problems, management has a range of corrective options, including discontinuing the program. Such may not be the case in the public sector.
Yet, despite these unique challenges, many Australian’s public sector entities are considering incorporating the ROI Methodology into their evaluation systems. The Australian Department of Defense has already adopted the ROI methodology to measure the impact of a new human resources information system.
Measuring the return on investment in people, programs, and processes is now a commonplace global practice. With origins in the private sector, the fiscal responsibility trend is increasing in the public realm as public policies, government regulations, executive-level agendas, taxpayer demands, and privatization of government organizations and agencies drive the need to measure the ROI of people, programs and services in the public sector. In addition, an emerging breed of government managers with a business mindset is pressing to replace ineffective evaluation methods, especially in light of increased costs for many programs and initiatives. Each of these driving forces brings a renewed focus on measuring ROI in the public sector.
The ROI methodology has proven to be a feasible, credible and sound approach to meeting the accountability requirements of the public sector network, its internal clients and the research community. With the increased acceptance and public sector utilization of ROI, much of the focus has now turned to best practices for ROI implementation. The following 11 best practices represent the state-of-the-art with those organizations that have successfully implemented ROI.
Best Practice #1:The ROI methodology is implemented as a process improvement tool and not a performance evaluation tool for the HR staff.
HR staff acceptance is critical for the implementation of this process. No individual or group is willing to create a tool that will ultimately be used to evaluate his or her performance. Consequently, many organizations recognize that ROI is a process improvement tool and communicate this posture early. The process shows not only the success of a particular project, program, or solution, but also provides detail into how the project can be revised to add additional value. Barriers and enablers to success are always identified.
Best Practice #2:The ROI methodology generates a micro-level scorecard with six types of data.
As Figure 1 shows, the data represents a balanced, micro level scorecard of performance for individual programs. In all, it represents both qualitative and quantitative data, often taken at different time frames and from various sources.
This approach expands upon Donald Kirkpatrick’s four-level framework for categorizing evaluation data (i.e. reaction, learning, behavior, and results, as described in Evaluating Training Programs: The Four Levels,© 1998, Berrett-Koehler Publishers) to incorporate a fifth level of evaluation, which serves to capture the financial impact or return-on-investment (ROI) of HRD programs.
The ROI model includes techniques for isolating the impact of an HR or training solution and also provides for a sixth data measure: Intangible Benefits… which are those benefits that have not been converted to monetary value. These entail such measures as increased morale, improved teamwork, or increased job satisfaction.
The first type of data—participants’ reaction—is measured on almost all HR functions and programs, usually with questionnaires and surveys. Learning measurements check to ensure that participants gain new skills, absorb knowledge, and grasp how to make the HR program successful. Measuring application and implementation determines if participants are executing desired program behaviors successfully. Measuring impact focuses on the actual results achieved directly from the program involvement. Typical measures include output, quality, costs, time, and employee satisfaction. ROI is the ultimate level of evaluation, where the impact measures are converted to monetary benefits and compared with the costs. ROI is usually expressed as a percentage or benefit/cost ratio. In addition to tangible, monetary benefits, most HR programs will produce intangible, non-monetary benefits—benefits from implementation and impact that are not converted to monetary value.
Best Practice #3:ROImethodology data are being integrated to create a macro scorecard for the HR function.
As more and more studies are conducted, data are rolled up to create a macro level scorecard, showing the value of the function.
As shown in Figure 2, the individual micro scorecard evaluation data are integrated into the overall macro level scorecard. This approach requires a few similar questions to be asked each time. These are then integrated, typically by using technology, to create the HR macro level scorecard.
Best Practice #4:ROI impact studies are conducted very selectively, usually involving five to 10 percent of all programs and solutions.
Programs that are usually targeted for levels four and five are those that are strategically focused, expensive, high profile, controversial and reflective of management’s interest. This does not mean that other programs are not evaluated. It is recommended that all programs be evaluated at level one and the vast majority at level two, but only a few select programs are taken to levels three, four and five. Most importantly, those involving the actual ROI calculation— taken to the fifth level—are evaluated at all five levels.
Best Practice #5:ROI evaluation targets are developed, showing the percent of programs evaluated at each level.
Organizations are targeting the desired number of programs to evaluate at each level, expressed as a percent. Target levels are developed reflecting the resources available and the feasibility of evaluation at each level. Targets usually begin at 100 percent of programs at level one and conclude with five to 10 percent of programs at level five. Figure 3 shows a typical profile of targets developed by a best practice organization.
Best Practice #6:A variety of data collection methods are used in ROI analysis.
ROI evaluation is not restricted to a particular type of data collection method such as monitoring of business data. To develop a complete profile of the six types of results data generated from the ROI methodology, data can be collected at different times and gathered from different sources using a variety of methods, including:
- Follow-up surveys and questionnaires,
- On-the-job observation to capture application and use,
- Tests and assessments measuring the extent of learning,
- Interviews to measure reaction and implementation,
- Focus groups,
- Action plans, and
- Performance records monitoring to show improvement in various performance records and operational data.
The challenge in data collection is selecting appropriate methods for given settings and specific programs, within time and budget constraints.
Best Practice #7: For a specific ROI evaluation, the effects of HR are isolated from other factors.
HR professionals often overlook the step of isolating the effects of the particular HR program or solution. Although a difficult issue, best practice organizations realize there must be some method in place to show the direct contribution of the HR program to make a business linkage to a specific HR effort. Many best practice organizations are currently using a variety of techniques to tackle this issue with each impact study. These techniques include:
- Comparison groups—a pilot group of participants in an HR program is compared with a group not participating in the program,
- Trend lines—projected program values are compared with the actual data during program implementation and at the program’s conclusion, and
- Estimations by participants, stakeholders, and independent experts.
While many factors influence performance data after a program is implemented, these specific techniques can pinpoint the degree of improvement directly related to the program. In reality, this step is critical to helping executives understand the relative contribution of HR. Otherwise, there’s a temptation to slash the budgets of major programs because there’s no clear connection between the program and the business value.
Best Practice #8:Business impact data are converted to monetary values.
For some programs, intangible, non-monetary benefits have important value, such as improved public image, increased job satisfaction, increased organizational commitment, reduced stress, and improved teamwork. These days, it may not be enough to report HR program value simply in terms of intangible benefits or in terms of outcomes expressed as quality improvement, cycle time reduction, turnover reduction, or enhancement in customer loyalty.
Defining the actual value of the improvement in monetary terms is essential for ROI analysis because an ROI calculation compares the monetary value with the cost of the program. Best practice organizations are using a full array of approaches to develop monetary values.
Best Practice #9:The ROI methodology is being implemented for about three to five percent of the HR budget.
One of the common fears of ROI implementation is the excessive cost in both time and direct funds. Best practice agencies report that they can implement the
ROI methodology for roughly three to five percent of the total budget, using appropriate evaluation targets discussed in number five. Available cost savings approaches include the following:
- Plan for evaluation early in the process
- Build evaluation into the training process
- Share the responsibilities for evaluation
- Require participants to conduct major steps
- Use shortcut methods for major steps
- Use sampling to select the most appropriate programs for ROI analysis
- Use estimates in the collection and analysis of data
- Develop internal capability to implement the ROI process
- UtilizeWeb-based software to reduce time
- Streamline the reporting process
When implementing ROI, many public sector organizations have migrated from a very low level of investment (around one percent or less) to the three to five percent level by a process of gradual budget enhancements. These enhancements sometimes come directly from the cost savings generated from the use of the ROI methodology.
Best Practice #10:ROI forecasting is being implemented routinely.
Senior executives are sometimes asking for a forecast of ROI before a project begins. Consequently, best practice organizations are routinely using ROI forecasting approaches to enhance the decision making process. The credibility of the process is greatly increased by the use of conservative adjustments and built-in steps to secure input from the best experts.
Best Practice #11:The ROI methodology is used as a tool to strengthen/ improve the HR process.
A significant payoff for using the ROI process over time is that it transforms the role of HR in the organization. Application of the process increases HR alignment with business needs, improves the efficiency of design, development, and delivery, and enhances the value of the HR function in the organization.
Furthermore, it builds respect, support and commitment from internal groups, including senior executives and major program sponsors.
Collectively, these best practices are evolving as hundreds of organizations are using ROI each year. The best practices underscore the progress in the evolution of ROI application and use in both the public and private sector. Yet the best process, tool, technique, or model will only be successful if it is properly used and becomes a routine part of the HR function. Implementation issues include assigning evaluation roles and responsibilities, building the necessary skills, developing transition plans and goals around the process, and preparing the environment, individuals and support teams for this type of comprehensive analysis.
The ROI Methodology, developed by Dr. Jack Phillips, is a comprehensive step-by-step process used to assess, measure, and evaluate training, performance improvement, human resources, quality, and technology. The ROI Institute is a research, benchmarking, and consulting organization that provides workshops, publications, and consulting services on the ROI Methodology™. It is the only organization that supports, updates, refines, and distributes information and provides services on this Methodology. Dr. Jack Phillips serves as chairman of the ROI Institute. Dr. Patti Phillips serves as president and CEO.
