![\"Proving](\"https:\/\/old.insources.com.au\/images\/Blog\/Proving_ROI_in_2014.jpg\")
It is difficult to imagine a world of learning and development without technology, and investment in technology continues to grow at astonishing rates. Its growth is inevitable and its use is predestined. But these investments attract attention from executives who often want to know if they’re working properly.<\/p>
Does it make a difference? How does it connect to the business? Does it really add the value that we anticipated? Is it as effective as facilitator-led learning? These are some of the questions training and education professionals must answer to show the impact learning technologies have on talent development.<\/p>
A fundamental change<\/strong> Technology, with its many forms and features, is here to stay. However, some concerns must be addressed about the ac- countability and success of technology-based learning.\u00a0<\/p> The need for business results<\/strong> Udell goes on to say that mobile learning should connect to any of those measures, and he takes several of them step-by-step to show how, in practical and logical thinking, a mobile learning solution can drive any or all of these measures. He concludes by suggesting that if an organization is investing in mobile learning or any other type of learning, it needs to connect to these business measures. Otherwise, it shouldn’t be pursued. This dramatic call for accountability is not that unusual.<\/p> Credible connections<\/strong> Unfortunately, the majority of results presented in technology case studies are void of measurements at the levels needed by executives. Only occasionally are application data presented\u2014measuring what individuals do with what they learn\u2014and rarely do they report\u00a0a credible connection to the business. Even rarer is the ROI calculation. In a recent review of award-winning e- learning and mobile learning case studies\u00a0published by several prestigious organizations, not one project was evaluated at the ROI level where the monetary value of the impact was compared with the program’s cost. They used the concept of ROI to mean any value or benefit from the program.<\/p> Mislabeling or misusing ROI creates some concerns among executives, who are accustomed to seeing ROI calculated in a very precise way by the finance and accounting team. Only two or three were evaluated on the cost savings of technology-based learning compared with facilitator-led learning. This may not be a credible evaluation. Reasons for lack of data<\/strong> These obstacles keep the proponents from developing metrics to the levels desired by executives:<\/p> \ufffcBecause these barriers are perceived to be\u00a0real, they inhibit evaluation at the levels de- sired by executives. But they are myths for the most part. Yes, evaluation will take more time and there will be a need for more planning. But the step-by-step process of the ROI methodology is logical.<\/p> Case study Summary<\/strong>. This project involves a mobile learning application for sales associates of a large software firm specializing in software solutions for the trucking industry. Sales associates were provided a mobile learning solution for their iPads that was designed to teach them to describe and sell an upgrade to its most popular software product, ProfitPro.<\/p> Measuring results<\/strong>. The first 25 people signed up within three days of the program’s an- nouncement. Level 1 reaction data were collected at the end of the fifth module. Re- actions were as expected, averaging 4.3 on a five-point scale.<\/p> Level 2 learning seemed appropriate, and quiz scores were above targets. The average score was 20.8 out of a possible 25.<\/p> Level 3 application data seemed to be on track. The skills of identifying pricing options and explaining implementation and support were off a little, but overall the objectives were met. As expected, there were some barriers and enablers to success. The barriers were minimal. However, there was a concern that 9 percent of sales associates were not encour- aged by their managers to use the program. The number one enabler was management encouragement.<\/p> The Level 4 impact data comparing the experimental group of 25 sales associates with the control group of 22 revealed that al- most all of the control group members (19 For the control group, the average amount of monthly sales per associate was $3,700 Next, the fully loaded costs were included to make the ROI calculation credible. The benefit-cost ratio and ROI were then calcu- lated. ROI was measured as 311 percent, which exceeded the objective.<\/p> In addition to the tangible sales increase converted to money, several intangibles were connected to the program:<\/p> Accountability<\/strong> Owners and developers must build in ac- countability, measure successes, report results to the proper audiences, and make adjustments and improvements. This brings technology-based learning to the same level of accountability that IT faces in the imple- mentation of its major systems and software packages. IT executives must show the impact, and often the ROI, of those implementations. Technology-based learning should not escape this level of accountability.<\/p> By Tamar Elkeles, Patti Phillips, and Jack Phillips<\/p><\/div>
Learning technologies have been used in the workplace for more than 20 years, but it is only in the past few years that their impact could be described as a “fundamental change.” More recent evolutions of learning technol- ogy bring significant change in how we grow and develop our current and future employ- ees. These include:<\/p>
Most would agree that any large expenditure in an organization should in some way be connected to business success. Even in non- business settings, large investments should connect to organizational measures of output, quality, cost, and time\u2014classic measurement categories of hard data that exist in any type of organization.
In Learning Everywhere: How Mobile Content Strategies Are Transforming Training, author Chad Udell makes the case for connecting mobile learning to business measures. He\u00a0starts by listing the important measures that are connected to the business, including:<\/p>
Those who fund budgets are adamant about seeing the connection between investing in learning technologies and business results. These executives realize that employees must learn through technology, often using mobile devices. And they know that employees must be actively involved and engaged in the process and learn the content. But more importantly, employees must use what they have learned and have an impact on the business.<\/p>
Evaluation of technology-based learning rests on six levels of data. Level 0 represents inputs to the process, and Levels 1-5 encompass outcomes from the process.<\/p>
Credible connections to the business were rare. Only one study attempted to show the impact of mobile learning using comparison groups. Even there, the details about how the groups were set up and the actual differences were left out. When the data are vague or missing, it raises a red flag.<\/p>
In our analysis of technology-based learning programs, several major barriers emerged.<\/p>![\"pyramid6\"](\"https:\/\/old.insources.com.au\/images\/Blog\/pyramid6.jpg\")
<\/strong>
This case study highlights the key issues in calculating the impact and ROI of a mobile learning solution on business results.<\/p>
of 22) were selling the upgrade even though they did not participate in the program. But the difference between the two groups was impressive.<\/p>
and the average time to first sale was 21 days. Conversely, the experimental group mem- bers took an average of only 11 days to make their first sale and sold on average $7,500 per month\u2014a difference of $3,800. The difference was then annualized, producing an improve- ment of $1.14 million.<\/p>
The appropriate level of evaluation usually is achievable within the budget and it is feasible to accomplish. It is a relatively simple process. The challenge is to take the initiative and be proactive\u2014do not wait for executives to force the issue.<\/p>