| Introduction | | | | work on projects also but would not initially attend the |
| Return on Investment (ROI) is a monetary | | | | 3 day program. |
| measurement that is used to evaluate the efficiency | | | | The following data was collected from the program: |
| and effectiveness of an investment made by an | | | | - Level 1: 95% of the participants were satisfied overall |
| organization. Investments take many forms - financial, | | | | with the 3 day customized program. |
| human capital, equipment, and training programs - to | | | | - Level 2: 100% pass rate on the exam (85% or higher |
| name just a few. This paper will focus on the use of | | | | was considered passing). It should be noted that the |
| ROI and the Phillips ROI Methodology(TM) to measure | | | | exam was not an overly difficult exam and as long as |
| the effectiveness of a project management training | | | | the attendees participated and paid attention in the 3 |
| program completed within XYZ Law Firm. The 5 levels | | | | day class they would be able to pass the exam. |
| of evaluation which will be reviewed within the case | | | | Frankly, the real measurement was in the Action Plans |
| study include: | | | | that were produced. |
| - Level 1: Reactions (the "smiley" sheet): Did participants | | | | - Level 3: 100% of the participants submitted Action |
| like the training they received? | | | | Plans and completed follow up questionnaires. Action |
| - Level 2: Learning: Are participants confident that they | | | | plans were completed after the class and followed up |
| have learned something from the training program? | | | | on within 6 months. The action plan included information |
| - Level 3: Behavior/Application: Are participants able to | | | | on: |
| apply what they learned in the training program back | | | | - Participant's goals (individual improvement efforts) |
| on the job? | | | | - How participants intended to meet those goals |
| - Level 4: Results/Business Impact: Did the training | | | | - Action steps to be taken |
| show improvement in efficiencies, productivity, profits, | | | | - Support and/or resources needed |
| costs, reduced turnover? | | | | - Timeline for completion |
| - Level 5: ROI: Did the training program show a positive | | | | How participants will know if they are successful in |
| ROI? | | | | meeting those goals |
| Why the Interest in ROI for Project Management | | | | How participants will evaluate that success |
| Training Programs? | | | | After the 6 month time period, participants were asked |
| The challenges surrounding training and its | | | | to complete a Continuous Improvement Plan for |
| effectiveness within the organization have become | | | | moving forward. This follow up to the Action Plan |
| more complex over the years. Today, the challenge is | | | | included: |
| even more significant for learning and development | | | | - Results/accomplishments to date |
| professionals. Return on Investment (ROI) as a tool for | | | | - Next steps to accomplish |
| evaluating project management training is becoming an | | | | - Timeline for doing so and support needed |
| expectation of senior executives within organizations. In | | | | Level 4: Data collected from participants and managing |
| today's tight economy with reduced resources and | | | | partners and senior attorneys on the impact to the |
| tighter budgets, learning and development professionals | | | | business. The data collected from Level 4 showed a |
| are finding it increasingly necessary to show the | | | | 40% increase in the number of successful projects. |
| monetary value of the project management programs | | | | Successful projects went up from 70 successful out |
| they are bringing to the organization. Today, the | | | | of every 100 (smaller projects included in the 100 along |
| success of project management training programs is | | | | with more strategic projects) to 98 successful out of |
| measured by the financial contribution of the program | | | | 100. |
| to the organization. It is not surprising then that ROI | | | | - 40% increase in the number of successful projects |
| measurements for project management training | | | | - On time and within budget and scope |
| programs are often requested. With such a large | | | | - Projects met set quality standards |
| focus on project management in all kinds of | | | | - Resources tracked and effectively allocated |
| organizations, there is an increasing desire to show the | | | | - Standard processes implemented for all projects |
| monetary benefit of investing in project management | | | | - Lessons learned captured and best practices shared |
| training programs which often reach a wide variety of | | | | Phase 3: Data Analysis |
| employees and represent a significant investment for | | | | In converting the data collected to a monetary value, |
| organizations. | | | | we found that cost savings (program benefits) were |
| The case study example below will follow the Phillips | | | | as follows: |
| ROI Methodology(TM) and encompass the four major | | | | - More efficient use of resources: $285,000. |
| phases of the methodology: | | | | - Projects on time and within budget and scope: |
| - Evaluation Planning | | | | $250,000 |
| - Data Collection | | | | - Increase in better tracking of client projects: $450,000 |
| - Data Analysis | | | | Program benefits are measured in Level 4: Business |
| - Reporting | | | | Impact. To convert to a monetary value, a dollar |
| Case Study Background: XYZ Law Firm | | | | amount was put to every resource based on salary |
| XYZ Law Firm was interested in developing a project | | | | time/fringe, etc. If projects did not go out of scope, no |
| management mindset within the firm. There was | | | | additional resources were needed, which was a cost |
| support from the managing partners and funding was | | | | savings. More efficient use of resources also meant |
| expected to be approved. The goal of the training | | | | that projects had fewer people assigned to them. The |
| was to ensure that completing projects at the law firm | | | | $450,000 in increase from better tracking of client |
| was not a severe drain on resources. There was | | | | projects was the amount that would have been lost |
| much anxiety behind any project which needed to be | | | | from the bottom line had those projects that were |
| completed and the champions of the training program - | | | | measured not been completed successfully. |
| which included a few of the partners, attorneys and | | | | The total cost of the program was $320,000. This |
| the paralegals - were confident that with some basic | | | | included planning for the ROI study, food and beverage |
| project management training, this anxiety could be | | | | for participants in the program, facilitator costs, |
| controlled and projects done within the firm would be | | | | development of the customized program, time out of |
| more easily managed within the law firm. The whole | | | | the office for training, salary/fringe of participants, etc. |
| concept of "project management" scared individuals. | | | | Any cost associated with the program or planning for |
| The managing partners of the law firm specifically | | | | the program was included in the total costs. The |
| wanted to be updated on project status and they | | | | biggest component of the cost was salary/fringe of |
| often felt that they were unsure what projects were | | | | participants - especially since the attorneys and |
| being worked on and how they were progressing. A | | | | paralegals are billable resources. |
| significant goal was to ensure implementation of best | | | | Intangible benefits, those benefits that did not have a |
| practices and standard processes and increase | | | | dollar value associated with them, were also |
| knowledge sharing. | | | | considered. They included: |
| Overall the goals of the training program included: | | | | - Reduced anxiety over involvement in and managing |
| - Increased likelihood of successful projects | | | | projects. |
| - Ability to implement strategic plans into action | | | | - Improved teamwork and reduced conflicts |
| - Improved monitoring and controlling of projects | | | | - Satisfaction of managing partners and senior |
| - Proactive risk management | | | | attorneys over status reporting on projects |
| - Improved time management and teamwork | | | | These intangible benefits were able to be tied back to |
| - Efficient utilization and tracking of resources | | | | the program and certainly contributed to the success |
| - Standards around status reporting | | | | of the projects. |
| - Implementation of best practices and standard | | | | During phase 3, we calculate the ROI of the project |
| processes | | | | management training program, using the ROI formula: |
| Some of the projects that the law firm worked on | | | | ROI = Net Program Benefits / Program Costs x 100 |
| included: | | | | ROI = $665,000 / $320,000 x 100 |
| - IT/infrastructure projects | | | | ROI = 2.08 x 100 |
| - A variety of projects for their corporate clients, such | | | | ROI = 208% |
| as mergers and acquisitions and major contract | | | | For every $1.00 invested, $2.08 is returned after the |
| negotiations | | | | costs are recovered. This was significantly over the |
| - Annual recruiting projects (which included recruiting | | | | 20% ROI set at the beginning of the project during the |
| from law schools worldwide) | | | | planning of the study. |
| - Long range capital strategic projects | | | | Phase 4: Reporting |
| Many such projects always took longer than originally | | | | A report of the results of the study was presented to |
| planned for, went out of scope and went over budget | | | | the managing partners of the law firm. Components of |
| and resource commitment. This was certainly a | | | | the study were presented to the participants and |
| problem for the client projects the law firm worked on | | | | others in the law firm. The project was considered a |
| as such projects were under fixed price agreements | | | | success and it was decided that the 3 day |
| with the client. If the project went off track, it would | | | | customized project management training program |
| impact the profitability for the law firm for that | | | | would be rolled out firm-wide. Additionally, a ½ |
| particular engagement. | | | | day program was created for the managing partners |
| Phase 1: Evaluation Planning | | | | of the firm to provide them training around how to |
| Based on the goals of the law firm, a three day | | | | support project initiatives. A future focus was to |
| customized basics of project management training | | | | develop a portal to house all project information. That |
| program was developed. Prior to the start of the | | | | was a "to do" for the IT/Application Development |
| program, data was collected on past project initiatives, | | | | department and would b e a key strategic project for |
| specifically around: | | | | them. |
| - Budget | | | | Summary |
| - Time | | | | For training and learning departments, the PMO, or lines |
| - Scope | | | | of business, showing the ROI of the project |
| - Resource utilization | | | | management training programs conducted within the |
| - Quality | | | | organization, whether in-house developed programs or |
| Much of this data was tracked, although it was not in | | | | vendor-provided programs, enables you to show the |
| one central location and required conversations with | | | | value the programs have on the business in monetary |
| many members of the law firm - from managing | | | | terms that executives can understand, respect, and |
| partners through to paralegals and administrative | | | | champion. |
| assistants. | | | | The six types of data collected during the Phillips ROI |
| In the first phase of the project, we developed a plan | | | | Methodology (TM) process (see list below) enable for |
| around the data we would be collecting. The plan | | | | consistency in the measurement of project |
| included the following components for Level 1 - 4: | | | | management training programs, thereby bringing |
| - Level 1: Satisfaction survey to participants | | | | credibility to the process. |
| - Level 2: A test administered to participants to | | | | - Reaction and value of program |
| measure the learning from the 3-day program. | | | | - Learning and confidence |
| - Level 3: Action Plan follow-up and questionnaires | | | | - Application and implementation |
| including information on how the participants were | | | | - Business impact |
| applying the skills they learned. Level 3 questionnaires | | | | - ROI |
| also went to managing partners and senior attorneys | | | | - Intangible benefits |
| to get their feedback and perception of the skills being | | | | To begin measuring your project management training |
| applied. | | | | programs, you might start with a brand new program |
| - Level 4: Collection of all data for business impact | | | | or review the current programs that are in place - |
| We also developed an ROI Analysis Plan to include | | | | following the criteria for measuring project |
| the items we intended to measure against to show | | | | management training programs discussed earlier in this |
| improvement based on the 3-day training program. | | | | paper. If you review a current program, gather data on |
| The data items to be collected at Level 4, which are | | | | the initial purpose of the program when it was first |
| the impact measures targeted for improvement | | | | developed and implemented. Does it still meet that |
| included: | | | | purpose? What value do you perceive the individuals |
| - Improved performance on projects - including | | | | taking the project management training program are |
| improved quality of end result | | | | achieving? How much has been invested in the |
| - Efficient utilization and tracking of resources | | | | program since the beginning? Should this program |
| - Increased percentage of successful projects (on | | | | remain in place "as is," or should it be updated, or |
| time, within budget, within scope) | | | | possibly be eliminated from your offerings in its entirety |
| - Implementation of best practices and standard | | | | or in part to accommodate new needs among your |
| processes | | | | project managers? An ROI study will enable you to |
| We set the ROI (hurdle) rate at 20%. | | | | look at the value of your current programs and enable |
| Phase 2: Data Collection | | | | you to better plan new programs. |
| A pilot group of 20 junior attorneys, paralegals and | | | | References |
| administrative assistants were selected to go through | | | | Phillips, J. and Phillips, P. (2007) Show Me the Money: |
| the program first and measure the benefits of the | | | | How to Determine ROI in People, Projects and |
| training. In this way, an isolation factor - an important | | | | Programs, San Francisco: Berrett-Koehler Publishers, |
| component of doing an ROI study - was available to | | | | Inc. |
| compare against a similar group of attorneys, | | | | Phillips, J., Phillips, P., Stone, R., and Burkett, H. |
| paralegals and administrative assistants who would | | | | |