Value Realization Audit
Where We Started
The Alberta Pensions Services Corporation (APSC) administers the pension programs for all public sector pension programs in the province of Alberta. Following the implementation of a difficult and challenging systems project, a post-implementation review was commissioned to assess the impact of the project and whether promised project benefits were realized.
What They Needed
APSC undertook a significant modernization program, embarking on the replacement of its core pension administration system. The original development effort was envisioned as a program involving three separate, sequential projects. The initial project implemented base functionality, replacing existing legacy technology with what was expected to be a modern, more-robust, stable and usable system. The project was seen as essential in being able to sustain service delivery, and anticipated seamless end-to end automation, with a high level of automation and workflow. Later stages promised increased benefit, including the implementation of online services for both employers and members.
The implementation of the first phase of the project was significantly more challenging than anticipated. Significant changes were encountered, as well as numerous design, development and implementation issues. After the implementation of the first phase, significant operational and usability challenges were experienced. This began an extensive period of operational transition, updating processes, structures and workflows in order to work around issues and optimize organizational performance despite limitations of the system.
As a consequence of the difficulties encountered in the first phase of the project, the subsequent phases that had been initially planned were not undertaken. The functionality was limited to only what was implemented in the first phase. Given the challenges and implementation difficulties associated with the project, APSC initiated an audit to review the project results, and in particular to assess the degree to which the benefits promised at the outset of the program were realized, and to evaluate the ultimate impact of the project on the organization.
What We Did
We were engaged as the audit consultants to lead the benefits realization review. Many audits and post implementation reviews focus on what happened during the project, with an emphasis on evaluating project delivery and project management challenges. Results typically focus on findings and recommendations, with view to identifying future project improvements. This audit was unique in that it focussed less on the project management and delivery results than on the attainment of benefits. The specific desire was to examine the degree to which the business case as promised was implemented, and assess the consequences of the project for the organization.
Given the experience during the project and in particular the implementation process, there was a significant degree of frustration, animosity and resentment in the organization with respect to the project. Popular perception was that it was a failed project, and in particular that—despite efforts to proactively address change management—the implementation in particular was flawed.
In conducting a benefits realization review for APSC, we conducted an extensive assessment of documentation associated with the project, including presentations and commitments initially made at an executive level and to the boards of member pension plans. We also reviewed detailed documentation associated with the development and implementation of the system. We conducted extensive one-on-one interviews with staff and other stakeholders involved in the project, and conducted focus groups to review the impact of the system on front-line staff.
What Happened
One of the challenges that emerged in conducting the benefits realization review was that a formal business case for the project was never prepared. In order to assess the degree to which expected benefits were realized, we first needed to reverse engineer what a business case would have looked like if compiled at a pre-approval stage. By reviewing and cross-referencing commitments and estimates articulated in early correspondence and board-level presentations, we were able to build a comparative assessment of expected impacts.
Having completed the review, we were able to conduct an analysis of the impacts of the project on the organization. This included a review of planned and actual costs, evaluation of operational impacts as a result of the implementation effort, and an assessment of the benefits that were realized. In addition to reviewing benefits in terms of operational staffing requirements, expected FTE savings and financial impacts, we were also able to assess and incorporate consideration of a number of intangible impacts.
While there were positive impacts that were realized compared to initial expectations, these were primarily the result of significant post-implementation operational adjustments that were not anticipated or expected. The benefits, while real, were less a result of the project as much as a consequence of work done after the project, in order to put processes and structures in place that allowed the new system to work.
Because of our work on the benefits realization review, we were able to provide concrete, objective and defensible analysis of impacts, which were more positive than expected given frustrations that were experienced and expressed regarding the project. This provided not only useful insights to satisfy the audit objectives, but also delivered concrete recommendations for improving the management of future projects and assessing and evaluating expected benefits.
