Project Evaluation Reports and Lessons Learned
A completed project usually culminates in the preparation of a report concerned with project performance over the period of the project life. For larger projects this report may consist of the covering letter, cover page, summary, contents, introduction, purpose, methodology, discussion or findings, conclusions and lessons learned, recommendations and possibly numerous appendices.
Some ways of collecting the necessary project performance data for evaluation are surveys, questionnaires, interviews, focus groups, observation, and through study of project documentation, diaries and logs. It is particularly useful to maintain a Lessons’ Learned Log. There is no single way to structure the body of our reports (sometimes referred to as discussion, findings or observations), but one of following may be appropriate:
Knowledge Areas. The PMI PMBOK identifies the following ten knowledge areas in terms of which project success can be evaluated:
- Project Integration Management coordinates the other areas to work together throughout the project.
- Project Scope Management is a set of processes used to ensure that the project includes all of the requirements and no new requirements are added in a way that could harm the project.
- Time Management involves processes to ensure that the project is completed on schedule.
- Cost Management involves processes to ensure that the project is completed on budget.
- Quality Management ensures that the project meets its requirements, or does what it is expected to do.
- Human Resource Management includes all of the processes used to develop, manage and put the project team together.
- Communication Management determines what information is needed, how that information will be sent and managed, and how project performance will be reported.
- Risk Management involves identifying, managing and controlling risk of a project.
- Procurement Management is the group of processes used to acquire the materials and services needed to complete the project.
- Stakeholder Management concerns the identification, engagement and control of those who are affected by the project or may affect the project.
Lifecycle. Another way to assess project performance is to do so as per the project lifecycle phases, one example of which is CDEF process, where:
1. Conception when the project idea is evaluated and if the proposition is appropriate a project manager is appointed and a project charter prepared.
2. Development when the project planning team prepare a detailed plan for the implementation of the project.
3. Execution when the project plan is implemented, progress monitored and controlled to produce the final deliverable(s).
4. Finish when the project is closed.
Six Ps Perspective. Not “Proper Preparation Prevents Piss Poor Performance”, which of course is true, but in this instance six Ps means we assess project performance from the following perspectives:
1. Purpose. Did the project realise its purpose; its rationale for being undertaken? What business case benefits have been realised at this early point?
2. Parameters. How did the project perform in terms of scope, time, quality, cost, risk and whatever other parameters or objectives were relevant?
3. Participants. Were stakeholders (including project team members) effectively managed and their needs satisfied.
4. Processes. How effective and efficient were the processes used during the project? Such processes may include those for estimating, project approval, outsourcing, problem solving, decision making, communications, reviews, risk and issues management, variations, monitoring and control etc.
5. Product. Was the final deliverable, product or service, satisfactory in all respects?
6. Politics. Politics are a fact of organisation life and project managers are political beings by virtue of their position. What impact did politics have on the project?
Report Recommendations. Project reports always culminate in conclusions that may be followed by recommendations. If so, this section of our report is probably the most useful part, where we make suggestions based on our report’s conclusions. Recommendations in project reports tend to be less tentative than those of academic reports. For example, a vague recommendation such as “Perhaps the procedure for recording variations should be updated in due course” will not be as effective as would a more action-oriented and specific recommendation that might read thus “The project sponsor is to update and publish the procedure for recording variations by 1 June” (which may indeed spawn a small project).
Recommendations must flow logically from our report’s conclusions. In fact, it is usually helpful to involve others in their identification. Also, our recommendations should match any terms of reference, be prioritized since they require resources, be consistent with our organisation’s mission and goals, core values and desired culture, be readily understood, and be capable of implementation.
Novopay Project Report. Click here for the naughty Novopay report – the Ministerial report on New Zealand’s notorious teachers’ payroll venture. The lessons learned at page 13 of this report are rather generic and would be applicable to all projects. Although not mentioned, removing the intended pilot and phased rollout from project implementation would not have helped. The system was supposed to cost the taxpayer somewhere in the region of $200 million over a 10-year period. Yet fixing Novopay has already cost us taxpayers an extra $30 million as at April 2014, and new problems are emerging.
Doubtlessly, there are other “Novopays” out there that would benefit from this debacle. Actually, the report following our INCIS (Integrated National Crime Information System) IT project disaster some ten years ago resulted in similar lessons, namely:
- The business case needs to adequately address technology and risks.
- Over-ambitious technology should be avoided.
- Technology needs to be firmly fixed at the time of contract.
- Adequate resources in terms of skilled and experienced governance and management are imperative.
- Appropriate governance and management structures should be in place.
- The contract and the form of the contract needs to be appropriate.
- Sound quality and risk management processes must be in place.
- Human resources problems are important and need to be addressed.
- An undue degree of reliance on the contract should be avoided.
- An appropriate structure should be in place for monitoring of the project.
The lessons learnt from the INCIS project led the Inquiry to make the following main recommendations:
- The business case should reflect overall business strategy and should address technology resources and risks as well as financial issues.
- Projects should normally use proven technology. Where it is necessary to use unproven technology, this should be reflected in an increased risk management process.
- It is essential that a project be properly resourced in terms of skilled and experienced governance and management.
- The appropriate form of contract should be signed only when the Government agency is satisfied that it has addressed and resolved all relevant issues including technology, resources and risks.
- There must be comprehensive quality and risk management processes in place.
- The appointment of key personnel is critical and care needs to be taken in the appointment process.
- Any conflict or dispute that adversely affects the project must be addressed promptly and in an effective way.
- To maximise the prospects of success, a project needs to have tight and effective management.
- The Government needs to define structure and role for approval and monitoring of large IT projects.
- The approval and monitoring structures need to be resourced to enable them to effectively meet the expectations of Government.
I notice that the Novopay report’s claims “There are numerous examples of high complexity, high value projects that have delivered effectively, on time, and on budget.” I challenge the authors to name even one.
Cynics might say – Novopay is about chronically underbidding the requirements to win selection. Having party. Key people taking cheques and moving jobs before anything hits the fan. Screw up the work. Milk the variations. Crash the project. Cry foul and then get a fat upgrade when it’s to late to change direction.
A book “Dangerous Enthusiasms” written by Robin Gauld and Shaun Goldfinch of Otago University describes how big IT projects usually don’t deliver on their promised benefits. They analyse five failed IT projects including INCIS, which was abandoned as a failure at the $100 million point. Hey – Novopay might warrant a new edition of their book. The “enthusiasms” referred to in the title of their book are:
1. Idolisation – general adoration for things IT.
2. Technophilia – more complex IT is the way to go.
3. Lomanism – naïve belief in IT salespeople’s promises.
4. Managerial Faddism – must have latest IT gadgetry.
“Dangerous Enthusiasms” also mentions eight ways or habits to ensure project failure:
- Prefer a huge project scope.
- Change deliverable specifications often during project.
- Have an enormous and complex contract document.
- Rely on advise of salespeople and use lots of consultants.
- Ensure project takes a long time to ensure technology becomes outdated.
- Believe everything you’re told about progress.
- Assume bugs will be ironed out once project goes live.
- When failure threatens, never terminate project, but rely on promised fixes.
- Continue to throw lots of money at project.