oracle brm projects: do you really learn your “lessons learned€¦ · a questionnaire should be...

Do you really learn your

“Lessons Learned?”

214.333.2000 • www.ssglimited.com

801 East Campbell Rd. • Suite 350 • Richardson • TX 75081

Published by SSG LimitedAuthor: Craig Cochrane

We all know intuitively what it means to learn from our past experiences. But areyou really reaping the benefits of learning the hard way? This white paper is written from the perspective of a software development service provider and Oracle Billing and Revenue Management (BRM) project manager. It explains howto approach BRM projects with simple evaluation processes. When used consis-tently, these will help you to efficiently document your project experiences so youcan leverage your Lessons Learned long term.

Oracle BRM Projects:

2

DO YOU REALLY LEARN YOUR “LESSONS LEARNED”

When was the last time you looked for a Lessons Learned document concerning a project your

organization completed last year? Yesterday? Yeah, right! The majority of companies doing

software development perform formalized evaluations of their project performance—but few have

adopted an approach that ensures they will truly benefit from the value of their own experience.1

We all know intuitively what it means to learn from our past experiences – typically it means learn

“the hard way.” However, in order for your organization to really reap the benefits of learning,

you must use a consistent process (i.e., applied project after project) that participants buy into to

instill lasting change in the organization’s behavior.

Although the evaluation and disposition of Lessons Learned is normally completed during the project

closeout phase, they should be captured throughout a project. A Lessons Learned ‘log’ (essentially

a project "journal") should be established in a dedicated location within the project's repository

(every project has a repository, right?). A key element of the process is to ensure all team members

feel they can contribute to the process. Team members are encouraged to provide entries.

Lessons Learned entries should address an action or issue in which the outcome had a significant

positive or negative effect on the project's ability to deliver successfully. Every Lesson Learned should

include: a name for the issue or action, a one-line description of the issue or action, and a one or

two-sentence statement characterizing the impact to the project. Optionally, a recommendation

concerning how to address an issue or promote a positive action in the current project and/or

future projects may be provided. In any case, there are guidelines, of course, for handling

contributors’ thoughts. In particular, ideas must be accepted without judgment initially. In some

cases, it may be appropriate to implement the recommended action on the current project and not

wait to influence future projects!

Date of Entry Originator Description ofevent or action

Impact on project Recommendation toresolve/avoid (negative) orencourage/enforce (positive)

xx/xx/xx John D. Project kickoffmeeting not performed

Project goals & objectives not well understood by team;roles & responsibilities notwell known

Conduct a project kickoff meetingwhich includes all participants andcontributing stakeholders.

1 – Knoco Ltd survey. Conducted summer 2009. 74 businesses responded; 76% claimed to have a LessonsLearned process with an additional 7% working on a formalized process. However, less than half felt they had en effective process.


801 East Campbell • Suite 350 • Richardson • TX 75081

As stated earlier, a Post Delivery Review (PDR) should be conducted during the close-out stage for

each project to formalize the Lessons Learned process. Too many organizations incorrectly refer to

the Lessons Learned process as a “post-mortem analysis.” This discredits the entire activity by giving

it a negative connotation rather than acknowledging the potential value.

The PDR evaluation involves 3 elements:

t Feedback from the key customer's stakeholders - This is an opportunity to understand

how your customers perceive your performance

t Feedback from the project team - This is basically an internal self-examination of your

own performance.

t Feedback from the end users - This is feedback from the actual users regarding the quality

of the delivered system with respect to characteristics such as usability, performance,

fulfillment of need, etc.

The PDR should be conducted soon after the project is completed while the experiences are

fresh on team members’ minds. Since the delivered system must be operational for some time

before the users can really make a meaningful assessment, the collection of user satisfaction is

addressed as a separate topic.

Feedback from the key customer's stakeholdersThe first part of the Post Delivery Review is the collection of feedback from the customer

organization. To be effective, the delivery team and customer team must have a relationship in

which the customer team feels comfortable discussing the topics frankly and objectively.

A questionnaire should be used to provide structure to this process. The questionnaire facilitates

a face-to-face dialogue where the goal is to obtain a solid understanding of how the customer

perceives the project was performed, not just to obtain the completed form. The questionnaire

must be a short, pre-defined form, i.e., one page maximum. It should use a five-point scale

format as shown below. This type of structure enables consistent summarization and comparison

across respondents within a customer and across all customers.

1

“Please check appropriate answer:”

StronglyAgree

SomewhatAgree

NoOpinion

SomewhatDisagree

StronglyDisagree

3

Copyright SSG, Ltd. - 2013

4

The questionnaire should start with two or three General Assessment questions. For example,

“I would use this project team again for my next project,” will quickly cut to the respondent’s

real feelings. The questionnaire should then contain eight to ten questions regarding more

specific topics of interest, e.g., “The project team consistently meets schedule commitments,”

or “Contract issues were consistently managed in a timely and competent manner.” Finally,

the questionnaire should wrap up with two to three open ended queries requiring statements

in response, e.g., “What did you like best about working with the delivery team?/What did

you like least about working with the delivery team?”

It is critical to the efficacy of the PDR that the responses to the questionnaire are obtained in a

face-to-face dialogue with members of the customer team. It is not just a form to be simply

emailed to respondents with a request to complete and return to the sender. At a minimum,

customer personnel to be interviewed should include: the sponsor, business/operations group

members, validation team members and end user group members.

Feedback from the project teamThe cornerstone of an organization's continuous process improvement

program is its commitment to self-examination.

Part two of the PDR is an internal assessment by the delivery team. It starts with a “Just give me

the facts“ summary of the project.

Financial Performance:

Compared to budget/forecast -

t Did the project attain its cost goal?

Work performance:

Compared to estimates -

t Did the project labor actual expenditures correspond to estimates

Schedule performance:

Compared to the approved schedule -

t Did the project deliver items in accordance with the schedule?

Technical performance:

Compared to the baseline requirements list -

t Did the delivered solution fulfill all key requirements

2



The Project Manager provides this information to the team. Then a more qualitative assessment of

development practices employed on the project is conducted:

Compared to company best practices and standards -

t Was a complete and accurate definition of requirements prepared?

t Were changes to requirements managed effectively throughout the project?

t Did the design process result in an optimal architecture? (i.e., looking back, was it the

best answer?)

t Were implementation standards used consistently?

t Was an effective integration strategy developed & implemented to facilitate an effective,

systematic build of the system?

t Were tools used effectively?

t Was an effective testing strategy developed and implemented resulting in an optimal set

of test cases performed?

Finally the project management practices used on the project are assessed:

t Was a complete and accurate project plan developed and implemented for the project?

t Were roles and responsibilities clear? (i.e., who was going to do what and when)

t Was the status of tasks tracked against the project schedule effectively?

t Was the status of expenditures tracked against the project budget effectively?

t Were effective risk assessment and mitigation actions performed regularly?

t Were interactions with the customer effective?

t Were team member career development tasks performed effectively? (i.e., performance

feedback, training)

The Lessons Learned log maintained throughout the project provides a means to refresh the

team’s collective memory as well as start the discussion on recommendations for the future.

The customer feedback is included to support the internal evaluation. This must be filtered of

any negative attribution to specific individuals. Too often, the PDR becomes a forum for team

members to air their gripes against management. To avoid this, facilitated brainstorming

techniques should then be used to gain consensus as well as rankings of (1) the root causes

of problems, and (2) keys to positive outcomes and ways to encourage those in the future. The

results of the PDR should be published to the company’s knowledge base where it is available

for all of the project participants and, of course, non-project members of the organization.

5


6

The final step in getting feedback from the project team is the real challenge. It’s based on the

Dennis the Menace conundrum: when Dennis hears his mom yelling for him to come home,

he immediately begins to ask himself “Hmmm … did I do something I shouldn’t have? Or did

I not do something I should have?” In order to really derive organizational value from the

experiences of the project team, the lessons learned resulting from the PDR (regarding both

positive results and negative consequences) must be evaluated against the guiding policies

and procedures for the organization. Of course, that assumes the organization has such

practices and uses them consistently—but that’s a topic for another discussion. For each Lesson

Learned, the root cause is evaluated to determine if there are provisions in the company’s

defined process regarding the lesson learned topic and to what degree were the process

provisions implemented during the project. The results can be summarized in a simple plot.

The diagram is not as important as the thought process behind the diagram. Obviously, those

items that are in the “process poorly defined and poorly implemented” quadrant are easy

candidates for developing guidelines to improve performance in the future. Similarly, lessons

in the “well defined and well implemented” quadrant reinforce the value of standardized

processes. The lessons in the “well implemented and poorly defined” category may be the

result of an individual’s outstanding efforts. These are candidates for inclusion in the corporate

procedures so another team on different project can achieve the same benefit. The harder

questions involve those items that are in the “well defined and poorly implemented” quadrant.

Those questions typically require management consideration to affect future performance.

Take away points -

1. Lessons Learned should be captured throughout a project. A centralized location in the

project’s repository is a great mechanism to capture team members’ contributions.

2. A structured method of collecting and analyzing Lessons Learned information must be

consistently applied over time.

Poorly defined

Well implemented

Poorly implemented

Well defined

1

4 2

3



3. Performance assessment information for the project comes from the customer as well as

honest self-examination by the project delivery team.

4. The value of customer feedback is the dialogue, not the form.

5. The Lessons Learned process is an investment. Real value is realized in changes to the

organization’s guiding policies and procedures, not a bunch of dusty old Lessons

Learned documents sitting in someone’s email archive folder.

About SSGSSG is a software processional services firm that specializes in building trust-based partnerships

and solving complex problems in business-critical applications for our clients. Our ideal client

has made significant investments in technology and values a partner that they can depend on

to bring a depth of experience and solve their problems right the first time.

Our Billing and Revenue Management Practice specializes in delivering optimized Oracle BRM

solutions by providing deployment, enhancement, customization and support services in the most

cost-effective manner. The practice represents 15 years of experience with the implementation

of Oracle BRM, a fully convergent, real-time enterprise revenue management application that

is perfectly suited for communications, media and cloud services provisioning.

Within our Data Management Practice, we blend our technology and business expertise,

leveraging a proven and repeatable methodology to deliver expert Oracle DW/BI project

consulting. And in partnership with Informatica, we deliver enterprise data integration software

and services helping organizations gain great business value from all their data – on the

premises, with partners or in the cloud.

Our clients would tell you that they count on us for straight-talk, hard work and to quickly solve

problems at reasonable rates. They trust us to put their business challenge before our interests

and not to just make more work for ourselves at the cost of their budget.

We have a unique culture that highly values:

t Bringing integrity back to the professional services industry

t Doing the job right the first time

t Solving problems so they stay solved

t Remaining at the forefront of today’s technology

t Producing real business value for our clients

t Helping each other, our partners and the community

t Having fun and enjoying life

7


oracle brm projects: do you really learn your “lessons learned€¦ · a questionnaire should be...

Documents