Independent Evaluation: Insights from Public Accounting

Abigail Brown (Harvard)Jacob Alex Klerman (Abt Associates)

Presentation for Improving the Quality of Public Services, Moscow, June 2011Independent Evaluation: Insights from Public AccountingThis Paper/Presentation Reviews the history, theory, and experience in independent audits for insights about independent evaluationsOur thesis: In both fieldsThe disjunction between who oversees the audit/evaluation and the ultimate consumers of the resultsLeads to problematic incentives for the auditor/evaluator and bad outcomesBut that recognizing the disjunction and the experience in auditing suggest several feasible changes that would improve the situation 2Brown & KlermanMay 9, 2011May 9, 20112Brown & KlermanOutlineThe Auditing Problem(s)Some Auditing SolutionsThe Analogy to EvaluationDiscussion3Brown & KlermanMay 9, 2011May 9, 20113Brown & KlermanOwners and ManagersProvide the capital

OwnersManagersManagers run the firm day to dayMay 9, 20114Brown & KlermanBut Who Watches the Mangers?Owners are too busy to watch wellBut need to watchMalfeasanceInsufficient effortHire/FireIncentive compensation (bonuses, stock options)May 9, 20115Brown & KlermanOwnersManagersEnter Independent AuditorPaid by owners(corporate funds)

To watch (and report on) managersAuditorMay 9, 20116Brown & KlermanOwnersManagersOrigins in 1840s EnglandRequired by 1934 (U.S.) Securities Exchange ActBut Why Should this Work?Managers Choose auditorsthis time and next timeand how much to pay themOversee auditors contract, including scopeCan give auditors other (non-auditing/consulting) workSo why should auditors report truthfully to owners?Why not just report what managers want reported?May 9, 20117Brown & KlermanPaid by owners(corporate funds)

To watch (and report on) managersAuditorOwnersManagersOutlineThe Auditing Problem(s)Some Auditing SolutionsThe Analogy to EvaluationDiscussion8Brown & KlermanMay 9, 2011May 9, 20118Brown & KlermanFour Partial SolutionsProfessionalismReputationLiabilityOwner OversightMay 9, 20119Brown & KlermanAICPA Code of ConductProfessionalismReputationLiabilityOwner OversightA distinguishing mark of a profession is acceptance of its responsibility to the public. In discharging their professional responsibilities, members may encounter conflicting pressures from among each of those groups. In resolving those conflicts, members should act with integrity, guided by the precept that when members fulfill their responsibility to the public, clients' and employers' interests are best served. ..Those who rely on certified public accountants expect them to discharge their responsibilities with integrity, objectivity, due professional care, and a genuine interest in serving the public. May 9, 201110Brown & KlermanA Simple Equilibrium ArgumentProfessionalismReputationLiabilityOwner OversightSuppose there are two types of auditors independent and not independentThe auditor selected by management is publicly observableWhat would owners conclude if management chose a not independent auditor?So management has to choose an independent auditorMay 9, 201111Brown & KlermanSue Them ProfessionalismReputationLiabilityOwner OversightAuditors can be sued for errorsHave been large payoutsShould be a strong financial incentive for independenceBut, liability requiresClear standards for malfeasancewhich the profession has tried hard to avoid Strong liabilitywhich the profession has successfully weakenedMay 9, 201112Brown & KlermanSOX/Sarbanes-Oxley Act of 2002ProfessionalismReputationLiabilityOwner OversightIf the problem is too little owner (too much management) involvement in the audit, fix that directly! SOX requires that an Independent Audit Committee of BoD/Board of Directors (i.e., excluding management)Hire the auditorOversee the auditApprove other business with the auditor (to prevent side payments) May 9, 201113Brown & KlermanOutlineThe Auditing Problem(s)Some Auditing SolutionsThe Analogy to EvaluationDiscussion14Brown & KlermanMay 9, 2011May 9, 201114Brown & KlermanEvaluation is Like AuditingEvaluationAuditingGovernment agencyCorporationIdeally wants both the appearance of independentevaluationauditand the ability to claim thatthe program worksthe books are correct/ we are making profits15Brown & KlermanMay 9, 2011May 9, 201115Brown & KlermanWhat are the implications of experience in independent auditing for independent evaluation?Applying these Strategies to EvaluationProfessionalismReputationLiabilityOwner OversightMay 9, 201116Brown & KlermanSame Problems as in AuditingProfessionalismReputationLiabilityOwner OversightAlso problematic in evaluationEvaluation is also a businessStakes are high: own employment and bonusPerhaps professionalism more effective in evaluationPublic release of results and methodsResults published in academic journals Results reviewed and used by academicsMay 9, 201117Brown & KlermanDoes Reputation Matter?ProfessionalismReputationLiabilityOwner OversightTo some extentBut clients also seem to value (perhaps value more) a reputation for Client serviceNot revealing clients secretsNot publishing without permissionMay 9, 201118Brown & KlermanLiability Does Not Seem OperativeProfessionalismReputationLiabilityOwner OversightThere does not seem to be any statutory basis on which to sue evaluators for failure to be independentWho would sue?For what?What are the damages?Ideas for making this work?May 9, 201119Brown & Klerman0wnerOversight Seems PromisingProfessionalismReputationLiabilityOwner OversightCurrently, agency Selects the evaluatorOversees the evaluationReviews the report and any presentationsUp to and including (indefinitely) preventing publication and presentationFor evaluation seems easier to changeBy analogy with SOX Audit Committee reformsMay 9, 201120Brown & KlermanBy Analogy with SOX, Audit Committee ProfessionalismReputationLiabilityOwner OversightExpand membership of TWG/Technical Working Group to includeCongress (e.g., CBO, GAO)Higher executive (e.g., OMB, agencys Inspector General, agencys PA&E) General policy community (academics, think tanks from both sides)Change role of TWGSelect the evaluatorOversee the evaluationReview the report and any presentationsDecide what and when to release reports

May 9, 201121Brown & KlermanEncourage Independence by ProfessionalismReputationLiabilityOwner OversightAgency comments on, but need not approve, materials going to TWGRemove restrictions on evaluator contact with the TWGEncourage reporting of attempts by agency to subvert independence of the evaluationMay 9, 201122Brown & KlermanOutlineThe Auditing Problem(s)Some Auditing SolutionsThe Analogy to EvaluationDiscussion23Brown & KlermanMay 9, 2011May 9, 201123Brown & KlermanOwner Oversight in PracticeExplicitly specify which tasks are independent evaluationsSometimes in statutespecific evaluations, certain evaluation fundsSometimes by programs wanting strong evidence of effectivenessDefine independent evaluation in the FAR/Federal Acquisition Regulations; i.e., A TWG that includes key consumers, and Rules for how the TWG would operateLimit other business contact between independent evaluators and agencies (to prevent side payments)24Brown & KlermanMay 9, 2011May 9, 201124Brown & KlermanPlus, Facilitate Market for ReputationPublicize deviations attempts to subvert independenceBy the agency, andBy the evaluatorDistinguish independent evaluations from othersIn making funding decisions (e.g., official policy reviews by NAS/IOM)At conference presentations and at journal publication

25Brown & KlermanMay 9, 2011May 9, 201125Brown & KlermanOutlineIntroductionThe Auditing Problem(s)Some Auditing SolutionsThe Analogy to EvaluationDiscussion26Brown & KlermanMay 9, 2011May 9, 201126Brown & KlermanAbigail Brown (Harvard)Jacob Alex Klerman (Abt Associates)

Presentation for Improving the Quality of Public Services, Moscow, June 2011Independent Evaluation: Insights from Public AccountingEvaluation is Like AuditingEvaluationAuditingGovernment agencyCorporationIdeally wants both the appearance of independentevaluationauditand the ability to claim thatthe program worksthe books are correct/ we are making profits28Brown & KlermanMay 9, 2011May 9, 201128Brown & KlermanOwners and ManagersProvide the capital

OwnersManagersManagers run the firm day to dayFor most of human history, these were the same.May 9, 201129Brown & KlermanOwners and ManagersSeparation of owners (i.e., stockholders) and control (i.e., management) is a cornerstone of the modern corporationAllowing:Poor entrepreneurs to start companies/implement their ideasOwners to invest all of their money and to diversifyMay 9, 201130Brown & KlermanOwnersManagersAnd the Issues Are SubtleMay 9, 201131Brown & KlermanPaid by owners(corporate funds)

To watch (and report on) managersAuditorOwnersManagersPure fraud is clear, butHow much to spend looking?Where to look?Management often has a desired spin; owners may want something elseGEs earnings smoothingLehmans Repo 105Good accounting requires judgmentWhen judgment is required, which way does the accountant lean?

But ProfessionalismReputationLiabilityOwner OversightTheir own private (financial) interests are largeContinued employment, repeat audit business, consulting businessAuditings professional ethos emphasizes confidentiality and client (i.e., management) serviceNot independence (and serving owners)May 9, 201132Brown & KlermanDeviation Should be CostlyProfessionalismReputationLiabilityOwner OversightNow suppose management suggests creative accountingPerhaps with some side payment to the auditor (future auditing, consulting work, etc.)Auditor weighsSide payment vs.Loss of reputation as an independent auditor if side payment is discoveredMay 9, 201133Brown & KlermanDoes this Reputation Work?ProfessionalismReputationLiabilityOwner OversightYes: Arthur Anderson collapsed after criminal conviction in relation to Enron scandalNo: Other accounting scandals did not cause similar collapse (Arthur Anderson was the exception)Some evidence of collusion among firms (rather than competition on independence)Too few to fail (currently only four large accounting firms)May 9, 201134Brown & KlermanWill SOX Work?ProfessionalismReputationLiabilityOwner OversightDuh! Wasnt this obvious?This was best practice before SOX; but inconsistently appliedWhy not? And what does that suggest? Management often controls the BoDManagement often suggests BoD membersWho are not independentrelatives, interlocking BoD, other business relationshipsAnd board membership is lucrative, so members want to stay in managements good gracesMay 9, 201135Brown & KlermanEvaluation is Like AuditingEvaluationAuditingGovernment agencyCorporationIdeally wants both the appearance of independentevaluationauditand the ability to claim thatthe program worksthe books are correct/ we are making profits36Brown & KlermanMay 9, 2011May 9, 201136Brown & Klerman