statistical quality control the experience of the banco de portugal antónio agostinho maria josé...

Statistical Quality ControlThe experience of the Banco de

Portugal

António AgostinhoMaria José Valério

Banco de PortugalStatistics Department Statistics Audit Unit

Q 2008 - European Conference on Quality, Rome July 2008

Statistics Quality Control: the experience of the Banco de Portugal

Q 2008 - European Conference on Quality, Rome, July 2008 2

1. The Statistics Department2. Quality Control Procedures3. Statistics Audit4. Institutional Cooperation5. Conclusions and Prospects

Ahead

Outline



1.Statistics Department Created in 1997 to cope with rising statistical requirements (in particular in the framework of the Monetary Union)

Centralized statistical function allowing:

The concentration of the available statistical skills;

The increase of capacity to face the challenges resulting from building up the European System of Central Banks;

The improvement of efficiency gains and the enhancing of statistical quality control at all levels of the statistical process (collection, processing, analysis and dissemination);

The adoption of consistent methodological approaches; and,

An easier data sharing and coordination among all the staff in charge of statistical production.



1.Statistics Department

Employees with university degreeEmployees with university degree 88 ( 56% ) 88 ( 56% )

Other employees Other employees 69 ( 44% ) 69 ( 44% )

Total 157Total 157

National Financial Accounts and Securities Statistics Division

Total: 23 (Univ. Deg. 17)

National Financial Accounts and Securities Statistics Division


Balance of Payments and Int.Invest. Position Statistics Division


Balance of Payments and Int.Invest. Position Statistics Division


Monetary and FinancialStatistics and Central Credit

Register DivisionTotal: 20 (Univ. Deg. 11)

Monetary and FinancialStatistics and Central Credit

Register DivisionTotal: 20 (Univ. Deg. 11)

Central Balance-SheetDivision


Central Balance-SheetDivision


Statistics Dissemination UnitTotal: 10 (Univ. Deg. 8)

Statistics Dissemination UnitTotal: 10 (Univ. Deg. 8)

Central Credit RegisterInformation Unit


Central Credit RegisterInformation Unit


Statistics Audit UnitTotal: 3 (Univ. Deg. 3)

Statistics Audit UnitTotal: 3 (Univ. Deg. 3)

MethodologicalDevelopments Unit


MethodologicalDevelopments Unit


Executive ManagementTotal:2 (Univ. Deg. 2)

Executive ManagementTotal:2 (Univ. Deg. 2)

Staff Support UnitTotal: 13 (Univ. Deg. 3)

Staff Support UnitTotal: 13 (Univ. Deg. 3)

December 2007

Organization Chart



1.Statistics Department

Four Divisions Four different areas of statistical

production

Vertical organization in each Division

Regular hierarchical statistical production meetings (on

a monthly basis)

Internal technical task forces to deal with transversal

issues



2.Quality Control Procedures

Quality control procedures along all the phases of

statistical production process: Data collecting Data processing and analysis Statistics dissemination



2.Quality Control Procedures Data Collecting -

1

Data collecting is based on legal provisions Involvement of the reporting institutions is pursued For more complex changes in collection systems working

groups are created Detailed reporting instructions and handbooks are delivered

to the reporting institutions Appointment of contact persons – respondents for each

specific statistical data submission (training sessions and

regular meetings)



2.Quality Control Procedures Data Collecting -

2

Specific designed software is made available to the

reporting institutions - IT tests and consistency tests on

elementary data BP net – a secure electronic communication system

between the Banco de Portugal and the other financial

institutions Feedback to reporters – the CCR and CBS experiences Statistical reporting indicators



2.Quality Control Procedures

Quality Assessment Reports – Major Financial Groups

Cross-checking data reported by MFIs: MFIs’ report to money and banking statistics (MBS)

versus accounting data MFIs’ report to MBS versus report for Central Credit

Register MFIs’ report to MBS versus data collected/reported for

securities statistics Securities portfolios versus securities issues MFIs’ report to MBS versus report for BoP statistics

(external assets and liabilities) High level periodical statistical meetings

Data Collecting - 3



2.Quality Control Procedures Data processing and analysis - 1

Full compliance with the international methodological

framework: concepts and definitions scope of statistics classification and sectorization systems valuation rules recording principles

Adjustments and/or appropriate substitute measures

are developed when the source data are not in

accordance with the standards



2.Quality Control Procedures Data processing and analysis - 2

Validation tests performed on individual and/or

aggregated data and also at different phases of data

processing – source data, intermediate data and final

statistical data: internal consistency temporal consistency cross-check control

Additional control in the framework of Financial

Accounts compilation



2.Quality Control Procedures Statistical Dissemination -

1

Meeting the users’ needs and the statistical reporting

requirements of International Organizations Main features of the statistics dissemination:

Release in time according to a pre-announced schedule

Clear and understandable presentation Workshops with media and media references

monitoring Monthly, free of charge, to all users at the same

time, in the BdP website:• Statistical Bulletin (also available in paper and

CD-ROM)• BP stat – allows online access to the interactive

database of BdP Assistance to users – contact points in BdP website and in

the Statistical Bulletin



2.Quality Control Procedures Statistical Dissemination - 2

BP stat | Statistics online | facilities: Portuguese and English versions; Data under two formats: time series and

multidimensional analysis; Data in hierarchical structure – tree of statistical

domains; Data and metadata may be accessed, downloaded or

printed; Internet links to the relevant metadata references; Predefined and/or personalised tables (“Favourites”

option); “Alert” messages by email to the users; and, Glossary, search, calendar and help options are

available.



2.Quality Control Procedures Quality Control

Matrix

Outside the

production cycle

Appointment of contact persons Detailed reporting instructions and handbooks

Regular meetings with the reporting institutions

Training sessions Feedback information to the reporters

"Quality Assessment Reports“ High level periodical statistical meetings Internal technical task forces

During the

production cycle

BP net – electronic communication

system

Specific designed software - IT tests -

Validation tests: internal consistency temporal consistency cross-check control

Regular statistical production meetings

Consistency tests on elementary data

Additional control in the framework of Financial Accounts

Totally automated Judgment needed



3.Statistics Audit Main purposes

(1)

Analyse and evaluate the different phases of statistical production procedures in place, from source data collection up to final statistics compilation and dissemination

Analyse the organizational and functioning aspects, in particular evaluate the burden and the share of functions and responsibilities in the statistical production process

Evaluate the efficiency of the procedures in place, identifying the good practices as well as improvement needs



3.Statistics Audit Main purposes

(2)

Contribute to enhance methods and applied

techniques and to improve the statistical

production quality by issuing, when it applies,

suggestions/ recommendations on current practices

or by suggesting new procedures and/or

organizational arrangements

Encourage and promote the share and

comparison of good practices and procedures

among the different areas of the Statistics

Department



3.Statistics Audit Methodological approach – AK

list (1)

A. Initial appointment with the responsible of the

unit in charge of the statistics under audit

B. Collect, organize and analyse the

documentation made available by the unit

C. Submission of the “Self Assessment

Questionnaire”

D. Fieldwork

E. Definition of a flow chart on the information

circuit

F. Preparation of the Check List of procedures



3.Statistics Audit

G. Joint analysis with the unit’s manager of the Self Assessment Questionnaire

H. Graphical assessment of the statistical production system (radar chart)

I. Report on the statistical audit process (1st draft) – submission to the unit under audit

J. Joint evaluation of the comments from the unit under audit

K. Final report on the statistical audit process – submission to the Head of the Statistics Department

Methodological approach – AK list (2)



3.Statistics Audit

The self assessment questionnaire is the basis for

the presentation of the radar chart

Two sets of answers to the questionnaire – one, in

the beginning of the audit process, from the unit

under audit and another, at the end, from the

Statistics Audit Unit – are contrasted graphically in a

statistical production assessment diagram

Self Assessment Diagram (1)



3.Statistics Audit

Self Assessment Diagram(Securities Issues Unit (SIET)

0

1

2

3

4

5Staff Functions Share

System Adequacy

User Satisfaction

Overall Relevance

Transparency Degree

Overall Coverage

Methodological Soundness Degree

Elementary Data QualityReporting Compliance

Intermediary and Final Data Validation

Dissemination Punctuality

Temporal Consistency

Consistency with other Statistics

Dissemination Clearness and Coverage

Metadata Accessibility

SIET Unit

Accessibility Relevance

Serviceability

Accuracy and Reliability

Integrity

Methodological Soundness

1 - Very low2 - Low3 - Medium4 - High5 - Very high




3.Statistics Audit

Self Assessment Diagram(Securities Issues Unit (SIET) versus Statistics Audit Unit (NAE))

0

1

2

3

4

5Staff Functions Share

System Adequacy

User Satisfaction

Overall Relevance

Transparency Degree

Overall Coverage

Methodological Soundness Degree

Elementary Data QualityReporting Compliance

Intermediary and Final Data Validation

Dissemination Punctuality

Temporal Consistency

Consistency with other Statistics

Dissemination Clearness and Coverage

Metadata Accessibility

SIET Unit

NAE

Accessibility Relevance

Serviceability

Accuracy and Reliability

Integrity

Methodological Soundness

1 - Very low2 - Low3 - Medium4 - High5 - Very high




3.Statistics Audit Overall assessment of the statistical audit

operations (1)

The overall assessment is quite significant:

1. The deep interest and good collaboration revealed by those intervening in the statistical production systems audited

2. These operations allow a detailed assessment on the current features of the statistical systems in place

3. The possibility of conveying upwards the desires and aspirations as well as the shortcomings faced by the responsibles of the units under auditing, which, otherwise, might encounter more constraints before being acknowledged



3.Statistics Audit Overall assessment of the statistical audit

operations (2)

4. The share in all Department of procedures and good practices in place

5. Most of the suggestions/ recommendations raised by the statistical auditors and accepted by the units under audit are already being implemented or are included in the unit’s planning for the years ahead

6. The relevance of the follow-up of the implementation of suggestions/ recommendations



4.Institutional Cooperation

More efficient statistical production processes:

Facilitates data sharing and good practices’

diffusion; Promotes the use of harmonized statistical

methodologies - thus improving statistical data comparability;

Helps reducing reporting burden and data redundancy; and,

Enhances the clarification of responsibilities and areas of competence of the parties involved.




Some highlights: Banco de Portugal became a national statistical

authority in the same level playing field as Statistics Portugal, without prejudice of their independence;

This new law emphasizes the importance of ensuring an efficient cooperation and the need for them to develop models of cooperation considered as adequate to fulfil their role in the NSS; and,

The principles of the European Statistics Code of Practice are also formally adopted in this new legal framework.

New Statistical National Law (May 2008)




- In the framework of the High Council of Statistics (HCS) – the body on top of the Portuguese NSS;

- Other bilateral/multilateral initiatives:

1) 1998 - Protocol with Statistics Portugal (SP) and Ministry of Finance (MF) in the field of IMF Special Data Dissemination Standard (SDDS);

2) 1998 - Protocol on the compilation of Portuguese National Accounts - shared responsibility between SP (non financial accounts) and BdP (financial accounts) – updated in 2001 and 2007;

3) 1999 – Protocol with SP for the joint conduction from 2000 onwards, in partnership, of a Quarterly Survey on non Financial Corporations;

4) 2004 – Two protocols in the field of Balance of Payments, with SP and General Directorate of Tourism, focusing on a survey of cross-border movement of travellers and a survey on international tourist expenditure (the last was conducted in 2007);

In national terms (1)




5) 2006 - Protocol with SP, (the first was signed in 2000) for the conduction of a Survey to the Households Indebtedness and Wealth;

6) 2006 – Protocol with SP and MF, on the compilation of General Government Statistics, Public Debt Statistics and analysis of Excessive Deficit Report;

7) 2006 – Joint work with SP for the development of a unique File of Statistical Units for the whole NSS;

8) 2006 – Protocol with SP, MF and Ministry of Justice to define a single electronic data submission with data from the financial statements of non financial corporations – IES (formally created by Decree-Law n. 8/2007 of 17th January).

In national terms (2)




- With ECB and the other EU national central banks in the framework of ESCB;

- With Eurostat and the EU national statistical offices under the umbrella of the Memorandum of Understanding (MoU);

- In the framework of CMFB (currently the Head of the BdP Statistics Department is also part of the CMFB Executive Body);

- With other international organizations namely IMF, BIS, World Bank and OECD;

- Technical assistance – several activities in particular, but not exclusively, with the national central banks ant the national statistical offices of the Portuguese-speaking Countries.

At international level



5.Conclusions and Prospects Ahead

Further improvements: Define a set of structured quality indicators, to be

compiled on a regular basis, on the quality of the statistical production;

Improve the already existing quality manuals, exploring further these documents both in a user’s oriented perspective and in a statistical quality control framing;

Enlarge the feedback information currently produced and returned to the reporting institutions;

Define mechanisms to have an adequate feedback from statistics’ users (on statistical data already currently available and on the users’ future needs).

The appraisal on the work developed so far is quite significant.



Thank You

for your Attention

statistical quality control the experience of the banco de portugal antónio agostinho maria josé...

Documents

statistics quality control

financial statistics

statistics audit unit

portugal q

european conference

staff support unit total

statistical process

executive management