European Conference on Quality in Official Statistics

8-11 July 2008

Mr. Hing-Wang Fung

Census and Statistics Department

Hong Kong, China (Email: hwfung@censtatd.gov.hk)

2

A Further Step in

Quality Assurance

for the Official Statistics of

Hong Kong, China

3

Endeavours to Ensure Quality of Official Statistics

Department’s vision statement:“To provide high-quality statistical services, contributing to the social and economic developments of Hong Kong.”

Pre-requisites: good management of statistical systems an enabling working environment for

continuous quality improvement

4

Quality Assurance Review Programme

Started with the documentation of each process of statistical systems

Next Steps Development of Data Quality Assessment

Framework Improvement on statistical processes of

statistical systems

Key features involve promoting self-assessment and enhancement

of individual statistical systems; and conduct of in-depth third party review by an

independent quality assurance team

5

Recent Developments in Quality Assurance Review

Developed a set of general guidelines on preparation of proper documentation of statistical systems containing: essential items that should be covered; salient points that should be noted;

and good examples and practices for each

statistical process

6

Recent Developments in Quality Assurance Review (Cont’d)

Conducted self-assessment of documentation by the subject

professionals in accordance with the set of general guidelines

7

Recent Developments in Quality Assurance Review (Cont’d)

Developed a central database on good practices in respect of documentation for use within the Department

8

Data Quality Assessment Framework

Quantitative through a set of quality i

ndicators modelled on the DESAP

checklist promulgated by Eurostat

Qualitative through a series of

open-ended questions under the Data Quality Assessment Framework developed by the IMF

+

Existing New

Documentation Review

Assessment of quality level achieved for statistical products

9

Preliminary Set of Quality Indicators

Criteria for devising the quality indicators: Manageable in size Relevant to the statistical products compiled Easy to compile Easy to interpret Allowing for comparison over time Representative for the quality dimensions being

adopted Adaptable

10

Preliminary Set of Quality Indicators (Cont’d)

Devised 19 quality indicators under 6 quality dimensions Translated to 19 assessment questions

With pre-defined response categories 15 of them in five-point ordinal scales with brief

explanation for each point, reflecting quality in descending order from 5 to 1

Nearly half of these 15 questions (i.e. 7 questions) are under Relevance and Accuracy

4 of them either for assisting assessors in determining rating for some of the 15 questions or giving hints to the subject professionals on area for improvement

11

Preliminary Set of Quality Indicators (Cont’d)

Dimension Element Indicator

1. Relevance Identification of users 1. Means to identify user groups and their frequency

2. Information available on the usersAssessment of user

satisfaction3. Means to collect information on user satisfaction

4. Information available on user satisfactionMonitoring user needs 5. Means to identify users’ needs of information

6. Information available on users’ needs of information

2. Accuracy Assessment of non-responses

1. Unit non-response rate

2. Classification of unit non-responses for adjustments

3. Imputation rate of key variableAssessment of sampling

error4. Coefficient of variation of the statistics

Assessment of revision between provisional and final statistics

5. Extent of revisions between the provisional and final key statistics

in five-point ordinal scales, reflecting quality in descending order from 5 to 1

12

Preliminary Set of Quality Indicators (Cont’d)

Dimension Element Indicator

3. Timeliness Production time 1. Time lag between the reference period and the release date of the preliminary results

2. Time lag between the reference period and the release date of the report

4. Accessibility Assistance to users on application of statistics

1. Percentage of enquiries meeting the performance standards and targets of C&SD on the waiting time

Metadata accessibility 2. Ease of getting metadata by the users

5. Comparability Comparability over time 1. Extent of comparability of the statistical product over time

Comparability across domains

2. Extent of comparability of the statistical product across different domains

6. Coherence Coherence of results for different frequencies

1. Extent of coherence of the statistics based on results for different frequencies

Coherence with other related statistical products

2. Extent of coherence of the statistics within the same socio-economic area

in five-point ordinal scales, reflecting quality in descending order from 5 to 1

13

Illustration of Data Quality Assessment Mechanism

Step 1: Set a benchmark for each of the 15 assessment questions by respective subject professional, specifically for each of their statistical products

e.g. How do you appraise the time lag between the reference period and the release of the report for the statistical product?Name of report: Report on Annual Survey of Industrial Production

CheckBox1CheckBox1CheckBox1CheckBox1CheckBox1

Rating Response Benchmark

Time lag between the reference period and

the release date of the report

1 There is a substantial time lag 16 months

2 There is a large time lag 14 - <16 months

3 There is a certain time lag 12 - <14 months

4 There is a small time lag 10 – <12 months

5 There is a very small time lag < 10 months

14

Illustration of Data Quality Assessment Mechanism (Cont’d)

Step 2: Moderate all the benchmarks by quality assurance (QA) review team

Step 3: Moderate and endorse the set of benchmarks of each statistical product by respective senior officer

Step 4: Answer the assessment questions by respective subject professional based on the set of benchmarks endorsed by senior officer

Step 5: Mark the responses of questions in the assessment diagram and compile the product quality assessment score by QA review team

CheckBox1CheckBox1CheckBox1CheckBox1CheckBox1

15

012345

Identification of users

User satisfaction

User needs

Unit non-response rate

Imputation rate

Coefficient of variation

Revisions between provisional and finalstatistics

Time lag for preliminary resultsTime lag for report

Waiting time for enquiries

Metadata

Comparability over time

Comparability across domains

Coherence for different frequencies

Coherence within same area

Illustration of Data Quality Assessment Mechanism (Cont’d)

Product Quality Assessment Diagram

Statistical Product: Statistics on the operating characteristics of quarrying, manufacturing, and electricity and gas sectors

16

01

23

4

5

A

B

Illustration of Data Quality Assessment Mechanism (Cont’d)

In this example,

the score is 49.

100x )( diagram theofarea Total

)( indicators individual of ratings thejoiningby formedArea

Score AssessmentQuality Product

BA

A

Its value ranges from 4 to 100.

17

Illustration of Data Quality Assessment Mechanism (Cont’d)

Interpretation of the results of assessment based on the product

quality assessment diagram and score: Quality of the statistical product is not poor but still far from

perfect. Quality level of the statistical product varies a lot among the

different dimensions. Relevance is the weakest dimension, in particular the

identification of users. Accuracy level is assessed to be just on the average and hence

some effort should be put in improving this area, in particular the reduction of non-responses.

The performance of the product in other dimensions are above average.

18

Limitations of the Product Quality Assessment Diagram and Score

Modification of the diagram and formula for computing the score necessary if some indicators are not applicable

Equal weights assumed for the 15 indicators

The score is ordinal in nature

Different permutations of the ratings for the 15 indicators may result in same score

Sequence of indicators may affect the value of score

19

Way Forward

To supplement and refine the set of quality indicators of statistical products and conduct the assessment on a regular basis

To further promote a culture of systematic improvement drive within the department

To improve the statistical processes, concepts and methodology of each statistical system through continuous assessment

20

Thank you !