Direction and system changes impacting on data editing and

imputation at Statistics New ZealandPaper by Emma Bentley and Felibel Zabala,

presented by Emma Bentley

May 2011

Drivers for change

Economic and social changesProduce relevant and timely statisticsMeet users needsSustainable systems, flexible enough to adapt to changing needs and usesMaximise efficiency and value

2

Organisational direction

Statistics 2020 – Te Kāpehu Whetū

Strategic Plan 2010-20

Direction requires organisational change, to achieve our vision

3

“An informed society using official statistics”

Strategic Priorities

Lead the NZ Official Statistics System so that it effectively meets the country’s needs for relevant, trustworthy, accessible informationObtain more value from official statisticsTransform the way we deliver our statistics

Create a responsive, customer-focused, influential, sustainable organisation

4

A model of sound practice for statistical production, with standard process and infrastructure across collections.

Delivery Transformation

Generic Business Process Model (gBPM)Methodological standard for E&IStandardised processing platformsStandardised tools

5

Delivery Transformation

Improve efficiency of data processingPerform less manual editingBetter use of quality measuresIncrease use of administrative data

6

Generic Business Process Model

Guide to achieving standard processesBenefits• Speed up development and creation of new statistics• Focus more time on complex and in-depth analysis• Decrease differences between business units• Staff can rotate around business units more easily

7

Methodological Standard for E&I

E&I NetworkComprehensive source of objectives, principles, requirements and guidelinesEnsure effective and efficient data editing processes and strategiesApproval and adherence to the Standard - continual improvement of process phase of the gBPM.

8

Methodological Standard for E&I

All outputs must have a documented E&I planAutomate editing processes where possible, minimising manual interventionImplement efficient editing techniques such as macro and selective editingEnsure users are better informed about the quality of our data and statistical outputsEnsure staff have access to E&I documentation and training

9

Processing platforms

Need for new infrastructure to create our desired environment A platform is a logical cluster of functionality that enables components to be put together to provide a complete end-to-end systemPOSS – social statisticsBESt – administrative and micro-economic data

10

Platform requirements

Support continuous improvement, flexible and user-configured, user-maintainedNew developments should use standard E&I toolsProduce process flags at each step to indicate which value has been manipulated and by which methodAutomate the E&I process where possible

11

Future state

More platforms based on common infrastructure and survey clusters developed

12

Overseas Trade in BESt

Overseas Trade and sub-annual survey series, retirement of 8 legacy platformsIncreased amount of automated data editingIntroduction of quality measuresReuse of existing packages, new builds for methodologies not currently in BEStMembers of different teams involved in planning, development, testing – IT, subject area, Methods teams

13

Standard Methodology Toolbox

Tools used in the statistical process endorsed and supported by Statistical Methods divisionWork with IT to ensure the Toolbox aligns with the applications and software that they support and maintainIT understanding and support important for implementation of methodological tools in the new platformsToolbox is flexible as future requirements progress

14

Standard Methodology Toolbox

E&I tools• Banff - BESt• CANCEIS - POSS• SELEKT• SEVANI• Logiplus

Graphical editing tool in future?Subject areas need to increase capability for these tools, Toolbox is a one-stop-shop of information

15

Evaluation process for tools

Joint Methods/IT project on how to assess and implement methodology toolsEnsures technological compatibility and methodology need considered together and early in any evaluation processNew process consists of evaluation, procurement and implementationSELEKT and SEVANI used to evaluate evaluation phase

16

Standard process quality measures

Another joint Methods/IT projectImplement a standard set of quality measures for business surveys covering the gBPM using Analysis CubesIf successful, potential implementation in BEStMethodological Standard supports better use of quality measures for continuous improvement

17

Culture change

Statistics 2020 – making the strategic direction accessible to all staffMulti-team involvement in these projects that are promoting change

18

Culture change

Standardise processes, align to the gBPMGetting subject areas to accept current level of manual intervention is inefficientMaximise common methods and infrastructure, minimise maintenance costsMore efficient processing, more analysisGain more from available data

19

Future initiatives

Migrate more social surveys to POSS platformMigrate more micro-economic sources to BESt platform, develop macro editing functionsComplete macro-economic platform buildImplement quality frameworkEnhance ToolboxExpanded use of admin data sources and long term direction

20