standardising & industrialising “end to end” flows of statistical metadata within the...
Post on 15-Dec-2015
220 Views
Preview:
TRANSCRIPT
Standardising & industrialising “end to end” flows of statistical metadata within the
statistical production process
Initial practical steps at the ABSHelen Toole
Jennifer MitchellAlistair Hamilton
Structure of presentation
• Context– ABS IMTP (Information Management Transformation Program)
– Nascent international progress toward “industry” architecture for production of official statistics
• Metadata Registry/Repository (MRR) &Statistical Workflow Management (SWM)– Vision– Proof of Concept (PoC)
• Metadata “Census”• Learnings & next steps
3
IMTP Vision
An environment in which Australian Governments and the Community can easily
find, access, and combine statistical information which can then be used
confidently as an evidence base for policy, to target service delivery and to inform
decision making
IMTP• Drivers : Changed needs, changed expectations, changed opportunities
(eg data deluge) & threats (eg maintain relevance)
• Standardising & industrialising production is fundamental to delivering suites of outputs that are extensive, timely, flexible, sustainable and readily integrated for multifaceted analysis– A necessary enabler, although not sufficient on its own
• Required transformation is multifaceted– Business model, business process & practice, business applications,
organizational structure, wider national statistical system
• Producers face shared challenges, & opportunities, internationally– Strategic vision of the High-level group for strategic developments in business architecture in statistics– The case for an international statistical innovation program - Transforming national and international statistics systems
Business Process & Information• Enterprises require “process-centric” & “data-centric” perspectives on their
core business– This point is explored in more detail in GSIM presentation (Session VI)
• In our industry various classes of information are
– the core product (eg statistics), and – the core raw material (eg data)
• “Process” & “Information” are pillars for standardisation & industrialisation within IMTP– Appears fundamentally similar to thinking from Statistics Netherlands around steady
states and transformations, including information/metadata to describe & drive transformation
• Relevance of METIS CMF (Common Metadata Framework) Part C – Metadata and the Statistical Business Process
Strategic VisionFrom the High Level Group on Business Architecture in Statistics (HLG-BAS)
The road to industrialisation & standardisation
GSBPM
GSIM
Common
Reference
Model
Conc
eptu
alPr
actic
al
DDI
SDM
XSim
ulated Form
s
Builder
Simulated
Registry
Search
MRR
Harmonised
Methods Harm
onised
Tech
nology
SWM
SEMANTIC
REFERENCE
MODEL
Model (for Proof of Concept) of how vision might be
actualised locally
Unresolved discussion in ABS : Where would CORA + CORE constructs be positioned?
MRR + SWMfuture business context
Integrated “Statisticians’ Workbench” for Statistical Production (“Process Dashboard”)
Applications and services supporting statistical production
Statistical Workflow Management System (SWM)[Enables metadata driven processes & ensures efficient flows of metadata in production process]
Metadata* Registry/Repository (MRR)[Register & store all metadata used (input, output, guide, enabler) in statistical production process]
* More generally “Statistical Information” – including data and metadata
Diagram borrows heavily from Statistics Sweden’s presentation at MSIS 2011Tentative anatomy of a new generation of IT-architecture to support GSBPM-processes
Access / UserManagement
CorporateDirectory
BP InstanceRepository
Process Execution Engine
RulesEngine
RulesRepository
Statistical Workflow
MRR + SWM Conceptual Diagram
ResolutionServiceID Service
SchemaRepository
CentralisedData
Repository
CentralisedMetadataRepository
MetadataRegistry
ServicesRegistry
MetadataRepositories
DataRepositories
OtherServices
MRR
Challenges in reaching the future state
• Must support needs of 100+ statistical business processes spanning all statistical subject-matter domains– How can we ensure the information models supported, and services provided, by the MRR will meet the future
needs of each of these production processes?• The statistical business processes are necessarily heterogeneous in statistical frameworks, methodologies , required outputs
• Which existing needs and methods will need to be supported in future?– Many existing needs and methods will be harmonised during transformation
• It is not feasible to transform every single statistical business process and every single application from “As Is” to “To Be” at the same time– How to maintain consistency and integration during the period of transition where “legacy” processes and
applications (with “legacy” information requirements) need to be supported along side processes and applications transformed to a “standardised and industrialised” basis?
• Maintenance of business continuity (timely and quality assured delivery of agreed statistical outputs to the nation) cannot be risked during transition
• Require– extensive analysis (eg thorough understanding of “As Is” and “To Be”)– testing (eg Proofs of Concept)– etc (stakeholder communication and engagement, co-ordinated planning and project management,…)
GSBPM
GSIM
Common
Reference
Model
Conc
eptu
alPr
actic
al
DDI
SDM
XSim
ulated Form
s
Builder
Simulated
Registry
Search
MRR
Harmonised
Methods Harm
onised
Tech
nology
SWM
SEMANTIC
REFERENCE
MODEL
Common GenericIndustrialised Statistics
10/11 MRR Proof of Concept
Create common
frame
Select sample
Common Frame
Create survey frame
QEWS Frame
Forms design and approval
Forms
Load sample to
PIMSLabel files
Dispatch
SignificanceEditing
Time series analyses
NAB and FAS sign off
Time series to PPW
Collection and IFU
Data
Paradata(Collection
Information)
Clean Data
Time Series Databases
Published Data
Business Process Steps
Business Output and Input Artefacts
Derivation Processes
Sample
MRR Proof of Concept 2010/11Core case study was elements of statistical business process for Quarterly Business Indicators Survey (QBIS)
Simplified End-to-end Process For QBIS
Create common
frame
Select sample
Common Frame
Create survey frame
QEWS Frame
Forms design and approval
Forms
Load sample to
PIMSLabel files
Dispatch
SignificanceEditing
Time series analyses
NAB and FAS sign off
Time series to PPW
Collection and IFU
Data
Paradata(Collection
Information)
Clean Data
Time Series Databases
Published Data
Sample
Statistical Workflow Management
Metadata Registry and Repository
Ultimately want the metadata in the MRR to drive reuse in the above processes in conjunction with rules and processes stored in the SWM
Categories
Codes
Universe (population/ scope)
QBIS 2010 quarter 3
Question scheme (modules/parts)
Concepts
Variables
Study Unit (collection cycle)
Resource Packages
Categories
Codes
ANZSIC 06 (industry classification)
Questions
Standard Question Wording
Proof of Concept : Supported object types*
Interviewer instructions
SequencingData sets
Process metrics
Collection Instrument
Object type was supported in MRRObject type was simulated (not fully modelled)
* Relationships (eg between objects) are also an object type in their own right
Metadata Census• Initially conceived 2010.Q2 as project to understand all existing
Metadata Stores in ABS– Identify & analyse all Metadata Stores, – Classify types of metadata,– Identify what types of metadata are kept in which stores for which
applications
• Synthesise findings and provide empirical “bottom up” input to– MRR requirements and design– International “OCMIMF” collaboration project which is developing the
Generic Statistical Information Model (GSIM)• Maintain currency of information gathered
– reference when planning and managing transformation
What was Found (1)
• ABS has hundreds of systems/applications which:– Store data– Store metadata about data– Store metadata associated with data in other systems,– Run processes across data/metadata in systems– Duplicate data/metadata in other systems
• Production of comprehensive, integrated findings from the Metadata Census was not feasible within the given time and resource allocation
What was Found (2)• Inconsistent use of terminology within ABS
• Inconsistent modelling (at conceptual, logical and physical levels) of some types of metadata
• eg classifications• eg concepts -> QDT “Properties”, CPCF Mat “Properties”, CPCF Mat “Concepts”, DER/QDT
“Concepts”, etc.
• Inconsistent and insufficient support for versioning of metadata
• Loss and redefinition of metadata throughout statistical process– One example
• A variable in the ABS Input Data Warehouse has meaning from previously defined metadata around concepts, questions and qualifiers.
• During processing, in particular moving data to ABS Output Data Warehouse, links to the earlier metadata isn’t carried through, and so is lost.
• As these links are lost, the metadata is being redefined repeatedly throughout the statistical process.
Second phase• Focus in depth on metadata for a specific statistical business process
– QBIS used as example
• Collate information in order to create an object model describing objects that would be registered in the MRR during the Proof of Concept– GSIM has not yet reached a level of detail and common agreement which would provide
a “top down” path for describing these objects• Current target for GSIM to reach required level of agreed detail is December 2012
– The model to be used in the meantime is termed the ABS Transitional Model• Anticipate ABS Transitional Model will be fundamentally compatible with GSIM in most regards• Anticipate adjusting ABS model for alignment with GSIM where appropriate
– Model for Proof of Concept was small in scope (the six primary object types)• ABS Transitional Model expected to grow to 20-40 object types by June 2012
– Design of ABS Transitional Model (and GSIM) recognises SDMX and DDI as valuable technical standards supporting implementation & interoperability• Seek to support “crosswalks” to information models underpinning SDMX and DDI where these are
relevant and fit for purpose
Metadata Census : Conclusions
• Things can get very complicated very quickly• Comparing ‘To Be’ to what currently exists makes it
even more so.– Especially as what currently exists is inconsistent
• Practical analysis of statistical information (primarily metadata) flows throughout the statistical business process is invaluable input– GSBPM is a key reference point for processes– GSIM will be a key reference point for information
• Practical analyses in the meantime help build a better GSIM!
top related