professional standards authority conference – 28 march 2014 regulatory design: the role of...
TRANSCRIPT
Professional Standards Authority conference – 28 March 2014
Regulatory design: the role of programme theory in evaluating healthcare regulation
Kieran Walshe and Denham Phipps
University of Manchester, UK
• Programme theory in evaluation research
• Applying programme theory to understanding the Care Quality Commission’s regulatory model
• Conclusions and reflections
Programme theory
• “A plausible and sensible model of how a programme is supposed to work” which “provides the hypothesized links between program features and planned outcomes to be tested” (Bickman 1996)
• Derived from existing knowledge/research and from programme actors – policymakers, leaders etc
• Makes programme design choices explicit/visible and open to contestation/adjudication
• In “what works” – helps distinguish between theory and implementation, to understand mechanism and context, and to generalise
CQC study objectives and methods
• Develop a “logic model” to map programme theories underlying the regulatory model
• Use it to develop a framework for identifying and prioritising topics and questions for evaluation
• Document review – mainly CQC publications/reports
• Interviews with circa 20 CQC staff in London and Preston
• Review of the research literature and comparison of practice with four other regulators (two in health elsewhere, two in other settings in UK)
Regulatory model: components
• Registration
• Compliance
– Standard setting
– Information gathering and risk assessment
– Inspection and reporting
• Enforcement
• Information provision
Registration
• Three main models of registration:
– Threshold – tool for driving improvement, deterring poor providers from registering, leverage, assurance
– Relationship – get to know provider, build knowledge base, establish relations, triage or risk assess
– Administrative process – collect basic data, register and then pass on to others to regulate
• Consequences – foregone opportunities, deferred regulatory load
Compliance: standard setting
• Four main models of standard setting:
– Mechanism to frame/set stakeholder expectations
– Mechanism for improvement through self-enforced compliance
– Mechanism for compliance through measurement and enforcement
– Mechanism for differentiating performance of providers
• Consequences: generic standards, minimal/median /maximal level, content and focus, understanding and implementation
Compliance: information gathering and risk assessment
• Three main models of proportionate regulation:
– Using information to determine when to use regulatory interventions
– Using information to focus/direct the content of regulatory interventions
– Letting providers know about likely regulatory intervention
• Consequences: timely, meaningful, predictive information and intelligence, capacity to use/interpret it
Compliance: inspection
• Four main models of inspection:
– Driver for improvement in advance of inspection
– Measure of compliance to support enforcement
– Measure of performance to support improvement
– Driver for other providers to improve
• Consequences: direct observation and proxy measures, reliability and validity of judgements, inspection workforce competence
Enforcement
• Four main models of enforcement:
– Informal enforcement or prospect of enforcement
– Incentive to drive compliance
– Symbolic action to drive compliance
– Driver for other organisations to achieve compliance
• Consequences: resource to enforce, scale of penalty, effects of frequency, spillover effects, capacity to change
Information provision
• Three main models of information provision:
– Information to be used by other stakeholders in their decision making
– Information to be used by providers in compliance and improvement
– Information as a mechanism for public accountability
• Consequences: limited volume, utility and timeliness of data, usability by stakeholders
Summary of current regulatory model
• Registration as an administrative process
• Compliance
– Standard setting to measure compliance and support enforcement
– Information gathering and risk assessment to inform or direct regulatory intervention
– Inspection and reporting to measure compliance
• Enforcement as an incentive for compliance
• Information provision to support decision making and accountability
Current regulatory model: key issues
• Consistent with a mission as a “safety-net” regulator focused mainly/only on compliance at the bottom of the performance distribution
• Some problems even so with the model
– Regulatory standards - measurability and meaning
– Attempts proportionate regulation without predictive information and capacity to use it
– Inspection process – reach and workforce competence
– Information provision – little useful information to share
Conclusions and reflections
• For CQC – a contribution towards their rethinking of their regulatory strategy/philosophy and model
• Programme theory evaluation – most useful (and most difficult) when theory(ies) in use unclear, ambiguous, complex/complicated
• Place for programme theory in regulatory design – not just post hoc evaluation
• Wider question of how best to integrate evaluation into regulatory innovation/policy development/practice