![Page 1: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/1.jpg)
Professional Standards Authority conference – 28 March 2014
Regulatory design: the role of programme theory in evaluating healthcare regulation
Kieran Walshe and Denham Phipps
University of Manchester, UK
![Page 2: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/2.jpg)
• Programme theory in evaluation research
• Applying programme theory to understanding the Care Quality Commission’s regulatory model
• Conclusions and reflections
![Page 3: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/3.jpg)
Programme theory
• “A plausible and sensible model of how a programme is supposed to work” which “provides the hypothesized links between program features and planned outcomes to be tested” (Bickman 1996)
• Derived from existing knowledge/research and from programme actors – policymakers, leaders etc
• Makes programme design choices explicit/visible and open to contestation/adjudication
• In “what works” – helps distinguish between theory and implementation, to understand mechanism and context, and to generalise
![Page 4: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/4.jpg)
![Page 5: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/5.jpg)
CQC study objectives and methods
• Develop a “logic model” to map programme theories underlying the regulatory model
• Use it to develop a framework for identifying and prioritising topics and questions for evaluation
• Document review – mainly CQC publications/reports
• Interviews with circa 20 CQC staff in London and Preston
• Review of the research literature and comparison of practice with four other regulators (two in health elsewhere, two in other settings in UK)
![Page 6: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/6.jpg)
Regulatory model: components
• Registration
• Compliance
– Standard setting
– Information gathering and risk assessment
– Inspection and reporting
• Enforcement
• Information provision
![Page 7: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/7.jpg)
Registration
• Three main models of registration:
– Threshold – tool for driving improvement, deterring poor providers from registering, leverage, assurance
– Relationship – get to know provider, build knowledge base, establish relations, triage or risk assess
– Administrative process – collect basic data, register and then pass on to others to regulate
• Consequences – foregone opportunities, deferred regulatory load
![Page 8: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/8.jpg)
Compliance: standard setting
• Four main models of standard setting:
– Mechanism to frame/set stakeholder expectations
– Mechanism for improvement through self-enforced compliance
– Mechanism for compliance through measurement and enforcement
– Mechanism for differentiating performance of providers
• Consequences: generic standards, minimal/median /maximal level, content and focus, understanding and implementation
![Page 9: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/9.jpg)
Compliance: information gathering and risk assessment
• Three main models of proportionate regulation:
– Using information to determine when to use regulatory interventions
– Using information to focus/direct the content of regulatory interventions
– Letting providers know about likely regulatory intervention
• Consequences: timely, meaningful, predictive information and intelligence, capacity to use/interpret it
![Page 10: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/10.jpg)
Compliance: inspection
• Four main models of inspection:
– Driver for improvement in advance of inspection
– Measure of compliance to support enforcement
– Measure of performance to support improvement
– Driver for other providers to improve
• Consequences: direct observation and proxy measures, reliability and validity of judgements, inspection workforce competence
![Page 11: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/11.jpg)
Enforcement
• Four main models of enforcement:
– Informal enforcement or prospect of enforcement
– Incentive to drive compliance
– Symbolic action to drive compliance
– Driver for other organisations to achieve compliance
• Consequences: resource to enforce, scale of penalty, effects of frequency, spillover effects, capacity to change
![Page 12: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/12.jpg)
Information provision
• Three main models of information provision:
– Information to be used by other stakeholders in their decision making
– Information to be used by providers in compliance and improvement
– Information as a mechanism for public accountability
• Consequences: limited volume, utility and timeliness of data, usability by stakeholders
![Page 13: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/13.jpg)
![Page 14: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/14.jpg)
Summary of current regulatory model
• Registration as an administrative process
• Compliance
– Standard setting to measure compliance and support enforcement
– Information gathering and risk assessment to inform or direct regulatory intervention
– Inspection and reporting to measure compliance
• Enforcement as an incentive for compliance
• Information provision to support decision making and accountability
![Page 15: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/15.jpg)
Current regulatory model: key issues
• Consistent with a mission as a “safety-net” regulator focused mainly/only on compliance at the bottom of the performance distribution
• Some problems even so with the model
– Regulatory standards - measurability and meaning
– Attempts proportionate regulation without predictive information and capacity to use it
– Inspection process – reach and workforce competence
– Information provision – little useful information to share
![Page 16: Professional Standards Authority conference – 28 March 2014 Regulatory design: the role of programme theory in evaluating healthcare regulation Kieran](https://reader030.vdocuments.net/reader030/viewer/2022032804/56649e4f5503460f94b45bd6/html5/thumbnails/16.jpg)
Conclusions and reflections
• For CQC – a contribution towards their rethinking of their regulatory strategy/philosophy and model
• Programme theory evaluation – most useful (and most difficult) when theory(ies) in use unclear, ambiguous, complex/complicated
• Place for programme theory in regulatory design – not just post hoc evaluation
• Wider question of how best to integrate evaluation into regulatory innovation/policy development/practice