Methods for assessing fidelity and quality of delivery of smoking cessation behavioural support

Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie

Centre for Outcomes Research and Effectiveness, University College London

Health Behaviour Research Centre, University College London3National Centre for Smoking Cessation and Training

• Why is there substantial variability in quit outcomes across services?

• Is it the content of support delivered?

• Many services have structured behavioural support treatment manuals– Specify recommended content and format of sessions

– Help guide and standardise practitioner behaviour

• Practitioners from same NHS SSSs, operating under same manual have widely varying success rates (Brose et al. 2012)

Actually following the treatment manual?

• Two aspects of delivery need to be examined:

1. What manual specified content is actually delivered?

- Fidelity of delivery

- extent to which interventions are delivered as intended (i.e. recommended practice actual practice)

2. How is this content actually delivered?

- Quality of delivery

- Appropriate + comprehensive delivery of intervention content

*** Does this affect outcomes?

AimsTwo linked studies

Study 1:

1.Develop a reliable method for assessing fidelity of delivery of behavioural support in practice.

Study 2:

1.Develop a reliable method for assessing quality of delivery of key technique, setting a quit date (i.e. goal setting)

2.Examine the association between quality of goal setting and likelihood of clients making a quit attempt.

Study 1

assessing fidelity of delivery

‘WHAT is delivered’

METHODS: Sample and materials

-National telephone behavioural support service (i.e. quit line)

-Six practitioners

- All trained to NCSCT standard

- All aware of treatment manual

-Service treatment manual (i.e. recommended practice)

-Outlines recommended content/format of pre-quit, quit day, and post quit sessions

-Transcripts of 64 audio-recorded, consecutively delivered behavioural support sessions (i.e. actual practice).

- Pre-quit sessions (n=27)

- Quit-day sessions (n=16)

- Post-quit sessions (n=21).

Procedure

• Two coders identified + categorised BCTs in treatment manuals and session transcripts using taxonomy of 43 smoking cessation BCTs.

• Inter-rater reliability assessed using % agreement

• Examined what % of manual-specified BCTs were present in transcripts of practice sessions (recommended vs. actual practice)

RESULTS

Reliability: average percentage agreement was high (87.1%; range: 78.4% - 95.6%)

Fidelity: Note: <50% = ‘low fidelity;’ 51-79%= ‘moderate;’ 80-100% =‘high’

SESSION TYPE NUMBER OF BCTS IN

MANUAL

AVERAGE NUMBER (%)

MANUAL BCTs DELIVERED

SD/RANGE

PRE-QUIT(n=27)

22 10 (46%) SD 16.9Range: 14-82%

QUIT-DAY(n=16)

25 9 (35%) SD 14.8Range: 8-60%

POST-QUIT(n=21)

28 12 (42%) SD 16.3Range: 8-82%

ACROSS SESSIONS

25 (average) 10.3 (41.2%) -

Study 2

‘HOW’

assessing quality of delivery of

‘setting a quit date’ (i.e. goal setting)

association between quality of goal setting

and quit attempts.

METHODS

•Sample & Materials:

-Same quit line service and practitioners

-New set of transcripts: 85 pre-quit behavioural support sessions

- 85 clients

- Pre-quit sessions as this when quit date likely to be set

-Outcomes: whether client made a quit attempt as planned by following session (yes/no)

PROCEDURE

1) Development of quality rating scale:

-Quality= appropriate + comprehensive delivery of technique

-Two researchers independently examined guidance documents to identify components of high quality ‘setting a quit date’

- NCSCT training standard, standard treatment programme, training module, quit line service treatment manual

-Extracted components featured across all guidance documents description of ‘gold standard’ goal setting

- Basis for content of rating scale

0 Absence of goal setting

+1 Prompts goal setting (i.e. encourage client set quit date)

+1 Agrees clear quit date (dd/mm/yy)

+1 Agreed quit date within appropriate time frame (1-2 weeks)

+1 Quit date allows appropriate time to obtain medication

+1 Emphasise goal is to not smoke a single cigarette/puff

+1 Provide relevant normative information and examples

-1 Inappropriate goal setting (i.e. unclear date, incorrect time frame)

-1 Undermines client commitment to quit date (i.e. imply flexibility)

Points allocated for appropriate delivery

Points deducted for inappropriate delivery

Score range: -2 to 6

Higher score implies more comprehensive + appropriate delivery

• Scored sections of transcripts in which ‘setting a quit date’ occurred

using scale

• Inter-rater reliability assessed using Cohen’s Kappa (weighted)

• Examined association between quality scores and whether client made

a quit attempt as planned

– Multilevel logistic regression

• both using total quality score as predictor

• And individual scale component as predictor

RESULTS

Inter-rater reliability: Weighted kappa = 0.68 ‘substantial agreement’ (Landis & Koch,1977)

Average quality score:

- overall: 1.6 (SD 1.2) low quality (scale range: -2 to 6)

- clients made attempt (n=18; 21.2%): 2.2 (SD.70)

- clients did not make attempt (n=67; 78.8%): 1.4 (SD 1.27)

Association with quit attempts:

Higher quality goal setting in pre-quit sessions significantly increased likelihood of clients making quit attempt as planned

[p<.001 OR 2.60 (95% CI: 1.54 to 4.40)]

Only 1 scale component independently associated significantly with quit attempts:

-Setting a clear quit date (dd/mm/yy): p<.001, OR 36.9, 95% CI: 4.52 to 302.11)

-Opposite of this: Setting inappropriate quit date (i.e. unclear date): p<.001, OR .03, 95% CI: .24 to 2.88)

DISCUSSION

•It is possible to reliably assess fidelity and quality of delivery of smoking cessation behavioural support in practice

•Preliminary analysis of national telephone quit line service:– Less than half of what is specified in treatment manuals is not being

routinely delivered in practice

– Quality of ‘setting a quit date’ is low, but higher quality significantly increases likelihood of initiating quit attempts

– Particularly if a clear date is set.

– Deconstruction of technique into individual components: identify specific aspect driving outcome (i.e. ‘active ingredient’)

• Limitations:– Limited generalisability- single service examined; other modalities

– Service manual extremely comprehensive (22-29 BCTs per session!) feasible? Relevant?

• Implications & Future work:– Identification of training needs + aspects of service delivery to be targeted

for improvement

– Expand quality scale to other behaviour change techniques

– Need to examine how practitioners actually use manuals in practice

– And whether manuals are fit for purpose

• Conclusions: Stop smoking services should establish routine procedures for monitoring fidelity and quality of behavioural support delivered.

Thank you for listening.

Fabi.lorencatto@ncsct.co.uk

The team:

Fabiana Lorencatto, Robert West, Carla Bruguera, & Susan Michie

Acknowledgements:

Collaborating services

Funding:

National Centre for Smoking Cessation and Training; Department of Health