The Inter-rater Reliability and Intra-rater Reliability of Bedside Ultrasounds of the

Femoral Muscle Thickness

Daren K. Heyland, MD, MSc, FRCPCProfessor of MedicineQueen’s University, Kingston General HospitalKingston, Ontario

A Randomized Trial of Supplemental Parenteral Nutrition in Under and Over Weight Critically Ill Patients:

The TOP UP Trial

Hypothesis: Increased energy and protein delivery to underweight and overweight critically illpatients (Body Mass Index [BMI] <25 or >35) will result in improved 60 day survival compared to usual care.

• Multicenter pilot study• Randomized trial of 160 critically ill adult patients from 8 tertiary care ICU’s in

Canada, United States, and Europe.• Patients randomized to one of 2 interventions: enteral nutrition (EN) alone or enteral

nutrition plus parenteral nutrition (supplemental PN group). • Patients stratified on the basis of admission BMI: <25 or >35, medical or surgical

admission diagnosis, by site and if EN was administered between ICU admission and randomization

Primary outcome: 60 day mortality.Secondary outcomes: 28 day mortality, hospital mortality, duration of stay (ICU and hospital), multiple organ dysfunction (SOFA and PODS), duration of mechanical ventilation, development of ICU acquired infections, functional status at hospital discharge, and 3 and 6 month survival and health-related quality of life.

Weekly Ultrasounds

• There is emerging evidence that muscle mass and muscle function predict morbidity in surviving patients and that muscle mass at ICU admission may predict length of hospital stay.

• We propose to evaluate the effect of differential amounts of protein and energy provided to study patients on muscle mass and function.

• We can postulate that the beneficial effect of enhanced energy and protein provision is mediated by the preservation (or attenuated deterioration) of muscle mass and increased function in these better fed patients, which would ultimately result in positive outcomes.

• We will evaluate muscle mass in all study patients using non-invasive bedside ultrasound of the femoral muscle

Testing Ultrasound Reliability before TOP-UP

Before we performed weekly ultrasounds on the study population, we conducted a trial of the ultrasound protocol to allow us to:

•standardize the training of all Study Investigators performing the ultrasound assessment•test the feasibility of the procedures •determine ‘normal’ values to which we can compare our measures in the study population•determine intra-rater (trainer) and the inter-rater (trainee) reliability.

The ultrasound reliability trial involved:• the 7 participating TOP UP sites• healthy volunteers• a standardized protocol:

o ultrasound trainers were to perform ultrasounds twice on each patiento a trainee repeated the measurement on the same patient

The objective was to evaluate the inter-rater reliability and intra-rater reliability of bedside ultrasounds of the femoral muscle measuring muscle thickness in healthy volunteers before using this tool in detailing overall muscle mass in ICU patients.

Reliability Protocol1) Trainer to complete a 2/3rd and midpoint ultrasound on each leg2) Trainer to repeat a 2/3rd and midpoint ultrasound on each leg (intra-rater reliability test)3) Trainee to complete a 2/3rd and midpoint ultrasound on each leg (inter-rater reliability test)

The overall muscle thickness was calculated as the average of the readings measured at the border between the lower third and upper two-thirds between Anterior Superior Iliac Spine (ASIS) and upper pole of the patella as well as the reading at the midpoint between the ASIS and the upper pole of the patella averaged over the right and left legs.

Volunteers’ Demographics

Variable N Mean Std Dev Minimum Maximum

Age

Height

Weight

BMI

64

78

78

78

30.6

171.4

71.2

24.1

8.4

11.5

16.4

4.4

21.0

135.0

46.0

16.9

55.0

196.0

136.4

40.7

Intra-rater Reliability ResultsSite Subjects Between Subject Variance Within Subject Variance ICC

Erasme University Hospital, Brussels, Belgium

10 0.2307 0.01380 0.94

Grey Nuns Hospital, Edmonton, Alberta, Canada

0 NA NA NA

Royal Alexandra Hospital, Edmonton, Alberta, Canada

10 0.2425 0.001018 >0.99

Nouvel Hôpital Civil, Strasbourg, France

4 0.2567 0.000199 >0.99

University of Alberta HospitalEdmonton, Alberta, Canada

12 0.05866 0.003180 0.95

University of Colorado, Aurora, Colorado, USA

5 0.2869 0.000613 >0.99

University of Texas Health Science Center, Houston, Texas, USA

5 0.1774 0.003837 0.98

Pooled 46 0.2648 0.004554 0.98

ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)

Intra-rater Reliability Results (Continued)

The mean difference between trainer measurements (95% CI) = 0.037 (0.010 to 0.063) p=0.0077

Mean difference between trainer measurement 1 and trainer measurement 2

Trainer measurement 1 Trainer measurement 2

• The paired t-test was used examine the average difference between the first and second trainer measurements.

• The paired profile plot visualizes the difference in the paired measures.

Inter-rater Reliability ResultsSite Subjects Between Subject Variance Within Subject Variance ICC

Erasme University Hospital, Brussels, Belgium

10 0.2194 0.02900 0.88

Grey Nuns Hospital, Edmonton, Alberta, Canada

13 0.3217 0.000769 >0.99

Royal Alexandra Hospital, Edmonton, Alberta, Canada

10 0.2305 0.02072 0.92

Nouvel Hôpital Civil, Strasbourg, France

18 0.2623 0.009972 0.96

University of Alberta HospitalEdmonton, Alberta, Canada

12 0.03587 0.01360 0.73

University of Colorado, Aurora, Colorado, USA

5 0.1714 0.02746 0.86

University of Texas Health Science Center, Houston, Texas, USA

5 0.1704 0.03334 0.84

Pooled 73 0.2584 0.01580 0.94

ICC: Intra-class correlation coefficient, ICC = between subject variance / (between subject variance +within subject variance)

Inter-rater Reliability Results (Continued)

There was a small but statistically significant difference between the trainer and trainee results: Mean (95% CI) = -0.061 cm (-0.100 to -0.022), p= 0.0028

Mean difference between trainer measurement and trainee measurement

Trainer Trainee

• The paired t-test was used examine the average difference between the first trainer and the trainee measurement.

• Paired profile plots are provided to visualize the difference in the paired measures.

Conclusion

• There is excellent inter and intra-rater reliability for ultrasound measurements of the femoral muscle to determine overall muscle thickness in healthy volunteers.

• A sample of ‘normal’ values is now available to compare measures from a study population.

• Further evaluation of this technique must be validated in critically ill patients.

• Efforts to link the ultrasound measurements to ICU outcomes should be undertaken.

Questions