copdtrainer: a smartphone-based motion rehabilitation training system...

COPDTrainer: A Smartphone-based Motion RehabilitationTraining System with Real-Time Acoustic Feedback

Gabriele Spina†, Guannan Huang†, Anouk Vaes††, Martijn Spruit††, Oliver Amft††ACTLab, Signal Processing Systems, TU Eindhoven, The Netherlands

††CIRO+, Hornerheide 1, 6085 NM Horn, The Netherlands{g.spina, amft}@tue.nl

ABSTRACTPatient motion training requires adaptive, personalized exer-cise models and systems that are easy to handle. In this paper,we evaluate a training system based on a smartphone that in-tegrates in clinical routines and serves as a tool for therapistand patient. Only the smartphone’s build-in inertial sensorswere used to monitor exercise execution and providing acous-tic feedback on exercise performance and exercise errors. Weused a sinusoidal motion model to exploit the typical repet-itive structure of motion exercises. A Teach-mode was usedto personalize the system by training under the guidance ofa therapist and deriving exercise model parameters. Subse-quently, in a Train-mode, the system provides exercise feed-back. We validate our approach in a validation with healthyvolunteers and in an intervention study with COPD patients.System performance, trainee performance, and feedback ef-ficacy were analysed. We further compare the therapist andtraining system performances and demonstrate that our ap-proach is viable.

Author KeywordsRehabilitation training; wearable computing; patientmonitoring; exercise quality.

ACM Classification KeywordsH.5.2 Information interfaces and presentation (e.g., HCI):Miscellaneous

INTRODUCTIONCardiopulmonary fitness is a well-known condition for ourlong-term health and wellness [11]. In particular, patientssuffering from the widespread cardiovascular diseases (CVD)and chronic pulmonary obstructive disease (COPD) can ben-efit from physical training. Nevertheless, CVD and COPDpatients have special requirements regarding fitness training,related to their physical ability, determining type and inten-sity of exercises, and practical systems to support them. Forexample, generally healthy people could regularly jog and

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected]’13, September 8–12, 2013, Zurich, Switzerland.Copyright © 2013 ACM 978-1-4503-1770-2/13/09...$15.00.http://dx.doi.org/10.1145/2493432.2493454

run, and even over-train without immediate health conse-quences. In chronic patients, both over-training and under-training could lead to quick and detrimental worsening ofthe health condition, resulting in exacerbations and hospi-talisation, or death [2]. In addition, chronic patients oftenfear to exercise wrongly [16], if not under therapist supervi-sion. While therapists can recommend exercises for the pa-tient’s independent training, both therapist and patient haveno means to assess the exercise performance during indepen-dent training. Ubiquitous and on-body systems could enablepatients to perform additional physical training on their own,in addition to the supervised training with a therapist.

Fitness and sports studies revealed a series of challenges formonitoring and coaching, when using ubiquitous and on-body sensing systems. In a recent survey, Kranz et al. [10]identified usability improvement, instruction quality, andlong-term motivation as core design aspects of fitness trainingsystems. Usability improvement refers to reduced labour inmaintaining log-books or other manual records during train-ing. Instruction quality refers to the guidance a trainee is pro-vided with, to adequately execute an exercise. We believe thatsystem feedback could prevent injuries, or worsening condi-tions in patients. During rehabilitation exercise training, forexample, different errors can co-occur and should be identi-fied accordingly. Moreover, it is essential that an error esti-mation algorithm can handle different exercises with minimaladjustments to support training variety.

Adequate feedback depends on individual skills and fitnesslevel, which is particularly varying in chronic patients cor-responding to their rehabilitation progress. Thus, error es-timation algorithms should be adjustable to a patient’s indi-vidual capability level. Until now, many error monitoringapproaches focused on individual exercises or specific multi-sensor training devices that helped to stratify error conditions.However, attaching multiple devices is often too difficult forpatients to train individually. While on-body sensors couldbe comfortable during exercise training, their cost and han-dling is challenging for patients. The widespread adoption ofsmartphones provides a platform for healthcare applicationsthat is directly available to patients.

In this work, we introduce a smartphone-based motion reha-bilitation training system, intended for individual exercisingof chronic patients. The system processes motion sensor dataonline on the phone and provides real-time acoustic feedbackregarding the exercise performance and quality. We inves-

Session: Health II UbiComp’13, September 8–12, 2013, Zurich, Switzerland

597

tigate whether exercise model parameters describing typicalrehabilitation exercises can be derived from a smartphone’sinternal sensors to reliably support patient training and pro-vide real-time feedback. In particular, this paper provides thefollowing contributions:

1. We introduce a training approach, where the trainee hasto attach a holster carrying the smartphone only. Afteran initial rehabilitation exercising session with the thera-pist (Teach-mode), individual training (Train-mode) can beperformed.

2. We validated our system with healthy individuals perform-ing six limb movement exercises as they are commonlyprescribed for COPD patients. We model errors using mo-tion parameters and classify nine performance classes (in-cluding “correct” and eight error conditions). Subse-quently, we evaluate our system in an intervention studywith seven COPD patients. We assess system recognitionperformance regarding exercise and performance classes.

3. In further analyses of the COPD patient study data, we de-termine patients’ training performance, error trends, andfeedback efficacy. Furthermore, we compare the therapisttraining error assessment against the sensor-based mea-surements. We confirm that the smartphone-based train-ing system can achieve similar performance than when as-sessed by a therapist.

Our approach integrates into this clinical rehabilitation rou-tine by incorporating a Teach-mode, where training is per-formed under therapist supervision. During Teach-mode, oursystem derives motion parameters that are subsequently usedduring the Train-mode to estimate training performance andquality. Hence, the system could serve as a novel tool fortherapists and their chronic patients to improve training op-tions, both in the rehab center and at home. The smartphoneserves as single training device, thus reducing starting barri-ers for rehabilitation training, including cost, availability, andhandling of devices.

RELATED WORKA few works assess the quality of exercise activities beingperformed, especially for clinical applications. Analysing ex-ercises performance is usually done by means of cameras [9],depth cameras [4] or optical motion capture systems in com-bination with passive markers (Vicon, OptiTrack). In generalvision-based systems allow users to easily extract a humanskeleton automatically, but require constrained environmentsto install and calibrate cameras. Various ambient and on-bodydevice developments identified opportunities for continuoustraining and coaching in fitness and sports outside the lab,such as the Ubifit Garden [5], MOPET system [1], and Triple-Beat [6].

Smartphones are being widely deployed and provide severalintegrated sensors to analyze data in real-time and providetraining performance feedback. Thus, smartphones could beused as stand-alone systems to minimize costs hurdles in ap-plications. For example smartphones were used as a mobileexercise skill assessment tool (GymSkill) to support personal

health and fitness [12]. GymSkill monitors exercise qualityperformed on a balance board and provides feedback accord-ing to various parameter including regularity of movements.Muehlbauer et al. [14] exploited arm worn smartphones torecognize and count upper body resistance training exercisesfrom a acceleration sensors. In [15] the authors introduced analgorithm based on dynamic time warping, which uses accel-eration data to evaluate the number and duration of correctlyrecognized repetitions. The application provided real-timefeedback on the duration of repetitions and was studied inhealthy individuals. Further parameters, including the rangeof motion and efficacy of the feedback were not considered.

Wearable distributed sensors and other dedicated deviceswere used in several exercise and sports studies. Strohrmannet al. [17] assessed performance level, training assistance andfatigue monitoring of runners. Tseng et al. [18] used ac-celerometers and compass sensors in a rehabilitation gameto increase motivation. The system provided scores on move-ment quality. A fixed rule-set was used to recognise activ-ities. Chang et al. in [3] proposed a system to recognizemotion patterns and count repetitions of a limited set of free-weight exercises using acceleration data from a glove and achest belt. The system did not provide feedback on execu-tion quality since start and end of a repetition were not de-tected. Although their counting algorithm showed good re-sults, it needed re-training to obtain accurate results for dif-ferent exercise speeds. Moreover, training data was requiredto obtain pattern models off-line. Velloso et al. [19] used fiveXsens sensors and a Kinect camera to derive pattern mod-els during an exercise demonstration performed by an expert.The system then detected mistakes and guided users on im-proving their performance. Three weight training exerciseswere studied, however system recognition performance to de-tect training errors were not analysed. The deployment of theKinect camera constrained the field of view to a 2 m distancefrom the system. Limbs should not be pointed directly at thecamera or be occluded by the body, which limits the exercisesthat could be monitored.

The related works discussed above focused mainly on gymmonitoring applications for healthy subjects. Often theseapproaches relied on multi-sensor information and patternrecognition methods, requiring individual learning of motionpattern models. Our work aims at describing different train-ing exercises with the same exercise quality parameters anda sinusoidal model. In our approach, a smartphone servesas single measurement, estimation, and feedback device forassessing patient exercise performances. We evaluate ourmethod’s recognition performance for classifying executionerrors, which is necessary to deploy the system in practiceand especially in a clinical application. To the best of ourknowledge, no existing commercial or academic work ex-ploits smartphones to assess training quality in chronic pa-tients. Likewise, recognition performance for classifying ex-ecution errors was not evaluated in similar systems for a clin-ical application.


598

SMARTPHONE-BASED TRAINING APPROACHThe ability to perform particular motion exercises differs be-tween trainees, due to individual motion constraints. Chronicpatients often suffer from limiting pathologies and muscleweakness, thus may not be able to perform exercises at thesame speed or range of motion of another trainee. To use thesmartphone as an exercise monitoring and feedback device inthis user group, it was attached in a holster to a body part orlimb involved in the exercise. The phone’s integrated iner-tial sensors could thus capture the motion performance. Afterperforming one exercise, the holster could be moved to a po-sition designated for the next exercise. Illustrations on thephone’s screen guided the trainee during attachments. Here,we describe our training approach that includes Teach-modeand Train-mode operation as illustrated in Fig. 1 and our mo-tion exercise modeling approach.

Figure 1. COPDTrainer training approach. Our smartphone app ismeant to enable individual training options of a patient, besides thetraining with therapists. (1): Patient/therapist selects Teach-mode orTrain-mode. (2): Select an exercise, here Arm extension. (3): Begin ex-ercising after pressing start. (4): After exercising different summaryscreens are shown, depending on the operation mode.

Teach-mode operationThe Teach-mode allows therapists to personalise the systemfor a trainee under direct supervision, e.g. during the regularphysiotherapy practicing times. Any selectable exercise canbe performed and the trainee learns from the therapist howto attach the phone and perform a particular exercise. Illus-trations are shown on the screen after selecting an exercise toremind the patient about the exercise execution independentlyof the operation mode. In Teach-mode, the therapist initiallyguides the patient during the first trials to perform the exer-cise accurately. Subsequently, the Teach-mode recording isstarted by pressing a large button on the phone’s screen. Apreset number of exercise repetitions (ten in the current im-plementation) will then be acquired from the phone’s inertialsensors. From the recorded data, all necessary exercise modelparameters, i.e. mean and variance of the duration and of therange of motion of the limb during the ten repetitions) areestimated and stored for further use in the Train-mode. The

derived parameters are shown on the smartphone, such thatthe therapist and trainee can review them. If the therapistjudges that the trainee did not perform the exercise with suf-ficient quality, the session could be repeated. Moreover, thesystem checks consistency of the exercise repetitions and canthus reject a Teach-mode session that shows extensive execu-tion variability. These choices consider the clinical routines,where therapists have only 30 to 45 min per patient for assess-ment, therapy, and exercise training. Thus, complex interac-tions with the device were avoided.

Train-mode operationDuring Train-mode, the derived exercise models are arrangedin a todo-list for the trainee to complete. This mode is in-tended to be used by the trainee to exercise without therapistsupervision, i.e. at the rehab center or at home. After select-ing an exercise to be performed and starting the Train-mode,inertial motion data is recorded from the phone’s sensors andprocessed in real-time to count the exercise repetitions anddetect errors. While training, the smartphone system will pro-vide acoustic feedback on the counted repetitions and notifywhen errors occur. E.g., if the trainee had practised an exer-cise with the therapist before, but starts to perform repetitionsfaster than during the Teach-mode, the system will providethe feedback “move slower”. This feedback could preventinjuries from repetitive erroneous movements. Finally, afterthat the configured number of repetitions were detected, thesystem will ask the trainee to stop and displays a summary ofthe execution performance.

Motion exercise modelingBased on the observation that many fitness exercises have arepetitive structure, from training with free weights to cardiofitness motion, we consider a sinusoidal motion model. Foreach exercise, a representative motion feature could be cho-sen that represents a sinusoidal pattern. The feature can bebased on a single raw sensor axis or fused from several sen-sors, such as orientation estimates. For example, in a lateralarm abduction exercise, where the phone is attached to thewrist, the anterior-posterior orientation angle could be usedas motion feature. Figure 2 shows an example waveform forseveral repetitions of an exercise.

Advised by three therapists and after consulting COPD guide-lines, we derived speed of motion (corresponding to the pe-riod frequency) and range of motion (corresponding to thefeature amplitude) from the sinusoidal pattern of each exer-cise repetition. In Kinesiology speed and range of motion,together with their relative tolerances and the number of rep-etitions, are considered standard measures for exercise mon-itoring [7]. Estimating movement speed during exercises isuseful to educate patients in breathing techniques (i.e. by ex-ercising the patient can learn how to breath with a correcttiming). Based on these exercise quality parameters, we de-rived performance classes, such that the classes are applicableto various exercises that are performed by repetitive move-ments. During the Teach-mode, exercise repetitions are usedto represent repetition range and duration parameters usingtwo normal distributions. In the Train-mode, these model pa-rameters are used to identify nine performance classes. The


599

classification approach is further described in the followingsection.

ALGORITHMS IMPLEMENTATIONHere, we describe the algorithm implementation for segment-ing and evaluating exercise performances in a stand-aloneAndroid 1 smartphone application.

Teach-mode implementationFor any exercise to be monitored, the smartphone body at-tachment and the motion feature representing the sinusoidalpattern need to be chosen. Subsequently, during the Teach-mode, the application records inertial sensor data from ac-celerometers, magnetometers, and the orientation API, andstores the data for the later analysis. Since the Teach-modeis performed under therapist supervision, no real-time feed-back will be provided. Once the trainee completes an exercisesession with a preset number of repetitions, the applicationloads the stored data and extracts the exercise model param-eters. The following processing steps were applied to derivethe model parameters.

Filtering. The selected motion feature was filtered using amoving average to remove tremor-induced noise and sensornoise. The window size was set proportional to the amountof data acquired (i.e. WindowSize = DataSize

ScalingFactor , withScalingFactor determined empirically and set to 80). We ob-served that this approach provided consistent results acrossthe different exercises. Since the number of repetitions is pre-configured, we assume that the total data amount recordedis proportional to the movement speed during the exerciseexecution: the faster a trainee performs the exercise, thelower muscular tremor would be, and thus, data averagingcan be reduced. Bounds were applied to the averaging win-dow size, to prevent ineffective averaging for very fast/slowrepetitions (15 ≤WindowSize ≤ 31 sa.)

Period estimation. By estimating the position of positive andnegative peaks in the filtered motion feature, exercise repe-titions were counted. For the Arm abduction exercise, Oywill be maximal when the arm is raised to shoulder height. Itreaches its minimum value when the arm returns to the neutralposition (arm aligned to the trunk). We used an adaptive hill-climbing algorithm to detect positive and negative peaks (de-noted in Fig. 2 by ppi and npi, respectively), given a peakthreshold θ.

The hill-climbing algorithm is a popular first choice amongoptimisation algorithms. While there are many alternatives,such as simulated annealing or tabu search, hill-climbing canachieve sufficient or better results if runtime is constrained,such as in the real-time system targeted here. Figure 2 illus-trates an example motion waveform with the exercise param-eters. In particular a maximum (minimum) was selected, ifboth sides (uphill hu and downhill hd) are greater than thepeak threshold θ. In this work, we estimate θ during theTeach-mode. Starting from an initial setting θ = θin, θ wasadjusted in steps of 1.2 until the preset repetition number wasobtained. We set the number of exercise repetitions to ten.1Android OS, Google. http://www.android.com.

The fitted θ = θopt was subsequently applied during Train-mode operation.

Figure 2. Example waveform of the Pitch motion feature (Oy) selectedfor the Arm abduction exercise. The hill-climbing parameter θ is fittedduring system runtime (Teach-mode) for each exercise. The horizontaldashed line indicates the estimated duration d

2, the vertical line indicates

the estimated range of motion r. Online feedback is provided in theindicated region.

Peak correction. While the hill-climbing algorithm has lowruntime complexity, the detection of local maxima and min-ima remains susceptible to detecting additional peaks (inser-tion), e.g. during vibrations, or to missing peaks (deletion)if the signal amplitude decreases. In situations where thereis one insertion or deletion error in sequence, the alternatingorder of positive and negative peaks is interrupted. If two con-secutive positive or negative peaks were derived a peak cor-rection algorithm was applied. This peak correction worksby first removing redundant peaks and then inserting miss-ing peaks that were missed during the first iteration of thehill-climbing algorithm. If two consecutive positive (nega-tive) peaks were detected, the algorithm compare them andremoves the one having a smaller (greater) amplitude, thusrestoring the alternating of maxima and minima.

Subsequently, time intervals ∆t between two consecutivepeaks were used to determine if there could be peaks miss-ing. If an interval was larger thanm∆t+σ∆t, wherem∆t andσ∆t are mean and standard deviation of the intervals, it wasassumed that peaks might be missing. Missing peaks weresearched by applying hill-climbing within the section delim-ited by the consecutive peaks. A smaller threshold θ (reducedby 20% of its initial value) was then used. The hill-climbingwas iterated while adjusting θ until a peak was found, or alower limit of θ < θopt

2 was reached.

Exercise model parameter estimation. The following fiveparameters were derived: number of repetitions (n), meanand standard deviation of repetition duration d (md, σd), andmean and standard deviation of the range of motion r (mr,σr). The repetition duration was derived from the time inter-val ∆t between two adjacent minima. The range of motionwas derived from the magnitude difference between adjacentnegative and positive peaks. The number of repetitions couldbe obtained by counting the number of maxima. Since capa-bilities between trainees vary a small initial peak threshold θinwas fixed per exercise to detect small motion, while avoiding


600

Exercise Motion feature θin [θmin − θmax] [dmin − dmax] [rmin − rmax][degree] [degree] [samples] [degree]

Ex1: Arm abductions Pitch Oy 25 [22.5-22.5] [3.07-4.56] [70.44-82.78]Ex2: Elbow circle Pitch Oy 15 [13.5-18.89] [1.55-2.64] [24.01-42.57]Ex3: Elbow breathing Azimuth Ox 15 [13.5-13.5] [2.5000-3.21] [40.11-68.12]Ex4: Knee extensions Pitch Oy 15 [13.5-13.5] [2.58-3.51] [58.28-87.03]Ex5: Leg lifts Pitch Oy 10 [3.13-9] [2.52-3.34] [7.6-21.91]Ex6: Steps up Pitch Oy 6 [3.13-9] [3.49-4.91] [13.03-36.36]

Table 1. Overview on the rehabilitation exercises and parameter ranges estimated in this work. All parameters were dynamically fitted during systemruntime to personalize the training.

capturing vibration and noise. Table 1 provides an overviewon the exercises and parameters.

Train-mode implementationData segmentation. In contrast to the Teach-mode, Train-mode operation requires online period estimation and sub-sequent performance analysis. A sliding window was usedto segment the incoming data stream. The sliding windowsize was set to cover two average repetitions based on the pa-rameters estimated in the Teach-mode (2md) with an overlapof 75% between consecutive windows. The overlap ensurestimely feedback during a newly detected repetition.

Period estimation. The hill-climbing algorithm was appliedin the sliding windows, configured according to the Teach-mode. Due to the overlap in sliding windows, duplicate peakdetections had to be corrected by matching the peak locationsacross the sliding windows.

Exercise performance class estimationBased on the peak detection, repetitions could be counted.To provide timely feedback, i.e. before the trainee starts asubsequent repetition, the first half of a repetition was eval-uated to estimate duration and range of motion estimates. Inpreliminary tests, we observed that the error incurred by con-sidering only half of a repetition was negligible. The derivedduration di and range of motion ri estimates were used tocompare with the parameters estimated in the Teach-mode,i.e. md, σd, mr and σr. For duration and range of mo-tion, each repetition performance is estimated based on aGaussian distribution N(md, σd) and N(mr, σr). The per-formance of each exercise is classified into 3 class types: in-between, under and above the ranges: [md − 2σd,md + 2σd]and [mr− 2σr,mr + 2σr]. In total, there are then nine differ-ent classes to which each performed exercise repetition couldbe associated.

While we aimed here at a generic system that can deal withvarious exercises, some of them may not benefit from all per-formance classes. For the exercise Steps up, only three per-formance classes (Correct, Too fast, Too slow) were expected,since step height was fixed by the stairs used, thus rangeof motion was not relevant for this exercise. After the per-formance class corresponding to the ongoing repetition wasevaluated, an audio feedback was provided to the trainee, tonotify if a repetition was erroneously performed. The nineperformance classes and feedback are listed in Table 2.

# Performance class Condition(feedback)

1 Correct di ∈ [md − 2σd,md + 2σd],(-) ri ∈ [mr − 2σr,mr + 2σr]

2 Too fast di < md − 2σd,(move slower) ri ∈ [mr − 2σr,mr + 2σr]

3 Too slow di > md + 2σd,(move faster) ri ∈ [mr − 2σr,mr + 2σr]

4 Too large di ∈ [md − 2σd,md + 2σd],(move lower) ri > mr + 2σr

5 Too small di ∈ [md − 2σd,md + 2σd],(move higher) ri < mr − 2σr

6 Too fast and too large di < md − 2σd,(move slower and lower) ri > mr + 2σr

7 Too fast and too small di < md − 2σd,(move slower and higher) ri < mr − 2σr

8 Too slow and too large di > md + 2σd,(move faster and lower) ri > mr + 2σr

9 Too slow and too small di > md + 2σd,(move faster and higher) ri < mr − 2σr

Table 2. Performance classes, feedback and condition used to identifyexercise quality.

EVALUATION STUDYTo test and evaluate our training system, two sets of exper-iments were conducted. Initially, the system was validatedwith healthy participants using a a scripted protocol where allperformance classes have been equally represented. Subse-quently, the training system was evaluated in an interventionstudy with COPD patients. The study was performed duringnormal therapy training sessions to evaluate the overall sys-tem performances and the feedback efficacy. Here we sum-marise the considered exercises and present the study proto-cols used with healthy participants and COPD patients.

Training exercisesSix exercises were chosen according to the COPD guide-lines and in consultation with therapists. The exercises se-lected (as illustrated in Fig. 3) do not require additionalequipment to be performed, thus can be used also out-side of the rehab center. Due to their utility in exercis-ing the breathing, the therapists considered these exercisessuitable for daily training at home. The exercise set con-sist of three upper body exercises and three lower body ex-ercises. The upper body exercises, arm abductions (AA),elbow circles (EC) and elbow breathing (EB), increasestrength in arm and shoulder muscles, while the lowerbody exercises, knee extensions (KE), leg lifts (LL) andstep-ups (SU), train lower body muscles.


601

Figure 3. Illustration of the exercises selected for the training systemevaluation with COPD patients. The patient is wearing the smart-phone (red circle) on limbs that are involved in the different exercises.During the active part of the movements the patients were recommendedto breathe in (inhale) and during the return phase to breathe out (ex-hale).

Data collection and exercise featuresBoth healthy participants and COPD patients were asked towear the smartphone in a holster during the exercises. Theholster was attached to the right wrist during the upper bodyexercises and to the right ankle during the leg exercises us-ing Velcro straps. Three acceleration signals (Ax, Ay, Az)and three Euler orientation angles (Azimuth Ox, PitchOy and Roll Oz) were recorded from the 3D accelerom-eter and magnetometer sensors embedded in a SamsungGalaxy SIII phone. Data was acquired at the sampling fre-quency of ∼50Hz using the Android Software DevelopmentKit (SDK). From the available motion signals, we chose ori-entation estimates as features to describe the sinusoidal exer-cise pattern. According to the phone orientation at the body,Azimuth orientation angle (Ox) was used for elbow breathingand Pitch (Oy) for all other exercises. Orientation estimatescan be efficiently derived on smartphones, thus the approachcan be used even on entry-level phones.

Validation with healthy participantsIn this validation, we quantified the accuracy of the system indetecting the nine performance classes listed in Tab. 2. Fourhealthy participants (all male, 26 ± 2.5 years old) were indi-vidually asked to perform nine exercise sessions of at least tenrepetitions in the university sport center. An expert personaltrainer instructed the participants to perform the exercises asrequired according to the protocol and supervised the wholedata collection. After a first set of repetitions to familiarize

with the exercise, ten continuous repetitions were recordedfor the evaluation. The data gathering started with the ac-quisition of one correct session of ten repetition during theTeach-mode. Afterwards participants were asked to performthe repetitions using the Train-mode.

Evaluation with COPD patientsSubsequently to the validation, we tested the training sys-tem in a pilot intervention study with COPD patients. Forthis evaluation, seven COPD patients (age 60 ± 10 years)were recruited while they were following a rehabilitation pro-gram in a specialized center for pulmonary diseases. The pa-tients ranked a COPD GOLD score between II and III, FEV1

was between 37% and 69% and FEV1/FV C was between27% and 69%. After being instructed on the exercise exe-cution, one Teach-mode session for each exercise was per-formed with one therapist guiding the patient in order to per-form correct movements and rejecting sessions that were notperformed correctly. After completing the Teach-mode for allthe exercises the therapist did not interact anymore with thepatients to simulate individual training exercise executions.The patients started the Train-mode session by touching thestart button on the smartphone’s screen and then began withthe the training. During the exercise execution, the applica-tion gave audio feedback if an erroneous repetition was de-tected and asked the patients to terminate the session whenten repetitions were counted. While the patients were per-forming the exercises, one therapist judged each repetition,according to the movement errors defined before and anno-tated the performance for each repetition in a table. Afterthe recordings, the annotations were digitized. Furthermore,a ground truth was derived after the recordings by reviewingthe raw sensor data timeseries and labeling error classes foreach exercise repetition based on the data. The ground truthlabels were aligned according to the peaks and valleys in thesinusoidal exercise pattern. Speed and range of motion ofeach repetition were then calculated according to Tab. 2.

In total each patient was asked to perform three sessions often repetitions per each of the six exercises. Patients couldtake a break in between if they wished to do so. Data wasacquired in two sessions at separate days. During the firstsession four patients (P1, P2, P3, P4, 1 male and 3 female)were examined and during the second session the remaining3 patients (P5, P6, P7, 2 male and 1 female). In the firstsession, patients (P1-P4) were specifically asked to alternatetheir training such that at least one other exercise was donein between Teach-mode and Train-mode performance of anexercise. With this protocol, we intended to avoid excessiveeffects of exercise memorisation. In the second session, pa-tients (P5-P7) were not asked for alternating exercises. In oursubsequent analysis we did not observe performance differ-ences between the two sessions.

RESULTSThis sections describes the results of our system validationwith healthy participants and the evaluation with COPD pa-tients. Performance of the system to detect execution errors isdetailed for both investigations. For the patient study, we fur-thermore analysed feedback efficacy. Finally, we compared


602

the therapist’s performance in detecting erroneous exerciserepetitions with the ones of our system.

Validation with healthy participantsWe formally evaluated our system with a dataset acquiredfrom healthy participants, using a scripted exercise and per-formance protocol. According to the protocol, participantswere asked to perform all exercise movement errors, besidesthe correct execution of each exercise. Thus, the script andsupervising trainer provided reference information for theground truth used. Here we detail the recognition accuracyfor all nine performance classes and six exercise classes.

Recognition confusion among performance classes. Theaverage performance classification accuracy of the systemwas 96.2%, confirming that applicable feedback would havebeen provided for almost every exercise repetition. The con-fusion matrix for providing matching feedback (i.e. classifytoo slow when the recorded motion was slower than in theTeach-mode) for all performance classes is shown in Figure 4.The matrix illustrates very good accuracies for all perfor-mance classes, above 90%. Lowest accuracy (92%) was ob-served for performance class 9 (Too slow & too small), wherethirteen repetitions where misclassified. A further analysis at-tributed these classification errors to challenges in performingLeg lifts (Ex. 5).

Figure 4. Recognition confusion for providing matching feedback acrossall exercises and validation study participants (healthy adults). Num-bers in cells show class-relative accuracy (first line) and number of in-stances (second line).

Recognition accuracy for exercise classes. To further anal-yse correctness of the feedback, Fig. 5 shows the accuracydistribution across exercises and performance classes. It canbe observed that most of the erroneous feedback in perfor-mance class 9 was given during exercise Leg lifts (Ex. 5),where the system showed lowest accuracy (77.5%).

Evaluation with COPD patient in intervention studyWe evaluated system performance for counting exercise rep-etitions and recognizing the trainee errors performed in agroup of seven COPD patients. During the intervention studyrecordings, we observed that the therapist could not accu-rately assess all exercise repetitions, because it was difficult

Figure 5. Recognition accuracies for exercises and performance classesin all validation study participants (healthy individuals). For exerciseStep up (Ex. 6), no instances for performance classes 4-9 were recorded.

to observe the patient’s movements and annotate the exerciseperformances simultaneously. A ground truth labeling wasderived in a post-recording step according to the raw sen-sor measurements, as explained in Section Evaluation withCOPD patients. We needed to exclude two sessions of Step-ups performed by Patient 1, three sessions of Leg lifts, by Pa-tient 3, and one session of Knee extensions by Patient 7, dueto data recording issues. Patient 6 could only complete thefirst session of Elbow circles due to shoulder pain. In total,1176 exercise repetitions were acquired and analyzed.

Accuracy in counting repetitions. Redundant feedback (in-sertions due to noise or vibration) and missing feed-back (deletions due to missing peaks) were further consideredin this analysis. Figure 6 shows examples of the two errortypes. Both insertion and deletion errors were considered fordetermining repetition counting performances. We denotednp as the number of repetitions performed and nc the num-ber of repetitions counted by the application. The repetitioncounting error incurred by our system for each exercise ses-sion is represented by ec = |nc − np|. The error ratio foreach exercise and patient are shown in Fig. 7. Even thoughtthe average counting error across all exercises for Patient 1was acceptable (ec = 9%), repetitions of P1 were challeng-ing to identify, in particular during Elbow circle (ec = 20%)and Leg lift (ec = 31%) exercises. By grouping the patients,the two exercises challenging for repetition counting were in-stead Leg lift (ec = 9.6%) and Step-up (ec = 6.3%). For allother exercises, the error incurred by our training system wasbelow 3%. The overall counting accuracy (accc) across allpatients and exercises was set to accc = 100%− errortot%,with accc = 96.7%.

Accuracy of system feedback. Feedback caused by inser-tion errors represented repetitions that were actually not per-formed by the user (no ground truth reference available). In-sertions were not further considered for the feedback analysis.Deletion errors were handled and taken into account whencomputing feedback accuracy, however deletions are not vis-ible in the confusion matrix.


603

Figure 6. Timeseries plots from Leg lift (top) and Elbow circle (bottom)exercises illustrating insertion and deletion errors. Crosses indicate thelocal maxima and minima identified. Arrows indicate insertion (top) anddeletion errors (bottom).

Figure 7. Exercise repetition counting error distribution in the patientintervention study.

The confusion matrix illustrating the system accuracy for pro-viding matching feedback is depicted in Fig. 8. Overall, anaccuracy of 87.5% was achieved. Correctly performed repe-titions (performance class 1) were recognized at 90%. ForToo large (class 4) and Too fast and too small (class 9) alower accuracy (63% and 64%, respectively) with regard tothe healthy participants validation, was obtained. Further-more, the distribution of the error across exercise repetitionswas investigated and is shown in Fig. 9. It can be observedthat errors occurred mostly during first and last repetition ofeach exercise session (17% and 21% of the total errors, re-spectively). We assume that these repetitions may have beeninfluenced by interactions of trainee and smartphone to startand stop recordings.

Figure 10 further analyses the system’s feedback with re-gard to individual exercises and patient. As the result shows,

Figure 8. Recognition confusion for providing matching feedback acrossall exercises and intervention study patients. Numbers in cells showclass-relative accuracy (first line) and number of instances (second line).

Figure 9. Patient performance error distribution for all exercise repeti-tions. Most errors occurred during first and last repetitions.

for Patient 3, lowest accuracy (84%) was obtained. Stepsup (Ex. 5) were the most challenging to recognise, resultingin an accuracy of 85%.

Efficacy of the smartphone-based intervention. We eval-uated the efficacy of feedback provided during the exercisesessions. Figure 11 shows how patients reacted to feedbackduring the training. After an audio feedback, a total of 297repetitions were performed correctly. Feedback for 57 rep-etitions was ignored, with subsequent repetitions performederroneously too. By analyzing changes in exercise perfor-mances, we observed that in 119 cases feedback caused achange in the following movement (both, correct and othererroneous performance). In 71 cases the feedback was ig-nored, not causing any change in behavior. In only 8 casesmight the feedback have led to performing errors in the fol-lowing repetition.

Comparison of therapist and training system. From litera-ture it is known that therapists may disagree amongst them orcould make errors when assessing training performance [8].We analysed the therapist’s assessment in comparison to our


604

Figure 10. System feedback accuracy per patients (P1-P7) and exer-cise (Ex1-Ex6) in the intervention study.

Figure 11. Feedback efficacy of the training system during the patientintervention study.

ground truth. With this analysis, we could estimate an ade-quate performance level for our training system. Our analysisshowed that the therapist performs better than our trainingsystem for the performance class Correct (96% vs. 90%).For all error performance classes, the training system showsbetter performance. Overall, performances match each otherclosely suggesting that the training system is suitable for pro-viding exercise feedback.

DISCUSSIONOur training error classification technique is based on a sinu-soidal motion model, which showed good results in detectingmovement errors for both, healthy adults and chronic patients.We chose this method over others for two reasons: (1) usingmachine learning techniques requires a training set to obtainthe classifier model. In particular, a sufficient number of ex-ercise error instances would be required. For our application,it was not feasible to let patients perform exercise errors dueto the risk of injuries. (2) With machine learning techniques,it is difficult to differentiate variations in performance of thesame exercise from execution errors. Hence, we formalizederror classes by considering deviations from the correct exe-cution using our sinusoidal model.

We consider that other exercises can be integrated in our train-ing system by selecting a suitable motion feature that reflectsthe sinusoidal pattern and estimating the initial hill-climbingparameter threshold. Subsequently, speed and range of mo-tion can be considered to determine the performance classes.The relevance of a Teach-mode was confirmed by the param-eter ranges obtained after estimating exercise model param-eters for every patient. Hence, we expect that our exerciseanalysis approach is scalable to other patient groups requir-ing exercise training feedback systems. Figure 4 shows excel-lent system accuracies in the validation with healthy subjectsfor all performance classes. The remaining classification er-rors could be explained by the fact that participants could notalways maintain speed and range of motion, while perform-ing all repetitions of an exercise. Lowest accuracy for the er-ror combination Too slow & too small (performance class 9)could be attributed to the challenge in performing individualexercises, e.g. Leg lift (Ex. 5).

With regard to the clinical analysis, errors in counting Stepsup occurred for all patients (Fig. 7). This can be explainedby the fact that patients often toed the step before completingit. For Patient 1, the lowest counting accuracy was obtained.Patient 1 was the oldest and most severely affected partici-pant in the study and for this patient the exercises were moredifficult to perform fluently. From literature it is known thattremor is larger for slow and small movements [13], whichmay have resulted in peak detection errors and lower count-ing accuracies for Elbow circles and Leg lift exercises. Whenanalyzing exercises within a small range of motion, the algo-rithm was more sensitive to noise and resulted in wrong peakdetection. The lowered recognition accuracies (Fig. 8) of ourintervention study compared to healthy participant validationcould indicate the effect of vibrations due to muscle tremor.

Teaching and training were performed very close in time. Thevariation of how patients would perform the exercises at homemay be higher and exercise memorisation may have dimin-ished. Nevertheless, we think that patients could benefit fromthe COPDTrainer app when training at home: it is probablethat patients will perform more errors at home and thus thesystem’s guidance feedback on movement errors will be moreeffective compared to the clinical setting. For this study, thetherapist’s presence during all study sessions was required toensure patient safety. During the train-mode, the therapist didnot aim at influencing the exercise performance of patients inany way. Moreover, the therapist was a well-known personto the patients. Still, patients may have been affected in theirbehaviour in subtle form since they were watched by anotherperson. This would differ from the setting at home. We alsoconsidered video recording the exercise repetitions, but apartfrom the unmet safety requirements, cameras may imply evenstronger physiological stress to patients. They were used totrain in presence of a therapist while not in presence of a cam-era. Subsequent clinical studies should aim at a step-wise re-duction of supervision through therapists or other means, inorder to accustom patients with the new technology.

This paper focuses on evaluating performance and feedbackefficacy of a patient training system using a smartphone only.


605

Our primary goal was to derive an easily reusable modellingapproach. We did not investigate in long-term usability, moti-vational aspects, and user acceptance. Nevertheless, we havetaken these aspects into account by choosing the smartphoneas sensing and feedback platform. While in our study, pa-tients did not express issues in handling the phone, it is clearthat further app design refinement could support patients. Webelieve that smartphone apps can be optimised for wide useracceptance. A wearable device would incur less weight tocarry and be more comfortable during training. In contrast,patients are already familiar with smartphones. Patients couldimmediately use them remotely, even in developing countrieswhere distances may not allow them to attend a rehab centerregularly. Our training and modelling approach can be usedas well with wearable sensor bracelets or smartwatches thatintegrate audio or vibration feedback.

CONCLUSION AND FURTHER WORKIn this work, we investigated a novel smartphone-based exer-cise training and feedback system for chronic patients usingsmartphone-integrated sensors only. Our approach integratesinto therapy routines, where an initial training session with atherapist is used to derive exercise quality parameters. In sub-sequent patient training sessions, the system can provide in-stant acoustic feedback on the detected exercise performance.Our validation with healthy participants showed an overall ac-curacy of 96.2%. We implemented an intervention study withseven COPD patients in their regular rehabilitation environ-ment to assess the viability of our approach. Repetitions werecounted at 96.7% accuracy and trainee performance classifi-cation rate was 87.5%.

Based on our result, we concluded that a smartphone-basedtraining system can be used to asses the performance and ex-ecution quality of a rehabilitation exercises in COPD patients.Based on the system performance and feedback efficacy, webelieve that our approach and developed methods will be a vi-tal basis for future investigations on training systems for dif-ferent patient groups. Additional steps are needed to confirmthe clinical relevance and integration into clinical practice.In this regard, we consider our work as a pilot study, pro-viding the basis for validating COPDTrainer in a clinically-supervised intervention at the patient’s home.

ACKNOWLEDGEMENTThis work was supported by the EU FP7 Marie CurieNetwork iCareNet under grant number 264738 and theiCare4COPD project of Agentschap NL under contract num-ber PNE101005.

REFERENCES1. Buttussi, F., and Chittaro, L. Mopet: A context-aware and

user-adaptive wearable system for fitness training. Artif IntellMed 42, 2 (February 2008), 153–163.

2. Celli, B. R., MacNee, W., and , A. T. S. R. S. T. F. Standardsfor the diagnosis and treatment of patients with copd: asummary of the ats/ers position paper. Eur Respir J 23, 6 (Jun2004), 932–946.

3. Chang, K.-H., Chen, M. Y., and Canny, J. Tracking free-weightexercises. In Proceedings of the 9th international conferenceon Ubiquitous computing, UbiComp ’07, Springer-Verlag(Berlin, Heidelberg, 2007), 19–37.

4. Chang, Y.-J., Chen, S.-F., and Huang, J.-D. A kinect-basedsystem for physical rehabilitation: A pilot study for youngadults with motor disabilities. Research in DevelopmentalDisabilities 32, 6 (2011), 2566 – 2570.

5. Consolvo, S., McDonald, D. W., Toscos, T., Chen, M. Y.,Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., LeGrand,L., Libby, R., Smith, I., and Landay, J. A. Activity sensing inthe wild: a field trial of ubifit garden. In CHI 2008: Proceedingof the twenty-sixth annual SIGCHI conference on Humanfactors in computing systems, ACM (New York, NY, USA,2008), 1797–1806. SESSION: Personal Health.

6. de Oliveira, R., and Oliver, N. Triplebeat: enhancing exerciseperformance with persuasion. In Proceedings of the 10thinternational conference on Human computer interaction withmobile devices and services, MobileHCI ’08, ACM (NewYork, NY, USA, 2008), 255–264.

7. Floyd, R., and Thompson, C. Manual of StructuralKinesiology. 2011.

8. Harms, H., Amft, O., Appert, M., Mller, R., and Troster, G.Wearable therapist: Sensing garments for supporting childrenimprove posture. In Ubicomp 2009: Proceedings of the 11thInternational Conference on Ubiquitous Computing, ACMpress (2009), 85–88.

9. Havens, T., Alexander, G., Abbott, C., Keller, J., Skubic, M.,and Rantz, M. Contour tracking of human exercises. InComputational Intelligence for Visual Intelligence, 2009. CIVI’09. IEEE Workshop on (2009), 22–28.

10. Kranz, M., Moller, A., Hammerla, N., Diewald, S., Plotz, T.,Olivier, P., and Roalter, L. The mobile fitness coach: towardsindividualized skill assessment using personalized mobiledevices. Pervasive Mob Comput (2012).

11. Marshall, S. J., and Ramirez, E. Reducing sedentary behavior:A new paradigm in physical activity promotion. Am J LifestyleMed 5, 6 (2001), 518–530.

12. Moller, A., Roalter, L., Diewald, S., Scherr, J., Kranz, M.,Hammerla, N., Olivier, P., and Plotz, T. Gymskill: A personaltrainer for physical exercises. In Pervasive Computing andCommunications (PerCom), 2012 IEEE InternationalConference on (2012), 213–220.

13. Morrison, S., Mills, P., and Barrett, R. Differences in multiplesegment tremor dynamics between young and elderly persons.The Journals of Gerontology Series A: Biological Sciences andMedical Sciences 61, 9 (2006), 982–990.

14. Muehlbauer, M., Bahle, G., and Lukowicz, P. What can an armholster worn smart phone do for activity recognition? InWearable Computers (ISWC), 2011 15th Annual InternationalSymposium on (2011), 79–82.

15. Pernek, I., Hummel, K. A., and Kokol, P. Exercise repetitiondetection for resistance training based on smartphones.Personal Ubiquitous Comput. 17, 4 (Apr. 2013), 771–782.

16. Spruit, M. A., Troosters, T., Trappenburg, J. C. A., Decramer,M., and Gosselink, R. Exercise training during rehabilitation ofpatients with copd: a current perspective. Patient Educ Couns52, 3 (Mar 2004), 243–248.

17. Strohrmann, C., Harms, H., Troster, G., Hensler, S., andMuller, R. Out of the lab and into the woods: kinematicanalysis in running using wearable sensors. In Proceedings ofthe 13th international conference on Ubiquitous computing,UbiComp ’11, ACM (New York, NY, USA, 2011), 119–122.

18. Tseng, Y.-C., Wu, C.-H., Wu, F.-J., Huang, C.-F., King, C.-T.,Lin, C.-Y., Sheu, J.-P., Chen, C.-Y., Lo, C.-Y., Yang, C.-W., andDeng, C.-W. A wireless human motion capturing system forhome rehabilitation. In MDM 2009: Tenth InternationalConference on Mobile Data Management: Systems, Servicesand Middleware, IEEE (2009), 359–360.

19. Velloso, E., Bulling, A., and Gellersen, H. Motionma: Motionmodelling and analysis by demonstration. In CHI ’13 (2013).


606

copdtrainer: a smartphone-based motion rehabilitation training system...

Documents