presentation of evidence on data challenges · presentation of evidence on data challenges cheryl...

Presentation of Evidence on Data ChallengesCheryl Grandinetti, FDABrian Perry, Duke UniversityRay Dorsey, University of RochesterPhil Coran, MedidataChris Miller, AstraZeneca

June 15, 2017

Day 1 Data bundlesData Integrity, Data Collection, Data Management and Data Access

Data Attribution, Data Monitoring, Audit Trails and Data Security

Data Analysis and Interpretation, Making Data Available to the FDA

Lessons Learned Related to Data IssuesInterviewees’ Recommendations Regarding Data Issues

Phase I Interviewees: Sponsors (15)Focus of research:

• All involved in drug development • Almost all were involved in biologics

Use of devices:• All used movement sensors • 73% had used smartphone apps • Some had used biosensors, pressure sensors, video

cameras, audio sensors, global positioning systems (GPS), adherence monitors, and electronic patient-reported outcome devices

Phase I Interviewees: Investigators (8)All worked in an academic institution or academic health system that has research and education responsibilities

Role in current trials: PI=7 Sub-I/co-PI=1

Focus of research: 75% involved in medical devices 50% involved in drug trials

Use of devices: All used movement sensors All used smartphone apps Some used biosensors, pressure sensors, video cameras, audio

sensors, and GPS

Phase II Interviewees: Subject matter experts

20 experts interviewed: Device manufacturers (n=10) Data management experts (n=2) Biostatisticians (n=4) Data security experts (n=4)

Device manufacturers included software and hardware developers of: Inertial sensor devices (n=5) Glucose monitors (n=2) Mobile data hubs (n=1) Wearable cardiac and respiratory monitors (n=1) Mobile health applications (n=1)

One commercial device company was represented among manufacturers

Data Integrity, Data Collection, Data Management and Data AccessRay Dorsey, University of Rochester Medical Center

Brian Perry, Duke University

Our Working Definition of Data IntegrityData integrity requires that data cannot be modified or corrupted in an undetectable way during its generation and flow (derived from ISACA definition) We have defined data integrity narrowly as a discrete issue for the

purposes of this project. Recognize that data integrity in the broader sense is woven

throughout most of the data issues we will consider i) completeness and consistency as part of data collection and data origination, and ii) data integrity as part of data transmission and data security

Sponsors and Investigators Approaches to Addressing Data Integrity

A number of investigators and sponsors were not involved in data integrity aspects, relying on external analytic groups to ensure data integrity & maintenance of the audit trail Assessed the analytics group’s dataflow procedures Ensured that external groups used approved, secure data transfer

platforms (encrypted, de-identified)Some used real-time or instant data upload via Wi-Fi/Bluetooth to help ensure that data were not altered prior to transferring to the study databases

I mean, we could see the data in real time, not that we were looking in real time all the time … it essentially went from the [wearable] patch to the computer system and then, via satellite, to our data analytic group in [name of US city]. - Sponsor

Sponsors and Investigators Approaches to Addressing Data Integrity

A few pilot tested data collection and transfer platforms One interviewee described using an application

programming interface (API) to screen for incomplete data as a means for ensuring that all datasets were complete and any missing data were identified and flagged.

Two not concerned with data integrity Perceive mobile devices to be more reliable than other

forms of data collection

Biostatisticians’ Concerns with Data Integrity

There’s a much higher distance between the data collection and the data analysis in terms of steps that have to be taken. The data are first recorded in the device, and then transferred to phone, and then phone

uploads it to the [device manufacturer’s] cloud, and then someone picks them up from the [device manufacturer’s] cloud and puts them

on a data storer, and then they’re finally analyzed. So if you think each of the steps can be affected in some way, then the probability of having some data corruption is much higher than you have it in a

standard clinical trial. - Biostatisticians

Manufacturers’ & Managers’ Approaches to Addressing Data Integrity

Make the following as simple as possible: Design of the technology User engagement with the device Data retrieval mechanisms and procedures System controls

Conduct human factor testing to understand the challenges users may have in engaging with the technology

Develop mechanisms for clients (i.e., sponsors) to report problems with data may help to identify misuse or malfunctioning devices


Provide detailed user instructions

Run pilot simulations of data transfer and retrieval procedures

Design software to automatically check for integrity of data collected by a device (e.g., checksums)

Review time stamps linked to data Include multiple time stamps—time of capture and time

received


Additional suggestions related to security (described in more detail later) A “security umbrella protocol” built into an app that

manages three levels of security: authentication, encryption, and non-repudiation (transferring the right data from the right source) Limiting data access via a closed, cloud-based app for

secure data capture and storage Hard-wired transfer of data removes potential of altering

data during transfer

Our Working Definition of Data Collection

Data collection is the process of gathering and measuring information on targeted variables in an established systematic fashion, which then enables one to answer relevant questions and evaluate outcomes.

Investigators and Sponsors Approaches to Addressing Data Collection

Investigators and sponsors comments focused on: The lack of familiarity with device and data among

patients, regulators, IRBs, and study personnel The need for high-quality data for regulatory purposes The need to reduce patient and staff burden and promote

retention How to ensure participant compliance How to deal with unforeseen technical issues

How have investigators and sponsors addressed concerns about lack of familiarity?

Conduct small-scale studies to build institutional knowledge and experience with the mobile deviceWork with implementing partners who are familiar with the use of mobile devices in clinical trialsAllow time upfront to provide hands-on experiential training for staff and to identify technical issues

How have investigators and sponsors addressed quality and regulatory concerns?

Will mobile devices provide high quality data necessary for regulatory purposes? Many interviewees believe they can

Will these data be accepted by regulators?I think one of the biggest concerns that it is very prevalent within

the company itself is—it is very uncertain whether or not regulators will accept the data coming from devices due to just

the novelty of it all and the lack of strong guidance in implementation and use and expectations on their part. So, while

we were very excited about the possibilities that devices like these can bring to clinical trials, there's major concerns.

—Sponsor

What approaches have investigators and sponsors taken to reduce patient and staff burden?

Passive data collection, use few devices Avoiding the collection of unstructured data (increase personnel time required for analysis and interpretation)Using simple devices without a lot of additional functionality (ensure interoperability of multiple devices)

What approaches have investigators and sponsors taken to ensure patient compliance?

Sensors could be used to monitor if and when patients are actually using the technologyA belief that by offering full-time user support, patients would be better equipped to comply with study proceduresRaises question about who owns the data

So they had the app on their smartphone, and they record their own symptoms. And if all they want to do is look at it themselves in terms of the summary graphs and things like that, then that's fine…if they want to show the handset data to friends and family, to health professionals, primary care

physicians, whoever it is…if they're happy with it, to have the summary data streamed through into electronic care records of hospitals they're in contact with...But all stages of data is owned by the patient.—Investigator

Challenges: Remote data capture (i.e., collecting data outside of a

controlled research setting) Asking patients to be responsible for collecting data Volume and variety of data captured

Data Collection: Biostatisticians’ Perspectives

Challenges:Technical challenges due to device malfunction or

problems with the remote data collection infrastructure Not having traditional source documents The inability to confirm that the intended user was the

originator of the dataUsing mobile devices reduces researcher control on how,

when, and where the data are collected

Biostatisticians' Comments on the Challenges of Remote Data Capture

If you want to capture it in terms of variance, when you’re in free living conditions, your sources of variance are much higher than in

a controlled environment. — Biostatistician

Suggestions: Include geolocation metadata (but introduces additional

concerns) Increase size of data sample (either by increasing N or T)

may help to address variance in data

Biostatisticians' Comments on the Challenges of Remote Data Capture

So how do you still find signal [meaningful information] and that much variance? Well, you have to up the N. […] So yeah, to reduce bias in

free-living condition, just jack up your N, number of patients, and also, you can jack up your T, the time at which you are measuring things, because that also is a measuring of how much information you’re

collecting. — Biostatistician

Biostatisticians’ Comments on Asking Patients to Collect Data

Challenges: Operational challenges

• Losing the device• Misusing the device• Damaging the device• Forgetting their access codes

Potential for falsifying data Too much participant training not representative of

daily living BYOD increases potential variability in data quality and

integrity


Suggestions: Biostatisticians suggested that researchers ensure that

the technologies:• Are simple • Are appealing to the target population• Are durable and can withstand user abuse• Match the users’ capabilities• Fit within the users’ daily routines

If BYOD are used, researchers must • Prevent the downloading of any firmware or software• Record the device model and firmware information

with the data elements


Suggestions (continued): Other suggestions focused on reducing participant

burden as much as possible and included:• Have participants manually enter the least amount of

data necessary• Avoid making participants change passwords too

frequently • Avoid making participants use more devices than

necessary• Rely on passive engagement with mobile technologies

– Use automatically gathered metadata recorded by devices whenever possible

Biostatisticians’ Comments on the Volume and Variety of Data Captured

Challenges: More data better in all situations General difficulties in storing and cleaning large datasets Variety of data captured could pose new analytical

challenges

Suggestions: Ensure that the sampling rate and amount of data

collected match the research question Ensure that data collection does not detract from the

real-world nature of remote data capture by overburdening users

Suggestions (continued): Include a biostatistician in the initial study design

conversations to • Determine the appropriate sampling framework• Develop a data management and analysis plan

Avoid collecting data information that will not be used to answer the research question(s) Ensure that data collected will inform a clinically

meaningful outcome Include tech-savvy informatics experts to help manage

larger data sets

Biostatisticians’ Comments on the Volume and Variety of Data Captured

Our working definition of data management

Data management is administrative process by which the data from mobile devices are acquired, validated, stored, protected, and processed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of the data users.

Investigators’ and sponsors’ comments on data management

Major change from traditional data collection—some vendors store raw study data on their own servers and the investigators and sponsors may not have direct access to those data The vendor writes algorithms and provides aggregated

data to the research team upon request.A few interviewees said they relied on an external data management team for managing their study data, specifically for identifying errors and cleaning datasets

Our working definition of data access

Data access is the ability of sponsors and device manufacturers to view or use information collected by a mobile device that is relevant to the clinical investigation.

Device Manufacturer’s Comments on Data Access

Various approaches to enabling sponsors and other users to access data, the types of data that they provide, and their views on data ownership Providing clients with interpreted data, but noting that the

sponsor has 100% ownership of the data and can access it any time Providing access to both the raw and the transformed

data Providing transformed data

Device Manufacturer and Security Experts’ Comments on Data Access

Data collected by a device will often flow through the manufacturer’s server prior to being made available to the client (sponsor) Depending upon the device and user agreements, data

may be accessed and used by the manufacturer in various ways

Some manufacturers produce customizable APIs or closed-data hubs that allow clients (sponsors) to communicate with and access data directly from the devices used in the study This removes the manufacturer from the data chain

Security Experts’ Comments on Data Access

Manufacturers are granted access to data collected by their devices when users register their devices during activation For researchers to be able to access these data, a

program must be developed to request access to specific data from the participants’ accounts on the manufacturer’s server

Specific data governance policies may be set up to establish data access rights (e.g., the specific time in the study when data can be accessed and the specific individuals who can access the data)

Data Integrity, Data Collection, Data Management and Data Access

Brief questions and comments

Data Attribution, Data Monitoring, Audit Trails and Data SecurityPhil Coran, Medidata Solutions


Our Working Definition of Data Attribution

Data attribution is the act of establishing a particular individual / participant as the creator of data

Sponsors’ and Investigators’ Comments on Data Attribution

Many were concerned about how to ensure that mobile devices were not used by unintended users and used only by the patient

Yet, most said they were uncertain of how study personnel ensured that only intended patients used the device

Sponsors’ and Investigators’ Suggestions for Ensuring Correct Data Attribution

Monitor trends in patient data

Use technology that is only relevant to particular patient populations

Enable location tracking capabilities on the technology

Program monitoring algorithms that scan for and “flag” outlying data points

Program a unique passcode to download and use the technology

Make the technology only accessible through the use of a unique biomarker

Have study personnel download apps

Biostatisticians’ Comments on Using Trend Data to Ensure Attribution

Viewed the approach as a good idea, but also said Is just one tool to address attribution Post hoc analyses reveal inconsistencies in trend and

therefore have limited benefit It takes a significant amount of work to determine

attribution because the data are complex and there are currently no tools to automate analysis There’s a shortage of statisticians qualified to tackle the

challenge of determining attribution using trend data or other approaches Some participants will always try to beat the system

Biostatisticians’ Comments on Data Attribution

Alternative approaches to monitor and ensure attribution: Password protected devices Biometric authentication (fingerprints, eye scans) Site monitoring Risk-based monitoring Clustering techniques Programs that can detect randomness Retrospective questionnaires about device use Monitor users’ experiences with weekly check-ins

I think one thing we’ve done is retrospective questionnaires. So, you elect to do that every week. You can ask, “Was your device used by

someone else? So do you think there’s a chance your Fitbit was used by your child?” Like this opens the door to much longer

conversation, which is not what this interview is about, but when you do trials in free-living conditions, you really have to put an effort into monitoring and user experience. So you have to be able to reach out

to the patient and make sure that the patient can reach out to you easily, if they found problems with the tech. So maybe reach out with

questionnaires or make sure that the person reaches out to you every week with a report. — Biostatistician

Device Manufacturers’ Comments on Data Attribution

Very limited ability to ensure intended user is the only one entering data into the device Either implant the device Securely attach it to a patient via a hospital band (which is

something that one manufacturer has done) Some technologies are less prone to potentially being

used by individuals not enrolled the study, simply because the technology is not relevant to individuals without a specific condition (e.g., glucose monitoring)


However, manufacturers may be more concerned with non-compliance than ensuring attribution Develop technologies individuals want to use regularly

may allay attribution challengesBut the real question here, and I think this is where the regulatory

bodies need to talk a little bit more, is what's the motivation for the patient to put the wearable on someone else? ... So the

bigger challenge is non-compliance rather than attribution… And so what our goal is to work with the advocacy groups, work with

the sponsors to understand what can we give patients that doesn't modify behavior, but also engages them and gets them

excited? …and we believe that will help the attribution. — Manufacturer


Alternative approaches to monitor and ensure attribution: Password protect devices Use a multi-layered authentication process Have site staff assign sensor with specific IDs to patients,

and link the sensor IDs to the patient IDs as part of the enrollment process Monitor trends from baseline

• Build a patient profile, e.g., how they perform on a regular basis

Asking patients to do a particular task in a particular order at the start of the day

Our Working Definition of Study Monitoring

Study monitoring is the activity performed by a study monitor, examining the data for completeness, consistency and accuracy

Sponsors’ and Investigators’ Comments on Data Monitoring

Overall concerns related to remote data monitoring:What information should be monitored throughout the trial How frequently this data should be monitoredWho is responsible for monitoring data How to monitor big data How to monitor for safety Whether and how to share data with a Data Safety

Monitoring Board (DSMB)

Types of Data that Sponsors and Investigators Reports Monitoring

Data that could be emblematic of glitches in the technology (e.g., inaccurate time stamps, erroneous or impossible data points)Missing or outlying dataTiming, frequency, and density of data from deviceDate and time of technology upload, distribution and returnLocation or movement of patients via mobile deviceTrendsEnrollment and continued participationPatient-reported acceptability of the technology

Investigators’ and Sponsors’ Comments on Intermittent & Continuous Monitoring

Use application programming interfaces (APIs) to pre-program monitoring algorithms Several said they conduct intermittent monitoring of the

data (e.g., monthly, weekly, per batch transfer, at the end of the study) Others said they conduct continuous remote

monitoring (depends on study design)• Look at trends• Identify missing or outlying data• Monitor for recruitment and fulfillment of study

procedures

Investigators’ and Sponsors’ Comments on Intermittent & Continuous Monitoring

Interviewees commented that substantial investment in monitoring data at the beginning of the study may be helpful in identifying technical problems before they become too detrimental for the study

The one thing that we’d done that was relatively smart, was that we’d said, “There is a risk here. We’re doing something new. It’s for the first time. What we ought to do is put a very early interim read to make sure that we are actually getting the data that we expect and that it’s robust,” and that was where the issue was picked up. We then resolved it, and the issue has not come back…—Sponsor

Investigators’ and Sponsors’ Comments on Who is Responsible for Monitoring

For most, the clinical research associate or coordinator conducted most of routine data monitoringOthers said an external analytics team monitored their dataInvestigators, data managers and analysts were also suggested as staff who should take some action to monitor data throughout the studyDSMB: some believed it would be difficult for DSMBs to interpret the larger complete datasets

Investigators’ and Sponsors’ Comments on How to Monitor for Safety

Review trends in remote data outputs, but few doing this

A few said avoid monitoring for safety outcomes because the technology used wasn’t considered a medical or diagnostic device

Some relied on patients reporting their own adverse events, or that personnel monitored patient records rather than relying on the mobile device to monitor for safety outcomes

Security experts’ comments on monitoring data

All security experts stated that no one needs to view the data to ensure that they are being received as expected or to monitor for missing or out-of-range data Servers can be programmed with data monitoring

software to automatically scan for data quality and to send alerts to data managers when issues arise

Defining an Audit TrailFDA’s eSource guidance defines an audit trail as a process that captures details such as additions, deletions, or alterations of information in an electronic record without obscuring the original record. An audit trail facilitates the reconstruction of the course of such details relating to the electronic record.

Device Manufacturers’ Comments on Ensuring Data Record is Accurate

Ensuring that data are not inadvertently corrupted during collection, transfer or storage Use a closed, cloud-based system for secure data

retrieval and storage that requires authentication and log in and maintains a full audit trail of all activity Ad hoc monitoring of data pulled directly from device to

data retrieved from manufacturer’s server Collect time of data capture and data receipt

Device Manufacturers’ Comments on Ensuring Data Record is Legible

Numerous data outputs are used to present data in a clear and standard format—that are readable by most computers—and to reduce the risk of misreading the data element: CSV (Comma Separated Values) Excel XML (Extensible Markup Language) DFD (Data Flow Diagram Graphic) SAS

Device Manufacturers Comments on Ensuring Data Record is Original

Data output from technologies can be traced back to original source data from “converted” data

One manufacturer said that no changes should ever be made to the raw data However, if changes are made, they should be made with

an electronic audit trail so that the raw (original) data are not changed in any way

Our working definition of data securityData security refers to protective measures that are applied to prevent unauthorized access to electronic systems. Data security also protects data from corruption. Data security is expected in transit (transmission) and at rest (storage) so that the data is not corrupted or lost and in the event of such corruption or loss, the detectability.

Sponsors’ and investigators’ practices for keeping data secure

Secured data transfer and storageHIPAA mandated firewalls and security measures such as encryption and password protectionInvolved the IT department to ensure security during storageRestricted accessTracked trendsUsed an algorithm that creates security flags

Data security — vendors Methods for ensuring that vendors’ security measures meet sponsor and/or regulatory requirements were: Having a corporate audit or compliance team that vets

vendors’ security measures and work with companies that have validated their security measures

Investigator and Sponsor current approaches to ensure privacy & confidentiality

Password protect devices and appsDo not store PHI with dataStore PHI and clinic data on separate servers, which could be linked by an algorithm, if needed

This [ensuring privacy and confidentiality] is a really, really important thing that we take incredibly seriously, actually. In this case…every

device is labeled with a serial number. The device manufacturer only has the serial number. So, they don’t know any patient

identifiable information. And actually, the core data that was being returned to the device manufacturer, its primary use was to

essentially get a heartbeat of the device itself so [to] know that the device is functioning correctly, getting the diagnostics from the

device, of which the usage data is kind of one diagnostic. —Sponsor

Device Manufacturers’ comments on data security

Accessing data directly from their mobile devices is limited because they have: Ensured that the data and servers are encrypted Installed “no write access” to the raw data stored on the device and

on the server

Some devices require data to be transferred via a hardwire link between the device and the study server, which reduces the risk of data being accessed and altered while being transferred

Can provide customizable APIs or a closed server (or central hub) so sponsors can connect directly to devices to retrieve data rather than through the manufacturer’s server

Security experts’ comments on data security

Nearly each of the security experts noted that any data stored on a mobile device, or transferred wirelessly from a device to a server, or even stored on a secure server, can be hacked if someone was motivated enough

Yet, the likelihood of hacking given current safeguards, is low And may pose no greater risk for security than current

paper-based data collection.

Ultimately security is done in layers… it really comes down to what layers do you want to accept and how far do you have to take it.

— Security expert

Data security experts’ comments on securing the storage of data on mobile devices

Automatically encrypt data when stored locally on the device/app

Limit the actual amount of data stored on the device

Retrieve data from mobile technologies as soon as possible

Access data directly from the device rather than retrieving data through the manufacturer’s server

Limit the variability of devices and technologies used in the study so that researchers have more ability to manage security issues

Data security experts’ comments on securing the storage of data on mobile devices

Restrict the users’ access to data so they cannot tamper with them or view them

Program software that would require users of a device or app to enter a passcode to use the technology

Program software to include automatic timeouts for infrequent use

Educate users to be more conscious of security issue

Data security experts’ comments on securing the transfer of data from the device to the server

Use an encrypted file transferring platform (FTP), such as HTTPS connections to transmit data wirelessly

Consider using a hardwired transfer (i.e., plugging a device directly into a computer that can access the secure server to download data from the device) because it removes the potential that data could be accessed by someone outside of the study during transfer—experts warned, however, that this approach increases participant burden

Use an encrypted FTP and encrypt the payload (i.e., the batch of data being sent)

Data security experts’ comments on securing the transfer of data from the device to the server

Perform “checksums” or “two phase commits” to ensure that the data sent matches the data received and that no data were lost or changed during transfer

Include “Certificate Pinning” software on the mobile device and on the server This would ensure that the device(s) used in the study

would only be able to communicate with servers with the correct pin, and servers, likewise, would only be able to communicate with devices that had the correct pin

Data security experts’ comments on securing data during storage on server

Require multiple layers of authentication to access the data

Use a server that can only be accessed through a Virtual Private Network (VPN)

Maintain a log of who accesses the data server and when (which will be helpful in the case of security breaches)

Work with a security vendor or commercial company that has invested in secured data storage for their business

Data security experts’ comments on additional security procedures

Review or audit security procedures for server hosts to ensure they have adequate SOPs for routinely reviewing who has access to servers and for disabling access for individuals when required

Assess the technical and security awareness of participants prior to enrollment and either only enroll participants with a certain level of expertise or use this information to weigh study results

Educate participants about the importance of data security

Instruct participants on how to secure their data

De-identify data as soon as possible or refrain from capturing identifiable data at all via the mobile device

Data security experts’ comments on BYOD

Challenges associated with variability in technologyBecause of the different standards, particularly software that's doing some

level of encryption and maybe it's expecting to do a high level of encryption, some phones may not do that well. Some device manufacturers

may not do that very well…So, really it comes down to just the variety in the ecosystem. I mean there's too many variables when you're doing bring

your own device. — Security expert

Suggestions: Establish minimum qualifications for eligible BYOD

technology and operating systems Use a specialized API that will be able to request data

from the various manufacturer servers

Data security experts’ comments on BYOD

Challenges associated with variability in user practicesI don't know if there are a lot of measures you can take because you can't

install something on people's phones that prevent them from using Facebook while they're in this trial. I mean no one's going to accept that, so

you'll always risk what I refer to as data leakage. … The user error is not preventable. — Security expert

Suggestions: Select specific commercial devices that allow sponsors to

lock specific features of the device using an enterprise mobility management system Establish minimum eligibility requirements for user

privileges that participants would have to agree to, to enroll in the study

Data Attribution, Data Monitoring, Audit Trails and Data Security


Data Analysis and InterpretationMaking Data Available to the FDAChris Miller, AstraZeneca


Our Working Definition of Data AnalysisData analysis is the process of extracting, compiling, and modeling raw data to obtain information that can be applied to formulating conclusions, predicting endpoints or outcomes, or supporting decision making.

The part of data analysis that our project team is considering for this project will include only how units of objective data, commonly captured by mobile devices, should be prepared for Statistical analysis Regulatory submission

Investigators and sponsors comments on data analysis

For some – the same rules apply as in traditional data collectionFor others – we must go beyond usual approach Statisticians who typically analyze clinical trial data lack

the experience and expertise in analyzing huge datasets Big data experts must be contacted – those from other

fields who have experience in—and who are not afraid of—dealing with massive amounts of data

Approaches sponsors & investigators have used to identify meaningful data:

Conducted pilot studies to explore correlations between patient-reported outcome data and outputs from mobile devicesObtained patient and clinician feedback on information they perceived as meaningfulWorked with (bio)engineers to ensure mobile devices were developed so they can collect data the researchers (and patients and clinicians) felt were meaningfulFocused on identifying relevant time points to collect data based on clinical or standard measures, which are established prior to implementing the study

Investigators’ and sponsors’ concerns and difficulties with data analysis plans

Summarizing continuous data at the right level Enough But not too much

Determining what to do with the vast amount of data mobile devices collectUnfamiliarity with data output from mobile devicesBe aware of data fishing given the novelty of, and the vast amount of, data

collected from mobile devices

Investigators’ and sponsors’ approaches data analysis plans

Conducted small-scale feasibility trials to become familiar with data outputs and how to analyze such dataPartnered with big data experts from other fields who are experienced in analyzing huge datasets or external analytics groups that have experience managing and analyzing data collected by mobile devices Developed the analysis plan prior to trial initiation, and included variables to be assessed and how missing data will be addressed

Biostatisticians’ comments on data analysis and interpretation

Most biostatisticians described new analytical challenges with data captured with mobile devices: Size and complexity of datasets Confounding and variance in free-living data collection Missing data Determining clinical meaningfulness Using different devices and BYOD

It makes us think about there could be additional sources of variability compared to a very controlled setting and sometimes that's good,

because your inferences and conclusions can be more generalized…but it can often mean that a greater variability makes it

harder to show a treatment effect. — Biostatistician

Be mindful that, “just because I can, doesn’t mean I should or need to” Do not collect the data if they will not be used

Develop an analysis plan prior to data collection to determine what data to collect, how they will be used to inform study endpoints, and how they will be analyzed

If appropriate, consider reducing overall sample size because sampling at higher frequencies

Clearly define and document data cleaning procedures, e.g., how outlying information is defined and identified

Biostatisticians’ Suggestions: Large, complex datasets

Biostatisticians’ Suggestions: Large, complex datasets

What you usually do is Winsorize your data stream…if you have multiple measurements over time, you can use those as baseline for winsorizing. You can define outlier per patient at the cleaning time and remove things that are outliers…That’s something that’s

enabled by the fact that you have multiple measurements. You couldn’t do that with single measurement, even though that’s

outlier for the norm. — Biostatistician

Biostatisticians'’ Suggestions: Confounding and variance in free-living data

Increase the number of data points Increase the size of the participant population or the frequency of

data collection

Consider collecting Metadata in datasets: Provides researchers with the ability to monitor if and when remotely-

captured data are being uploaded as expected Identify issues that might affect how data are analyzed

Document all decisions so reviewers can understand steps taken to clean and analyze the data

Other Suggestions from BiostatisticiansMissing data: It should be simple to impute missing data given amount

of data collected by mobile technologies, and Missing data can be informative because it may be

interpreted as non-use

Clinical meaningfulness: Discuss with regulators during protocol development to

determine what information must be collected to make a regulatory claim for the effectiveness of a new therapy

BYOD Do not assume data from different devices are equivalent Record firmware and device version number

Biostatisticians’ comments on sharing data with FDA

General challenges: Presenting novel data with unknown meaningfulness Defining “source data”, given that provisioned mobile

devices are returned to the sites, wiped clean, and can’t be inspected A lack of clarity about what needs to be presented to the

FDA

Suggestions: Propose and negotiate a plan in advance for what data

and records to present to the FDA Provide FDA with information about the device model,

device firmware, and operating system


Challenge: Concerns related to free-living dataWhat types metadata does the FDA want to see?

Suggestions: Provide FDA with all the data, software, specific codes,

and all libraries used to enable them to run analyses of free-living data


Challenge: Concerns related to patient-generated data How to document attribution for patient-generated data How patient-generated data are validated The need to document the integrity of the data in the audit

trail The lack of source documents


Challenge: Concerns related to high sampling frequency data Volume of data makes it hard to visually spot check if

anything is “wrong” The shortage of storage capacity at the FDA for huge

data files May require new models to transfer the data

Suggestions: Talk to regulators in advance and agree on clinically

relevant endpoints, sampling frequency, and analysis plan• Provide FDA with all of the data, but only analyze what

was agreed upon upfront


I think, if I were going to use a new device, or a device that was going to collect information in a different, more frequent way, I'd go talk to the regulator ahead of time. To say, so what are we going to do with this new toy? Is this going to help us or is it going to get in the way? And agree on the relevant clinical

question. Are we going to take the average over every day? Are we going to look at the area under the curve, or whatever the

answer is? And then, once everybody agrees on that, well then the presentation at the end of the trial is sort of now taken care of, because everybody knows what to expect. — Biostatistician

Data Analysis and InterpretationMaking Data Available to the FDA


Lessons Learned Related to Data IssuesInterviewees’ Recommendations Regarding Data IssuesCheryl Grandinetti, FDA


What would sponsors and investigators do differently in future studies using mobile devices for data capture?

Specifically related to data issues: Encourage funding for innovative partnerships in

informatics and technology Leverage vendor expertise and don’t do everything

yourself Have all stakeholders engaged from the beginning Use smaller, more flexible sites that have experience with

technology Provide feedback to patients on how their symptoms are

doing over time so that patients are engaged in the process

Sponsors & Investigator Recommendations

“Just do it”—start by trying the mobile device in a trial and gain experience and knowledge by doing itStart by first incorporating devices into low-risk data collection opportunities, such as early-phase trials or supplemental studiesCan gain experience, experiment, and learn

without putting the trial at risk

Sponsor & Investigator Recommendations

Develop a hypothesis involving mobile device data up-front, rather than opting for a “fishing expedition”Make it easy for patients to stay engaged with the trial by simplifying the user interface and tapping into patients’ feelings of altruismReduce patient burden by adding a passive monitoring componentManage unforeseen circumstances Ramping studies up slowly and investing time and

resources upfront to assess the quality of initial data

Sponsor & Investigator Recommendations

Research teams should be non-competitive and share data about device usageIt’s beneficial to collaborate with others outside of health care and clinical trials and with those with different backgroundsBe open with regulators

Whenever possible, cut out the middleman by establishing a direct connection from the device to the server, without needing to go through multiple authorizations along the way

Ideally devices could store at least a couple days’ worth of data in the event that the user is not able to sync device, so that the data are not lost

Take a minimalist approach to data collection and patient interaction with a device Use as few devices as possible

Manufacturer and Data Manager Recommendations

Biostatistician RecommendationsInvolve a statistician beginning on Day 1 of research planning, to provide input on study design and analysis

Don’t collect data that you will not use to answer your research question

Consider the need to ensure patient privacy Be aware of the metadata being collected by the device

and use it to your advantage or don’t collect it at all

Security Expert RecommendationsEngage the services of a security expert

Weigh the costs and benefits to increased security (e.g., consider added burden on users and impact on over compliance)

Look for obvious factors that could compromise security, such as internal security policies and personnel issues

Separate personally identifiable data from research data and control access to each of these subsets

Encrypt the payload, not just the transmission channel

Data should only go on the server and not go out until the study is complete

Lessons Learned Related to Data IssuesInterviewees’ Recommendations Regarding Data Issues


www.ctti-clinicaltrials.org

THANK YOU.

presentation of evidence on data challenges · presentation of evidence on data challenges cheryl...

Documents