WG5 P02 Proposal 2014Qualification of Standard Scripts
Proposal through CSS 2014http://www.phusewiki.org/wiki/index.php?title=File:FDA_Scrips.ppt
• Anyone should be able to submit a script, according to a check list
• Categorize scripts according to complexity– Complexity:
low, medium, high, software– Output:
tabulated data, analysis data, table, figure, listing
• Metadata for script should indicate– Type of output:
tabulated data, analysis data, table, figure, listing– Study design:
parallel, crossover, etc– State of qualification:
?
Proposal through CSS 2104
• Test data– Overall project should have minimum test data (SDTM &
ADaM)– Scripts can propose new test data, must pass (Data fit?
Open CDISC?)– Share program to produce test data, never binary test data
• 2 levels of qualification to match script complexity/output– Light vs. Heavy qualification– Common elements include
• header• good programming practices• clearly declared scope of script (e.g., study design(s))• test data matches scope & passes "FDA Data Fit"
assessment (?)• documentation inputs/outputs/dependencies/usage
Proposal through CSS 2104
• Heavy qualification– Beta package includes
minimal elements for contribution• Specification & Documentation (could be in pgm
header)• Test data (Data Fit? or Open CDISC or other, as
appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests
reproduced– Draft
• Write qualification plan, Review tests for completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
– Test• Peer Review: Write qualification report, incl. log/output
from tests– Final
Proposal through CSS 2104
• Light qualification– Beta package includes
skip if >1 yr production use without ERROR– Draft
minimal elements for contribution• Specification & Documentation (could be in pgm
header)• Test data (Data Fit? or Open CDISC or other, as
appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests
reproduced• Write qualification plan, Review tests for
completeness/suitability (e.g., Branch testing – are all conditional blocks/combos tested?)
– Test• Peer Review: Write qualification report, incl. log/output
from tests– Final
Proposal through CSS 2104
Peer Review Checklist Heavy
Light
Requirement specification X ?
Documented or perhaps only documented in header X
User Guide X X
SDTM/ADaM used in input/output X X
Open CDISC validator or Data Fit used to check input/output
X X
GPP in source X X
Run according to Requirement specification X ?
Tested by qualification plan, tests & results all Peer reviewed
X ?
Tested by End users X ?
Robust without red errors in contributor's production environment
X X
Robust and used in FDA (other) scripts repository, ranked ******
X
Proposal meaningful terms in blueQualificationhttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx
• Qualification Instructions (see embedded template ð) – Certification phase of Qualification applies to new scripts
and tests– Confirmation phase applies to updates of existing scripts
• States:
Contributed, Development, Testing, Qualified
• Roles– Contributor: Anyone with appropriate skills & interests– Developer: CSS Working Group 5 volunteer familiar with
objectives**– Tester: CSS WG 05 volunteer familiar with objectives**– Environment Tester: Anyone in industry community able to
set up automatic test replication in their work environment– Reviewer: Author of white papers, designers of script
targets
** suggests a quick-start onboarding page in CSS Phusewiki
WG05-P02 Qualification plan (template).do
ProposalQualification
• Metadata for script should indicate– Whitepaper & output ID– Programming language & version (e.g., R 3.1.1)– Type of output:
tabulated data, analysis data, table, figure, listing– Study design:
parallel, crossover, etc– State of qualification:
Contributed, Development, Testing, Qualified
• Test Data requirements– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each
script– include expected results from these data for defined
tests/checks
ProposalQualification
• End-user Objectives– Clear overview of purpose and resources– Inspire confidence from first sight– Ease of use, clear messaging from first run– Reproducible results– Consistency of scripts, learning first one makes remaining
familiar– Ease of converting users to contributors
• Contributor Objectives– Standardize routine steps– Modularize routine components– Automate testing, issue identification– Centralize & consolidate information & results
ProposalQualification
• Transitions
Contributed is the original State of all scripts– to Development, checklist includes
by Developer & Reviewer• R & D confer on suitability of contribution. Suitable
starting point?[ may require analysis details, specs, from contributor ]
• D reviews components• D works with Contributor to complete minimum
components[ including Test Data and Coverage of defined tests ]
• D adds standard parameter, dependency checking• D writes Qualification instructions .docx (see template,
above)– to Testing, checklist includes
by Tester• Review Qualification instructions, consider coverage of
tests• Execute Qualification instructions• Work with Developer to complete execution
successfully
ProposalQualification
• Transitions
continued– to Qualified
by Tester & Environment Tester & Reviewer• T updates posted test outputs from
certification/confirmation• E updates local tests and executes (posting PASS/FAIL
results)• R confirms script output matches intention &
qualification process covers important elements and considerations. Also provides user (rather than technical) feedback?
• Achieve "Qualified" state when all tests in all test environments PASS (i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved
ProposalQualification
• Efforts Required– Finalize Qualification states, roles, workflow, checklists, and
templates – Design test structure in google code– Develop scripts that will allow Environment Testing– Develop general components (e.g. parameter, dependency
checking)– Identify Environment Testers based on
• Host environment• SAS or R version
– Identify opportunities to automate qualification. E.g.,• Docx format for Qualification instructions is not easily
machine readable• Environment Testers to post results back as machine
readable• Script green-light/red-light qualification matrix on Phusewiki
Doing now what patients need next