conformity assessment: activities & systems lisa carnahan nist standards coordination office...
TRANSCRIPT
CONFORMITY ASSESSMENT: ACTIVITIES & SYSTEMS
Lisa CarnahanNIST Standards Coordination Office
Standards [email protected]
Topic Map
• Background• Definition of Conformity Assessment• Conformity Assessment: Needs and Confidence
• Discussion of conformance confidence and its relationship to risk, and cost
• General factors to consider in designing a conformity assessment system
• Actors, activities and relationships• Actors in conformity assessment• Activities in conformity assessment
• Example models for conformity assessment
Conformity Assessment“demonstration that specified requirements relating to a product,
process, system, person or body are fulfilled”
ISO/IEC 17000
So you want confidence that your purchased product or service conforms…how much confidence?
• The need for conformity assessment is primarily driven by risk • The perception of risk associated w/ non-conformity drives
the need for regulatory and market confidence• A successful CA system provides that amount of confidence at
minimal cost
Factors in CA System Design The risks associated with non-compliance should be
proportional to the rigor and independence of the CA system. System over-design will add too much cost. System under-design will result in too little confidence of
compliance. Penalties associated with non-compliance may reduce the
needed rigor and independence of the conformity assessment system.
Timely mechanisms that effectively remove non-compliant products from the market may also reduce the needed rigor and independence of the system.
Risk and Conformity Assessment--How Much Confidence is Needed?
Per
ceiv
ed R
isk
Independence and Rigor of Conformity Assessment
Supplier’s Declaration
1st party conformity assessment
Certification3rd party conformity
assessment
Testing by Accredited Labs
Qualified Products List based on
ISO Guide 65 Certification
Certification BodyAccredited toISO Guide 65
Testing by Accredited Labs
Qualified Products List based on
ISO Guide 65 Certification
Supplier Declarationof Conformity
Testing by Accredited Labs
List of Declared Products
Supplier Declarationof Conformity
Testing by Accredited Labs
Supplier Declarationof Conformity
Supplier Declaration of Conformity
Listed Products Certification
Relationship of CA Types and Confidence
Confidence
Time, $$, Resources
Conformity Assesment Actors• Consumers• Manufacturers (resellers, integrators, etc.)• Accreditation Bodies (ABs)
• Accreditation bodies for testing laboratories• Accreditation bodies for certification bodies
• Testing Laboratories• Certification Bodies• Scheme owner • ISO 9000 Registrars• Inspection BodiesDefinition: certification scheme owner: person or organization that is responsible for developing and maintaining a specific certification scheme (3.2) NOTE The certification scheme owner can be the certification body itself, a governmental authority, trade association, group of certification bodies or other. ISO/IEC CD 17067
The Parties – Who Done it?
Conformity Assessment can be conducted by:• first party – seller or manufacturer • second party – purchaser or user• third party – an independent entity that has no interest
in transactions between the 1st and 2nd parties
Components of Conformity Assessment
• Testing• Supplier’s Declaration of Conformity• Certification• Accreditation• Surveillance
Testing
Use When critical characteristics can be evaluated via measurement under specified conditions
Activities Testing
Who does it 1st, 2nd or 3rd parties
Relationship to other components of CA
• Test report may be used for evidence of conformance in supplier’s declaration
• Test report may be used for evidence of conformance in a certification system
• Test report may be used in surveillance
Related Standards ISO/IEC 17025 (testing laboratories)
Supplier’s Declaration of Conformity
Use • Risk associated with nonconformity is low• Adequate penalities (consequences) exist for
placing nonconformant product in the market• Adequate mechanisms for removing
nonnconformant product
Activities • May use testing• May use quality system approach• Supplier attests to conformity
Who does it 1st party
Relationship to other components of CA
• May use test report as evidence of conformity
Related Standards ISO/IEC 17050 Parts 1 and 2
Certification
Use Risks associated with non-conformity are moderate to high
Activities • Evaluation of evidence of conformity• Compliance decision• Attestation of conformity• Surveillance
Who does it Conducted by only a 3rd party
Relationship to other components of CA
• Certifer may be accredited• Test report used as evidence of conformance• May require accredited testing laboratories
Related Standards • ISO/IEC Guide 65 (certification bodies)
Accreditation
Use • Higher confidence for conformity assessment bodies (testing or certification)
Activities • Evaluation of competence to perform testing or certification activities within scope
• Evaluation of conformity to management & technical requirements• Attestation of conformity and comptence• Surveillance of conformity assessment bodies
Who does it 3rd party
Relationship to other components of CA
• May be required by scheme owner for testing and/or certification bodies
• May be required by certification body for testing laboratories• May be required by regulator for testing and/or certification bodies
Related Standards • ISO/IEC 17011
SurveillanceUse • To enhance confidence in ongoing conformity
• The frequency and rigor should be balanced with the cost and confidence needs. (This is typically resource intensive.)
Activities • May be performed through inspection• May be performed through testing• May be performed through audit• May be performed pre-market or post-market• These activities may be announced or unannouced• These activities may be done in conjunction with each other
Who does it • 3rd party
Relationship to other components of CA
This is a key part of a certification program or a registration system (e.g., ISO 9000 series).
Related Standards Required in ISO/IEC 17011 Required in ISO/IEC Guide 65
Conformity Assessment - ISO Guides and Standards Testing and Calibration Laboratories
Product Certification Bodies
Management Systems Registrars - Quality and Environment
Inspection Bodies
Mutual Recognition Arrangement (ILAC, APLAC, EA, IAAC)
Multilateral Recognition Arrangement (IAF)
Multilateral Recognition Arrangement (IAF)
Mutual Recognition Arrangement (ILAC and IAF, APLAC, EA, IAAC)
Accreditation Bodies (ISO/IEC 17011)
Accreditation Bodies (ISO/IEC 17011)
Accreditation Bodies (ISO/IEC 17011)
Accreditation Bodies (ISO/IEC 17011)
Accredited testing and calibration laboratories (ISO 17025)
Product certification Bodies (ISO/IEC Guide 65)
Registrars (ISO/IEC 17021)
Inspection Bodies (ISO 17020)
Samples (Test methods and sampling methods)
Products and services (Appropriate product or service standards)
Companies or organizations (ISO 9000, ISO 14000, or equivalent)
Products (Appropriate product standards)
1 2 3 4
A
B
C
D
Conformity Assessment Hierarchy
Who Watches the Watchers?
Manufacturers
Certifier(s)/Inspection Body(ies)/Laboratory(ies)
Accreditor(s)
Scheme ownersets overall
requirements of the CA system.
Supplier’s Declaration Example - IPV6 Conformity Assessment
AccreditedIPV6
Testing Labs
IPV6 Vendor
ProcurementAgency
Equipment
$
Results
SDoC *
Lab Accreditor
IPV6 Technical Specifications
$
$+
+ Assessment and accreditation
* Supplier’s Declaration of Conformity per ISO/IEC 17050 parts 1 and 2
Accredited Testing Lab examples: NIST Cryptographic Module Validation Program
20
ACBAuthorized
Certification Body*
Authorized Testing Body*
ACBAuthorized
Certification Body*
Authorized Testing Body*
Third Party Testing & Certification Example: HHS EHR Certification Program
Self developer/Vendor
Product successfully passes testing Product successfully
achieves certification
ONC reviews and posts certified
product to CHPLONC-ACBAuthorized
Certification Body*
NVLAP-Accredited Test Lab*
ANSI (as an AB)Authorized Accreditor
ONC
accredits
performs testing against Criteria
certifies tested products
approves
accredits
NIST NVLAPNational Voluntary
Laboratory Accreditation Program
*ONC-ACB and NVLAP Accredited testing bodies may be part of the same organization provided a firewall exists between the testing and certification operations
auth
oriz
es
accredits
Source: Carol Bean, HHS EHR Certification Director, NVLAP Health IT Program Workshop
Mullti-model Approach Example: FCC Participation Mutual Recognition Agreement Equipment Authorization Program
21
Verification
(Self-approval)
Verification
(Self-approval)
Certification (Approved by FCC or TCB)
Certification (Approved by FCC or TCB)
DoC(Self-approval usingan accredited testing lab)
DoC(Self-approval usingan accredited testing lab)
The type of approval is specified in the rulesfor the particular type of device
SDoC(Self-approvalDatabase by ACTA)
SDoC(Self-approvalDatabase by ACTA)
Minimum
Maximum
Source: William Hurst, P.E. Federal Communications CommissionOffice of Engineering and TechnologyLaboratory Division
Telecommunications Certification Body (TCB) = accredited third-partycertification body
Equipment Authorization Types
22
Verification SDoc DoC Certification2 Most ISM
Equipment PC’s & Peripherals PC’s & Peripherals1
TV & FM Receivers
Most Receivers Most Receivers
All Other Digital Devices
TV Interface Devices
TV Interface Devices
Pt-to-Pt Microwave
Consumer ISM Equipment
Consumer ISM Equipment
Broadcast Transmitters
Telephone Equipment
Telephone Equipment1
Aux. Broadcast Transmitters
Most transmitters
INMARSAT Equipment
Scanning Receivers
406 MHz ELT Access BPL
CATV Relay Transmitters
(1) The FCC Lab no longer certifies this equipment. However, this equipment may be certified by an accredited third-party certification body (TCB).
(2) For several products the manufacturer is given the option to use either DoC or Certification.
Source: William Hurst, P.E. Federal Communications CommissionOffice of Engineering and TechnologyLaboratory Division
NIST Conformity Assessment Guidance for Agencies
• reducing overlap and duplication and increasing efficiency• working with private sector
NTTAA directs NIST to coordinate Federal agencies in:
Federal agencies maintain their authority and responsibility to make regulatory, procurement and federal assistance decisions
NTTAA does not indicate a preference for any specific approach in conformity assessment
NIST advises Federal agencies on development of appropriate conformity assessment systems including the use of international CA standards
NIST Recommends a Risk-Based Approach to Conformity Assessment System Design
Consider risks associated with non-compliance when determining the necessary rigor of a system• Over-design can be costly; may delay products to market• Under-design reduces confidence; may prevent market
acceptance of the product
Marketplace consequences, regulatory penalties and effective recall processes may be considered in determining needed level of rigor in conformity assessment systems
FedRAMPrequirements for Provisional Authorization
ISO/IEC 17020 + FedRAMP competency requirements
ISO/IEC 17011 + technical requirements
FedRAMP Program Built on International Standards
Cloud Service Providers
Third Body Assessment Organization
(Inspection Body/ies)
Accreditor(s)
FedRAMP PMO
ISO/IEC 17011; Conformity assessment -- General requirements for accreditation bodies accrediting conformity assessment bodies ISO/IEC 17020; General criteria for the operation of various types of bodies performing inspection
Oversight &Communication
What is FedRAMP?
26
FedRAMP is a government-wide program that provides a standardized approach to security assessment, authorization, and continuous monitoring for cloud products and services.
This approach uses a “do once, use many times” framework that will save cost, time, and staff required to conduct redundant agency security assessments.
Policy on Security Authorization of Information Systems in Cloud Computing Environments
December 8, 2011 OMB Policy Memo
The Office of Citizen Services and Innovative Technology (OCSIT), within the General Services Administration (GSA), is responsible for managing FedRAMP, to provide a unified and government-wide risk management framework that addresses these problems.
27
FedRAMP’s Purpose
28
Problem:• A duplicative, inconsistent, time
consuming, costly, and inefficient cloud security risk management approach with little incentive to leverage existing Authorizations to Operate (ATOs) among agencies.
Solution: FedRAMP• Uniform risk management approach• Standard set of approved, minimum
security controls (FISMA Low and Moderate Impact)
• Consistent assessment process• Provisional Authorizatoin
FedRAMP Executive Sponsors
29
NIST Role• NIST Cloud Computing Program: build a U.S.
Government Cloud Computing Roadmap • Technical Advisor on FedRAMP• Collaborated with Federal CIO Council Security
Working Group to develop FedRAMP concept• Collaborate with GSA to develop and implement a
formal conformity assessment program
consistent independent, third-party assessments of security controls implemented by Cloud Service Providers
• Technical Experts regarding FISMA compliance– Special Publications (SP) 800-53 and 800-37– Federal Information Processing Standards
(FIPS) 199 and 200• Advise Joint Authorization Board on compliance
requirements
FedRAMP Goals
The goals of FedRAMP are to:
1. Accelerate the adoption of cloud solutions through reuse of assessments and authorizations
2. Increase confidence in security of cloud solutions
3. Achieve consistent security authorizations using a baseline set of agreed upon standards and accredited independent third party assessment organizations
4. Ensure consistent application of existing security practices Increase confidence in security assessments
5. Increase confidence in security assessments
6. Increase automation and near real-time data for continuous monitoring
31
FedRAMP Stakeholder Roles and Interaction
32
FedRAMP and the Security Assessment and Authorization Process
33
• Maintains Security Baseline including Controls & Continuous Monitoring Requirements
• Maintains Assessment Criteria
• Maintains Active Inventory of Approved Systems
Ongoing A&A (Continuous Monitoring)
Continuous Review of Risk• Oversight of the Cloud
Service Provider’s ongoing assessment and authorization activities with a focus on automation and near real time data feeds.
3 Provisional Authorization
Grant Provisional Authorization• Joint Authorization Board
reviews assessment packages and grants provisional authorizations
• Agencies issue ATOs using a risk-based framework
2Assessment
Independent Assessment• Before granting a provisional
authorizations, Cloud Service Provider systems must be assessed by an approved, Independent Third Party Assessment Organization
1
Ongoing A&A Activities Will Be Coordinated Through:
1. DHS – CyberScope Data Feeds2. DHS – US CERT Incident Response
and Threat Notifications3. FedRAMP PMO – POA&Ms
Authorizations:1. Provisional ATO - Joint
Authorization Board2. ATO – Individual Agencies
Independent Assessors to be retained from FedRAMP approved list of 3PAOs
Consistency and Quality Trustworthy & Re-useable Near Real -Time Assurance
FedRAMP Third Party Assessment Organization (3PAO) Conformity Assessment Process
34
FedRAMP requires CSPs to use Third Party Assessment Organizations (3PAOs) to independently validate and verify that they meet FedRAMP security requirements.
Benefits of leveraging a formal 3PAO
approval process:
Creates consistency in performing security assessments among 3PAOs in accordance with FISMA and NIST standards • Ensures 3PAO independence from
Cloud Service Providers in accordance with international standards
• Establishes an approved list of 3PAOs for CSPs and agencies to choose when satisfying FedRAMP requirements.
FedRAMP worked with NIST to develop a conformity assessment process to qualify 3PAOs.
This conformity assessment process will qualify 3PAOs according to two requirements:
(1) Independence and quality management in accordance with ISO standards; and(2) Technical competence through FISMA knowledge testing.
Overview of 3PAO Role• Performs Initial and Periodic Assessments of CSP
Security and Privacy Controls• Independent, Cannot Help CSP Prepare Documents!• Reviews CSP Documents for Accuracy• Develops Security Assessment Plan (SAP)• Conducts Security Testing
• Use Test Case Workbooks• Manual Tests• Automated Tests
• Develops Security Assessment Report (SAR)
35
FedRAMP Phases and TimelineA phased evolution towards sustainable operations allows for the management of risks, capture of lessons learned, and incremental rollout of capabilities
36
FY12
FY12 FY12 FY13 Q2 FY14
Pre-Launch Activities Initial Operational Capabilities (IOC)
Full Operations Sustaining Operations
FedRAMP Finalizes Requirements and Documentation in Preparation of Launch
Launch IOC with Limited Scope and Cloud Service Provider (CSP)s
Execute Full Operational Capabilities with Manual Processes
Move to Full Implementation with On-Demand Scalability
Key Activities
• Publish FedRAMP Requirements (Security Controls, Templates, Guidance)
• Publish FedRAMP Compliance Guidance for Agencies
• Accredit 3PAOs• Establish Priority Queue
• Authorize CSPs
• Update CONOPS, Continuous Monitoring Requirements and CSP Guidance
• Conduct Assessments & Authorizations
• Identify Scale Operations to Authorize More CSPs
• Implement Electronic Authorization Repository
• Scale to Steady State Operations
Outcomes • Initial List of Accredited 3PAOs
• Launch FedRAMP in to Initial Operating Capabilities
• Initial CSP Authorizations
• Established Performance Benchmark
• Multiple CSP Authorizations
• Define Business Model• Measure Benchmarks
• Authorizations Scale by Demand
• Implement Business Model
• Self-Sustaining Funding Model Covering Operations
• Privatized Accreditation Board
Gather Feedback and Incorporate Lessons Learned
IOC Launch: June 6, 2012
Additional information
Testing
Money
Product or Service
Contract
DOMESTIC AND INTERNATIONAL TRADE
CONFORMITY ASSESSMENT
ProductCertification
SupplierStandards andSpecifications
Buyer, User Standards andSpecifications
Supplier'sDeclaration
of Conformity
Inspection
InternationalMutual Recognition
Arrangements and Agreements
LaboratoryAccreditation Body
ProductCertification
Body
InspectionBody
Regulation
GovernmentRegulatory
Body
Government
Personnel
PersonnelCertification
Body
AccreditationBody
RegistrarAccreditation Body
CalibrationLaboratory
Testing Laboratory
TestMethods
Management SystemRegistrar
Management Systeme.g.
ISO 9000 - Quality
AccreditationBody
AccreditationBody
jh-20031218