Establishing test labs for emerging technologies

Keith MainwaringIndependent Consultant

REGIONAL STANDARDIZATION FORUM FOR BRIDGING THE STANDARDIZATION GAP (BSG)Seoul, Korea (Rep. of), 24 October 2017

WTSA Resolution 44

• ITU-T to develop a programme to “assist developing countries on developing strategies in establishing national/international test laboratories for emerging technologies”

Resolution 44 - “Bridging the standardization gap between developing and developed countries”, WTSA Hammamet, 25 October – 3 November 2016

Benefits of conformity assessment

• Give confidence that products requirements are met• Benefits the user as they can make better purchase decisions• Benefits the supplier as products may more easily gain market acceptance• Helps level the industry playing field & encourages competition• WTO Technical Barriers to Trade Agreement recognises “the important

contribution that…conformity assessment schemes can make…by improving efficiency of production and facilitating the conduct of international trade”

• Provides governments and regulators with best practices

Conformity assessment scheme

• Should be related to the degree of risk associated with non-compliance with requirements considering such aspects as safety, health or environmental impacts, durability, compatibility and suitability for intended use

• Cost should not be excessive

• Should ensure transparency and neutrality

Testing

• 1st party – the supplier

• 2nd party – the user

• 3rd party – an independent body

ISO define product certification as a 3rd party conformance assessment activity

Mandatory / Voluntary• EU

– Radio Equipment Directive (RED)

• USA – FCC Title 47 Part 2 Subpart J

• China– Network Access License (NAL)– SRRC (State Radio Regulation of China) Type

Approval

• Bahamas• UAE• Brazil• Iran

• Global Certification Forum (GCF) • PTCRB (originally the PCS Type Certification

Review Board)• CTIA• WiFi Alliance• WiMax Forum• Bluetooth• Metro Ethernet Forum• Broadband Forum (BBF)• HomeGrid Forum (HGF)• IEEE Conformity Assessment Program (ICAP)• CableLabs• IPv6 Forum

EU Radio Equipment Directive (RED)

• Defines a harmonised regulatory framework for the approval of radio equipment in the European Union.

• Based on supplier declaration of conformity to basic requirements intended to ensure that the equipment is safe to use and does not cause interference with other equipment.

RED – Essential requirements

• Low Voltage Directive (LVD) (2014/35/EU) but with no voltage limit;

• Electro-Magnetic Compatibility (EMC) Directive (2014/30/EU); and

• Shall be constructed to avoid harmful interference.

RED - Additional requirements

• interwork with accessories, in particular with common chargers; • interwork via networks with other radio equipment; • be able to be connected to interfaces of the appropriate type throughout the EU; • cause no harm to the network or its functioning nor misuse network resources, thereby causing an

unacceptable degradation of service; • incorporate safeguards to ensure that the personal data and privacy of the user and of the subscriber are

protected; • support certain features ensuring protection from fraud; • support certain features ensuring access to emergency services; • support certain features in order to facilitate its use by users with a disability; and• support certain features in order to ensure that software can only be loaded into the radio equipment where

the compliance of the combination of the radio equipment and software has been demonstrated.– But these requirements are exceptional and to date the only additional essential requirements in place are for equipment

accessing emergency services (maritime, inland waterway and avalanche beacons)

RED – Conformity assessment procedures

• Internal production control

• EU-type examination that is followed by the conformity to type based on internal production control

• Full quality assurance

RED – Internal production control

Manufacturers must:• ensure and declare on their sole responsibility that the essential requirements have been

met;• document how the essential requirements have been met (including test results); • take all measures necessary so that the manufacturing process ensures compliance of the

manufactured apparatus with the essential requirements;• affix the CE marking to the equipment; and• prepare a declaration of conformity which must then be kept together with the technical

documentation for 10 years after the equipment has been placed on the market and shall be made available to the national authorities upon request.

RED –EU Type Examination

• A notified body which examines the technical design of the equipment and verifies and attests that the design of the equipment meets the essential requirements. The equipment itself is not examined in any way.

• The manufacturer’s application to a notified body shall include technical documentation on the design, manufacture and operation of the equipment in sufficient detail for the notified body to assess whether the equipment complies with the essential requirements and supporting evidence such as reports of tests that have been conducted by the manufacturer or another testing laboratory on the behalf of the manufacturer.

• The notified body prepares an evaluation report for the manufacturer and, if it considers that the requirements of the directive have been met, issues an EU-type examination certificate to the manufacturer. The manufacturer shall keep a copy of this certificate for 10 years along with the technical documentation.

RED – Full quality assurance

• Manufacturer must operate an approved quality system for design, manufacture, final product inspection and testing that has been assessed and is subject to surveillance by a notified body.

• Testing may be performed by the manufacturer or by a third party. No formal accreditation is required to carry out the tests.

• Manufacturers remain responsible in all cases for the compliance of their apparatus and declare on their sole responsibility that the requirements have been met.

USA – Code of Federal Regulations 47

• Part 2 Frequency allocations and radio treaty matters; general rules and regulations• Part 15 Radio frequency devices• Part 18 Industrial, scientific and medical equipment• Part 22 Public mobile services• Part 24 Personal communications services• Part 25 Satellite communications• Part 27 Miscellaneous wireless communication services• Part 68 Connection of terminal equipment to the telephone network• Part 90 Private land mobile radio services• Part 95 Personal radio services

FCC Procedure

Source: FCC

Depending upon capabilities the equipment may be subject to:

• Verification – testing is required but does not need to be performed by an FCC Recognized Accredited Testing Laboratory– supplier declares compliance and maintains supporting documentation– applicable, for example, to TV and FM receivers; industrial, scientific and medical equipment; and mainframe computers

• Declaration of conformity– requires testing by an FCC Recognized Accredited Testing Laboratory– supplier prepares a compliance information statement to be supplied with the product and maintains supporting documentation– applicable, for example, to personal computers and peripherals; citizens band radio receivers; and TV interface devices

• Certification – requires testing by an FCC Recognized Accredited Testing Laboratory– issued by Telecommunications Certification Body based on test results submitted by the supplier – applicable, for example, to mobile phones

Bahamas

• The Utilities Regulation and Competition Authority (URCA) has adopted the FCC system (i.e. Part 15 of the Code of Federal Regulation 47)

• An application for type approval must contain:– a completed application form for each make and model; – a signed and dated test report for each device, issued by an accredited test laboratory; – a copy of the FCC’s Grant of Equipment Authorization Certification for the device; – a detailed technical specifications of the device; – a letter of authorization, if the application is made by anyone other than the

manufacturer; and, – the type approval application fee in accordance with URCA’s Fee Schedule in force at the

time of the application, currently US$100.

United Arab Emirates

• Approval scheme based on adoption of European standards

• Based Supplier Declaration of Conformity with supporting documentation

• But wider in scope that EU Radio Equipment Directive– Requirement approval of network equipment such as UMTS Base

stations as well as mobile terminal devices and also equipment to be connected to fixed lines (e.g. digital leased lines, primary rate ISDN, analogue PSTN, X.21, X.25 and xDSL)

Brazil

• Anatel specifies approval requirements for equipment to be put into service in Brazil

• Reference is often made to international standards, in particular those of ETSI

• But all testing has to be conducted in Brazil by a laboratory accredited by the National Institute of Metrology, Quality and Technology (Inmetro)

Iran

• Requires both radio and fixed line telecommunications equipment to be type approved, including network equipment, such as routers and switches

• Also requires functional testing (such as of signalling and performance) in addition to testing to ensure safety, EMC and that radio devices do not cause interference. Supplier declaration of conformity is only accepted for functions that cannot be tested in Iran and some devices that have minimum potential of interference, such as Bluetooth and terminal Wifi devices and small and indoor antennae.

• Testing has to be performed in an ISO/IEC 17025 accredited laboratory vetted by the Communications Regulatory Authority (CRA) in Iran or by one of a few laboratories in other countries that have been approved by the CRA.

China

• Employs three type approval schemes administered with individual marking requirements: – The China Compulsory Certification (CCC) scheme ensures no harm to users and is

administered by the General Administration of Quality Supervision, Inspection and Quarantine (AQSIQ);

– The Network Access License (NAL) is applicable to telecommunications equipment– SRRC (State Radio Regulation of China) Type Approval is required for radio equipmen.

• The testing for the NAL and SRRC Type Approval must be conducted in a laboratory in China authorised by the MIIT and accredited by the CNCA (Certification and Accreditation Administration).

Voluntary certification schemes

• Global Certification Forum (GCF) • PTCRB (originally the PCS Type Certification Review Board)• CTIA• WiFi Alliance• WiMax Forum• Bluetooth• Metro Ethernet Forum• Broadband Forum (BBF)• HomeGrid Forum (HGF)• IEEE Conformity Assessment Program (ICAP)• CableLabs• IPv6 Forum

Global Certification Forum (GCF)

• Operates a certification scheme for any type of product incorporating 3GPP (GSM. 3G UMTS, LTE) and 3GPP2 (CDMA2000) mobile connectivity

• Certification process is based on a supplier’s declaration of conformity as defined in ISO/IEC 17050

• Membership consists of mobile network operators (currently 131 in number), manufacturers (81) and other industry stakeholders including testing laboratories and test system manufacturers (69 observer members).

GCF certification process

• Qualification– Manufacturer must meet ISO 9000 quality assurance standard and is regularly reviewed by a third party certification body. All testing must be

performed at an accredited ISO/IEC 17025 test facility.

• Definition • Validation • Assessment

– Manufacturer assesses whether the device has met the appropriate certification criteria. Assessment may require conformance, interoperability and field trial tests to be performed. Testing must be performed by a GCF Recognised Test Organisation (RTO). To qualify for RTO status an organization must be a member of the GCF in a category other than associate member. Four categories of RTO are defined for:

• Conformance testing;• Field trials;• Interoperability testing; and• 3GPP2 cabled IoT testing.

– Laboratories must be accredited in accordance with ISO/IEC 17025 and only perform tests within the scope of their accreditation. Commercial testing laboratories performing field trial and interoperability testing must also follow GSMA TS.11 guidelines.

• Declaration • Contest

PTCRB (originally the PCS Type Certification Review Board)

• Established by mobile network operators in USA in 1997

• Sets requirements based on standards for the certification of E-UTRA / LTE devices operating in FDD and TDD Bands and UTRA devices operating in FDD Bands II, IV and V

• Accredits 3rd party testing laboratories

• Testing laboratories must have ISO/IEC 17025 accreditation

• The PTCRB authorised test laboratories validate and manage the test cases of the certification program

PTCRB certification process

• Manufacturer submits a request for certification via the PTCRB certification database;• Manufacturer selects a PTCRB authorized test laboratory;• The selected laboratory determines which tests are to be performed depending upon the

device capabilities;• The laboratory performs the tests and submits the documentation of results to the

certification database;• The manufacturer submits documentation to the certification database;• The manufacturer pays the CTIA fee;• The CTIA ensures that all requirements have been met; and• The PTCRB issue a certification notice to the manufacturer.

CTIA managed certification programs

• LTE/CDMA Wireless Device – makes use of the Global Certification Forum certification criteria

• Battery– verifies conformance of products to the IEEE 1725 and IEEE 1625 standards for

rechargeable batteries– requires manufacturing site authorization based on on-site audit and analysis performed

by a CTIA Authorised Testing Laboratories (CATL) as well as testing by a CATL

• Battery life • Device hardware reliability • Bluetooth compatibility

CTIA Authorised Testing Laboratories (CATLs)

• ISO/IEC 17025 accredited by an ILAC member with scopes matching those of the CTIA certification testing

• Undergo a CTIA audit • Meet any program specific requirements (such as being required to attend all

face-to-face meetings of the associated certification program working group) • Sign the CATL License and Service Agreement• Pay the annual CATL administrative fee of US$1,500 for each authorized location

[“Policies and procedures for CTIA Authorised Testing Laboratories” Version 1.6, July 2016].

WiFi Alliance

• Wi-Fi CERTIFIED ™ Programs covering products based on the IEEE 802.11 standards• The WiFi Alliance has two membership categories:

– Contributor• Develop, test and certify products• Participate and vote in working groups• US$ 15,000 annual fee

– Implementer• Access finalized documents• US$ 5,000 annual fee

• Testing performed in Wi-Fi Alliance Authorized Test Laboratories– Currently 20 test labs “accredited” by the WiFi Alliance located in Spain (1), USA (4),

Germany (1), Japan (2), Korea (3), China (5) and India (1).

WiMax Forum

• WiMAX Forum Certification Programs certify the interoperability of IEEE 802.16e products:– WiMAX Forum Certified

• Designated laboratory – Bureau Veritas ADT, Taiwan, China– WiGRID Certified

• Designatated laboratory – Powertech, Canada

– WiMAX Forum Certified – AeroMACS• Designatated laboratory – Powertech, Canada

• Both of the designated certification laboratories are also WiMAX Certification Bodies (WCB).

Bluetooth SIG

• Bluetooth Product Qualification and Declaration scheme

• Uses an on-line test plan generator that a product developer can use to generate the test cases applicable to aproduct’s capabilities

• Executing the test cases generates a test evidence report which allows a test declaration to be made

• The declaration or listing process involves purchasing a declaration ID and signing a declaration of compliance

• Bluetooth SIG has two categories of membership:– Adopter

• “develop, qualify, list and sell”

• Free

– Associate• “influence, grow and save”

• participate in working groups

• membership fee: US$7,500 if company annual revenue is less than US$100 million, otherwise US$35,000

Metro Ethernet Forum

• Certification programs for Carrier Ethernet equipment, services and professional staff

• Iometrix has been nominated by the MEF to perform testing for the MEF technology and services certification programs

• Iometrix perform pre-qualification technical assessment of products and provide test plans (approved by the MEF), configuration guides and engineering support to equipment manufacturers in preparation for certification testing. Test reports are provided to the manufacturer and are only made public at their discretion.

Broadband Forum (BBF)

• Produces interoperability and conformance test specifications for technologies such as xDSL, G-PON, EPON, G.hn, TR-069 and Gfast and also arranges testing event.

• Approved Test Laboratory (ATL) program. The requirements for becoming an BBF ATL include the following:– BBF member– Actively involved in the BBF– 3rd party laboratory– ISO/IEC 17025 accredited– Subject to review by the BBF– Demonstrates knowledge of technology under test and testing specifications

• There are currently three ATLs.

Home Grid Forum (HGF)

• Certification scheme for products implementing the ITU –T G.hn

• Certification test plans covering interoperability and performance

• HGF Accredited Test Houses (ATH)

– operated by ALLION Labs in China

• HGF certification testing is only available to companies that are members of the HGF

IEEE Conformity Assessment Programs (ICAP)

• Phasor Measurement Unit (PMU)• Precision Time Protocol – Telecommunications

– IEEE 1588-2008

• Precision Time Protocol – IEEE 1588 Power Profile• Service Interoperability in Ethernet Passive Optical Networks (EPON)

– IEEE 1904.1-2013– Currently seeking test laboratories

• Interconnection of Distributed Energy Resources (DER)• Camera Phone Image Quality

– IEEE P1858

• Nuclear Power Electrical Equipment.

CableLabs

• Guidelines for the certification of cable network components to ensure the interoperability of cable devices such as cable modems, set-top boxes, digital TVs and telephony devices– Cable Labs – Certification Wave Requirements and Guidelines, October 2017

• “Certification” - customer premises equipment (CPE) • “Qualification” - non-CPE devices and network elements • Testing tools and other devices used by cable network operators may be

tested by CableLabs and “verified for interoperability”• Cable Labs have nominated Kyrio to perform testing

IPv6 Forum – IPv6 Ready Logo Program

• Conformance and interoperability testing program for IPv6 Core Protocols, DHCPv6, IPsec, SNMP and Customer Edge Routers• Also experimental test specifications and tools for 6LoWPAN, IKEv2, IMS UE, MIPv6, MLDv2, NEMO and SIP• To certify a product one can either download and execute the self-testing tools and interoperability test scenarios or submit the

product to an IPv6 Ready Logo Recognized Laboratory for testing• The Recognised Laboratories are:

– BII – Beijing Internet Institute, China;– CableLabs, USA;– IRISA – Institut de Recherche en Informatique et Systemes Aleatoires, France;– JATE – Japan Approvals Institute for Telecommunications Equipment, Japan;– TEC – Telecommunication Engineering Centre, India;– TTA – Telecommunications Technology Association, Korea; and– UNH-IOL – University of New Hampshire InterOperability Laboratory, USA.

• Test results are submitted to an examination laboratory (one of the recognized laboratories) and to the IPv6 Ready Logo Technical Group for review

ITU-T Study Groups

• Test specifications

• e.g. SG 11 Protocols and test specifications

ISO Conformity Assessment

• ISO/IEC 17025:2005/Cor1:2006 General requirements for the competence of testing and calibration laboratories

• ISO/IEC 17011:2004 General requirements for accreditation bodies accrediting conformity assessment bodies

– See “Guidelines for developing countries on establishing conformity assessment test labs in different regions” ITU, May 2012

Accreditation

• ILAC – International Laboratory Accreditation Cooperation– calibration laboratories (using ISO/IEC 17025), testing laboratories (using

ISO/IEC 17025), medical testing laboratories (using ISO 15189) and inspection bodies (using ISO/IEC 17020)

• IAF – International Accreditation Forum– global association of Conformity Assessment Accreditation Bodies– operates a worldwide Mutual Recognition Arrangement among

Accreditation Bodies (ABs) so that the results issued by Conformity Assessment Bodies (CABs) accredited by IAF AB members are accepted globally

Considerations for establishment of labs (1)

• Type of testing – mandatory type approval / voluntary?

• If mandatory – are regulations in place and specifications available?

• Which technologies are of particular interest?

• Are SDOs or industry forums working in areas of interest?

• Are new test specifications needed? Which organisation is best suited to produce them?

• Would international collaboration beneficial?

Considerations for establishment of labs (2)

• What is the business model? Is there sufficient demand for services?• What testing equipment is required? • What laboratory facilities are required? • How many staff are required and what are the staff training requirements?• Is it necessary for the laboratory to be accredited and recognised

internationally?• What are the costs (premises, equipment, utilities, staff, training, membership of

standardisation organisations or industry forums, participation in specification activities (travel and hosting meetings), accreditation fees….)?