certification testing guidelines for wfa system...

Certification Testing Guidelines for WFA System Interoperability

Test Plans

Nov 05, 2013

Copyright 2013, Texas Instruments Israel Ltd. of 24 Printed specifications are not controlled documents. Updated version is on network only, verify version before using.

Contents 1 Overview ...................................................................................................................................... 4

2 Background .................................................................................................................................. 4

2.1 Definition of the DeviceUnder Test (DUT) .................................................................................................. 4

2.1.1 Define your Access Points Under Test (APUT) .................................................................................... 4

2.1.2 Define your Station Under Test (SUT) ................................................................................................ 4

3 The Testing Tools, Methodology, and Approach ........................................................................... 5

3.1 IxChariot Software(if used) ......................................................................................................................... 5

3.1.1 IxChariot scripts .................................................................................................................................. 5

3.2 Sigma Software ........................................................................................................................................... 5

3.3 Basic System Test Configuration ................................................................................................................ 6

4 How to prepare the DUT for WFA certification.............................................................................. 8

4.1 TI'srecommendation: .................................................................................................................................. 8

4.2 WFA Certification process: ......................................................................................................................... 8

4.3 DUT's preparation processfor certification: ............................................................................................... 8

4.3.1 DUT'sprerequisites for certification: .................................................................................................. 8

4.3.2 DUT's applications required for certification: .................................................................................... 8

4.3.3 DUT's at WFA lab ................................................................................................................................ 9

5 TGn SUT certification .................................................................................................................. 10

5.1 Basic information: ..................................................................................................................................... 10

5.2 Device preparation: .................................................................................................................................. 10

5.2.1 Environment preparation on DUT (should be done once, skip if you already performed): ............ 10

5.2.2 SUT role start .................................................................................................................................... 10

5.2.3 IxChariot Endpoint(should be started in case of using IxChariot as the traffic generator) .............. 11

5.3 Special instructions: .................................................................................................................................. 11

6 TGn APUT certification ............................................................................................................... 12



6.2.1 Environment preparation on DUT(should be done once, skip if you already performed): ............. 12

6.2.2 APUT role start.................................................................................................................................. 12

6.2.3 IxChariot Endpoint(should be started in case of using IxChariot as the traffic generator) .............. 12



7 WiFi Direct .................................................................................................................................. 15




7.2.2 DEVUT (P2P) role start ...................................................................................................................... 15


7.3.1 Test 6.1.11 – may fail while Sigma's run........................................................................................... 16

7.3.2 Test 6.1.12– Two issues due to which the test may fail while Sigma's run: ..................................... 16

8 WMM-PS SUT Certification ......................................................................................................... 18



8.2.1 Special commands for Power Save mode enabling .......................................................................... 18



8.3.2 SUT role start .................................................................................................................................... 19


9 WPSv2 SUT certification ............................................................................................................. 20



9.2.1 SUT role start .................................................................................................................................... 20


10 WPSv2 APUT certification ........................................................................................................... 21



10.2.1 SUT role start .................................................................................................................................... 21

10.3 Special instructions ................................................................................................................................... 21

Appendix A : ASD ........................................................................................................................... 23

Appendix B : Start Station shell script example ............................................................................... 23


1 Overview The purpose of this document is to ensure readiness of TI's customers for the WFA certification process.

2 Background The goal of the Wi-Fi Alliance (WFA) is to ensure interoperability among IEEE 802.11a/b/g/n products that

support the extended security features of Wi-Fi Protected Access 2 (WPA2) from multiple manufacturers, and to

promote this technology within both the business and consumer markets. To this end, the WFA has developed

interoperability test suite. Working in conjunction with an authorized test lab, these tests are performed on

vendor products.

The document concerns the following certifications:

WMM-PS (WMM-Powersave_testplan_v2-1.pdf)

TGn (TGnInteropTP_2.0.35.pdf)

WPS (Wi-Fi_Protected_Setup_Test_Plan_v2.0.10.pdf)

WiFi Direct (wfa_wifi_direct_interoperability_test_plan_version_1.0.9.pdf)

2.1 Definition of the DeviceUnder Test (DUT) An accurate and precise selection of the definition for you device will impact the WFA test plans to be

accomplished. As well, some device's limitations, like slow/week Host processor may have an impact on device's

performance. In such cases, an "Application Specific Device" (ASD) document will be required.

2.1.1 Define your Access Points Under Test (APUT)

Example:

The APUT is a wireless (WiFi)AP which supports both 2.4GHz and 5GHz bands with 20MHz channel width on

2.4GHz and 40MHz on 5GHz.The APUT does not support enterprise authentication,but personal only.

2.1.2 Define your Station Under Test (SUT)

Example:

The SUT is a wireless (WiFi) Station, which supports both 2.4GHz and 5GHz bands with 40MHz channel width in

5GHz.

The station can be categorized according to the following device types:

Cellular Audio Terminal Audio Network Camera PDA DLNA Multimedia Adaptor

1. VOIP Phone 2. Wi-Fi Phone 3. Converged Cell phone 4. DVD Player 5. Tablet PC


3 The Testing Tools, Methodology, and Approach This section defines tools, methodology, and approach for testing during WFA certification.

3.1 IxChariot Software(if used) IxChariot software from IXIA should be used. The IxChariot includes a large set of standard, editable

scripts,which can be used to define a particular traffic flow between two “endpoints”. Script's definition,

testconfiguration, test execution and results reporting are managed through the IxChariot “console”, which

inour case will be separate machine from the units under test.

Unless otherwise specified, default parameters are used. Usually, tests run for 1.5 minutes.

3.1.1 IxChariot scripts

The scripts, which are used during the testing, should be changed as following:

- The buffer size should be changed to 1456 bytes - The Data rate (throughput) should be changed according to the ASD document (if exists) - The packet size in the InquiryL.scr script should be changed to 1000B as alignment to Sigma's definition

3.2 Sigma Software As an alternative to IxChariot, the Sigma Automation Suite may be used. This tool suite provides configuration,

test control, traffic generation, and results analysis services. The test plan, in its entirety, can be executed in a

fully automated manner through the WFA distributed Sigma Command Scripts and the Sigma Unified CAPI

Console. Additional information is available through the test tools page on the member website.


3.3 Basic System Test Configuration The basic test configuration for infrastructure tests is depicted in the following figure and is a physical

connection diagram (note there are two logical networks to separate chariot traffic from configuration traffic)

Figure 1: Chariot Basic System Test Configuration


The stations and the IxChariot Server operate as IxChariot endpoints, with the tests configured and controlled

from the IxChariot Console. In practice the IxChariot Console and three RADIUS servers may be run on the same

machine. The Devicescape AS runs on a Linux OS machine. STAs run under Windows XP with SP2. In the above

configuration, the RADIUS servers are Devicescape, HostAPd, Radiator, and Microsoft.

When a DUT is being tested in a Station mode, only one station is typically presents, namely the Station Under

Test (STAUT). When the DUT is being tested in the AP mode, only one AP is typically present, namely the AP

Under Test (APUT).

An alternative test bed arrangement is to use the Sigma approach whereby full automation of the certification

tests can be accomplished.

Figure 2:Sigma System Test Configuration


4 How to prepare the DUT for WFA certification The following paragraph describes the testing methodology and guidelines for product's preparation.

4.1 TI'srecommendation: o Read this document before going to external certification o Run Pre-certification testing on the Customer DUT

4.2 WFA Certification process: o Register to be a WFA member - https://www.wi-fi.org/about/become-member o Select an authorized lab for testing and certifying your product - http://www.wi-

fi.org/authorized_labs.php o Choose the right testing programs (STA/AP/p2p) for your device.

The selection of the role should be according to device’s capabilities and purpose, for

example a p2p device, must run WiFi TGn for STA, WPS for STA, WiFi Direct and may skip AP

testing programs and STA-WMM-PS certification.

4.3 DUT's preparation processfor certification:

4.3.1 DUT'sprerequisites for certification:

o The DUT is fully functional;

o The performance of the DUT is similar to what TI publish in the test results;

o DUT has passed pre-certification tests. In case that there are modifications on the SW/HW from TI configuration (host processor, major code changes etc), this is a mandatory step;

o In case the Throughput performance is degraded and cannot be improved due to slower host or any other reason, a customer should prepare an ASD according to the device's performance and category – see Appendix A.

o The DUT file system should be ready to read and write. In addition, the file system must save modifications after reboot, so the customer or lab engineer will be able to download files to it with ftp/tftp/sdcard or any other procedure.

4.3.2 DUT's applications required for certification:

o Sigma – provided by TI support after signing a legal document.

Note: The Sigma must be adapted to customer’s platform environment

o IxChariot Endpoint – Download from Ixia website

o IW – Download from opensource community and compile to customer’s environment

o Busybox – Download from opensource community and compile to customer’s environment.

https://www.wi-fi.org/about/become-member

http://www.wi-fi.org/authorized_labs.php

http://www.wi-fi.org/authorized_labs.php


4.3.3 DUT's at WFA lab

o Usually you are requested to ship two DUTs to the WFA lab – make sure that those are tested devices

o Device should be sent to the lab with the operation instructions

o It is recommended to support the WFA lab during the certification process, mostly during the initial phase.


5 TGn SUT certification

5.1 Basic information: Test Plan: TGnInteropTP_2.0.35.pdf

ASDdocument number: None

Testing tool: Sigma and Chariot are available for testing

5.2 Device preparation: Required files (should be supplied by TI Support team):

ti_sigma_files.tar

5.2.1 Environment preparation on DUT (should be done once, skip if you already performed):

o Create a directory on the DUT to contain the package files (e.g./home/root)

mkdir /home/root cd /home/root

o Upload the “ti_sigma_files.tar"file to the above directory ("/home/root")

Use ftp/tftp/sdcard, for example:

tftp -g 20.1.1.10 -r ti_sigma_files.tar -l ti_sigma_files.tar

o Extract the "ti_sigma_files.tar " tar -xvf ti_sigma_files.tar

o Run the file "ti_prep_linux.sh" sh ./ti_prep_linux.sh

Note: This operation will create a folder “test_engine” with the relevant Sigma’s files.

5.2.2 SUT role start

o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./sta_start.sh – An example of this script is shown in Appendix B. sh ./start_sigma.sh tgn

o Configure current date (should be defined after each reboot) date <YYYYMMDDHHMM> (example: date 201111071225 for 07Nov2011 12:25 ).


5.2.3 IxChariot Endpoint(should be started in case of using IxChariot as the traffic generator)

o Upload the IxChariot .tar file and extract it(should be done once, skip if you already

performed before) cd /home/root/chariot tftp -g 20.1.1.10 -r endpoint.tar tar –xvf endpoint.tar

o Start IxChariot Endpoint (should be run after every reboot)

cd /home/root/chariot ./endpoint &

5.3 Special instructions: o For all tests that Chariot scripts are used, limit scripts buffer size to maximum 1456 bytes for

avoiding TCP overflowing stuck. For example, script RTP1 uses buffer size of 65535 bytes by default.

Other scripts may not specify buffer size, but use the OS “default” value that may vary

between operational systems.

o Test 5.2.14 Multicast traffic, when using Sigma. Add the multicast route to the SUT: route add –net 224.0.0.0 netmask 240.0.0.0 dev wlan0

o Test 5.2.15 Pre-authentication. Add background scan in the network configuration profile: wpa_cli -iwlan0 set_network 0 bgscan ‘”learn:5:-65:5”’

o Test 5.2.24(test 2) The maximal configurable STAUT’s clock is 203801010101 (1-Jan 2038)

o Tests 5.2.43-44, section 5.3

Tests are not supported (optional and not supported).

o General approvals from the WFA regarding the 802.11n certification if using Chariot: o Chariot Script INQUIRYL

Buffer size for 802.11n Chariot Script INQUIRYL (“Inquiry Long” ) for DT3 throughput tests should be changed from 100Bytes to 1,000Bytes, this is the default size the Sigma uses.

o Test 5.2.47 Start UDP Filesndl-HT Chariot script on TID0 to STA1 x 1 pair (Uplink 1)


6 TGn APUT certification

6.1 Basic information: Test Plan: TGnInteropTP_2.0.35.pdf

ASD document number: None

Testing tool: Sigma and Chariot are available for testing


ti_sigma_files.tar

6.2.1 Environment preparation on DUT(should be done once, skip if you already performed):



o Upload the “ti_sigma_files.tar"file to the above directory ("/home/root") Use ftp/tftp/sdcard, for example:




o Note: This operation will create a folder “test_engine” with the relevant Sigma’s files.

6.2.2 APUT role start

o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./ap_start.sh – An example of this script is shown in Appendix B. sh ./start_sigma.sh tgn

o Configure current date (should be defined after each reboot) date <YYYYMMDDHHMM> (example: date 201111071225 for 07Nov2011 12:25 ).

6.2.3 IxChariot Endpoint(should be started in case of using IxChariot as the traffic generator)

o Upload the IxChariot .tar file and extract it (should be done once, skip if you already


performed before) cd /home/root/chariot tftp -g 20.1.1.10 -r endpoint.tar tar –xvf endpoint.tar

o Start IxChariot Endpoint (should be run after every reboot)

cd /home/root/chariot ./endpoint &

6.3 Special instructions: o For all tests that Chariot scripts are used, limit scripts buffer size to maximum 1456 bytes for

avoiding TCP overflowing stuck. For example, script RTP1 uses buffer size of 65535 bytes by default.

Other scripts may not specify buffer size, but use the OS “default” value that may vary

between operational systems.

o When using chariot Console to run traffic, the routing on the APUT should be enabled: echo 1> /proc/sys/net/ipv4/ip_forward

o For dynamic IP address, configure the DHCP Server with range of IP addresses in the udhcpd.conf file.

o The Access Point is configured via hostapd text configuration file hostapd.conf. Use editor “vi” to change value of parameters:

vi /home/root/hostapd.conf.

o Security configuration options:

Open – no security:

#wep_default_key=0

wpa=0

WEP40:

wep_default_key=0

wep_key0=9876543210

WPA-PSK:

wpa=1

wpa_passphrase=12345678

wpa_key_mgmt=WPA-PSK

wpa_pairwise=TKIP


WPA2-PSK:

wpa=2



rsn_pairwise=CCMP

Mixed WPA+WPA2:

wpa=3



wpa_pairwise=TKIP

rsn_pairwise=CCMP

o Test 4.2.10Multicast traffic,when using Sigma.

Add the multicast route to the APUT:

route add –net 224.0.0.0 netmask 240.0.0.0 dev wlan1

o Tests 4.2.2,4.2.4,4.2.15

Should be executed in PSK

o Test 4.2.7

Should be excluded, according to the test plan, since this test is an enterprise test, which is

not supported (the test 4.2.8 tests the same scenario with PSK)

o Tests 4.2.11,4.2.12,4.2.17-18

Excluded.The APUT does not support enterprise security.

o Tests 4.2.19,4.2.35-37,4.2.39,4.2.41-42

Optional,not supported.

o General approvals from the WFA regarding the 802.11n certification if using Chariot: o Chariot Script INQUIRYL

Buffer size for 802.11n Chariot Script INQUIRYL (“Inquiry Long” ) for DT3 throughput tests should be changed from 100Bytes to 1,000Bytes, this is the default size the Sigma uses.


7 WiFi Direct

7.1 Basic information: Test Plan:WFA Wi-Fi Direct Interoperability Test Plan Version 1.0.8.pdf


Testing tool: Sigma


ti_sigma_files.tar










7.2.2 DEVUT (P2P) role start

o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./sta_start.sh – An example of this script is shown in Appendix B. sh ./start_sigma.sh p2p


7.3 Special instructions:

7.3.1 Test 6.1.11 – may fail while Sigma's run

The testbed sniffer has no ability to analyze data packets transmitted with SGI

o The problem is described in WFA document: "Sigma-ReleaseNotes-7.1.0", p.5, "Known issues" table:

“Sniffer seems not able to capture SGI frames in some cases”

o To avoid this problem, add the following lines in the Sigma's scrip "STA-6.1.11.txt" in

UCC folder: $STA1_control_agent!sta_set_11n,interface,$STA1_IF,greenfield,disable!DEFAULT

$STA1_control_agent!sta_set_11n,interface,$STA1_IF,sgi20,disable!DEFAULT

7.3.2 Test 6.1.12– Two issues due to which the test may fail while Sigma's run:

The testbed sniffer has no ability to analyze data packets transmitted with SGI.


“Sniffer seems not able to capture SGI frames in some cases”

o To avoid this problem, add the following lines in the Sigma's script "STA-6.1.12.txt"in

UCC folder: $STA1_control_agent!sta_set_11n,interface,$STA1_IF,greenfield,disable!DEFAULT

$STA1_control_agent!sta_set_11n,interface,$STA1_IF,sgi20,disable!DEFAULT

A race condition during the test.


“The test case 6.1.12 in test planhas some race conditions – Sigma check fails in some instances and requires tester to do manual sniffer check.”

o In case of test's failure due to sniffers analysis of PVB, EOSP or More bit during a

UPSD session, the following should be done to handle this problem:

Comment the sniffer analysis of this part of the test in the ”P2P-6.1.12.tx";

#wfa_sniffer!sniffer_control_field_check,filename,P2P_6112_6,srcmac,$P2P_IF_ADDR_DUT,MoreData_Bit,0!ID,$RET


#wfa_sniffer!sniffer_control_field_check,filename,P2P_6112_6,srcmac,$P2P_IF_ADDR_DUT,EOSP_Bit,1!ID,$RET

#wfa_sniffer!sniffer_control_field_check,filename,P2P_6112_7,srcmac,$P2P_IF_ADDR_DUT,pvb_bit,0!ID,$RET

#wfa_sniffer!sniffer_control_field_check,filename,P2P_6112_7,srcmac,$P2P_IF_ADDR_DUT,pvb_bit,1!ID,$RET

Let test to run until its completion and performthe steps required after this point in the test;

Analyze the aboveis part of the test manually after tests completion.


8 WMM-PS SUT Certification

8.1 Basic information: Test Plan: WMM_Power_Save_Testplan_V_2-1-3.pdf


Testing tool: Sigma

8.2 Device preparation:

8.2.1 Special commands for Power Save mode enabling

o Forced Power Save mode echo 1 > /debug/ieee80211/phy0/wlcore/forced_ps

o Full UPSD mode: echo 15 > /debug/ieee80211/phy0/netdev:wlan0/uapsd_queues

o Mix UPSD mode: echo 3 > /debug/ieee80211/phy0/netdev:wlan0/uapsd_queues


ti_sigma_files.tar












o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./sta_start.sh – An example of this script is shown in Appendix B. sh ./start_sigma.sh tgn

8.4 Special instructions: o Test 5.7 connectionshould be performed after capture is started


9 WPSv2 SUT certification

9.1 Basic information: Test Plan: Wi-Fi Protected Setup Test Plan_v2-0-10.pdf

ASD document number: 788

Testing tool: Manual



o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./sta_start.sh

o Seta static IP address on the wireless interface: ifconfig wlan0 X.X.X.X netmask X.X.X.X

9.3 Special instructions:

o The wpa_supplicant configuration utility “wpa_cli” includes several commands that are used for WPS tests:

o wpa_cli -iwlan0 wps_pin any - to start WPS PIN method o wpa_cli -iwlan0 wps_pbc - to start WPS Push Button Configuration

o In some test cases as in test 5.1.8 where there is an expected message and it does not

appear, run the supplicant with debug level to get the requested trace: o wpa_supplicant -d -e/home/root/entropy.bin -iwlan0 -Dnl80211 -

c/home/root/wpa_supplicant.conf -N


10 WPSv2 APUT certification

10.1 Basic information: Test Plan: Wi-Fi Protected Setup Test Plan_v2-0-10.pdf

ASD document number: 788

Testing tool: Manual



o Start the driver and the Sigma on the DUT (should be done after every reboot): cd /home/root sh ./ap_start.sh

o Seta static IP address on the wireless interface: ifconfig wlan1 X.X.X.X netmask X.X.X.X

10.3 Special instructions o For dynamic IP address configure the DHCP Server with a range of IP addresses in the

udhcpd.conf file.

o The APUT is configured via hostapd configuration file hostapd.conf. Use editor “vi” to change value of parameters:

vi /home/root/hostapd.conf.

o Security configuration options:

Open – no security:

wpa=0

WEP40:

wep_default_key=0

wep_key0=9876543210

WPA-PSK:

wpa=1



wpa_pairwise=TKIP


WPA2-PSK:

wpa=2



rsn_pairwise=CCMP

Mixed WPA+WPA2:

wpa=3



wpa_pairwise=TKIP

rsn_pairwise=CCMP

o The APUT should be restarted to get the new parameters:

sh ap_stop.sh sh ap_start.sh

o The hostapd configuration utility hostapd_cli includes several commands that may be used for WPS tests:

hostapd_cli -iwlan1 -p /home/root/wlan1 wps_pbc hostapd_cli -iwlan1 -p /home/root/wlan1 wps_pin any

o Test 4.1.13 hostapd_cli -iwlan1 -p /home/root/wlan1 wps_pin <enrolee’s UUID><PIN> 300 <enrolee’s MAC Address>

o Tests 4.2.1, 4.2.2, 4.2.4– PIN’s integrity: Incorrect PIN

hostapd_cli -iwlan1 –p/home/root/wlan1 wps_check_pin <abnormal PIN>

Correct PIN

hostapd_cli -iwlan1 -p /home/root/wlan1 wps_pin any <correct PIN>


Appendix A : ASD In the case that device'sthroughput performance is lower comparably to TI'sreported throughput performance and cannot meet WFA throughput requirements, the customer should consider the preparationof an ASD (Application Specific Document) document according to DUT's performance. The customer should get a reference for ASD from WFA, or use ASD ready from WFA website. Need to make sure that the ASD fits the device's category and type. ASD is a test plan, based on an official WFA test plan, with some mitigationto meet the DUT's limitations. The approval of ASD, by WFA,for the throughput changes only, will take 3 working days. The approval of ASD that changes the test plan or test method beyond the throughput, will take 21 working days.

Appendix B : Start Station shell script example

Below is an example for the sta_start.shscript:

cp /etc/wpa_supplicant.conf /home/root/wpa_supplicant.conf cp /etc/p2p_supplicant.conf /home/root/p2p_supplicant.conf iw phy0 interface add p2p0 type managed wpa_supplicant -B -e/home/root/entropy.bin \ -iwlan0 -Dnl80211 -c/home/root/wpa_supplicant.conf -N \ -ip2p0 -Dnl80211 -c/home/root/p2p_supplicant.conf


IMPORTANT NOTICE

Texas Instruments and its subsidiaries (TI) reserve the right to make changes to their products or to

discontinue any product or service without notice, and advise customers to obtain the latest version of

relevant information to verify, before placing orders, that information being relied on is current and complete.

All products are sold subject to the terms and conditions of sale supplied at the time of order

acknowledgment, including those pertaining to warranty, patent infringement, and limitation of liability.

TI warrants performance of its products to the specifications applicable at the time of sale in accordance with

TI’s standard warranty. Testing and other quality control techniques are utilized to the extent TI deems

necessary to support this warranty. Specific testing of all parameters of each device is not necessarily

performed, except those mandated by government requirements.

Customers are responsible for their applications using TI components.

In order to minimize risks associated with the customer’s applications, adequate design and operating

safeguards must be provided by the customer to minimize inherent or procedural hazards.

TI assumes no liability for applications assistance or customer product design. TI does not warrant or

represent that any license, either express or implied, is granted under any patent right, copyright, mask work

right, or other intellectual property right of TI covering or relating to any combination, machine, or process in

which such products or services might be or are used. TI’s publication of information regarding any third

party’s products or services does not constitute TI’s approval, license, warranty or endorsement thereof.

Reproduction of information in TI data books or data sheets is permissible only if reproduction is without

alteration and is accompanied by all associated warranties, conditions, limitations and notices.

Representation or reproduction of this information with alteration voids all warranties provided for an

associated TI product or service, is an unfair and deceptive business practice, and TI is not responsible nor

liable for any such use.

Resale of TI’s products or services with statements different from or beyond the parameters stated by TI for

that product or service voids all express and any implied warranties for the associated TI product or service,

is an unfair and deceptive business practice, and TI is not responsible nor liable for any such use.

Also see: Standard Terms and Conditions of Sale for Semiconductor Products.

www.ti.com/sc/docs/stdterms.htm

Mailing Address:

Texas Instruments

Post Office Box 655303

Dallas, Texas 75265

certification testing guidelines for wfa system...

Documents