OCAP-RI Vendor Forum

August 10, 2011

Presenters: NDS and CableLabs Dev Teams

Location: CableLabs Summer Conference – Keystone, CO

Agenda

• Introductions• Ground Rules• Release Roadmap• NDS Topics• CableLabs Topics• Miscellaneous Topics

Production / Debug Builds

• CableLabs occasionally tests releases in both debug and “release” build configurations

• In “release”, compiler optimizations are enabled and much logging is disabled

• Changes in timing can exponse thread synchronization bugs in the stack

Harder to debug

Extension-based Builds

• Stack can be configured with OCAP base and any combination of OCAP extensions

DVR, HomeNetworking, FrontPanel, etc

• Application developers must be careful to structure their apps so as not to trigger accidental class loading

• Class file verification can trip you up JVM spec (http://java.sun.com/docs/books/jvms/second_edition/html/ClassFile.doc.html)

Extension-based app examplepublic class App{ public void doSomething() { MyInterface myIf; if (System.getProperty(“ocap.api.option.dvr”) != null) { myIf = new MyIntefaceDVRImpl(); } else { myIf = new MyIntefaceBaseImpl(); } }}

Testing Overview

• QA - Bug tracking• Trunk and 1.1.4 Branch• Unit Testing

CTP attended/unattended, CTP smoke, atelite, DLNA & UPnP CTT, JUnit

• Integration Testing Integration Xlets, AutoXlet, CATS TDK

• MSO Guides – ODN, Trio, Barcelona • QA: TestLink• We want your tests - Appeal for your tests - Please

Bug Tracking

Private Database - OCORI•A mix of publicly visible issues, and some are not•To protect confidential information, exposed case by case•Mirrors the issues in the private data

Public database - IT•For these issues, all information in the private databases is also in the public database

Trunk and Branch Testing Schedule

Trunk and 1.1.4 vendor branch •Daily Guide smoke tests (trunk)•Weekly Guide smoke tests (1.1.4 branch)•Weekly CTP test runs•Full regression run prior to releases

Internal Branch Testing•CTP tests are run prior to merging back into the trunk

CTP Unattended Tests Test Setup• 5600+ unattended CTP tests.• 1 ATE dedicated to testing the RI.• Tests run against Windows and Linux platforms.• Run against all test suites requires 3 days.

Test Schedule• Test all suites against the trunk (frequency: once per week).• Test all suites against the 1.1.4 branch (frequency: once per week).• Test all suites or a subset of tests against developers branches before merge to trunk (frequency: as needed).• Test subset of suites against separable builds: Base, DVR only, HN+DVR, HN only (frequency: when ATE available).

Attended CTP Tests

•400+ CTP tests require operator interaction•Attended CTP tests can be roughly divided into two groups:

•Tests which demonstrate interactivity.•Tests which require visual inspections.

•Automated attended CTP testing addresses the tests which require only visual inspection.

•Test run is automated.•Screenshot and verification question are captured and saved for each test.•After test run completes, operator analyzes saved screenshots and verification questions.

•Goal of automated attended CTP testingIncrease scope of tests run for each release cycle while using QA resources more efficiently.

Automated Attended CTP Testing utilizes programs developed by the RI team to capture test results, incorporate results into a spreadsheet and pull aggregated results in to a GUI for analysis by QA.

DLNA CTT v 1.5.57.03

•Unit tests which check the conformance of the Ri against the DLNA specification•Run the DMS (server) tests. Player tests are restricted by PC features supported by the RI.•267 unit level tests are run.

UPnP CTT 2.0.18.5079

•Unit tests which check the conformance of the Ri against the UPnP specifications•110 unit level tests are run.

JUnit Tests

•Modified version of the JUnit test runner•Tests are available as part of the distribution•Executed by development engineers as part of their workflow.

Integration Tests

• $OCAPROOT/apps/qa/org/cablelabs/xlets• Each test application should have a readme.txt in its

directory.• Currently undergoing clean-up and rework

Latest status is found in

$OCAPROOT/apps/qa/docs/IntegrationTestInfo.doc• New test applications will be added going forward for

integration testing of new features.

List of tests run with every RI release:Test Xlet Number of tests Autoxlet?

DvrTest 85 YesFileAccessPermissionTest - Create 60 YesFileAccessPermissionTest - Read same org 4 YesFileAccessPermissionTest - Write same org 4 YesFileAccessPermissionTest - Read diff. org 4 YesFileAccessPermissionTest - Write diff. org 3 YesFileAccessPermissionTest - Remove diff. org 4 YesFileAccessPermissionTest - Remove same org 4 YesPermissionTest - Unsigned 118 NoPermissionTest - Signed withOUT PRF 119 YesPermissionTest - Signed with PRF 112 YesPermissionTest - Dually Signed withOUT PRF 117 YesPermissionTest - SuperHost (monitor app) 72 YesPropertiesTest - Unsigned 56 NoPropertiesTest - Signed 56 YesPropertiesTest - SuperHost (Monitor app) 56 YesTuneTest 5 YesPowerKeyTest 6 YesHSMScene 10 YesUserPreference 33 YesFocusTest 6 YesFrontPanelResourceTest 7 YesFrontPanel 138 YesTimeTest 1 NoAppVersionTest 7 NoDvrExerciser - HN 4 NoEAS 8 No

Integration Test Automation

• Autoxlet – a framework that automates xlet launching and dispatch of remote control events.

XletDriver – AUTOSTART application that loads xlets and sends remote control events based on the XML script file.

• To automate a xlet, the xlet should implement the Driveable interface.

DispatchEvent(…) provides a method for receiving and interpreting key events.

• Uses IXC communication for logging and result reporting.

Deterministic xlet results are provided using JUnit-like asserts.

• $OCAPROOT/apps/qa/Autoxlet/

MSO Guide Tests

Guides•ODN•Barcelona•Trio

Run a subset of the tests used by the MSOs for their own testing

No. Name of Test Description Pass/ Fail

Comments

1 Measure time Time taken to load and acquire guide data

1 minute 50 seconds sec; guide data was available immediately

2 Tuning without the guide

Direct tune to ch 114. Tune up and down 15 channels. Direct tune to 132.

Fail Chan up/down skips channels. OCORI-3176

3 Front panel time

Compare time on the emulator with current time.

Pass

4 Time shift buffer

Test trick-play functionality at various speeds.

Pass

5 Tuning with guide

Click on guide button use the arrow keys and select key to tune to the desired channel

Pass

6 Record one channel using

guide

From guide, select channel. From guide, select record. Verify and delete recording.

Pass

7 Record Multiple Programs

(two tuner test)

Record channel A. Navigate to channel B and record. Wait three minutes. Attempt to navigate to channel C. Verify message box is displayed. Select stop recording from the message box. Navigate to channel A and stop recording. Verify recordings on DVR menu. Verify trickplay functionality. Delete recordings.

Pass

8 Parental Lock From the menu navigate to the parental settings, option. Lock movies. Attempt to tune to ch. 114. Verify that the parental lock code must be entered before channel can be tuned.

Fail Movie lock doesn't work. correct PIN not accepted. (OCORI-2939)

Home Networking InterOp Testing

List scenarios run at the May Interop•Discover and playback RI recordings on your STB•RI discover and playback recordings from your STB•Trick modes•Delete recordings

TDK and CATS

•Will run the same tests as NDS talked about earlier•At this moment, still working through installation issues•Our goal is to run all the TDK tests prior to each RI release

TestLink

•Web based test case management tool•Archive of test results•Repository for test procedures

TestLink Results Entry

04/22/2323

Testing Overview - Summary

• QA - Bug tracking• Trunk and 1.1.4 Branch• Unit Testing

CTP attended/unattended, CTP smoke, atelite, DLNA & UPnP CTT, JUnit

• Integration Testing Integration Xlets, AutoXlet, CATS TDK

• MSO Guides – ODN, Trio, Barcelona • QA: TestLink• We want your tests - Appeal for your tests - Please

Questions and Suggestions

•RI QA is a small team with limited resources•Do you have tests to contribute?•Do you have suggestions for new tests?