Designing Tests for Smart TVs

Ceren Şahin Gebizli

Vestel Electronics, R&D

Test Architect, M.Sc.

ISTQB CTAL-TA

Nordic Testing Days 20177-9 June 2017, Tallinn, Estonia

Challenges

• Short time-to-market

• Limited resources

• Large code baseo 5M LOC in total

• Large modelso Thousands of states and transitions

• Importance of User

Perception

2Image ref: https://www.vestel.com.tr/

Traditional Testing

3

• Checklists based on experiences

• Lack of coverage information

• Lack of inputs for test automation

• Long time for preparing and executing

tests.

• Some critical faults are missing

Model-based Testing

Test Model

AB

C

DE

MBT Tool

A,E,B,A,EA,E,B,C,DC,D,E,B,AC,A,E,B,A

...

A,E,B,A,EA,E,B,C,DC,D,E,B,AC,A,E,B,A

...

Test Cases

• Focus on features that are mostly used

• Focus on scenarios that are mostly error-prone

• Focus on scenarios that reveal different failures

Effective test case generation;

4

System Models used for MBT

Hierarchical Markov chains defined with the MaTeLo tool (http://www.all4tec.net)

transitionprobabilities

states that can comprise sub-

models

finishstate

start state

5

Longevity Tests

6

• Detecting the mostly used

modules of TVs by getting

data from real users

• Designing a usage model

• Execution of each

generated test case on

test automation system

• Executing test cases

simulates 1 year usage of

product in 23 days.• 1 test case ~1.5 h

Image ref: https://www.vestel.com.tr/

Usage Profile

7Image ref: https://www.vestel.com.tr/

Selected Modules with

Probabilities

8

Software Module # of Visits Calculated Proability

Portal 1900 0.146

Youtube 2250 0.173

HBBTV 500 0.038

Media Browser (Video) 1750 0.134

Media Browser (Audio) 400 0.03

Media Browser (Picture) 100 0.007

PVR Ready 1000 0.076

Channel List 1750 0.134

EPG 2000 0.153

Teletext 1250 0.096

HDMI-SCART 100 0.007

Test Model

9

Test Automation

10

• Embedding test scripts to the corresponding

transitions.

• Hundreds of test cases and test scripts are

generated automatically by Model Based Test

tool.

• Test scripts are executed automatically on our

VesTA automation system.

Test Automation

11

Test Automation

12

Test Automation

13

Results and Future WorkTraditional methods:• Test cases preparation : 2d

• Test scripting of these cases : 3d

• Number of test cases: 300

• Test execution time : 15h (%100 automatic)

• Faults: 4

No idea about simulating the yearly usage

Model-based Testing:• Test design and scripting in parallel: 1d

• Number of test cases: 79

• Test execution time : 1.5h for each test case (%100 automatic)

• Faults: 10

Detection of new critical faults by simulating the 1 year usage of end-users in 23 days.

14

Conclusions

• Consumer electronics domain

• Context of an industrial case study for MBT of a

Smart TV system

• User profiles give an idea about most important

modules, features and test scenarios.

• Generated automatic tests can be run for different

software versions as regression test.

15

Thanks! Questions welcome..

16