testing with vs2010 - a bugs life

Delivering Awesome Web Applications

A bug’s life

Testing with VS 2010


Testing the world with Visual Studio 2010

How often do you get a bug report that you can’t reproduce?

How often do you struggle to find the source of the bug in your code?

How can you streamline the testing process and make sure you don’t repeat the same bugs?

With Visual Studio 2010 & Team Foundation Server (TFS) 2010 there are a lot of new features for testers and developers that will solve these problems.

In this session you will learn how to: automate more, reproduce bugs easier, maintain your tests and configuration and discover problems sooner.

Abstract


In every day software development, we often encounter bugs that are difficult to reproduce and even more difficult to find. This problem is accentuated when it’s a reoccurring bug in the system. Visual Studio 2010 & Team Foundation Server (TFS) 2010 introduces heaps of new features that will help testers and developers in squashing these bugs once and for all.

In this session you will learn how to:

Reproduce bugs

Automate your tests to find reoccurring bugs

Maintain your tests and configuration

Discover problems sooner

Abstract


SSA @ SSW

Loves C# and .NET (Java not anymore)

Specializes in

Windows Forms ASP.NET TFS testing Automated tests

Peter Gfader


How to report bugs

What to test to find bugs?

Tests• Unit tests• Integration test

Testing, reporting bugs and fixing bugs

Test & Lab Manager (MTLM) VS2010

Missing in VS2010

Agenda


“Pass by my office then I show you”


Email

Web application

TFS web access

How do you report bugs?


We love TFS

And we love emails

TeamCompanion glues those worlds together

http://www.ssw.com.au/ssw/standards/rules/rulestobetterprojectmanagementwithtfs.aspx#TeamCompanionWorkItem

How we report bugs




Instructions to tester

Reproduce steps

Screen captures

Stacktrace (Windows Forms, WPF, ASP.NET, Silverlight)

Win7 Recorder?

How do you report bugs (Instructions)?


Reproduce steps

Screen captures

Stacktrace (Windows Forms, WPF, ASP.NET, Silverlight)

Win7 Recorder ???

URL (Page)

Version number

Browser (Firefox or IE)

http://www.ssw.com.au/ssw/standards/Support/BugReportOrEnhancement.aspx

Instructions to tester




Code

UI

DB

Deployment

Reports

Performance

Security

Whole system

User requirements fulfilled

What do we want to test?


Theory


What is the problem here?

[TestMethod]public void TestCalculation(){ var orderManager = new OrderManager(); using (var db = new NorthwindDataContext()) { var order = db.Orders.FirstOrDefault(); var actual = orderManager.GetTotalIncGST(order); var expected = order.OrderItems.Sum(oi => oi.Qty * price) * 1.1; Assert.AreEqual(actual, expected); }}

1. What are we testing?

2.


“A unit test is a fast, in-memory, consistent, automated and repeatable test of a functional unit-of-work in the system.”

“A unit of work is any functional scenario in the system that contains logic. It can be as short as a function, or it can span multiple classes and functions, and it provides internal or business value to the system under test.”

“Art of Unit Testing” Roy Osherove

What is a Unit test?


It talks to the database

It communicates across the network

It touches the file system

It can't run at the same time as any of your other unit tests

You have to do special things to your environment

Configuration files, registry, ...

Michael Feathers

What is NOT a Unit test?


Integrates different pieces

UI Business Layer Database

Not in memory

Touches

File system Registry Database Other shared resources

Integration test


Run much slower

not run in-memory

Less consistent

Dependent from external resources

Has side effects

Harder to prepare

E.g. Database setup

Integration test


Unit tests – ALWAYS

Integration tests – “allow” to be red

Why is this important?


Readable

Understand! What? Scenario? Expectation?

Maintainable

We change our application

Trustworthy

All We are bug free

Attributes of a good Unit test


Exam!

Click icon to add picture


[TestMethod]public void ActualEndTime_PlusOvertimeHardcoded_Valid(){ // Arrange EmployeeShift newShift = new EmployeeShift(); newShift.EndTime = new DateTime(2000, 01, 01, 8, 0, 0); newShift.OvertimeAfter = 1.5; // Act DateTime actual = newShift.ActualEndTime;

// Assert Assert.IsTrue(actual == new DateTime(2000, 01, 01, 9, 30, 0));}


[TestMethod]public void GetDirtyObjects_Insert2Shifts_DirtyObjectsFound(){ using (var db = new NorthwindDataContext()) { // Arrange db.CreateNewEmployeeShift( 1 // EmployeeId "01/01/2000 02:00", "01/01/2000 10:00“); db.CreateNewEmployeeShift( 2 // EmployeeId "01/01/2000 15:00", "01/01/2000 19:00”);

// Act var dirtyObjects = db.GetDirtyObjects(empShiftIdToVerify);

// Assert Assert.IsTrue(dirtyObjects.Count() > 0); }}


[TestMethod]public void TestCalculation(){ var northWindDataContextFake = MockRepository.GenerateStub<INorthwindDataContext>(); var orderitemsFake = Helper.CreateListOfOrders(); var orderFake = Helper.CreateNewOrder();

northWindDataContextFake.Stub( d => d.Orders.FirstOrDefault()).Return(orderFake); northWindDataContextFake.Stub( d => d.OrderItems).Return(orderitemsFake);

var orderManager = new OrderManager(); using (var db = northWindDataContextFake) { var order = db.Orders.FirstOrDefault(); var actual = orderManager.GetTotalIncGST(order); var expected = order.OrderItems.Sum( oi => oi.Qty * oi.Price) * 1.1;

Assert.AreEqual(actual, expected); }}


Hard to see

Naming convention

Unit test

Test single small “unit of work”

Integration test

Test integration, functionality

Unit test vs. Integration test


Test & Lab Manager (MTLM)

VS2010

TestingReporting bugsFixing bugs


Coding done

Internal testing done

Deploy to staging

“Test please” to Tester…

VS2008Scenario #1


No plan (no user stories)

Lucky to find bugs

Waste a lot of time on irrelevant things

This is called “Exploratory testing”

Tester tests


Subject: Bug found!!

“I cant quite remember what I did but I got this after a while”


Hi Ryan

The developer tests and replies...


Bug never gets fixed


Dev: We have 2 additionally reported bugs

PM: No time to fix bugs...

We must SHIP!

We call this the “Vista method”

Finally at the end of Test phase


2 weeks development BANG

Tester has no plan – Exploratory testing

Tester tests everything

“Unable to reproduce” = “Works on my machine”


Ship a buggy product

Problems


VS2010 #1 Tester uses Test Lab Manager


Add screenshot


#2 Tester files bug


1. Start VS2010

2. Open “My work items”

#3 Dev sees bug


More details...


Bug in TFS

Video.wmv

Detailed steps

IntelliTrace (drill through to Visual Studio)

Automated recording

Lots of details (you don’t need if you get intelliTrace)

Stack traceSystem info

What’s good?


#4 Dev rewinds debugging session Dev spots bugs


From recorded test we create an automated test

Don’t fix bug yet#5 Create automated test


Changes to the test

//Arrange ApplicationUnderTest appUT = ApplicationUnderTest.Launch( @”bin\x86\Debug\AvalonCalculator2.exe");

//Act // SNIP SNIP SNIP

//Assert Assert.IsTrue(appUT.Process.Responding, "App not responding");


#6 Dev fixes bug


VS2008: Developer tests manually his bug fix

VS2010: Replay coded UI test

#7 Dev verifies bug fix


Automated test is GREEN


Easy to reproduce the bug

Create functional test (coded UI test)

Automate the test from recording

Good for future changes (Regression testing)

What’s good with coded UI tests


Fall asleep during a movie

IntelliTrace



Suddenly wake up because of a surprise!

IntelliTrace



Suddenly wake up because of a surprise!

Rewind back to see what happened...

IntelliTrace


Was called “Historical Debugging”

Easy to spot the bug

Record method calls with parametersAdvanced call stack

Get intellitrace logs from your testers

Record / Trace testers execution

Use same source code to debug as the tester had to test

Get intellitrace logs from your build server

What’s good in IntelliTrace


Hi Ryan

Done – Fixed and test added

PS: you don’t need to test this again

PPS: testers are becoming less useful with 2010

Peter

#8 Email to tester


Everyone is happy


MTLMHow can I get it?


Email

Web application

TFS web access

Test & Lab Manager

Which one would you use now?


When you get a bug report!

1. Create test

2. Test is RED

3. Fix bug

4. Test is GREEN

5. Send a “Done - Test created”

When do you create a test?


Reproduce bug

Spot bug

“Works on my machine”


Generate test from bug

Dev uses “Re-play” button instead of manual test

Recap Scenario


Every time I change something I want to test (verify) that I didn’t introduce a bug

Deployment Bug fix New feature New report added

I love automated tests


Integrated web UI tests

Windows UI tests, when they do it for Web, they can do it for Windows apps (Windows Forms, WPF)

RowTests

Data driven tests with attributes next to the test

What I want in VS2012


[TestMethod][DeploymentItem(@"TestData\validConditions.txt")][DataSource("System.Data.OleDb", "Provider=Microsoft.Jet.OLEDB.4.0; DataSource=|DataDirectory|; Extended Properties='text;FMT=TabDelimited;HDR=YES'", @"validConditions#txt", DataAccessMethod.Sequential)]public void CodeParserTryParseCondition_Valid_ValidConditionsFromFile(){ // Arrange string condition = TestContext.DataRow[0].ToString(); CodeParser codeParser = new CodeParser();

// Act bool successFul = codeParser.TryParseCondition(condition);

// Assert Assert.IsTrue(successFul);}


[TestMethod][Row("Target.IsDirty")][Row("Target.IsValid“)][Row("(1 != 2)")]public void CodeParserTryParseCondition_Valid_ValidConditionsFromFile( string condition){ // Arrange CodeParser codeParser = new CodeParser();

// Act bool successFul = codeParser.TryParseCondition(condition);

// Assert Assert.IsTrue(successFul);}


Roy Osherove

Book http://www.artofunittesting.com/ Blog http://weblogs.asp.net/ROsherove/ Unit test definition

http://weblogs.asp.net/ROsherove/archive/2009/09/28/unit-test-definition-2-0.aspx

Michael Feathers

Bloghttp://www.artima.com/weblogs/index.jsp?blogger=mfeathers

What is not a unit test?http://www.artima.com/weblogs/viewpost.jsp?thread=126923

Resources

http://www.artofunittesting.com/

http://weblogs.asp.net/ROsherove/



http://www.artima.com/weblogs/index.jsp?blogger=mfeathers

http://www.artima.com/weblogs/viewpost.jsp?thread=126923


Compare Visual Studio editionshttp://www.microsoft.com/visualstudio/en-us/products/2010/default.mspx#compare

Visual Studio pricing optionshttp://arstechnica.com/microsoft/news/2009/10/visual-studio-2010-simplified-to-four-skus-beta-2-arrives.ars

Resources

http://www.microsoft.com/visualstudio/en-us/products/2010/default.mspx#compare

http://www.microsoft.com/visualstudio/en-us/products/2010/default.mspx#compare

http://blogs.msdn.com/habibh/archive/2009/10/22/intellitrace-is-not-available-why.aspx

http://blogs.msdn.com/habibh/archive/2009/10/22/intellitrace-is-not-available-why.aspx


Good blog about VS2010http://blogs.msdn.com/habibh/

How Does VS2010 Historical Debugging Work?http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/06/16/how-does-vs2010-historical-debugging-work.aspx

Resources

http://blogs.msdn.com/habibh/

http://www.wintellect.com/CS/blogs/jrobbins/archive/2009/06/16/how-does-vs2010-historical-debugging-work.aspx




[email protected]

http://peitor.blogspot.com

3 things…


Thank You!

Gateway Court Suite 10 81 - 91 Military Road Neutral Bay, Sydney NSW 2089 AUSTRALIA

ABN: 21 069 371 900

Phone: + 61 2 9953 3000 Fax: + 61 2 9953 3105

[email protected]

testing with vs2010 - a bugs life

Technology

reoccurring bugs

bugs instructions

bugs easier

bugs lifetesting

functional unit

unit of work

critical bugs founddo

unit testsyou