My Past Test Experience

By Qingsong Yao

• About Our Team• Katmai NewDateTime• Sparse Column• Dev10 SQL Express Setup• SQL Server Apollo• Dac In-place Upgrade• Dac Logical Import/Export

Our Team’s Engineering Excellent

• Family is more important than work.• Encourage People to do innovation, spend time on

thinking and reading.• What we did– Team Study Group– Tracking Time Spend every day– Weekly Matrix Afternoon– Root Cause Analysis– Test Robustness

• Reduce 30,000 meta tests with 12 testshell tests• Merge many test suites into single test suites

Katmai Date and Time

• How we make decision?– Always think in term of customer’s interests.– Technique difficulty is never an excuse.

• How we make customer happy?– .Net and SQL Server Fully Aligned– Add new Built-in Function IsoWeek.– Rewrite and reviewed BOL

• How we build our feature successfully?– Build a team, then we will build the feature.

Sparse Column

• How we build the right features?– Closely work with customer (Bi-weekly meeting, provide early

build for testing).• Can we rely on partner for testing?

– No, SharePoint use our feature three years later.• How dev improve the quality?

– QO dev write adhoc test, and is our bug bash winner.• What is the effective way of finding bugs?

– Stress testing, SEI PipeLine, and TestQP• How can we do better?

– Provide better weapon (testing library).

Dev10 SQL Express Setup• What is the goal?– Make sure user have the best setup experiences

• How we make decision?– We take risk to make user experience better.

• How you do testing?– Scenario based testing plan.– Manual test and sign-off by China Vendor Team.

• Why we don’t do test automation?– Spend more time on testing– Find UI bugs.– Use the feature as real user.

SQL Apollo• How you do functional tests?

– Use Declarative Testing Idea, Focus on what, not how.• How you do scenario based testing?

– Collection all possible DW database queries and running on the lab for every build with no human involvement.

• How dev write tests?– Modify TestQP to cover all new QE code paths.

• How to reuse existing tests?– Parse and stored all queries in existing tests into central database.– Use the same way as scenario testing.

• How you do primtive performance testing?– Use Perf Team’s Perf Test Automation Framework– Measure CPU Instructions instead of CPU/Run Time– Run Perf tests on the lab every week.

DAC In-place Upgrade

• What’s your testing methodologies?– Functional; Model based; User Scenario;

Exploratory testing; Risk based testing; Manual• How we take customer into account?– We find many internal customers, and work with

them together.– We did not make everyone happy in one release.

However, we committed to continue improving our libraries.

DAC Logical Import/Export

• What is your testing strategy?– Writing ATDD tests and testing as real user:• Testing data is real• Testing target is real.

• Do you have some demo?

Why we need DogFood by oureselve?

• Feel the pain of customer.• We learn experiences from the beginning. • Understand what is real user scenario and real

requirement of our customer.• We get tests for free.• Our sales can use the experiences as case study

to sale our product .• We get the benefit of cloud. No more

deployment day any more.

Thanks• My managers Jinghao Liu and Rajendra Gokhale• My Colleagues:Xin Zhang, Xuerong Tang, June Lou, Charlie Carson, Aditya Sarpotdar; Anna Wawrzyniak;Charlie Carson;Jason Short; Preetha Subbarayalu; Shireesh Thota ;Renato Ferreira ; Alex Zhang; Frode Kristensen; Gjorgji Gjeorgjievski; Jesus DeLaTorre; Yiwei Zhang; Antony Zhong; Hongyi Wang; Hugh Qu; Min He; Peng Li;Srini Acharya; Peter Carlin; Jun Fang;Mostafa Elhemali: Dave Poole, Qingqing Zhong; Artem Oks;Jinwen Wang; Yuelong Jiang;Rami Koudsieh;