edge case testing for the database professional vicky harp

Edge Case Testing for the Database ProfessionalVicky Harp

About Me – Vicky Harp

Product Manager at Idera Community Manager at Idera

http://community.idera.com Twitter: @vickyharp Email: [email protected]

Let’s talk bugs

All software has failure conditions Corruption Tampering

Most software has bugs Errors in logic Errors in execution

Let’s talk bugs

Grey areas Scalability Platform and Hardware Compatibility Language and Regional Support

If a customer experiences a problem, it’s no longer a grey area – it’s a bug

The Blame Game

Blame the developer All development done as sysadmin on a local

Developer edition server with no test data Data access layer design treated as an afterthought Cowboy code

“Cool tricks” that rely on insecure, unstable, or deprecated functionality

Continuing to use deprecated features Absurdly low fault tolerance Overuse of ad hoc SQL

The Blame Game

Blame the DBA No access to reasonable development environments No availability of QA environments QA environments used as semi-production Uneducated use of obscure configuration options,

trace flags, and data topology Database compatibility modes

Failure to maintain server as a stable code platform Mismatched tempdb collation Triggers in msdb

The Blame Game

Blame the designer Preposterous object names

Trailing spaces Special characters

Over/Under-normalization User objects in system databases Underdocumentation

The Blame Game

Blame management No time allotted for QA Bugs are expected to be fixed too quickly Insufficient personnel Insufficient hardware Unreasonable expectations

Setting Boundaries

Define expectations for what is and is not supported in your application

Document, communicate, enforce, and maintain these boundaries

Example: SQL 2005 SP2 through SQL 2008 R2 on case insensitive, US-regionalized English language instances

Enter QA

All software, internal or external, application or database, needs QA

Professionals test their code Not testing your code is unprofessional

Creating Test Cases

Most testing done in a small environment is ad hoc Did this change work?

More robust testing relies on test cases and use cases

Use Cases

Expected behavior for the application Example:

When User clicks the Cancel button: The window will close No data will be changed

When User clicks the OK button: The window will close Data will be saved If there is an exception, retry twice before returning an

error message

Test Cases

Test definition with expected output Example:

Precursors: Product is running on a supported platform but the SQL Server is offline

User clicks Cancel Expected: Window closes

User clicks OK Expected: Exception returned to user

Using Test Cases

Almost no limit to how highly documented test cases can be

If you’re starting from 0, a good start is to start making checklists Things to check before accepting a build Things to check after failing over a cluster

Test Plans

A collection of test cases is a test plan The best case scenario is to write the test

plan before the feature Realistically this often needs to be done with

an application in-situ

Types of Test Cases

Main Success Cases Edge Cases Corner Cases Others? It depends

Main Success Cases

What most people think of when they think of testing

Test case that supports a use case Example:

Use Case: User runs sales forecasting report Main Success Case: Report returns correct data

with no error messages within 30 seconds

Edge Cases

Test cases which explore the boundaries of an application

The boundary may be based on the use case or based on technology Based on use case: User clicks all the buttons on

a view Based on technology: What is the behavior of the

application on SQL 2000?

Edge Cases

Ideally edge cases should be identified at design time This is part of good programming practice,

regardless of language

DBAs should be aware of where their configuration is introducing new edge cases that may not have been accounted for

Edge Cases

Most common edge for databases is datatypes Name field backed by an nvarchar(100) column

Edge case: NULL Edge case: 0-length string Edge case: 100 character Unicode string

Primary key on a smallint column Edge case: key seeded at 0 and 32,768 rows inserted Edge case: key seeded at -32,768 and 65536 rows

inserted

Corner Cases

The intersection of two cases Not necessarily the intersection of two edges Example:

Mail merge – inserting a customer’s name into a message, then saving the text to a column

Corner cases Special character in either name or message, plus Extremely long string for either name or message

Corner Cases

SQL Server is especially prone to corner cases Shared resources are a common source of

corner cases Shared disks Shared tempdb Shared memory

Be aware that configuration decisions can introduce corner cases that a developer cannot possibly anticipate

Common Edge Cases

Data Types Language and Regionalization Date/Time Considerations Performance Usability/Maintainability Security Integration Recoverability Compatibility

Data Types

Column and variable types int, bigint, nvarchar, char, xml, etc.

Most basic of all test cases, so no excuse not to test this

Data Types

Max and min values Collation and sort order Special characters NULL values

Data Type Overflows

Overflow during aggregation select sum(intcolumn) select sum(cast(intcolumn as bigint))

Overflow or truncation with isnull Use same data types or coalesce()

Data Types

Let the engine help you out – use table constraints! Name your constraints such that any error

message returned will help you understand the problem

Language/Regionalization

Behavior when using different languages and regional settings

Particularly affects numeric and date fields, may also cause other issues

AfrikaansAlbanianAmharicArmenianArabicAssameseAzeri (Latin)Bangla (Bangladesh)BasqueBengali (India)Bosnian (Cyrillic)Bosnian (Latin)BulgarianCatalanChinese (Simplified)Chinese (Traditional)CroatianCzechDanishDariEnglishEstonianFinnishFilipino

FrenchGalicianGeorgianGermanGreekGujaratiHausa (Latin)HebrewHindiHungarianIcelandicIgboIndonesianInuktitut (Latin)IrishisiXhosaisiZuluItalianJapaneseKannadaKazakhKhmerKiSwahiliKonkani

KoreanKyrgyzLaoLatvianLithuanianLuxembourgishMalay (Brunei Darussalam)Malay (Malaysia)MalayalamMalteseMaoriMarathiMongolian (Cyrillic)NepaliNorwegian (Bokmål)Norwegian (Nynorsk)OriyaPersianPolishPortuguese (Brazil)Portuguese (Portugal)PunjabiQuechua

RomanianSerbian (Cyrillic)Serbian (Latin)Sesotho sa LeboaSetswana (South Africa)SinhalaSlovakSlovenianSpanishSwedishTamilTatarTeluguThaiTurkishTurkmenUkrainianUrduUzbek (Latin)VietnameseTing VitWelshYoruba



Make your code language and region agnostic 2012-01-07 11:00:00 AM 1000 instead of 1,000

Use “set language” set language us_english

As a DBA, seriously consider the region settings for your SQL Server Mismatch between SQL and Windows can cause

heartache

Collation Conflicts

Using different collations on the same database or on the same server can cause conflicts

Use a COLLATE statement when comparing two strings

Rule of thumb: if you are going to normalize case with LOWER or UPPER, use COLLATE

Never assume rows will come back in a specified order

Date/Time Considerations

Inaccuracies in scheduling and in date arithmetic

Can become a real nightmare on a WAN


Daylight savings time When DST ends, the clock goes from 1:59 AM -

1:00 AM SQL Agent will pause for an hour

February and Leap Day Day-of-year calculations after the 60th day


Timezones Not all timezones are on even hour boundaries

so you need to be able to account for 30 and 15 minute differences

Date Arithmetic Region settings can affect functions like

datepart() Date math functions can and do overflow. It takes

less than 25 days in milliseconds.

Performance

This is a very large topic DBAs sometimes take this more seriously

than developers Making your development environment look

and behave more like production goes a long way toward identifying these cases

Unless it is a single-user application, test with multiple users

Usability/Maintainability

How hard is it to track down bugs? Are error messages helpful? Classic problems:

Huge stored procedures TSQL embedded in other application code Obfuscated table names Encrypted stored procedures

Security

Doing all dev work with an administrator login is not realistic

Default databases for logins Compliance issues xp_cmdshell

Integration

Does this application play well with others? Do two projects that live together in prod

have separate test environments? Does maintenance overlap?

Recoverability

Are you backing up your data? Can it be restored?

Is your application mirror or cluster aware?

Compatibility

Compatibility with SQL Server and Windows versions

Non-default configurations Database compatibility modes

The master database will be set at 80 compatibility on a 2000 instance that has been upgraded to 2005

DMF functions, COLLATE statements, and JOIN statements are all subject to failure in certain compatibility modes

Know what you support and be sure those collations exist in your test environment

How To Test?

Dev and Test Environments Ad Hoc testing Automated testing Bug tracking

Dev and Test Environments

You should have a dev and/or test environment

This environment should be more challenging than production, not less

Having a difficult dev environment means that code is more likely to perform in prod

Dev and Test Environments

Example Korean language and regionalization case sensitive collation large number of databases many agent jobs long database and object names with special

characters

Ad Hoc Testing

Set aside time to “poke around” in code Enter long strings into fields Input negative numbers Change the clock and see what happens on

Leap Day Set high seed values for primary keys and

change default values for tables

Automated Testing

Can be as simple as SQL Agent jobs Developers can add test cases to their

product Many tools

If you do not have source code, just profile Build up a list of test cases that need to be

run before going to production

Bug Tracking

Post it notes won’t cut it Start with a spreadsheet Keep track of how long bugs take to fix Try to prevent bug fixes from taking over your

schedule

Wrap-Up

Define your application boundaries Define use and test cases Test your software Refine, repeat, improve

Discussion

More Information

Contact me: Forums: community.idera.com Twitter: @vickyharp Email: [email protected]

Slides will be available at http://community.idera.com

mailto:[email protected]

edge case testing for the database professional vicky harp

Documents