testistanbul conferences 2012 test automation at the scale of google baran ozgul sr. test engineer -...
TRANSCRIPT
![Page 1: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/1.jpg)
TestIstanbul Conferences 2012
Test Automation at the Scale of Google
Baran Ozgul
Sr. Test Engineer - Google AdSense
Olivier Gaillard
Sr. Software Engineer in Test - Google AdSense
![Page 2: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/2.jpg)
Guessing Game
How many test cases are executed per year at Google?
100 x 10^9
~14 000 years of test execution per year
![Page 3: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/3.jpg)
TestIstanbul Conferences 2012
Agenda
• Case study: AdSense Frontend
• Google's Innovation Culture
• The Google Scale
• Developer Engagement
• Testing Infrastructure
• Web Testing at Google
• Key learnings
• Q&A
![Page 4: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/4.jpg)
GOOGLE'S INNOVATION CULTURE
![Page 5: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/5.jpg)
TestIstanbul Conferences 2012
Google's Innovation Culture
"Launch early, launch often"
"The only true failure is not attempting the audacious."
• Take risks
• Launch and iterate
![Page 6: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/6.jpg)
TestIstanbul Conferences 2012
Launch and iterate
Google+: 100 new features in first 90 days
Search: Idea-to-user in less than 48 hours
Chrome: A new version every 6 weeks
![Page 7: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/7.jpg)
THE GOOGLE SCALE
![Page 8: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/8.jpg)
TestIstanbul Conferences 2012
The "Google Scale"
• 60 offices, 30+ countries
• 1 giant code base everything built at latest check-in
• 11K people in R&D
• 10s of thousands of changed code lines / day
• 10s of products, 100s of feature teams
• Weekly to sub-daily release cycles
How is this possible?
![Page 9: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/9.jpg)
TestIstanbul Conferences 2012
Paradigm Shifts!
• Developer owns code
• Test engineering = quality assurance
• Reliance on people to scale
• Test to prevent bugs
• Spread thinly
• Developer owns code + test • Test engineering =
Productivity enablers
• Reliance on common test infrastructure to scale
• Capacity to fix bugs fast
• Staff high impact projects
![Page 10: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/10.jpg)
DEVELOPER ENGAGEMENT
![Page 11: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/11.jpg)
TestIstanbul Conferences 2012
Developer engagement
![Page 12: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/12.jpg)
TestIstanbul Conferences 2012
Developer engagement
![Page 13: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/13.jpg)
TestIstanbul Conferences 2012
Developer engagement
![Page 14: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/14.jpg)
TestIstanbul Conferences 2012
Test Engineering
"Engineering productivity"
• Test leadership• Process automation• Code health signals• Test evangelism
![Page 15: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/15.jpg)
TestIstanbul Conferences 2012
Test Leadership
• Make existing code testable
• Implement the first N tests, lead by example
• Use data to define testing priorities
• Remove testing bottlenecks
• Make it faster to write and run the tests
![Page 16: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/16.jpg)
TestIstanbul Conferences 2012
Process automation
• Continuous integration
• Automated releases
• Release tracking
• Production monitoring
• Automated code quality checks
![Page 17: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/17.jpg)
TestIstanbul Conferences 2012
Code health signals
Latency (ms)
Code coverage
If you don't measure it, you can't control it.
![Page 18: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/18.jpg)
TestIstanbul Conferences 2012
Metrics: Incremental Code Coverage
![Page 19: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/19.jpg)
TestIstanbul Conferences 2012
Test Evangelism
• Codelabs & Techtalks ("yt: code health talks")
• Testing on the Toilet (TotT)
• Internal & external test automation events (GTAC)
• Testing intergrouplet
• Test certification
Make testing part of the development culture
![Page 20: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/20.jpg)
TestIstanbul Conferences 2012
Test Certified Programme
• An indication test automation maturity• Show value every stage
• “Test Certified” Levels
1: Setup tools – get familiar with testing
2: Write tests for all new code
3: Get all code under test
4: Improve test coverage
5: Invest in testable code
![Page 21: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/21.jpg)
ADSENSE FRONTEND
Case Study
![Page 22: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/22.jpg)
TestIstanbul Conferences 2012
Case Study: AdSense Frontend
Before
• Time until next release: months
• Build broken for weeks
• 5+ release candidates
• Reliance on manual testing for release confidence
• Near absent test automation coverage
• One man to release them all
![Page 23: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/23.jpg)
TestIstanbul Conferences 2012
Case Study: AdSense Frontend
Now
• Developers write tests for their features
• Affected tests run at every code change
• Manual testing: only exploratory
• Build gets fixed in hours
• weekly releases
• <2 release candidates per release
• 6 auto-staging releases per day
• Push-button releases
![Page 24: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/24.jpg)
TEST INFRASTRUCTURE
![Page 25: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/25.jpg)
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
minutes hours days
code review
Push to production
![Page 26: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/26.jpg)
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
minutes hours days
Push to production
code review
![Page 27: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/27.jpg)
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
code submittedimpacted tests run
minutes hours days
Push to productionbuilcops/build breakers
notified
code review
![Page 28: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/28.jpg)
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
code submittedimpacted tests run
minutes hours days
Push to productionbuilcops/build breakers
notified
push to production dry runcode review
![Page 29: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/29.jpg)
TestIstanbul Conferences 2012
Test enablers
• test engineers worked on optimizing this process
• makes it easier to create/maintain testso developers are more likely to create more tests
• lots of these steps are using a common infrastructureo search for "continuous integration google"
![Page 30: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/30.jpg)
WEB TESTING
![Page 31: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/31.jpg)
TestIstanbul Conferences 2012
Browser Automation
Selenium 2.0/WebDriver
• Core browser automation platform (others exist)
• A collection of language specific bindings to control a browser
• Emulate true user actions
• Support for major browser manufacturers
![Page 32: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/32.jpg)
TestIstanbul Conferences 2012
Browser Automation
WebDriver driver = new FirefoxDriver(); driver.get("http://www.google.com"); WebElement element = driver.findElement(By.name("q")); element.sendKeys("Cheese!"); element.submit(); assertTrue(driver.getTitle().contains("Cheese"));
How does this scale?
![Page 33: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/33.jpg)
TestIstanbul Conferences 2012
Browser automation
Test Runner
Shard 1Browser 1
Test Browser
Shard 1Browser 2
Test Browser
Shard NBrowser M
Test Browser
Shard NBrowser1
Test BrowserOutput
Aggregator
![Page 34: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/34.jpg)
TestIstanbul Conferences 2012
Browser automation (AdSense Frontend)
Test Runner
Shard 1Chrome
Test(s) Browser(s)
Server
Database
...
Shard 1Firefox
Test(s) Browser(s)
Server
Database
...
Shard 1InternetExplorer
Test(s) Browser(s)
Server
Database
...
![Page 35: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/35.jpg)
TestIstanbul Conferences 2012
Browser Automation: Test enablers
• Test engineers worked on making it easy to write/maintain browser testso developers are more likely to more tests
• Browser tests run at every affecting code changesome costly tests are throttled
![Page 36: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/36.jpg)
KEY LEARNINGS
![Page 37: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/37.jpg)
TestIstanbul Conferences 2012
Key Learnings
• Push testing up-streamo Early developer testing brings biggest winso Position ourselves as productivity enablerso Make/Hire developers passionate about testing
• Invest in a great testing infrastructureo Reduce the cost of writing tests to zeroo Automate as much as possible
![Page 38: TestIstanbul Conferences 2012 Test Automation at the Scale of Google Baran Ozgul Sr. Test Engineer - Google AdSense Olivier Gaillard Sr. Software Engineer](https://reader037.vdocuments.net/reader037/viewer/2022102923/5516b69b550346f0208b54a9/html5/thumbnails/38.jpg)
TestIstanbul Conferences 2012
Thanks
"Google Testing Blog""Google Test Automation Conference (GTAC)""Google Engineering Tools""WebDriver" or "Selenium"
"GTAC""Clean code talks""Web Testing at Google GTAC""Build Software at Google Scale GTAC"