testing and software writer a year later
TRANSCRIPT
RAILSCLUB 2015
Testingand Software Writer a year later
Simon BagreevHead of Ruby development
Rambler&Co
RAILSCLUB 2015
whoami
status_200
semmin
Simon Bagreev
RAILSCLUB 2015
US
RAILSCLUB 2015
RailsConf 2014
RAILSCLUB 2015
TDD
● doesn’t test anything
● complicates design
● holy trinity (speed, coverage, ratio)
● adds cost, not value
RAILSCLUB 2015
Software Writer
● does the whole thing work?
● testing at the higher level (system tests)
● adequate testing
RAILSCLUB 2015
More problems
● no testing
● false confidence
● holy wars
● ignored failing tests
● endless test suite (due to units)
● “virtual” QA department
RAILSCLUB 2015
No testing
● no proof that features
work
● no way to catch
regressions
● impossible to maintain
RAILSCLUB 2015
No testing - TODO
● start with user stories
● write acceptance specs
● move to system tests
● create “visual” tests
RAILSCLUB 2015
False confidence
● many (unit) tests
● runs fast
● 100% coverage
RAILSCLUB 2015
About testing
“Testing shows the presence, not the absence of bugs.”
Edsger W. Dijkstra
NATO Software Engineering Conference, 1969
RAILSCLUB 2015
Value of unit testing
“Most software failures come from the interactions between objects rather than being a property of an object or method in isolation.”
“ Functional testing typically finds twice as many bugs as unit testing ”
James Coplien
Why Most Unit Testing is Waste
RAILSCLUB 2015
Coverageclass Event
def initialize(started_at)
@started_at = started_at
end
def today?
(Time.now.beginning_of_day..Time.now.end_of_day).cover? @started_at
end
end
RAILSCLUB 2015
100% coverageRSpec.describe Event do
subject { Event.new(start_time) }
context 'today' do
let(:start_time) { Time.now }
it 'todays event' do
expect(subject).to be_today
end
end
end
RAILSCLUB 2015
100% coverage
no expectation
RSpec.describe Event do
subject { Event.new(start_time) }
context 'today' do
let(:start_time) { Time.now }
it 'todays event' do
subject.today?
end
end
end
RAILSCLUB 2015
● mutation testing● higher level testing
● “Create system tests with good feature coverage (not code coverage)”
James Coplien
Why Most Unit Testing is Waste
False confidence TODO
RAILSCLUB 2015
Holy wars
● RSpec vs MiniTest
● Capybara vs Cucumber
● AR fixtures vs FactoryGirl
● stubs vs mocks
● PhantomJS vs Selenium
● etc...
RAILSCLUB 2015
Ignored failures
“Oh, you have to invoke Maven with this flag that turns off those tests — they are tests that no longer work!”
James Coplien
Why Most Unit Testing is Waste
RAILSCLUB 2015
“Rotten” tests
“If you want to reduce your test mass, the number one thing you should do is look at the tests that have never failed in a year and consider throwing them away.”
James Coplien
Why Most Unit Testing is Waste
RAILSCLUB 2015
rspec-rotten, FTW
“rotten” – haven’t changed status (failed/passed /pending) in extended period of time
https://github.com/rambler-digital-solutions/rspec-rotten
RAILSCLUB 2015
rspec-rotten
● uses custom rspec formatter● subscribes to rspec notifications● after every test run saves stats into json file:
{
"id": "./spec/controllers/calendar_controller_spec.rb[1:1:1]",
"state": "failed",
"date": "2015-09-21T22:04:39.063+03:00",
"location": "./spec/controllers/calendar_controller_spec.rb:13",
"description": "render the :show template"
}
RAILSCLUB 2015
rspec-rotten
Notifies you about “rotten” tests:
RAILSCLUB 2015
Endless unit testsWeinberg’s Law of Decomposition
RAILSCLUB 2015
Endless unit testsAcceptance test
RAILSCLUB 2015
Endless unit testsUnit testing
RAILSCLUB 2015
Endless unit tests
(Testing) is identifying and resolving problems that arise in the system as a whole rather than in individual components. Testing is done by groups, as an individual programmer lacks the total knowledge needed to test large sections of the system. The emphasis in testing should never be on individual modules.
Edsger W. Dijkstra
NATO Software Conference, 1969
RAILSCLUB 2015
Additionally
● hard to maintain (tests are code too, may contain bugs)
● hard to change (tends to become “binary”)● may never occur in prod
RAILSCLUB 2015
Many unit tests TODO
● mutation testing
● rspec-rotten
● “Software Writer’s” approach: “If this test fails,
what business requirement is compromised?”
RAILSCLUB 2015
“Virtual” QA dept
“My unit tests pass, QA / PM / Product Owner will take care of the rest” ... Nope
another team
RAILSCLUB 2015
“Virtual” QA TODO
● focus on acceptance specs (outside-in)
● “visual” tests for end-user regressions
RAILSCLUB 2015
rspec-visual, FTW
"visual" testing with rspec via screenshot comparison
https://github.com/rambler-digital-solutions/rspec-visual
RAILSCLUB 2015
rspec-visual step 1
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Create “visual” test
RAILSCLUB 2015
rspec-visual step 2
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Take a screenshot
RAILSCLUB 2015
rspec-visual step 3
# spec/features/visual/home_page_spec.rb
describe 'home page', type: :feature, visual: true do
it 'home_page' do |example|
visit '/'
take_screenshot(page, example)
should look_like example.description
end
end
Use look_like matcher to compare with “stable”
RAILSCLUB 2015
rspec-visual step 3
# rspec/visual/.../look_like_matcher.rb
RSpec::Matchers.define :look_like do |expected|
# ... set file paths
match do |actual|
# ... copy to stable folder if not found
system "compare -metric PAE -subimage-search -dissimilarity-threshold 1 \
#{expected_image_file_path} #{actual_image_file_path} #{diff_file_path}"
end
# ... customize failure message
end
look_like matcher
RAILSCLUB 2015
rspec-visual result
!=
RAILSCLUB 2015
rspec-visual diff
RAILSCLUB 2015
rspec-visual result
!=
RAILSCLUB 2015
rspec-visual diff
RAILSCLUB 2015
rspec-visual gotchas
● dependencies (Capybara, Poltergeist + PhantomJS, imagemagick)
● only supports Poltergeist● dynamic data● speed
RAILSCLUB 2015
To conclude
As a Software Writer:
● focus on what really matters● throw away tests that have no business
value● do “visual” testing● use Thought-Driven Development
RAILSCLUB 2015
Contacts
В группе компаний Rambler&Co всегда есть открытые вакансии для тех, кто
хочет профессионально расти и развиваться, занимаясь тем, что по-
настоящему нравится
www.rambler-co.ru/jobs
RAILSCLUB 2015
Links
RailsConf 2014 Keynote: Writing Software by DHH: http://bit.ly/1LwrENX
Why Most Unit Testing is Waste, James Coplien: http://bit.ly/1kBMor1
Why Most Unit Testing is Waste part 2, James Coplien: http://bit.ly/1OuzrOc
NATO Software Engineering Conference 1969: http://bit.ly/1iwANKT
rspec-visual: https://github.com/rambler-digital-solutions/rspec-visual
rspec-rotten: https://github.com/rambler-digital-solutions/rspec-rotten
RAILSCLUB 2015
Thanks
Questions?