why test automation guys should have hacker mindset?!
DESCRIPTION
Presentation from meetup occurred in Chicago, May 24TRANSCRIPT
Why Test automation guys should have hacker
mindset?!
By Peter Kartashov @2011 http://at4qa.blogspot.com
a talk about being test automation engineer
• lazy one who got tired doing testing manually• crazy about automating everything• tester who wants to know and manipulate apps inside of that interfaces• looks for new opportunities to break apps• relies on testability
A few silly observations about Good test automation engineer:
By Peter Kartashov @2011 http://at4qa.blogspot.com
• lazy one who got tired doing testing manually• crazy about automating everything• tester who wants to know and manipulate apps inside of that interfaces• looks for new opportunities to break apps• relies on testability
Great test automation engineer is same as Good one
+ plus +• experience, • effectiveness• flexibility
By Peter Kartashov @2011 http://at4qa.blogspot.com
Challenges (technical)
1. You will never have stable app and interfaces2. You will have different ways to automate (interfaces,
approaches, frameworks)3. You will have to either reuse or grow new own framework4. You will never have smooth path5. Be ready to use these words in your vocabulary:
“workaround”, “hack”, “backdoor”, “crutch”…
If you have neither these symptoms – There is a good chance you are doing either wrong or trivial test automation
By Peter Kartashov @2011 http://at4qa.blogspot.com
Challenges (technical)
Technically making enterprise wide automation implies thinking over solid architectural solution that involves:• Infrastructure• Integration with other systems and practices (CI, test
management, version control, etc)• Automation framework• Persistent storage test data, logs and historical information
By Peter Kartashov @2011 http://at4qa.blogspot.com
Challenges (organizational)
1. Struggle to find out right scope, resources and solution (notorious ROI)
2. You will have to prove that you are useful (not money wasters)3. Some people will consider you as neither tester nor developer4. If a group of automation engineers – a right person should
manage and drive this team (test automation lead with background). If you looked after by test manager or manual test engineer only – personally I don’t envy you.
If you have neither these symptoms – There is a good chance you are doing either wrong automation or you are lucky
By Peter Kartashov @2011 http://at4qa.blogspot.com
Testability
Testability is a property allowing you to manipulate with your software project by calling its interfaces.
Some most popular Issues: Some interfaces will be open, some will be closed. Some interfaces will be stable, some will be freaky to any code commit It is normal to have a choice on which level to automate your app. As functional and E2E testing is pretty complex, you will likely utilize different levels of testing. Another challenge related to testability is cross-platform (cross-browser) implementation
By Peter Kartashov @2011 http://at4qa.blogspot.com
Testability: examples of mixed implementationAPI level testing is utilized to manipulate system components internally, transmit data and trigger events. Verification of results is running on API-level, then verification on GUI as double check
Integration between 2 components is verifying in module integration level (API). Integration with GUI layer is checking in GUI as checkpoints.
Change data in database (SQL scripting); checking results in API as checkpoint.
Vice versa: change data using UI, validate on database side (SQL querying). Additional validation can be incorporated on API level
Check Web service I/O, check queue object, check POST/GET Header and Body
By Peter Kartashov @2011 http://at4qa.blogspot.com
Hacks, workarounds and tools
You will never have one tool or/and language covering all your needs.
So that you have to be creative and all-round to
resolve blockers quickly
By Peter Kartashov @2011 http://at4qa.blogspot.com
Examples: Reflection
By Peter Kartashov @2011 http://at4qa.blogspot.com
Reflection allows you to work with native code of running application in runtime, i.e. you can intercept and evaluate currently running code on machine (like attaching to currently running COM object)Java and .Net platforms allows to enable reflection so that testing framework “understands” running code of compiled application.
import java.lang.reflect.*; public class DumpMethods { public static void main(String args[]) { try { Class c = Class.forName(args[0]); Method m[] = c.getDeclaredMethods(); for (int i = 0; i < m.length; i++) System.out.println(m[i].toString()); } catch (Throwable e) { System.err.println(e); } } }
*Don’t use reflection in production code – main concerns are performance degradation and security
Examples: OS manipulation
By Peter Kartashov @2011 http://at4qa.blogspot.com
Use internal Scripting capabilities: WMI, PowerShell, bash, Win Registry
E.g. VB Script utilizing WMI object to monitor CPU usage on hosts:
Examples: using tools and libs
By Peter Kartashov @2011 http://at4qa.blogspot.com
Firebug to watch DOM rendering, scripting errors and memory leaking
… and evaluate scripts. I recommend using Java script to traverse, manipulate and hack over Web
Examples: using tools and libs
By Peter Kartashov @2011 http://at4qa.blogspot.com
To simplify your Java Script injection – I recommend utilizing jQuery
Examples: using tools and libs
By Peter Kartashov @2011 http://at4qa.blogspot.com
WinDbg to get memory dump, call critical section, emulate stack overflaw…
Core skills of good test automation engineer
By Peter Kartashov @2011 http://at4qa.blogspot.com
+ Knowledge of internal mechanisms of interacted objects (e.g. DOM model, COM object, MFC, Pure Java). Basically you have to be able to invent efficient object recognition model with high robustness on intermediate changes and proper control identification (avoiding concurrent identification) + Knowledge of test automation tool(s) itself + Familiarity with programming language and scripting + Knowledge of development practices, principles (like single responsibility Open/Closed, reusability, OOP, procedural approach…) and common design patterns (singleton, decorator, factory, wrapper, etc)+ Regular expressions. That’s a very powerful tool for us. Must to know concepts and how to use. Ideally you are ready to write any expression at fly. + Knowledge of performance testing, modeling and identification of bottlenecks (for load testing)+ Knowledge of theory and practices on relational databases.+ Good knowledge of one on-demand RDBMS, querying and performance analysis+ XML – based technologies and service model (XML, XSLT, XSD, DTD, WSDL)+ Experience in version control systems+ Experience in tuning application/web/DB servers+ Broad knowledge of operation systems (server and consumers solutions) as advanced user or as administrator.
By Peter Kartashov @2011 http://at4qa.blogspot.com
In general, Test automation engineer should quickly solve technical challenges utilizing right tools and libraries.
Finally, the effort on building automation has to be empowered by testing aptitude and passion to break software with keen tests.
Aren’t all these slides about Hackers?
Thank you!
Questions?
By Peter Kartashov @2011 http://at4qa.blogspot.com