Test Automation Framework Development Challenges
Test
automation framework development is a multi-stage process. And passing
through each stage involves multiple challenges to be addressed. Key
challenges to be addressed are detailed below:
Clear vision
Clear
vision of what needs to be achieved out of this automation must be
defined and documented. To formulate the clear vision, the following
needs to be identified:
- Testing Model - To be adapted (Enterprise, Product , Project)
- Types of testing under the scope based on the testing model
- Prioritizing
the automation activity based on product/application/module (for
example: Conducting functional testing and then performance testing)
- Identify the areas that need to be automated (for example: Registration, Order processing)
- Challenging
validations that need to be taken into account (for example:
Communicating from windows to Linux machine and executing the tests)
- Critical and mandatory functionalities
Tool Identification and Recommendation
Tool identification process is a crucial one, as it involves critical factors to be considered, which include:
- Creating
a standard tool evaluation checklist which needs to be created by
considering types of testing, teams involved, licensing cost of the
tool, maintenance cost, training and support, tool's extensibility,
tool's performance & stability etc.
- Testing requirements which may include types of testing such as Functional, Performance, and Web Service etc.
- We may need to acquire multiple tools to perform different types of testing on the product/project line.
Framework Design - Appropriately Pick and Choose
Framework design involves identifying requirements from multiple areas. At a high level, this includes (not limited to):
- Identification of necessary utility/components related to application functionalities
- Types of input data store to be communicated for data flow
- Communication between the utilities/components (for example: data check-point components communicating to the logger etc)
- Communication
between the systems and utility/component development related to the
same. (for example: communicating from windows to Linux environment)
- Tool
extending capabilities - Developing utilities/components for the
validations not supported by the identified test automation tool, if
any.
Approach to Framework Development
Pre-requisites and Assumptions
- User is aware of the basics of test automation
- User has planned the test automation activity, by considering the scope, objectives, requirements, schedule and budget
- User
has gone through the process of "Test Automation Tool Build or Buy" and
has taken a decision of buying a tool or getting a open-source tool
Identify Testing Scope
Each
organization believes in its own requirements in software test
automation. Considering the organization's requirements, test automation
activities can be performed with three different scopes:
- Enterprise-oriented - Test automation to support different product lines and projects in the organization.
- Product-oriented - Test automation activities focused towards specific product line of applications
- Project-oriented - Test automation effort focused towards specific project and its test process.
Identify Testing Types
Subsequent
to the testing scope identification, product/application/modules under
the testing scope need to be identified. Based on the
product/application/module requirement, types of testing that need to be
performed are identified. For example,
em Scenario: For an 'Enterprise-oriented' testing scope, product A would
require a functional testing, product B would require a web-service
testing, a product C would require a performance testing and also
complete project management etc.......
Priority must be assigned to each type of testing, based on the schedule for product release.
Identify Requirements to be automated
Testing
requirements and their nature is studied for the
product/application/modules. Each requirement has its own actions,
validations for testing. For example,
Scenario 1: For an
application, form validation functionalities, database validation and
accessibility functionalities needs to be validated.
Scenario 2:For
an application, all the web-service methods needs to be validated. This
would also include the delay time for the request's reponse from third
party systems
All the identified requirements are assigned
priority. This would help in identifying 'Build-Verification Test' (BVT)
requirements that should never fail.
Evaluate Test Automation Tool
Identified testing types and requirements, acts as a base criterion for test automation tool evaluation.
Checklist
- An exhaustive evaluation checklist needs to be created which is
in-line with the requirements and the tool is evaluated against this
checklist for positive results. Checklist needs to cover (not limited
to):
- Our requirements
- Types of testing
- Teams involved
- Licensing cost of the tool
- Maintenance cost
- Training and Support
- Tool's Extensibility
- Tool's Performance & Stability
Identify
Tools - Next task would be identifying different industry-standard
tools based on the requirements. This can also include tools to be
purchased or open source tools. Identified tools must be evaluated
against the checklist.
Sample Run - Tools claim that the tool
supports specific requirements, but finally when we try creating our
scripts it fails. Best way to evaluate is to create a sample run, this
includes different types of actions and requirements we need to perform.
Create sample scripts and execute the same for results.
Rate and
Select Tools - Based on the sample run, supportive tools could be
identified and rated. Also there may be scenarios, where multiple tools
satisfy the requirements. In the said scenario, we may need to
choose more than one tool, for test automation.
Implementation and
Training - High rated tools will be procured/open-source licensed.
Training needs to be conducted for the project team, on how to use these
tools.
Identifying Requirements can be Automated
Every
tool has its own limitations. A feasibility study needs to be conducted
for the requirements against the tools. This study would result in
listing requirements that can be automated. Also based on the nature of
the requirements, automation feasibility needs to be identified.
Design Test Automation Framework
Generally,
testers start creating test scripts based on the scenarios. This
includes multiple actions to be performed against each objects. This
approach leads to an ad-hoc test script creation and duplicate testing
effort, i.e. testers, would create test scripts for a single action in
different scenarios.
Our approach takes a different path as
explained below. For designing a framework, various elements need to be
taken into consideration. Utilities/Components (re-usable) would be
designed for the following elements that include (not limited to):
- Actions to be performed - Identification of actions to be automated for each object of the application
- Communicating Systems - Study of different internal systems, third-party systems and their communication methodology
- Business
Rules - List of business layers and any specific algorithm has to be
studied. A separate function needs to be created for each specific
algorithm.
- Database Communication - Database validation and check point validations
- Communication
with additional automation tools - In the scenario, where we would
require communicating with different automation tool. All the
communication requirements needs to be identified and designed
- Data retrieval - Retrieval of data from multiple input data stores
- Schedulers - Functionalities related to invoking of relevant scripts based on scheduler configuration
- Tool
Extensibility - Overcoming tool limitations. Components for
actions/validations for which the tool does not provide any support
- Device Communication - Device communication and data transfer related actions/validations
- Log - User-defined logs for analysis
- Error Handlers - Error handlers to handle known and unknown errors and log the information
- Custom Messages - Display of relevant defined messages
- Result Presentation - Customized and presentable reports on completion of test execution
Test automation framework would be designed based on the listed factors, using the following guidelines.
- Application-independent.
- Easy to expand, maintain, and perpetuate.
- Encapsulate the testers from the complexities of the test framework
- Identify and abstract common functions used across multiple test scripts
- Decouple
complex business function testing from navigation, limit-testing, and
other simple verification and validation activities.
- Decouple test data from the test scripts
- Structure scripts with minimal dependencies - Ensuring scripts executing unattended even on failures
Design Data Input Store
Types
of input data files supported by the tools, needs to be identified.
Based on the requirements, input files can be categorized as (not
limited to),
- Objects Identifier - Object identification syntax respective to the tool, mapped to the logical object name. For example,"A username textbox in the registration page mapped to logical object name - regUName"
- Scenarios/Workflows/Transactions
based input - Complete set of input data for different
scenarios/workflows/transaction. Each scenario/workflow/transaction
translates to "n" number of test cases. This test case based user input
benefits the team during future enhancements, in a way that multiple
input data can be added using the Test Case ID. For example, "A complete
financial transaction order processing, which invokes web services
methods for order processing. In this case, input data is created based
on test case id". TestCase (TC) 1 would be entering account details, TC2
Order details etc ...
- Custom Message - This can contain custom messages to be displayed for known and unknown errors.
- Driver
- File can contain list of file/transaction/workflow id's to be
referred to, for a selected batch execution/group of test case
executions
For all the files types, file format needs to be identified and prototyped based on the input data storage.
Develop framework
Framework
development is facilitated using the same set of identified tools.
Scripting language supported by the test automation tool is used to
create the components. Tool extensibility utility/component can be
developed using a different language. Utility functions/components
created based on framework design is explained in Step 7.
In addition to the re-usable components driver scripts and worker scripts needs to be created.
- Driver
Scripts - Scripts that execute a set of transactions, by invoking
relevant re-usable utilities/components for each test case. Driver
scripts can be mapped to a group of test cases related to a
scenario/transaction/screen/window.
- Worker Scripts - Scripts
that execute the driver scripts. Worker scripts are group of
driver/individual scripts to execute in a batch mode. Worker scripts
produce the final results, for the executed batch.
Approaches for developing re-usable utilities/components include:
- Record/Replay - Used to identify the object recognition pattern, of the specific tool. Very minimal usage
- Screen/Window/Transaction - Used to identify different scenarios to execute scripts in batches.
- Action/Keyword
- Invokes the relevant utility/components to perform actions on
specific objects. Driver scripts are created based on this input
- Data Driven - Input to keyword driven approach, where validation needs to be performed using multiple input combinations.
Populate Input Data Store
Input
data store needs to be populated based on the file structure defined in
Step 6. Data can be populated either manually or in an automated
fashion from different data-sources. Test data would be populated based
on parent-child hierarchy. For example,
"A transaction would be a parent hierarchy and Input to the test cases would be the child"
Configure Schedulers
Scheduler
requirement needs to be identified. Schedulers can be configured to run
a worker script (batch script) on a specific time period. This approach
benefits in a way that even a business user can configure the scheduler
and make the test execution happen.
Key Benefits of Framework
Standard process in Production
- Test automation processes, a single standard is established across
the organization. This helps the organization as they follow the
standard processes as compared to pre-empted ad-hoc processes, which
yield no results.
Free from dependencies
- Complete coding and component usage standards are defined in production. Organization benefits include:
- Independency from the individual coding standards and the utilities/components created
- Complete documentation helps the organization in inducting the new members with minimal effort
Complete Coverage
- Requirements are collected from an overall organization's perspective
(for eg: Product suite on multiple technologies .net and java etc).
Overall coverage of re-usable components which includes (data
communications, system communications, schedulers, loggers, reporters
etc)
This overall coverage minimizes the testing effort during the
later stages of the releases, for the entire product suite across the
organization.
Future Enhancements Support
- Organizations
need not worry about testing future enhancements. Only the validations
related to the enhancements need to be added to the existing base
framework, and that too with minimal effort.
Cost Estimation
- At the end of Step 6, the complete cost for the framework development can be estimated. This cost includes,
- Acquisition cost - In the procurement process of the tool, following cost needs to be considered:
- Tool Cost
- Cost based on number of licenses, based on our requirements
- Tool Support Cost (On-line, Telephone)
- Version Upgrade Cost
- Training
- Training cost incurred for training test engineers, business users,
developers and creating supportive training documentations must be taken
into consideration
- Environment - Cost involved in setting up the system environment (Hardware and Software) must be taken into consideration
- Development - Development cost can be calculated based on the components designed in the framework development - Step 6
- Maintenance
- Each tool has its own maintenance requirements. Some tools may demand
for part-time, some tools demand for a dedicated resource maintaining
the tool. Based on the tool's requirements, this cost needs to be
calculated
http://www.thoughtworks.com/insights/articles/guide-test-automation
