Workflow Driven Tests

7-May-2002

Some commonly touted benefits of test automation include repeatability, reliability, and increased coverage. Unfortunately, simply automating tests will not guarantee that these benefits will materialize. One of the largest barriers to realizing these benefits is an overwhelming maintenance load driven by the workload needed to keep individual tests useable & accurate, and the inability to reuse automation code throughout the enterprise.

Implementing development best practices thru an automation framework can significantly reduce the maintenance load associated with automated tests (see Enterprise Framework for Test Automation white paper). However, there remains a critical issue of the limited availability of testware developers. People with both QA and software development skills are not easy to come by.

How do you make the most out of your limited testware development resources?

Data driven tests allow input and verification data to be accessed from a source other than the code. This information is typically stored in spreadsheets and/or data tables. Data driven tests are normally more flexible because one test script can handle multiple data scenarios. In addition, code maintenance workload is reduced because the maintenance is done on the values in the data tables. Data tables are intuitively easier to understand and modify than programming code.

However, moving everything into data tables is not the answer to all code maintenance problems. Some people have a fallacy of thinking that if two aspirin are good, then ten aspirin must be better. This fallacy can be applied to data tables as well. Putting everything you can into data tables can cause table maintenance issues that are greater than the saving realized in code maintenance. After establishing that it can be done, ask if it should be done. When is this threshold reached? It depends on your specific situations, internal processes and corporate culture.

Additionally, data driven tests do not work well where the results returned are dynamic and require unexpected decision-making. Two examples of this are exception handling and multi-path processing requiring looping conditions. The data driven model works well in many cases, just keep in mind not in all.

It has been proposed that data driven tests will eliminate the need for testware developers. This is incorrect. As with any software project, development resources are needed throughout the development lifecycle. Many issues can effect the completion of a data driven test. Incorrect entries in the data table, coding errors in test script functions, GUI map maintenance, and problem isolation are just four examples. Identifying that a test has failed does not require extensive knowledge, but identifying the root cause of that failure requires a testware development resource.

A key use of data driven tests is in verifying workflow processes. A workflow is a logical sequence of tasks that are performed to exercise a business or test process. Workflow tests simplify code maintenance by modularizing navigation, entry actions, and test data usage.

This paper describes how to implement workflow driven tests.

Complex software normally means more errors, higher maintenance costs, and software with a limited life. Reducing this complexity is a key way of creating successful software. The same holds true for testware. Using an approach that helps create reusable modular code will reduce complexity, while at the same time increasing maintainability & flexibility.

When building testware you have to take into account test data, navigation steps, business rules, data entry mechanics, and the GUI objects used in the application. Bundling this information together increases the complexity of the testware. Isolating this information into five segments will help produce reusable, modular code. Each of these segments requires their own development and maintenance focus. This forms the basis for establishing workflow driven tests.

Figure A

Identify the unique physical description of every GUI object (attributes, properties, class type, etc.) used in the workflow and then tie a unique logical name to each description. This is known as building a GUI Map. GUI changes will have an impact on your test code, but this mapping construct keeps the effects to a minimum. Key to this process is the logical naming convention. Once you have established the logical names, you do not want to change them.

Identify the data that will drive the workflow, satisfy data entry requirements, and provide expected results for each validation point. Also identified is the method you will use to manage your test data. This simplifies the creation of test scripts, helps minimize data management & maintenance issues, and increases the possibility that this data can be shared between test cases.

Identify how to navigate through each defined workflow test process. The data required in building the workflow is heavily dependent on the GUI Map and Test Data segments. Manipulating workflows can be performed while minimizing or eliminating the need to change test code. Generally, this information is stored in a Workflow File.

The control engine is designed to read each row of data from the workflow file. Each row of data is analyzed to determine what actions to apply to the application during the execution of the workflow process. This includes data entry, navigational actions, and test verification. Creating common data entry processes ensures consistency throughout all tests. If a data entry process is changed, then limited code changes can ensure that all effected workflows will perform properly. A majority of testware code is kept in a function library.

Identify the actions to take when certain conditions are met. These can include synchronization events, checkpoints, database validation, and special workflow actions (looping, pausing for manual intervention, unique function calls, etc.). This isolates the identification & maintenance of process rules. This information is contained within test scripts and function libraries.

Figure B

The workflow model components can be categorized as either requiring coding or not. The components of the coding category consist of the Test Script (a.k.a. Workflow Engine), FuncLib, and ResultLib. The components of the non-coding category consist of the GUI Map, Test Data Database, and Workflow file. The goal is to have the majority of test case maintenance efforts occur in the non-coding area.

As an application evolves through its lifecycle, changes to its workflow processes are quite likely. In fact, they should be expected. Isolating your test information into these segments reduces the impact that workflow process changes have on your tests. Maintenance will be more focused (identifiable & limited scope) with minimal impact to other segments.

Figure C

This script library contains functions that manipulate screen objects. These functions are able to exercise all object class/method combinations. Some examples of this are: Press a Push Button, Select a Radio Button, and Check a Check Box. Modification of this library occurs when new object class/method combinations are identified. Function libraries are designed based on enterprise and project requirements. If the software development team follows standards, then a global library can be built to handle the common objects and actions that occur in all applications across an enterprise. When common processes are shared, maintenance costs will be affected positively. However, each application (project) may have unique business processing needs that require special objects and actions to be kept local to the application.

This contains a physical to logical mapping of GUI windows and objects. Each object is uniquely defined by a physical description that includes information such as the object’s physical id, window id (where it resides), class id (what type of object), and other attributes that help in the unique identification. Once the physical description is defined, the testware developer will associate a unique logical id to it. There are automation tools that provide a facility to manage this process and automatically provide logical names. We highly recommend establishing a logical ID naming convention. As shown in figure C, the logical identifiers established in the GUI Map are used throughout the workflow components (test scripts, workflow files, and test data databases). The GUI Map is a high change segment. Consistent object naming conventions should be a practice followed by your application development group. Keeping up with GUI name changes could be painful if this practice is not followed (see Framework white paper).

This library contains functions that create execution logs. Modifications occur only if new output formats are required.

If a test data database is incorporated into your workflow process, there can be an endless number of uses for the database. It can contain input and verification data, business process dynamics, special business rule scenarios, and so on. One use would be to create a table for each navigated window within the workflow process (or the entire application). Each object of a window that requires an action makes up a column in its associated table. The table and object names are given the same logical names mapped in the GUI Map. This information must be kept in synch with the GUI Map.

Each workflow has a Workflow Engine. The Engine performs the user & verification actions that are contained within a Workflow File. An Engine is created when a new workflow has been identified. The code in the Engine is modified when business rules are created or modified. In addition to processing a Workflow File, this script also contains additional business rules (verification actions) for that workflow. This allows the Engine to handle variations in the workflow (multiple workflow files or special use of test database data).

This is a sequential list of navigation steps thru a workflow process. It contains information about what windows to use, which buttons to press, what fields to enter data into, where to get that data, and what to verify. Some examples are:

A simple web-based application is shown below (client maintenance).

The following steps are performed:

1. Press the Add Client button in the Main window.
2. In the Add Client window perform the following:
1. Enter a name in the First Name field.
2. Enter a name in the Last Name field.
3. Enter a phone number in the Phone Number field.
4. Check the Call Back checkbox.
5. Press the OK button.
3. In the Update Status window perform the following:
1. Check for "Client Added" in the Status field.
2. Press the OK button.
4. In the Main window perform the following:
1. Find the client name that was just added.
2. Press the Edit link next to the name.
5. In the Edit Client window perform the following:
1. Verify that all fields match what was entered from step 2.
2. Press the Cancel button.
6. The Main window should now be in focus.

The wndAddClient table contains the following:

In this case, the workflow rows where RowAction = SKIP are used for readability (documentation). Row 4 populates all fields from the wndAddClient table. Row 10 calls a script that finds the name in the main window and presses the Edit link. Row 11 verifies all fields against those in the wndAddClient table.

The example below performs the same actions as the Workflow File above. The only difference is that all fields, actions, and values are explicitly listed. The same Workflow Engine can process both files.

Changes to an application’s workflow will require maintenance. The level of maintenance can be categorized as low, medium, or high.

• Low – A simple change to the workflow’s navigation sequence is made.
• Workflow File –navigation sequence is modified
• Medium – A new object is added to an existing window, a new window is created, or a new workflow is identified.
• GUI Map – new object identified
• Test Data Database – modified to reflect new objects, or the new table and it’s objects
• Workflow File – navigation sequence is modified, if new object directly accessed (new object to an existing window does not require a workflow file change), or the new window is added to workflow scenario.
• High – A new object is identified as having a class that is not handled by current functions or a new special condition exists (finding a string in a dynamic list, new exception handler required, etc.).
• GUI Map – new object identified if a new class
• FuncLib – code to handle the new object class/method combination is added.
• Test Data Database – modified to reflect new objects
• Workflow File – navigation sequence is modified.
• Workflow Engine – customized to handle any special decision making actions, or handle exceptions, etc.

Consistent coding practices by the software development team will help ensure that the numbers of high maintenance occurrences are minimized. Most of your efforts will then be focused on low and medium maintenance issues (non-coding).

STAMP was presented with a test automation project by the Residential Funding Corporation’s (RFC) quality control department. With this project came a list of test automation requirements specified to meet their current and long-term enterprise wide needs. The following is a subset of those requirements:

1. The method and code must be reusable across all similar systems within RFC.
2. Minimize the maintenance of test scripts when changes to the application occur (GUI, workflow, data content, and business rules).
3. Provide a means to simply route and archive test results (detail and summary) for management and development without the need of excessive tool licenses. These results should be viewable by anyone using RFC’s standard browser.
4. Simplify the process of logging defects into the defect-tracking tool of choice.
5. Automated workflows must be creatable by any test or business analyst resource with or without programming knowledge.
6. Minimize the effort required in creating and maintaining test data.
7. Incorporate software development best practice methods in the design, implementation and support of the test automation enterprise within RFC.
8. Test automation must run in all known test environments without any direct changes to the test scripts or process.
9. Minimize the effort for knowledge transfer, both in transmitting and receiving the knowledge.

The application under test was a functionally mature web-based product. Future releases of this product were already planned which included minor functional enhancements and major middleware and backend changes in an attempt to improve performance. The development group was the initial user of the testware suites, followed by the business analysts who perform the final acceptance test verification before release.

It took 5 months to provide the method, develop the first couple of test suites (diagnostic and navigation suites) and implement the processes to support multiple test environments. It took only 3 months to develop four additional test suites that provided a near complete set of end-to-end workflow test processes.

However, the project was not executed in eight consecutive months. Test automation was added in increments of four to six-week assignments, with weeks and month-long breaks in the development cycle. To support such a schedule from a consulting firm perspective, we had to have four different testware developers work on the project. Our least experienced developer took on the last leg of the project. He was able to support the existing scripts, and begin development of new scripts, within two weeks of his arrival (the test plan was developed prior to our last engagement).

All the suites developed are still being used a year after they have been developed and are planned to be used to help test the product after a technology conversion to .NET.

A second win was the transfer of the methodology and common scripting to a new data entry product that was being developed within RFC. Within seven weeks of our engagement on this new project we had a comprehensive list of navigational and data entry scripts built and providing service. After the initial release of the product, test automation was no longer supported. Nearly five weeks had elapsed since the test automation was executed again and nearly 16 requirement changes had occurred in the application during that time. With no release notes, or understanding of what might have changed, it took us only 48 hours to update all the workflow processes and run the tests in three separate test environments. The sixteen requirement changes had no effect on our test automation code. Changes were required to the GUI Map, Workflow file, and test data tables.

This paper has described Workflow Driven Tests, a modular approach to test automation. This methodology creates a high level of code reuse (potentially across all applications in the enterprise); simplifies maintenance by driving the testing of high change areas thru data tables/files; and enables testware to evolve along with the applications being tested.

Tim Nelson is a Senior QA Test Automation Engineer. He has over 25 years of IT and programming experience within the test and QA industry. He has a BS in EE from the University of Minnesota, with minors in both computer science and business. His experience has extended across several different engineering and business development sectors. He has engineered diagnostic and automated test systems for hardware gate and block model simulators to the custom design of enterprise wide test automation frameworks for client/server and Web based technologies. He is now a principle owner in a Software Test and QA consulting firm (STAMP Technologies, LLC) that is developing a new generation of highly qualified software test and automation specialists. Tim can be reached at tdnelson@stamptech.net.

Michael R. Waller is a Senior QA Consultant. He has over 20 years of IT experience including software QA, test automation, software development, and operations. His business sector experience spans manufacturing, financial, and software development companies. He has an MS in Management from Florida Institute of Technology and a BS in Math/Computer Science from Boise State University. Mike can be reached at mwaller@stamptech.net.