TAC
WorM (Workflow Management) Features
·
Administration Settings
o The administrator has a variety of configuration
options to support customized processes
§
Allow anyone to
delete Test Cases vs. allow only those with admin[1]
authority to delete them
§
Allow login with
no password vs. password required
§
Allow Feature
Tree restructuring through drag and drop of nodes (moving Sub-Features and Test
Cases to new parents)
§
Force changer of
Test Cases to enter the reason for changes vs. allow it but don’t force it.
§
Force users to
enter time metrics for running test cases vs. allow but don’t require it
·
Test Case setup
time
·
Test Case
execution time
·
Test Case tear
down time
·
Defect
investigation time
§
Encrypt
Repository files vs. no encryption
§
Allow limited
automatic login to TAC Manual Assistant for Testing (TAC MAT): when running a
test, allow the system to “remember” the user’s login for X hours.
§
Tickler default
settings when first installed (allows an organization to customize the way
Tickler works for all users.)
·
Show Test Cases
assigned for review by this user
·
Show Test Cases
owned by this user awaiting review
·
Show Test Cases
owned by this user awaiting maintenance
·
Show automated
test results owned by this user that are unhandled (i.e. no one has signed off
on Failed and/or blocked results.)
·
Enter number of
days backwards we should check for Unhandled automation results.
·
Hours between
Tickler checks (every time WorM is started, it checks to see if Tickler should
be run automatically. If there are X
hours since the last time it was run, it will be triggered.)
·
Turn automatic
execution of Tickler off / on.
§
Defect default
processing
·
Number of
expected defects in this release (optional)
·
Allocate defects
o By proportion: As Test Planner (TP) objects are
created, the number of defects they are ‘assigned’ to find varies
proportionally with the number of expected defects vs. the number of Test Cases
in the TP.
o Manually by the test manager, taking into account
risk areas, etc.
·
Estimated
Standard Investigation Time for defects (average number of hours and minutes
that it is expected to take to investigate / document / process each defect
(for the test team only.)
§
File Repository
Path (location on a server that the File Repository will be stored.
§
Artifact Save
Base Path: a location on the server to be used as the base directory for
storing zipped up artifact[2]
files for both automated and manually run Test Cases.
o
§
Access
§
SQL Server
o Multiple Projects / Applications are supported
§
There is
complete separation of name spaces
o Ownership can be transferred from one user to another
§
A single object
can be transferred
§
All owned
objects can be transferred
§
A user can be
deleted from the database
o Lists can be configured to simplify and customize
selections for users. Available lists:
§
Apps / Projects
§
Automation Sets
(groups of automated tests run as a single entity)
§
Supported
Automation Tools (commercial or home brew)
§
Releases (per
App / Project)
§
Builds [Cycles]
(per Release)
§
Test
Environments
§
File Categories
(for the File Repository cataloguing)
§
Data-Driven
script names (Parent and Child)
§
Test Types
(classification of different Test Cases)
§
User Names /
Roles (accessible only to those with Administrator privledges.)
·
Features /
Sub-Features
o T-WorMS' hierarchical Feature Tree allows an
organization to easily map requirements to the software features and
sub-features that represent them.
o The Feature Hierarchy is used to organize Test Cases
into a logical and maintainable order
o Features can be added and deleted as the project
evolves, allowing early start with no penalties for changes
o Features can be hot linked or reference linked back
to requirements documents to provide traceability
§
Direct
connection to Requirements Management tools may be supported in the future
through their APIs.
§
Controls for
both a Requirements Document and the location in the document are provided.
o Quantitative risk assignment is supported in three
dimensions
§
Business risk
§
Technical Risk
§
Project risk
o Feature history table shows evolution of feature. Each change to a Feature is tracked by
requiring (Sub-)Feature changer to enter the reason for the change.
o Each Feature / Sub-Feature is an object owned by the
person who created it (may be transferred to another user)
§
Assigning a
(Sub-)Feature to another person may be recursive (i.e. the ownership of all
sub-features and tests below the transferred one may also optionally be
transferred to the new owner if they are owned by the original person.)
o Feature defect mechanism allows the suspension of
associated Test Cases from running when a feature is broken, not available or
not ready for test
o Features may be deleted, but only if they are a leaf
node (i.e. no Sub-Features or Test Cases are directly owned by the
(Sub-)Feature.
o Test Case reports (in Excel) can be created for all
Test Cases directly owned by the (Sub-)Feature.
o All tests directly owned by a (Sub-)Feature can be
marked for review
§
May be
recursively applied to all Test Cases in the hierarchy below the selected
(Sub-)Feature.
§
Allows the
selection of reviewers and date for review (see description on Test Case review
below.)
o Owner of (Sub-)Feature and the user who last changed
the Feature are tracked.
o Feature Description (a required field) is a Rich Text
Edit that allows formatting the text and inclusion of images, objects, etc.)
o Future release will allow reversion to earlier
versions of the (Sub-)Feature
·
Test Cases
o Test cases are "assigned" to (Sub-)Features
o Risk factor is automatically inherited from owning
(Sub-)Feature
o Test information fields are highly configurable to
provide more or less data and control as needed
§
Includes several
Rich Text Edit fields to allow individualized formatting and inclusion of
images if desired
·
Detailed
Description (Common[3]
) [Optional]
·
Test Comments
(Common) [Optional]
·
Test Specific
(Specific) [Optional]
·
Expected Results
(Common) [Optional]
§
Standard Edit
fields
·
Estimated[4]
setup time for Test Case (Specific) [Optional]
·
Estimated tear
down time (Specific) [Optional]
·
Estimated
execution time (Specific) [Optional]
·
DiRT record[5]
associated with this Test Case (Specific) [Optional]
·
Relative Order
o An integer between 0..100
o Defaults to 50
o This field is used to indicate relative execution location
for the Test Case
o For automated Test Cases, in general, the smaller
this number, the earlier in the suite the Test Case will be run[6].
o For manual Test Cases, this order is merely advisory.
§
Drop-down List
boxes
·
Test Environment
(Specific) [Optional]
·
Test Type
(Specific) [Optional]
o Multiple modes for Test Cases are supported:
§
Manual
·
Execution is
expected to be performed by manual testers
§
Automation
Candidate
·
Execution is
eventually expected to be performed though automated means.
·
Until such time
as the Test Case is actually automated, it is treated identically to the manual
tests.
·
All automated
tests start out as Automation Candidate; a test owner may not directly change
the execution mode to Automated. Only a
user with Automator authority may change a Test Case from Automation Candidate
to Automated.
§
Automated
·
Execution will
be performed via some kind of automation tool rather than manually.
·
Automated test
cases are still owned by the manual tester; Automators do not take ownership
simply because they created an automated way to execute them.
o Flags are used to indicate Test Case status
throughout life cycle
§
Needs
Maintenance
·
Set by a QA
Engineer or Automator to indicate that the Test Case needs to have work
performed on it.
·
This does not
imply that the test case may not be run.
If the Test Case is broken to the extent that it cannot be run, it
should be either suspended or marked as NOT Runnable.
§
Needs Review
·
Indicates that
this Test Case has been assigned to one or more users for review.
·
A review must be
accepted by the owner of the test (or other authorized person) for the Needs
Review Flag to be toggled off.
§
Runnable[7]
·
This flag
indicates that the test case is ready to run.
·
If the Test Case
is manual or an Automation Candidate, when this flag is set to TRUE (checked),
the test will be reported in Test Planners as ready to execute (assuming that
it is not Suspended (see below.)
·
If the Test Case
is automated, it will automatically be available to be executed via automation
assuming that the Executable flag is also checked and the Test Case is not
Suspended.
·
When a Test Case
is created, this flag defaults to FALSE.
·
The user may
edit the Test Case any number of times without tracking changes as long as this
flag remains FALSE.
·
Once the Test
Case is ready for use, the Runnable flag should be set to TRUE.
·
Once this is
done, each time the Test Case is edited, the reason for change must be entered
and the Test Case is versioned on each change.
·
Once versioning
starts it does not stop, even if this flag is toggled back off.
·
This flag
generally should not be set to FALSE (unchecked) once it has been set to
TRUE.
o However, local processes may be designed to use this
flag to control execution. It is better
to use the Defect / Suspension mechanism to control the actual running of the
test.
·
If there are
problems with the Test Case, a Defect record should be opened against it (see
below.)
o If the Test Case should not be executed due to a
defect in the Test Case, the Defect Record should be set to Suspend the Test
Case.
§
Executable
·
This flag is
only used for Automated Test Cases and only the Automator may toggle it.
·
If this flag is
set to TRUE (checked) the Test Case is eligible for automated execution
assuming that it is also set to Runnable and is NOT Suspended.
·
This flag is
initialized to FALSE.
·
It should be
toggled to TRUE after a Test Case is fully automated and the automation has
been unit tested.
·
This flag
generally should not be set to FALSE after it has been set to TRUE.
o However, local processes may be designed to use this
flag to control execution. It is better
to use the Defect / Suspension mechanism to control the actual running of the
automated test.
§
Suspended
·
This flag is
automatically toggled from off to on based on the Defect mechanism.
·
Only the
Automator may manually change this flag in the Define Automation screen.
·
Creating a Test
Defect record against a Test Case does not automatically suspend the test. There is a setting in the Defect record to
indicate whether the Test Case should be suspended or not.
·
Creating a
Feature Defect record against any (Sub-)Feature in the hierarchy above a Test
Case may suspend that Test Case.
o Creating a (Sub-)Feature Defect Record for the
(Sub-)Feature that directly owns the Test Case will always cause it to be
Suspended.
·
Closing a Defect
Record (either Test or Feature) does not necessarily un-suspend the Test Case.
o The Test Case suspension remains until such time as
ALL defect records [that called for Test Case suspension against it] are
closed.
o Tests may have files attached to them (automatically
made available at run time from the File Repository)
§
For tests that
are automated, the run-time system will automatically download the attached
files to the local workstation before the test runs.
§
For manual and
automation candidate Test Cases, the MAT tool will automatically download all
attached files when the test is started.
o A tabular manual script can be stored as part of the
test object
§
A script is a
three column table
·
Test Step
·
Data to use
·
Expected result
§
Scripts are versioned
each time they are changed.
§
In the next
version of WorM, a user will be able to revert the script back to an earlier
version.
§
A script may be
printed out in tabular form.
o Test history table shows all changes made to the Test
Case
§
After the test
is marked Runnable, each change to the Test Case cause it to be versioned.
§
In the next
version of WorM, a user will be empowered to revert a Test Case back to an
earlier version.
o Integrated Test Case review field makes good process
easy
§
The owner of a
Test Case or any authorized user (Test Lead, Manager, etc.) may assign one or
more people to review the Test Case.
§
Each person who
is assigned to the review gets notification in their Tickler report of their
assignment.
§
A specific due
date is assigned for the review.
§
As the Test Case
is reviewed, the user adds comments to a running history showing all past
review comments with timestamps.
§
After completing
the review, the reviewer's name is automatically removed from the Review list.
§
When the last
reviewer name is removed from the list (i.e. the last review has been
performed), a note indicating the reviews are done is placed in the Tickler
report of the Test Case owner.
§
The owner (or
authorized person) may accept the existing reviews and remove the Test Case
from review at any time.
§
New reviewers
may be assigned at any time.
o Pre-requisite tests can be identified to ensure that
a Test Case is only executed when all of its pre-requisite tests have been
passed.
§
For automation,
pre-requisites are enforced.
·
If an automated
test has defined pre-requisites, the run-time system checks to make sure that
each and every one of the listed pre-requisite Test Cases both has run and has
passed before allowing the Test Case to run.
·
If one or more
pre-requisite tests did not run or did not pass, the Test Case will not be
allowed to be run. Instead, it will be
marked as blocked and logged as such.
§
For manual Test
Cases, pre-requisites are only advisory.
§
The interface
makes pre-requisite Test Case selection easy.
·
A warning is
given at selection time if the relative execution order is reversed (i.e. the
pre-requisite test has a larger relative order than the Test Case being
edited.)
o Execution frequency guidelines can be set
§
Each Test Case
is set to have an execution frequency between 0 (execute every build) and 4
(run only when running Full Regression.)
§
For automation,
the execution frequency is used as a filter.
·
All automated
Test Cases that are theoretically available for running are filtered to select
only those Test Cases that are included in the specified execution frequency.
·
This scheme
allows a user to slice and dice the entire automation test base to select tests
to run.
o The owner of a Test Case can recommend that it be run
in one or more Automation Sets.
§
An Auto Set is a
group of automated Test Cases that is run as a single entity.
§
The Automator
actually selects which Test Cases go in which Auto Sets.
§
The
recommendation of a test owner is advisory only, as there may be other criteria
known to the Automator in actually selecting the tests.
o Reports
§
A report may be
created to show all of the Run Results ever created for this Test Case (both
automated and manual.)
§
A button is
available to see the time metrics recorded for all manual runs of a Test
Case. Pressing the button generates this
report, allowing the user to refine the times to more closely resemble reality.
§
A report (in
Excel) can be generated showing all fields in the Test Case record.
§
See the main
Report section below for a complete list of all the T-WorMS reports.
o DiRT is integrated with WorM to collect, store and
track Test Data.
§
A single DiRT
record from a Virtual Table may be attached to a Test Case.
§
Because a DiRT
record may contain other DiRT records to any desired recursive level, an almost
infinite amount of data may be attached to any Test Case as a single reference.
§
There is a
button in the Test Case definition screen to automatically open up the DiRT
tool (if not already open) and bring up the attached record in Edit mode.
§
There is also a
button to bring up the record in View only mode.
§
See DiRT section
for more information.
o Ownership of a Test Case may be transferred to
another user.
o The Test Owner and Last Changer are both listed in
the interface.
o A Test Case can be deleted under certain
circumstances:
§
If a user has
Admin permissions
§
If the Admin
allows, a QA Engineer may delete Test Cases up until the time they are marked
Runnable.
§
When a Test Case
is deleted, it is placed in an archive.
·
In the next
version of WorM, a user will be able to un-archive a Test Case.
o From a Test Case, a user may elect to go to the
Automation Test Evaluation screen.
Pressing this button causes all automated run results for the Test Case
to be loaded into the Evaluation Tree and highlights the first one.
o A button is available and enabled when the Test Case
is Linked to other Test Cases. Pressing
the button brings up a report that shows all Test Cases that are linked to it.
·
Feature Tree
o In the Test Definition screen, this tree view control
contains the hierarchical structure containing all of the project Features,
Sub-Features, and Test Cases that are defined in the project (subject to
filtering.)
o A filter dialog may be invoked to filter the test
cases.
§
(Sub-)Features
are not filtered. Instead, when Test
Cases are filtered, all Features and Sub-Features that own no Test Cases are
removed from the tree automatically.
§
Filtering may be
done by creating a SQL predicate in the Filter dialog.
o Drag and Drop capabilities
§
A number of
actions may be performed in the tree via Drag and Drop (these may also be
performed via right mouse click menus)
·
A Test case or
branch starting from any Sub-Feature may be moved to another owning
(Sub-)Feature by drag and drop with no keyboard key pressed.
·
A Test Case or
branch starting from any Sub-Feature may be link copied to any other
Sub-Feature by Drag and Drop with the Ctrl key pressed.
o Each Sub-Feature is created new in the new position.
o Each test case will be link copied (the Specific
portion of the Test Case is created new, while the Common portion of the test
is merely shared.)
o The original test case is unchanged (other than a
link to it being added)
·
A Test Case or
branch starting from any Sub-Feature may be cloned to another Sub-Feature. A clone means that an entire new Test Case is
created for each one that is moved.
There is no linking involved.
This is done by Drag and Dropping the branch with the Shift key pressed.
·
A Test Case may
be moved to a different position under its parent.
o This is only available when the tree is not sorted
alphabetically.
o The tree has a right mouse menu, containing items to
control the following:
§
The tree may be
ordered alphabetically, or it may be ordered in the same order it was created
in.
§
The tree may be
expanded or collapsed.
§
The tree may be
refreshed (reloaded).
§
An individual
branch of the tree may be expanded.
§
A user may
search for a test case by
·
Specific Key
value
·
Sub-string in
the Test Case name
§
A user may
duplicate an existing Test Case
·
A completely new
Test Case is created (no linking is done)