Analyzing Test Results in QTP

When a run session ends, you can view the run session results in the Test Results window. By default, the Test Results window opens automatically at the end of a run. If you want to change this behavior, clear the View results when run session ends check box in the Run tab of the Options dialog box.

The Test Results window contains a description of the steps performed during the run session. It displays a single run iteration.

After you run a component, the Test Results window displays all aspects of the run session, including:

• a high-level results overview report (pass/fail status)
• the data used in all runs
• an expandable tree of the steps, specifying exactly where application failures occurred
• the exact locations in the component where failures occurred
• detailed explanations of each step pass or failure, at each stage of the component

Note: The Test Results window can show results with up to 300 levels in the tree hierarchy. If you have results with more than 300 nested levels, you can view the entire report by manually opening the results.xml file.

Update test object descriptions

The Update Options tab contains the Update test object descriptions check box. Selecting this option instructs QuickTest to update the test object descriptions for your business component according to the properties currently defined in the Object Identification dialog box for each object class. You can use this option to modify the set of properties used to identify an object. When you use this option, all values are updated, even if they are parameterized or use regular expressions.

Tip: You can also update individual test object descriptions from the object in your application using the Update from Application option in the Object Repository window or Object Repository Manager.

Note: If the property set you select in the Object Identification dialog box for an object class is not ideal for a particular object, the new object description may cause future runs to fail. Therefore, it is recommended that you save a copy of your component before updating it, so that you can return to the previously saved version, if necessary.

This option can be especially useful when you want to record and debug your component using property values that are easy to recognize in your application (such as object labels), but may be language or operating system dependent. After you debug your component, you can use the Update Run Mode option to change the object descriptions to use more universal property values.

For example, suppose you design a component for the English version of a part of your application. The test objects are recognized according to the test object property values in the English version, some of which may be language dependent. You now want to use the same component for the French version of this part of your application.

To do this, you define properties that are non-language dependent. These properties will be used for object identification. For example, you can identify a link object by its target property value instead of its text property value. You can then perform an update run on the English version of this part of your application using these new properties. This will modify the test object descriptions so that you can later run the component successfully on the French version of your application.

Tip: If you have a component that runs successfully, but in which certain objects are identified using Smart Identification, you can also use the Update test object descriptions option to update the test object description property values.

When you run the component with Update test object descriptions selected, QuickTest finds the test object specified in each step based on its current test object description. If QuickTest cannot find the test object based on its description, it uses the Smart Identification properties to identify the test object (if Smart Identification is enabled). After QuickTest finds the test object, it then updates its description based on the mandatory and assistive properties that you define in the Object Identification dialog box.

Note: Test objects that cannot be identified during the update process are not updated. As in any run session, if an object cannot be found during the update run, the run session fails, and information about the failure is included in the Test Results.

Any properties that were used in the previous test object description and are no longer part of the description for that test object class, as defined in the Object Identification dialog box, are removed from the new description, even if the values were parameterized or defined as regular expressions.

If the same property appears both in the test object's new and previous descriptions, one of the following occurs:

• If the property value in the previous description is parameterized or specified as a regular expression and matches the new property value, QuickTest keeps the property's previous parameterized or regular expression value. For example, if the previous property value was defined as the regular expression button.*, and the new value is button1, the property value remains button.*.
• If the property value in the previous description does not match the new property value, but the object is found using Smart Identification, QuickTest updates the property value to the new, constant property value. For example, if the previous property value was button.*, and the new value is My button, if a Smart Identification definition enabled QuickTest to find the object, My button becomes the new property value. In this case, any parameterization or use of regular expressions is removed from the test object description.

Permissions Required for Business Process Testing

The Quality Center Project Administrator can control access to a project by defining which users can log in to it and by specifying the types of tasks each user may perform. The Quality Center Project Administrator can assign permissions for adding, modifying, and deleting folders, components, steps, and parameters in the Business Components module of a Quality Center project.

Note: To modify application areas, you must have the required permissions for modifying components, and adding, modifying, and deleting steps. All four permissions are required. If one of these permissions is not assigned, you can open application areas only in read-only format.

You need to make sure you have the required Quality Center permissions before working with business components and application areas.

QTP for Business Process Testing Terminology

The following terminology, specific to QuickTest Professional for Business Process Testing, is used in this help file:

Application Area—A collection of resources and settings that are used for the creation and implementation of business components. These include function libraries, shared object repositories, keywords, testing preferences, and other testing resources, such as recovery scenarios. An application area, provides a single point of maintenance for all elements associated with the testing of a specific part of your application. You can define separate application areas for each part of your application and then associate your components with the appropriate application areas.

Business Component (or Component)—An easily-maintained, reusable unit comprising one or more steps that perform a specific task. Business components may require input values from an external source or from other components, and they can return output values to other components.

Also known as Keyword-Driven Component.

Manual Component—A non-automated business component created in Quality Center. In QuickTest, you can view and work with manual components only after converting them to business components.

Scripted Component—An automated component that can contain programming logic and can be edited in QuickTest using the Keyword View, the Expert View, and other QuickTest tools and options.

Keyword View—A spreadsheet-like view that enables tests and components to be created, viewed, and debugged using a keyword-driven, modular, table format.

Function Library—A document containing VBScript functions, subroutines, modules, and so forth. These functions can be used as operations (keywords) in components. You can create and debug function library documents using the QuickTest function library editor.

Business Process Test—A scenario comprising a serial flow of business components, designed to test a specific business process of an application.

Component Input Parameters—Variable values that a business component can receive and use as the values for specific, parameterized steps in the component.

Component Output Parameters—Values that a business component can return. These values can be viewed in the business process test results and can also be used as input for a component that is used later in the test.

Local Input Parameters—Variable values defined within a component. These values can be received and used by a later parameterized step in the same component.

Local Output Parameters—Values that an operation or a component step can return for use within the same component. These values can be viewed in the business process test results and can also be used as input for a later step in the component.

Roles—The various types of users who are involved in Business Process Testing.

Automation Engineer—An expert in QuickTest Professional automated testing. The Automation Engineer defines and manages the resources that are needed to create and work with business components. The Automation Engineer creates application areas that specify all of the resources and settings needed to enable Subject Matter Experts to create business components and business process tests in Quality Center. The Automation Engineer can create and modify function libraries, and populate a shared object repository with test objects that represent the different objects in the application being tested. The Automation Engineer can also create and debug business components in QuickTest.

Subject Matter Expert—A person who has specific knowledge of the application logic, a high-level understanding of the entire system, and a detailed understanding of the individual elements and tasks that are fundamental to the application being tested. The Subject Matter Expert uses Quality Center to create and run components and business process tests.

Understanding the Application Area

The application area is the foundation upon which components are built. An application area provides a single point of maintenance for all elements associated with the testing of a specific part of an application.

In the application area, you can define specific settings that are relevant for testing a particular part of your application. For example, you can define settings that instruct QuickTest to load specific add-ins at the start of a run session, run a component only on specified applications, activate a recovery scenario under particular conditions, and so forth. You can also specify the keywords that are available to any component that is associated with that particular application area.

An important aspect of application areas are the resource files that can be used by a component. After you create these resource files you store them in the same Quality Center project used by the Subject Matter Experts who create and run the business process tests for the specific application. Typical resource files include function libraries and shared object repositories.

You create function libraries that contain functions, or operations (also known as keywords), that can be called by a component. These functions contain programming logic that encapsulates the steps needed to perform a particular task, and they enhance the functionality of the component that calls them. You can use QuickTest's built-in function library editor to create these function libraries. You can also use the QuickTest Function Definition Generator to insert basic function definitions, and then complete each function by adding its code.

After you associate function library files with an application area, you can prioritize them according to relevance. By associating a function library with an application area, any component based on that application area will have access to all public functions defined within that function library.

You also create, populate, and maintain shared object repository files that are used by QuickTest to identify the objects in your application. You define and modify test object information in shared object repositories using the QuickTest Object Repository Manager. After you associate shared object repository files with the application area, you can prioritize them according to relevance. By associating a shared object repository with an application area, any component based on that application area will have access to all of its test objects and other elements.

You can create multiple application areas—each one focusing on a particular part (area) of the application being tested. For example, for a flight reservation application, one application area could be created for the login module, another application area for the flight search module, another for the flight reservation module, and still another for the billing moduleIn addition to creating and maintaining the resource files associated with the application areas, you can also use QuickTest to debug components and their associated function libraries. You can also create components in QuickTest, although this is more often done by Subject Matter Experts using Quality Center.

Understanding Business Process Testing

Business Process Testing enables structured testing of an application by combining test automation and automatically generated, easy-to-understand test documentation. Business Process Testing is not dependent on the completion of detailed testing scripts. This enables applications to be tested manually before automated tests are ready. This also enables business process tests to be created and implemented more quickly than other automated tests, enabling potential performance issues to be detected earlier in the development process, before downtime can occur.

Components are easily-maintained, reusable units that perform a specific task. They are the building blocks of business process tests. Each component is comprised of several application steps that are logically performed together in a specific order. For example, in a Web application, a login component might be comprised of four steps. Its first step could be to open the application. Its second step could be to enter a user name. Its third step could be to enter a password, and its last step could be to click the Submit button on the Web page. By creating and calling functions stored in function libraries, you can enhance the component with additional logic to test important details of the login task.

By design, each component tests a specific part of an application. When combined, components are incorporated into a business process test in a serial flow representing the main tasks performed within a particular business process. For example, a business process test for a flight reservation application may include a login component, a flight finder component, a flight reservation component, a purchasing component, and a logout component. The flight finder, flight reservation, and purchasing components might be reused several times within the same business process test to test multiple reservation scenarios. The test might also include a component that resets the application between flight reservations, enabling the test to perform multiple iterations of flight reservations. The task of creating and running components and business process tests is generally performed by Subject Matter Experts working in Quality Center.

Due to the modularity and reusability of components, they can be used in multiple business process tests. For example, the same login and logout components could be used in conjunction with an analysis (report) component that tests the report and graph generation process in the application, or with a frequent flyer component that tests the business process of subscribing to a frequent flyer program.

QuickTest provides two types of components: business components and scripted components. Business components (also known as keyword-driven components) are fully integrated with both QuickTest and Quality Center, enabling both you and Subject Matter Experts to create, modify, and run them. Scripted components are more complex components containing programming logic. Due to their complexity, scripted components can be created and modified only in QuickTest. Subject Matter Experts can view scripted components in Quality Center and incorporate them in business process tests, but they cannot modify them.

Note: Although you can also use QuickTest to create scripted components for use in business process tests, this help focuses on the functionality and features associated with business components. For information on the differences between scripted components and business components, as well as information on working with scripted components.

Before automated testing resources are available, Subject Matter Experts can define manual steps in the Design Steps tab of each component (using the Quality Center Business Components module). They can add these manual components to a business process test and run the steps manually using the Quality Center Manual Runner. As they define components, Subject Matter Experts can add comments in the Discussion Area of the Details tab (in the Quality Center Business Components module). This enables them to enter any additional information or remarks that they want to communicate to you, the Automation Engineer, such as requests for new operations, future changes planned for the component, or alternative tests in which the component can be used.

During this design phase, you can work with the Subject Matter Experts to define which resources and settings are needed for each component. You can then create individual application areas for the various parts of your application based on real testing needs. The application area specifies the settings and resource files used by components when working with business process tests. When a Subject Matter Expert creates a component, the component is always associated with a particular application area, enabling it to access these settings and resource files. After you create the application area and define its settings and resource files, the Subject Matter Expert can incorporate these automated testing resources in business component steps, convert any existing manual components to automated components, and create new automated components.

QuickTest Professional for Business Process Testing

Business Process Testing is a role-based testing model. It enables Automation Engineers and Subject Matter Experts to work together to test an application's business processes during the application's development life cycle.

Automation Engineers are experts in automated testing. They use QuickTest to define the resources and settings needed to create components, which are the building blocks of business process tests.

Subject Matter Experts understand the various parts of the application being tested, as well as the business processes that need to be tested, however they may not necessarily have the programming knowledge needed to create automated tests. They use the Business Components and Test Plan modules in Quality Center to create keyword-driven business process tests.

Integration between QuickTest and Quality Center enables the Automation Engineer to effectively create and maintain the required resources and settings, while enabling Subject Matter Experts to create and implement business process tests in script-free environment, without the need for programming knowledge.

Note: Each organization defines the roles of Automation Engineer and Subject Matter Expert according to its needs. This help assumes that you are performing the role of the Automation Engineer as defined above, and that the role of Subject Matter Expert is performed by other personnel in your organization. However, these roles are flexible and depend on the abilities and time resources of the personnel using Business Process Testing. There are no product-specific rules or limitations controlling which roles must be defined in a particular organization, or which types of users can do which Business Process Testing tasks (provided that the users have the correct permissions).

Handling Run Errors in QTP

The Run Error message box displayed during a run session offers a number of buttons for dealing with errors encountered:

• Stop—Stops the run session.
  The run results are displayed if QuickTest is configured to show run results after the run.
• Retry—QuickTest attempts to perform the step again.
  If the step succeeds, the run continues.
• Skip—QuickTest skips the step that caused the error, and continues the run from the next step.
• Debug—QuickTest suspends the run, enabling you to debug the component and any associated function library that contains a function called by the component.

You can perform any of the debugging operations described in this section. After debugging, you can continue the run session from the step where the component or function library stopped, or you can use the step commands to control the remainder of the run session.

• Help—Opens the QuickTest troubleshooting Help for the displayed error message. After you review the Help topic, you can select another button in the error message box.
• Details—Expands the message box to display additional information about the error.

How to use Watch in QTP??

You can view the current value of any variable or VBScript object in your function library by adding it to the Watch tab. As you continue stepping into the subsequent steps in your function library, QuickTest automatically updates the Watch tab with the current value for any object or variable whose value changes. You can also change the value of the variable manually when the function library pauses at a breakpoint.

To add an expression to the Watch tab:

Perform one of the following:

• Click the expression and choose Debug > Add to Watch
• Click the expression and press Ctrl+T
• Right-click the expression and choose Add to Watch from the context menu
• In the Watch tab, paste or type the name of the object or variable into the Name column and press Enter to view the current value in the Value column

Note: You can add an expression to the Watch tab from a function library (and not from a business component).

Tip: You can also use the Variables tab to view the current values for all variables up to the current step in the function library.

Enabling and Disabling Breakpoints

You can instruct QuickTest to ignore an existing breakpoint during a debug session by temporarily disabling the breakpoint. Then, when you run your component or function library, QuickTest runs the step containing the breakpoint, instead of stopping at it. When you enable the breakpoint again, QuickTest pauses there during the next run. This is particularly useful if your component or function library contains many steps, and you want to debug a specific part of it.

You can enable or disable breakpoints individually or all at once. For example, suppose you add breakpoints to various steps throughout your component or function library, but for now you want to debug only a specific part of your document. You could disable all breakpoints in your component or function library, and then enable breakpoints only for specific steps. After you finish debugging that section of your document, you could disable the enabled breakpoints, and then enable the next set of breakpoints (in the section you want to debug). Because the breakpoints are disabled and not removed, you can find and enable any breakpoint, as needed.

An enabled breakpoint is indicated by a filled red circle icon in the left margin adjacent to the selected step.

A disabled breakpoint is indicated by an empty circle icon in the left margin adjacent to the selected step.

Note: Breakpoints are applicable only to the current QuickTest session and are not saved with your component or function library.

You can:

• Enable/disable a specific breakpoint
• Enable/disable all breakpoints

To enable/disable a specific breakpoint:

1. Click in the line containing the breakpoint you want to disable/enable.
2. Choose Debug > Enable/Disable Breakpoint or press Ctrl+F9. The breakpoint is either disabled or enabled (depending on its previous state).

To enable/disable all breakpoints:

Choose Debug > Enable/Disable All Breakpoints or click the Enable/Disable All Breakpoints button. If at least one breakpoint is enabled, QuickTest disables all breakpoints in the component or function library. Alternatively, if all breakpoints are disabled, QuickTest enables them.

Using Breakpoints

You can use breakpoints to instruct QuickTest to pause a run session at a predetermined place in a component or function library. QuickTest pauses the run when it reaches the breakpoint, before executing the step. You can then examine the effects of the run up to the breakpoint, make any necessary changes, and continue running the component or function library from the breakpoint.

You can use breakpoints to:

• suspend a run session and inspect the state of your site or application
• mark a point from which to begin stepping through a component or function library using the step commands

You can set breakpoints, and you can temporarily enable and disable them. After you finish using them, you can remove them from your component or function library.

Note: Breakpoints are applicable only to the current QuickTest session and are not saved with your component or function library.

Debugging Components and Function Libraries

In QuickTest, after you create a component or function library (including registered user functions), you should check that they run smoothly, without errors in syntax or logic. To debug a function library, you must first associate it with a component via its application area and then debug it from that component.

To detect and isolate defects in a component or function library, you can control the run session using the Pause command as well as various step commands that enable you to step into, over, and out of a specific step.

You can use the Start from Step command to begin your debug session at a specific point in your component. You can also use the Run to Step command to pause the run at a specific point in your component. You can set breakpoints, and then enable and disable them as you debug different parts of your component or function library.

When the component or function library run stops at a breakpoint, you can use the Debug Viewer to check and modify the values of VBScript objects and variables. Also, if QuickTest displays a run error message during a run session, you can click the Debug button on the error message to suspend the run and debug the component or function library.

You can also use the Run from Step command to run your component or function library from a selected step to the end. This enables you to check a specific section of your application or to confirm that a certain part of your component or function library runs smoothly.

Exporting Test Results from QTP

You can export the run results from the Test Results window to an HTML file. This enables you to easily view the run results when you are not in a QuickTest environment. For example, you can send the HTML file containing the run results in an e-mail to a third-party who does not have QuickTest installed. You can select the type of report you want to export, and you can also create and export a customized report.

To export the run results:

1. Choose File > Export to HTML File. The Export to HTML File dialog box opens.
2. Select an Export range option:
• All—Exports the results for the entire component.
• Selection—Exports run result information for the selected branch in the test results tree.
3. Select an Export format option:
• Short—Exports a summary line (when available) for each item in the test results tree. This option is only available if you selected All in step 2.
• Detailed—Exports all available information for each item in the test results tree, or for the selected branch, according to your selection in step 2.
• User-defined XSL—Enables you to browse to and select a customized .xsl file. You can create a customized .xsl file that specifies the information to be included in the exported report, and the way it should appear. For more information, see Customizing the Test Results Display.

Note: The Export format options are available only for run results created with QuickTest version 8.0 and later.

4. Click Export. The Save As dialog box opens, enabling you to change the default destination folder and rename the file, if required. By default, the file is named <name of component> [<name of run results>], and is saved in the run results folder.
5. Click Save to save the HTML file and close the dialog box.