Results

Run Results reports what happened in a run, at test suite, test case, and machine level. Drill into a step to read its analysis, check its logs, and compare it against an earlier run.

Prerequisites

• A test plan has to have completed one run.

Find a run

The Run Results page lists every run. Search by name, or click Sort by to order the list by Title, Created Date, Updated Date, or Last Run, ascending or descending. Show Filters filters by Created By, Last Run Date, Created Date, Updated Date, and Labels.

Pick a level

Results open at test suite level. The dropdown switches between Test Suites, Test Cases, and Test Machines.

• Test Suites: expand a suite to see the test case results inside it. The parallel indicators on each row show how many suites run in parallel, and how many test cases run in parallel within each suite
• Test Cases: every test case from every suite, in one list. Hover over a failed one for a brief reason
• Test Machines: every machine in the plan. Click one to see the suites it contains

Tip

To see the test cases in a machine, go to the Test Suites view and apply a Test Machine filter.

Filter within a run

Filters apply in 3 places on an open run.

• The Test Runs panel takes its own filters, through the Filters icon on the panel
• The test list takes filters at whichever level the dropdown is set to, through the Filters icon
• The donut chart in Run Overview filters by status when you click a segment, on top of any filter already applied

Read a test case result

Click a test case to open it. The header above the step list shows the total run duration, the browser version, and the operating system version. The step list sits on the left, headed by the step count, and each step shows its status icon. Select a step to open its details on the right, and drag the divider to resize either pane.

A healed step also shows the auto-heal icon at the end of its row. When a run contains healed steps, a banner above the step list shows how many, with thumbs up and thumbs down icons to rate the heal. See Healer.

The step header shows the step status, start time, duration, step number, and step action. For a failed step, the ERROR and ANALYZER VERDICT sections sit below it. See Debug.

The THIS RUN section shows the screenshot captured for the step. The expand icon shows it full size, and the download icon saves it. Below the screenshot, the step details list:

Field	What it shows
Element	The element the step targets. A step without one reads This step targets no element
Locator	The locator type and value used for the element
Run Type	The type of run that produced this result
Test data type and Test data	The data type and the value passed to the step
Start time and Duration	When the step started and how long it ran
Step level timeout and Plan level timeout	The wait limits applied to the step
Error code and Error message	The error raised during execution, if any

Two more controls sit in the step header:

• Affected tests opens Affected Instances, with counts across Test Cases, Step Groups, Test Suites, and Test Plans. This is what the element in this step is used by
• More details opens Test Case Overview, with the step breakdown across Passed, Failed, Not Executed, and Stopped, plus the run message and Machine Details

Step details panels

The icons in the step header open side panels with more about the selected step. Click the close icon to return to the step details.

Panel	What it contains
Locators	The locator search for a healed step. Appears on healed steps only. See Read the locator trace
Logs	Console and network logs, for the step or a wider scope, with a download option
Step Settings	Maximum wait time, prerequisite, and whether the step stops the test case on failure, is ignored in the test case result, or uses visual testing, Smart Execution, or accessibility testing
Metadata	The step’s test data and type, its ID and action, whether it uses a password, whether it was migrated, any additional data, and its step type and priority

See Debug for reading the logs and investigating a failure. For a root cause explanation and suggested fixes, click Analyze with Agent in the action bar. See Analyzer.

Compare two runs

Comparing runs answers the question a single result cannot: what changed since this test last worked.

1. Click Compare Runs on the Test Case Results page.

   

2. Select a run from the list at the top of either panel.

Each panel shows the run’s duration, date, step count, and failed count, then every step with its own duration. The comparison spans the steps either side of the failure as well, because the step that reports an error is often not the step that caused it.

Note

The run list includes Authoring Run, holding the values captured when the test case was authored, and each earlier run by its run ID. Every entry shows its run type and its status as Passed, Failed, or Healed, so you can pick the last run that behaved as expected without opening runs one by one.

Compare a single step

1. In the THIS RUN section of the step, click Compare Steps. The step under comparison appears in the breadcrumb.

   

   The left panel defaults to Authoring Time. The right panel shows the current run with its date, time, and status, such as Passed or Healed. Select a different run from the list at the top of either panel to change the comparison.

2. Select a mode from the View list.

   Mode	What it shows
   Compare with Baseline	The baseline run and the current run side by side
   Overlay Wipe	Both screenshots in one frame, with a divider to wipe between them
   Current Step Only	The current run’s screenshot on its own
   Element Source	A diff of the captured page source across the 2 runs, with the locator value for each

   Overlay Wipe loads the authoring capture and the execution capture into a single frame separated by a vertical divider. Drag the handle to wipe between them. Stacking the 2 makes a shifted control, a modal that did not close, or a reflowed layout visible at once, rather than something to find by comparing 2 panels.

   

3. Click Screens for screenshots only, or Screens + Details to show the step details beneath each one.

   Screens + Details lists the run type, start time, duration, step level timeout, plan level timeout, error code, error message, element name, and locator for each run. A value that was not captured shows a dash.

   Note

   Screens + Details is disabled in Overlay Wipe, since that mode renders one combined frame rather than 2 panels.

4. Select a step from Change Step to compare a different one. Each entry shows the step number, name, duration, and status, including Not Executed for steps the run never reached.

   

5. Click the download icon on a panel to save that screenshot, or the download icon at the top right to save the full comparison.

Note

Where a step targets no element, the left panel reads No recorded evidence captured for this step.

Run overview

The Run Overview section summarizes the selected run at every level, showing the Last Run, Test Plan Settings, and Accessibility Test Overview.

View Runs lists all runs, More Settings opens the additional settings, and View Report opens the accessibility testing report.

Rerun from a result

Select a run in the Test Runs panel to see its results.

1. Click Rerun in the top-right corner.

2. Select a rerun option:

   • All Test Cases: reruns every test case in the selected run
   • All Failed Test Cases: reruns only the test cases that failed in the selected run
   • Select Cases for Re-Run: reruns only the test cases you select

3. Click Start execution.

A run ID has a maximum rerun limit of 10.

When a test suite or a test machine has a prerequisite, its Rerun options setting decides what happens to the prerequisite during a rerun. Always run Pre-Requisite runs the prerequisite again in full before the failed test cases. Only execute failed Pre-Requisite iteration(s) runs only the prerequisite iterations that failed, so passed test cases are not run again. Set it on the test suite or the test machine.

Watch the execution video

A video of the run plays on the test case result, so you can watch what the test did rather than infer it from screenshots. Open the test case result and play the video alongside the step list.

Reach it from Ad-Hoc Runs in the utility panel as well, through View Details on the run.

Note

A run still in progress has no video. Headless execution records none at all, and a test plan set to Reset session for every test case produces no single video for the plan.

Export a report

Click the Export icon and select PDF, MS Excel, or JUnit. A PDF opens a dialog for its screenshot options, JUnit downloads immediately, and Excel arrives as an emailed link.

An exported report carries the names things had when the run executed, so exporting the same run twice reproduces the same document. See Reports for each format in full, and for the Allure and custom PDF reports generated outside the application.

Keyboard shortcuts

Action	macOS	Windows
Test case level	Option + C	Alt + C
Test suite level	Option + S	Alt + S
Test machine level	Option + M	Alt + M
Run history	Option + H	Alt + H
Filters	Option + F	Alt + F
Run overview	Option + O	Alt + O

Renamed and deleted items

The Run Results page shows values as they were when the run executed. Renaming or deleting a test plan, test suite, test case, environment, test machine, or test data profile does not change what an earlier run displays.

User names are the exception. A report always shows a user’s current name, so renaming a user updates reports that already exist.

Note

Runs that executed before this release fall back to each item’s current name, and show a hyphen where the item has since been deleted.