Test Evidence – What it is, Why & How to Capture?

Building confidence in any endeavor hinges on solid proof. In the world of software development, that proof comes in the form of test evidence. It’s not just about checking boxes; it’s about capturing concrete data and observations that verify your software’s quality, performance, and security. 

This evidence lays the foundation for stakeholder trust, fuels continuous improvement, and protects user experience. So, how do you gather this crucial evidence?

Let’s dive into the what, why, and how of an effective test evidence. Also, you can find out how automation can play a role in this.

What is Test Evidence?

Test evidence is the necessary foundation of high-quality software. It comprises the documentation and artifacts generated throughout the testing process, serving as absolute proof of executed tests and their outcomes. 

This includes 

• test plans, 
• reports, 
• defect logs, 
• code coverage data, 
• automated scripts, and 
• screenshots.
  

By providing unparalleled transparency, traceability, and accountability, test evidence empowers all stakeholders – developers, testers, and managers – to build robust, reliable software. 

Imagine a scenario where, during regression testing, carefully documented test evidence revealed a possible security vulnerability. Thanks to this evidence, swift action was taken to prevent a catastrophic data breach. 

Why the need for Test Evidence?

You can ask: Is keeping a record of the tests conducted an essential part of software testing? Of course, Yes!

It helps to keep track of the steps taken and the outcome of the test. This documentation helps to identify and fix any issues, thus ensuring that the software is reliable and functions as expected. 

It is also essential for compliance purposes, as it helps to show that the testing was done according to the established standards and regulations. 

Having clear and reliable test evidence enhances the credibility of the testing process

5 Approaches Implemented for Testing and Collecting Test Evidence.

Here are the approaches you need to know:

1. Manual Recording 

This classic method involves testers capturing screenshots, videos, and logs manually during testing. It’s straightforward and requires minimal setup, but it can be time-consuming, prone to errors, and needs more consistency. Think of it as jotting down notes in a paper notebook.

Example: Consider a scenario where you are evaluating a mobile application through manual recording of screenshots. Every action within the user flow is documented by images that capture the registration process, login screens, and all features of the application. Instead of fully automated documentation, this semi-automated approach allows visual representation at every stage— to capturing individual moments with a camera shutter click after click.

2. Semi-Automated Recording 

Here, tools assist testers by automating repetitive tasks like capturing screenshots or recording video steps. This frees testers to focus on complex interactions while controlling key moments. Imagine using a voice recorder to capture your testing narrative alongside automated screenshots.

In a case where: Testing a web application using automated test tools that are able to log all of your activities with the UI during the test. When you move from one page to another or carry out actions such as filling out forms or clicking buttons, it takes screenshots and saves them while keeping a record of what you have done. 

At the same time, you vocally explain your testing strategy, which is captured together with those automatic snapshots. In this way, you can concentrate on important interactions instead of boring capturing tasks.

3. Automated Testing 

This technique uses pre-written scripts and triggers to run tests automatically. It is very effective and reduces the chances of human error. The downside is that it demands an initial investment in script development plus an inflexibility in handling unanticipated situations.

Let’s understand this with the help of low-code test automation platform Testsigma. It allows testers to develop and handle automated tests even when they lack considerable programming background. Here, you can easily create test scripts by recording actions and using pre-built functions, allowing the tests to be more resilient to change in a simpler way.

So we can say for example: When a shopping website is being tested, automated tests are used to mimic real user actions. These tests involve adding items to a virtual cart, going through the checkout process, and completing a payment. These tests are designed to run automatically after any website changes to ensure that the website performs consistently. The automated checks simulate a mechanical being following specific steps to make sure the website functions properly under normal conditions.

With Testsigma, you can configure the tests to record screenshots, videos and logs for any test that is executed – they can also be selected for tests that pass or fail.

4. Backend Snapshot 

Rather than stressing the user interface, this approach tracks information as well as any modifications in the system backend. This feature is highly useful for detecting performance issues or understanding the internal workings of an application— it gives a view into the working of the software like an X-ray would reveal underlying details.

Let us say: Suppose you are testing a cloud-based application that handles user data. During stress testing, you capture snapshots of the database’s state at various load levels. You monitor how the system responds to increased user activity, capturing metrics like database query times and server CPU usage. This backend-focused approach helps identify performance bottlenecks and ensures data consistency, much like examining internal mechanisms without directly interacting with the user interface.

5. Application Instrumentation 

The innovative approach of embedding sensors in the code allows monitoring of events and metrics during execution at a microscopic level. The work becomes easier now: uncovering the hidden reasons behind poor performance or inaccessible bugs would be similar to planting tiny cameras within the software to watch every action closely.

Example: Let’s say you are a developer working on a gaming app. You would be embedding code that tracks gameplay metrics: the code watches players’ completion of levels, spots bug locations based on player actions, and determines playtime at various stages. 

All these data points are collected in real-time and sent back to your development team for analysis— almost as if tiny sensors were being placed within the software itself, providing keen observations into user behavior as well as application effectiveness (or lack thereof) without direct observation. This enables quality enhancements through feature updates or bug fixes based on what is revealed by these details.

💡Remember, the best approach depends on your project’s needs and resources. Consider combining these methods for a complete and efficient testing strategy!

How do you capture Test Evidence?

The method for capturing test evidence will vary depending on the chosen approach (manual, semi-automated, etc.) and the types of evidence you need. 

However, there’s a general step-by-step guide to provide a framework:

1. Define evidence requirements:

• Identify: What types of evidence are needed for your project? (e.g., screenshots, videos, logs, code coverage reports)
• Target: What specific interactions or scenarios should be captured?
• Format: How should the evidence be stored and documented? (e.g., naming conventions, file formats)
  

2. Choose your tools:

• Manual: Screenshot tools, video recording software, note-taking apps
• Semi-Automated: Testing frameworks with screenshot/video capture features
• Automated: Test automation frameworks with integrated evidence collection
• Backend Snapshots: Monitoring tools for database changes, API logs
• Application Instrumentation: Code profiling tools, tracing frameworks
  

3. Configure evidence capture:

• Set triggers: Specify when and how evidence should be automatically collected (e.g., on test failure, specific user actions)
• Customize templates: Define consistent naming and formatting for your captured evidence.
• Integrate with testing workflow: Ensure capturing evidence happens seamlessly within your testing process.
  

4. Execute tests and monitor evidence:

• Run tests: Manually or using automation frameworks
• Observe evidence collection: Ensure tools are functioning correctly and capturing the required data
• Make adjustments: Adapt evidence capture as needed based on testing needs.
  

5. Organize and document evidence:

• Store: Securely store your captured evidence in a centralized location (e.g., test management tool, cloud storage)
• Label and categorize: Organize evidence for easy retrieval and traceability
• Document findings: Associate evidence with test cases, defects, and relevant observations.
  

6. Review and utilize evidence:

• Analyze evidence: Review captured data to identify bugs, track progress, and assess test coverage.
• Share evidence: Communicate findings with stakeholders using clear and concise reports.
• Improve testing: Leverage evidence to refine test cases, optimize testing strategies, and enhance software quality.
  

By following these steps and adapting them to your specific needs, you can effectively capture valuable test evidence that supports robust software development and quality assurance.

How was it Tested?

When something goes wrong with software and a bug slips into production, it’s easy to get defensive or blame others. But that doesn’t help anyone. 

Instead, everyone involved – developers, testers, and stakeholders – should focus on understanding “how it was tested.” That means going beyond just saying whether a test passed or failed. 

We must dig into the details, explore edge cases, document unexpected behaviors, and capture data like logs and screenshots. 

We also need to foster a transparent and collaborative environment where everyone can share their observations and concerns. By doing this, we can turn “how it was tested” from a negative question into a tool for continuous improvement. 

The Importance of Artifacts in Test Evidence

Test plans, scripts, and cases might not be the starring act, but they’re the crucial scriptwriters! While execution reveals the drama, these artifacts define the plot, characters, and twists. 

Understanding how tests will be done guides evidence collection, ensuring we capture the right scenes and clues. 

They pave the way for meaningful analysis, letting us interpret the execution’s performance and refine the testing play for future acts. So, embrace these artifacts as the unsung heroes of your testing evidence saga.

Build a Body of Testing Evidence

Building a robust body of evidence is a collaborative act. It requires collaboration:

• Planners craft clear blueprints (test plans) for the testers to follow.
• Testers diligently record every interaction, capturing the performance with screenshots and logs and documenting their observations.
• Analysts interpret the data, identify patterns and insights, and share their findings to improve future performances.

Don’t Go It Alone

Collaboration is paramount in building a robust body of testing evidence. Testing is a multifaceted process that benefits from diverse perspectives and skill sets. Teams should foster an environment where collective insights and experiences contribute to the testing strategy. 

Collaboration not only helps identify potential blind spots but also enhances the overall effectiveness of the testing effort. 

• Engaging team members, 
• stakeholders, and 
• subject matter experts ensure a complete approach, 
• elevating the quality of the testing evidence and 
• reducing the risk of oversight.

Record Everything

Thorough documentation is the backbone of a comprehensive testing evidence repository. Every aspect of the testing process, from test plans to execution details, defects, and outcomes, should be meticulously recorded. This includes manual steps, automated scripts, and any observations made during testing. 

By maintaining detailed logs, screenshots, videos, and reports, teams create a traceable and transparent record of the testing journey. 

This documentation not only serves as proof of testing activities but also aids in 

• analyzing, 
• debugging, and 
• improving software quality throughout the development life cycle.

Other Forms of Evidence

While documentation is crucial, testing evidence extends beyond traditional records. Other forms of evidence may include

• user feedback, 
• performance metrics, and 
• system logs. 

User feedback provides valuable insights into real-world user experiences, helping identify usability issues and areas for improvement.

Performance metrics offer quantitative data on system responsiveness and scalability. 

System logs, especially when capturing backend processes, contribute to a more holistic understanding of the application’s behavior. 

Incorporating these diverse forms of evidence enriches the testing body, offering a more nuanced and complete picture of the software’s performance and quality.

Tools for test evidence details

While capturing and managing test evidence is crucial for software quality assurance, choosing the right tool depends on your specific needs and the types of testing you perform. 

Here’s a breakdown of the different tools:

Testsigma: 

Primary function: The test automation platform Testsigma is able to automate end-to-end UI testing for web, mobile and desktop applications, and APIs, without the need to write any code.

Test evidence: During the execution of tests, Testsigma takes screenshots along with logs, videos and complete reports.

Features: Cloud-based execution is brought together with AI-powered maintenance and collaborative test management within Testsigma.

Strength: A practical approach towards test automation that requires minimal scripting yet fits various needs typically associated with testing web applications.

TestRail:

• Primary function: Test case management and execution platform.
• Test evidence: Focuses on organizing and documenting evidence after testing. This includes screenshots, logs, defect reports, and comments associated with test cases.
• Features: TestRail allows storing evidence files, attaching them to test cases or runs, and linking them to defects for traceability. Users can also write detailed notes and observations within the platform.
• Strength: Provides a central repository for managing and analyzing test evidence across different projects and teams.
  

Selenium:

• Primary function: Framework for automating web browser interactions.
• Test evidence: Captures evidence dynamically during test execution. This includes screenshots, page sources, console logs, and JavaScript errors.
• Features: Selenium offers various libraries and plugins for capturing screenshots, recording videos, and accessing browser logs. Developers can integrate these features into their automated test scripts.
• Strength: Automates evidence capture, reducing manual effort and ensuring consistent data collection.

Appium:

• Primary function: Framework for automating mobile app testing.
• Test evidence: Like Selenium, Appium captures evidence during test execution, mainly screenshots, app logs, and device logs.
• Features: Appium leverages device capabilities and platform-specific libraries to capture mobile-specific evidence like network logs and crash dumps.
• Strength: Provides automated evidence capture specifically for mobile environments.

Conclusion

The Final Brick – Test Evidence!

Test Evidence is not just a bunch of pics and logs. It’s more like a colorful tapestry woven from every move, every watch, every win, and every failure during the testing journey. 

It’s like a report that tells you how it was tested, not just in the good things, but in the tiny details of the process, the little warnings of risks, and the surprises of unexpected behavior.

We must embrace the whole picture, including what worked and what didn’t. This way, we can learn and improve our software quality and development process.

Frequently Asked Questions

Is test evidence necessary in Agile?

Yes, test evidence is crucial in Agile! Even with fast iterations, understanding “how it was tested” remains vital. It’s like leaving a trail of breadcrumbs for future improvements.

Without evidence, it’s like navigating a maze blindfolded. You might stumble upon solutions, but replicating success becomes a guessing game.