In this month&#8217;s release, our product team has packed some cool new features like GraphQL support, native AWS integration, and testing capabilities to better simulate real-world usage conditions in an effort to continue to revolutionize your testing experience.&nbsp;



Let&#8217;s dive into the details:



<ol><li>Optimized <a href="https://testsigma.com/blog/test-graphql/" target="_blank" rel="noreferrer noopener">GraphQL testing</a> (Beta): Say goodbye to overfetching and underfetching. Testsigma&#8217;s support for GraphQL simplifies testing while enhancing accuracy. It lets you request multiple resources in a single query call, reducing back-and-forth between your app and the server. This optimization not only saves time but also minimizes bandwidth usage.</li></ol>



<ol start="2"><li>Native AWS Integration: Seamlessly merge your testing and deployment efforts with our AWS integration. Our native integration allows you to trigger Testsigma Test plans within your CI/CD pipeline, ensuring a smooth workflow based on Test Plan Results.</li></ol>



<ol start="3"><li>Network Throttling: Experience your website&#8217;s performance under real-world conditions of limited bandwidths and network speeds. Simulate scenarios with low network speeds and analyze your website&#8217;s performance to ensure optimal user experience, even in challenging network environments.</li></ol>



<ol start="4"><li>Geolocation Testing: Accurate and contextually relevant translations, culturally appropriate content, functional adaptability and more, localization features enable you to address diverse user preferences and languages. With our Geolocation testing capabilities, you can effortlessly test these features for your web and mobile apps, ensuring quality releases for users across the globe.&nbsp;</li></ol>



Stay tuned for more updates from our Product Team. And if you haven&#8217;t experienced the effect that Testsigma can have on your release velocity, <a href="https://testsigma.com/signup">try it for free</a>.

August ’23 Product Digest The latest on Testsigma

August ’23 Product Digest: The latest on Testsigma

We&#8217;re excited to bring you some amazing new features in this month&#8217;s release. From desktop application recording to on-demand partial test runs and seamless integrations with popular CI/CD tools, we are introducing new features and capabilities to enhance your test automation and streamline your QA processes.&nbsp;



Let&#8217;s dive in:



1. Rapidly automate Windows desktop app testing with our Desktop Recorder: Our new desktop recorder allows you to record and generate test cases for Windows apps and effortlessly add new UI elements to your Windows projects. 



2. Test hotfixes quickly with partial test plan execution: No need to re-run entire test plans after hotfixes. With Testsigma, you can now execute partial runs of test plans or test plan subsets for when you need quick results for that bug fix on release days.



3. Build automation made easy with Jenkins Integration: Integrate Testsigma into your CI/CD pipeline for smooth build automation and continuous testing. Our Jenkins integration allows you to run tests and get results at every merge without leaving your pipeline.&nbsp;



4. Unify your test reporting with Azure DevOps Integration: Report and maintain your tests in a single place with Azure DevOps integration. Access all your test reports and results from your Azure DevOps dashboard. Push bugs, epics, tasks, or stories directly from Testsigma while testing, streamlining your workflow.



Stay tuned for more updates from our Product Team. And if you haven&#8217;t yet, <a href="https://testsigma.com/signup">try Testsigma for free</a> and see the difference it makes to your SDLC and release quality.



<div class="cta-banner-v2">
<div class="title-with-description ">

Automate your tests for Web, Mobile, Desktop and APIs on Cloud with Testsigma. No setup required.
</div>
<a href="https://testsigma.com/signup" target="_blank" class="cta-btn-v1" rel="noopener">Try for free</a>
</div>

July ’23 Product Digest: The latest on Testsigma

This month, our team has introduced to Testsigma a range of innovative features that will revolutionize the way you approach testing.&nbsp;



Let&#8217;s dive right in:&nbsp;



1. Embrace targeted testing by using For Loop to filter Test Data Profiles: With the new Test Data Profile Filtering in <a href="https://testsigma.com/docs/test-cases/step-types/for-loop/">For Loop</a> feature, you can effortlessly run repetitive actions on a selected subset of your Test Data profile. Say goodbye to the unnecessary hassle of testing through the entire test data profile and embrace targeted testing like never before.



2. Develop deterministic tests with Mobile Debugger [Beta]: With <a href="https://testsigma.com/docs/debugging/results-on-local-devices/">Testsigma&#8217;s Mobile Debugger</a>, you can now debug issues on your local mobile devices step-by-step. Run your tests in Debug mode, and you&#8217;ll be able to execute them live, closely analyze each step, locate issues, and fix them in real-time—all in a single session. Gain 100% confidence in the accuracy of your test authoring and eliminate flakiness from your automated testing.



3. Execute Mobile Test Suites in Parallel: With<a href="https://testsigma.com/docs/test-management/test-plans/run-tests-in-parallel/"> parallel testing</a>, you can run tests on multiple OS and device combinations simultaneously, multiplying your test execution capacity and significantly reducing the time needed to complete your testing cycles.



4. Jira-Xray Integration: If your team relies on Jira Xray for release reporting and planning, Testsigma&#8217;s <a href="https://testsigma.com/docs/integrations/product-management/xray/">Jira-Xray Integration </a>will make it a breeze to push execution information to Xray with just a flip of a switch. This lets you seamlessly bridge the gap between testing and release management, and experience a smoother and more streamlined testing process.



With these new features, we&#8217;re continuing to empower you to achieve optimal testing efficiency and accuracy. If you&#8217;re not already a Testsigma user,&nbsp;



<button class="btn bg-primary cta-btn" style="margin: auto"><a href="https://testsigma.com/signup" target="_blank" rel="noopener noreferrer" style="color:#fff">Sign up for a free trial</a></button>



And embrace effortless automated testing with Testsigma.&nbsp;

June '23 Product roundup - latest on Testsigma

June ’23 Product roundup – latest on Testsigma

When it comes to software delivery, automation is directly proportional to efficiency. With its ability to automate building, testing, and software delivery, Jenkins is the go-to choice for CI/CD pipeline automation for many teams. With low-code end-to-end test automation with Testsigma and our native Testsigma-Jenkins integration, you can incorporate your simple English tests—for web, mobile apps, and APIs—into your build automation workflows—to test continuously and at scale.



Here&#8217;s how you can get started:



<ol><li>Installation and Setup: To get started, you&#8217;ll need to install the Testsigma Test Plan Run plugin in your Jenkins instance. You can find the plugin on the <a href="https://plugins.jenkins.io/testsigma/" target="_blank" rel="noreferrer noopener">Jenkins Plugins page </a>or download it directly from the Testsigma website.&nbsp;</li></ol>



Once installed, you&#8217;re ready to integrate Testsigma tests with Jenkins.



<ol start="2"><li>Configuring the Jenkins Pipeline: Add the <a href="https://testsigma.com/docs/test-management/test-plans/schedule-plans/" target="_blank" rel="noreferrer noopener">Testsigma Test Plan Run</a> plugin as a stage to trigger your Testsigma test plans:<ol><li>Click on the &#8220;Build&#8221; tab within your pipeline configuration.</li><li>Click on &#8220;Add Build Step&#8221; and select &#8220;Testsigma Test Plan run.&#8221;</li><li>Configure the plugin by providing your Testsigma API Key, Test Plan ID, maximum wait time for task completion, and the file path for the JUnit report XML file.</li><li>Save the changes to complete the configuration.</li></ol></li></ol>



<ol start="3"><li>Testing and Execution: Once your pipeline is set up, trigger a build to see the build verification in action. Check the console log for the execution details, and for more comprehensive results, log into your Testsigma account and deep dive into the test plan status.&nbsp;</li></ol>



With Jenkins&#8217; automation capabilities and Testsigma&#8217;s simplified testing, you automate your testing pipeline to maximize efficiency and deliver high-quality software with confidence. Automate your testing pipeline today and experience the benefits of a smoother, more efficient release workflow with Testsigma.



<div class="cta-banner-v2">
<div class="title-with-description ">
Build and run tests for Web, Mobile, Desktop and APIs under one platform with Testsigma.
</div>
<a href="https://testsigma.com/signup" target="_blank" class="cta-btn-v1" rel="noopener">Try for free</a>
</div>

Embrace Effortless Pipeline Automation with Testsigma-Jenkins Integration

Effortless Pipeline Automation with Testsigma-Jenkins Integration

Engineered with a focus on testing efficiency, our product team has introduced some really cool features this month. Let’s dive right in!



<ol><li>Optical Character Recognition (OCR) for Advanced Text Extraction: The HTML canvas element is used to draw graphics, on the fly, via scripting. But during test automation of such DOM elements, users can neither rely on coordinates-based action nor on visual testing since it varies over screen sizes and does not assert any element inside the canvas image. With Testsigma&#8217;s <a href="https://testsigma.com/docs/addons/ocr-extraction/" target="_blank" rel="noreferrer noopener">OCR-based text extraction </a>via add-ons, you can now extract content from inside the canvas tag. This powerful feature allows you to interact with any text/components inside the canvas image. Read here &#8211; <a href="https://testsigma.com/blog/ocr-test/" target="_blank" rel="noreferrer noopener">OCR Test</a>.</li></ol>



<ol start="2"><li>Disable Test Cases in a Test Plan: To address the challenges faced when running known-to-fail test cases, Testsigma now allows users to <a href="https://testsigma.com/docs/test-management/test-plans/disable-test-cases-in-test-plans/" target="_blank" rel="noreferrer noopener">selectively disable test cases </a>within a test plan. By disabling these cases, you save valuable execution time and achieve a result decoupled from the known failures, ensuring efficient and accurate testing.</li></ol>



<ol start="3"><li>Support for Flutter Apps: Flutter is a crowd favorite for cross-platform app development, and Testsigma now supports recording and execution for apps built using the <a href="https://testsigma.com/tutorials/test-cases/mobile-apps/build-tests-using-recorder/" target="_blank" rel="noreferrer noopener">Flutter framework</a>. Seamlessly record and execute tests on your Flutter mobile apps, enabling comprehensive testing for both iOS and Android platforms.</li></ol>



<ol start="4"><li>Support for Shadow DOM Element Capturing and Execution: With <a href="https://testsigma.com/docs/elements/web-apps/shadow-dom/" target="_blank" rel="noreferrer noopener">Shadow DOM</a>, hidden DOM trees can be attached to elements in the regular DOM tree, which allows for isolated testing of specific HTML document sections. Testsigma&#8217;s latest release introduces support for capturing and executing test cases on Shadow DOM elements.</li></ol>



<ol start="5"><li>Mailbox Functions for URL Extraction and Verification: This new mailbox function lets you extract and verify URLs from emails. Whether you need to extract and click on specific URLs or validate email URLs, Testsigma provides flexible options such as text matching, partial URL matching, and regular expressions.</li></ol>



With these new feature updates, we’re continuing to make test automation a breeze.



If you’re not already a Testsigma user,



<div class="cta-banner-v2">
<div class="title-with-description ">
Build and run tests 10x faster for web, mobile, desktop and APIs under Testsigma&#8217;s unified platform.
</div>
<a href="https://testsigma.com/signup" target="_blank" class="cta-btn-v1" rel="noopener">Try for free</a>
</div>



If you’d like to see the difference that hassle-free automation can make to your release cycles, <a href="https://testsigma.com/request-demo" target="_blank" rel="noreferrer noopener">let us know</a>.

May ’23 feature spotlight The latest on Testsigma

May ’23 feature spotlight: The latest on Testsigma

We&#8217;re excited to announce our latest release, packed with new features to take your testing to the next level. We built these features with testing efficiency in mind for quality engineering teams delivering quality releases at speed.



Let&#8217;s begin!



<ol><li>Powerful new API testing capabilities .Our brand new API testing interface comes with powerful features that support real-time API response validation, testing of different API methods (GET, PUT, POST, DELETE, PATCH), and data-driven API testing. <a href="https://testsigma.com/automated-api-testing" target="_blank" rel="noreferrer noopener">Learn more</a>.&nbsp;</li></ol>



<ol start="2"><li>Concatenate dynamic test data You can now dynamically generate and use multiple different sets of data and concatenate them with Testsigma&#8217;s test data concatenation feature. It lets you easily concatenate any base strings with dynamic user properties and run the same test for as many iterations as you need. Building dynamic tests saves you time and reduces the risk of errors. <a href="https://testsigma.com/docs/test-data/types/concat/" target="_blank" rel="noreferrer noopener">See documentation</a>.&nbsp;</li></ol>



<ol start="3"><li>Download and white-label PDF reports Generate detailed reports for management reporting or your own audit and analysis for each build. With this feature, you can export reports to PDF, complete with test cases, test steps, and screenshots. Plus, you can include your firm&#8217;s <a href="https://www.goldenflitch.com/blog/how-much-does-logo-design-cost" target="_blank" rel="noreferrer noopener">brand logo</a> to ensure the report follows any copyright and company guidelines. Note: This feature is only available to Enterprise users (<a href="https://testsigma.com/docs/reports/export-reports/export-report-pdf/" target="_blank" rel="noreferrer noopener">See documentation</a>).</li></ol>



<ol start="4"><li>Nested Step Groups Group the test steps that are generally carried out together, and reuse the group everywhere. For instance, in a data-driven test case, you can iterate over user profiles one by one and perform different actions based on the profile type. By creating nested step groups, you can save time and reduce the risk of errors. <a href="https://website.testsigma.com/docs/test-cases/step-types/nested-groups/" target="_blank" rel="noreferrer noopener">Learn more</a></li></ol>



<ol start="5"><li>Disable Test Cases in a Test Plan Sometimes, you know that certain test cases are bound to fail due to known existing bugs. To avoid wasting time on these failures, you can now disable test cases that you don&#8217;t want to run at the moment. This feature allows you to decouple your results from known failures and gives you more control over your testing. <a href="https://testsigma.com/docs/test-management/test-plans/disable-test-cases-in-test-plans/" target="_blank" rel="noreferrer noopener">Know more</a></li></ol>



<ol start="6"><li>Enhance test data mid-execution with new NLP Our new NLP feature lets you write values to test data profile columns during test case execution. For example, if you&#8217;re iterating over a series of SKUs, you can log the price of each SKU from the UI and write it to the desired column for a test data profile. Later, you can use this test data profile in another test case to verify if the prices in another pin code or city are the same. This feature makes it easy to add data from the app being tested to your test data profile or log status for certain actions. <a href="https://testsigma.com/docs/test-data/add-values-to-test-data-profile/" target="_blank" rel="noreferrer noopener">Know more</a></li></ol>



With these newly released updates, we&#8217;re continuing to enable teams to set up and scale test automation quickly and without coding skills.&nbsp; If you&#8217;re not already a Testsigma user,



<div class="cta-banner-v2">
<div class="title-with-description ">
Automate your tests for Web, Mobile, Desktop and APIs on Cloud with Testsigma. No setup required.
</div>
<a href="https://testsigma.com/signup" target="_blank" class="cta-btn-v1" rel="noopener">Try for free</a>
</div>



If you’d like to see the difference that hassle-free automation can make to your release cycles, <a href="https://testsigma.com/request-demo" target="_blank" rel="noreferrer noopener">let us know</a>.

April '23 feature spotlight New features to supercharge your testing

April ’23 feature spotlight: New features to supercharge your testing

George Ukkuru is a seasoned technocrat and AVP of quality engineering, DevOps, and SRE @Marlabs Inc. Over the course of two decades, he has helped Fortune 500 companies implement Agile testing practices. He has also authored a number of books on quality engineering and test automation. 



In this QnA, Ukkuru lays out the scope and depth of his experience without even trying. He makes the points that non-functional testing should be higher up in your priority, that manual test data provisioning keeps you from truly reaping the rewards of automation, and that you must never miss an opportunity to celebrate good work. 



<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#Q1_What_changes_have_you_seen_in_SDLC_and_QAs_role_in_it_over_the_course_of_two_decades">1 Q1: What changes have you seen in SDLC and QA’s role in it over the course of two decades?</a></li><li><a href="#Q2_Whats_your_battle-tested_process_for_finding_and_resolving_bugs_in_Productionnbsp">2 Q2: What’s your battle-tested process for finding and resolving bugs in Production?&nbsp;</a></li><li><a href="#Q3_What_are_some_testing_anti-patterns_you_wish_never_existed">3 Q3: What are some testing anti-patterns you wish never existed?</a></li><li><a href="#Q4_If_you_had_to_restart_your_career_today_what_key_lessons_would_you_take_from_the_past">4 Q4: If you had to restart your career today, what key lessons would you take from the past?</a></li><li><a href="#Q5_To_aspiring_leaders_what_is_your_advice_on_how_to_be_a_good_manager">5 Q5: To aspiring leaders, what is your advice on how to be a good manager?</a></li></ul></div>
<h3 id="q1-what-changes-have-you-seen-in-sdlc-and-qa-s-role-in-it-over-the-course-of-two-decades">Q1: What changes have you seen in SDLC and QA’s role in it over the course of two decades?</h3>



In the last two decades, I have seen:



1. The emergence and dismantlement of the TCOE model, aka Independent Verification and Validation. It started in the early 2000s. Today, TCOEs have been converted to a hybrid model as teams adopt Agile &amp; DevOps. 2. Automation used to be a nice-to-have skill. When I started my career, automation was used for automating UI-based regression testing. Today, automation is a must-have skill, and the focus has moved from regression testing to complete test lifecycle automation.



The type of applications that are being tested has changed as well, from (now) legacy client server-based applications to microservice-based or cloud-based architectures. This changes how teams test for quality. 



The focus has shifted to non-functional testing, including running security tests, API &amp; workflow-based performance testing, Chaos Engineering, compatibility testing, etc. As technology advancement continues, we’ll see newer, more nuanced types of testing.



<h3 id="q2-what-s-your-battle-tested-process-for-finding-and-resolving-bugs-in-production">Q2: What’s your battle-tested process for finding and resolving bugs in Production?&nbsp;</h3>



Once a bug&#8217;s been found on production, I have several practices to fast-track its resolution:&nbsp;



1. Continuously review production tickets and identify defect trends, in collaboration with the Operations Team.2. Use an algorithm that identifies the best developers for determining the root cause and fixing the defect.3. Organize regression packs based on the type of fixes going into production and define when to use them.4. Monitor social media feedback, application logs, Mobil App Store feedback to identify the issues faced by the end users and take proactive actions to identify patterns.



<h3 id="q3-what-are-some-testing-anti-patterns-you-wish-never-existed">Q3: What are some testing anti-patterns you wish never existed?</h3>



I see several anti-patterns in software testing. But the ones that peeve me the most are:



1. Heavy focus on UI Automation. The effort and cost to maintain UI automation is disproportionately higher than the rewards, I think. UI changes frequently as the products evolve. Teams need to strike a fine balance between UI and API-based tests. API Tests are faster to design, execute and maintain.



2. Manual test data provisioning. The test data is ported to test environments after manually masking data from obfuscation. That is slow. You can&#8217;t truly automate for DevOps using this approach. Instead, teams should use synthetic data provisioning and self-service mechanisms to accelerate and automate data provisioning.



3. Late Identification of Non-Functional Defects. This is usually a result of a lack of clarity around non-functional testing requirements. Performance or security tests are seen as a part of &#8216;hardening&#8217; or release-level sprints. Unfortunately, performance and security bugs often need architectural changes to fix, and keeping non-functional testing for the end of the cycle can cause schedule and cost overruns.



I advise teams to look at behavior-driven testing techniques to document security and performance requirements. Engineers should also integrate and execute API-based performance tests and SAST/DAST scans to the DevOps pipeline.



<h3 id="q4-if-you-had-to-restart-your-career-today-what-key-lessons-would-you-take-from-the-past">Q4: If you had to restart your career today, what key lessons would you take from the past?</h3>



Master at least one programming language, preferably Java, Python, or C#, so that you can design and create scripted tests.



Focus on taking up courses around test automation, either with open source or commercial tools.&nbsp;Or both.



Spend time learning about practices and concepts in software testing. ISTQB Foundation syllabus should be a good starting point.&nbsp;



Learn about Cloud, Agile, and DevOps principles and practices. Learn the basics of tools like Jira, Azure DevOps, Jenkins, etc.



Your communication skills matter more than you think.&nbsp;Work towards improving them. 



<h3 id="q5-to-aspiring-leaders-what-is-your-advice-on-how-to-be-a-good-manager">Q5: To aspiring leaders, what is your advice on how to be a good manager?</h3>



Take up formal courses on Agile or Project Management. Refresh your skills regularly to stay relevant.



This is my style, but I advise new managers to get actively involved in the project. Remember that you are not a people manager. Understand what the project objectives are and how it is going to impact a business or end-user. Work from there.



Understand the technologies involved so you can grasp the nuances of the project. This will help you make decisions and manage risks.



Learn the art of remote management. Here are some practices that work for me:-



<ul class="list_style_disc"><li>Have frequent meetings with your direct reports, preferably over video.</li><li>Do NOT micro-manage and follow-up every hour. Trust your team. Move away from a task-based and adopt a Goal-based approach while assigning work.</li><li>Spend time with your team to understand their strengths, passions, and weaknesses. Assign work based on their strengths and provide training and mentoring to help employees overcome their weaknesses.</li><li>Keep your team in the loop about the happenings in your department and the larger organization. This will help take care of rumors and avoid perceptions.</li><li>Provide feedback on the work assigned without delays. Don&#8217;t be a bottleneck.</li><li>Never miss an opportunity to appreciate and celebrate good work.</li></ul>



For more experience-led insights from George Ukkuru, check his book on software testing estimation on Kindle.

Quality Engineering Discussions 5 Questions with George Ukkuru

Quality Engineering Discussions: 5 Questions with George Ukkuru

Rahul Parwal is a software engineer, speaker, mentor, and writer out of Jaipur, Rajasthan. He has dabbled in software development, testing, and automation, and often shares his learnings in <a href="https://testingtitbits.blogspot.com/" target="_blank" rel="noreferrer noopener nofollow">his blog</a>.



In this QnA, Rahul makes the point that a randomly failing test is always worth investigating and that assumptions are dangerous in testing.



<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#Q1_What_are_your_most_and_least_favorite_things_about_test_automationnbsp">1 Q1: What are your most and least favorite things about test automation?&nbsp;</a></li><li><a href="#Q2_It_sounds_great_on_paper_but_what_would_it_really_take_for_a_mid-sized_org_to_incorporate_continuous_testing_in_their_release_cyclenbspnbsp">2 Q2: It sounds great on paper, but what would it really take for a mid-sized org to incorporate continuous testing in their release cycle?&nbsp;&nbsp;</a></li><li><a href="#Q3_What_are_some_Severity_1_bugs_that_ALMOST_slipped_past_you_and_how_did_you_make_sure_they_never_did_so_again">3 Q3: What are some Severity 1 bugs that ALMOST slipped past you — and how did you make sure they never did so again?</a></li><li><a href="#Q4_How_do_you_manage_failed_tests_and_tests_that_fail_randomly">4 Q4: How do you manage failed tests and tests that fail randomly?</a></li><li><a href="#Q5_Whats_your_most_memorable_deployment_andor_release_day_Good_OR_bad_just_as_long_as_no_one_dies_at_the_end8230">5 Q5: What’s your most memorable deployment and/or release day? (Good OR bad, just as long as no one dies at the end&#8230;)</a></li><li><a href="#Bonus_If_you_could_give_just_ONE_piece_of_advice_to_fledgling_software_testers_what_would_it_be">6 Bonus: If you could give just ONE piece of advice to fledgling software testers, what would it be?</a></li></ul></div>
<h3 id="q1-what-are-your-most-and-least-favorite-things-about-test-automation">Q1: What are your most and least favorite things about test automation?&nbsp;</h3>



One of the things I like about test automation is that it can be scaled over multiple platforms, devices, interfaces, environments, etc. quickly and reliably. Test automation, if planned and executed correctly, can be a great service to testing and QA.



There is, however, The Silver Bullet Syndrome, where everyone wants to ascribe the responsibility of succeeding with test automation success to a tool. “Let’s find the one tool to solve all of our problems.” It doesn&#8217;t work.



<figure class="wp-block-image is-resized"><img src="https://lh4.googleusercontent.com/ghN79ATI0DLYh5CNL3-LxlBUzyTg_fCE011vqAikrZkvyJqUZWPufXB3-U_BQNUtUCnw_qjdmjD0g4OSQGM_kurFoNBWGnPFVwH-dpYhyrWL36RWBwbVm4P-ILV8z8KCIp6Q_--H" alt="Definition" width="600"/></figure>



The concept of automation extends well beyond a tool. Teams attempting test automation should consider a tool’s capabilities and limitations. But first, they should establish realistic expectations, sort out people constraints, and have the support of management. Without all three, adopting test automation will just be an uphill battle.



<h3 id="q2-it-sounds-great-on-paper-but-what-would-it-really-take-for-a-mid-sized-org-to-incorporate-continuous-testing-in-their-release-cycle">Q2: It sounds great on paper, but what would it really take for a mid-sized org to incorporate continuous testing in their release cycle?&nbsp;&nbsp;</h3>



The best way to start is by evaluating and understanding your context, team, and environment.



&#8220;No matter how they phrase it, it&#8217;s always a people problem,&#8221; as Sherbie&#8217;s Law says.&nbsp;CI/CD and consequently, continuous testing, needs a shift in the team&#8217;s mindset. If you push a whole bunch of processes and tools on people without getting them on board, you are setting them all up for failure.&nbsp;



In practice, my advice is to start with small experiments and do one thing at a time. Don&#8217;t expect overnight results, and <ul class="list_style_disc"><li>Make it a team sport. Establish channels of collaboration. Hold knowledge-sharing sessions. Try Pair and mob testing to reduce knowledge silos within the team.</li><li>Plan before you execute. At each stage of the product life cycle, plan what and how you&#8217;ll test.</li></ul> 





<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://lh6.googleusercontent.com/k81_BbjtH20lB1vpqnVsN9Sij_GDLcaZa6KtTE9P6hAF17sur2tq9Xj81j2CmxqWgPDEiMy3hrjeHj6x0ftLAlMia_XmvetYDw_WDirOJ3O5WbQPBHdPgkzVj4R9MG9rsfFRSw-M" alt="Plan before you execute" width="600"/></figure></div>



<h3 id="q3-what-are-some-severity-1-bugs-that-almost-slipped-past-you-and-how-did-you-make-sure-they-never-did-so-again">Q3: What are some Severity 1 bugs that ALMOST slipped past you — and how did you make sure they never did so again?</h3>



I have had two such incidents. I’ll share both, along with the lessons I learned from them:



<ul class="list_style_disc"><li>Software loading was failing if a specific application is running in parallel. Lesson: Test your software in real user conditions. Remember to test how your app works with shared resources such as memory, space, network resources, etc.</li> <li>The application gave a timeout if the form submission exceeded 15 minutes. Lesson: Actual users are not as fast as an automation bot/script. Users will take will never perform a form submission as quickly as a script, so use waits judiciously.</li></ul>



<h3 id="q4-how-do-you-manage-failed-tests-and-tests-that-fail-randomly">Q4: How do you manage failed tests and tests that fail randomly?</h3>



A failing test establishes my confidence in a test I’ve designed. The amazing Angie Jones once tweeted something which sums up my feelings on failed tests.



<figure class="wp-block-image is-resized"><img src="https://lh5.googleusercontent.com/0SG95qwUYRPHWIFFupruMFOcQE4y1wgMeyTfDrUNOwAO7sHAERCa3PC7e_SEGVzhCbnRKyOxrHyizTYGGGv3cd5OpLg8LYL0E3yTlkVrTb5mQuJLLrXEWSrwhfRaC0ckYHeCUAy8" alt="How do you manage failed tests and tests that fail randomly" width="600"/></figure>



Randomly failing tests are interesting. They give me a reason to go sleuthing, to find and uncover variables that might be affecting my product and its functionality. My bug analysis and exploration skills have greatly improved as a result of investigating such bugs. Whenever I see a randomly failing test, I triangulate failure conditions by varying:



<ul class="list_style_disc"><li>Data</li><li>Behavior</li><li>Options &amp; Settings</li><li>Software &amp; Hardware Environment</li><li>Configuration &amp; Dependencies, etc.</li></ul>



A randomly failing test is usually a symptom of a much larger problem. Sometimes, developers refer to such cases as corner cases, but I&#8217;ve learned to never ignore a random failure without investigating it. It might just be the tip of the iceberg of a whole load of flaky tests.



<h3 id="q5-what-s-your-most-memorable-deployment-and-or-release-day-good-or-bad-just-as-long-as-no-one-dies-at-the-end">Q5: What’s your most memorable deployment and/or release day? (Good OR bad, just as long as no one dies at the end&#8230;)</h3>



In my first project as a software tester, I was testing for a product release of a large enterprise financial application. The stakes were very high since this product was developed to sunset a legacy system that had been in place for decades.



During testing, we were going through all the flows on a mocked simulated data set that was providing data as per our testing layer. After months of rigorous testing, on the release day, we started getting errors from the monitoring system.&nbsp;



There were gaps in the mocked simulated data and the actual production data. We hadn’t questioned the authenticity of the test data during the testing cycle. We ended up with a release rollback.



Some lessons that I learned that day:



<ul class="list_style_disc"><li>Never assume anything &#8211; What if the data is wrong?</li><li>Learn how to create your own testing data. This way, you understand the sources, flows, and formats of the data.</li><li>If you use mock/simulated data for testing, be sure to run a significant number of tests using real data as well.</li></ul>



<h3 id="bonus-if-you-could-give-just-one-piece-of-advice-to-fledgling-software-testers-what-would-it-be">Bonus: If you could give just ONE piece of advice to fledgling software testers, what would it be?</h3>



Like Paul Muad’Dib of Dune, “Learn how to learn.” Testing is a continuous search for information: Both about your product and the tests that you run it through. None of us are born with knowledge. Trust that you can learn new ways to test. Be a continuous learner.

Quality Engineering Discussions: 5 Questions with Rahul Parwal

In this series, real (and really good) QA practitioners use their experience to support—or debunk what you might know about software quality.



James Espie is a test specialist, a quality engineering proponent, and a continuous learner from Auckland, New Zealand. He shares his insights and sporadic bursts of inspiration in a hilarious newsletter called Pie-mail. If you haven’t seen it, <a href="https://jpie.nz/newsletter/" target="_blank" rel="noreferrer noopener nofollow">you should check it out</a>.



<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://lh4.googleusercontent.com/vrj-e9P8NUEQgfsq9LUdxmQ8ewzOaZ-ulDqcGjCq1n3YY6ZFTKlHCdoiDoFQ5mIBfDti4pDJkt8dKeg9_PheRwVMUTYlap00qW4kWixVFozXgywTSOAvrqU0jTeagyaiZjSAMiCb=s1600" alt="Postman pie-mail" width="600"/><figcaption>&nbsp;Postman jokes. Do we need to say more?</figcaption></figure></div>



In this quick QnA, Espie makes the point that software quality begins with people, that more testing doesn’t necessarily mean better quality, and that context matters in metrics.



<div id="toc_container" class="no_bullets">Table Of Contents<ul class="toc_list"><li><a href="#Q1_What8217s_the_most_common_organizational_bug_that_you8217ve_seen">1 Q1: What&#8217;s the most common organizational bug that you&#8217;ve seen?</a></li><li><a href="#Q2_What_are_some_testing_anti-patterns_you_wish_never_existed">2 Q2: What are some testing anti-patterns you wish never existed?</a></li><li><a href="#Q3_Metrics_question_What_are_some_QA_or_testing_metrics_that_you_ALWAYS_want_to_track_Why">3 Q3: Metrics question. What are some QA or testing metrics that you ALWAYS want to track? Why?</a></li><li><a href="#Q4_What_are_some_Severity_1_bugs_that_ALMOST_slipped_past_you_and_how_did_you_make_sure_they_never_did_so_again">4 Q4: What are some Severity 1 bugs that ALMOST slipped past you — and how did you make sure they never did so again?</a></li><li><a href="#Q5_Whats_your_most_memorable_release_day_Good_OR_bad_just_as_long_as_no_one_dies_at_the_end8230">5 Q5: What’s your most memorable release day? (Good OR bad, just as long as no one dies at the end&#8230;)</a></li></ul></div>
<h3 id="q1-what-s-the-most-common-organizational-bug-that-you-ve-seen">Q1: What&#8217;s the most common organizational bug that you&#8217;ve seen?</h3>



An &#8220;organizational bug&#8221; I see often is teams that are ‘agile’ until it comes to testing.



Design, product, development functions are all collaborating and working together. But when it’s time to test, the team switches to ‘waterfall mode’. They pass the ticket to the tester and move on to something else.



Often agile teams don’t have a clear picture of where a test specialist should engage. The fix is to encourage teams to <a href="https://testsigma.com/automated-continuous-testing-platform">include testing at every stage of the development process</a>. But this is easier said than done.



One thing that’s worked for me is encouraging developers and testers to test together.



When a developer has a change ready to merge, they pair up with a tester. The developer demonstrates to the tester that it “works on my machine”. We call this process “WOMMing”. I’ve also called it “show and tell”.



(The term WOMMing was coined by one of my former colleagues at <a href="https://www.linkedin.com/company/pushpay-ltd/" target="_blank" rel="noreferrer noopener nofollow">Pushpay</a> &#8211; I don’t know who exactly to attribute it to though.)



During WOMMing, the role of the tester is to tease out edge cases, ask questions, and be critical of the change. It’s useful because it helps find bugs early. Additionally, it encourages the developer to learn some testing skills.



Of course, it’s not a silver bullet. Testers need to be pulled into other phases of the development life cycle. But if a team is at a point where testers aren’t included at all, pairing with developers could be a great way to start.



<h3 id="q2-what-are-some-testing-anti-patterns-you-wish-never-existed">Q2: What are some testing anti-patterns you wish never existed?</h3>



I wish massive end-to-end UI testing suites did not exist.



I see this often. Organizations use automation to test every screen, function, and feature in their product. In my experience, you only need a few E2E UI tests. As a test suite starts to get bigger, the value from each extra test starts to diminish.



Some things that have stood out to me include:



<ul class="list_style_disc"><li>Tests that check the same screen over and over, with only slight variations in data.</li><li>Tests that check things that are already covered by tests at other layers; Unit or API tests, for example.</li><li>Tests that check features that are of little to no business value.</li></ul>



These don’t seem like a good use of time. Also, each extra test will make the test suite take longer &#8211; and, potentially, more brittle.



A lot of organizations fall into the trap that you need to automate everything. Testers become test automation engineers and end up building more and more automated tests. I’m not convinced; I think a tester&#8217;s time could be better spent elsewhere.



<h3 id="q3-metrics-question-what-are-some-qa-or-testing-metrics-that-you-always-want-to-track-why">Q3: Metrics question. What are some QA or testing metrics that you ALWAYS want to track? Why?</h3>



First I’ll say, I am really careful with metrics.



If care is not taken, and teams don’t understand why metrics are important, they can be tempted to ‘game the system’. <a rel="noreferrer noopener" aria-label=" (opens in a new tab)" href="https://www.theguardian.com/business/ng-interactive/2015/sep/23/volkswagen-emissions-scandal-explained-diesel-cars" target="_blank">There’s a really interesting (and shocking) example from Volkswagen from a few years back, where they ‘cheat’ to achieve emissions targets</a>.



That isn’t to say metrics aren’t important, but they do need to be treated with respect. Personally, I like <a href="https://www.cloudbees.com/blog/dora-devops-metrics-bandwagon" target="_blank" rel="noreferrer noopener" aria-label=" (opens in a new tab)">DORA metrics</a>:



<ul class="list_style_disc"><li>Deployment Frequency (how often does a deploy happen)</li><li>Mean Lead Time for changes (how long it takes to get a change to production)</li><li>Mean Time To Recovery (how long it takes to recover from an incident)</li><li>Change Failure Rate (how many changes fail in production)</li></ul>



My approach to metrics like these is simple: find out what the numbers are and then work to improve them. For example, it is useful to know the Mean Time To Recovery. If it’s half a day, then that’s a useful and interesting piece of information. Once we know that, then we can look at ways to improve that time and measure it again.



My suggestion with metrics though is:



<ul class="list_style_disc"><li>Be clear what action is being taken to improve a certain metric, and that it makes sense.</li><li>Measure, and measure again after improvements have been made. But, don’t set arbitrary values as targets. “Make our time to recovery faster” is far better than “make our time to recovery less than ten minutes”</li><li>Make sure everyone has an understanding of why you want to measure a specific thing. Context matters!</li></ul>



Hopefully, these suggestions will help to avoid becoming like the team at Volkswagen 🙂



<h3 id="q4-what-are-some-severity-1-bugs-that-almost-slipped-past-you-and-how-did-you-make-sure-they-never-did-so-again">Q4: What are some Severity 1 bugs that ALMOST slipped past you — and how did you make sure they never did so again?</h3>



This is a story about one that did slip past us. But, we got away with it.



My team worked on a billing product for a SaaS company. We had a seemingly simple task, replacing a couple of our products with new ones. The products also had new prices.



Now, the data structure was overcomplicated. There were two tables, product, and price. Each product had a start and an end date. It was related to a price, which also had its own start and end date. (You might see a problem here already!)



To make matters worse, the product dates were stored in NZT. But the price dates were stored in the local timezone for that region. We missed this in testing.



When we went to production, we ended up with a product that ‘ended’ at midnight NZT. Its associated price didn’t ‘start’ until midnight GMT. So, for a window of twelve hours, nobody in the UK could buy our product. Yikes!



We got lucky. Our product was a B2B product, and the 12 hour period fell on a Sunday. So it was very unlikely anyone was trying to make a purchase. After the fact, we checked the usage statistics &#8211; zero hits. Pretty thankful we made the call to change the price over a weekend!



We ran a post-mortem to figure out what could prevent this from happening again. We decided that the fix for it was to refactor our database so that everything used the same time zone. But, this was going to be a hefty piece of work &#8211; that one change had significant downstream effects.



There was other information to take into account:



<ul class="list_style_disc"><li>Price changes are a rare occurrence for this organization.</li><li>Product changes are even rarer.</li><li>There was an ongoing piece of work to replace this billing system &#8211; in a couple of years, we expected it to be gone.</li></ul>



We opted to do no mitigations. That’s kind of counter-cultural, but it was the right thing to do. I guess my point is, sometimes, doing nothing is entirely reasonable!



<h3 id="q5-what-s-your-most-memorable-release-day-good-or-bad-just-as-long-as-no-one-dies-at-the-end">Q5: What’s your most memorable release day? (Good OR bad, just as long as no one dies at the end&#8230;)</h3>



The world of CI/CD highlights the difference between deploying and releasing. We deploy software all the time. Silently behind feature flags or otherwise inaccessible by customers. Then, when it’s ready for customers, all we need to do is flip a switch. Then, it’s released.



My team did this recently for a new integration feature. All our code was sitting in production, behind a feature flag. Once we were confident everything was in place, all we had to do was switch it on. Very unglamorous, and no ceremony.



We returned to the office the next day. Our lead engineer went to check the logs first thing that morning,&nbsp; to see if anyone had used the newly ‘released’ feature. And sure enough, one person had!



Now I know, it doesn’t sound like much. But, for us, it was incredible. After months of work over that feature, we had our first real live user!



The joy comes not from releasing the feature but seeing people use it and get value from it.



This is one reason why having good monitoring and logging is so important. You can see what your users are doing, and celebrate the success!



<hr class="wp-block-separator"/>



For more test insights by James Espie, check out <a href="https://jpie.nz/" target="_blank" rel="noreferrer noopener">JPie.nz</a>

Quality Engineering Discussions: 5 Questions with James Espie