Mainframe Testing | How to Do, Challenges & Best Practices

Mainframe testing comes toward the end of a testing pipeline. It evaluates software, applications, and services built for Mainframe Systems. By doing so, these tests confirm that a site or app is reliable enough for a public launch. This form of testing uses a set of validation techniques and verification approaches to verify that the app functions flawlessly in an actual user’s hand. 

In this article, we’ll be delving into a 101-style introduction to Mainframe Testing. Start here to understand what mainframe testing is, where & run to run these tests, common methodologies, testing procedures, challenges, troubleshooting, best practices and tools associated with this essential element of Quality Assurance. 

What is Mainframe?

The mainframe is a high-performing, high-speed computerized system used for large-scale processing – operations that require massive resource availability and security. Mainframes are built and used because they are secure, scalable, and reliable, more so than regular computers. They are often used in retail, finance, health and pet insurance, healthcare, and other sectors to crunch massive datasets continuously and without errors.

If you google mainframes, you’ll see that, among other abilities, a mainframe can execute millions of instructions each second. 

In case you’re interested, here are the main features of a mainframe system (features that make them so wildly useful)

• Maximum input/output bandwidth: In case the mainframe encounters a glut of excessive input and output, there are choke points in the links between drives and processors. This prevents the system from becoming overwhelmed.
  
• Reliability: Mainframe systems are designed so that they can frequently subject themselves to graceful deterioration and service while continuing to operate as expected.
  
• Reliable single-thread: This allows performance to remain critical for real-world database activities.
  
• Maximum Input/Output Connectivity: It is by virtue of a mainframe’s maximum input/output connectivity that indicates that it can deliver massive transactions at a time. 

What is mainframe testing?

Mainframe testing is the practice of testing software so that it works well on mainframe systems. Therefore, mainframe tests are created on the basis of the mainframe systems. Its primary goal is to verify software reliability, performance and quality, especially when it’s working with certain mainframes. Generally, once mainframe testing is done, the software is ready for deployment.  

Here’s the deal. When you’re running mainframe tests, you only need to know about navigating CICS(Customer Information Control System) screens – custom-built for niche user scenarios. In case the code is modified via COBOL, JCL(Job Control Language) or any other language, it does not affect the emulator or the tester. All changes (ones that work) made to one terminal emulator will be equally applicable to all others. 

• The mainframe application or job batch is tested via test cases created on the backs of the project requirements (including user expectations). 
• Mainframe tests check the deployed code by putting in multiple data combinations into the input file. 
• All apps on the mainframe are accessible via the terminal emulator – which is the only software you need to install on the client machine.

Mainframe Attributes

Virtual Storage

This attribute lets a processor replicate the main storage, which is larger than the actual storage attached to the processor. By doing so, the processor can use memory to store and execute tasks requiring all capacities. Here, disk storage appears to be larger than the real device storage. 

Multiprogramming

The mainframe computer can run multiple programs simultaneously, but only one program can have control of the CPU at one time. This enables efficient use of the CPU. 

Batch Processing

By this attribute, all tasks are executed in units or “jobs”. Each job triggers the execution of one or more programs in a pre-set sequence. 

The task order is decided by the job scheduler, which manages the order to maximize average throughput. Jobs are scheduled according to priority and class. 

Data for batch processing is fed into the system via JCL(Job Control Language), which describes the batch job – data, resources, scripts. 

Time Sharing

Time-sharing systems allow each user system access via the terminal device. You don’t submit jobs to be scheduled to run in the future. The user just pushes commands that are executed immediately. 

This “Interactive Processing” lets users communicate with the computer directly. 

SPOOLing(Simultaneous Peripheral Operations Online)

The SPOOL devices stores output from the application. This output is used to control external devices, if required. By doing so, this process leverages buffering to user external devices more efficiently. 

Where is mainframe testing done?

For the most part, mainframe tests check the deployed code. Scripts use multiple datasets to create input files. All applications must be tested for complete compatibility with the mainframe to go into production. 

The mainframe application, alternatively called batch task, is tested against requirements-driven test cases. Testers access the application under test through the terminal emulator. 

How to do Mainframe Testing

• Create a plan based on requirements documents from the business team. The plan lays out how each element or process in the application will be reoriented in the release cycle.
  
• Once the QA team receives the requirements-based plan, they’ll identify which processes will be impacted by changes. It’s the norm for about 20-25% of the application to be impacted by customizations.
  

So, in summary, mainframe tests run in two parts:

Test the Requirements: Testing an app for its functionality. You can also test the changes highlighted in the requirements documentation. 

Test the Integration: Verify the entire end-to-end workflow of the application that receives and/or sends data to the impacted application. Basically, this is Regression Testing. 

Mainframe Testing Methodology

Scenario: A financial app carries, among its many features, a member enrollment section. This feature draws data from online and offline users who enroll. To test this feature, the app will be tested via online testing & batch testing (as explained earlier). 

• For online testing, you access data through the app screen via which members can enroll. This database is verified by checking the data entered by users through this screen.
  
• Offline enrollment data can come from a third-party website or actual paper forms filled up by users. This data (the “batch”) is fed into the main app’s database through “batch jobs”. The format of this data is used to create the input flat file, which is then used to direct the sequence of batch jobs.
  
  So, to test this feature, you utilize the following methodology:

• The first job in the batch job sequence checks the data entered for accuracy. For eg., are there any special characters in fields that do not allow them?
  
• The second job checks if the data is consistent according to business standards. For eg., a nominee can only be a family member or friend, not an organization.
  
• The third job checks if the app modifies the data into a format accepted by the app database. For eg., if the app only stores an ID number and deposit amount, then it should delete the user name, date of entry, etc.
  
• The fourth job checks that the data is accurately loaded into the database. 

Each job is tested and validated separately. The integration between the jobs is verified by feeding the input flat file to the first job, which validates the database. 

Mainframe Testing Procedures to Follow

Shakedown/Smoke Testing

Verify that the code is deployed in the right environment and has no critical issues. 

System Testing

• Batch testing validates results on output files and data changes triggered by batch jobs within each test. 
• Online testing works at the front end of the mainframe application. It checks if the entry fields are operational. 
• Online-Batch Integration Testing checks systems using batch processes and online applications. Tests data flow & interaction between online screens and batch jobs. 
• Database testing checks the databases from where you get the data for the mainframe application. This is also where the data is validated for layout and storage. 

System Integration Testing

These tests validate the function of systems interacting with the particular system under test. These particular systems are not impacted by requirements but they utilize data from the system being tested. Therefore, you have to test the UI and relevant alert messages (Job successful, Job failed, Updated, etc.)

Regression Testing

Regression tests are run after every other test is complete and the code is to be changed. It checks if the changes have affected the existing, perfectly operational functions. The tests verify that batch jobs and online screens interacting with the system under test are not touched by the current release. 

Performance Testing

Ferrets out bottlenecks in the areas meant for high user interaction & processes, like front-end data. It also upgrades online databases and explores the app’s scalability. 

Security Testing

Check the level at which the application can fend/counteract attacks on its security. At this point, you test Mainframe security and Network security. Security testing checks the validity of the software’s integrity, data privacy, authorization & authentication mechanisms as well as its availability. 

Above tests can be done manually, and if they are repeated frequently, automation is recommended.

Below, we will discuss tools for automating mainframe tests.

Benefits of Mainframe Testing

• Prevents duplication of tests. 
• Serves to improve the app’s overall UX, which increases its market appeal and chances of success. 
• Cuts down on production downtime by ensuring the software functions well under a wide range of user conditions.
• Helps retain customers by giving them flawless, glitch-free user journeys. 
• Reduces overall IT costs by ensuring that the app doesn’t glitch, and needs minimal repairs. 

Mainframe Testing Challenges and Troubleshooting

Here we discuss some challenges faced during mainframe testing and how to troubleshoot them.

A mismatch between requirements and the test handbook

If testers are going by a user handbook or training guide, they might still come across tests and issues in requirements that do not exist in the handbook. 

To prevent this, the QA team should be included in the dev pipelines right from the requirements gathering phase. Testers can then check and state if stated requirements are actually testable, given their current resources. And, if they’re able to see requirements early on, testers can save time, money, and effort by creating specified, targeted tests that the software actually needs to verify it’s functions. 

Finding required test data

Certain tests need specific datasets, which need to be found and taken from a larger, more expansive database. Writing tests that do this aren’t exactly easy, especially since the required data isn’t always easily identifiable. 

The solution is to find/create tools that set up data as required by tests. Create queries beforehand to summon existing data. If there’s a problem, you can trigger a request for creating or cloning the data you need. 

No impact analysis

Be prepared for the fact that the changes in code can completely change and reorient the system – in terms of aesthetics and/or performance. These changes are often necessary, whether they apply to test cases, data, or test scripts. 

To ensure code changes don’t impact the whole system detrimentally, use impact analysis and strategies to manage scope changes in testing. 

Ad-hoc request

Many errors caught in testing indicate that the entire application needs end-to-end testing. But, demands for such tests detail the entire test pipeline and timeline, because you need to create new end-to-end tests, which aren’t exactly run in two lines of code. You have to devote significant resources to trigger them in an existing test cycle. 

You need backup plans for such scenarios – regression scripts, automation scrips, skeleton scripts, etc. Again, testers should be included in the brainstorming phase so that they can understand and build backup mechanisms that prevent the test pipeline from delay or bloating. 

Best Practices For Mainframe Testing

• Do a dry run of each job under test by using empty input files. Do this for every job to be impacted by the tests.
  
• Finish the test task setup before you start the test cycle. By doing so, you’ll find JCL errors and save execution time.
  
• Create test data, and check that the sets are complete well before the test cycle.
  
• Set auto-commit to “NO” when accessing DB2 tables via SPUFI – which is what the emulator uses to use said tables. This prevents unwanted updates to the system.
  
• Confirm technical inventory and get test + use case data beforehand. Project and test management is key to successful mainframe testing. 

Get here the list of mainframe testing tools

Automation Tools for Mainframe Testing

Testsigma

Testsigma is a unified, fully customizable software testing platform that works out of the box. It is designed to help teams build end-to-end tests 10x faster for web, mobile apps, & APIs. You can use Testsigma to create test scripts in plain English scripts – scripts that self-heal and require low or no maintenance. 

Used by 9000+ customers to run tests worldwide (25 million tests & counting), Testsigma stands out by letting you create automation tests fast. Write test steps in English, so it’s a perfectly usable tool for non-tech personnel. 

You can run tests in your local browser/device or run across 800+ browsers and 2000+ devices on our cloud-hosted test lab. Also, You can also view step-wise results for each test and analyze real-time reports & dashboards at the individual, test suite & device levels. Moreover, Testsigma’s intelligent AI automatically fixes broken scripts, heals dynamically changing elements, and suggests fixes for test failures.

Automate your mainframe tests, 10x faster, with Testsigma

QTP (Quick Test Professional)

QTP, now UFT is used to run functional and regression tests of web-based software. The tool can run automated functional tests without a tester having to monitor the system. 

QTP is ideal for functional, regression, and service tests. It is often used to test client servers, as well as web apps. You can find a UFT extension in the Chrome Store, which makes it incredibly easy to use. 

UFT supports newer technologies like JDK 1.8, and XenDesktop 7 and browsers like Windows 8.1, Windows Server 2012, and Safari.

Subject7

Subject7 is a cloud-based SaaS test automation platform providing end-to-end testing. You can shape tests through a series of commands. This platform needs no scripts and is language-agnostic. It also comes with an intuitive UI that is easy to use for non-technical testers as well as professional test automation engineers. 

Subject7 integrates with multiple DevOps and CI/CD tools – REST, JIRA, Jenkins, etc. It uses open source platforms for testing, like Selenium & Appium. The tool is also equipped with features for enabling collaboration, email notification, video records of tests, etc. 

The tool can also store all your tests (manual and automated) in an in-built repository. It supports test for APIs, browsers, native mobile apps & more. Since Subject7 does not require scripts but commands, QAs can create tests and run them at a moment’s noticed. 

You get parameterization, data-driven test steps, modularized test steps, clouds runs, scheduled test execution and alerts for pass/fail milestones. 

Last but absolutely not the least, the platform offers robust test reporting and archiving, including video recording for both functions. 

Conclusion

Mainframe testing is not easy, which is why it’s usually senior testers who create and run these tests. Use this article as your first stepping stone to writing and executing tests that work. And while you’re at it, don’t forget to pick the right tool for the job. 

Frequently Asked Questions

What is QA in mainframe?

QA in mainframe would mean Quality Assurance for applications built for mainframe systems.

Is portability testing a functional test?

No. Portability testing is a non-functional approach that checks how easy or difficult it is to move (or “port”) an application or software component between different environments.