Mainframe Testing | How to Do, Challenges & Best Practices

Mainframe systems are heavy systems that perform high-speed computing tasks often reaching millions of transactions per second. They are deployed behind the application’s interface by one end connecting to the application and the other to the database. Mainframe testing aims to test these systems for their reliability and performance making sure they provide accurate outputs on the application’s end.

What is Mainframe?

The mainframe is a high-performing, high-speed computerized system used for large-scale processing – operations that require massive resource availability and security. Mainframes are built and used because they are secure, scalable, and reliable, more so than regular computers. They are often used in retail, finance, health and pet insurance, healthcare, and other sectors to crunch massive datasets continuously and without errors.

If you google mainframes, you’ll see that, among other abilities, a mainframe can execute millions of instructions each second. 

In case you’re interested, here are the main features of a mainframe system (features that make them so wildly useful)

• Maximum input/output bandwidth: In case the mainframe encounters a glut of excessive input and output, there are choke points in the links between drives and processors. This prevents the system from becoming overwhelmed.
  
• Reliability: Mainframe systems are designed so that they can frequently subject themselves to graceful deterioration and service while continuing to operate as expected.
  
• Reliable single-thread: This allows performance to remain critical for real-world database activities.
  
• Maximum Input/Output Connectivity: It is by virtue of a mainframe’s maximum input/output connectivity that indicates that it can deliver massive transactions at a time. 

What is Mainframe Testing?

Mainframe testing is the practice of testing software so that it works well on mainframe systems. Therefore, mainframe tests are created on the basis of the mainframe systems. Its primary goal is to verify software reliability, performance and quality, especially when it’s working with certain mainframes. Generally, once mainframe testing is done, the software is ready for deployment.  

Here’s the deal. When you’re running mainframe tests, you only need to know about navigating CICS(Customer Information Control System) screens – custom-built for niche user scenarios. In case the code is modified via COBOL, JCL(Job Control Language) or any other language, it does not affect the emulator or the tester. All changes (ones that work) made to one terminal emulator will be equally applicable to all others. 

• The mainframe application or job batch is tested via test cases created on the backs of the project requirements (including user expectations). 
• Mainframe tests check the deployed code by putting in multiple data combinations into the input file. 
• All apps on the mainframe are accessible via the terminal emulator – which is the only software you need to install on the client machine.

Get here the list of mainframe testing tools

Importance of Mainframe Testing

Mainframe testing is important for the business due to various aspects.

Ensures the system doesn’t lack performance

Mainframe systems handle humongous transactions each second. A small dip in its performance can impact millions of users hurting the reputation of businesses and often responsible for monetary loss for them and their users. Mainframe testing ensures that the system’s performance is always up to the standards and there are fallback mechanisms in place for the time of failures. 

Ensures the system is scalable

Mainframe systems are used for high-processing tasks by big organizations to ensure there are no lags and downtime for the end-user. Such systems often keep increasing their load as they introduce newer functionalities over time. Scalability, therefore, becomes a much-needed characteristic that cannot be compromised at any cost. Mainframe testing ensures that the system is highly scalable and can serve traffic load as required at any time.

Ensures the system is always secure

Mainframe systems deal with critical data in large amounts. Any security lapse in this system can create a lot of issues including legal compliances due to breach of personal data. Mainframe security testing ensures that the current system is secure and there are no backdoors to enter the system illegally.

Ensures the system is always compliant

Mainframe systems are used by big businesses such as banks, hospitals, and airlines that have to remain compliant with a lot of standardizations and regulations. One of the phases of mainframe testing focuses on ensuring that the system follows these regulations and there are no compliant or legal issues. It helps businesses save costs in damages and legal processes.

Ensures a better user experience

A smoothly working system without any scope for failures, highly scalable, secure, and reliable are all good signs of excellent user experience. When all these metrics are kept in perfect shape, businesses save a lot of money and build their reputation over time as well.

Mainframe testing is an essential part of dealing with a mainframe system. Whether the business has set up its own system or using one over the cloud, mainframe testing should be a part of its testing cycles ensuring a robust infrastructure as it deals with sensitive data and processes.

Classification of Mainframe Testing Method

Mainframe testing methods can be classified into multiple types targeting different areas of the application.

Online testing

Online testing refers to the testing of middleware which is responsible for building customer transaction application programs. Somewhat related to web page testing (user-facing real-time interactive sessions), online mainframe testing also deals with the functionality of existing screens or when newer screens are added into the system.

Batch job testing

Mainframe systems are big and execute large amounts of data. It is common to see large batches of data sent to this system for quick processing and saving CPU cycles. Batch job testing aims at testing the functionality of how reliably a system can execute batch jobs. The reliability parameters include the accuracy of output generated after execution, the ability of the system to handle failures, and how efficiently the data is processed.

Database testing

Since mainframe systems work on large data sets in a very small amount of time, the database attached to it will face many read-and-write operations in the same timeframe as well. This puts high pressure on the database and any inconsistency in it can ripple out to big failures including the failure to hold ACID properties. Database testing ensures that no such inconsistencies occur in the database and while the database faces high quantities of read and write operations, it is always stable and holds on the foundational properties due to which accurate data is preserved and represented on the application’s end.

Check here –Database Testing Tools

Middleware testing

The middleware part of a mainframe system is critical in ensuring that the application works as expected. If there is an issue in the middleware, the data would not flow correctly resulting in inconsistent presentation on the application layer or the database. Middleware testing ensures that the middleware of a mainframe system is always robust, consistent, and reliable leading to no failures or bugs on both ends.

These four classifications of mainframe testing ensure that the system is tested from all ends and there exists no point of failure.

Where is Mainframe Testing Done?

For the most part, mainframe tests check the deployed code. Scripts use multiple datasets to create input files. All applications must be tested for complete compatibility with the mainframe to go into production. 

The mainframe application, alternatively called batch task, is tested against requirements-driven test cases. Testers access the application under test through the terminal emulator. 

Pre-Requisites to Start Mainframe Testing

Before starting the mainframe testing journey, a few things (knowledge) have to be acquired because the mainframe system is complex and contains a lot of sophisticated elements without any user interface.

• Mainframe basics: Knowledge of mainframe basics is extremely important to work on them. They are entirely different than regular personal computers, have different methods of working, and pose unique challenges that can only be tackled when the knowledge of the system is of high quality.

• Credentials: Mainframe testing can only begin if the tester has access to the mainframe system. For this, terminal login credentials are required so the tester can be authorized to execute tests.

• Obtain permissions: For the credentials given for testing, ensure that all the required permissions are granted. In case permission is skipped, test cases will fail without any flaw in the tests or the system generating false positives.

• Mainframe programming languages: Mainframe testing is possible only through the mainframe programming languages. Knowledge and expertise in languages such as COBOL and RPG are important to interact with the mainframe system.

• Knowledge of databases: Similar to the programming languages, the databases are also specially designed to handle the data capacity of the mainframe system. Knowledge about the foundational structure and method of working is important to start interacting with it during the testing.

• Expertise in tools: Expertise in specialized tools for mainframe interactions is important to conduct mainframe testing efficiently. For instance, ISPF is a tool that provides a menu-driven interface for user interaction with the operating system. These tools work differently than what an application tester is used to and therefore require good practice before starting the mainframe testing.

Mainframe test automation is impossible to execute without knowing certain concepts and control over special tools that are all designed specifically for these large systems. Practicing them on large systems and gaining immense knowledge will increase the quality of testing and hence the overall architecture.

How to do Mainframe Testing ?

• Create a plan based on requirements documents from the business team. The plan lays out how each element or process in the application will be reoriented in the release cycle.
  
• Once the QA team receives the requirements-based plan, they’ll identify which processes will be impacted by changes. It’s the norm for about 20-25% of the application to be impacted by customizations.

So, in summary, mainframe tests run in two parts:

Test the Requirements: Testing an app for its functionality. You can also test the changes highlighted in the requirements documentation. 

Test the Integration: Verify the entire end-to-end workflow of the application that receives and/or sends data to the impacted application. Basically, this is Regression Testing. 

Mainframe Testing Methodology

Scenario: A financial app carries, among its many features, a member enrollment section. This feature draws data from online and offline users who enroll. To test this feature, the app will be tested via online testing & batch testing (as explained earlier). 

• For online testing, you access data through the app screen via which members can enroll. This database is verified by checking the data entered by users through this screen.
  
• Offline enrollment data can come from a third-party website or actual paper forms filled up by users. This data (the “batch”) is fed into the main app’s database through “batch jobs”. The format of this data is used to create the input flat file, which is then used to direct the sequence of batch jobs.
  
  So, to test this feature, you utilize the following methodology:

• The first job in the batch job sequence checks the data entered for accuracy. For eg., are there any special characters in fields that do not allow them?
  
• The second job checks if the data is consistent according to business standards. For eg., a nominee can only be a family member or friend, not an organization.
  
• The third job checks if the app modifies the data into a format accepted by the app database. For eg., if the app only stores an ID number and deposit amount, then it should delete the user name, date of entry, etc.
  
• The fourth job checks that the data is accurately loaded into the database. 

Each job is tested and validated separately. The integration between the jobs is verified by feeding the input flat file to the first job, which validates the database. 

Mainframe Testing Procedures to Follow

Shakedown/Smoke Testing

Verify that the code is deployed in the right environment and has no critical issues. 

System Testing

• Batch mainframe testing validates results on output files and data changes triggered by batch jobs within each test.
• Online testing works at the front end of the mainframe application. It checks if the entry fields are operational. 
• Online-Batch Integration Testing checks systems using batch processes and online applications. Tests data flow & interaction between online screens and batch jobs. 
• Database testing checks the databases from where you get the data for the mainframe application. This is also where the data is validated for layout and storage. 

System Integration Testing

These tests validate the function of systems interacting with the particular system under test. These particular systems are not impacted by requirements but they utilize data from the system being tested. Therefore, you have to test the UI and relevant alert messages (Job successful, Job failed, Updated, etc.)

Regression Testing

Regression tests are run after every other test is complete and the code is to be changed. It checks if the changes have affected the existing, perfectly operational functions. The tests verify that batch jobs and online screens interacting with the system under test are not touched by the current release. 

Performance Testing

Ferrets out bottlenecks in the areas meant for high user interaction & processes, like front-end data. It also upgrades online databases and explores the app’s scalability. 

Security Testing

Check the level at which the application can fend/counteract attacks on its security. At this point, you test Mainframe security and Network security. Security testing checks the validity of the software’s integrity, data privacy, authorization & authentication mechanisms as well as its availability. 

Above tests can be done manually, and if they are repeated frequently, automation is recommended.

Below, we will discuss tools for automating mainframe tests.

Benefits of Mainframe Testing

• Prevents duplication of tests. 
• Serves to improve the app’s overall UX, which increases its market appeal and chances of success. 
• Cuts down on production downtime by ensuring the software functions well under a wide range of user conditions.
• Helps retain customers by giving them flawless, glitch-free user journeys. 
• Reduces overall IT costs by ensuring that the app doesn’t glitch, and needs minimal repairs. 

Mainframe Testing Challenges and Troubleshooting

Here we discuss some challenges faced during mainframe testing and how to troubleshoot them.

A mismatch between requirements and the test handbook

If testers are going by a user handbook or training guide, they might still come across tests and issues in requirements that do not exist in the handbook. 

To prevent this, the QA team should be included in the dev pipelines right from the requirements gathering phase. Testers can then check and state if stated requirements are actually testable, given their current resources. And, if they’re able to see requirements early on, testers can save time, money, and effort by creating specified, targeted tests that the software actually needs to verify it’s functions. 

Finding required test data

Certain tests in the mainframe testing cycle need specific datasets, which need to be found and taken from a larger, more expansive database. Writing tests that do this aren’t exactly easy, especially since the required data isn’t always easily identifiable. 

The solution is to find/create tools that set up data as required by tests. Create queries beforehand to summon existing data. If there’s a problem, you can trigger a request for creating or cloning the data you need.

No impact analysis

Be prepared for the fact that the changes in code can completely change and reorient the system – in terms of aesthetics and/or performance. These changes are often necessary, whether they apply to test cases, data, or test scripts. 

To ensure code changes don’t impact the whole system detrimentally, use impact analysis and strategies to manage scope changes in testing. 

Ad-hoc request

Many errors caught in mainframe testing indicate that the entire application needs end-to-end testing. But, demands for such tests detail the entire test pipeline and timeline, because you need to create new end-to-end tests, which aren’t exactly run in two lines of code. You have to devote significant resources to trigger them in an existing test cycle. 

You need backup plans for such scenarios – regression scripts, automation scripts, skeleton scripts, etc. Again, testers should be included in the brainstorming phase so that they can understand and build backup mechanisms that prevent the test pipeline from getting delayed or bloating.

Best Practices for Mainframe Testing

• Do a dry run of each job under test by using empty input files. Do this for every job to be impacted by the tests.
  
• Finish the test task setup before you start the test cycle. By doing so, you’ll find JCL errors and save execution time.
  
• Create test data, and check that the sets are complete well before the test cycle.
  
• Set auto-commit to “NO” when accessing DB2 tables via SPUFI – which is what the emulator uses to use said tables. This prevents unwanted updates to the system.
  
• Confirm technical inventory and get test + use case data beforehand. Project and test management is key to successful mainframe testing. 

Mainframe Testing in Testsigma

Testsigma

Testsigma is a unified, fully customizable software testing platform that works out of the box. It is designed to help teams build end-to-end tests 10x faster for web, mobile apps, & APIs. You can use Testsigma to create test scripts in plain English scripts – scripts that self-heal and require low or no maintenance. 

Used by 9000+ customers to run tests worldwide (25 million tests & counting), Testsigma stands out by letting you create automation tests fast. Write test steps in English, so it’s a perfectly usable tool for non-tech personnel. 

You can run tests in your local browser/device or run across 800+ browsers and 2000+ devices on our cloud-hosted test lab. Also, You can also view step-wise results for each test and analyze real-time reports & dashboards at the individual, test suite & device levels. Moreover, Testsigma’s intelligent AI automatically fixes broken scripts, heals dynamically changing elements, and suggests fixes for test failures.

Automate your mainframe tests, 10x faster, with Testsigma

Conclusion

Mainframe testing is not easy, which is why it’s usually senior testers who create and run these tests. Use this article as your first stepping stone to writing and executing tests that work. And while you’re at it, don’t forget to pick the right tool for the job. 

Frequently Asked Questions

What is QA in mainframe?

QA in mainframe would mean Quality Assurance for applications built for mainframe systems.

What are Mainframe Attributes ?

• Virtual Storage
• Multiprogramming
• Batch Processing
• Time Sharing
• SPOOLing(Simultaneous Peripheral Operations Online)

Database Testing

Database Testing: What it is, Why & Best Practices

Test Data Management

What Is Test Data Management? | Importance & Best Practices

Test Data Management Tools https://testsigma.com/test-data-management-tools Software Quality Assurance https://testsigma.com/guides/quality-assurance/ Debugging Tools

Top 10 Debugging Tools : The Only List You Need!

Business value of Test Automation

Business value of Test Automation – How to measure and drive it?