Data Driven vs Keyword Driven Frameworks

Data Driven vs Keyword Driven Frameworks for Test Automation

What is a Test Framework? 

A test framework is a set of guidelines which can be followed to create test cases and related processes. These guidelines can be about coding practices, storage and retrieval of test data and test results, interaction with external resources and many other things.

A test framework usually contains internal libraries and reusable code modules which provide a foundation for test automation and can be leveraged to build test automation systems for different types of applications.

Benefits of a Test Automation Framework

A test automation framework offers several benefits, some of which can be mentioned as follows:

  • Efficiency: A test automation framework provides a structure for the test automation tasks like test creation, test execution, organization of tests, test data creation or reporting.

    This helps reduce confusion, duplication of efforts and redundancy. Thus, using a test automation framework makes building a test automation system more efficient.

  • Easy Maintenance: As all test cases are created following the same structure, debugging and isolation of bugs becomes easier.

  • Reduced Manual Interactions: The main purpose of automating any process is to reduce manual labour and, implementing a test automation framework helps reduce manual labour that was involved in the execution of test cases.

  • Flexibility to Run Selective Test Cases: A test automation framework generally provides the flexibility to execute a single test case, a suite of test cases, or all test cases.

    Such flexibility helps during development and debugging sessions of test scripts and also for selective testing, for e.g. if a feature needs to be tested, only test cases related to that feature can be selected and executed.

Different Types of Test Automation Frameworks:

Test automation frameworks can be broadly classified as follows:

  • Linear: In linear frameworks, test scripts are created in a record and playback manner which generates sequential test steps with hardcoded test data.

    Test scripts can be generated quickly with this framework, but they aren’t much reusable other than repeating the execution of the same test case with the same test data.

    This provides an easy way to learn a new tool or understand a test workflow since it doesn’t need coding expertise for a beginner, but not good for scaling purpose

  • Modular: In modular frameworks, the application can be divided into different modules which can be tested independently.

    Test scripts can be reused to an extent, thus reducing the test script development time. Programming knowledge is required to work with these frameworks.

    Using object-oriented concepts, an abstraction layer can be developed which can help in easier maintenance of test scripts.

  • Library Architecture: In library architecture frameworks, common tasks needed for test scripts are consolidated into a library of functions, which can then be called in different test scripts.

    Test scripts developed using this method are more maintainable and reusable. Technical expertise is needed to work with these frameworks.

  • Data-Driven Framework: In data-driven frameworks, test data is separated from test scripts and stored in an external resource such as text file, excel spreadsheet, CSV file or database table.

    These frameworks provide the flexibility of executing the same test script with multiple data sets.


  • Keyword Driven Framework: In keyword driven frameworks, the test logic is divided into keywords and functions. A sequence of keywords is used to define the test scripts and these keywords are further defined as functions to implement the desired behaviour.

    An external table is generally used to maintain the mapping of keywords and functions, therefore this framework is also known as Table Driven Framework. For e.g. HP QTP and Selenium are widely used for keyword-driven testing.

  • Hybrid: In hybrid frameworks, more than one framework is used to achieve the desired objectives. The combination should be designed in such a way that weaknesses of one framework are compensated with the strengths of other frameworks.

  • Behaviour Driven: These frameworks allow the test cases to be written in such a format that they could be easily reviewed by non-technical stakeholders for a project for eg. Clients, Business Managers, Product Managers, Project Managers etc.

The UI that could be reviewed is in a human-friendly language that could be easily understood. For eg. Cucumber, Specflow, Testsigma etc.

We shall now discuss the data-driven framework and the keyword-driven framework for test automation, in detail. 

Need for Data-Driven Framework for Test Automation


In non-data driven testing frameworks, test data is embedded in test scripts which makes it difficult to update the test data at any point of time. This becomes a festering problem in a product’s lifecycle; test data may need to be updated several times for various reasons.

In data-driven testing frameworks, test data and test scripts are separated which makes it easier to maintain and update the test data at any point of time without affecting the test scripts. Similarly, any changes to the test scripts can be made without affecting the test data. 

For example, if you need to test a video player, a very basic testing script would look something like this:

For non-data driven framework for testing:

Load A.mp4

Click the Play button

For data-driven framework for testing:

Load media

Click the Play button

The Test Data file would look something like this:

A.mp4

B.mov

C.avc

Benefits of Data-Driven Framework for Test Automation

Data-driven frameworks offer several benefits such as:

  • Less number of Test Scripts: As the same test scripts can be executed with multiple test data sets, the total count of test scripts required is greatly reduced.

  • Less Error-Prone: As test data and test scripts are different entities, each can be updated independently without interfering with others. This reduces the chances of introducing errors in the test automation system while updating either test data or test scripts.

  • More Flexibility: Since test data is stored externally, it is easier to perform positive and negative testing using the same test scripts just by updating the test data.

  • Easier Maintenance: Data-driven test automation systems are easier to maintain as compared to non-data driven test automation systems, where test data and test scripts are tightly coupled and cannot be updated independently.

Challenges with Data-Driven Framework for Test Automation

Using a data-driven framework does come with its own challenges. A few of which are:

  • Programming Expertise: Testers need to have the good programming knowledge to write the test scripts.

  • Planning is Important: Test automation systems need to be well planned in terms of test scripts, test data, and their interactions.

  • Getting Started Takes Time: Getting started with a data-driven framework does take more time as compared to non-data driven frameworks where test scripts can also be made using record and playback manner. 

Now, there are automation frameworks that offer codeless testing and easier integration with data files for data-driven testing. For e.g. Testsigma is one such tool.

Here, no programming expertise is needed as the test cases can be created in Simple English and test data can be configured in the tool itself to enable data-driven testing.

Need for Keyword Driven Framework for Test Automation

In non-keyword driven frameworks, writing and maintaining the test scripts becomes the prime responsibility of few automation experts in the team, whereas, the non-automation experts in the team who have a much better knowledge of the product are left out.

This creates a bottleneck in the testing team’s productivity. Team members creating the test scripts are always under pressure to write more scripts to quickly build the test automation system.

On the other hand, team members not creating the test scripts don’t have much faith in the quality of test scripts created as they know the product better than the team members creating the test scripts and try to depend on manual testing. 

Keyword-driven frameworks help in eliminating these bottlenecks and create a balance where all team members with and without the programming knowledge can contribute to creating the test automation system of the product. 

In keyword driven frameworks, test scripts are a sequence of keywords which are mapped to functions defining specific behaviours.

Team members who are well-versed with the product can use these keywords to create the test scripts and team members who are automation experts can implement the functions corresponding to keywords to define the required behaviours. 

Mapping of keywords and their corresponding functions can be maintained in an external resource such as a table.

Anyone writing the test scripts only needs to refer to this external resource to see which keywords are available and anyone implementing the functions needs to update this external resource with the newly defined functions corresponding to the keywords. 

Since writing the test scripts using keywords and implementing the functions mapped to keywords can be done independently, the team can work on both in parallel, which also helps in boosting the team’s productivity.

For example, if you need to test a video player, a very basic testing script would look something like this:

For a non-keyword driven framework:

Load media

Click Play button

For a keyword-driven framework:

Load

Play

Keywords and Functions Mapping would look something like this:

KeywordFunctionDescription
LoadloadMedia()Loads the media in the player.
PlayClick play()Clicks the play button in the player.

Benefits of Keyword Driven Framework for Test Automation

Few of the benefits of keyword-driven frameworks are:

  • Application Abstraction: Test scripts can be designed with/without the application under test. This is very useful when test cases need to be developed before the application is ready for testing.

  • Enhanced Team Productivity: As keyword driven frameworks allow everyone on the team to be part of the automation journey, it helps in boosting the team’s productivity and confidence.

  • More Suited for Agile: Keyword-driven frameworks are very helpful when products are developed using Agile methodologies.

    When there is a change in the product requirements, you only need to update the function’s implementation for accommodating the new behaviour rather than changing every test script.

  • Platform Independent Test Scripts: If the product behaviour is expected to be the same on different platforms, the same test scripts can be reused.

    Platform-specific changes would be needed only in the implementations of functions mapped to these keywords.

  • Code Reusability: Since multiple test scripts can be defined using the same set of keywords, the code is reused to a large extent.

Challenges with Keyword Driven Frameworks

Using keyword-driven frameworks has its own challenges, such as:

  • Keyword Management: Inventing and managing unique meaningful keywords for various types of actions becomes a daunting task for large projects. This sometimes poses problems for scaling.

  • Planning and Management are Crucial: If a keyword-driven framework is not planned ahead it could lead to the creation of keywords that are not useful in the long-run or may need too many edits every time a keyword is added.

    Thus, thinking about what keywords to implement and how becomes a crucial step to make the framework productive and efficient.

  • Programming Expertise is Required: To implement the functions mapped to keywords, you still need team members who have good knowledge of programming and test automation to make the whole system work together.

    Though, this problem can be easily solved if a tool that does not require coding expertise is used. For eg. Testsigma.

Difference Between Data Driven and Keyword Driven Frameworks

We can summarize the differences between Data-Driven and Keyword Driven Frameworks as follows:

  • Need for Programming Expertise in the Team: Though both the frameworks need programming expertise in the team, keyword-driven frameworks have the ability to also leverage non-programming expertise in the team, such as manual testers who lack programming expertise but are well versed with the product.

    This allows everyone in the team to contribute in building the test automation system of the product they are working on. On the other hand, data-driven frameworks don’t allow such flexibility.

    To build a test automation system using a data-driven framework, you need programming experts in the team who can write test scripts using a programming language.

    Non-programming experts of the product don’t have many opportunities here to contribute in building the test automation system of the product they are working on.

  • Need of Product for Developing Test Scripts: It is easier to develop test scripts using the keyword-driven framework when the product development is not complete, only the implementation of keywords depends on actual product development.

    On the other hand, to develop test scripts using data-driven frameworks, the actual product is needed.

  • Planning: Keyword-driven frameworks need more extensive planning as compared to Data-driven frameworks. With data-driven frameworks, you only need to plan for what test data and test scripts are needed.

    With keyword-driven frameworks, you need to plan for keywords and their implementations along with test data and test scripts.

  • Management: If not properly planned, test automation systems using keyword-driven frameworks could be much harder to manage as compared to data-driven frameworks.

  • Maintenance: A well-planned keyword-driven test automation system is easier to maintain as it has well-defined layers of abstraction between test data, test scripts, keywords and their implementation.

    On the other hand, a data-driven test automation system has abstraction only between test data and test scripts.

Selecting the Right Test Automation Framework

Different factors can be considered in selecting the right testing framework for your product. Let us explore them.

  • Same behaviour needs to be tested with different sets of test data: If your product needs to test the same behaviour with different sets of test data, a data-driven framework can be a good option.

  • Test data is expected to change often: If you know test data is expected to change, there is no point in keeping it embedded in your test scripts. A data-driven framework is your need here.

  • Programming expertise in the testing team: If all the members in your testing team are equally good at programming, you can choose a non-keyword driven framework.

    If your testing team has members who are not good at programming but have good knowledge of the product to be tested, you can choose a keyword-driven framework so that everyone can contribute in creating automation for the product.

  • Product behaviour is expected to change: If you know there can be a change in the way some features of the product are expected to behave, a keyword-driven framework can be your option as you would only need to update the implementations of few functions at one place rather than changing multiple test scripts. 

Depending on your need you can also choose to have a hybrid framework, where you can combine data-driven and keyword-driven frameworks to take advantage of both.

For example, if you need to test a video player, a very basic testing script would look something like this:

Load

Play 

Test Data file would look something like this:

A.mp4

B.mov

C.avc

Keywords and Functions Mapping would look something like this:

KeywordFunctionDescription
LoadloadMedia()Loads the media in the player.
PlayClick play()Clicks play button in the player.

Managing and scaling such frameworks over time may bring its own challenges along with the benefits.

Conclusion:

Different types of testing frameworks may be suitable for different types of products and teams.

Before finalizing any framework for building your test automation system for the product, it is important to make an analysis of what you need from a framework and what are your strengths as a team to work with the framework.

This would help you in choosing the framework that’s right for your team and product.

Sign up now to write your first test step!
Go!