Consumer Driven Contract Testing | What , Tools & Example

Introduction

To understand consumer driven contract testing. We first need to understand in what scenarios we would use contract testing. The most common example of contract testing is with APIs. Those who come across the concept often have experience with API integration testing. Whereby you test the API from the perspective of how the API is built. Building the tests from the documentation generated from the generated swagger docs or from “contract” documents in some form of wiki.

This approach allows you to confirm at the time of building the API and from the viewpoint of the developers creating the endpoints, that the API works as expected. 

Think about your experience, or imagine if you don’t have personal experience, working with distributed teams who are working on their own services and applications. Commonly teams build microservices split by domain or business area. Each team builds their services or applications independently within their own timelines and focus on testing their own parts. 

When it comes to integrating two parts, there may be a delay in one team delivering or they make an assumption about how the other team is going to consume the service. This leads to communication challenges and human error. No fault of either party, just the way things work when people and time are involved. This is where consumer driven contract testing comes in, a communication tool within software development.

What is Consumer Driven Contract Testing?

Consumer-driven puts the ownership of tests on the “consumer” (e.g. Web or Mobile App). The application consuming an API (or event/message) defines the contract of how they are requesting the service. This is then shared with the “provider”, the owners of the service, who replay the request on their side. This means that the service is being tested using real scenarios by the actual consumers 

To break down the definition, let’s explore the individual parts separately.

What is a consumer

A consumer could be a web app, mobile app or another service. Anything that consumes an event or API service, can be a consumer. Within consumer driven contract testing they are the creators of the contracts. Consumers drive the implementation and/or changes from the service.

What is a provider

A provider is a service providing a message or API. Providers verify the contracts are compatible with how the service has been implemented. Contract testing provokes conversations about changes, the provider still has control over the implementation. They can make informed decisions based on interactions from the consumers and understand the behaviour of all consumers.

What is a broker

So where do you store the contracts? In something called a broker. The broker is where the contracts are stored and verified. The broker holds all the information about versions, tags and environments.

Glasses on a contract by Mari Helin.

Why Consumer Driven Contract Testing?

Consumer-Driven Contract (CDC) testing emerges as a compelling solution to the stumbling blocks often faced with automated end-to-end (E2E) tests involving interconnected components. E2E tests can be sluggish, prone to breaking, expensive, and a maintenance nightmare. CDC testing comes as a savior by focusing on the interactions between components.

1. Isolation and Contracts: CDC testing isolates components and tests their interactions using predefined contracts. Both consumers and providers agree upon these contracts. The contracts are then validated against the provider’s real instance, partitioning a complex system into manageable, testable units. This leads to quicker, simpler, and more stable tests, instilling confidence in the release process.
2. Reduced Reliance on E2E Tests: CDC testing reduces the reliance on extensive E2E tests. While some E2E tests remain crucial for assessing system performance in real environments, CDC primarily covers functional interactions between components. This balance streamlines testing efforts and ensures efficient coverage.

Initially designed for RESTful APIs, CDC testing’s versatility extends to various consumer-provider systems and messaging protocols beyond HTTP. Consumer-Driven Contract testing revolutionizes the testing landscape by emphasizing component interactions, contract-based validation, and optimized testing approaches. It’s a pathway to robust, reliable software releases without the pitfalls of traditional E2E testing.

Consumer Driven Contract Testing Design Blocks

Consumer-Driven Contract (CDC) testing offers a unique approach to testing software interactions, with a focus on collaboration and precision. Here’s how CDC testing breaks down:

1. Consumer Tests with Provider Mock: Consumers begin by creating integration tests against a provider mock. These tests, run within the CI pipeline, set expectations for provider responses, effectively defining the desired contract.
2. Contract Generation: Contracts emerge from the provider mock’s expectations after successful tests. CDC frameworks like Pact use JSON specifications to outline request-response pairs, forming the basis of the contract.
3. Complementary Nature: Contracts and API specifications work together. While API specs describe structure and format, contracts ensure specific interactions result in expected responses, promoting accuracy and clarity.
4. Provider Contract Verification: Provider-side tests validate contracts against actual responses, highlighting discrepancies. This allows quick identification of integration issues, improving debugging efficiency.

Consumer-Driven Contract testing revolutionizes the testing process by enhancing collaboration, reducing uncertainty, and fostering precise communication between consumers and providers.

Reasons to Choose Consumer Driven Contract Tests Over End-to-End Tests

1. Isolation and Faster Feedback Loop

Consumer-Driven Contract (CDC) tests emphasize testing interactions between components in isolation. Unlike end-to-end (E2E) tests involving the entire system, CDC tests focus on specific contracts agreed upon between consumers and providers. This isolation speeds up testing and provides quicker feedback on potential issues.

2. Enhanced Stability and Reduced Breakage

CDC tests contribute to improved stability compared to E2E tests. E2E tests can break easily due to changes in various components, but CDC tests are less susceptible because they validate components based on predefined contracts. This minimizes the chances of unexpected failures when modifications are made.

3. Cost-Effective Maintenance

E2E tests often demand extensive maintenance efforts, as changes to any component can ripple through the entire system, necessitating updates to multiple tests. CDC tests, however, focus on specific contracts, limiting the scope of updates required when components change. This targeted approach reduces the maintenance burden and associated costs.

4. Efficient Testing of Large Systems

Executing E2E tests for large systems can be time-consuming and complex outside dedicated testing environments. CDC tests offer a solution by breaking down these systems into manageable components with their contracts. This enables thorough testing of individual components, ensuring the overall system functions seamlessly when integrated.

5. Increased Developer Confidence

CDC tests provide a high level of confidence for developers before releases. By validating interactions between components based on real contracts, developers gain assurance that their code aligns with the agreed-upon expectations. This confidence encourages smoother releases and reduced post-release issues.

6. Applicability Beyond RESTful APIs

While CDC testing originated with RESTful APIs, its principles can be extended to various consumer-provider systems and messaging protocols beyond HTTP. This adaptability makes CDC tests a versatile choice for different software architectures.

In conclusion, Consumer-Driven Contract tests offer a compelling alternative to traditional End-to-End tests. By focusing on isolation, stability, cost-effective maintenance, efficient testing, enhanced developer confidence, and versatility, CDC tests address many of the shortcomings of E2E tests, resulting in more streamlined and reliable testing processes.

Consumer Driven Contract Testing Tools

Below are the tools that support Consumer-driven contract testing:

1. Testsigma: Checkout this tutorial to see more on how to do CDCT in Testsigma.

Automate your Rest API tests, with your web, mobile, desktop tests in the same place, with Testsigma

2. Pact.io: Has support for most common languages, checkout pact github for more details.
3. spring cloud contract testing (https://spring.io/guides/gs/contract-rest/) 
4. saucelabs (https://saucelabs.com/products/api-testing/contract-testing)

I haven’t compared the tools myself but maybe that’s for another blog post. Some tools have adapted schema testing to do similar things such as k6 (https://github.com/grafana/k6-jslib-k6chaijs-contracts). 

Consumer Driven Contract Testing Example

A good way to get started with CDC testing is when you have control over both consumer and provider within your team. This way the team can understand the full end-to-end process of CDC testing. Another example of when CDC testing can be used is when your team is purely in charge of the Frontend or Backend. Where you are the consumer or the provider.

In this scenario the need to communicate and coordinate changes is very important. By introducing consumer driven contract testing, there are less assumptions made and contract changes can provide a conversation starter. Bear in mind, this doesn’t give all control to the Frontend consumer as the provider won’t ever verify the contract proposal.

This will however encourage a conversation to compromise on any contract changes requested. Sometimes it could just be a misunderstanding or out-of-date API documentation which needs clarifying.

Provider Driven Contract Testing Example

An example I stumbled upon recently while running a session on contract testing was using provider driven contract testing with 3rd party dependencies. If the 3rd party application provides OpenAPI documentation for their service, bi-directional contract testing can be used. Bi-directional provides the ability to use OpenAPI documentation alongside API tests using tools like postman or schemathesis (which is pretty cool as it auto generates the tests from the OpenAPI docs).

Contract testing against 3rd party applications means you can identify the changes easier. This approach won’t stop the 3rd party from breaking the contracts but would at least save some time in debugging where the change has occurred. 

Another example where provider driven contract testing makes sense, is when you already have built an API. At this point there will already be a suite of API tests where the test results can be published to the pact broker for verification alongside the OpenAPI specification.

Bi-directional contract testing offers the ability to use existing tests to create contracts and verify them. Without having to duplicate code and effort. However there are some drawbacks with this method. The abilities to use matchers and state become challenging which could cause the tests to be more flaky.

Conclusion

To summarise my thoughts above, I believe if you own the services internally within the organisation then you should be using Consumer Driven Contract Testing. If you have external dependencies which expose their API documentation then Provider Driven Contract Testing is useful. If you have lots of tests already using mocks then you can use bi-directional method but it comes at a cost.

Low Code Alternative

If you haven’t dived into code before, contract testing could be a bit daunting. As a guest author for testsigma, I tried out their REST API solution. I set up my first test within minutes, added it to a test plan and could schedule the test runs with notification via email on failure shortly after. I only had to look at the documentation once to get the environment url working.

Then to find out you can easily integrate with your most popular CI tools via a curl command makes everything super simple. As I mentioned, if you are new to working with RESTful APIs and think native consumer driven contract testing is a little too advanced (I thought the same when I first came across the concept) then checkout Testsigma low code solution which supports contract testing according to the documentation here: CDCT with Testsigma.

I love tools where you can get setup quickly and the documentation offers great support. While being able to verify the json schema and status codes easily. I even was able to check if an `/auth` endpoint returned the right token format using the regex verification type.

Frequently Asked Questions

Why consumer-driven contract testing?

Consumer-driven puts the ownership of tests on the “consumer” (e.g. Web or Mobile App). The application consuming an API (or event/message) defines the contract of how they are requesting the service. This is then shared with the “provider”, the owners of the service, who replay the request on their side.

Thus, when you need the consumer to define the contract to be honoured by an API, the consumer-driven contract testing is needed.

Is contract testing same as API testing?

Contract testing is a sub-type of API testing. It is done by issuing a contract to both endpoints and ensuring that they honor it.