QA vs UAT: Difference between QA and UAT Testing

Allow me to start this post with a quote. A quote from one of the greatest curious minds the world has ever got. 

“I learned very early the difference between knowing the name of something and knowing something.” – ― Richard P. Feynman

Advice

1. Read until the end of this slightly dry yet actionable content
2. Be skeptical 
3. Make sense of what is written
4. Use it in your work and see what works and what’s not
5. Develop your own theory

Starting with writing Why!

Before we proceed, I have some questions for you. I know that only you, the reader, have the right answer to those questions. 

1. Why do you want to know QAT vs UAT or QA vs UAT?
2. Why did you frame QAT vs UAT or QA vs UAT?
3. Why is it important to know for you at this point of time?

If I am not wrong, such curiosities and search terms framings just don’t come out of the blue. There must be a solid reason behind this.

Give yourself ~2 minutes and take a pen and paper or open a text editor of your choice and state the top 3 reasons by writing ‘I want or need to know QAT vs UAT or QA vs UAT because…

Reason 1:

Reason 2:

Reason 3:

Be honest with yourself , and please write your answers first. Read them to yourself and continue from there.

Now assuming that you are done with writing your reasons, let me share the reasons that I thought and let’s see if I can match your reasons to a good extent. 

Common Why’s 

Some of my quick and best guesses(es) about why you want to know about QAT vs. UAT are below. I think that the chances are that…

• You are preparing for a Software Testing interview and
  • You found that QA Testing vs UAT is a frequently asked question
    • You want to know and probably mug up(?) the answer
• You are brand new to software testing and
  • You are hearing a lot about these two terms QAT andUAT
    • You are curious to know
      • What these terms are
      • What these signify
      • How are these useful
• You have been in the software testing business for a long time, maybe from Waterfall, V-Model age and
  • You still need to be convinced if you fully understand QA Testing vs UAT.
• You might be setting up a new IT/Software testing business and
  • A new prospect just sent out a proposal and mentioned QA, QAT, UAT, AUT, etc
  • You may be hearing these terms for the first time
  • Trying to make sense of what all this means to proceed further with consulting someone.
• You come from Agile, DevOps, TestOps, and CI/CD & Continuous Testing eras and
  • You are dubious about what and how exactly QAT vs. UAT works in today’s world
• You, like me, are
  • Writing an article or blog post on the same/similar topic 
  • Doing some research and analysis
• Other reasons ( please specify in the comments box)
• Don’t prefer to disclose
• Just like that

Assumptions and Assertions

Before I jump on writing a single word on QAT vs. UAT and start a conversation with you (the reader), my first and foremost job as a tester is to explain my assumptions and assertions.

I am making three assumptions…

Assumption 1: QAT stands for QA Testing, and QA stands for Quality Assurance. Quality Assurance in software development context which is also referred as SQA.

Assumption 2: UAT stands for User Acceptance Testing in software development

Assumption 3: You (the reader) are here because you have some interest in knowing what QAT vs. UAT is.

Assertion 1: AT (Assurance Testing) is out of the scope of this conversation

Acknowledgment

In the following paragraphs and words, I will define QAT and UAT and explain my way of handling the ‘vs’ aspect of the original topic QA vs UAT. 

It would be a shame on me if I don’t, at least once, mention Toyota, when it comes to quality.

Unless I am terribly wrong with my studies about quality and work of the early pioneers in the quality space, I think I can safely say that Toyota, must have inspired many industries’ thought-leaders and practitioners. They further have guided many software and technical professional organizations to give different definitions of QA (Quality Assurance) and designed and provided processes, frameworks, books of knowledge , templates, systems, and what not around quality assurance and related processes in the last decades. 

Also, I am thankful to many people and organizations who misunderstood or underestimated the QA, User Acceptance, and Testing and allowed so many people to continue searching and browsing the internet to find out some useful answers.

Wordifying QA vs UAT

I am hoping that we all know the meaning of ‘vs’ and hence I am further wordifying the topic as ‘QAT’ versus ‘UAT.’

Now, the foundation stone of the further discussion on this topic of QAT vs UAT shall depend upon how carefully we use the preposition ‘versus’. In the context of the topic, the original issue and my assumptions, now may take shape as 

1. Quality Assurance Testing in opposition to or in contrast with User Acceptance Testing.
2. Quality Assurance Testing against User Acceptance Testing.

Which one of #1 and #2  should I be picking to move ahead from here?

My preference is to pick the first derivation and let me simplify it further for you.

Quality Assurance Testing in contrast with User Acceptance Testing.

So far, so good? A mutual (virtual and mental) agreement is important because whatever I am going to say about incoming lines is based on the premise that we both agree on the most favorable derivation of QAT vs UAT or QA vs UAT.  Does it make sense? If Yes, good. Let’s dig deeper.

Let’s surface the contrast.

Now, after setting a foundation, let’s move towards building blocks of understanding the contrast in the coming paragraphs. I will…

1. Explain what I know from my  Quality Assurance Testing experience
2. Explain what I know from my User Acceptance Testing experience

The ultimate purpose of the explanation is to bring awareness to the following…

1. Understanding the ideas and terms 
2. Common perceptions about those ideas and terms
3. Their usefulness and relevance to us (testers)

Quality + Assurance + Testing

What is Quality?

एकं सद् विप्रा बहुधा वदन्ति । ऋ. 1.164.46

English Translation:- Scholars express the same truth in different forms.

Quality is perceived and defined differently by different people. Yet, everyone understands what is meant by “quality.”

1. Generally speaking, Quality is the merit of something. 
2. The relevant dictionary meaning of Quality is “the degree of excellence.” However, this definition is relative. The supreme test in this evaluation process lies with the customer / end-user of the product.
3. Quality can also be defined as ‘The totality of the features and characteristics of a product or service that bear on its ability to satisfy stated or implied needs.’ Ref: Oxford
4. Definition of Quality from some of the industry pioneers 

• “Quality is value to some person.” (Gerald M. Weinberg)
• “Quality is “fitness for use.”” (Joseph M. Juran)
• “Quality is conformance to the specifications.” (Philip B. Crosby)
• “Quality means a predictable degree of uniformity and dependability with a quality standard suited to the customer.” (W. Edwards Deming)
• “Quality is value to some person, at some time, who matters.” (James Bach and Michael Bolton extending on Gerald M. Weinberg’s definition)

Again, time for some action on your part. Give yourself ~2 minutes and take a pen and paper or open a text editor of your choice. 

1. Write (don’t copy paste) these definitions, at least the first three to start with. 
2. Observe what might be going on in your mind when you are writing.

Assuming you’re  done, let me tell you that all of these above definitions of Quality resonate with me. It took time to understand and internalize and I am yet to explore a lot in software product development so that I become qualified enough to devise a new definition of Quality now. In a nutshell, I am happy to test in different situations per all these great definitions.

What is Assurance?

• In the sense of confidence
  • Conviction about something
• In the sense of Insurance
  • Certainty or guarantee about something

What is Quality Assurance?

1. Quality assurance (QA), as I have closely observed while working with companies who have QA COEs, ISO, and CMMi standards, adopts a more extensive set of activities. 
2. Those activities span the software inception, design, development, testing, and delivery process and aim to ensure Quality throughout, at least from the software producer’s stand point. Please remember that the ultimate evaluation is in buying customers and / or end-user hands.
3. In cases where a small but influential section of leadership, management and practitioners are in sync and understand that ‘Quality is everyone’s responsibility.’ I found that QA aims to inspire and build excellence in each work product coming out of each iteration of the software development life cycle, where each work product leads to producing quality software.
4. In some cases, QA is seen as Gatekeeping (in a helpful manner but sometimes over-engineered and causing unhealthy conflicts among teams) mechanism. Even in those cultures, QA still aims to build quality but I found that it is not more on ‘Everyone’s responsibility’ mindset and actions. 

What is Testing?

1. Simply put, Software testing involves evaluating an idea or a product through exploring, experimenting, experiencing, and identifying potential problems where the identified problems might badly impact, for example, your product’s valuation, sale, reputation, usage, and user experience. 
2. We should remind ourselves that (Software) QA differs from (Software) Testing. Software testing shall be integrated into your overall Quality Assurance framework but is not equivalent to QA. 

However, it is common to see how we use these terms interchangeably (the true synonymy problem) even w/o giving a thought! 

The misconception often leads to the imagination about the tester’s role as a gatekeeper of Quality, development process auditor, and accountable for preventing bugs from moving to production.

QA Tester or QA Personnel? 

Sometimes, I contextually play a QA personnel role. I do it when I consciously accept a gatekeeper, a quality-related processes writer, quality assuring practices designer, or auditor role. Again all from a software producer standpoint. My hands-on experience with ISO, QMS, CMMi, and PMO practices design often help me become a QA personnel.

One of the popular notions of QAT (Quality Assurance Testing) is built around the ideas below…

Quality (situational and often subjective) can be assured (guaranteed, made certain) with the help of…

1. dedicated QA (Quality Assurance) function comprising testers
   1. Who can and should do something additional to testing
      1. Including 
         1. Gatekeeping
         2. Policing
         3. Process Auditing 

Or something similar that gives management the confidence of assuring Quality!

Recommended Read: https://developsense.com/blog/2010/05/testers-get-out-of-the-quality-assurance-business  

QA Testing

Now assuming that by now you have good literacy around Q, QA, and T. 

Please take a pen and paper or a text editor of your choice. Let’s play a permutation – combination (or vice versa) and word game for the words Quality, Assurance, and Testing!

Ready? Remember, since (Assurance Testing) is out of the scope, Quality of Assurance Testing is not an option!

1. Assurance of Quality of testing.
2. Testing for Assurance of Quality.
3. Quality of testing for Assurance.
4. Anything else that you can think of?

Whatever you choose, the common term is ‘Testing,’ and let’s now talk about 5W1H of Testing.

5W 1H of Testing (Qualitatively!)

Whether a Quality Assurance environment is given to you or not, it is critical to understand that as testers, our first job is to test qualitatively. Here are some time-tested guide words that, if practiced consistently and well, shall help you test appropriately, timely, and inexpensively.  

Why: Testing primarily provides information on perceived behavior, risks, and issues about the product under test. Testing is done with the intent that the information gathered and provided on time, will be useful for key stakeholders in making timely and inexpensive decisions to achieve a specific goal.

What: We test everything that matters per discovered, defined, and/ or evolving scope. The scope is usually guided and determined by the business goals, including profit, operational successes, regulatory & compliance goals, market-driven priorities, release schedules, and money.

When: Testers today understand that gone are the days when testers tested using traditional SDLC phased and labeled cycles to decide when to test. These testers also know that shifting right or left is not new. Skilled, expert, adaptable, and opportunistic testers test whenever they can by hunting the opportunities to test. Avoid the urge to test only when code is delivered to you unless… (curious to know what you think here, do add your thoughts in comment.)

Where: In the software product context, we test at strategically and/or tactfully carefully chosen layers to bring effectiveness, efficiency, and credibility to our testing process. E.g. from a web application standpoint, some humanly operable key layers include GUI, API, CLI, and Database. 

Who: We test for stakeholders but are not limited to the high stake decision-makers only. We test for our next immediate information consumers, including developers, business analysts, product owners, IT / DevOps teams, project managers, and fellow testers as well just to name a few. The information takes a more meaningful and consumable shape when it flows from our immediate fellows to the people in the hierarchy. 

How:  We deploy different techniques, scripted and no-scripted, automation-assisted approaches to test appropriately to capture the required information as quickly and inexpensively as possible. We deploy those approaches to test ideas, specifications, code, configuration, functions, features, integrations, and an integrated suite of applications and, if lucky, often an overall system too. 

QAT in a nutshell

In a nutshell, I would like to define QAT as

You perform context-driven, purpose-driven, risk-based and qualitative testing in a Quality Assurance focused environment. It may seem counterintuitive but in my overall experience sometimes the QA culture is well defined and in some cases it is assumed to have and in some cases it is built over the time. 

In all such settings, you might sometimes be asked to wear a non-tester hat to do something useful for the organization, like process engineering, auditing, and sometimes gatekeeping.

The problem arises when we fall into the trap of understanding QA as testing or testing as QA. It is a BIG NO. 

What is UA (User and Acceptance)?

Let’s recall what Richard P. Feynman said once,

“I learned very early the difference between knowing the name of something and knowing something.” 

What you just read above is a full form of the acronym UAT.

First and foremost, User Acceptance Testing, at least in my world-view, is the beginning of a product’s acquisition, use, feedback loop, change requests as it might seem relevant to the target users, and finally a time based acceptance, by those users. 

Most of the time, we, the testers, desire to perform UAT in a production-like environment with most likely user scenarios, data sets, and favorable or unfavorable conditions. Of course we do that but sometimes we forget that we are NOT the users. We, at the best, in the best of our spirit, can be a very close representative of the end-users.

Most of the time, our thought-land for user emulation is not that wide and open!

A product that first goes into the end-users hands is the beginning of deep diving into understanding what “Customer Acceptance” might mean now and more importantly the degree of satisfaction with that Acceptance.

To avoid becoming cryptic and sounding like a pure theorist, let me deep dive into each word that makes ‘User Acceptance.’

User

20% of users generate 80% of the revenue!

Simply put, a user (especially someone who matters) uses a software product to get something meaningful or valuable done and desires to feel smart and accomplished during or after using those products/services/ideas. 

A user may not necessarily be a buying customer, though there is a high possibility that she might be. For example, I am a buying customer and end-user of a handful of apps on my mobile phone.

User’s Story?

Where there’s a user, there’s a story!

Alas! Only a few are available to listen to the user’s stories in today’s tools, frameworks, and templates-driven fast delivery-focused world. Most of the time, people write and while they write wonderfully, it doesn’t feel like a story.

A story is more than – “As a User, I want, So that!”

Again I would like to quote Richard Feynman, who emphasized that. 

…knowing the name of something doesn’t mean you understand it. In order to talk to each other, we have to have words, but we often talk in fact-deficient, obfuscating generalities to cover up our lack of understanding.

In a later blog post, more on User Stories and how to tell, write and test. But here is a food for thought for you.

Acceptance

A development (all-inclusive) team’s understanding of the user’s acceptance expectations (desires) of the product continues building as the stories (user requirements) and product evolve.

In the context of user stories, most of the acceptance usually tends to focus on Acceptance Criteria and very often the criteria focus on the technical implementation. Very few, that I have seen, focus on user behavior and goals (as a story) to accomplish first followed by means of accomplishing. It sometimes feels like Cart before the horse. I am not blaming people for this because listening to and writing such stories and acceptance criteria is hard. 

Also, as a tester, you must have heard many things like ‘Think like a user’ and ‘Putting yourself in users’ shoes,’ Right? 

I prefer the second one to try to imagine what is the criteria that can make a user accept the product with a delightful experience, happiness, a smile on her face, and finally resulting in a word of mouth publicity. 

Remember: It is hard to think like a user, while it might be easy to assume to ‘think like a user’. There is a difference.

UAT (User Acceptance Testing)

The known

While I see the following as a credible source to refer to get a traditional and formal view of UAT. I directly copied pasted from this source and I didn’t feel a need to change as such. 

User Acceptance Testing (UAT), which is performed on most UIT projects, sometimes called beta testing or end-user testing, is a phase of software development in which the software is tested in the “real world” by the intended audience or business representative. This type of testing is not intended to be menu-driven, but rather to be performed by business users to verify that the application will meet the needs of the end-user, with scenarios and data representative of actual usage in the field.

Source: https://uit.stanford.edu/pmo/UAT 

The Confession:  I had been in the teams doing UAT the way defined above. I have set up UAT teams from scratch and instructed / coached them performing UAT this way. I sometimes became a business representative and felt proud. 

I can imagine most of us might be doing UAT this way. As long as such a way of imagining and performing UAT helps avoid slipping costly bugs into the production, keep your people happy not burning the night oil and users are happy using your product, kindly continue.

However, in my experience, this is one side of the coin which looks beautiful, and the other side, where I burned the midnight oil, found what we missed in UAT, which caused issues including delaying deployments. That side is yet to explore for some of us, I guess.

The other side of the coin

I offer my perspective on UAT based on my experience. I believe that, from a user’s acceptance standpoint, UAT largely means..

‘Testing for the sole purpose of determining if a product is ready and to what extent to kick off in the market and further can be used at the desired scale.’ 

In the phased approach of SDLC, I think it was inevitable to do it towards the end once the fully integrated product is ready to use, even by the testers. 

Nowadays, in the ever-growing iterative yet very short and increased bug-tolerant software delivery cycles based SDLC (Agile and DevOps), UAT is done more frequently. In fact in each iteration, each integration, deployment and delivery there is a continuous acceptance testing in small and testing in large. 

Give yourself a break and read this interesting post: https://testing.googleblog.com/2011/03/how-google-tests-software-part-five.html 

Users operate, testers test!

While I do understand what ABCT (Anybody Can Test) might mean to someone. I don’t think that users can test like testers, and also if the development team can operate like users! 

I know by experience that in the UAT

• The intended users (usually a small group of favoring and devil’s advocates) and the designated domain and subject matter experts (including testers)
• Within the real-world situations yet controlled environments
• Largely following 80/20 principle
  • Operate like The Users. 

This is where they come up with appreciation for what is good in the product and they also uncover the problems that matter to them and they want to get those problems addressed!

The real acceptance

The important thing to note is that when it comes to UAT, it is primarily aimed at customer or end-user evaluation based on how they operate and define quality, especially on the operations side! Recall What is Quality?

I may be exaggerating but I think that the real acceptance or reluctance to accept can be truly determined when intended users use the product in the field. The acceptance should reflect in your organization’s bottom line, reputation, increased word-of-mouth publicity, resulting in new customer acquisition, retention, and growth.

Automation in QAT and UAT

A necessity or a luxury?

Context, Purpose, Value, and Quality of Life matter!

A little experiment

I performed a little experiment while typing the following words using Google Docs. 

I misspelled ‘mistage’, and luckily my instance of Google Doc was configured (knowingly or unknowingly) to detect and correct the misspelling automatically. As a result of that setting, the word ‘mistage’ almost instantly got detected and changed to the word “mistake.”  

So, spell check and automatic correction happen in real-time and are fully automated.

It is not Eureka now. It is simply an example of Automation. No?

What’s the big deal, you should be asking? Let me ask you this

1. Could I do all spell checks on my own? 

• Yes! I could. At least for the words I know and can type, I have a dictionary.

1. Should I do all the spell check and correction on my own? 

• Yes and No and Maybe and that decision(s) depends upon
  • Purpose
  • Time
  • Efficiency
  • Budget
  • Accuracy expectations
  • Risk management for the issues that may arise from my failures in timely detecting and precisely correcting a spelling.

So Can vs. Should. 

In my overall experience, after working with a handful of not-so-social media-known and non-famous yet brilliant automation thinkers, I have no hesitation in saying the following.

Automation, as I prefer to think today

‘Automation is one of the solutions to those specific problems that arise during software testing, which should be early, quickly, cheaply, and consistently solved by machines executing a set of well-designed and maintained scripts. Such problems, beyond problem identification, solution development, and execution, should be handled by someone like machines other than skilled testers because they must use their time and the organization’s resources wisely.

If the above statement is not easy to understand, please forgive me. Consider Cost, Value, and Energy regarding Automate or Not to Automate decision-making. As I learned from experts, I prefer ‘Should we automate’ over ‘Can we automate’! I prefer thinking of ‘Just In Time’ or ‘Use and Throw’ Automation or a Long term investment.

Let’s look into what Bill Gates once mentioned.

“Automation applied to an efficient operation will magnify the efficiency. … Automation applied to an inefficient operation will magnify the inefficiency.”

Automation in QAT and UAT

From my point of view and professional experience, Automation plays an important role in the overall software development and delivery life cycle (including testing) during QAT and UAT. Automation is usually done using open-source, commercial, or both tools. Some key examples of Automation during QAT and UAT are as follows.

Analysis and Design side

• Automating application modeling
• Automating test ideas (in some cases test cases) generation
  • Combinatorics
• Automating test data creation
• Data analytics for customer reviews and issues for improved test design

Predesigned and/or fuzzy execution side

• Automating execution of smoke (a.k.a Build verification tests) and sanity checks
  • Smoke: I used to automate checks that constitute key business rules using a keyword-driven automation tool. 
  • Sanity: I used to automate and execute tests for key functions and data integrations to determine if the application is stable and integrated sufficiently to move ahead for testing complex scenarios
• Automating processes
  • Batch jobs
  • ETL (Extraction, Load, and Transform)
• Automating load injection (Load testing)
• Automating a11y checks
• Automating vulnerability scans (Security testing)
• Automating necessary repetitive and mundane tasks
• Regression tests execution (One of the most popular and yet highly misused)
• Automating Web APIs testing including performance

Reporting and Analytics

• Automatic and custom reports generation & delivery on test execution completion

Some day and possibly in some other post, I would like to share my thoughts on AaS (Automation as a Service) way of thinking on which I am working these days. Here is a glimpse.

Automate your end-to-end tests for web, mobile, desktop, and APIs, 5x faster, with Testsigma

Conclusion

Remember that someone said, “Testing at its best is, Sampling.” 

1. To find the contrast between QAT and UAT, first you will have to assume that the UAT process, in your context, is out of QA influence! No? This means your quality assurance activities are not applied to the user acceptance testing process, are you good with making that assumption?
2. Practically all you are doing is ‘Sampling’ and In QA culture you do sampling across the SDLC and STLC because this is what Quality Assurance Activities will make as a mandate or guidelines. 
3. Second, If your organization is still on phased SDLC, you may want to hold on to the premise of QAT in contrast to UAT, but if you are in DevOps or Agile or a tester who tests based on context-driven principles, you know what you can do at what time. 
4. I don’t advise waiting until the end to start UAT because that seems unrealistic. 

Nowadays, in the ever-growing iterative yet very short and increased bug-tolerant software delivery cycles based SDLC (Agile and DevOps), UAT is done more frequently. In fact in each iteration, each integration, deployment, and delivery, there is continuous acceptance testing in small and testing in large. 

5. I also advise looking at all key written or non-written acceptance criteria critically, having timely discussions with subject matter experts, actual users, and business analysis / product owners.

In nutshell, I find it way better to start operating the product like a user within the QAT life cycle and find out the ‘Wow’, ‘Ok’ and ‘Oh, No’ things earlier in SDLC. 

Probably the time has come where we all start thinking QAT and UAT rather QAT vs. UAT

Question for you: Did I answer your question QAT vs UAT or QA vs UAT?

Frequently Asked Questions

Does QA or UAT come first?

QA or quality assurance is supposed to be done through out the SDLC, while UAT is done once the product is ready to be used by the target user. 

What comes before UAT testing?

All types of testing – unit, component, integration, system – that the testing or the quality assurance team does before it is approved for UAT comes before UAT testing. After UAT testing, release testing, and production testing may be done. 

What is dev vs QA vs UAT vs prod?

Dev stands for “Development”
QA stands for “Quality Assurance”
UAT stands for “User Acceptance Testing”
Prod stands for “Production.