What is Bebugging? | Benefits, Tools & Best Practices

“Overconfidence by the programmer could be attacked by a system that introduced random errors into the program under test.”

This quote starts the introduction of the term “Bubugging” officially for the first time in 1970 by Gerald M. Weinberg. The idea behind introducing this term was to understand whether the team of programmers is effective or not in finding the bugs in the system. In general, the rate at which a real bug is found and rectified cannot be necessarily represented by a constant formula. Some days are good while some are really bad. This makes researchers think about deducing some technique where the team has some data regarding their efficiency and whether any improvements are required or not.

The result of this deduction is the introduction of bebugging to the world, a concept that resonates with fault injection with slightly different motives. Its current stand helps define the quality of the system, explore the faulty areas, and motivate the team to work more productively in the future.

What is bebugging?

Bebugging is a concept in the software development lifecycle to knowingly introduce bugs into the system. To understand the motive behind bebugging, let’s compare it with a surveillance system because after all, software testing too is a type of surveillance.

A surveillance system is a network of cameras all attached to multiple screens in a room, also called a control room. It may have one or two men always constantly looking at the screens to identify any suspicious activity. The more sensitive the area (for instance, a military zone), the more efficient these men should be (or expected to be). But the problem is, how do we measure the quality of their surveillance? What if there was suspicious activity at night, and they failed to notice? 

Representational image to illustrate a surveillance room

The most naive suggestion would be to create something suspicious in front of a camera knowingly and see whether the team in the room responds to it. This method, however naive it sounds, is the most effective method in this scenario and hence, the motivation behind bebugging as well.

Bebugging introduces bugs into the system to check the efficiency of the programmers. While the idea is to get a statistical analysis of the effectiveness of the team, the goal is to deduce whether we have actual unknown bugs in the system or not. This can be achieved by checking whether all the known bugs were explored by the team. If not, then there must be real bugs. Therefore the methodologies of our work need to change or we need to train the team more for the hidden scenarios. It is a type of fault injection technique and is sometimes also termed defect seeding or defect feeding as the term “bebugging” is often confusing. The reasons for naming the process “bebugging” could relate to “debugging” as it explores the bugs in the system which is the opposite of bebugging. However, evidence of any such relation is unknown.

Why perform bebugging?

Bebugging the software means the team wants to know whether the bug can be discovered by the target team or not. The target team can be programmers or testers depending on the scenario. Hence, a clear reason we use bebugging is to know the effectiveness of the teams in their respective domain. It helps expose weak areas in the team and redefine the areas where actual bugs could be present.

If we connect this main motive behind bebugging, it is quite clear that all these changes will ultimately impact the quality of the software and help keep maintenance to a minimum in the future.

Advantages of bebugging

If we could just list down the benefits of bebugging, it would help us understand the broader view of adopting this method in our cycles.

• Better quality software: As mentioned in the previous section, bebugging helps enhance the quality of the software by defining the weak areas and statistical analysis of our work.

• Covers weak areas of test suites: When bebugging is done, a known bug is introduced. Since it is a bug, it should ideally be caught by the test suites otherwise we can deduce that our test suites do not cover all the scenarios. Hence, we can work on strengthening them.

• Increases test coverage:  When the test suites are exposed, so is the area in the code base that was not yet covered (or just partially covered). Working to cover this increases the test coverage of the system as well.

• Explore undetected bugs: When the known bugs are explored, they also expose the connected weak holes in the system. These holes may have gone unexplored to the end-user without bebugging and therefore enhancing the quality of the software even more.

• Wide scope: Bebugging is a software testing technique that provides methods acceptable to any domain. Since we are just introducing new bugs, the process can be followed anywhere. Therefore, bebugging does not restrict a team to a certain area. 

• Covers weak areas of the team: Undetected known bugs mean that the team is not working efficiently on their tasks. What follows is a quick feedback and research on the areas that need improvement for the future.

• Motivates team to work: Defects introduced with bebugging once explored give an idea to the target team that they are working well. This motivates them to work harder with more energy and provides them with excitement factor.

• Gives more control to the team: Fewer weaker areas mean the team is watching over a larger code base. Any future discrepancies could therefore be easily detected in the majority of the areas which gives better control to the testing team over the application.

• Saves cost and time: Needless to say, whatever we discussed in this section will save a lot of tester’s and programmer’s time in the future, especially from the maintenance perspective.

The reader may include other advantages that are de facto with respect to any testing domains.

Disadvantages of bebugging

With a long list of advantages, bebugging does come with a few disadvantages as well that a tester should know before implementing it in their projects.

• Risky: Bebugging injects defects and bugs into the application. This is a risky job as these defects can snowball into something big unknowingly that will take additional time and effort from the tester.

• Need expertise: Due to the high-risk nature of bebugging, they should only be conducted by experienced individuals.

• Need accuracy: Highly accurate models are required for better analysis and effectiveness. This is time-consuming and hard to figure out in the first place.

These disadvantages are not many and therefore it may look like bebugging is a profitable adaptation. However, the damage that can be done by a wrong bebugging process can certainly overshadow all the advantages and result in huge time and money losses.

Preparations and pre-requirements for bebugging

The disadvantages of bebugging make us aware of the risks we take when we incorporate it into our testing and development cycle. This doesn’t mean we skip it altogether. If it increases the quality of our application without any dependencies, we just need to find a way to proceed ahead.

Analyzing these disadvantages, it is easier to gather that by preparing well enough the team can cover the weak points easily. These preparations and pre-requisites can be listed below:

• Experienced employees only: The team behind bebugging works with analyzing scenarios where defects need to be injected, types of defects, their implications over the source code, their seriousness, and much more. All this needs experienced employees exclusively and this is what the bebugging team should constitute.

• Gather data: The best defects can be introduced only when we have some data backing up those defects. Therefore, gather a variety of data, possibly from various sources, and insert defects only after relying on them.

•  Prepare post-release past bugs: Bugs in the past that were not detected by the team and eventually made their way to the end-user have special importance in testing. They showcase the possible holes in the system and just closing the explored bugs doesn’t necessarily mean that the weak points are closed too. Hence, the testing team should have a report of such bugs and the bebugging process should contain similar bugs as well.

• Requirements of the application: The bebugging method is best used when it synchronizes with the application’s requirements. Injecting bugs that do not mean anything is just a waste of time. The team should therefore prepare a document of requirements before starting the process.

• Test results: For the same reasons we gather past post-release bugs, we also need to accumulate the past test results. A lot of the time team finds patterns in test failures over a series of reports. This pattern can prove to be very advantageous in crafting the bug similar way.

• Bebugging tool: Bebugging can relate to many angles of an application. It may also depend on the type of application under test. Hence, a suitable tool needs to be selected prior to starting the process.

A few prerequisites may also depend on the team that will initiate the process. For instance, a team with members that have never done bebugging is not desirable. This may require a couple of hiring in such a case. The same goes for the tool to be used due to which training of the team members may be required. Hence, it is recommended to extend this list of prerequisites by brainstorming among the team members as well.

Bebugging techniques

There are two techniques to conduct bebugging on the application:

• Runtime injection.
• Compile injection.

Runtime injection

A runtime injection is the process of injecting faults into the system while it is in the running state. This is achieved by using software triggers that are initiated based on the condition they are set into. There are two types of software triggers used in runtime injection:

• Time-based triggers: This type of trigger can be set to run at a particular time or frequently in a set interval.
• Interrupt-based triggers: This type of trigger uses a hardware-based mechanism to interrupt at a time when that action is executed with the hardware.

An example of runtime error

Both of these methods can be used to apply triggers with the following methods:

• Network level bug: This type of bug is related to networks such as corrupting the receiving packets to show irrelevant information.

• Memory level bug: This type of bug corrupts the main memory or secondary memory by impacting the storage hardware such as registers and cache. 

• System call bug: This type of bug corrupts the system calls that are provided by the operating system using kernel.

If these are not your requirements to achieve bebugging, then probably the user wants to take the path of compile injection.

Compile injection

When the fault is injected into the source code rather than at the runtime, it is termed compile injection or compile fault injection. An example of compile injection is an error in the logic. Compile injection can be done in the following ways:

• Code mutation.
• Code insertion.
• Logic alteration.

Compile injections are often targeted to test the test suites and their effectiveness because a test suite should work against faulty logic if it was constructed when the logic was correct. Although, that doesn’t mean it is solely used for test suites. It can be used for multiple purposes as would any fault injection technique.

All these techniques may seem manual. This is true. A careful fault insertion requires manual precision and the tester should know what is being inserted at what place and the consequence of the same. However, automation has still tried to cover this topic under its realm which creates an interesting combination.

Where to implement bebugging?

Now that we are well aware of what is bebugging and how we implement it in the software cycles, we need to know about the most effective targets for optimum results out of it. For this, we need to answer one question – what is it that our complete application depends on?

While some experts may believe that a laid down system will eventually work as a generic method, that is never the case. Each application is different in its own way and each application brings its own vulnerabilities when published. Not everyone should focus unnecessarily on the security of the application when they are never dealing with data or any other sensitive information.

So, the question is to explore the pillars of your application and what impacts it the most. Let’s take an example of a mobile application or a web application that provides cloud storage services. With respect to it, what would be the pillars of this application?

1. Storage-based applications consider each user as valuable because once a user is enrolled, they rarely move to other platforms due to too much overhead. Hence, each user should be able to open the application (native or web) on their device. Therefore, cross-browser testing is important and bebugging should include that on priority.

2. Cloud storage depends on payment systems because if payment is not completed, the user will not be able to enroll and hence the organization loses business.

3. Uploading services should work flawlessly as the complete business model depends on the uploading of data. Here, the user will initiate an upload request with multiple files at once. A good scenario is to consider failures of a few files and still continuing with others without exiting.

4. A similar scenario should be considered for downloading services as well.

5. Finally, security is extremely important in such cases because the storage services contain sensitive and personal data. Bebugging should keep security issues in the list while injecting faults to know if we are working effectively towards security or not.

After combining these factors, we extend this list by analyzing the past records in our application and similar applications as ours. For instance, there are multiple cloud storage services on the internet and many of them have been through various issues in the past. The most popular is the history of data breaches. Hence, it should be assumed that our application is vulnerable to similar weaknesses and bebugging should be considered for closing those gaps as well.

Once these two factors are considered, the testing team gets the exact location where faults need to be injected and the type of defects as well. 

Bebugging and Test Automation

Test automation has always helped testers and developers cut down the time of their work and wrap things quicker with accuracy i.e. no or extremely minimum errors. Bebugging has not been one such field where test automation could extend its arms and help in a similar way. This is because of the methods applied in the bebugging. They may be deduced by artificial intelligence, but writing the script for the same is not a good idea. Moreover, the bugs need not be the same all the time which is an additional problem in constructing scripts.

For now, test automation exists in bebugging but holds authority only on the things that do not require manual inspection. These involve memory space related errors where automation can block some memory and behave as if the memory is not sufficient. Apart from this, many researchers have tried to devise new test automation methods with new and effective strategies but they cannot be implemented practically in big applications. This includes automated integration of fault injection into the ASIC design flow, Fault Injection in Actuator Models for Testing of Automated Driving Functions, and An Automated Fault Injection Technique Based on VHDL Syntax Analysis and Stratified Sampling. 

On the other hand, artificial intelligence is something that has a promising future with automated fault injection techniques. With effective machine learning models, they will be able to understand past behaviors, and defects, and join them with current requirements and possibilities of areas where bugs can be inserted. Although, this is not currently in either research or development.

Tools used for bebugging

The following tools will facilitate the bebugging process of the team:

• Bstorm
• The Mu Service Analyzer
• Holodeck
• Xception

They may not be a complete solution and the tester would still want to use manual methods to accomplish bebugging successfully. 

Test Automation Tools like Testsigma do help in this regard. Providing artificial intelligence to combine with testing, testers can get their work done easily without writing any programming scripts at all. High-risk jobs like bebugging can be accomplished quickly and safely due to the use of the English language with accurate reports as the platform uses real devices.

Best practices for bebugging

Bebugging is a risky business. One wrong bug can take apart the work of weeks from the developers and other teams. Hence, testers are advised to follow best practices to reap the benefits without causing any harm to the application. In the same line of thought, below are the five most important and critical practices to be followed.

Do not target the production environment directly

If the bug is for the production environment, then it should be injected into the non-production environment first. This will help the tester understand the impact and the consequences of the bug. Also, it will show the behavior after the bug is injected in a controlled environment that can be reverted easily anytime. If the bug is related to customer-specific actions such as number of active users, requests, etc. then load testing should be able to accomplish that in non-production first.

Document the implications of bug insertion

Before inserting the bug, know the implications of it around the application. Document these implications and get them reviewed by a senior member of the team or present them in front of the team to get ideas out of them. Different people will be able to present different areas as per their experience getting affected by that bug. It will help understand and control the bug.

Learn from the past and set examples for the future

An extremely important point to note is the past mistakes we have made, the bugs that entered production, similar application experience, etc. The more we learn about our past, the better we can predict the type of bug and what bug needs to be injected (a concept used by machine learning). Document why this bug was selected and its reference to the past. It will also help future testing sessions on bebugging.

Keep risks to a minimum

As repeatedly reminded in this post, bebugging is prone to high risks. A small mistake can damage the application which will take a lot of the team’s time to fix. Hence, it needs to be done carefully. To keep this in check, it is always advisable to document the risks associated and keep them to a minimum. Due to its very nature, we can never eliminate the risk of bebugging completely. Hence, the risk calculation helps us deduce what could go wrong if the bug insertion turns the other way.

Control the bug by setting boundaries

A good practice to keep the risk to a minimum is building up walls that detect the bug’s behavior and not letting it jump through it. This is a safe net built up to know if the bug is acting in the controlled state or expanding to high-risk areas. If the testers have calculated the implications, they may know up until which point the fault can be reversed and traced. Along this area, the boundaries can be set and if breached, the testers can look back and re-iterate the process again with optimizations.

These best practices will help the testers perform bebugging without any side effects.

Conclusion

The rate at which the bugs are resolved can never give an idea of how well our development and testing teams are doing. Some days are good and we receive a couple of bugs while some are bad that can fill up the week’s schedule for each tester on the job. This raises the question “How effective is our team?” and “Are we efficient enough to make the application so secure it never fails our expectations?”. Rarely anyone can answer this based on past experiences.

While the above questions are important, the only method to know the reality is by actually injecting bugs into the system and verifying whether our team is able to identify them or not. This process is called bebugging and surprisingly, it has been of great importance for testers, developers, and organizations since its inception. The only problem we face is that bugs can disrupt a lot of inner workings of an application which makes this a very risky job to do.

Through this post, we tried to convey the importance of bebugging and how to perform it in the most idealistic way. If the testers prepare themselves and take all the precautions, bebugging can explore holes in the system that could have been extremely hard to uncover before the release. With this, we hope this guide helps you in your future endeavors and serves as a reference when you perform bebugging next time.

Frequently asked questions

What is the difference between debugging and bebugging?

Debugging and bebugging are two opposite processes with goals of execution that are poles apart. The main aim of debugging is to expose the bug (unknown bug) from the system, find its root cause, explore the area of impact, deduce the corrective actions, and finally implement those corrections in the code.

Bebugging, on the other hand, is the process of inserting bugs into the source code knowingly. The main aim of bebugging is to identify the effectiveness of the system we have constructed, the team’s efficiency, and whether there exist any weaker areas in the application where an unknown bug could hide and get transferred to the end-user.