What is Coverage, and Why does it Matter?

What is Coverage, and Why Does it Matter?

We hear a lot about coverage in testing, but what exactly does it mean to improve coverage levels? 

Businesses commonly define coverage by the percentage of code covered, the number of tests run, or the number of use cases covered. However, these measurements only capture the surface of quality improvement. Coverage is the designing of sufficient tests to verify that the design and code correctly implements the requirements.

When most people speak of coverage, they are referring to the first definition, where coverage is used as a measurement of the comprehensiveness or thoroughness of some testing. For coverage to be measurable, it must be based on a Model. This model can be a diagram, flowchart, sequence diagram or any other kind of pictorial representation. We can count the body and lines, or the combinations of them, to trace pathways. These pathways can be then be used to define a target coverage. 

Coverage in Quality Modeller

Quality Modeller counts three elements in models:

1. Nodes
2. Edges
3. Combinations of Edges

Nodes

Edges

Combinations of Edges

These can be thought of as our coverage items. We can count each of these items and set an individual target. By doing this, we can begin to measure our levels of coverage by using the formula below.

Quality Modeller generates paths that cover nodes, edges or combinations of edges for the minimum number of pathways for the maximum level of coverage. 

If a tester were to do this by hand, they would need to invent paths that go through our coverage items, and keep track of them. They would continue creating new paths until they had no coverage items left to incorporate. As you can imagine, this would be a laborious process!

We can choose our coverage target by selecting a coverage profile. For each profile, algorithms will generate the minimum number of paths, or test cases, to achieve this target goal.

Coverage Profiles in Quality Modeller

Quality Modeller uses algorithms to determine pathways through our coverage items. There are four coverage levels available:

• Exhaustive coverage, which covers every possible route through the model.
• High coverage, which covers every combination of edges out of each decision matched with edges that enter that decision.
• Medium coverage, which covers all decision outcomes / all the arrows in the model. You might also know this as decision or branch coverage.
• Low coverage, which covers all the nodes in the model

Let’s look at each in detail.

What is The Difference Between High and Medium Coverage

Medium coverage will cover all the edges in a model, while High coverage will cover all in and out edges. In the model below, there are 3 edges, and 2 pairs of edges. Both a Medium and High coverage profile will generate 2 paths.

When we add an additional coverage item, we now have 4 edges and 4 pairs in our model. We need 2 paths in order to cover all the edges at least once (Medium), but we need 4 paths to cover all the in and out edges (High).

Strong vs Weak Functional Coverage

Each coverage profile has a different strength of functional coverage. Weak coverage means you have tested some aspect of some decision blocks, whereas strong coverage means you have fully tested some decision blocks. Strong and weak in this context are not equivalent to good and bad - it depends on what your testing needs are.

Weak functional coverage means that for each decision block, the number of in/out pairs (functional variations) are considered. A functional variation is considered weakly covered if it is included in at least one (successful) test case.

Strong functional coverage means that for each decision block, a functional variation is considered strongly covered if and only if all functional variations for that decision block are included in at least one (successful) test case. If one of the functional variations are not included in a test case, then none of the functional variations for that decision block are considered fully covered.

Relationship between Coverage Profiles

The relationship between coverage profiles in Quality Modeller is hierarchal. Exhaustive coverage includes, or automatically covers, High coverage test cases. High coverage includes all Medium coverage test cases, and Medium coverage includes Low coverage test cases.

The coverage profile becomes stronger and generates more test cases as it goes higher in the level of coverage. However, more test cases does not automatically mean better. We want to prioritize the quality of our test cases over the number of them.

What Coverage Profile Should I Use?

There is no one size fits all when it comes to choosing which coverage profile to use. Like most things in testing, it highly depends on your needs. There are two helpful questions you can ask yourself to determine which profile to use:

1. How many tests can I afford to build and run?
2. Do these tests give me sufficient confidence?

When considering which profile to use, consider the time and financial costs to run larger test suites. Running more tests will be expensive and isn’t always necessary to thoroughly test your system. Determining the minimum coverage that makes you confident in your tests is vital. With Quality Modeller’s tagging feature, you can customize your coverage profile to assign different levels of coverage to each system component. This is a good way to achieve the balance between having full coverage of your system and a manageable amount of test cases.

Summary

Coverage is the designing of sufficient tests to verify that the design and code correctly implements the requirements. To verify system requirements, we can model the application and use algorithms to generate possible paths, or test cases, through our coverage items. Our coverage items consist of our nodes, edges, and combination of edges - together, these items show the design of the system as well as behavioural requirements. 

In Quality Modeller, there are different coverage profiles to select from, depending on how rigorously each component of the application needs to be tested. One size does not fit all for coverage - choosing your target coverage depends on the cost to run the test cases, as well as deciding if your test suite gives you confidence that your application has been thoroughly tested.