How to measure test coverage against available requirements - Step by step guide to measure the test coverage

Test coverage is a useful tool for finding untested parts of a code base. Test coverage is of little use as a numeric statement of how good your tests are. Its very important to measure the test coverage because without measurements you can't measure the quality.

For example , you can say if your test coverage is less than 90% you can't go to production.  Its very use full measurement to fine the gaps in written test cases against the available requirements

• Test coverage is is a test review activity, which performed to analyze the missing test coverage to enhance the test quality

• Test coverage measures the amount of testing performed by a set of test.
• Determining test coverage is best served with three measurable categories

• Mapping requirements to test cases
• Test case status
• code coverage analysis, plus a fourth subjective category.

 Benefit of Test coverage:

• It creates additional test cases to increase coverage.
• It helps in finding areas of a program not exercised by a set of test cases.
• It helps in determining a quantitative measure of coverage, which indirectly measure the quality of the application or product. 
• Increasing productivity and efficiency by highlighting tests that may be redundant, i.e. testing the same structure with different data (as there is possibility of finding the defects by testing the same structure with different data).

When to Perform Test Coverage Review

•  Ideally test coverage review activity should be performed before test execution phase

Who will perform the Test Coverage Review 

Test coverage review activity can be performed by different ways

1. Test Coverage review can be performed by peer review, a team member within the same team can perform the review activity
2. An independent resource who is domain expert can perform the the test coverage review
3.  Team lead in a team can perform the test coverage review

How to Perform Test Coverage Review 

• Analyze the  requirements ( SRS) or User Stories if you working in Agile
• Identify the test conditions from the requirements
• Identify the test scenarios (basic flows , alternative flows, field validations) , read the article to know about test attacks (Test Heuristics and data attacks for your application under test( Tips for better application quality))
• Analyze the written test cases 
• Map the written test cases with the identified the test conditions or test scenarios and requirements. Identify the gaps between written test cases and requirements.
• Create defects against all missing test cases.
• Calculate the test coverage index

Check list

• Have all the scenarios specified in the requirement – both explicit and implicit, been converted into Test conditions?
• Have the related areas that could possibly be affected by the implementation of the requirement been identified and included in the test cases? (Identify the impact areas and check with the test cases)
• Has the test data set, if required been generated appropriately?
• Has the field level validation been covered appropriately?
• Have the required negative scenarios been identified in the test conditions?
• Have the steps been correctly given in appropriate sequence for each test scenario? Steps/Actions should state very clearly the sequence of actions to be carried out on the system by the user. All statements should be definite. Ensure that terms like  “If”, “In case” etc are not used.
• Have the Expected Results been identified correctly?

• Expected Results should clearly state how the system should respond to the user actions given in each step/action.    
• Ensure that too many things are not included to be verified under one expected output.
• Ensure that separate cases are written for multiple verification of the application’s behavior.
• Vague statements like “Appropriate message/value/screen” etc, should not be part of expected result. Every detail should be clearly spelt out.

• Test cases written for multilingual
• Browsers are mentioned in test cases
• Regression test cases are properly created
• Read this article to more understand about test attacks Test Heuristics and data attacks for your application under test( Tips for better application quality)

 Defect Severity

• Critical
• Major
• Medium
• Minor

Defect Types

• Missing TC - Basic Flow
• Missing TC - Alternative Flow
• Missing TC - Validations
• Missing Test Step
• Invalid TC
• Invalid Expected Results
• Invalid Test Steps 

How to calculate the coverage index

•  Divide your requirements into use case or user stories depends on your working structure
• Against each use case or user story report the defects if against each missing test case or invalid test case
• Put the defect severity , defect type 
• Based on the reported defect put some status against user story or use case
• Assign some weight to user stories or use cases depends on reported defects against them
• Calculate the weight  , that is your test coverage index it could be like ( 50%, 70%, 90% etc...)
• Higher index means high quality of test coverage, low number means low quality of test coverage.

Test Coverage Index= Sum ( Status*weight)/ Total User Stories(except out of scope)

Status	Weight	Defect Severity							Comments
		Critical		Major		Medium		Minor
Full Coverage	100	0	and	0	and	0	and	0	No defects found
Above Average	90	0		0		1	or	1	One minor or one medium defect
Average	70	0		1	or	>1	or	>1	One Major or more than one medium or minor defect
Below Average	50	1	or	>2	or	>3	or	>5	One critical defect or more than 2 major or more than 3 medium or more than five minor defects
Poor	25	>1	or	>3	or	>5	or	10	More than one critical defect or more than 3 major or more than five medium or 10 or more than 10 minor defects
No Coverage	0								No test case found
Out of Scope	This status have no impact on test coverage

Now you have check list, defect type, defect severity. Follow the below example to calculate the Test Coverage Index

Example

 Let's suppose you have three use cases in your project.

1. Login Process
2. Create Users
3. Logout Process

Use Case	Status	Defects Count
		Critical	Major	Medium	Minor
Login Process	Full Coverage	0	0	0	0
Create User	Above Average	0	0	0	1
Logout Process	Above Average	0	0	1	1

As mentioned above weight will be assigned against status

Full Coverage=100

Above Average= 90

Test Coverage Index : 100+ 90+90/3  =  94%

So you have test coverage of 94%

Please share your feed back with your suggestions or improvements