Skip to main content

Functional Coverage

Introduction#

The functional coverage capability in Vitaq AI, monitors, measures and then analyses whether important values, sets of values, actions, sequences of values or sequences of user journeys have been exercised during testing. It provides essential metrics for our machine learning AI to influence decision making on next allowable Test Action selection and auto-generation of test data from Vitaq Activity Variables. The reporting coverage view helps testers determine whether Vitaq has achieved its test goals defined by coverage targets that are set on Test Actions, Activity Variables and Sequences.

Vitaq automatically creates Functional Coverage metrics for Test Actions, Test Activity Variables and Test Sequences (user-journeys or use-cases).

Note:

Functional coverage is different to code coverage. Code coverage checks what lines of codes or expressions and branches in the code have been tested. Functional coverage verifies the functionality of the code based on its functional requirements specification. Combining code coverage and functional coverage is highly recommended to provide an overall metric for testing your system.

When 'use AI' is selected in the test activity settings, Vitaq will use the coverage targets as the user-defined goal to achieve 100% coverage with an optimal set of tests.

Enabling automatic functional coverage in Vitaq#

Vitaq will auto-collect coverage metrics into a postgresql database for Test Actions, Test Activity Variables and Test sequences (for user-journeys or use-cases) by selecting the 'Use Coverage' check box in the Test Activity Settings form.

Test Action Coverage#

To define your test action goals, click on the Action you want to include in Coverage and select the Action Properties tab on the right. Enter the Action Coverage target number in the form in the tab. For example if you want Vitaq to select the Products Test Action 10 times or more during a Test Activity run then set it to 10 as shown below.

You will notice the coverage symbol appears on the test action with the target value you have set.

Note

If you do not have a coverage target for a particular action (you don't care how many times that particular action is visited during Vitaq test activity runs, then set the value to 0. No target is then displayed and hence it will not influence the AI driven test automation. A '0' target action will still be executed during test activity runs. If you do not want the action to be executed (because you have not developed the test action script for it yet or that part of your app or software under test is not yet ready, then disable it by clicking in the Enabled box to remove it.) It will then have a 'no entry sign'.

Sequence Coverage (User-journey or use case)#

To define user-journey (or use case sequence) coverage, select the sequence tab on the left.

Click on the + to start creating your user-journey (or use case sequence) coverage. First enter the name you want to refer to your sequence coverage with (this will be the name it will be recorded in the coverage database as) and the target (the number of times you want to see that user journey exercised during Vitaq Test Automation). Below we have used userJourneyBackpack as the name. You then need to click on each action in turn to define the sequence of actions that define your sequence coverage. Notice that incrementing numbers appear on the action selected, denoting the order the actions appears in your sequence coverage definition. If you make a mistake, do not click ok, just start again from the +. To change a created sequence coverage you have to delete it and restart from +.

Note

If you have not set any targets on your Test Action Coverage when defining sequence coverage that includes these actions, then Vitaq will automatically create a target of 1, so that the sequence coverage can be filled.

Record hit on Error#

When Vitaq AI executes Spec Action Scripts into the Application Under Test through WebdriverIO, there will be circumstances in which an exception is thrown. This might be because of a failing assertion (e.g. checking the value of a field), by an error in the Application Under Test, by an error in the Javascript code or by an explicit “throw” statement. When this happens Vitaq treats this as an error in the test action.

By default Vitaq will not record coverage for a Test Action and hence any Sequence (User Journey) containing the Action that experiences that exception (error).

To change Vitaq AI default behaviour and have it record QA Functional Coverage for Test Actions and hence Sequences that ‘error’, you need to set the ‘Record hit on Error’ in the Test Activity settings file.

Test Activity Variable Coverage (Data)#

The user interface provides the ability for auto-generating Test Activity Variable Coverage. For each data type (lists, integers etc) of Test Activity Variables, Vitaq helps you to construct your Data Coverage Targets.

To define targets Functional Coverage has the concept of 'bins'. A Bin is just a counter that counts the number of times that it has been hit i.e. How many times Vitaq has auto-generated that data value (or values in a range of values.)

Double click on the activity Variable name that you want to define targets for, Vitaq will provide a form for you to enter your Coverage Targets. If your Test Activity Variable is a list of text, then the form will split the list into individual bins for each item of the list as shown below for our userNames data.

warning

If you do not set a target for your Activity Variable bins but ask Vitaq to 'record' (see below recordCoverage) the data coverage of that Activity Variable you will get a Vitaq error in the WebdriverIO log such as:

Attribute Error 'Monitor' object has no attribute 'userNames_item'

Where you have created a record Coverage for the userNames Activity Variable but the bins are left at 0, so no targets are set.

For Numbers (Integers or Reals), you have to add your 'bin' structure by defining what range of values you want to represent in each bin. In the shopping web Activity we are interested in auto-generating more searches for indexes 1-2, so we create bins and bin targets to represent that and give them the targets we want to hit.

recordCoverage#

To make sure that you monitor and measure your data functional coverage for Test Activity Variables into the coverage database, you need to use the recordCoverage method in the Test Action script where you are asking Vitaq AI to automatically generate the data using the requestData method.

You need to use the recordCoverage method like this in your webdriverIO spec Action script. Please note that the Activity Variables are denoted as an array and placed in [""], even if you are only recording coverage for a single Test Activity Variable as shown below. You can ofcourse include multiple Activity Variables by listing then in the [] array separated by a , such as ["activityVariable1","activityVariable2","activityVariable3"].

await vitaq.recordCoverage(["userserNames"]);

Here we are recording Functional Coverage for the Vitaq Test Activity Variable AcceptedUserNames.

Warning

If you fail to record Coverage by not including the method in your script with the appropriate Test Activity Variable, you will not get any Coverage measured for that variable and hence the Vitaq AI algorithms will not be able to achieve 100% Coverage.

Warning

If you record Coverage for a Test Activity Variable before generating data (using the requestData method) then Vitaq AI will error but continue informing that you have not yet generated any Data to measure coverage on.

Using the Vitaq Coverage viewer#

To view and analyse your functional coverage results, Vitaq provides an integrated coverage viewer. To open the viewer click on the coverage icon to the right of the settings icon which is in the middle at the top of the Vitaq browser window next to the Projects > Test Activity navigation as shown below highlighted in yellow.

The coverage viewer associated with that Test Activity will launch in a separate tab as shown below.

The name of the automatically generated Vitaq functional coverage results database will be diaplayed in the Test: field. If you have run Vitaq multiple times with the same Test Activity, then you can select previous functional results by selecting them from the drop down menu from their data and number.

The Total Coverage view is the accumulated coverage results for all test runs contained in the coverage database.

Note

If you set a coverage target to 0, then Vitaq coverage measurement will show this as 100% covered even if it shows zero hits because you are denoting it as a "don't care" and hence will sum up in the accumulation of coverage results.

To analyse individual test runs, you can select the individual tests tab and see the list of all test runs. If you wish to delete certain test runs or all test runs from the database then select the individual test run check box on the left side of the individual test run line and then click on the delete selected red dustbin at the top of the viewer. To analyse the individual coverage results in detail click on the right arrow '>' to open out the coverage results for the individual tests.