Skip to main content

AI-driven Test Automation

Introduction#

Vitaq is equipped with AI machine learning algorithms that are coverage led. When you run Vitaq in AI mode, it will use the coverage model (defined by you the user) as the goal to try to achieve 100% coverage for. It will do this by working out what is the most optimal set of tests to achieve this target.

The variable nature of your Vitaq Test Activity will provide many next allowable action selections, and many different test data inputs. By informing the Vitaq AI machine learning algorithm what you care about, i.e., what user journeys you want to test and what test data you want generated, Vitaq will continuously run until it achieves 100% coverage of your coverage model.

Selecting AI mode#

To select Vitaq to run in AI mode, go to the settings and click on the 'Use AI selection' box as shown below

You can also use the vitaqai service configuration or command line switches as shown in Vitaq Ai WebdriverIO Service

Quick guide: The Vitaq AI can be guided by use of two sliders Variability and Variability Decay.

Setting Variability to low and Variability Decay to high => Fastest run-time to maximise Coverage

Setting Variability to high and Variability Decay to low => Slowest run-time to maximise Coverage but allows for more variability to be deployed to 'explore' around the user-journey test space to help find defects that may not be exposed when just focusing only on action sequences and Test Activity Variable data that you define as goals.

The power of Vitaq is delivered by its highly variable nature. Action to next allowable (or not allowable for negative tests) actions and highly varied test data generation.

To make sure you still get variability when driving Vitaq using its AI machine learning algorithms you can set the variability level between low and high. When set to high Vitaq will 'explore' with highly variable action selections and data generation whilst still focusing on delivering 100% coverage with an optimal set of tests. When you set variability to low, the Vitaq AI machine learning algorithm will be more 'directed' in its test selections to achieve 100% coverage.

The Variability Decay is used to influence the rate at which the Vitaq AI machine learning algorithm will reduce the variability to 'home in' on achieving its coverage target goals. Set to maximum decay and it quickly reduces the variability. Set to minimum and then it will slowly focus in on each run, reducing the variability slowly each run.

Once Vitaq has achieved 100% coverage of your defined coverage goals in the coverage model it will stop running. Vitaq AI uses the total coverage at the end of each seed to inform and guide it through the next seed.

When using Vitaq AI in conjunction with exhaustive actions it is possible to get seeds that run for a very long time or even indefinitely, even though in non AI mode they always stop. This is because the AI is selecting which action to go to and may never choose the action that causes the seed to stop. In this way it is also possible to have a lot of coverage covered by a single seed. It is recommended to use call limits on the leaf actions (those with no next actions) to ensure that the Test Activity run stops.

Debugging Sequences in AI mode#

To help debug very large Test Activities that have deep conditional logic in their Test Action Scripts, and a lot of possible user journeys we have created the feature to provide users with the ability to run a single user-journey sequence in AI-mode. To access this single sequence run capability, you must have AI-mode selected in the settings form and you must have created a Sequence in the Sequences tab on the left of the Test Activity diagram window.

To run a single sequence you can either use the command line switch or configuration file as shown in You can also use the vitaqai service configuration or command line switches as shown in Vitaq Ai WebdriverIO Service. Vitaq will run in AI-mode, executing Test 'seed' runs to achieve the Coverage target for the sequence selected.

The Vitaq coverage viewer will only display the coverage results for the sequence selected. i.e., a single sequence.

The AI machine learning will iterate through the number of seeds needed to achieve the sequence coverage target. As it runs (in single sequence run mode), it will choose the next action defined in the sequence. If the Test Action logic disables this action from being selected, then it will continue to try for a number of seeds in the expectation that any data needed will be auto-generated to 'enable' the next allowable action in your sequence in any conditional logic defined in your Test Action Script.

Randomised Seed#

When testing websites or mobile apps that 'remember' a username or piece of entry data, you may want to auto-generate a different test data input on every seed run of Vitaq AI running in AI-mode. The default AI running mode of Vitaq is start the seed value selected from 1 and increment on each successive seed. So if you run Vitaq repeatedly during testing mode, you will be repeating the seed selection starting from seed value 1 again. The effect of this, as you now should know, is that you will get the same value auto-generated. Same seed, same value, different seed, different value.

To change this default mode, we have provided the ability to select a randomised seed in the settings form or in the configutation or command line switch sjown in You can also use the vitaqai service configuration or command line switches as shown in Vitaq Ai WebdriverIO Service. This chooses a seed number from a very, very large value range, hence giving a different value each time Vitaq AI starts a seed tun.

This behaviour provides data generation aligned with the way the popular 'Faker' JavaScript library works, which is used by test developers.

Selecting the Randomised seed can also be useful if you want to run Vitaq in Exploratory Testing mode but driven by the AI Machine Learning algorithms to maintain a focus on the user journeys that you care about.