Skip to main content

AI-driven Test Automation

Introduction​

Vitaq is equipped with AI machine learning algorithms that are coverage led. When you run Vitaq in AI mode, it will use the coverage model (defined by you the user) as the goal to try to achieve 100% coverage for. It will do this by working out what is the most optimal set of tests to achieve this target.

The variable nature of your Vitaq Test Activity will provide many next allowable action selections, and many different test data inputs. By informing the Vitaq AI machine learning algorithm what you care about, i.e., what user journeys you want to test and what test data you want generated, Vitaq will continuously run until it achieves 100% coverage of your coverage model.

Selecting AI mode​

To select Vitaq to run in AI mode, go to the settings and click on the 'Use AI selection' box as shown below

You can also use the vitaqai service configuration or command line switches as shown in Vitaq AI WebdriverIO Service

Quick guide: The Vitaq AI can be guided by use of two sliders Variability and Variability Decay.

Setting Variability to low and Variability Decay to high => Fastest run-time to maximise Coverage

Setting Variability to high and Variability Decay to low => Slowest run-time to maximise Coverage but allows for more variability to be deployed to 'explore' around the user-journey test space to help find defects that may not be exposed when just focusing only on action sequences and Test Activity Variable data that you define as goals.

The power of Vitaq is delivered by its highly variable nature. Action to next allowable (or not allowable for negative tests) actions and highly varied test data generation.

To make sure you still get variability when driving Vitaq using its AI machine learning algorithms you can set the variability level between low and high. When set to high Vitaq will 'explore' with highly variable action selections and data generation whilst still focusing on delivering 100% coverage with an optimal set of tests. When you set variability to low, the Vitaq AI machine learning algorithm will be more 'directed' in its test selections to achieve 100% coverage.

The Variability Decay is used to influence the rate at which the Vitaq AI machine learning algorithm will reduce the variability to 'home in' on achieving its coverage target goals. Set to maximum decay and it quickly reduces the variability. Set to minimum and then it will slowly focus in on each run, reducing the variability slowly each run.

Once Vitaq has achieved 100% coverage of your defined coverage goals in the coverage model it will stop running. Vitaq AI uses the total coverage at the end of each seed to inform and guide it through the next seed.

When using Vitaq AI in conjunction with exhaustive actions it is possible to get seeds that run for a very long time or even indefinitely, even though in non AI mode they always stop. This is because the AI is selecting which action to go to and may never choose the action that causes the seed to stop. In this way it is also possible to have a lot of coverage covered by a single seed. It is recommended to use call limits on the leaf actions (those with no next actions) to ensure that the Test Activity run stops.

Debugging Sequences in AI mode​

To help debug very large Test Activities that have deep conditional logic in their Test Action Scripts, and a lot of possible user journeys we have created the feature to provide users with the ability to run a single user-journey sequence in AI-mode. To access this single sequence run capability, you must have AI-mode selected in the settings form and you must have created a Sequence in the Sequences tab on the left of the Test Activity diagram window.

To run a single sequence you can either use the command line switch or configuration file as shown in Vitaq AI WebdriverIO Service. Vitaq will run in AI-mode, executing Test 'seed' runs to achieve the Coverage target for the sequence selected.

The Vitaq coverage viewer will only display the coverage results for the sequence selected. i.e., a single sequence.

The AI machine learning will iterate through the number of seeds needed to achieve the sequence coverage target. As it runs (in single sequence run mode), it will choose the next action defined in the sequence. If the Test Action logic disables this action from being selected, then it will continue to try for a number of seeds in the expectation that any data needed will be auto-generated to 'enable' the next allowable action in your sequence in any conditional logic defined in your Test Action Script.

Randomised Seed​

When testing websites or mobile apps that 'remember' a username or piece of entry data, you may want to auto-generate a different test data input on every seed run of Vitaq AI running in AI-mode. The default AI running mode of Vitaq is start with a seed value of 1 and increment on each successive run. So if you run Vitaq repeatedly during testing mode, you will be repeating the seed selection starting from seed value 1 again. The effect of this, as you now should know, is that you will get the same value auto-generated. Same seed, same value, different seed, different value.

To change this default mode, we have provided the ability to select a randomised seed in the settings form as shown below or you can also use the vitaqai service configuration or command line switches as shown in Vitaq AI WebdriverIO Service. This chooses a seed number from a very, very large value range, hence giving a different value each time Vitaq AI starts a seed run.

This behaviour provides data generation aligned with the way the popular 'Faker' JavaScript library works, which is used by test developers. Vitaq delivers 'random stability', meaning that if you run a test with the same seed, you will get the same data generated and the same sequence of actions (until you edit the test). To replicate this in 'Faker' you should always use the randomness seed.

faker.seed(123);

Having a hard coded value for the seed in Faker will result in the same values getting generated by Faker for every seed run in Vitaq. Instead, the seed for Faker can be generated from a test activity variable in Vitaq. Since the generated value of this variable will vary with the Vitaq seed being used it will cause Faker to generate different values for each seed run, but they will always be the same value for a given seed. e.g.:

// Generate a random seed value for Faker
// ... where fakerSeed is an integer test activity variable
faker.seed(await vitaq.requestData('fakerSeed'))

Note that data generated with Faker can not be recorded in the coverage.

Selecting the Randomised seed can also be useful if you want to run Vitaq in Exploratory Testing mode but driven by the AI Machine Learning algorithms to maintain a focus on the user journeys that you care about.