There has been a lot of press commenting on the benefits of Cypress vs WebDriverIO. We launched our version 1 of Vitaq AI Test Automation tool at the Selenium Conference in London a couple of years ago. This was our first proper foray into the world of web and mobile app testing. For several years we had steadily built up a good run-rate business in the embedded software test automation space with our Vitaq tool.

Where we came from​

Our biggest claim to fame was that between 30-40 million cars, out of the 85 million cars coming off global production lines each year, were tested by our tool. Vitaq had become the leading AutoSAR (the operating system software in automotive control systems) test automation solution. We also had many electronic systems companies using Vitaq for a variety of applications. From testing video broadcast codecs to aerospace control systems to wireless network protocol stacks.

Our small self-funded company had made a name for itself in the mission and safety critical testing space. Our unique approach was finding functional bugs that other approaches missed. We had many tens of companies and many hundreds of users but we just could not get it to grow to hundreds of companies and thousands of users.

Each time we applied the tool to a different application problem, it turned into a custom software engineering task. Then we discovered the mobile and web application testing market thanks to King the developers of Candy Crush. They were making $4.2 Million a day at the time. So failure was not an option! They wanted to apply Vitaq to a very complex mobile application development project.

One ring to rule them all​

That was our Eureka moment! We discovered the beauty of WebDriver. Exactly what was lacking in the embedded electronics systems world. A world-wide web consortium standard interface to provide a platform and language-neutral wire protocol. It has become the industry standard way for out-of-process programs to remotely instruct the behavior of web browsers. Better still, the brilliant opensource WebDriverIO takes this global standard and extends it with all the robustness you need to proactively solve complex test automation problems for both web and mobile application testing. In the darkest depths of our non-scaling business despair, we had stumbled across the one ring to rule them all (my precious;).

Late to the party​

Over the last several years there has been an explosion of Test Automation companies coming to the web and mobile app market. Are we too late to the party? What we had noticed is the vast majority of these tools automate the running of manually created tests. Vitaq AI is focused on auto-generating the end-to-end functional tests themselves. Rather than writing test, after test, after test, Vitaq provides a Test Activity Diagram, a picture that generates 1000 tests! Tests that explore all of your user journeys in highly variable ways. It is this human like behaviour that help you find functional bugs that other approaches miss.

The only other tool that we see doing something similar to us today is Eggplant Test. One of our lead technical people left us to join them as we started to engage the web and mobile test automation space. With the tens of millions of dollars invested by their Private Equity backers, they got into the market 10x quicker than us. But to get payback for their corporate PE backers, they had to charge 10x more than us.

Standing on the shoulders of open source giants​

To really deliver on the opportunity we knew we needed to stand-out from the crowd. So just supporting Selenium and Appium would not be enough. We would need to provide a complete solution to both QA and Dev testers. Our initial teaching customers informed us on this, following our launch at the Selenium Conference in London. They asked us to stand on the shoulders of a one of the "next-generation" test automation frameworks. The big question was which one should we build on first and the choice was between Cypress vs WebDriverIO.

Cypress has over four times the number of daily downloads than WebdriverIO and has a great on-boarding experience. But it cannot support native mobile apps and because it runs in the browser using a "pre-primed" test lag, its architecture has fundamental limitations. It cannot support multiple tabs for example. With the release of Selenium 4.0, some of the gaps are closed with Cypress with the brand new Chrome DevTools Protocol API. This enables Selenium tests to not only mock server requests and intercept responses, but to fully leverage all of the possibilities of Chrome DevTools.

Conclusion​

Don't get us wrong, we love Cypress but we had to make a decision on where to apply our limited development resources first. Taking into account the architectural limitations of Cypress for web testing and the lack of native mobile app testing, then pitch all that against the superb progress that WebDriverIO 8.0, Selenium 4.0 and Appium have made. The WebDriverIO decision was easy to make. We have found working with the excellent team at WebDriverIO an absolute joy and have now just released the Vitaq AI service and registration page

Do you remember the "1000 songs in your pocket" story that Apple used to launch the iPod?​

A picture is worth a 1000 tests, so how about having "1000 tests in one diagram?" Are you fed up of writing spec file, after spec file, after spec file to capture all of your required tests? Difficult to reuse, not representative of how real users will journey through your app and painful to manage what you have or have not tested? That's why we set about developing Vitaq AI!

Test Activity diagram for Vitaq to test Vitaq

With Vitaq AI, we have simplified the problem for you. We use a single spec file to represent the testing to be executed on a specific page of the application under test, Vitaq AI then controls the order of execution of the spec files. At it's simplest this allows us to define a simple sequence of spec files to execute in a defined order but it also allows us to execute the spec files in any order that is allowed by your app under test, including doing things like executing the same spec file multiple times in a loop.

Ah, I hear you cry, but how do I define what is allowable in my application? Easy, we have a browser based graphical environment (all you hardcore 'code-only' folks bear with me) that allows you to capture what we call a "Test Activity Diagram", this defines the pages you can go to from any page in the application. With the Test Activity diagram comes many huge benefits:

• Vitaq AI can automatically generate endless test paths through your application, doing all of the allowable, but unexpected things that a real human user would do.
• Vitaq AI also has the ability to generate data to be used during testing to add to the range of the testing
• Targets can be specified for sequences, individual spec files and for variables and Vitaq AI will ensure that the targets are met
• Changes in the application can be quickly captured in the test-activity diagram and the testing will immediately reflect the changes. This makes it easy to grow and change the testing as the application evolves.

Test Activity diagram for Vitaq to test Vitaq showing a user journey

All of this means that it is easy to create a large amount of test activity, that is auto-generated on the basis of "what is allowable" rather than "coding test, after test, after test". This means that Vitaq AI can help you find bugs that other approaches miss.

When do you know you have done enough testing?​

That age old test problem of when is enough, enough? When do you stop? Because Vitaq AI automatically generates tests, it is also keeping track of what has been tested. Sure, you can (should) use code coverage to keep a track of which code has been executed. But (in the simplest case) if the code is always executed on the happy paths and known "unhappy paths" (i.e. those whose errors are handled), then code coverage will be high but we may have missed some nasty errors that were never imagined.

What we need to do instead is track the functional coverage. Yes, with our new tool, you get a wonderful new metric, functional coverage. We measure how often have we have executed a specific piece of functionality?

Functional Coverage view for Vitaq testing Vitaq

For this reason, Vitaq AI measures functional coverage. The functional coverage is triggered by one of three events:

• A specified sequence of spec files are executed, this might be end-to-end or it might just be a sequence of spec files within the app that you want to see executed
• A specified spec file is executed, maybe this is a less critical page, but still you want to make sure that it is tested (e.g. an "About Us" page).
• A test data variable or a specific value of the variable is used in a spec file (e.g. you have a list of unallowable passwords and you want to make sure that all values in the list are tried and rejected). By setting targets for the functional coverage we can ensure that the key functionality has been sufficiently tested. Functional coverage gives a more user centric view of the testing than code coverage does, since it tells us that the specific UI events that we are interested in testing have been triggered.

Vitaq AI also records a run history for every test executed and the pass/fail status of the spec file, which allows us to quickly and easily work through the testing that has been performed and identify where there may have been failures

Why do you need a test activity diagram?​

The test activity diagram is where we specify the relationship between the pages, specifically which pages you can go to from each page in the application. This has various advantages over the standard coding of spec file, after spec file, after spec file approach:

• Most importantly, the test activity diagram requires us to define what is allowable or possible in the application. This forces a change in the way we think about testing, from "What set of steps do I need to run to test feature X" to "I'm currently on page Y, what can I do on this page and where am I allowed to go from here". This change means Vitaq will end up doing things that real users might do, but that you don't expect them to do, like logging out half way through the checkout process!
• The test activity diagram is a graphical representation of your application under test and this is a powerful way of planning, discussing and analysing the testing with the team. It will help you to drive the testing logic out into the open for everyone to see.
• The test activity diagram will evolve alongside your application development to allow rapid response to changes and additions.

Why should hardcore code only testers care?​

Earlier I asked the hardcore "code-only" folks to bear with me. Creating test software is a difficult task, there is a continuous tension between the preference of some users for a low-code, graphical environment and that of other users to be able to do everything in code, sometimes these extremes exist within the same team.

Vitaq AI can work at both extremes, in the case of the preference for low-code, the test activity diagram can be created and edited in a graphical environment. The spec file structure can be created automatically using one of our utility scripts and the contents of the spec file can be captured using the Chrome DevTools Recorder and pasted into the spec files. Whilst this does require a little coding knowledge, it has minimised the amount of coding required.

For those who prefer to do everything in code, Vitaq AI has a rich API and there is a huge amount of control available, which allows the coder to interact with and modify the test activity diagram from the spec files.

We have been developing this for a long time. We hope you like it and it helps you solve the difficult problem of thoroughly testing complex web and mobile apps.

We sure would be delighted to receive your feedback. Both good and bad. They do say that you learn a lot more when you fail! But for you app testers, failure is not an option! Or your company could end up on the front page of the Wall Street Journal for all the wrong reasons.

You can register for an account right now. Get started using Vitaq AI, for free, by following the Getting Started Guide, you'll be up and running with Vitaq AI in under 5 minutes.

If you have any questions or comments please contact us at support@vertizan.com.