AI Testing

Bringing AI Testing to Mainstream

Due to advancing consumer technology and the need to process and understand ever increasing amounts of data the buzz around Artificial Intelligence and Machine Learning has gathered steam in the last few years. In the IT space, AI and machine learning (ML) are already driving innovation and digital transformation. They are embedded in how we sift through large volumes of data and content, and how we enhance, organize, connect, and deliver content and information to the end users.

Artificial Intelligence is increasingly finding its way into the mainstream with diverse applications like self-driving cars, intelligent digital assistants, detecting and developing new treatments and drugs, reviewing legal contracts and documents and customer service automation to name a few.

AI Testing has thus become important

Testing AI applications has some unique challenges that must be addressed to ensure good quality. E.g. how do we test software:

  • When the decision logic is not clearly defined
  • When the results may be wrong and that’s OK sometimes
  • That learns and adapts based on interactions
  • To find the problems that will matter when the input domain is complex and massive
  • That may need to collaborate or compete with other AI software

A fundamental principle of AI is that it learns from the data. That means any inaccuracies in the data will be reflected in the results. And any additional layers of prediction or analysis have to be added separately.

AI Testing vs. Traditional Testing

Traditional application testing is deterministic, with a finite number of scenarios that can be defined in advance. On the other hand AI systems require a limitless approach to testing. While traditional software test design ideas can be applied to testing AI based applications, there are several additional considerations that must be taken in to account.

ImpactQA’s AI Testing Approach

Broadly speaking, the field of AI distinguishes between rule-based techniques and machine learning techniques.
As such, the entire universe of AI can be split into these two groups:

  • A computer system that achieves AI through a rule-based technique is called rule-based system
  • A computer system that achieves AI through machine learning technique is called a learning system

As a natural consequence, test approach for both the systems also varies. To test Rule based systems, our focus is to adopt Pyramid approach which comprises of three broad steps;

  • General Case Test: Enables to test the system according to the standardized test or expected flow
  • Domain Specific Test: Enables to test the system for a specific product or a specific group
  • Limit Test: Enables to test the limits of the system

To test learning based system, we feel there is no approach that works best than the “Train and Test”. Train consists of training or teaching the system to produce a set of outputs by learning patterns in the data through progressive learning algorithms.

Test involves comparing the outputs to key performance indicators and verifying the efficacy of the system.

At ImpactQA , we try to keep the whole testing process as dynamic as possible to make testing process more meaningful and productive.