(1) Background: What is Cognitive Testing All About?
Cognitive testing is a new class of testing and, as the name suggests, it leverages machine learning, artificial intelligence, natural language processing, and other cognitive computing techniques. I recently participated in a workshop in an effort to identify testing cases that could benefit from applying IBM Watson’s cognitive intelligence to testing
In this blog, I will discuss the evolution of cognitive technology as well as the available platforms for hosting this technology, such as IBM Watson. I will also cover different methodologies involved in cognitive testing, the benefits of this testing, supported technologies, sector-specific use cases, and the costs and challenges of cognitive testing implementation.
Today, computers are getting smarter with both machine learning and the use of products or solutions developed through technologies like data analytics, business intelligence, and big data. Computers can mimic the human brain and draw inferences from existing data and patterns with the help of underlying software. The software is also able to insert this information back into its knowledge base for future inferences in the form a self-learning feedback loop. Cognitive intelligence renders a new class of problems computable. It addresses complex situations that are characterized by ambiguity and uncertainty that we can relate to humans.
Natural language and artificial intelligence studies have existed for nearly five decades. Now, these studies combined with predictive analytics have become the building blocks of cognitive analytics. Cognitive analytics systems are being utilized for computing with the goal of creating accurate models of how the human brain senses, reasons, and responds to stimuli.
(2) The Evolution of Cognitive Technology and Available Platforms
The evolution of cognitive computing is evidenced by the move from descriptive to predictive analytics, and the next step requires prescriptive analytics to make decisions based on this analyzed information. This would create opportunities for self-service testing platforms with self-healing capabilities. The advances in natural language processing would lead to more advanced and scriptless testing solutions. Smart speech-to-text processing would enable the business users to narrate requirements in natural language and be able to convert them into automated test suites using scriptless testing techniques. Neuromorphic systems and neuro-enabled firmware can be used for smart environment handling and intelligent test data management.
Some of the platforms available that offer cognitive intelligence include:
- OpenAI, a non-profit, artificial intelligence research platform to benefit humanity, unconstrained by a need to generate financial return.
- Google DeepMind, which specializes in building algorithms that are capable of learning independently from raw experience or data, and can perform a wide variety of tasks.
- IBM Watson, a technology platform using natural language processing and machine learning to reveal insights from large amounts of unstructured data.
(3) Methodologies Involved in Cognitive Testing
Cognitive testing leverages machine learning, artificial intelligence, natural language processing, speech-to-text, image recognition, and similar cognitive computing techniques. Cognitive testing uses heuristics to predict defects, to measure system performance, and to optimize the test coverage based on assessed risks.
Products such as IBM Watson, Google DeepMind, and Microsoft Oxford provide platforms for cognitive computing. The same can be used for solving test optimization problems. Some examples of how cognitive intelligence can be leveraged in testing include:
- Test prioritization
- Automated regression test bed selection and prioritization
- Failure prediction using log analyzers
- Test coverage optimization
- To compare product module patterns in production vis-à-vis test coverage
- Bridge the gap in test coverage
- Determine how much testing is enough
- Assess release readiness and provide a decision on halting regression
- Provide a risk index of product released to production
- Self-correcting and updating test suites
- Retire the obsolete test cases and code fixes in the pipeline
- QA Dashboard
- Reporting of metrics such as cost of defect and resource utilization
(4) Advantages of Cognitive Testing Compared to Current Methods
Capgemini has the opportunity to leverage IBM Watson’s cognitive intelligence in various application areas. IBM Watson considers the four key advantages as “Adaptive,” “Interactive,” “Iterative and Stateful,” and “Contextual.”
Adaptive: Cognitive testing is adaptive because it enables the system to learn as information changes and as both goals and requirements evolve. For instance, the system can suggest a number of test iterations and decisions about when to stop testing based on code quality and requirement changes,
- Interactive: The system interacts seamlessly with users so that they can easily define their test requirements. They may also interact with other processors, IoT devices, cloud services, and people.
- Iterative and Stateful: This feature helps the system “remember” previous interactions and returns suitable information for the specific application at a specific point in time. The testing sequence can be defined based on questions asked.
- Contextual: The software is able to understand the context. Sample contexts include meaning, time, location, appropriate domain, regulations, user profile, process, task, and goal.
(5) Sector Specific Use Cases
Cognitive testing applies across industry sectors. However, adopting this system relates to the use of agile or DevOps software lifecycle. Highly competitive and new age industries are more likely to adopt cognitive solutions whereas the more stable and highly regulated sectors are likely to prefer conventional testing solutions.
Telecom and financial services are key users of cognitive intelligence. Financial services use cognitive systems in domains such as Know Your Customer (KYC), credit ratings and loan decisions, and wealth management for portfolio optimization. Healthcare is another service area, but regulations may prevent machines from making decisions in regards to patient health.
(6) Technologies Supported
Cognitive testing applies to all types of applications and it is not just limited to web interfaces. The key principle behind cognitive testing is the ability of the system to access and analyze large volumes of data (both structured as well as unstructured). The system also uses machine learning to extract context-sensitive intelligence from the analyzed data and use it to predict future behavior. All kinds of applications—cloud based, IoT, mainframe, database, and mobile—can benefit from cognitive testing techniques.
Take, for instance, the following testing scenario: IBM Watson can crawl through defect data, various requirements and design documents, development code base, and production incidents to identify the likely points of failure. The intelligence gathered can then be used to tailor an appropriate test execution plan. Platforms such as Watson, whose fame began when it cracked the show “Jeopardy!” as an intelligent machine) have come a long way.
IBM Watson and other tools that form the basis of cognitive intelligence solutions can work on the code base as well. The tool can be configured to browse all leading technologies. There is a lot of intelligence to be derived by analyzing the unstructured information embedded in the code comments. In addition, applying cognitive computing techniques to measure logical accuracy and to derive boundary conditions for test data preparation can work to assess the quality of the code base. The relative age of the underlying code and the frequency of check-ins is also found to have a very strong relationship with defect density.
(7) Costs Involved
The methodology is expensive, but one does not need all the modules of a complex system such as Watson. Rather, one needs an analysis of the modules based on specific use cases, particularly in case of perpetual enterprise licenses. To solve short-term problems, one can use the SaaS model at costs as low as US$30 to US$80 (€28 to €75) per-person, per-month. Enterprise use could come at costs ranging from US$1 – 5 million (€0.9 to 4.7 million), including hardware costs.
(8) Challenges Associated in Implementing Cognitive Technologies
- The system requires training in the domain and application landscape, and this demands a significant time investment during initial stages. However, there are great economies of scale once the system is trained and the costs come down sharply.
- The cognitive computing system derives its intelligence from the requirements that are fed into the system. The challenges with incorrect requirements (lack of alignment with customer needs and ambiguous, inaccurate, invalid, or impossible requirements) could lead to incorrect test results.
- Complex architecture, combined with frequent technology and configuration changes, pose technical challenges in implementing this technology.
Given the pressures of the industry, it’s obvious that the adoption of cognitive technologies is driven by the potential for increased revenue, lower costs, faster time-to-market, improved competitive positioning, and enhanced customer experience. Specifically in relation to testing, the benefits include fewer defects, improved test efficiencies, and a better collaboration between the development and testing teams.
Main Author: Renu Rajani, Vice president, Capgemini Technology I P Ltd, email@example.com
Contributing Author: Manish Goyal, Solutioning Program Manager, Capgemini Technology I P Ltd, firstname.lastname@example.org