The impact of AI -- how to maximize value and minimize risk [Q&A]

Tech stacks and software landscapes are becoming ever more complex and are only made more so by the arrival of AI.

We spoke to David Gardiner, executive vice president and general manager at Tricentis, to discuss to discuss how AI is changing roles in development and testing as well as how companies can maximize the value of AI while mitigating the many risks.

BN: How can businesses guarantee their generative AI systems will output accurate/correct responses to build trust?

DG: It’s critical to understand that every application of generative AI will have a degree of variability. The very nature of generative AI comes with inherent unpredictability that cannot be removed. However, an unexpected response from AI is not an 'error' in itself -- it’s just a new type of output that workers must come to understand. It’s only when workers don’t understand these outputs, and how to handle them, that errors can be created in a workflow. If a business struggles to accept these facts, then it may struggle to justify the use of generative AI. It’s imperative that a business is able to relay this reality to its entire board and workforce to avoid misinformed expectations.

That being said, uncertainty can be managed -- and this is how businesses should go about building trust in AI. The first step a business should take in pursuing an application that leverages generative AI is determining the acceptable degree of variation for any given use case. Similarly to what businesses regularly consider in a standard business impact analysis or use case valuation analysis, leaders should question the value of the product and the impact of ‘failure’ (in this context, if the AI process produces an expected result). If there’s a very slim window for acceptable degree of variability, businesses must weigh whether it’s advisable to use generative AI in that case at all.

BN: Certainty cannot be 100 percent guaranteed when it comes to generative AI. How then can businesses work toward making their generative AI systems produce as few unpredictable outcomes as possible?

DG: The only way to truly measure the quality of an AI system is to test it with a large number of examples and measure the system’s range of responses. Subsequently, teams should perform pattern analysis of the responses to help identify common sources of unpredictability and target them for refinement, which is more effective than addressing isolated issues. It’s important to note that AI models evolve and drift over time, so testing the quality of an AI system should be a regular practice.

Once a business has deployed and activated generative AI systems within an application, it’s important to have checks and balances that mitigate paths of unpredictability that exceed the established acceptable degree of variation. As a core method of control, software development teams should embed multiple layers of validation and systems that can quickly identify when responses are taking a path too far from the expected one -- enabling the course correction of AI outputs in response.

Additionally, continuous monitoring and human oversight in critical scenarios can help catch and mitigate unreliable outputs before they impact users. For example, when AI outputs are used in high-stakes contexts like regulatory reporting or financial summaries, businesses should ensure they have human reviewers on their team who can intervene as needed to ensure accuracy.

BN: The CrowdStrike outage served as a stark reminder of the increasingly complex and interconnected nature of digital landscapes, and how one small mistake can result in massive damage. How can businesses best protect themselves from such an outage happening to them?

DG: If businesses take one thing away from the CrowdStrike outage, it should be this: your business must have efficient testing strategies in place.

The first step in determining the best place to aim testing efforts is taking the time to understand users’ needs and their approach to a product across several environments. I always advise software development and testing teams to create user profiles that simulate realistic scenarios, ensuring that every possible use case and environment is accounted for. Once user actions and strategies are aligned, testing efforts can be arranged to focus on core impact zones to identify problems early.

In addition to product-specific focus areas for testing, two essential matters that every software development team must stress test are a product’s ability to handle high-volume traffic and demand. Performance testing is a central aspect of any prepared business’ broader testing strategy, and it must be embedded throughout the software development lifecycle (SDLC) to identify bottlenecks and ensure systems can withstand real-world usage.

However, focusing on your own systems is not enough. Delta Air Lines, for example, learned this the hard way, with impacts from the CrowdStrike outage causing them an estimated ~$500 million lost. It’s critical that businesses ensure all third-party integrations are reliable and strong. To best determine this, I advise companies to identify and align with all third-party providers included in essential company applications and review their policies and procedures for handling defective code that came from them. Holistically analyzing infrastructure and integration in detail reduces risks from external parties, contributing to a more secure and streamlined ecosystem for your business.

BN: Many argue that the government needs to regulate AI to keep society safe, but could this stifle innovation?

DG: The regulation of AI is at a crucial juncture. With AI’s rapid advancement, potential harms -- like misinformation, intellectual property concerns, and black box decision-making -- are becoming more visible, especially in customer-facing applications. Contrary to the fear that regulation will stifle innovation, I believe that well-designed regulations actually create a safe, level playing field.

Take how trademark laws, for example, establish trust in brand authenticity. Similarly, AI regulations could provide legitimacy, allowing companies -- especially startups -- to innovate without the looming risk of unforeseen liabilities. The goal should be to establish clear guidelines on the responsible use of AI, fostering a landscape where innovation can thrive within safe boundaries.

The key to effective AI regulation is transparency and informed consent. Users should be made aware when they are interacting with AI and understand that it may provide inaccurate or ‘hallucinated’ information. Future regulations might mandate disclosures, enabling users to make informed choices when engaging with AI. Additionally, there is potential for a ‘right to be forgotten by AI,’ similar to data protections under GDPR, allowing individuals to restrict the use of their data in AI training.

Overall, balanced regulation focusing on transparency and safe usage -- rather than overly restrictive limits on the technology itself -- offers the best path to harness AI’s benefits responsibly while safeguarding society.

BN: With generative AI’s ability to develop quality code at a fast pace, do you expect generative AI to alter the viability of software programming as a profession?

DG: While generative AI has advanced to the point where it can create code at speed, this won’t diminish the need for skilled software developers. Instead, it will change the nature of the work, pushing developers and those testing the code to take on more strategic roles focused on quality and oversight.

When calculators gained widespread adoption in the early 1970s, they redefined certain tasks, but the need for human expertise wasn’t eliminated; it was just reallocated. Generative AI is poised to do the same: enhance what developers and testers can achieve, rather than replace them outright.

Developers will be expected to manage and validate the quality of code generated by AI, especially as the speed of output increases. This shifts the emphasis from software development to rapid quality assurance (QA) processes that ensure new accelerated coding workflows still meet high standards. In practice, AI won't necessarily reduce the need for developers; AI will amplify their need, as quality and reliability become even more critical and strategic in nature. And the same can be said for developers’ counterparts, testers, whose role is also shifting to adapt to the growing complexity and opportunity generative AI poses. By shifting focus to streamlining processes for faster feedback, reduced costs, and improved efficiency across the SDLC, testers are opening the door for more strategic quality and oversight.

Image Credit: Unsplash