Beyond traditional metrics: How to redefine AI success

In the past year, AI made great strides in moving past hype, so much so, that it’s hard to recall the early days of AI when the initial value of the technology was questioned. Today, as AI initiatives start to deliver widespread returns, enterprise CIOs are faced with competing forces of driving down core IT costs, while investing heavily in AI to drive business transformation.

A recent study of 2,400 IT decision makers, commissioned by IBM and developed with Lopez Research, underscores this optimism. The findings reveal that the vast majority of companies are making headway on their AI strategies, with nearly half already reporting positive financial returns from their deployments. The cost benefits have been especially pronounced for organizations using open-source AI tools -- 51 percent of surveyed companies harnessing open-source solutions reported seeing positive ROI, compared to just 41 percent of those that are not.

Nearly two-thirds of executives say they are increasing their AI spend this year, signaling growing confidence in the technology’s long-term value. Enterprise AI’s promise is beginning to bear undeniable fruit, and it’s critical for organizations to approach their AI investments more strategically to demonstrate impact on the most important priorities.

Measuring Change Beyond Traditional ROI Metrics

Incorporating AI for business transformation has been an imperative for organizations in nearly every industry for the past several years, yet many have struggled to justify AI investments using conventional ROI calculations. In fact, a Gartner study from late 2023 found the most common obstacle to greater AI adoption was respondents’ challenges in estimating and demonstrating the value of their solutions.

Today, this difficulty is decreasing as organizations establish better performance benchmarks. Forward-thinking organizations have opted to measure the success of AI investments in less conventional ROI metrics, including speed, time and unit costs. When asked about their primary metrics for calculating AI ROI, IT decision makers at these companies prioritized faster software development (25 percent), more rapid innovation (23 percent) and productivity time savings (22 percent) -- with hard-dollar returns a distant fourth at just 15 percent. Empowered with a better understanding of the benefits of their AI projects, fewer than one in 10 of these IT decision makers reported making no progress in their AI projects in 2024.

This shift in perspective enables organizations to better articulate the value of their AI initiatives. For example, tracking metrics like cycle time reduction and unit cost impact can provide IT executives with a clearer picture of AI’s influence on business operations for both efficiency and effectiveness. By comparing the pre- and post-AI implementation results, executives are better positioned to demonstrate the tangible benefits of AI, not only validating the investment but also providing a clear framework for continuous improvement.

While technological and financial metrics may have dominated pre-AI era discussions, the effect of AI on job roles and task execution has profound implications for the human aspect of work.  Organizations that will thrive through this dramatic change will understand and consider the overall human impact, such as time saved, engagement, and their overall happiness with their role, as a critical measure of success. AI implementation fundamentally changes how people work, and measuring this impact is essential for long-term success. Companies that actively monitor indicators such as employee satisfaction, skills development and workforce adaptability are also better positioned to realize the full potential of their AI investments.

The Open-Source Advantage

An explanation for the optimistic AI forecast may lie in the burgeoning use of open-source AI solutions. Initially, open-source models had been popular due to their lack of per-transaction API costs, helping organizations accelerate their AI ambitions cost-effectively. Today, the advantages extend beyond cost savings. Open-source models, like Granite and DeepSeek-R1, have improved significantly in performance over the past year, with some rivaling or even exceeding the capabilities of closed proprietary models, especially for enterprise-focused use cases. This improvement, combined with greater transparency and control over data, has made open-source AI increasingly attractive for even the largest enterprises.

The preference for open-source AI is already strong. More than 80 percent of surveyed IT decision-makers reported at least a quarter of their company’s AI solutions or platforms are currently based on open source, with the largest companies actually the most likely to adopt open-source.

Organizations are discovering that a more thoughtful approach to AI implementation -- one that leverages purpose-built, trusted smaller models requiring less computational power -- can deliver equivalent or better results to large, resource-intensive models. Smaller open-source models offer cost-effective solutions while also democratizing AI access, fostering collaboration, and enabling customization for specific use cases.

For companies seeking to optimize their AI investments, this approach represents a significant opportunity. When asked to cite the strategic changes they planned to make in 2025 to optimize their AI investments, nearly half of those surveyed listed the use of open source as a top priority, with 40 percent of those not yet using open source planning to begin doing so.

The Returns Ahead

The current enterprise AI landscape is characterized by growing maturity and pragmatism. As organizations move beyond the initial experimentation phase, they are implementing solutions that deliver multiple dimensions – financial, operational, and human.  The steady long-term vision of AI investment appears poised to pay off for an even greater share of businesses in the year ahead.

Image credit: denismagilov/depositphotos.com

Matt Lyteson is CIO VP Technology Platforms Transformation, IBM.