By Vincent Yates, Chief Data Scientist and Partner, Credera
In today’s business world, artificial intelligence is a major force for change. Yet, amidst the buzz and promise, a critical concept often remains shrouded in ambiguity—the AI maturity model. But what exactly is this model, and why does it command such significance for organizations eager to harness AI effectively?
Originating in government sectors, maturity models were initially devised to categorize vendors for substantial projects. They proved so effective that their use expanded across industries. At their core, these models aim to transform the “unknown unknowns”—challenges and variables we aren’t even aware we don’t know—into “known unknowns,” things we recognize we don’t understand.
Yet, herein lies a paradox. While maturity models offer a roadmap for progression, they often fall short of enabling the actual journey. Their static and overly generalized nature fails to address the nuanced and dynamic challenges of integrating AI into an organization’s fabric. This realization has prompted leaders, including myself as the chairman of the AI Global Council, to advocate for a reevaluation of the traditional AI maturity model.
The essence of our critique is not to undermine the utility of maturity models but to highlight their limitations. Our call to action focuses on shifting from a static, process-oriented evaluation to one that emphasizes outcomes and performance. The real measure of AI maturity should not be how well an organization aligns with a predefined model but how effectively it leverages AI to create enterprise value. This paradigm shift requires a keen understanding of the leading indicators that contribute to organizational value, grounded in strategies based on empirical data and performance outcomes.
How to assess your AI maturity model
To effectively assess where you stand on the AI maturity scale and plot your course forward, consider adopting the METAL framework—a blend of Method, Experimentation, Trust, Agency, and Latency.
1. Method: Define your AI strategy
First and foremost, establish a clear methodology for integrating AI within your organization. This involves identifying which teams and processes could benefit most from AI and setting clear objectives for its implementation. Consider where AI fits and where it doesn’t, ensuring that your approach is strategic and aligned with your overall business goals.
2. Experimentation: Embrace A/B testing
AI is inherently experimental. Unlike traditional software development, where success criteria are predefined, AI often explores unknown territories. Embrace A/B testing to compare different AI models and approaches directly. This real-world feedback loop is crucial for understanding how AI can best serve your needs. Zillow’s foray into AI with its Zillow Offers program serves as a stark example of the experimental nature of AI in business intelligence. Launched to streamline home selling by leveraging AI for price forecasting, the initiative faltered due to the unpredictable real estate market, leading to considerable losses and the program’s cessation in 2021, with about 2,000 layoffs. This outcome highlights the risks of depending on historical data for AI predictions. Oren Etzioni, former AI2 CEO, pointed out that “adversarial machine learning,” where models can’t adjust to new market conditions, played a role in its failure, emphasizing the need for ongoing testing and adaptation in AI endeavors.
3. Trust: Establish diagnostic measures
Trust in AI comes from transparency and understanding. Develop diagnostics to evaluate whether your AI models are performing as expected and generating trustworthy results. This involves not just technical metrics but also considering the impact on end-users and business outcomes. Trust is built on clear evidence that AI is enhancing decision-making and operational efficiency.
4. Agency: Leverage AI to make decisions
Agency is about acting on the insights and efficiencies AI offers. It’s the practical application and realization of AI’s benefits. Whether optimizing hospital bed allocations or improving customer service, demonstrate how AI directly contributes to better decision-making and operational improvements in your organization.
5. Latency: Adapt quickly to change
The AI landscape and customer preferences evolve rapidly. Your AI initiatives should be agile enough to adapt to these changes swiftly. “Latency” in this context means the ability to pivot and align AI strategies with shifting targets and market dynamics.
Understanding AI maturity requires looking beyond standardized models to consider the unique needs, capabilities, and goals of our organizations. For instance, the requirements for OpenAI, a leader in foundational AI models, differ vastly from those of a CTO managing technological strategies in a corporate setting. While foundational models demand a high caliber of research and data access, a CTO’s focus might lean more toward practical applications and vendor assessments. This divergence highlights why a one-size-fits-all maturity model falls short; it fails to accommodate the diverse landscapes in which AI is applied.
The true measure of AI maturity lies not in achieving a universal standard but in how effectively AI is integrated to solve real-world problems and create value in your specific context. This approach calls for a departure from traditional models and towards a more flexible, outcome-oriented perspective. By focusing on what AI success looks like for you and building your strategy from there, you can unlock the full potential of AI in a way that is meaningful and impactful for your organization.
About the Author
Vincent Yates is chief data scientist and partner at Credera, where he runs the data and analytics division. He spends his time transforming Fortune 500 organizations through data by leveraging his deep experience at some of the top data companies in the world. He got his start in data science after a Ph.D. program in statistics at the University of California, Berkeley.
