Pieter Van Schalkwyk

CEO at XMPRO

In today’s fast-evolving technology landscape, Agentic AI offers exciting possibilities for transforming industrial operations. These AI systems—powered by large language models that can observe, reason, plan, and act with increasing autonomy—could significantly improve how companies operate. However, engineering executives need to approach this technology with both enthusiasm and careful consideration.

Drawing on my experience developing Multi-Agent Generative Systems (MAGS) for industrial applications, I’ve identified ten critical questions executives should ask before implementation. These questions will help you assess whether agentic AI is right for your operations and how to implement it successfully.

✳️ Download resource: Agentic AI Readiness Assessment Rubric for Engineering Executives

Ten Questions Engineering Executives Need To Ask When Considering Agentic AI for Operations

1. What specific operational problems will agentic AI solve that existing technologies cannot?

Start with your operational challenges, not with the technology itself. Many organizations make the mistake of implementing solutions looking for problems rather than addressing concrete operational needs.

The true value of agentic AI comes from solving specific, high-value operational problems that resist conventional approaches. In asset-intensive industries, the most promising applications typically involve:

• Complex coordination challenges across multiple systems or processes
• Knowledge-intensive decision making that currently relies on scarce expert judgment
• Situations with significant operational uncertainty or variability
• Operations that can be managed against mathematical objective functions, optimizing for multiple variables simultaneously

For example, predictive maintenance applications show clear value beyond conventional analytics. AI agents can monitor equipment health, predict failures, and optimize maintenance schedules across interdependent systems. Process optimization represents another high-potential area, where agents continuously adjust parameters to maximize quality, throughput, or energy efficiency.

When evaluating potential use cases, prioritize those with clear, measurable impact on key performance indicators. These might include Overall Equipment Effectiveness (OEE), unplanned downtime reduction, energy efficiency, or quality improvements. Success requires starting with specific operational challenges and mapping precisely how agentic AI addresses these challenges.

Operational Problem Definition (mark your level with X)

Previous Digital Engineer Articles:

• The Digital Transformation Paradox: Bridging the Maturity Gap in Industrial Manufacturing
• The Carroll Industrial AI Agent Framework: Evaluating True AI Agency

2. How will agentic AI integrate with our existing operational technology and information systems?

Asset-intensive industries operate on complex, often legacy technology ecosystems. These span operational technology (OT) like SCADA systems and programmable logic controllers, alongside information technology (IT) systems such as enterprise asset management and resource planning tools. Integration with these existing systems represents one of the most significant implementation challenges.

Data integration forms the foundation of effective AI implementation. Agents need reliable, timely access to operational data from diverse sources to make informed decisions. You must understand how the proposed agentic AI solution will access and process data from disparate systems. This becomes especially crucial when dealing with proprietary protocols or legacy systems with limited connectivity.

System interoperability directly impacts implementation feasibility and costs. Consider these questions:

• Will the AI agents communicate directly with control systems or require middleware?
• Does the solution support relevant industrial protocols and standards?
• What APIs or integration frameworks does it offer?

Network architecture and deployment models significantly affect performance and security. Will agents run in the cloud, at the edge, or in a hybrid architecture? How will this architecture address latency requirements and cybersecurity concerns, particularly in sensitive OT environments?

Successful integration demands a detailed assessment of your current technology landscape and a clearly defined strategy. Prioritize solutions that demonstrate compatibility with existing systems and minimize disruption to operational processes.

Safety, Security and Reliability Framework

Previous Digital Engineer Article: Bounded Autonomy: A Pragmatic Response to Concerns About Fully Autonomous AI Agents

4. Is our data foundation adequate to support effective AI agent operation?

The effectiveness of AI agents depends fundamentally on data quality, accessibility, and governance. Many industrial organizations underestimate the data preparation requirements for successful AI implementation. This often leads to disappointing results or extended timelines.

Data quality challenges in industrial settings are pervasive and frequently underestimated. You must assess the quality, consistency, and completeness of your operational data. Common issues include sensor drift, inconsistent naming conventions, missing values, and poor timestamp synchronization. Consider how these data quality issues will be addressed before or during implementation.

Data accessibility across systems often represents a significant hurdle. Ask whether critical operational data sources can be accessed via APIs or standard protocols. Identify what barriers—technical or organizational—might impede data access. Check if sufficient historical data exists for initial agent training and configuration.

Data governance practices determine long-term success. Examine your policies regarding data ownership, access controls, privacy requirements, and retention. Consider how these governance frameworks will extend to data processed and generated by AI agents.

Companies frequently discover their data foundation needs significant improvement before agentic AI can deliver value. Conduct a thorough data readiness assessment as part of your evaluation process. This will help identify gaps and develop a data preparation strategy if needed. In some cases, implementing foundational data management capabilities may need to precede agentic AI deployment.

Data Foundation Quality

5. What is the true return on investment, considering both direct and indirect benefits?

Justifying investment in agentic AI requires a comprehensive approach to calculating return on investment (ROI). This calculation should capture both direct operational savings and broader strategic benefits. Resist oversimplified ROI calculations that fail to account for the full value proposition.

Direct operational benefits typically include reduced downtime, lower maintenance costs, and improved resource utilization. Energy savings, quality improvements, and potentially reduced labor costs for routine tasks also count. These benefits can usually be quantified in traditional ROI models based on established metrics.

However, the most significant value often derives from strategic and indirect benefits. These benefits prove harder to quantify yet potentially more transformative. They include improved decision quality, enhanced operational agility, and knowledge capture from retiring workforce. Reduced safety incidents and operating effectively with fewer experienced personnel amid skill shortages also matter greatly.

Establish clear, measurable objectives for agentic AI implementations tied to specific KPIs. Determine baseline metrics and set improvement targets with realistic timelines. Develop methodologies for measuring progress against these established baselines.

Time-to-value represents another critical consideration. While some benefits may take months or years to fully materialize, well-designed implementations should deliver incremental value along the way. Identify what early wins you can expect and how these will build toward the broader value proposition.

A robust business case should incorporate both short-term operational improvements and long-term strategic benefits. Clear measurement methodologies will help track progress and demonstrate value to stakeholders.

ROI Measurement Framework

Previous Digital Engineer Article: The Value-First Approach to Industrial AI: Why MAGS Implementation Must Start with Business Outcomes

6. Will our subject matter experts be empowered to configure and guide these AI systems, or will we create new technical dependencies?

The most successful agentic AI implementations leverage both domain knowledge from subject matter experts (SMEs) and technical capabilities from IT professionals. Finding the right balance proves crucial for long-term success and operational alignment.

SMEs possess the contextual understanding and operational wisdom essential for effective AI agent configuration. They understand equipment behavior nuances, process dynamics, and operational priorities that determine successful outcomes. Yet traditional development approaches often create a “translation gap.” This occurs when expertise must be filtered through IT professionals who may lack domain context.

Examine how the proposed solution enables SME involvement. Does the platform offer low-code or no-code capabilities that allow domain experts to directly configure agent behaviors? Does it provide interfaces where SMEs can define rules, constraints, and objectives without coding? Or does it rely entirely on IT professionals for implementation and adaptation?

The governance model surrounding agent creation significantly impacts long-term success. Understand how the solution balances agility and control. Does it provide mechanisms for SME input while maintaining appropriate security guardrails? How are agent configurations tested, validated, and deployed?

Solutions that empower SMEs while maintaining appropriate IT governance typically deliver faster value and better operational alignment. However, they also require thoughtful governance frameworks to ensure security, reliability, and consistency across the organization.

Subject Matter Expert Empowerment

Previous Digital Engineer Article: Beyond Code: How Parametric Agents Transform Industrial AI Implementation

7. How will we develop the necessary skills and organizational capabilities for successful implementation and management?

Implementing agentic AI requires new skills and capabilities across multiple organizational functions. The skills gap represents one of the most significant barriers to successful adoption. Yet many organizations underestimate this challenge during implementation planning.

Subject matter experts need new competencies to work effectively with AI systems. Deep coding expertise may not be necessary with well-designed interfaces. However, SMEs still need foundational understanding of AI concepts, data interpretation, and effective prompting techniques. They must also learn to evaluate agent performance against operational objectives. Consider what training programs you’ll provide to develop these capabilities.

IT teams require expanded capabilities beyond traditional enterprise systems. Managing AI platforms, securing OT/IT integrations, and monitoring agent performance demand new skills. Troubleshooting complex, distributed systems also requires specialized knowledge. Assess whether your organization has these capabilities internally or needs external expertise. Develop a training or hiring strategy to address these needs.

Leaders and managers need new supervisory skills for this technology. Overseeing operations that combine human and AI workers requires different approaches. Managers must learn to interpret AI recommendations, apply appropriate human judgment, and manage this new collaborative environment effectively.

Assess organizational readiness across these dimensions and develop comprehensive capability-building strategies. The most successful organizations view agentic AI adoption as an organizational transformation initiative. They understand it’s not merely a technology implementation project but a change in how work gets done.

Skills and Organizational Capabilities

✳️ Download our comprehensive assessment tool: Agentic AI Readiness Assessment Rubric for Engineering Executives

8. What type of agentic processes should we implement, from basic automation to cognitive decision-making?

Not all agentic AI applications are created equal. Understanding the different types and complexity levels of agents will help you match the right technology to each process need. This matching significantly impacts implementation success.

At the simplest level, you’ll find workflow automation agents enhanced with LLMs. These agents follow predefined processes but use natural language processing to handle inputs and outputs. They excel at content creation, information retrieval, and straightforward process automation. Think RPA on steriods. Though valuable, these represent just the beginning of what’s possible.

The middle tier includes specialized agents with deeper reasoning capabilities. These agents handle more complex tasks requiring domain expertise and can work independently within specific boundaries. They might analyze equipment patterns, optimize individual processes, or manage defined operational areas with some autonomy. Think of a support agent on a helpdesk.

The most sophisticated implementation involves cognitive decision agents in multi-agent systems. These agents can observe complex environments, reflect on information from multiple sources, plan sophisticated responses, and take coordinated actions. They truly replicate human-like decision-making processes for complex operational challenges. Advanced implementations incorporate shared memory spaces and shared decision spaces, allowing agents to build collective intelligence by accessing each other’s observations, reflections, and decisions across operational domains. Think of a team of autonomous agents that optimize a supply chain process for a production line in real time.

Each complexity level brings different requirements. Simple workflow agents need less data and infrastructure but deliver more limited value. Cognitive decision agents offer transformative potential but require better data foundations, strong governance, and advanced integration capabilities.

The key is starting with agent types appropriate to your current readiness level, while choosing a framework that will support your organization through all stages of maturity. Select platforms and tools that can scale from simple workflow agents to sophisticated cognitive decision agents with shared memory spaces. This strategic approach prevents the need to change platforms as your maturity increases, avoiding costly migrations and maintaining continuity in your AI implementation journey. Begin with simpler agents for well-defined processes, building experience and infrastructure. As your capabilities mature, you can implement more sophisticated cognitive agents for complex, high-value processes where they’ll make the most difference, all within the same technology framework.

Previous Digital Engineer Articles:

• Brain-Inspired AI Agents: How New Research Validates XMPro’s Cognitive Approach to Industrial AI
• Beyond Content: Understanding the Role of Cognitive Agents in Industrial Operations

9. How will our implementation approach balance standardization with customization?

Effective agentic AI implementations require finding the right balance between standardization and customization. This balance affects scalability, maintenance requirements, and the ability to deliver value across different operational areas.

Standardization brings significant benefits to your AI implementation. Standard agent templates, interfaces, and integration patterns reduce development time and maintenance costs. They ensure consistent security controls and governance across implementations. Standardized approaches also facilitate knowledge sharing among teams and make scaling easier across multiple operational areas.

However, operational environments often have unique requirements that demand customization. Process variations, equipment differences, and site-specific constraints exist across most industrial organizations. Forcing identical solutions across these environments can reduce effectiveness and user acceptance. Customization enables agents to address specific operational needs and deliver maximum value.

This balance extends to model selection and deployment architecture. Organizations with specific security or regulatory requirements may need solutions that support ‘Bring Your Own LLM Model‘ capabilities and edge deployment options, allowing them to maintain control over their AI models while still benefiting from standardized agent frameworks.

The most successful implementations establish clear boundaries between standardized components and customizable elements. Core architecture, security controls, integration patterns, and governance frameworks should remain standardized. Process-specific rules, decision thresholds, KPIs, and interfaces can be customized to meet local needs without compromising the integrity of the overall system.

Your implementation strategy should define this balance explicitly. Consider creating a standardized agent architecture with parameterized configuration capabilities. This approach allows customization of agent behavior without requiring code changes. Also identify which elements must remain consistent for security and governance reasons versus where flexibility supports operational efficiency.

Organizations that manage this balance well achieve faster deployment, easier scaling, and better operational alignment. They maintain governance while still addressing the specific needs of different operational areas. Investing time in defining this balance early will significantly improve your implementation success.

Implementation Approach (Standardization vs. Customization)

✳️ Download our comprehensive assessment tool: Agentic AI Readiness Assessment Rubric for Engineering Executives

10. What vendor evaluation criteria will ensure we select the right technology partner?

Selecting the right technology partner for your agentic AI implementation fundamentally impacts your success. The field is evolving rapidly with many vendors making ambitious claims. A structured evaluation approach helps separate marketing hype from genuine capability.

Technology capabilities form your primary evaluation dimension. Assess how well the vendor’s solution aligns with your specific use cases and requirements. Examine their agent architecture, integration capabilities, and security controls. Evaluate whether they offer the appropriate agent types for your operational needs – from simple workflow agents to cognitive decision makers. Review their track record of successful implementations in environments similar to yours.

Vendor expertise represents another crucial criterion. The agentic AI landscape includes diverse providers with varying specializations and backgrounds. Focus your evaluation on thought leadership, domain expertise, and proven implementation success rather than brand, size, or “agent-washing” marketing. Look for vendors who demonstrate deep understanding of your industry challenges and whose product evolution shows maturity and intentional development. Companies that have consistently pioneered innovations in your specific domain often deliver superior value through their specialized knowledge and purpose-built solutions. The most suitable partners typically show evidence of successful implementations in environments similar to yours and maintain strong relationships with their customers.

Implementation and support capabilities directly affect your time-to-value. Evaluate the vendor’s implementation methodology, typical timeline, and resource requirements. Assess their professional services capabilities and approach to knowledge transfer. Consider their ongoing support model, including response times, update frequency, and continuous improvement processes. Strong support accelerates value realization and reduces internal resource requirements.

Strategic alignment forms the final evaluation dimension. Look for partners whose vision and roadmap align with your long-term strategic direction. Consider their approach to co-innovation and willingness to adapt to your specific needs. Evaluate whether they view your relationship as a true partnership rather than just a transaction. Strong alignment leads to better outcomes as both organizations evolve.

Create a structured scorecard incorporating these dimensions with weightings based on your specific priorities. Consider running proof-of-value pilots to validate critical capabilities as part or the vendor assessment process. These steps will help you select a partner who will support your success long-term.