Enterprise AI Is Moving Beyond the Experiment Stage
Artificial intelligence adoption inside Indian enterprises is entering a more demanding phase.
After an initial wave dominated by experiments, copilots and proof-of-concept deployments, companies are increasingly integrating AI into core products and operational workflows. That shift is forcing technology teams to confront a more difficult question: can the infrastructure supporting these AI systems remain reliable, secure and affordable when usage scales?
At an Inc42 and Oracle CTO Dialogues discussion on “Building The Infrastructure Backbone For AI At Scale,” technology executives from fintech, agritech, logistics, D2C and cloud infrastructure argued that successful deployment requires much more than access to powerful GPUs. The Inc42 article was published through its BrandLabs division and identifies the session as an Inc42-Oracle initiative.
The discussion featured executives from GyanDhan, HYPD, NimbusPost, DeHaat, Pidge and Oracle India and was moderated by Sameer Dhanrajani, CEO of 3AI and AIQRATE.
‘AI at Scale Is Not a GPU Problem’
One of the clearest arguments from the discussion came from Vivek Gupta, senior director and head of technology cloud sales at Oracle India.
“AI at scale is not a GPU problem… It’s an engineering problem that needs to be solved.”
Gupta's point was that organisations can focus too heavily on acquiring compute capacity while overlooking a more fundamental requirement: clearly defining the business outcome the AI system is expected to deliver.
That distinction becomes increasingly important as AI moves from a controlled experiment to a production system serving employees or customers.
A pilot may demonstrate that a model can perform a particular task. A production deployment must additionally address reliability, latency, security, data access, monitoring and cost.
GyanDhan CEO: Solve the Problem, Not the AI Hype
Ankit Mehra, cofounder and CEO of GyanDhan, raised a related concern.
Based on his company's experimentation with AI workloads, Mehra said poorly framed problems could be a bigger obstacle than infrastructure scale itself.
“Really solve for the problem that you’re trying to solve rather than just running with the hype part of it.”
The implication is important for enterprise AI strategy: introducing AI is not automatically valuable simply because the technology is available.
Companies first need to establish what process they want to improve, what measurable outcome they expect and whether AI is actually the appropriate solution.
Data Becomes Harder to Control at Scale
Even a well-defined AI use case can struggle when the underlying data is unpredictable.
Mashiyat Hussain, engineering lead, Backend, at HYPD, highlighted the problem using a creator platform as an example. User-generated comments can arrive from different locations, backgrounds and formats, making the nature of incoming data difficult to anticipate.
“Data variety is a major problem for us.”
Hussain said the diversity of information received by the platform makes predicting and modelling the data challenging.
For businesses deploying generative AI, this matters because the quality and consistency of an AI system's output can depend heavily on the information supplied to it.
The problem is therefore not merely storing more data. Enterprises need mechanisms to clean, classify, govern and retrieve information in ways suitable for AI workloads.
DeHaat Sees a ‘Data-Plus-AI’ Challenge
Sanjeev Singh, VP engineering at DeHaat, described the evolution in similarly practical terms.
“Earlier it was only a data problem... Now it’s a data-plus-AI problem.”
His observation captures a broader change taking place in enterprise technology.
Traditional data-engineering challenges—such as moving, organising and processing information—have not disappeared. Instead, AI introduces another layer involving model inputs, outputs, context, governance and infrastructure requirements.
That means companies adopting AI may have to modernise their data foundations at the same time that they build the AI applications sitting on top of them.
There Is No Single AI Infrastructure Blueprint
Infrastructure choices can also differ substantially from one industry to another.
Nitish Gupta, SVP – technology at NimbusPost, said finding the appropriate infrastructure often requires experimentation, which itself can be expensive.
According to Gupta, different industries and applications have different requirements, making it difficult to prescribe one universal AI architecture.
A customer-service chatbot, fraud-detection platform and logistics allocation system, for example, can have very different requirements for latency, accuracy, data freshness and cost.
This is one reason infrastructure decisions increasingly become part of the economics of the AI product itself rather than a purely technical consideration.
Why Pidge Is Cautious About Fully Automated AI Decisions
The risks become more significant when an AI system directly influences real-world operations.
Shubhanshu Chouhan, CTO of Pidge, pointed to last-mile logistics, where decisions such as rider and delivery allocation have direct consequences for cost and performance.
“We cannot rely on an ambiguous, black-box algorithm like AI to always make the right decision.”
Chouhan warned that relying entirely on AI for such decisions could increase costs while making performance difficult to guarantee.
This illustrates an important difference between assistive and autonomous enterprise AI.
An AI system that drafts an internal document may tolerate occasional errors. A system automatically assigning deliveries, approving transactions or making other operational decisions can require substantially stronger safeguards, monitoring and fallback mechanisms.
Security and Governance Need to Start Before Deployment
Infrastructure is only one layer of the challenge.
During a subsequent fireside discussion, Ankit Gupta, CTO of Policybazaar for Business, argued that enterprise AI also creates a significant trust and information-security problem.
Poorly controlled prompts or interactions with AI agents can potentially expose sensitive enterprise information.
Gupta said security, identity, access controls and data governance therefore need to be incorporated into AI systems from the beginning rather than added after deployment.
That becomes particularly important as AI agents gain permission to retrieve corporate information or interact with other enterprise systems.
The more access an AI system receives, the more consequential its access-control architecture becomes.
AI Costs Can Rise Faster Than Expected
Another challenge is economics.
Nitin Kaushik, director of technology, Cloud Engineering at Oracle, said some customers are already consuming trillions of tokens per day.
He also described cases in which organisations felt their AI bills were beginning to approach—or even exceed—the business value they associated with their deployments.
That creates a fundamentally different infrastructure question.
The goal is no longer simply to ask whether an AI model can complete a task. Enterprises increasingly need to understand how much every inference or completed action costs at production scale.
This is helping make inference optimisation, model selection, caching, batching and AI-focused FinOps more important parts of enterprise architecture. Inc42's broader 2026 coverage similarly describes cost predictability and infrastructure observability as increasingly important as AI becomes critical business infrastructure.
Why AI Pilots Can Break When They Reach Production
According to Kaushik, three recurring infrastructure issues can become particularly visible when AI projects move beyond pilots: unpredictable cloud costs, fragmented data and infrastructure that is not optimised for AI performance.
Data silos can be particularly damaging because different systems may contain competing versions of the same information. If those inconsistencies are passed to an AI application, the reliability of its answers can suffer.
At the infrastructure layer, AI workloads can also have different networking and computing characteristics from conventional enterprise software.
Kaushik said Oracle's approach includes dedicated compute and memory, off-box virtualisation intended to reduce CPU networking overhead and RDMA-based networking designed for low-latency communication across GPU clusters. This is Oracle's description of its own infrastructure architecture, rather than an independent benchmark of comparative performance.
The Bigger Shift: From GPU Capacity to AI Engineering
The discussions reveal a broader change in how enterprises are thinking about AI.
During the early generative-AI boom, much of the infrastructure conversation revolved around access to GPUs and increasingly powerful models.
In 2026, the conversation is broadening.
Inc42's recent enterprise-AI coverage shows companies focusing more heavily on uptime, latency, governance, inference economics, data readiness and smaller or domain-specific models as AI applications move into production.
At the same time, India's underlying compute capacity is expanding rapidly. For example, Yotta Data Services told Inc42 it expected to scale its AI cloud beyond 40,000 NVIDIA Blackwell GPUs within four months and to roughly 85,000 GPUs by the end of the financial year, while expecting cumulative AI-infrastructure investment to exceed $7 billion by the end of FY27. Those figures are company projections reported by Inc42 rather than independently verified outcomes.
The availability of more compute, however, does not eliminate the engineering problems identified by enterprise technology leaders.
What Enterprises Need Before Scaling AI
The emerging enterprise playbook can be distilled into several connected requirements: start with a measurable business problem rather than an AI-first mandate; prepare and govern the underlying data; design identity, access and security controls before deployment; understand inference and token costs at realistic production volumes; choose infrastructure according to the use case rather than adopting a universal architecture; and retain monitoring or human intervention where AI decisions can materially affect operations.
The central lesson is that AI scale is becoming a systems-engineering challenge rather than simply a model-selection exercise.
GPUs remain essential infrastructure, but compute alone cannot compensate for fragmented data, poorly defined use cases, uncontrolled spending or weak security.
For enterprises attempting to turn successful AI experiments into dependable products, the infrastructure surrounding the model may ultimately matter as much as the model itself.
Source
Primary Source: Inc42 BrandLabs, How Enterprises Are Building Robust Infra Backbone For AI At Scale, published September 25, 2026. The article is based on an Inc42 and Oracle CTO Dialogues session.
Additional Context: Inc42's 2026 reporting on enterprise AI architecture, model strategy and India's expanding AI compute infrastructure.
