How Enterprises Are Building Robust Infra Backbone For AI At Scale

After the proliferation of AI adoption, enterprise India has started to ask whether its infrastructure is robust enough to survive the shift. With the country’s enterprise AI market expected to cross $71 Bn by 2030, boardroom mandates around AI are accelerating, cloud costs are spiralling, data pipelines are buckling under the weight of unstructured inputs, and CTOs are discovering that pilots that worked in isolation can collapse the moment they are pushed into production at scale.

Therefore, enterprises across sectors are moving beyond productivity tools into core product development and delivery processes, including coding copilots, automated testing, and AI-driven IT service management. 

As a result, the conversation and its implementation are now shifting from experimentation to predictability, governance and cost control.

To unpack this shift, India’s leading tech voices joined Inc42 and Oracle for CTO Dialogues, featuring a roundtable on ‘Building The Infrastructure Backbone For AI At Scale’, followed by a fireside chat and a session exploring where enterprise AI strategies often fall short.

The roundtable was moderated by Sameer Dhanrajani, CEO of 3AI and AIQRATE, and featured the following speakers:

• Ankit Mehra, cofounder and CEO, GyanDhan
• Mashiyat Hussain, engineering lead, Backend, HYPD
• Nitish Gupta, SVP – technology, NimbusPost
• Sanjeev Singh, VP engineering, DeHaat
• Shubhanshu Chouhan, CTO, Pidge
• Vivek Gupta, senior director and head of technology cloud sales at Oracle India

Building A Robust Infra Backbone For AI 

The roundtable, moderated by Dhanrajani, addressed the harder question of how enterprises actually operationalise AI at scale, with six leaders spanning cloud, fintech, D2C, agritech, logistics and last-mile delivery.

Oracle India’s Vivek Gupta opened the conversation by arguing that enterprises chasing AI at scale are often solving for the wrong constraint. 

“AI at scale is not a GPU problem… It’s an engineering problem that needs to be solved,” said Gupta. 

According to Gupta, the bottleneck for many companies when implementing AI is not compute availability, but the ability to define a clear business outcome before they begin building. That focus on the problem itself resonated with GyanDhan’s Ankit Mehra, who pushed back against the instinct to reach for AI as a default fix. 

Having run workloads at his own company for testing purposes, Mehra said he had seen more problems traced to poorly framed questions than to actual scale limitations.

“Really solve for the problem that you’re trying to solve rather than just running with the hype part of it,” Mehra added.

But even when the problem is clearly defined, the data feeding these systems can present a different set of challenges. 

HYPD’s Hussain brought the conversation to data itself, describing the unpredictability of user-generated content on a creator platform, with comments arriving from varied regions and formats and no consistent pattern to model against.

“Data variety is a major problem for us. We cannot always predict what kind of data we will receive. For example, on our platform, creators receive comments from people across different regions and backgrounds, making it difficult to predict the nature and variety of the data,” said Hussain.

That unpredictability becomes even more complex when AI is layered on top of existing data challenges, and the VP of engineering at DeHaat, Singh, reiterated the same. According to Singh, the nature of the problem has shifted over the past decade, from a pure data-management challenge to a compounded one.

“Earlier it was only a data problem, shuffling of data, creating or churning our money. Now it’s a data-plus-AI problem. So, we have faced both,” said Singh.

For businesses operating at scale, however, identifying the problem and managing the data is only part of the equation. The infrastructure itself has to be tailored to the use case. 

The SVP of technology at NimbusPost, Gupta, grounded the discussion in logistics, where infrastructure decisions vary sharply by use case and margin. Getting the right setup, he said, comes only through costly trial and error.

“The right choice would come out of experimentation, and that experimentation would need a good amount of money… infrastructure is something wherein different industries would have different use cases,” he said. 

And that experimentation becomes even harder when AI is deployed in live operational environments, where a wrong decision can directly affect costs and performance. 

Chouhan, CTO of Pidge, concluded on a note of caution around automation in last-mile operations. Rider and delivery allocation, he said, still cannot be handed fully to opaque models.

“We cannot rely on an ambiguous, black-box algorithm like AI to always make the right decision. If we put AI in that place, our costs will go up, and we cannot guarantee performance,” said Chouhan.

Hidden Costs And Risks Of Scaling AI

The conversation then moved from the roundtable to a fireside chat, where the focus shifted from the broader challenges of building AI infrastructure to the risks enterprises face as they scale these deployments from pilots.

Dhanrajani, also moderating the fireside, opened the AI infrastructure conversation by pressing his two panellists, Ankit Gupta, CTO of Policybazaar for Business, and Nitin Kaushik, director of technology, Cloud Engineering, at Oracle, on where enterprise AI strategies tend to break down first.

Building AI capability, Gupta argued, is not only an infrastructure and use-case problem. According to him, it is also a trust challenge that many organisations underestimate. He noted that, without the right controls, prompts and AI-agent interactions can create pathways to expose sensitive enterprise information.

“Earlier we used to say, it is garbage in, garbage out, but today, the risk is that poorly governed inputs can lead to sensitive information getting out. That is why security, data governance, identity, and access controls must be designed into AI systems from the start,” Gupta said. 

Oracle’s Kaushik picked up the thread when the conversation turned to cost, an issue Dhanrajani termed as the “elephant in the room” for most enterprises deep into their AI rollouts. 

Token consumption, Kaushik said, is rising in ways many organisations are not prepared for. “I have met customers that are consuming trillions of tokens a day. We hear cases where organisations say their AI bills are beginning to approach – or even exceed – the value they associate with the business impact they are receiving.” 

Infra Gap Behind AI Deployment

Kaushik followed the fireside chat with a deeper look at why AI deployments break down after the pilot stage. According to Oracle’s director of cloud engineering, companies face three recurring challenges behind AI implementation. 

Cloud costs are often unpredictable, and data silos can create multiple versions of the same dataset – undermining the consistency and reliability of AI outputs. At the same time, cloud infrastructure has traditionally been built to prioritise availability over performance. 

According to him, Oracle rebuilt from the infrastructure layer up with dedicated compute and memory, off-box virtualisation to reduce network load on the CPU, and RDMA-based networking designed to deliver low-latency communication across GPU clusters.

Taken together, the roundtable and fireside chat pointed to a common thread: enterprise AI’s biggest constraints are not just computational, but organisational. Enterprises are grappling with engineering discipline, data governance, cost visibility and trust, rather than mere GPU access.

Tech leaders across the sessions converged on the same view that these factors, alongside access to compute, will ultimately determine whether an AI deployment survives the move from pilot to production.

The post How Enterprises Are Building Robust Infra Backbone For AI At Scale appeared first on Inc42 Media.