Will inference change the game in AI apps and infrastructure?

Once AI models are trained on corporate data, they need to get put to work making decisions. That’s called inferencing, and it’s an essential shift if the AI economy is going to pay off. I’ve talked to several innovators in the space lately about their progress in the inferencing game, and where it fits into their growth prospects.

If you’d like to join me – and peers – for deeper conversations on innovation and leadership, get on this list for Fortt Knox Executive Communities, launching soon: mba.fortt.com.

d-Matrix

I spoke with d-Matrix CEO Sid Sheth recently about how the market’s attention is shifting toward what his team’s been working on for years:

We set out in 2019 to solve the inference compute problem because we had a thesis that you need something fundamentally different from what’s available in the market to do inference. It’s very different from training. And we set out to build the world’s most efficient inference computing platform, to make it commercially viable and sustainable.

And a few years later, ChatGPT happens, and here we are. Everyone is talking about inference. Back then I used to go to investors and they wouldn’t even know what the word inference meant, and I would have to explain it to them. But now it’s everywhere. Everyone’s talking about it, and it seems like there is a general acknowledgment that it is going to be one of the largest and toughest problems to go solve if you want AI to be truly successful and widespread.

I think it’s absolutely worth it. These smaller companies, some of them are going to go on to be bigger companies down the road. Some of them are going to get bought up by bigger companies that didn’t start early enough in figuring out some of the differences in the inference game, not only versus the training game, but versus how software was running before. We talked about this with DevRev and Vinod Khosla – we shared a little bit of that last week.

Just the entire software ecosystem is being reimagined and realigned based on which AI infrastructure piece is in control, which AI software control tower is in control. But then I think beyond that, at the level of inference, there are going to be, based on different business types, based on different workflows, different needs for resources. And there are going to be specialists within those categories that are able to super-serve that audience.

And as that shift toward inferencing in AI continues, there’s a chance that customers in the marketplace will migrate toward those specialists, and then they’ll have hand – maybe they end up consolidating, maybe they end up growing huge profits, but it’s going to shake up the industry, and we’re only just now beginning to see it.

Qualcomm

After Qualcomm’s earnings report this week, I spoke with CEO Cristiano Amon about his datacenter chip strategy, and why he thinks he’s got a shot at grabbing share from Nvidia and other entrenched names:

If you understand all of those projections that exist about AI, and I think it’s a massive SAM that exists, for addressable market of datacenters. But you expect AI to go from creation of AI to production of AI. Actually, all of those valuations are expecting that all those companies building datacenters, they’re going to generate a lot of profit from the AI.

And what it means is they have to go from training to inference. And I think the addressable market for Qualcomm is all of it. The question is pretty simple: There is a fast cadence of new technology. You’ve started to hear companies say, “I’m not going to use this GPU now, I’m going to wait for the next one.” Because every time you get more efficiency, more density, it changes the economics.

So our approach is very simple. We said we bring some unique things to the table. We’re designing a whole different architecture. Let’s talk about what comes after the GPU, if you want to say it that way, in terms of the efficiency of generating inferencing. We’re not going after training inferencing. And we think there are going to be massive clusters that are dedicated to inference, that we can offer a competitive solution in terms of driving a higher computer density and doing that with lower power. Because at the end of the day, if you want to look at all those projections that exist for datacenter? Power matters. That’s actually one of the factors that people say may slow down the rate of growth of compute.

Qualcomm’s probably going to have to be more than a little better than Nvidia to dislodge some of those developer groups that have really gotten entrenched on Nvidia infrastructure, and then some with AMD as well. So it’s a matter of an installed base and a developer habit that’s already starting to go. Then at the same time, I think about what happened with solid state memory and solid state storage and the way that’s had a huge impact in the datacenter.

Energy is a huge issue. Heat is a huge issue. If Qualcomm can create the kind of delta between themselves and Nvidia that Qualcomm did between themselves and Intel, where Apple, Steve Jobs at first with the iPod and then eventually the iPhone, wanted to use Intel chips instead of going with ARM based chips. But his engineers said, hey, look, these things are too hot, it’s not going to work. So I think it’s going to be a tall hill to climb for Qualcomm to break through in the datacenter. But if they can do in the datacenter what they did to the PC with the shift to mobile, then I think they have shot.

DigitalOcean

Closer to the customer-facing side, DigitalOcean provides that cloud infrastructure to small and medium businesses that want to use AI to drive business results. I spoke with CEO Paddy Srinivasan about how that’s starting to play out:

The beauty of inferencing is that you can actually measure the tangible outcomes from a business perspective. So one example that I walked through during the earnings call this morning is a company called Fal. Fal has generative media models hosted on DigitalOcean. And their customers are companies like Shopify, Canva or Perplexity. So if you are a Shopify merchant and you’re selling a product and you want someone to do a virtual try on, they use the models from, from Fal to project what it looks like before you make a purchasing decision. Those kinds of examples, the outcomes they are driving are increasing the conversion of that customer eventually doing business with that Shopify merchant.

So that’s a very tangible example of how generative media and AI is having an impact on consumer behavior as an example. That is going to deliver a lot of positive business outcomes for that Shopify merchant. It is not just saving a few hours with a coding agent for a developer, but it’s really starting to now percolate through different parts of the economy and driving positive economic growth for everyone.

Amazon’s got to be careful here with sealing off its ecosystem. I understand why they might be doing it. There’s all kinds of people saying – Wikipedia’s saying this – folks are coming out trying to crawl my site, these AI models, and it’s putting an enormous load on our infrastructure and they’re not paying for that. And that’s not really going to work. Plus, on the other end of it, it’s like the old mall model. If you go in this entrance, you’ve got to walk past the pretzels and the hot dogs and whatnot on the way to the store. And that’s kind of how everybody pays the bills.

So we want you to come to the mall, and therefore you’ve got to come in through this entrance. But then e-commerce comes along, direct to consumer comes along, and blows up your model anyway if you don’t adapt. Because that’s how capitalism works. And I think part of what Paddy was pointing out is that’s potentially starting to happen.

Shopify could be a spoiler here if Amazon doesn’t open up, because a lot of the same merchants that are selling on Amazon also have, say, a Shopify presence. And Amazon’s just one of the channels that they have to sell through. Now, if they can figure out how to optimize for AI and give AI more information through the Shopify platform than they’re getting from Amazon, because Amazon is sealing them off, and so those Shopify merchants still make that sale, but they make it through their Shopify store and not through Amazon? That commerce is still going to happen. Amazon’s just not going to benefit. And not only will it not benefit, its hold on the e-commerce ecosystem could slip. Not saying it’s going to happen, but I’m saying there’s a potential opening when there are these dramatic platform shifts like we’re seeing in AI right now.

If you’d like to join me – and peers – for deeper conversations on innovation and leadership, get on this list for Fortt Knox Executive Communities, launching soon: mba.fortt.com.