How AI startups think about compute cost versus gross margin

Why this is different from a normal SaaS margin conversation

Most software businesses have near-zero marginal cost per customer. A traditional SaaS company hosting a database and serving web pages might see gross margins in the 70-85% range once it scales, because adding another user costs almost nothing extra. AI startups, particularly those built on large language models or other heavy machine learning workloads, do not work like this. Every time a customer sends a query, the startup pays for GPU or specialised chip time to generate a response. That cost scales with usage in a way traditional software costs do not, which is why compute has become the central line item in how these businesses think about unit economics.

This matters for founders because it changes what growth actually means. In classic SaaS, more usage is almost pure upside. In AI-native products, more usage can compress margin if the underlying cost per query is not falling faster than the price customers are willing to pay.

What sits inside ‘cost of goods sold’ for an AI startup

For an AI company, cost of goods sold typically includes:

• Inference cost: the compute needed every time a model actually answers a query or processes a request, usually the largest and most variable cost
• Training and fine-tuning cost: building or adapting a model, which is more of a fixed or periodic cost depending on whether the startup trains its own models or uses a third-party API
• Model API fees: many startups do not train models at all but pay a foundation model provider per token or per call, which behaves like a variable cost that scales directly with usage
• Storage and data costs: vector databases, embeddings storage and data pipelines
• Human-in-the-loop costs: some products still rely on human review or moderation layered on top of model output

The mix matters enormously. A startup that has built its own smaller, efficient model and controls its own infrastructure has a very different cost structure to one that is a thin layer over a third-party frontier model API, even if both look similar from the outside as ‘AI products’.

Gross margin as a moving target, not a fixed number

Because chip prices, model efficiency and API pricing all move, an AI startup’s gross margin is rarely stable in the way a mature SaaS business’s margin is. Two forces pull in opposite directions:

Model and hardware costs generally fall over time as chips get more efficient and competition between model providers pushes API prices down. This tends to expand margin for a given product, all else equal.

Customer expectations rise at the same time. Users expect faster responses, longer context windows, more accurate outputs and multi-step reasoning, all of which increase compute consumed per query. This tends to compress margin even as unit costs fall, because the startup is doing more work per interaction, not less.

Investors assessing an AI startup’s gross margin therefore want to understand the trend line and the drivers behind it, not just a single snapshot figure. A margin that is improving because the team is genuinely getting more efficient is a very different story to one that looks fine today only because usage is still low.

Questions founders should be able to answer

When an AI startup talks to investors, or simply plans its own pricing, a few questions tend to come up repeatedly:

What is the cost per query or per unit of output, and how does that compare with what the customer pays per unit? This is the AI equivalent of contribution margin and is often more informative than a blended gross margin figure across the whole business.

How much of the cost base is fixed (owned infrastructure, committed cloud reservations) versus variable (pay-as-you-go API calls)? Fixed commitments can lower unit cost at scale but increase risk if usage does not materialise as expected.

Can the product get cheaper to run through smaller or distilled models, caching repeated queries, batching requests, or routing simple queries to cheaper models and only sending complex ones to expensive frontier models? This kind of engineering work, often called model routing or cost optimisation, is increasingly a core competency rather than a technical afterthought.

Is pricing usage-based, seat-based or a hybrid, and does that pricing model actually track the underlying cost driver? A startup charging a flat monthly seat fee while costs scale with usage volume can be caught out if a subset of customers use the product far more heavily than average.

Why this shapes valuation conversations

Because compute cost behaves more like a variable cost of production than a typical software expense, some investors assess AI startups with one eye on comparisons to businesses with a cost of goods sold, rather than assuming SaaS-style margins apply automatically. A startup that can show a credible path to margin expansion, through owning more of its infrastructure, negotiating better rates at scale, or engineering efficiency into the product, is generally viewed more favourably than one whose margin depends entirely on third-party model pricing staying favourable.

Founders raising in the UK should be ready to walk investors through this cost structure in detail, including how it might change as the business scales and as underlying model and chip costs evolve. For guidance on how compute infrastructure and government-backed capacity fit into the wider UK picture, the Department for Science, Innovation and Technology and the British Business Bank both publish material relevant to how AI infrastructure and funding interact in the UK market.