What Is Inference Cost for an LLM?

Why inference cost matters before you build anything

Most SMBs focus on the one-time cost of building an AI system and underestimate the ongoing cost of running it. Inference is the running cost. Every time a user sends a message, every time an agent calls a model, every time a document gets summarized, you're paying for inference.

For low-volume internal tools, the bill is negligible. For customer-facing products or high-frequency agents that call the model dozens of times per workflow, inference cost can easily exceed your infrastructure and development costs combined within a year. Getting this number wrong early means repricing your product or killing a feature later.

How inference cost actually works

Tokens are the unit of measurement. A token is roughly four characters of English text. Every prompt you send (input tokens) and every response the model returns (output tokens) gets counted and billed. Output tokens are almost always more expensive than input tokens because generating text is computationally heavier than reading it. On GPT-4o as of mid-2025, input runs around $2.50 per million tokens and output around $10 per million. Claude Sonnet 3.7 is roughly comparable. Smaller models like GPT-4o mini or Llama 3.1 8B cost a fraction of that.

Context window size directly affects your input token count. If you're using RAG and stuffing five retrieved documents into every prompt, or if you're running a multi-turn agent that carries a long conversation history, your input tokens balloon fast. A single agentic workflow that calls the model eight times with a 4,000-token context each call costs as much as 32,000 tokens per user request, not 500.

For private deployments on your own infrastructure or a dedicated cloud instance, the pricing model shifts. You're paying for GPU hours rather than per-token API fees. A dedicated A100 on AWS runs roughly $3 to $4 per hour. At high enough volume, this is cheaper than API billing. At low volume, it's more expensive. The break-even depends on your request rate, model size, and average context length.

When inference cost becomes a serious budget line

Inference cost is negligible for internal tools with fewer than a few hundred daily requests. It becomes significant when you're running customer-facing products, autonomous agents with multi-step reasoning, or document processing pipelines at scale. A logistics client processing 10,000 shipment documents per day at 2,000 tokens each hits 20 million input tokens daily before a single output token is counted.

Model choice also flips the math. Using GPT-4o for every task is like hiring a senior engineer to answer basic FAQ emails. Routing simple classification tasks to a smaller model like Llama 3.1 8B, and only escalating complex reasoning to a frontier model, can cut inference costs by 70% to 90% with no meaningful quality loss on the routed tasks.