What Actually Makes A Local LLM Fast, And When Does Running One Yourself Cost Less?

Synergised Consulting Ltd

3 November 20266 min read

Title card: Local LLM Speed And When To Own It, First 100 Days Operations

Last Updated: 2 November 2026

Whether running a language model on your own hardware beats paying per token in the cloud comes down to volume, not preference: a peer-reviewed 2026 study found cloud pricing roughly four times cheaper than a local server's electricity at 100,000 prompts a year, but around 7.6 times more expensive at midrange volumes ^[1]. And the speed of a local model is set almost entirely by memory bandwidth and quantisation, not by how fast the processor is. Both facts point to the same operating conclusion: the local-versus-cloud question is answerable with arithmetic before any hardware is bought, and the inputs are your own prompt volume, your prompt tier, and the memory your chosen model needs.

What Actually Sets Local LLM Speed

Speed when a model generates text is a memory problem, not a processor problem. When a model writes one token at a time, every step must read the model's weights out of memory, and the arithmetic units sit mostly idle waiting for that data to arrive. The practical ceiling on generation speed is therefore memory bandwidth divided by how many bytes each token requires the system to read, a relationship explained in detail by Belgavi.AI Lab's technical analysis of bandwidth-bound inference ^[3]. A workstation with a slower processor but faster memory will often outrun a machine with a bigger processor starved of bandwidth.

Worked through in practice: a business choosing between two machines for on-premise inference should compare memory bandwidth figures first and processor benchmark results second. Two recent consumer cards illustrate the point in measured data. The RTX 5090 has substantially more memory bandwidth than the older RTX 3090, and it is 1.76 to 1.86 times faster on current open-weight models in like-for-like tests ^[2]. The gap tracks the bandwidth almost exactly, which is what you would expect if memory, not compute, is the binding constraint.

The Second Lever: Quantisation

Quantisation is the second lever. Models are normally stored at full precision, but they can be compressed into fewer bytes per parameter with modest quality loss, and a compressed model means fewer bytes to read per token, which is faster as well as cheaper to fit in memory. The same compression also decides what fits at all: a model that exceeds the memory available cannot run, whatever its speed would have been.

What The Measured Data Says About Value

A published analysis of 117 real local-LLM builds, covering 298 measured results across 48 models with prices checked between 16 and 24 October 2026, found that a used RTX 3090 delivers about three times more generation speed per euro than an RTX 5090 on every current model measured on both cards ^[2]. The same dataset put the RTX 5070 Ti at 152 tokens per second per €1,000 of card cost running a 20-billion-parameter open-weight model, the best value in its comparison class. Newer is not better value; memory-appropriate is.

The dataset also records the cheapest speed upgrade available, and it costs nothing: speculative decoding, a technique where a small draft model proposes tokens the main model verifies, gave a median 1.76 times speedup across ten same-hardware before-and-after pairs, with gains ranging from 1.26 to 2.45 times depending on the model and prompt ^[2]. An owner who has already bought hardware can often more than recover performance with a configuration change than with a purchase.

When On-Premise Beats Cloud Pricing

A peer-reviewed study published in Frontiers in Computer Science in October 2026 worked the on-premise-versus-cloud comparison end to end, reusing refurbished office computers as local inference servers and comparing their electricity cost against cloud API pricing at three tiers of model quality ^[1]. At 100,000 prompts a year on a small model, cloud cost $48.75 against a local server's $207 annual electricity bill, roughly four times cheaper in the cloud. At midrange models the picture reversed: cloud came to $1,562 a year against roughly $207 of local electricity, about 7.6 times more expensive, saving over $1,350 annually. At premium model tiers cloud reached $12,000 a year, around 58 times the local energy cost.

The worked pattern for an owner is therefore volume-and-tier arithmetic, not a general rule. Count annual prompt volume, identify which tier of model the work actually needs, then place your volume against the study's three thresholds. Low volume on small models stays in the cloud; heavy volume on capable models is where owned hardware pays. The study's comparison assumes the hardware is already paid for through reuse, which is the honest starting condition for a small business anyway.

What Local Hardware Cannot Do

Local inference has genuine limits that the cost arithmetic does not capture. The memory reserved for conversation context grows in step with how much document or dialogue the model holds at once, so long-context work can exhaust a machine that comfortably fits the model itself, as engineering write-ups of production deployments describe ^[4]. Managed cloud providers also ship native tool calling, stable structured output and multimodal handling, while open-weight equivalents often need rigid schema enforcement and closer prompt engineering to reach the same reliability ^[4]. And cloud costs fall to zero when nobody is querying, while owned hardware depreciates whether it is busy or idle ^[4].

This is why the defensible pattern for a small business is routing, not replacement. In our own delivery work, agents run on a self-hosted server while the models behind them stay with hosted API providers, and that split is deliberate: the data and logic stay under our control, while the hardest reasoning runs on hardware nobody had to buy. The same shape applies here. Routine extraction, summarisation and anything touching private records are the candidates for local models; deep multi-step reasoning and long-context work stay on hosted endpoints until volume arithmetic says otherwise.

The Decision, In Order

The sequence that avoids both failure modes, buying hardware that idles and renting capacity that should be owned, is the same order every time. First, measure prompt volume and sort prompts by the model tier they genuinely require. Second, check speed expectations against memory bandwidth and quantised model sizes, not processor marketing. Third, place the volume against published break-even figures such as the Frontiers study's three thresholds ^[1]. Fourth, if hardware is justified, buy for memory bandwidth per euro rather than headline speed, and apply the free configuration wins such as speculative decoding before spending more ^[2]. Only then does local-versus-cloud become a purchase rather than a guess.