An H100 80GB rents for anywhere between $1.30 and $12.29 per hour in public price lists. That is not a cherry-picked sample. It is the spread across providers tracked by Coloprice as of November 6, 2026, where the median sits at $3.90. Runpod lists the same card at $2.89 in Secure Cloud and from $1.99 in Community Cloud, billed by the second. Eight times the price for identical silicon, and that is still not the uncomfortable part of the job.
The uncomfortable part is that GPU-hour price does not decide anything in a corporate deployment. It does not even make the top three arguments once the project reaches security review and the CFO. I am writing this after several deployments where we calculated savings first and built the gateway second. Every time, the savings turned out not to be the reason the thing got built.
What follows is a cost model with the assumptions written down, the architecture of an exit point, the code for a minimal proxy, and three failure modes I have seen in production.
What actually breaks with direct access
Calling a foreign API straight from a corporate network runs into three problems, and none of them is bandwidth.
Keys get revoked. Providers see requests from certain regions and treat them as a regional policy violation: first a 403 on individual models, then a full key block. Recovery means correspondence in which you explain why your office IP ended up where it did. Over an eight-month pilot this happens two or three times, and each occurrence is downtime for a specific team rather than an abstract risk.
Payments do not reconcile. A corporate card does not go through, so you need an intermediary, and an intermediary means a contract, closing documents and VAT. A new cost line appears that was never in the project budget, and it lives separately from the technical work.
The third one is what usually stops the project. If a prompt carries a name, a phone number or bank details, you cannot prove afterwards that it did not. With direct access from applications there is no place where that would be recorded: every developer put their own key into their own service, and nobody retains a trace of what exactly left the perimeter. That is no longer about a 403. It is about data protection law (152-FZ in Russia) and about having nothing to answer an auditor with.
Recalculating the cost of a single request
Take a typical profile: an internal assistant with retrieval over company documents. 1.5 million requests a month, averaging 900 input tokens and 350 output tokens. That is 1.35 billion input and 525 million output tokens.
At an open-model price of roughly $0.15 per million input and $0.60 per million output, you get about $203 for input and $315 for output, so roughly $518 a month. The order of magnitude matters more than the precision here: substitute your own price and the conclusion stays in the same range.
Then the arithmetic breaks the original framing. If the cache hits 30 percent of the time, you save about $155 a month, which is around thirteen thousand roubles. An engineer who owns the gateway, the routes, the updates and incident triage costs 250,000 to 300,000 roubles a month in Russia, taxes included. That is five percent of their cost, and only with a good cache hit rate. At these volumes a gateway does not pay for itself through token savings. It pays for itself through something else.
What it actually pays for
A single personal data leak costs a legal entity three to five million roubles under part 12 of article 13.11 of the Administrative Code when 1,000 to 10,000 data subjects are affected, five to ten million for 10,000 to 100,000, and for a repeat offence the fine is calculated from turnover. For comparison, the annual payroll of that same engineer is 3.0 to 3.6 million. One incident costs more than a year of their work, and that number excludes reputational damage and projects frozen during the investigation.
Which answers the original question. You do not install a gateway to save on GPU-hours. You install it so there is one place where it is visible what left, what stayed, and why a particular decision was made. Savings arrive as a side effect rather than as the goal.
What the exit point does
Put it where traffic is already aggregated but has not yet left the perimeter: between the applications and the outside world. From there it takes on four jobs that cannot be done inside a client application.
Masking happens first, before anything is sent. Not after the response, not in the logs, but in the request body before the upstream call: entities are replaced with placeholders backed by a reversible dictionary so the answer can be reassembled. It is also fair to admit that masking breaks some workloads. Which ones is covered below, because it is not a detail.
Next is routing by sensitivity. Ticket classification, field extraction from contracts and everything else on a short context goes to a small model inside the perimeter. Long analysis, code generation and image work go to a large model through a tunnel, but only after masking. The rule lives in configuration rather than in a developer's head, otherwise in a month nobody can say why a given request went where it went.
Then quotas. A limit per team, a separate counter for experiments, a monthly ceiling per project. Without this one enthusiast with a batch script consumes the budget in a week, and you learn about it from the invoice.
And the decision log, without which everything above is pointless. For every request: an identifier, which masking was applied, which route was chosen, what the response cost. This log is what you show an auditor. Its absence is usually what "no compliance" actually means, even when everything was formally approved.
A separate note on streaming, because almost every hand-built gateway trips here. Chat returns the answer over SSE, and if the gateway buffers the whole body to count tokens and write the log, the user stares at nothing for twenty seconds and then gets the entire text at once. That feels worse than a slow but live answer. Read usage from the final chunk and write the log after the stream closes, otherwise you get either added latency or lost records when the connection drops.
Code: a minimal proxy
Below is the base I usually start from. The token comes from the environment, every request has an identifier, the timeout is bounded, and retries happen only on transport errors so that the same request is not billed twice.
import os, uuid, asyncio
import httpx
from fastapi import FastAPI, HTTPException, Request
app = FastAPI(title="corporate-llm-gateway")
UPSTREAM = os.environ["UPSTREAM_URL"]
TOKEN = os.environ["UPSTREAM_TOKEN"]
@app.post("/v1/chat")
async def chat(request: Request):
payload = await request.json()
rid = request.headers.get("x-request-id") or str(uuid.uuid4())
payload = sanitize_pii(payload) # mask BEFORE leaving the perimeter
route = pick_route(payload, request.headers) # small model / large model / reject
async with httpx.AsyncClient(timeout=httpx.Timeout(30.0, connect=5.0)) as client:
for attempt in range(2):
try:
r = await client.post(
route.url,
json=payload,
headers={"Authorization": f"Bearer {TOKEN}", "x-request-id": rid},
)
if r.status_code in (429, 500, 502, 503, 504) and attempt == 0:
await asyncio.sleep(0.5)
continue
log_decision(rid, route.name, r.status_code, r.headers.get("x-usage"))
return r.json()
except httpx.TransportError:
if attempt == 1:
raise HTTPException(504, "upstream unavailable")
The sanitize_pii, pick_route and log_decision functions are deliberately left out: the first is an entity dictionary and regular expressions for your formats, the second is a rule table, the third writes to your logging system. What matters is not the implementation but the order. Masking and route selection run before the network call, not after it.
Three failure modes I have seen
A cache keyed on the wrong thing. Response caching looks like free money until the key omits the model version, the system prompt and temperature. This surfaces in support: answers meant for the basic tier reach premium customers because the key was built from the question text alone. Semantic caching is more honest about hit rates and more dangerous about consequences. Fifteen to thirty percent hits on a stream of similar requests is entirely achievable, but every hit is an answer to a similar question rather than to the same one.
Masking breaks code generation. Replace identifiers with placeholders and the model loses the structure: variable names and dictionary keys become identical labels, and the generated code does not compile. The fix is not to disable masking but to use different policies for different task types. Code goes through a route where source is allowed and personal data is not, while documents containing such data go through strict masking.
Failover that doubles the bill. When a route degrades, the gateway retries on a backup while the first request is neither cancelled nor untariffed, it is simply billed. On a stream of more than a million requests that is visible money and, worse, duplicated answers. You need an idempotency key and an explicit cancellation of the previous call on timeout.
How to test it
Synthetic pings are useless here. A test that shows something looks like this: record a real day of request traffic, replay it through the gateway, and drop one route halfway through the run. Then look at two numbers: the share of successful responses and the p95 latency. If both stay inside the contracted bounds, the architecture works. If they move, the problem is in the balancer or the timeouts rather than in the model, and that distinction matters during an incident review.
Test masking separately: run a set of requests with known entities and verify that none of them appears in the outbound log. That test should be written before the first incident rather than after it.
When you do not need a gateway
If you have one team, ten developers, no personal data in prompts and fewer than fifty thousand requests a month, you do not need a gateway. A hundred-line wrapper, a single key in a secret store and a ban on hardcoding keys solve the same problem more cheaply. I have seen projects that built a gateway "for the future" and then spent six months unable to explain who owns it.
A gateway becomes mandatory at exactly the moment a second application gets model access and the first personal data clause appears in a contract. Before that it only adds a service to your on-call rotation.
Takeaway
GPU-hour price is a poor metric for this problem because it answers "what does the hardware cost" while the decision is made on "what happens if data goes somewhere it should not". An eightfold spread on the same card is a good illustration of how secondary the first question is. Even if you pick the cheapest price, the answer to the second question does not change.
The practical conclusion is simple. Count the cost of one successfully handled request, including masking, routing and logging, rather than the cost of an hour of rental. And keep the exit point where the data boundary is: if you cannot show where a request was masked and where it went, you have neither compliance nor an SLA.
Sources
Prices and spreads were verified on November 11, 2026: the H100 80GB summary on Coloprice, rates on the Runpod pricing page. Fines are quoted from part 12 of article 13.11 of the Russian Administrative Code as amended with effect from May 30, 2025.
Top comments (0)