Look at the logs of almost any FAQ bot and you'll see the same pattern I keep running into: hundreds of people asking the same twenty questions, each time in slightly different words. "How long can I rent a bike?" and "What is the maximum rental period?" mean the same thing, yet the language model computes the answer from scratch every single time.
A semantic cache fixes that by comparing meaning instead of wording. In Semantic cache for Bedrock with ElastiCache and Lambda I build one for a fictional bike rental. Just as important as the code is the question of when such a cache becomes dangerous. This post sums up both.
Meaning instead of exact keys
An embedding model turns each question into a vector, so questions with similar meaning land close to each other. The cache looks up the nearest stored vector, and if the cosine distance is small enough, it returns the stored answer. Otherwise it asks the model and stores the result for next time.
This is not the same as prompt caching in Bedrock. Prompt caching still calls the model and only processes a repeated prompt prefix more cheaply. A semantic cache skips the model call entirely. The two work well together: prompt caching helps with every cache miss.
My setup has three levels, asked in order:
- An exact cache: normalize the question, hash it, one read, no embedding needed.
- A semantic level: create an embedding with Titan Text Embeddings V2 and run a filtered nearest-neighbor search in ElastiCache Serverless for Valkey.
- The model itself, whose answer is then written to both levels with a time to live.
Everything runs in one Lambda function, deployed with the Serverless Framework. The cache uses a public endpoint with TLS and IAM, so no VPC and no NAT gateway.
The dangerous part: false hits
A cache miss costs money. A false hit delivers a wrong answer with full confidence. Take "Do I need a helmet for the e-bike?" and "Do I need a helmet for the cargo bike?" To an embedding model these are nearly identical sentences, but the answers may differ completely.
A threshold alone can't solve this, because it's always one compromise for all questions. What helped me is a set of hard filters before the vector search. Valkey supports TAG filters ahead of the KNN query, so only entries that match exactly are searched at all. In the example these are the topic, a knowledge version (so old answers stop matching after a price change) and the bike type detected from the question. Almost every domain has a handful of such terms: plan names, product variants, countries, contract types.
Some questions never belong in the cache: anything personal like a booking number, anything time-sensitive like current availability, and follow-ups in multi-turn dialogs. The example has a deliberately strict rule that bypasses the cache for those.
Calibrate instead of guessing
Don't just copy a threshold from a blog post, including mine. Suitable distances depend on the embedding model, the dimension and your language. I collect 30 to 50 question pairs from real requests and label whether the same answer would fit:
const pairs = [
{ a: "How long can I rent a bike?", b: "What is the maximum rental period?", same: true },
{ a: "Do you have child seats?", b: "Can I book a child seat as an extra?", same: true },
{ a: "What happens if I get a flat tire?", b: "What happens if I bring the bike back late?", same: false },
];
A small script computes the distance per pair. All "same" pairs should fall below the boundary, all "different" ones above. If they overlap, a further hard filter is usually the better fix than a stricter threshold.
For me, a semantic cache is a business decision as much as a technical one. Which terms flip an answer and when content goes stale is something the domain team knows better than any vector index. And before building it, measure for a week how often questions really repeat.
What the full article covers
- Creating the ElastiCache Serverless cache and the vector index with TAG fields
- The full TypeScript code for policy rules, lookup, storage, the Bedrock call and the handler
- Hit-rate metrics with Embedded Metric Format, tests without AWS, and the calibration script
- A table of common mistakes and a cost calculation to see when the cache pays off
All details and the runnable project: Semantic cache for Bedrock with ElastiCache and Lambda
📖 This is the condensed version. The full article on sascha-ms.de goes deeper – with all code samples, diagrams and a short knowledge check.
👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.
📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (0)