Notes on building with LLMs
A few lessons from shipping LLM features to real users.
- 1 min read
- ~130 tokens
- LLMs
- Evals
- RAG
Shipping an LLM feature is less about the model and more about everything around it.
Start with evals
Before tuning prompts, write down 20–50 real examples and what a good answer looks like. Without that, every change is a guess.
Keep the model on a short leash
Give the model the smallest job that solves the problem.
Narrow tasks are easier to test, cheaper to run, and fail in predictable ways.
Retrieval beats cleverness
Most "the model is wrong" bugs are really "the model didn't have the right context" bugs. Fix retrieval first.