An LLM is fundamentally a next-token predictor. Given a sequence of tokens, it outputs a probability distribution over the entire vocabulary (~32K-128K tokens). The training objective is to minimize cross-entropy loss between the predicted distribution and the actual next token across billions of text examples. The model learns grammar, facts, reasoning patterns, and even code โ all as statistical regularities in token sequences.
Below ~10B parameters, models just predict tokens. Above ~50B, new abilities emerge that weren't explicitly trained: chain-of-thought reasoning, few-shot learning, code generation, translation between languages never paired in training data. This is why scale matters and is the foundation of the "scaling laws" (Chinchilla, Kaplan et al.).
Text is never fed directly to an LLM. It's first converted to tokens (sub-word units). Understanding tokenization is critical because:
| Aspect | Why It Matters | Example |
|---|---|---|
| Cost | API pricing is per-token, not per-word | "unbelievable" = 3 tokens = 3x cost vs 1 word |
| Context limits | 128K tokens โ 128K words (~75K words) | 1 token โ 0.75 English words on average |
| Non-English penalty | Languages like Hindi/Chinese use 2-3x more tokens per word | "เคจเคฎเคธเฅเคคเฅ" might be 6 tokens vs "hello" = 1 token |
| Code tokenization | Whitespace and syntax consume tokens | 4 spaces of indentation = 1 token wasted per line |
| Number handling | Numbers tokenize unpredictably | "1234567" might split as ["123", "45", "67"] โ why LLMs are bad at math |
Algorithms: BPE (GPT, LLaMA) โ merges frequent byte pairs iteratively. WordPiece (BERT) โ maximizes likelihood. SentencePiece/Unigram (T5) โ statistical segmentation. Modern LLMs use vocabularies of 32K-128K tokens.
| Parameter | What it controls | Range | When to change |
|---|---|---|---|
| Temperature | Sharpens/flattens the probability distribution | 0.0 โ 2.0 | 0 for extraction/code, 0.7 for chat, 1.2+ for creative writing |
| Top-p (nucleus) | Cumulative probability cutoff โ only consider tokens within top-p mass | 0.7 โ 1.0 | Use 0.9 as default; lower for focused, higher for diverse |
| Top-k | Hard limit on candidate tokens | 10 โ 100 | Rarely needed if using top-p; useful as safety net |
| Frequency penalty | Penalizes repeated tokens proportionally | 0.0 โ 2.0 | Increase to reduce repetitive output |
| Presence penalty | Flat penalty for any repeated token | 0.0 โ 2.0 | Increase to encourage topic diversity |
| Max tokens | Generation length limit | 1 โ 128K | Set to expected output length + margin; never use -1 for safety |
| Stop sequences | Strings that stop generation | Any text | Essential for structured output: stop at "}" for JSON |
Don't combine temperature=0 with top_p=0.1 โ they interact. Use either temperature OR top-p for sampling control, not both. OpenAI recommends changing one and leaving the other at default.
The context window determines how many tokens the model can process in a single call (input + output combined).
| Model | Context Window | Approx. Pages |
|---|---|---|
| GPT-4o | 128K tokens | ~200 pages |
| Claude 3.5 Sonnet | 200K tokens | ~350 pages |
| Gemini 1.5 Pro | 2M tokens | ~3,000 pages |
| LLaMA 3.1 | 128K tokens | ~200 pages |
| Mistral Large | 128K tokens | ~200 pages |
"Lost in the Middle" (Liu et al., 2023): Performance degrades for information placed in the middle of very long contexts. Models attend most to the beginning and end of prompts. Strategy: put the most important content at the start or end; use retrieval to avoid stuffing the entire context.
Training an LLM from scratch involves: (1) Data collection โ crawl the web (Common Crawl, ~1 trillion tokens), books, code (GitHub), conversations. (2) Data cleaning โ deduplication, quality filtering, toxicity removal, PII scrubbing. (3) Tokenizer training โ build BPE vocabulary from the corpus. (4) Pre-training โ next-token prediction on massive GPU clusters (thousands of A100s/H100s for weeks). Cost: $2M-$100M+ for frontier models.
A base model predicts tokens but doesn't follow instructions or refuse harmful content. Alignment methods bridge this gap:
| Method | How It Works | Used By |
|---|---|---|
| SFT (Supervised Fine-Tuning) | Train on (instruction, response) pairs from human annotators | All models (Step 1) |
| RLHF | Train a reward model on human preferences, then optimize policy via PPO | GPT-4, Claude (early) |
| DPO (Direct Preference Optimization) | Skip the reward model โ directly optimize from preference pairs, simpler and more stable | LLaMA 3, Zephyr, Gemma |
| Constitutional AI | Model critiques and revises its own outputs against a set of principles | Claude (Anthropic) |
| RLAIF | Use an AI model (not humans) to generate preference data | Gemini, some open models |
| Provider | Flagship Model | Strengths | Best For |
|---|---|---|---|
| OpenAI | GPT-4o, o1, o3 | Best all-around, strong coding, reasoning chains (o1/o3) | General purpose, production |
| Anthropic | Claude 3.5 Sonnet | Best for long documents, coding, safety-conscious | Enterprise, agents, analysis |
| Gemini 1.5 Pro/2.0 | Massive context (2M), multi-modal, grounding | Document processing, multi-modal | |
| Meta | LLaMA 3.1/3.2 | Best open-source, fine-tunable, commercially free | Self-hosting, fine-tuning |
| Mistral | Mistral Large, Mixtral | Strong open models, MoE efficiency | European market, cost-effective |
| DeepSeek | DeepSeek V3, R1 | Exceptional reasoning, competitive with o1 | Math, coding, research |
Chinchilla Scaling Law (Hoffmann et al., 2022): For a compute-optimal model, training tokens should scale proportionally to model parameters. A 70B model should be trained on ~1.4 trillion tokens. Key insight: many earlier models were undertrained (too many params, not enough data). LLaMA showed smaller, well-trained models can match larger undertrained ones.
Answer: The model becomes deterministic, always picking the highest-probability token (greedy decoding). Use for tasks requiring consistency (code generation, extraction, classification). Side effects: can get stuck in repetitive loops; technically temperature=0 is greedy, temperature=1 is the trained distribution, above 1 is "hotter" (more random). For near-deterministic with slight randomness, use temperature=0.1 with top_p=1.
Answer: LLMs don't "know" facts โ they model token probabilities. When asked about something rare or unseen, the model generates statistically plausible text rather than admitting ignorance. Solutions: (1) RAG โ ground answers in retrieved documents; (2) Lower temperature โ reduce sampling randomness; (3) Structured output forcing โ constrain output to valid formats; (4) Self-consistency โ sample N times, pick the majority answer; (5) Calibrated prompting โ explicitly instruct "say I don't know if unsure."
Answer: Context window is the tokens the model can process in a single inference pass โ it's completely stateless. There is no persistent memory between API calls. "Memory" in frameworks like LangChain is implemented externally by: (1) storing past conversation turns in a database, (2) summarizing old turns to compress history, (3) reinserting relevant history into the new prompt. Every call is independent โ the model has zero recall of previous calls.
Answer: RLHF: Train a separate reward model on human preferences, then optimize the LLM policy via PPO (reinforcement learning). Complex, unstable, expensive. DPO (Direct Preference Optimization): Skip the reward model entirely โ directly optimize from preference pairs using a closed-form solution. Simpler, more stable, cheaper. DPO is now preferred for open-source models (LLaMA 3, Gemma). RLHF is still used by OpenAI/Anthropic where they have massive human labeling infrastructure.
Answer: Research by Liu et al. (2023) showed that LLMs perform significantly worse when relevant information is placed in the middle of long contexts compared to the beginning or end. The model's attention mechanism attends most to recent tokens (recency bias) and the very first tokens (primacy bias). Practical implication: place the most critical context at the start or end of your prompt, never buried in the middle of a long document.
Answer: GPT-4o is a standard auto-regressive LLM โ generates tokens left-to-right in one pass. o1/o3 are reasoning models that use "chain-of-thought before answering" โ they generate internal reasoning tokens (hidden from the user) before producing the final answer. This makes them dramatically better at math, logic, and coding, but 3-10x slower and more expensive. Use GPT-4o for speed-sensitive tasks (chat, extraction), o1/o3 for complex reasoning (math proofs, hard coding, multi-step analysis).
For each token, compute 3 vectors via learned linear projections: Query (Q), Key (K), Value (V). The attention formula:
| Component | Role | Analogy |
|---|---|---|
| Query (Q) | What this token is looking for | Search query on Google |
| Key (K) | What each token offers/advertises | Page titles in search index |
| Value (V) | Actual content to retrieve | Page content returned |
| โdk scaling | Prevents softmax saturation for large dims | Normalization for numerical stability |
Why it works: Self-attention lets every token attend to every other token in O(1) hops (vs O(n) for RNNs). "The cat sat on the mat" โ the word "mat" can directly attend to "cat" and "sat" to understand context, without information passing through intermediate words.
Run h independent attention heads in parallel, each learning different relationship types. Concatenate outputs and project. GPT-4 likely uses ~96 heads. Each head specializes: head 1 may track subject-verb agreement, head 2 may track pronoun coreference, head 3 may track positional patterns.
| Variant | Key Idea | Used By | KV Cache Savings |
|---|---|---|---|
| MHA (Multi-Head) | Separate Q, K, V per head | GPT-2, BERT | 1x (baseline) |
| GQA (Grouped Query) | Share K,V across groups of Q heads | LLaMA 3, Gemma, Mistral | 4-8x smaller |
| MQA (Multi-Query) | Single K,V shared across ALL Q heads | PaLM, Falcon | 32-96x smaller |
| Sliding Window | Attend only to nearby tokens (window) | Mistral, Mixtral | Fixed memory regardless of length |
RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors in complex space. Advantages: (1) Relative position naturally emerges from dot products, (2) Enables length extrapolation beyond training length (with techniques like YaRN, NTK-aware scaling), (3) No additional parameters. Used by LLaMA, Mistral, Gemma, Qwen โ virtually all modern open LLMs.
Each transformer block has: (1) Multi-Head Attention โ (2) Residual Connection + LayerNorm โ (3) Feed-Forward Network (FFN) with hidden dim 4x model dim โ (4) Residual Connection + LayerNorm. Modern models use Pre-LayerNorm (normalize before attention, not after) and SwiGLU activation in FFN instead of ReLU for better performance.
Standard attention requires O(nยฒ) memory for the attention matrix. FlashAttention (Dao et al., 2022) computes exact attention without materializing the full matrix by using tiling and kernel fusion. Result: 2-4x faster inference, ~20x less memory for long sequences. FlashAttention 2 adds further optimizations. Essential for context windows >8K tokens.
Instead of one massive FFN, use N expert FFNs and a router that selects top-k experts per token. Only selected experts are activated (sparse computation). Mixtral 8x7B has 8 experts, activates 2 per token โ 47B total params but only 13B active per token. Result: 3-4x more efficient than dense models of same quality.
| Architecture | Attention Type | Best For | Examples |
|---|---|---|---|
| Decoder-Only | Causal (left-to-right only) | Text generation, chat, code | GPT-4, LLaMA, Gemma, Mistral |
| Encoder-Only | Bidirectional (sees all tokens) | Classification, NER, embeddings | BERT, RoBERTa, DeBERTa |
| Encoder-Decoder | Encoder bidirectional, decoder causal | Translation, summarization | T5, BART, mT5, Flan-T5 |
Answer: For large d_k, dot products grow large in magnitude (variance โ d_k), pushing softmax into regions with extremely small gradients (saturated). Dividing by โd_k normalizes variance to 1, keeping gradients healthy. Without it, training becomes unstable โ same principle as Xavier/He weight initialization.
Answer: During autoregressive generation, Key and Value matrices for past tokens are cached in GPU memory so they don't need recomputation on each new token. Without KV cache: generating token n requires reprocessing all n-1 previous tokens โ O(nยฒ) total work. With KV cache: each new token only computes its own Q, K, V and attends to cached K, V โ O(n) total. A 7B model with 8K context uses ~4GB just for KV cache. This is why GPU memory (not compute) is the real bottleneck for long-context inference.
Answer: MHA (Multi-Head Attention): Separate K,V per head โ maximum expressivity but largest KV cache. GQA (Grouped Query Attention): K,V shared across groups of Q heads (e.g., 8 Q heads share 1 KV pair). 4-8x smaller KV cache with minimal quality loss. Used by LLaMA-3, Mistral, Gemma. MQA (Multi-Query Attention): ALL Q heads share a SINGLE K,V pair. Maximum KV cache savings (32-96x) but slightly lower quality. Used by PaLM, Falcon. Industry has settled on GQA as the best tradeoff.
Answer: RoPE (Rotary Position Embedding) encodes position by rotating Q and K vectors in 2D complex planes. Key advantages: (1) Relative position naturally emerges from dot products โ Attention(q_m, k_n) depends only on m-n, not absolute positions. (2) No additional learned parameters. (3) Better length generalization โ techniques like YaRN and NTK-aware scaling allow extending context beyond training length. Sinusoidal encoding struggled with extrapolation and required absolute position awareness.
Answer: Standard attention materializes the full NรN attention matrix in GPU HBM (High Bandwidth Memory). FlashAttention uses tiling โ it breaks Q, K, V into blocks, computes attention within SRAM (fast on-chip memory), and never writes the full attention matrix to HBM. This reduces memory IO from O(Nยฒ) to O(Nยฒ/M) where M is SRAM size. Result: exact same output, but 2-4x faster and uses O(N) memory instead of O(Nยฒ). It's a pure systems optimization โ no approximation.
Answer: MoE replaces the single FFN in each transformer block with N parallel expert FFNs plus a learned router. For each token, the router selects top-k experts (usually k=2). Only selected experts are activated โ rest are skipped. Benefits: Train a model with 8x more parameters at ~2x the compute cost of a dense model. Tradeoffs: (1) All parameters must fit in memory even though only k are active. (2) Load balancing โ if router always picks the same experts, others waste space. Solved with auxiliary loss. (3) Harder to fine-tune โ expert specialization can be disrupted. Example: Mixtral 8x7B = 47B params but only 13B active per token.
The transformers library provides a unified API to load, run, and fine-tune any model architecture. It wraps 200+ architectures (GPT, LLaMA, Mistral, Gemma, T5, BERT, ViT, Whisper, etc.) behind Auto Classes that detect architecture automatically from config.json.
| AutoClass | Use Case | Example Models |
|---|---|---|
| AutoModelForCausalLM | Text generation (decoder-only) | LLaMA, GPT-2, Mistral, Gemma |
| AutoModelForSeq2SeqLM | Translation, summarization | T5, BART, mT5, Flan-T5 |
| AutoModelForSequenceClassification | Text classification, sentiment | BERT, RoBERTa, DeBERTa |
| AutoModelForTokenClassification | NER, POS tagging | BERT-NER, SpanBERT |
| AutoModelForQuestionAnswering | Extractive QA | BERT-QA, RoBERTa-QA |
| AutoModel (base) | Embeddings, custom heads | Any backbone |
torch_dtype=torch.bfloat16 โ half precision, saves 50% memory
device_map="auto" โ auto-shard across GPUs/CPU/disk
load_in_4bit=True โ 4-bit quantization via BitsAndBytes
attn_implementation="flash_attention_2" โ use FlashAttention for 2-4x faster inference
trust_remote_code=True โ needed for custom architectures with code on the Hub
The pipeline() function wraps tokenization + model + post-processing into a single call. Under the hood: tokenize โ model forward pass โ decode/format output. Supports these key tasks:
| Task | Pipeline Name | Default Model |
|---|---|---|
| Text Generation | "text-generation" | gpt2 |
| Sentiment Analysis | "sentiment-analysis" | distilbert-sst2 |
| Named Entity Recognition | "ner" | dbmdz/bert-large-NER |
| Summarization | "summarization" | sshleifer/distilbart-cnn |
| Translation | "translation_en_to_fr" | Helsinki-NLP/opus-mt |
| Zero-Shot Classification | "zero-shot-classification" | facebook/bart-large-mnli |
| Feature Extraction (Embeddings) | "feature-extraction" | sentence-transformers |
| Image Classification | "image-classification" | google/vit-base |
| Speech Recognition | "automatic-speech-recognition" | openai/whisper-base |
| Text-to-Image | "text-to-image" | stabilityai/sdxl |
The tokenizers library is written in Rust with Python bindings. It's 10-100x faster than pure-Python tokenizers. Key tokenization algorithms used by modern LLMs:
| Algorithm | Used By | How It Works |
|---|---|---|
| BPE (Byte-Pair Encoding) | GPT-2, GPT-4, LLaMA | Repeatedly merges most frequent byte pairs. "unbelievable" โ ["un", "believ", "able"] |
| SentencePiece (Unigram) | T5, ALBERT, XLNet | Statistical model that finds optimal subword segmentation probabilistically |
| WordPiece | BERT, DistilBERT | Greedy algorithm; splits by maximizing likelihood. Uses "##" prefix for sub-tokens |
โข Numbers tokenize unpredictably: "1234" might be 1-4 tokens depending on the model
โข Whitespace matters: " hello" and "hello" produce different tokens in GPT
โข Non-English languages use more tokens per word (higher cost per concept)
โข Always use the model's own tokenizer โ never mix tokenizers between models
datasets uses Apache Arrow for columnar, memory-mapped storage. A 100GB dataset can be iterated without loading into RAM. Key features:
Memory Mapping: Data stays on disk; only accessed rows are loaded into memory. Streaming: load_dataset(..., streaming=True) returns an iterable โ process terabytes with constant memory. Map/Filter: Apply transformations with automatic caching and multiprocessing. Hub Integration: 150,000+ datasets available via load_dataset("dataset_name").
The Trainer class handles: training loop, evaluation, checkpointing, logging (to TensorBoard/W&B), mixed precision, gradient accumulation, distributed training, and early stopping. You just provide model + dataset + TrainingArguments. For instruction-tuning LLMs, use TRL's SFTTrainer (built on top of Trainer) which handles chat templates and packing automatically.
accelerate abstracts away multi-GPU, TPU, and mixed-precision complexity. Write your training loop once; run on 1 GPU or 64 GPUs with zero code changes. Key feature: Accelerator class wraps your model, optimizer, and dataloader. It handles data sharding, gradient synchronization, and device placement automatically.
Every model on HF Hub is a Git LFS repo containing: config.json (architecture), model.safetensors (weights), tokenizer.json, and a README.md (model card). You can push your own models with model.push_to_hub(). The Hub supports: model versioning (Git branches/tags), automatic model cards, gated access (license agreements), and API inference endpoints.
| Library | Purpose | Key Feature |
|---|---|---|
peft | Parameter-efficient fine-tuning | LoRA, QLoRA, Adapters, Prompt Tuning |
trl | RLHF and alignment training | SFTTrainer, DPOTrainer, PPOTrainer, RewardTrainer |
diffusers | Image/video generation | Stable Diffusion, SDXL, ControlNet, IP-Adapter |
evaluate | Metrics computation | BLEU, ROUGE, accuracy, perplexity, and 100+ metrics |
gradio | Build ML demos | Web UI for any model in 5 lines of code |
smolagents | Lightweight AI agents | Code-based tool calling, HF model integration |
safetensors | Safe model format | Fast, safe, and efficient tensor serialization (replaces pickle) |
huggingface_hub | Hub API client | Download files, push models, create repos, manage spaces |
HF Spaces lets you deploy Gradio or Streamlit apps on managed infrastructure. Free CPU tier for demos; upgrade to T4 ($0.60/hr) or A100 ($3.09/hr) for GPU workloads. Spaces support Docker, static HTML, and custom environments. They auto-build from a Git repo with a simple requirements.txt. Ideal for: model demos, portfolio projects, internal tools, and quick prototypes.
from_pretrained and pipeline?Answer: pipeline() is a high-level convenience wrapper โ it auto-detects the task, loads both model + tokenizer, handles tokenization/decoding, and returns human-readable output. from_pretrained() gives raw access to model weights for: custom inference loops, fine-tuning, extracting embeddings, modifying the model architecture, or anything beyond standard inference. Rule: prototyping โ pipeline, production/training โ from_pretrained.
device_map="auto" and how does model sharding work?Answer: It uses the accelerate library to automatically distribute model layers across available hardware. The algorithm: (1) Measure available memory on each GPU, CPU, and disk; (2) Place layers sequentially, filling GPU first, spilling to CPU, then disk. For a 70B model on two 24GB GPUs: layers 0-40 on GPU 0, layers 41-80 on GPU 1. CPU/disk offloading adds latency but enables running models that don't fit in GPU memory at all. Use max_memory param to control allocation.
Answer: Datasets uses Apache Arrow โ a columnar, memory-mapped format. Key advantages: (1) Memory mapping: A 100GB dataset uses near-zero RAM โ data stays on disk but accessed at near-RAM speed via OS page cache. (2) Zero-copy: Slicing doesn't duplicate data. (3) Streaming: Process datasets larger than disk with streaming=True. (4) Parallel map: num_proc=N for multi-core preprocessing. (5) Caching: Processed results are automatically cached to disk. Pandas loads everything into RAM โ impossible for large-scale ML datasets.
Answer: Each instruction-tuned model is trained with a specific format for system/user/assistant messages. Llama uses <|begin_of_text|><|start_header_id|>system<|end_header_id|>, while ChatML uses <|im_start|>system. If you format input incorrectly, the model behaves like a base model (no instruction following). tokenizer.apply_chat_template() auto-formats messages correctly for any model. This is the #1 mistake beginners make โ using raw text instead of the chat template.
Answer: (1) Accept the model license on the Hub model page. (2) Create a read token at hf.co/settings/tokens. (3) For local: huggingface-cli login. (4) In CI/CD: set HF_TOKEN environment variable. (5) In code: pass token="hf_xxx" to from_pretrained(). For Docker: bake the token as a secret, never in the image. For Kubernetes: use a Secret mounted as an env var. The token is only needed for download โ once cached locally, no token is needed for inference.
Answer: Traditional PyTorch models use Python's pickle format, which can execute arbitrary code during loading โ a security vulnerability. A malicious model file could run code on your machine when loaded. safetensors is a safe, fast tensor format that: (1) Cannot execute code (pure data), (2) Supports zero-copy loading (memory-mapped), (3) Is 2-5x faster to load than pickle, (4) Supports lazy loading (load only specific tensors). It's now the default format on HF Hub.
Instead of modifying the full weight matrix W (d ร k), LoRA trains two small matrices: A (d ร r) and B (r ร k) where rank r << min(d,k). The adapted weight is W + ฮฑBA. During inference, BA is merged into W โ zero latency overhead. Only 0.1-1% of parameters are trained.
| Method | Trainable Params | GPU Needed (7B) | Quality |
|---|---|---|---|
| Full Fine-Tuning | 100% | ~80GB | Best |
| LoRA (r=16) | ~0.5% | ~16GB | Very Good |
| QLoRA (4-bit + LoRA) | ~0.5% | ~8GB | Good |
| Prompt Tuning | <0.01% | ~6GB | Task specific |
QLoRA (Dettmers et al., 2023) combines: (1) 4-bit NF4 quantization of the base model, (2) double quantization to compress quantization constants, (3) paged optimizers to handle gradient spikes. Fine-tune a 65B model on a single 48GB GPU โ impossible before QLoRA.
Answer: Rank r controls the expressiveness of the adaptation. Lower r (4-8): minimal params, fast, good for narrow tasks. Higher r (16-64): more capacity, better for complex tasks. Rule of thumb: start with r=16. If quality is poor, try r=64. If memory is tight, try r=4 with higher alpha. lora_alpha/r acts as the effective learning rate scaling.
Answer: Target the attention projection layers: q_proj, k_proj, v_proj, o_proj. Optionally add MLP layers (gate_proj, up_proj, down_proj). Targeting more modules increases quality but also memory. Research shows q_proj + v_proj alone gives 80% of the benefit. Use target_modules="all-linear" in recent PEFT versions for maximum coverage.
Answer: In full fine-tuning, training on new data overwrites old weights, causing the model to "forget" general capabilities. LoRA freezes the original weights โ the base model is untouched. Only the low-rank adapter is trained. This means the base knowledge is preserved, and you can merge/unmerge adapters to switch tasks.
Indexing phase (offline): (1) Load documents (PDF, HTML, Markdown, DB), (2) Chunk into segments (~500 tokens), (3) Embed each chunk with an embedding model, (4) Store vectors + metadata in a vector database.
Query phase (online): (1) Embed user query with same model, (2) Retrieve top-k similar chunks via ANN search, (3) Inject chunks into LLM prompt as context, (4) LLM generates a grounded response with citations.
| Strategy | How It Works | Best For | Tradeoff |
|---|---|---|---|
| Fixed-size | Split every N tokens | Generic text | May cut mid-sentence |
| Recursive character | Split by paragraphs, sentences, words | Most documents | LangChain default; good balance |
| Semantic chunking | Split where embedding similarity drops | Long documents | Groups related content; 10x slower |
| Document structure | Parse by headings, sections, tables | PDFs, HTML, Markdown | Preserves context hierarchy |
| Agentic chunking | LLM decides chunk boundaries | Highest quality | Expensive but best recall |
256-512 tokens for OpenAI embeddings, 128-256 for smaller models. Use 50-100 token overlap. Test with your actual queries โ measure retrieval recall, not just generation quality.
| Model | Dims | Max Tokens | Quality | Cost |
|---|---|---|---|---|
| OpenAI text-embedding-3-large | 3072 | 8191 | Top tier | $0.13/1M |
| OpenAI text-embedding-3-small | 1536 | 8191 | Very good | $0.02/1M |
| Cohere embed-v3 | 1024 | 512 | Top tier | $0.10/1M |
| BGE-large-en-v1.5 | 1024 | 512 | Great (open) | Free |
| all-MiniLM-L6-v2 | 384 | 256 | Good | Free |
| Technique | How It Works | When to Use |
|---|---|---|
| Hybrid Search | BM25 (keyword) + vector via Reciprocal Rank Fusion | Queries mixing keywords + semantic intent |
| Re-ranking | Cross-encoder re-scores top-N for precision | Always โ retrieve 20, re-rank to 5 |
| HyDE | LLM generates hypothetical answer, embed that | Short queries that don't match doc vocab |
| Parent-child chunks | Index small chunks, retrieve parent doc | When chunk boundaries lose context |
| Query decomposition | Break complex query into sub-queries | Multi-part questions |
| Self-RAG | Model decides whether to retrieve | When retrieval isn't always needed |
Process images, tables, charts alongside text: (1) Vision models โ use GPT-4o to caption images/charts, embed captions. (2) Table extraction โ extract as structured data. (3) ColPali โ directly embed PDF page screenshots without OCR.
| Metric | Measures | Target |
|---|---|---|
| Faithfulness | Are claims supported by context? | 0.9+ |
| Answer Relevance | Does answer address the question? | 0.85+ |
| Context Recall | Did retriever find all needed info? | 0.8+ |
| Context Precision | Is retrieved context relevant? | 0.8+ |
Answer: Overlap ensures sentences spanning chunk boundaries aren't lost. A 50-token overlap on 500-token chunks means each chunk shares context with neighbors. Without it, a key sentence split across chunks is invisible during retrieval. Tradeoff: more overlap = more storage + slightly redundant results.
Answer: Use RAGAS: (1) Faithfulness โ answer uses only retrieved context? (2) Answer Relevance โ addresses the question? (3) Context Recall โ retriever found all needed info? (4) Context Precision โ retrieved context relevant? Build a golden dataset of (question, ground_truth, expected_contexts). Use LLM-as-judge for scale.
Answer: HyDE: Short queries that don't match document vocabulary. LLM generates hypothetical answer, embed that instead. Query decomposition: Complex multi-part questions. Break into sub-queries, retrieve for each, combine. Use HyDE for vocabulary mismatch, decomposition for complexity.
Answer: Naive: embed query -> top-k -> stuff in prompt -> generate. Problems: poor recall, irrelevant chunks. Advanced: adds query transformation, hybrid search, re-ranking, parent-child chunks, metadata filtering. Advanced RAG achieves 20-40% higher faithfulness in production.
Answer: (1) Extract and caption โ tables to markdown, images captioned via GPT-4o, embed text. (2) Multi-modal embeddings โ CLIP for images alongside text. (3) ColPali โ embed entire PDF page screenshots without OCR. Approach 1 is most production-ready; approach 3 is newest.
| Metric | Best For | Range |
|---|---|---|
| Cosine Similarity | Text embeddings (normalized) | -1 to 1 (1 = identical) |
| Dot Product | When magnitude matters | -inf to inf |
| L2 (Euclidean) | Image embeddings | 0 to inf (0 = identical) |
Critical rule: Use the same metric your embedding model was trained with. OpenAI = cosine. Wrong metric can drop recall by 20%+.
| Algorithm | How It Works | Speed | Memory | Best For |
|---|---|---|---|---|
| HNSW | Hierarchical graph navigation | Very fast | High | Low-latency (<10ms) |
| IVF | Cluster vectors, search nearby | Fast | Medium | Large datasets, GPU |
| PQ | Compress vectors into codes | Medium | Very low | Billions of vectors |
| ScaNN | Google's anisotropic hashing | Very fast | Medium | Extreme scale |
| DiskANN | SSD-based graph search | Fast | Very low | Billion-scale on commodity HW |
Multi-layered graph. Top layers are sparse "highways" with long-range connections. Bottom layers are dense with local neighbors. Search starts at top, greedily navigates toward query, drops to lower layers for precision. Params: M (connections per node, 16-64), efConstruction (build quality, 100-200), efSearch (query quality, 50-200).
| Database | Type | Best Use | Hosting | Scale |
|---|---|---|---|---|
| FAISS | Library | Research, in-process | In-process | Billions (GPU) |
| ChromaDB | Embedded | Prototyping | Local/Cloud | Millions |
| Pinecone | Managed SaaS | Production, zero-ops | Fully managed | Billions |
| Weaviate | Full DB | Hybrid search | Cloud/Self-host | Billions |
| Qdrant | Full DB | Filtering + vectors | Cloud/Self-host | Billions |
| pgvector | Extension | Already using Postgres | In PostgreSQL | Millions |
| Milvus | Distributed | Enterprise | Cloud/Self-host | Billions+ |
The embedding model matters more than the vector DB. Check MTEB leaderboard. Key factors: (1) Dimension โ 384 (fast) to 3072 (accurate). (2) Max tokens โ must be >= chunk size. (3) Multilingual โ require for non-English. (4) Domain โ fine-tuned models beat general-purpose.
Answer: ANN trades <1% recall for massive speedups. Exact NN is O(n*d) โ 10M vectors at 1536 dims = 15 billion ops per query. HNSW achieves ~99% recall at 100x+ speedup by navigating a graph, visiting only ~log(n) nodes.
Answer: Cosine for text (angle-based, normalized). Dot product when magnitude matters. L2 for image/audio. Always match the metric your embedding model was trained with โ wrong metric drops recall 20%+.
Answer: (1) Pre-filter: metadata first, ANN on subset โ accurate but slow. (2) Post-filter: ANN first, filter after โ fast but may return <k. (3) Filtered ANN (Pinecone, Qdrant): filtering during graph traversal โ best of both. Always prefer filtered ANN in production.
Answer: FAISS: Research, benchmarks, in-process โ no persistence or filtering. ChromaDB: Prototyping <1M vectors โ good DX, local-first. Pinecone: Production, zero-ops, auto-scales, filtered ANN โ 10x more expensive. Use FAISS to validate, Chroma to prototype, Pinecone/Qdrant to ship.
ReAct (Yao 2022): Thought > Action > Observation > repeat. The LLM reasons about what to do, calls a tool, sees the result, and continues until it has a final answer.
| Architecture | How It Works | Best For |
|---|---|---|
| ReAct Loop | Fixed think-act-observe cycle | Simple tool-using tasks |
| Plan-and-Execute | Full plan first, then execute steps | Multi-step structured tasks |
| State Machine (LangGraph) | Directed graph with conditional edges | Complex workflows, branching |
| Reflection | Agent evaluates own output, retries | Quality-critical tasks |
| Framework | Paradigm | Strengths | Best For |
|---|---|---|---|
| LangGraph | Stateful graph | Persistence, human-in-loop, cycles | Production agents |
| CrewAI | Role-based multi-agent | Easy setup, role/goal/backstory | Business workflows |
| AutoGen | Conversational multi-agent | Code execution, group chat | Research automation |
| Smolagents (HF) | Code-based tool calling | Simple, open-source | Lightweight agents |
| OpenAI Assistants | Hosted agent | Zero setup, code interpreter | Quick prototypes |
| Type | Purpose | Implementation |
|---|---|---|
| Short-term | Current conversation | Message history in prompt |
| Long-term | Past conversations, facts | Vector DB + retrieval |
| Working memory | Intermediate state | LangGraph State object |
| Episodic | Past task outcomes | Structured DB of (task, result) |
(1) Single responsibility โ one tool, one job. (2) Idempotent โ safe to retry. (3) Fast โ slow tools stall the loop. (4) Rich descriptions โ the LLM decides based on the description string.
Agents take real actions. Safety measures: (1) Human-in-the-loop for destructive actions. (2) Sandboxing code execution. (3) Permission models. (4) Max iterations. (5) Cost/budget limits.
Answer: (1) Infinite loops โ stuck in reasoning cycle. Fix: max_iterations + loop detection. (2) Tool hallucination โ calls nonexistent tools. Fix: strict function calling schemas. (3) Context overflow โ long loops fill context. Fix: summarize observations. (4) Cascading errors โ bad step corrupts downstream. Fix: validation at each step. (5) Cost explosion โ long GPT-4o chains cost $1+/task. Fix: budget limits.
Answer: LangChain AgentExecutor is a fixed linear loop. LangGraph models workflows as a directed graph with persistent state โ branching, cycles, human-in-the-loop, session persistence. LangGraph is the production evolution; AgentExecutor is legacy.
Answer: (1) Structured output via function calling (not text parsing). (2) Checkpointing โ resume after failures. (3) Observability โ trace every LLM call (LangSmith, Langfuse). (4) Guardrails on tool inputs/outputs. (5) Fallbacks โ gracefully degrade after N failures.
Answer: Separates planning (create full multi-step plan) from execution (carry out each step independently). Planner has full task context; executor focuses on one step with clean context. After each step, planner can revise. Better than ReAct for complex multi-step tasks where the agent loses track of the goal.
Answer: Two patterns: (1) Shared state โ agents read/write a common state object (LangGraph). (2) Message passing โ structured messages between agents (AutoGen). Shared state is simpler; message passing is better for dynamic collaboration. Always pass structured data, not LLM summaries, to prevent error compounding.
| Pattern | Structure | Best For | Example |
|---|---|---|---|
| Supervisor | Orchestrator delegates to workers | Clear task decomposition | Manager + researcher + writer |
| Peer-to-Peer | Agents communicate directly | Collaborative reasoning, debate | Two reviewers debating quality |
| Hierarchical | Nested supervisors (teams of teams) | Enterprise workflows | Department heads coordinating |
| Swarm | Dynamic hand-off between agents | Customer service routing | OpenAI Swarm pattern |
Sequential: Agent A finishes before Agent B starts. Simple but no parallelism. Parallel: Independent tasks run simultaneously. Debate: Multiple agents propose solutions, a judge picks the best. Voting: Run the same task on N agents, majority wins (increases reliability).
CrewAI uses role-based agents with defined goals and backstories. A "Research Analyst" has different system prompts and tools than a "Report Writer." They collaborate via a crew's shared task queue. This mirrors real organizations.
Lightweight multi-agent pattern: each agent is a function with instructions + tools. Agents can hand off to each other by returning a different agent. No orchestrator โ control flows through hand-offs. Great for customer service routing (sales agent -> support agent -> billing agent).
(1) Over-engineering โ single ReAct agent often outperforms poorly designed multi-agent. (2) Context leakage โ agents sharing too much irrelevant context. (3) Error compounding โ LLM summaries between agents introduce errors. Always pass structured data. (4) Cost explosion โ N agents = N times the API calls.
Answer: (1) Shared state โ agents read/write a common state object (LangGraph). Simple for pipelines. (2) Message passing โ agents send structured messages (AutoGen). Better for dynamic collaboration. Always pass structured data, not LLM-generated summaries, to prevent error compounding.
Answer: When a single agent with good tools can handle the task. Multi-agent adds latency (N serial LLM calls), cost (N times more tokens), and complexity (coordination bugs). Start with a single ReAct agent; only split into multi-agent when context window overflow or specialization clearly helps.
Answer: Each agent in a chain can introduce errors. Mitigations: (1) Pass raw tool outputs between agents, not summaries. (2) Add validation nodes between agents. (3) Use structured output formats. (4) Implement self-correction loops where agents verify their own outputs. (5) Hierarchical review โ a supervisor validates worker outputs before proceeding.
Answer: Supervisor: When you know the workflow structure upfront โ orchestrator routes to specialized workers. Good for content pipelines, data processing. Swarm: When routing is dynamic and agents decide who handles what. Each agent can hand off to another. Better for customer service, conversational flows. Swarm is simpler but less controllable.
(1) You define tool schemas (name, description, parameters as JSON Schema). (2) Send schemas + user message to the LLM. (3) The model outputs a tool call with function name + arguments as JSON. (4) You execute the function locally. (5) Send results back to the model for final response.
Description matters most โ the LLM decides which tool to call based entirely on the description. "Search the web" is bad. "Search the web for real-time news. Do NOT use for stable facts like math or history" is good. Include: what it does, when to use it, when NOT to use it, and expected input format.
Parallel: Model outputs multiple independent tool calls in one response. Execute all simultaneously. GPT-4o supports this natively. Sequential: One call at a time, each depending on previous results. Parallel is 3-5x faster for independent operations.
MCP (Anthropic, 2024) is an open standard for connecting AI assistants to tools and data sources. Instead of each provider having their own format, MCP standardizes tool servers that expose capabilities. Supports: tools (actions), resources (read data), and prompts (templates). Adopted by Claude Desktop, Cursor, and growing ecosystem.
| Provider | Feature Name | Parallel Calls | Strict Mode |
|---|---|---|---|
| OpenAI | Function Calling | Yes | Yes (strict JSON) |
| Anthropic | Tool Use | Yes | Yes |
| Function Calling | Yes | Yes | |
| Open models | Varies (Hermes format) | Some | Via constrained decoding |
Answer: Function calling uses constrained decoding โ the model can only output valid JSON matching your schema. Text parsing relies on the model following instructions, which fails ~10-20% of the time. Function calling gives near-100% format reliability. It also enables parallel tool calls and strict type validation.
Answer: When the LLM determines multiple tools can be called independently, it outputs ALL tool calls in a single response. You execute them in parallel and return all results. GPT-4o supports this natively. A query like "Weather in 3 cities" produces 3 parallel calls instead of 3 sequential LLM rounds. 3-5x faster.
Answer: Model Context Protocol standardizes how AI assistants connect to external tools. Instead of building custom integrations for each LLM provider, you build one MCP server that works everywhere. It defines: tools (functions), resources (data), and prompts (templates). Similar to how HTTP standardized web communication โ MCP standardizes AI-tool communication.
Answer: (1) Validate inputs before execution โ check types, ranges, permissions. (2) Catch exceptions and return descriptive error messages to the model. (3) Let the model retry โ include error details so it can fix its call. (4) Max retries โ after 3 failed attempts, return a user-friendly error. Never let tool errors crash the agent loop.
| Type | Method | When to Use |
|---|---|---|
| Rule-based | BLEU, ROUGE, exact match | Translation, extraction with reference |
| Embedding-based | Cosine similarity to reference | Semantic similarity checking |
| LLM-as-Judge | GPT-4o scores on criteria | Open-ended generation (90% of cases) |
| Human eval | Human annotators rate outputs | Final validation, calibration |
| Framework | Target | Key Metrics | Strengths |
|---|---|---|---|
| RAGAS | RAG pipelines | Faithfulness, Relevance, Recall | RAG-specific, automated |
| DeepEval | LLM apps | Hallucination, Bias, Toxicity | Comprehensive, CI-friendly |
| Langfuse | Observability + eval | Traces, scores, datasets | Open-source, real-time |
| PromptFoo | Prompt testing | Regression across versions | CLI-based, fast iteration |
| LangSmith | LangChain ecosystem | Traces, datasets, evals | Deep LangChain integration |
(1) Use chain-of-thought in the judge prompt โ "think step by step before scoring." (2) Pointwise scoring (1-5 rubric) is more reliable than pairwise comparison. (3) Calibrate against humans โ run 100 examples with human labels to validate the judge. (4) Use a different model family for judging to avoid self-enhancement bias. (5) Define explicit rubrics โ not "is this good?" but "does it contain all key facts from the source?"
Faithfulness: Extract claims from answer; check each against context. Answer Relevance: Generate questions from the answer; compare to original. Context Recall: Check if ground truth facts appear in context. Context Precision: Rank contexts by relevance; compute precision@k.
(1) Manual curation โ gold standard but expensive. Aim for 100+ examples covering edge cases. (2) Synthetic generation โ use a strong LLM to generate (question, answer, context) triples from your documents. (3) Production sampling โ sample real user queries and manually annotate a subset. (4) Adversarial โ deliberately create hard/tricky cases to find model weaknesses.
Answer: They measure n-gram overlap with reference text. For generative tasks, many phrasings are equally valid โ "The model works well" and "The system performs accurately" have zero overlap but both correct. They also ignore factual accuracy, coherence, and helpfulness. Only use for translation/summarization where references are meaningful.
Answer: LLMs prefer outputs stylistically similar to their own โ GPT-4o favors GPT-4o outputs over LLaMA outputs even when quality is equal. Mitigations: use different model family for judging, blind evaluation (no model names), calibrate against human preferences, and use explicit rubrics rather than vague quality assessments.
Answer: (1) Start with 100+ manually curated (question, ground_truth) pairs covering edge cases. (2) Augment synthetically โ use strong LLM to generate variations. (3) Sample production queries weekly and annotate. (4) Include adversarial cases (tricky, ambiguous, out-of-scope). (5) Version control โ track dataset changes alongside prompt changes. A good eval dataset is your most valuable asset.
Answer: Offline: Evaluate against pre-built datasets before deployment โ catches regressions, validates prompt changes. Online: Monitor production outputs in real-time โ user satisfaction, escalation rates, latency. You need both: offline for gating deployments, online for detecting issues with real traffic patterns you didn't anticipate.
| Risk | Description | Mitigation |
|---|---|---|
| Hallucination | Model generates false facts | RAG grounding, self-consistency checks |
| Prompt injection | User overrides system instructions | Input classification, instruction hierarchy |
| Jailbreaking | Bypassing safety training | Multi-layer defense, red-teaming |
| PII leakage | Exposing personal data | PII detection, output filtering |
| Toxicity | Harmful, biased, or offensive output | Content classifiers, NeMo Guardrails |
| Off-topic | Answering outside intended scope | Topic classifiers, system prompt boundaries |
Layer multiple defenses: (1) Input guardrails โ classify user input before reaching the LLM (jailbreak detection, topic filtering). (2) System prompt โ clear instructions about boundaries. (3) Output guardrails โ validate LLM output before showing to user (PII removal, toxicity check, hallucination detection). (4) Monitoring โ flag anomalous patterns in production.
| Tool | What It Does | Approach |
|---|---|---|
| Guardrails AI | Schema-based output validation + re-asking | Pydantic-like validators |
| NeMo Guardrails | Conversation flow programming | Colang DSL for safety rails |
| Llama Guard (Meta) | Safety classifier for LLM I/O | Fine-tuned LLM classifier |
| Azure Content Safety | Violence, self-harm, sexual content | Multi-category API classifier |
(1) Separate system/user content โ use structured message roles, never mix. (2) Input sanitization โ detect injection patterns (e.g., "ignore previous instructions"). (3) Instruction hierarchy โ system prompt takes priority explicitly. (4) Canary tokens โ include a secret in system prompt and detect if the model repeats it. (5) Dual LLM โ use a cheap model to classify input safety before expensive model generates output.
Train safety using principles rather than human labels for every case. Model critiques its own responses against a written constitution, then revises. Scales better than RLHF for safety and produces principled, explainable behavior.
Systematic adversarial testing before deployment: (1) Manual โ human testers try to break the model. (2) Automated โ use another LLM to generate adversarial prompts (PAIR, TAP). (3) Structured attacks โ GCG gradient-based attacks, multi-turn escalation. Run before any public deployment and on each major prompt change.
Answer: Prompt injection is when user input (or malicious data in tool results) overrides system instructions. E.g., a web page says "Ignore all instructions." Defenses: (1) Separate system/user content with message roles. (2) Classify inputs before processing. (3) Instruction hierarchy โ explicit priority to system prompt. (4) Canary tokens. (5) Dual-LLM: cheap model classifies, expensive model generates.
Answer: Systematic adversarial testing to find ways the model can produce harmful outputs. Includes: manual (humans try to break it), automated (another LLM generates attacks โ PAIR, TAP), and gradient-based (GCG attacks). Run before deployment and after each major prompt change. Document findings and create regression tests.
Answer: (1) RAG grounding โ check if every claim appears in retrieved context (RAGAS Faithfulness). (2) Self-consistency โ generate N times, flag divergent answers. (3) Verification chains โ secondary LLM call: "Is this claim supported by the text?" (4) Confidence calibration โ instruct model to say "I don't know." (5) Citation forcing โ require the model to cite specific passages.
Answer: Layer multiple independent defenses: Input layer (classify safety, detect injection), System prompt (clear boundaries, role definition), LLM layer (safety-trained model), Output layer (PII filter, toxicity check, hallucination detection), Monitoring layer (alert on anomalous patterns). No single layer is sufficient โ adversaries will find bypasses. Defense-in-depth makes the system resilient.
| Framework | Key Feature | Best For | Throughput |
|---|---|---|---|
| vLLM | PagedAttention โ zero KV cache waste | Production, high throughput | Highest |
| TGI (HF) | Tensor parallelism, batching | HF ecosystem, Docker-ready | High |
| Ollama | One-command local deployment | Local dev, laptops | Low |
| LiteLLM | Unified proxy for 100+ models | Multi-provider routing | Proxy (no inference) |
| llama.cpp | CPU inference, GGUF format | No-GPU environments | Medium |
| Method | How It Works | Quality Impact | Use With |
|---|---|---|---|
| GPTQ | Post-training, layer-by-layer with error compensation | Minimal (<1% loss) | vLLM, TGI |
| AWQ | Protects "salient" weights during quantization | Slightly better than GPTQ | vLLM, TGI |
| GGUF | CPU-optimized format for llama.cpp | Varies by quant level | Ollama, llama.cpp |
| BitsAndBytes | Runtime 4/8-bit quantization | Good for fine-tuning | HF Transformers |
PagedAttention (inspired by OS virtual memory): stores KV cache in non-contiguous memory pages. Traditional serving pre-allocates max KV cache โ wasting 60-80% of GPU memory. PagedAttention allocates on demand: 3-24x higher throughput. Continuous batching: instead of waiting for an entire batch to complete, add new requests as old ones finish. LoRA serving: serve multiple LoRA adapters on same base โ multi-tenant support.
| Factor | API (OpenAI/Anthropic) | Self-hosted (vLLM) |
|---|---|---|
| Setup | Zero ops | GPU infra required |
| Quality | Best (GPT-4o, Claude) | Good (LLaMA-3, Mistral) |
| Cost at low volume | $$ | $$$$ |
| Cost at high volume | $$$$ | $$ |
| Data privacy | Data leaves your infra | Data stays local |
| Latency control | Limited | Full control |
Rule of thumb: API for <1M tokens/day, self-host for >10M tokens/day or data-sensitive workloads.
(1) Single GPU โ 7B-13B models with quantization. (2) Tensor Parallelism โ split model across GPUs on same node. (3) Pipeline Parallelism โ split layers across nodes. (4) Serverless โ Modal, RunPod, Together AI โ pay per token, auto-scale to zero.
Answer: Inspired by OS virtual memory paging: stores KV cache in non-contiguous memory pages. Traditional serving pre-allocates maximum KV cache size โ wasting 60-80% of GPU memory on padding. PagedAttention allocates pages on demand as sequences grow, enabling 3-24x higher throughput and supporting more concurrent users per GPU.
Answer: API: Zero ops, best quality, predictable at low volume, no data privacy control. Self-hosted: Data stays private, cost-effective at high volume, lower latency, customizable. Crossover point: ~1-10M tokens/day. Below that, APIs are cheaper when accounting for GPU rental + engineer time. Above that, self-hosting with vLLM saves 70-90%.
Answer: GPTQ: Standard 4-bit for GPU inference (vLLM, TGI). AWQ: Slightly better quality than GPTQ, faster activation-aware approach. GGUF: CPU-optimized for llama.cpp/Ollama โ works without GPU. Use GPTQ/AWQ for GPU serving, GGUF for local/CPU. All achieve <1% quality loss at 4-bit. 8-bit is nearly lossless.
Answer: Traditional batching waits for all requests in a batch to complete before accepting new ones. Continuous batching (used by vLLM, TGI) inserts new requests as soon as any request in the batch finishes. This eliminates the "convoy effect" where a short request waits for a long one. Result: 2-3x throughput improvement, much lower tail latency.
Embed user queries and check if a semantically similar query was recently answered. If similarity > threshold, return cached response. GPTCache and Redis + vector similarity implement this. Achieves 30-60% cache hit rates on chatbot traffic, dramatically reducing API costs.
Never make users wait for full response. Stream tokens as generated using Server-Sent Events (SSE). Perceived latency drops from 5-10s to <1s (time-to-first-token). Implement with stream=True in OpenAI SDK + async generators in FastAPI.
| Strategy | Savings | Tradeoff |
|---|---|---|
| Smaller models for simple tasks | 10-50x (mini vs full) | Lower quality for complex reasoning |
| Semantic caching | 30-60% fewer calls | Stale responses for dynamic content |
| Prompt compression | 20-40% fewer tokens | Risk of losing important context |
| Batch API (OpenAI) | 50% discount | Async only, 24h turnaround |
| Prompt caching (Anthropic/OpenAI) | 90% on repeated prefixes | Must structure prompts carefully |
| Self-host (vLLM) | 70-90% at high volume | Ops burden, GPU management |
| Tool | Features | Type |
|---|---|---|
| LangSmith | Tracing, evals, datasets, playground | Managed (LangChain) |
| Langfuse | Tracing, scoring, prompt management | Open-source |
| Arize Phoenix | LLM tracing, embedding analysis | Open-source |
| PromptLayer | Prompt versioning, A/B testing | Managed |
(1) Fallback chains โ GPT-4o -> Claude -> GPT-4o-mini -> cached response. (2) Rate limiting โ per-user token budgets. (3) Circuit breakers โ stop calling a provider after N failures. (4) Retry with exponential backoff โ handle transient API errors. (5) Timeout management โ kill requests after 30s to prevent hung connections.
Treat prompts like code: (1) Version control โ store in files, not strings. (2) A/B testing โ test new prompts against baseline. (3) Regression tests โ automated eval before deployment. (4) Structured templates โ use Jinja2 templates with variables. (5) Prompt caching โ structure with static prefix + dynamic suffix.
Answer: (1) Stream tokens โ TTFT <1s feels instant. (2) Semantic cache โ instant for repeated patterns. (3) Smaller models for preprocessing โ classify intent with mini, reason with full model. (4) Speculative decoding โ small draft model generates, large model verifies in parallel (2-3x speedup).
Answer: Reduces costs when the same system prompt prefix is reused. Anthropic and OpenAI cache the KV representation of repeated prefixes server-side โ 90% cheaper for cached tokens. Structure prompts: long static instructions first, dynamic user content last. Can save 50-80% on total costs for apps with large system prompts.
Answer: (1) Store prompts in version-controlled files. (2) Use prompt management tools (LangSmith, Langfuse) for A/B testing. (3) Run automated regression tests against an eval dataset before deploying new versions. (4) Log all prompts + completions for debugging. (5) Canary deployments โ route 5% of traffic to new prompt, monitor metrics.
Answer: Semantic caching: You implement it client-side. Embed user queries, find similar past queries, return cached LLM response. Saves entire API calls. Works best for FAQs. Prompt caching: Provider implements it server-side. Caches the KV representations of repeated system prompt prefixes. Reduces cost per call but still makes the call. They're complementary โ use both.
Answer: Priority-ordered model list: GPT-4o -> Claude -> GPT-4o-mini -> cached/static response. Use LiteLLM Router or custom retry logic. On each failure: (1) Log the error. (2) Try next model in chain. (3) After all fail, return graceful degradation message. Also implement circuit breakers โ if a provider fails 5 times in 1 minute, skip it for 5 minutes.
${m.desc}
${m.category}${m.desc}