advanced
14 min read
A2A Explained: How Agent-to-Agent Communication Differs from MCP
MCP answers "what can I call?" A2A answers "who can I hand this to?" Different questions, different failure modes.
Read tutorialpractice
Evaluation and Testing
Methods for measuring AI-system behavior, validating claims, detecting regressions, and turning observed failures into actionable evidence.
Tagged articles
Evaluation and Testing articles
63 articles in this tag.
advanced
17 min read
Advanced Retrieval Patterns: Multi-Query, Multi-Vector, Parent-Document, and Diversified Retrieval
Your recall metric looks healthy. Your answers are still wrong. That gap is the whole reason advanced retrieval patterns exist.
Read tutorial
advanced
17 min read
Agent Execution Environments: Filesystems, Shells, Browsers, Code, Databases, and Sandboxes
The demo worked because the agent was running on your laptop. It had your credentials, your network, your home directory, and your shell history. Move it…
Read tutorial
advanced
18 min read
Agent Improvement Loops: Traces, Failure Taxonomies, Regression Suites, and Hill Climbing
A prompt fix that passes the demo and fails two weeks later is not a bad prompt. It is an improvement process with no memory.
Read tutorial
advanced
11 min read
Agent Loop Pathologies: Repetition, Oscillation, Goal Drift, Plateaus, and Regression
An agent that has run forty steps and produced an answer no better than step three is not stuck. It is broken in a specific, nameable way.
Read tutorial
advanced
15 min read
Agent Observability and Artifacts: Traces, State Snapshots, Lineage, Cost, and Debugging
A user reports a wrong answer. You pull the trace. Every tool call succeeded, every span is green, latency is normal, and no error was logged anywhere. The…
Read tutorial
advanced
17 min read
Agent Skills and Procedural Capabilities: Prompts, Scripts, Assets, and Progressive Disclosure
Every agent builder hits the same wall. The first working agent is small: a system prompt, a few tools, a loop. Then someone needs release notes. Someone…
Read tutorial
advanced
15 min read
The Anatomy of an Agent Loop: State, Action, Observation, Feedback, and Termination
The demo works. The agent books the flight, summarizes the repo, files the ticket. Then you point it at a task with one extra branch, and it calls the same…
Read tutorial
intermediate
10 min read
Ask Before Answering: Self-Ask, Clarification, and Assumption Discovery
A user types three words into your assistant. The model returns a confident, well-formatted, entirely wrong answer to a question nobody asked. The user…
Read tutorial
advanced
17 min read
Chroma Context-1: What Is a Self-Editing Search Agent?
Hop three. The context window is full. Half of it is plausible garbage — passages that matched the query, ranked well, and turned out to be useless. The…
Read tutorial
intermediate
12 min read
Chunking and Structural Context: How Document Boundaries Change Retrieval
A retrieval system can only return what a chunk contains. Cut the answer in half, and no reranker will put it back together.
Read tutorial
advanced
11 min read
Codex Security Explained: Can an AI Agent Find, Validate, and Patch Vulnerabilities?
A scanner that flags ten thousand findings and a researcher who proves three are both "finding vulnerabilities." Only one of them has evidence you can…
Read tutorial
advanced
17 min read
Coding Agent Harnesses: Workspaces, Shells, Artifacts, Tests, and Long-Horizon State
The agent writes the function, re-reads it, decides it looks correct, and stops. The code is wrong. This is the most common failure in autonomous coding,…
Read tutorial
advanced
16 min read
Context Evaluation: Precision, Recall, Faithfulness, Citation Coverage, and Ablation
A RAG feature ships. The answers read well. Someone wires up a faithfulness evaluator, the number lands at 0.87, and within a week that single scalar is…
Read tutorial
advanced
16 min read
Context Windows and Attention Budgets: Signal, Noise, Relevance, and Token Tradeoffs
The request fits. It returns a worse answer than the shorter prompt you shipped last week.
Read tutorial
advanced
14 min read
Corrective Retrieval Loops: CRAG, Self-RAG, Query Rewriting, and Evidence Sufficiency
A vector search returns its nearest neighbors whether or not those neighbors answer the question. The pipeline reports no error, the generator receives…
Read tutorial
advanced
16 min read
Decomposition and Planning Prompts: Least-to-Most, Plan-and-Solve, and Self-Discover
The first sub-answer is wrong, and every later step inherits it with full confidence.
Read tutorial
advanced
15 min read
Designing a Deep Research Agent: Search, Evidence, Verification, and Bounded Revision
You have seen the failure. A research agent returns a polished report, the citations look real, and the first URL you check resolves to a page that does…
Read tutorial
advanced
13 min read
Designing a Workspace Operations Agent: Tools, State, Persistence, and Human Approval
The demo works. The agent reads the queue, drafts the update, and posts it. Then someone asks who approved the change, which run produced it, and how to…
Read tutorial
advanced
14 min read
Designing an Adaptive Knowledge Agent: Routing, Retrieval, Memory, and Context Assembly
The demo works on turn one. By turn five, the agent is answering a question you already answered, citing a document it retrieved three turns ago, and…
Read tutorial
advanced
12 min read
Embeddings, Vector Indexes, and Hybrid Search: Choosing a Retrieval Substrate
A retrieval system fails in a specific, repeatable way. You ship a demo that answers paraphrased questions beautifully, then a user pastes an error code, a…
Read tutorial
advanced
15 min read
Evaluator Design and Feedback Contracts: Graders, Judges, Disagreement, and Targeted Retry
An evaluator is only as useful as the set of distinct repair actions its output can select.
Read tutorial
advanced
14 min read
Few-Shot Example Engineering: Selection, Ordering, Diversity, and Boundary Cases
Your few-shot set passes every happy-path test, then folds the moment a real input arrives with an extra clause, an unusual label, or a phrasing you did…
Read tutorial
advanced
13 min read
How Hermes Agent Improves Itself: DSPy, GEPA, Skills, and Evolutionary Search
"Self-improving agent" is three different mechanisms wearing one coat. Pull the coat off and you find a memory that accumulates, a skill file that gets…
Read tutorial
advanced
12 min read
Is RAG Dead? What Actually Died—and What Replaced Naive RAG
The demo worked. Then the corpus grew, the questions started requiring synthesis across documents, and the same pipeline that felt magical last month now…
Read tutorial
advanced
10 min read
Jev and Laya Explained: How Non-Autoregressive “System One” Decision Models Work
You have a routing step in an agent pipeline. Four possible destinations. The LLM call costs a full generation, 1–2 seconds, and a token bill. The answer…
Read tutorial
advanced
13 min read
Andrej Karpathy’s LLM Wiki: A File-Based Alternative to Re-Reading Knowledge with RAG
Ask a system to synthesize five documents into one answer. It does the work. Ask the same question tomorrow, and it does the work again, from scratch, with…
Read tutorial
advanced
16 min read
Long-Horizon Context Management: Trimming, Summarization, Compaction, and Progressive Disclosure
The agent ran clean for forty steps. Then it re-ran a migration it had already finished, forgot the constraint the user stated at turn three, and quietly…
Read tutorial
advanced
16 min read
Long-Running Agent Loops: Checkpoints, Budgets, Progress Signals, and Recovery
An agent processes 200 units of work over four hours, dies on unit 201, and restarts from zero. The crash is not the failure. The failure is that nothing…
Read tutorial
advanced
19 min read
Memory Engineering for AI Agents: Short-Term, Semantic, Episodic, and Procedural Memory
The agent remembered everything and still got the user wrong. It recalled a preference the user had abandoned six months ago, re-derived a workflow it had…
Read tutorial
advanced
13 min read
Operational Loop Patterns: Turn-Based, Goal-Based, Time-Based, and Proactive Agents
A goal-based agent that never terminates is not a model failure. It is a contract failure.
Read tutorial
advanced
9 min read
Pi and the Minimal Agent Harness: Can Four Tools Be Enough?
Your agent has thirty tools, four MCP servers, and a plan-mode toggle. It still fumbles a routine refactor, and when it does, you cannot tell whether the…
Read tutorial
advanced
18 min read
Planning and Replanning Loops: Plan-and-Execute, Progress State, and Plan Repair
An agent that plans beautifully and then executes a stale plan into a wall is not a planning failure. It is a state failure.
Read tutorial
advanced
15 min read
Planning and Scheduling Harnesses: Task Graphs, Dependencies, ReWOO, and LLMCompiler
A plan that reads like a clean numbered list can still deadlock at runtime, because the list never told the scheduler which steps actually depend on which…
Read tutorial
advanced
13 min read
Prompt Chaining: Designing Multi-Stage Cognitive Workflows
The demo works. You paste a gnarly task into one prompt—extract the entities, reconcile them against the schema, flag the contradictions, write the…
Read tutorial
advanced
15 min read
Prompt Contracts: Objectives, Inputs, Constraints, Outputs, and Acceptance Criteria
A prompt that passed review, shipped, and then quietly broke a downstream consumer is not a prompt problem. It is a specification problem wearing a prompt…
Read tutorial
intermediate
13 min read
Prompt Engineering as Interface Design: From Vague Requests to Explicit Behavior
The prompt worked on your laptop. In production, the same prompt returns a paragraph on Monday, a bulleted list on Tuesday, and a confident fabrication on…
Read tutorial
advanced
12 min read
Prompt Ensembling and Candidate Selection: Self-Consistency, Best-of-N, and Consensus
A single greedy decode is right most of the time and wrong with total confidence the rest. The instinct is to sample more and let the answers vote. But…
Read tutorial
advanced
16 min read
Prompt Evaluation: Test Cases, Graders, Variance, and Regression Testing
A prompt is not a sentence you tune. It is a versioned artifact whose behavior is a distribution, and you cannot manage a distribution by reading three…
Read tutorial
advanced
11 min read
Python Threads, the GIL, and Free-Threaded CPython
A thread pool that scales on network calls and flatlines on arithmetic is not a mystery. It is the Global Interpreter Lock doing exactly what it was…
Read tutorial
advanced
14 min read
Query Engineering for Retrieval: Analysis, Decomposition, Rewriting, and Filters
Watch a real session. A user types: "Compare the latency and cost of the two reranking options we discussed, for our EU deployment." Your hybrid retriever…
Read tutorial
advanced
16 min read
ReAct Agent Loops: Tool Use Through Thought, Action, and Observation
An agent that acts without reasoning is not reasoning. It is guessing with a tool schema.
Read tutorial
advanced
11 min read
Reasoning Scaffolds: Scratchpads, Tables, Decision Matrices, and Code
The model gets the answer wrong, and it is not because it did not know something. It dropped a constraint. It merged two cases that should have stayed…
Read tutorial
advanced
10 min read
Recursive Self-Improvement in AI: What Anthropic Means by “When AI Builds Itself”
A coding agent is handed a fixed training script and a fixed success metric: make it run faster without breaking correctness. It rewrites, runs, times, and…
Read tutorial
advanced
15 min read
Reflection and Self-Refine: Building Generator-Critic Revision Loops
You add a critique-and-revise step. The output gets longer, more confident, more padded with caveats. Your eval score does not move.
Read tutorial
advanced
13 min read
Reflection vs Reflexion: Improving the Current Attempt vs Learning from Prior Attempts
A team adds a "reflect" step to their agent. Scores tick up a few points. Encouraged, they bolt on a memory buffer so the agent can "remember its…
Read tutorial
advanced
12 min read
Reframing and Attention Prompting: Step-Back, Rephrase-and-Respond, and System 2 Attention
That failure breaks the "bad answer means bad instruction" reflex. Sometimes the instruction is fine, and the bottleneck sits somewhere else: the model…
Read tutorial
advanced
17 min read
Reranking and Contextual Compression: Improving Evidence Before Generation
The retriever found the right document. The answer is still wrong. Somewhere between the vector store and the prompt, the evidence got buried under five…
Read tutorial
intermediate
11 min read
Role, Context, Style, and Negative Constraints: When Standard Prompting Techniques Matter
A prompt that grew from three lines to thirty is not a better prompt. It is a prompt nobody has tested.
Read tutorial
advanced
15 min read
Role Isolation, Approvals, and Independent Validation in Agent Runtimes
That is the shape of the failure I keep seeing in agent systems that look governed and are not. A team writes a planner prompt, a coder prompt, a reviewer…
Read tutorial
advanced
13 min read
Search and Tree Loops: Branching, Backtracking, Beam Search, MCTS, and LATS
A linear agent loop is a commitment device. It commits to one trajectory, and every later step inherits the consequences of the earliest mistake.
Read tutorial
advanced
15 min read
Self-Critique and Verification Prompts: Critique, Revision, Rubrics, and Chain-of-Verification
You ask the model to critique its own draft. It returns: "The response is clear and well-structured, but could be more specific." You ask it to revise. The…
Read tutorial
advanced
15 min read
Self-Harness: Can AI Agents Improve Their Own Harnesses?
The loop that improves the harness can also edit the evaluator, the permissions, and the rollback path. That is the whole problem in one sentence.
Read tutorial
advanced
14 min read
Source Routing and Adaptive RAG: Choosing Where and How to Retrieve
Always-on retrieval is a fixed policy applied to a variable question distribution. It over-retrieves on questions the model already knew, and it…
Read tutorial
advanced
13 min read
Thin vs Thick Agent Harnesses: How Much Scaffolding Does a Model Need?
Two teams run the same model on the same task. One ships reliably. The other drowns in tool-call latency, context bloat, and incidents nobody can explain…
Read tutorial
advanced
16 min read
Tool Engineering for AI Agents: Schemas, Descriptions, Contracts, Errors, and Idempotency
A failed agent run rarely looks like an interface problem. It looks like a model problem: wrong tool, malformed arguments, an error swallowed and retried…
Read tutorial
advanced
12 min read
Validation vs Verification vs Evaluation in AI Systems
The release gate is green. Schema validation passed, the golden set passed, the benchmark score ticked up, and the guardrails logged zero trips. Two days…
Read tutorial
advanced
10 min read
What Is an Agent Harness? The Software Layer Around an LLM
Two teams ship the same model, the same prompt, the same tool list. One agent finishes a multi-step task. The other stalls, loops, and quietly corrupts its…
Read tutorial
advanced
10 min read
What Is an AI-Native Startup? Lessons from Anthropic’s Founder’s Playbook
Two seed-stage teams ship an LLM-backed product. Same headcount, same model access, same demo polish. Eighteen months later, one compounds while the other…
Read tutorial
intermediate
13 min read
What Is Context Engineering? Building the Right Working Context for Every Model Decision
The prompt you spent a week tuning is not what the model reads. It reads everything in the window — and everything else is quietly voting.
Read tutorial
advanced
14 min read
What Is Graph Engineering for AI Agents? From Prompt, Context, Harness, and Loop Engineering to System Intelligence
The demo works. Then you add a second worker, a shared artifact store, and a verifier, and the system starts producing results nobody can attribute,…
Read tutorial
advanced
14 min read
Where Prompt Engineering Ends: Context, Harnesses, and Loops
A prompt is a spark, not an engine. The spark still matters — it just cannot carry the machine.
Read tutorial
intermediate
14 min read
Zero-Shot, One-Shot, and Few-Shot Prompting: What Examples Actually Change
The demo works. The real inputs don't. Same prompt, same model, and suddenly the output is the wrong shape, the wrong label, or confidently on the wrong…
Read tutorial