AI Agent Production Checklist

Audit how this project manages the agent's context window. Is there any measurement of how much context is used during a session, and any mechanism that compacts or summarizes history before hitting a threshold (~40 %)? Or does it just grow until the provider cuts it off? List where context accumulates uncontrolled and propose a concrete point to measure usage and trigger compaction.

Audit where this agent's long-lived state lives. Is data that doesn't fit in context (long history, project files, prior results) stored on disk or in a database and queried by the agent when needed? Or is everything crammed into the prompt? List what information is dragged through context when it could live outside, and propose how to externalize it and retrieve it on demand.

Audit how subagents are launched in this project. When the orchestrator delegates a task, does the subagent receive only the minimal context it needs for that specific task, or does it inherit the orchestrator's entire history? List where work is delegated and, for each, point out what context is being over-passed and propose the minimal subset it should receive.

Audit what happens to long tool responses in this agent. When a tool returns extensive output (a log dump, an API response, file contents), is that text summarized or dropped from history once used, or does it stay in full forever? List the tools that return large responses and propose, for each, how to summarize or purge their output after use.

Audit whether this project has an explicit strategy for long agent sessions. Is there compaction, external notes, checkpoints, or any deliberate plan for tasks that run for hours, or is it assumed that "the window is big enough"? Describe what would happen in a multi-hour session with the current design and propose a concrete context management strategy for those cases.

Audit the quality of the context this agent receives. Is the system prompt and the data passed to it curated —just enough, high-signal and relevant— or is documentation, files and examples dumped in "just in case"? List which parts of the context add little or nothing to the agent's typical task and propose what to cut to raise the signal-to-noise ratio.

Audit this agent's tool definitions. For each tool, do the name and description make it unambiguous what it does, when to use it and what parameters it expects? List tools with generic names, vague descriptions or overlapping purpose that a model could confuse, and propose a better name and description for each.

Audit this agent's tool inventory. Are there tools that overlap in purpose or are almost never used? Review, if possible, which tools are actually invoked and which aren't. List the candidates to remove or merge and, for each, explain why it's redundant and which tool would cover its function.

Audit the error messages returned by this agent's tools. When a tool fails, does the error explain what happened and what to do to fix it, or does it return a raw traceback or a context-free code? List the tools whose errors aren't actionable and, for each, propose an error message an LLM can read and use to correct itself.

Audit this agent's dangerous tools: the ones that write, delete, send or spend money. Are they behind a confirmation mechanism, a permission system or a dry-run, or does the agent execute them directly? List all tools with irreversible effects and, for each, state whether it has a brake and propose one if it doesn't.

Audit whether this project has measured the effect of the number of tools on agent performance. Has removing tools and comparing results been tested, or is it assumed that more tools is better? Identify tools that are candidates for removal and propose a concrete experiment: what to remove, what metric to observe and how to compare before and after.

Audit whether this agent can run code to solve tasks not anticipated by custom tools. Does it have access to an execution environment (a sandbox, an interpreter) and is that environment isolated from the real system? State whether the capability exists, whether the isolation is safe, and propose how to add or harden code execution in a safe environment.

Audit this project. When the AI agent completes a task, is there a mechanism that objectively verifies the work is correct —tests that actually run, build checks, output validation— or does the agent simply declare it's finished? List the areas considered complete without automated verification and, for each, propose a concrete check and where to add it.

Audit this project's deterministic sensors. Are there linters, type-checks and tests that run fast and automatically before a change is integrated? Are they wired into the agent's flow? List what deterministic checks are missing or don't run before integration and propose which to add and at what point in the flow to trigger them.

Audit the messages produced by this project's sensors (linters, type-checks, tests). When they fail, is the message written so an LLM understands it and self-corrects —stating which rule is broken and how to fix it— or is it cryptic? List the sensors whose output isn't fit for an agent to consume and propose how to improve their messages.

Audit where the agent's guidance lives in this project: CLAUDE.md, coding conventions, architecture docs, project rules. Is it versioned in the repository and up to date, or does it live in someone's head, a chat or outdated? List what knowledge the agent should have that isn't in the repo and propose what to document and where.

Audit where there's review —human or another agent— in this project. Identify the points where a silent agent failure would be expensive (production changes, migrations, customer communication, spending). For each, state whether there's review before the effect is irreversible and, if not, propose what kind of review to add.

Audit how quality checks are staggered over time in this project. Is there a distinction between cheap fast checks (linters, type-checks, unit tests) that run before integration and costly slow ones (e2e, integration, evals) that run later in the pipeline? List mislocated checks —costly ones running too early or cheap ones too late— and propose where each should go.

Audit this agent for the "lethal trifecta": access to private data, exposure to untrusted content (web, emails, third-party documents) and the ability to communicate externally. Do all three coincide in any flow? For each flow where they meet, explain the exfiltration risk and propose which of the three elements to break to close the combination.

Audit what identity and permissions this agent acts with. Does it operate with the identity of the specific user who made the request and only with their permissions, or does it use a service account with full access to all data? List the points where the agent accesses resources with more privileges than the user would have and propose how to restrict it to the correct identity.

Audit how this agent treats untrusted content: web pages, emails, third-party documents, user input. Is that content treated as data to process, or can it end up interpreted as trusted instructions by the model? List the points where external content enters the prompt without clear separation and propose how to delimit it and mark it as untrusted.

Audit where the code this agent generates or launches executes. Does it run in an isolated sandbox —ephemeral container, microVM— with no access to the real system, or in the same environment as the service? State what a malicious script generated by the agent would have access to under the current design and propose how to isolate execution.

Audit the credentials this agent uses to access external resources (databases, APIs, services). For each, does it have only the permissions the agent actually needs —read-only where it only reads— or are they broad credentials for convenience? List the over-permissioned credentials and propose the minimal permission set for each case.

Audit the AI-generated code in this project for typical security flaws: hardcoded secrets or credentials, endpoints or routes without authentication, database queries without limits or pagination, and injection via input concatenation. List each finding with its file and line, explain the risk and propose the concrete fix.

Audit what this agent logs for each run. Are reasoning traces, tool calls with their arguments and results, and intermediate decisions recorded —or only the final result? List what parts of the agent's journey go unrecorded and propose what to log and in what format to be able to debug a run afterwards.

Audit how this agent's success is measured. Does observability distinguish "ran without error" from "made the right decision", or are only exceptions and crashes monitored? State whether there's any way to know if the agent gets it right when it technically "works" and propose how to measure decision correctness, not just absence of error.

Audit whether this project can reconstruct, days later, what the agent did in a specific run. Are traces, input, output and the prompt and model versions of each run kept, with enough retention? State what's missing to investigate an incident reported a week later and propose how to close those gaps.

Audit this project's cost visibility. Can you tell how much each action, task or agent costs —in tokens and money— or do you only see the provider's total bill at month-end? State what cost granularity is missing and propose how to instrument per-run cost to be able to attribute spend.

Audit whether this project can detect gradual quality degradation of the agent —not hard failures, but slow drift. Is there any quality metric tracked over time that reveals a downward trend? State what's monitored today and what isn't, and propose what metrics to track and how to detect drift before it gets severe.

Audit this project's alerting. Are there automatic alerts when cost or error rate goes outside the normal range, or are metrics only looked at when someone decides to open the dashboard? State what conditions should trigger an alert, whether they exist today, and propose what alerts to add and with what thresholds.

Audit whether this project has its own eval set that decides whether the agent is ready to deploy. Is there a set of representative cases with success criteria, runnable repeatedly? State whether today's deploy decision is based on data or intuition, and propose how to build an initial eval set from the agent's real tasks.

Audit whether this project does error analysis: looking at real agent traces one by one, labeling failures and grouping them to find systematic failure modes. State whether that process exists or only aggregate metrics are reviewed, and propose how to start an error analysis: what sample of traces to review and how to categorize the failures.

Audit how this agent is evaluated. Does evaluation look only at whether the final answer is correct, or also at the trajectory —the sequence of decisions, tool calls and intermediate steps? State whether an agent that gets it right via a bad path would pass evaluation today and propose how to also evaluate process quality, not just the result.

Audit how prompt, model or tool changes are handled in this project. Is the eval set run before and after each change to compare impact, or are changes made and shipped by eye? State which recent changes were made without measurement and propose a process to evaluate before and after touching prompt, model or tool.

Audit the resilience of this agent's long tasks. Can it resume from a checkpoint if it crashes mid-task, instead of starting over? Does it have explicit stop conditions —step limit, time limit, cost limit— so it doesn't loop forever? State what's missing of the two and propose how to add checkpointing and stop limits.

Audit this project's rollback capability. If a new version of the prompt or model makes the agent's behavior worse, can you revert to the previous version quickly and in a controlled way? State whether prompts and model version are versioned and independently deployable, and propose how to enable fast rollback.