AI agents in production: what to set up before go-live

What an AI agent is

A language model on its own only produces text. An agent is a model that can also call tools, such as an API, a database query or an email function.

Anthropic separates two designs. In a workflow, "LLMs and tools are orchestrated through predefined code paths". In an agent, "LLMs dynamically direct their own processes and tool usage". The model decides the next step.

For client tools in the Claude API, the model returns a structured tool call. Your application runs the operation and returns the result. Enforce limits, approvals and logging in that handler. Server tools run on Anthropic's infrastructure. They need separate access and usage controls.

Anthropic advises "finding the simplest solution possible, and only increasing complexity when needed". If a fixed workflow does the job, use the workflow.

Start with one defined task

Write the task in one sentence. Say what goes in, what comes out and what counts as done. "Answer questions about our company" is too wide. "Look up the status of an order and reply to the customer with it" is a task.

Write down what the agent must never do. This list later becomes your tool limits and your test cases.

Set a stopping condition. Anthropic notes that it is common to include "stopping conditions (such as a maximum number of iterations) to maintain control". Without one, an agent can loop and keep calling the model and its tools.

Limit the rights of every tool

The OWASP Top 10 for LLM Applications (2025) lists Excessive Agency as LLM06. It names three causes: excessive functionality, excessive permissions and excessive autonomy.

Each cause has a plain countermeasure:

• Functionality: give the agent only the tools the task needs. OWASP suggests narrow tools instead of open ones, for example no tool that runs arbitrary shell commands.
• Permissions: use the least privilege. OWASP gives the example of a read-only database identity for an agent that only reads.
• Autonomy: require a person to approve high-impact actions before they run. Deleting data, sending money and sending email to customers are typical cases.
• Enforce rights in the downstream system, not in the prompt. OWASP calls this complete mediation: the target system checks authorization, not the model.

{
  "name": "get_order_status",
  "description": "Look up the status of one order by its ID. Read-only.",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" }
    },
    "required": ["order_id"]
  }
}

Plan for prompt injection

OWASP lists Prompt Injection as LLM01. In indirect prompt injection, the model reads outside content, such as a web page, a file or an email, and hidden instructions in it change what the model does.

An agent that reads emails and can send emails is exposed to this. Treat everything the agent reads as untrusted. OWASP advises to restrict privileges, require human approval for privileged operations and mark untrusted content clearly. It also advises to treat the model as an untrusted user when you test.

OpenAI makes the same point in its agent safety guide: "agents won't be perfect", so limit access with that in mind. It also advises to turn on tool approvals so that people can review each operation.

Test before every change

Change the prompt, the model or a tool, and the behaviour can change. You need test cases that run before each change goes live. These are often called evaluations, or evals.

OpenAI describes a trace as "the end-to-end record of model calls, tool calls, guardrails, and handoffs for one run". Its docs suggest to use traces to find problems, then move to datasets and repeatable eval runs "once you know what good looks like". They list checks such as whether the right tool was chosen and whether a change helped.

Anthropic recommends "extensive testing in sandboxed environments, along with the appropriate guardrails". Do not test an agent against live systems first.

A first set can be small. Take 20 to 50 real examples of the task. Add the cases from your "must never do" list. Add cases where the input holds hidden instructions. Run all of them after every change and keep the results.

Log, track cost, and plan the fallback

Log each run: the input, each model call, each tool call with its arguments and result, and the final output. OWASP notes that logging and monitoring "will not prevent Excessive Agency, but can limit the level of damage caused". They let you see what happened and fix it.

Logs can hold personal data from the inputs. Decide who may read them, how long you keep them and what you remove before storing. Write these choices down.

Track cost per task. For each Claude API response, record the token counts separately: uncached input, cache writes, cache reads and output. Apply the model's price to each category. Include any tool charges. Set limits. OWASP lists Unbounded Consumption as LLM10 and names rate limits, quotas, timeouts and queue limits as controls.

Plan the fallback to a person. Decide when the agent hands over: when it fails, when it reaches the stopping condition, or when a person rejects an action. Anthropic describes agents that "pause for human feedback at checkpoints or when encountering blockers". Make sure the task and the log of what the agent already did go to the person, so they do not start from zero.

The order of work

Check the controls at each step before moving to the next. Do not move to the next step until the current one works.

• Write the task, the "must never do" list and the stopping condition.
• Try a fixed workflow first. Use an agent only if the steps cannot be set in advance.
• Build the tools with the least privilege. Start with read-only tools.
• Add the approval step for every action that is hard to undo.
• Build the test cases and run them in a sandbox, not against live systems.
• Turn on logging, cost tracking and limits. Check that you can read a full run back.
• Run the agent next to the existing process. A person still does the work and compares.
• Switch over one step at a time. Keep the hand-over to a person working.