Back to Thoughts

The Hard Part of AI Agents Isn't Code

Jan 29, 20266 min readView raw

Most broken AI agents do not fail because the code is hard.

They fail because nobody decided what kind of system they were building before they started wiring tools to a model.

I have built agents for research, data extraction, web scraping, and orchestration. The failures had a pattern. The implementation usually worked in a demo. The design broke when real inputs, ambiguous state, bad tool responses, and production expectations showed up.

The hard part of AI agents is not code. It is deciding where the model gets freedom, where it gets structure, and what happens when it is wrong.

Link to this sectionAgent tutorials skip the decisions that matter

Most tutorials start with a framework.

Install the package. Define a tool. Connect a model. Run the agent.

That is useful for learning an API, but it hides the decisions that shape production behavior:

  • Should this be an agent or a workflow?
  • Which steps need an LLM, and which should be deterministic code?
  • Can the model choose tools, or does the application choose?
  • Should outputs be prose, JSON, typed objects, or events?
  • What happens when parsing fails?
  • How do you know the agent is improving?

If you skip those questions, you can still ship something. It will probably work for the happy path. Then a user asks the wrong kind of question and the system exposes every decision you avoided.

Link to this sectionStart with a workflow unless autonomy is required

The first fork is agent versus workflow.

An agent decides what to do next. It can choose tools, loop, recover, and adjust based on context.

A workflow follows a known path. Step one, then step two, then step three.

The rule I use now is strict: if the steps are known upfront, use a workflow. If the next step depends on reasoning over the current state, use an agent.

Most teams default to agents too early. They add autonomy, then spend weeks trying to constrain it back into predictability.

Start boring. Use a workflow. Graduate to an agent only when the system needs model-driven choice.

Link to this sectionGuardrails are not free

Too few guardrails and the agent fails on basics. It formats output incorrectly, misses required fields, calls tools in the wrong order, or invents state.

Too many guardrails and the agent stops adapting. It copies examples too literally. It follows stale instructions. It treats your prompt like a maze instead of a goal.

The balance is not a formula. It is an engineering loop:

  1. Start with the smallest useful instruction set.
  2. Observe real failures.
  3. Add constraints where the failures repeat.
  4. Remove constraints when the model no longer needs them.

Do not front-load a giant constitution because you are nervous. You will create a brittle system and still miss the failures that actually matter.

Link to this sectionThe questions to answer before writing code

Before building an agent, write down the decisions.

Link to this sectionDecide the architecture

Ask whether this should be an agent or a workflow.

If the process is predictable, use a workflow. If the system must decide what to do based on context, use an agent. Autonomy should be justified, not assumed.

Ask whether each agent has one clear responsibility.

Researcher researches. Parser parses. Orchestrator coordinates. When an agent owns too many responsibilities, failures become hard to localize.

Ask whether an LLM call is necessary.

If a Python function can classify, transform, validate, or fetch data deterministically, use the function. LLM calls add cost, latency, and variance. Spend them where judgment is needed.

Link to this sectionDecide the shape of outputs

If downstream code consumes the result, prefer structured output.

JSON, schemas, typed objects, and event payloads make integration easier to test. Prose is useful when a human is the final consumer, but it is a poor contract for application code.

Then decide how strict the structure needs to be. A schema helps only if you validate it and define what happens when validation fails.

Link to this sectionDecide the boundaries of autonomy

An agent can choose tools. It can call subagents. It can loop. It can ask for clarification. It can stop early.

Those choices need boundaries.

Can it spend money? Can it hit external APIs? Can it mutate production data? Can it retry forever? Can it expose partial results to users?

Autonomy without boundaries is not intelligence. It is an incident queue with better branding.

Link to this sectionDecide the failure path

Parsing will fail. Tool calls will return bad data. Search results will contradict each other. The model will be confident and wrong.

Decide the fallback before the failure happens:

  • retry with a smaller prompt
  • ask the model to repair the structured output
  • fall back to deterministic parsing
  • ask the user for clarification
  • stop and expose a clear error

The right fallback depends on the task. The wrong fallback is pretending failure is rare.

Link to this sectionModel choice is an architecture decision

Not every task needs the biggest model.

Use smaller models for narrow tasks: classification, extraction, formatting, routing, and simple transformations.

Use larger reasoning models when the agent needs to plan, compare trade-offs, handle ambiguity, or recover from unexpected state.

"Thinking mode" is useful when the task needs thinking. It is wasteful when the task needs reliable formatting.

Model choice affects latency, cost, reliability, and the amount of structure you need around the call. Treat it like a system decision, not a dropdown.

Link to this sectionSmall agents need clean ownership

The best agent architectures I have worked with look simpler than the diagrams people want to draw.

One orchestrator coordinates. Small specialized units handle specific jobs: research, parsing, fetching, validation, analysis.

The orchestrator delegates and combines. The specialized units execute and return. Clean boundaries make failures obvious.

The trap is turning every noun into an agent. If two roles always need the same context and always run sequentially, splitting them may just create context loss. Separate responsibilities because the boundary helps, not because the diagram looks better.

Link to this sectionMeasure behavior, not vibes

An agent is working only if you can define what working means.

Useful measures depend on the product, but common ones include:

  • task completion rate
  • validation pass rate
  • latency
  • cost per successful run
  • retry rate
  • tool error rate
  • human correction rate
  • user acceptance or approval rate

You do not need a perfect eval suite on day one. You do need a feedback loop. Otherwise every architecture discussion becomes taste with extra tokens.

Link to this sectionA pre-build checklist for AI agents

Before you write code, answer these:

  1. Is this an agent or a workflow?
  2. Where does the model make decisions?
  3. Which steps should be deterministic code?
  4. What is the output contract?
  5. How strict is validation?
  6. What can the agent do without human approval?
  7. What context does the agent need, and what should be excluded?
  8. What happens when parsing, tools, or external data fail?
  9. Which metrics prove this is getting better?
  10. Which assumptions should be revisited as models improve?

The code gets easier after this. Not because AI magically handles everything, but because the hard decisions are no longer hiding inside implementation details.

Frameworks will not save a confused architecture. The right questions will.

Keep reading

All thoughts
Gopal Khadka