A Practical Mental Model for Agent Architecture

An agent does not need a maze of orchestration code to look intelligent. A small, well-behaved loop can do most of the work:

user message
    ↓
model turn
    ↓
tool call or final message
    ↓
policy checks → tool handler → recorded result
    ↺

This article describes one way to build that loop. It is a design based on a particular set of priorities: let the model choose the next task-level action, keep the engine predictable, and make every tool call observable and recoverable. It is not a claim that every agent should be built this way.

The useful division of labour is simple:

• The LLM chooses what to do next.
• The engine keeps the task moving, enforces boundaries, and records what happened.
• Tools do the work that touches the outside world.

The engine should be boring. That is a compliment. When a task fails at two in the morning, predictable bookkeeping is much nicer to debug than a second model hiding inside the orchestration layer.

Three layers, three jobs

The engine layer

The engine owns the task loop. It assembles context, calls the model, parses the response, runs policy checks, invokes a tool, records the result, and repeats.

It does not decide that a vague request for “research” must become a search followed by three browser visits. That is a task-level decision for the model and its prompt. The engine does, however, decide whether a proposed action is allowed and how it may run. It can filter the visible tool list, require approval for a dangerous operation, serialize tool calls, retry a timed-out request, enforce a working-directory boundary, or stop a runaway loop.

That distinction is more accurate than saying the engine “does not make choices.” It does not make the model's high-level choices, but it still makes reliability and safety decisions.

The engine's job is therefore mechanical in a useful sense:

• maintain the task state;
• apply explicit policies;
• execute handlers and capture their results;
• persist and publish events;
• recover or fail in a known way.

Retries should follow similarly explicit rules. A timeout, a 429, and a malformed response are different problems. Transport failures may be retried with backoff. A structural failure usually needs to be recorded and surfaced. The engine should not quietly rewrite the model's query or add an instruction that the model cannot see. If the system changes what the model is allowed to see, that change should be represented in the context or tool list.

The prompt layer

The system prompt is the model's operating contract. It can be split into XML-tagged sections simply to keep a large prompt navigable; the tags are an organizational device, not an orchestration language.

A typical prompt may describe:

• output language and format;
• the agent loop and when to call tools;
• tool-call rules, such as one call at a time;
• error-handling and confirmation behaviour;
• the current environment and available CLI tools;
• browser capabilities;
• secret placeholders and prompt-injection rules;
• user preferences and available skills;
• the tool protocol and the expected final response.

These sections are rules and examples, not an if-else program. The model reads them and chooses its next action. That flexibility is the point, but it also means the prompt needs a clear contract. If a behaviour is a hard safety boundary, do not rely on wording alone; enforce it in the engine as well.

The tool layer

Tools are the engine's connection to the outside world. Each tool has two interfaces:

• ToolDefinition: the name, description, and parameter JSON Schema shown to the model;
• ToolHandler: the implementation called by the engine with validated arguments and execution context.

Most tools fall into four practical groups:

Group	Examples	Purpose
Communication	message, plan	Write to the conversation or update task state
Information	search, browser, vision	Read information from outside the process
Modification	file, shell, match	Change files, processes, or other local state
Generation	slides, generate	Produce a larger artifact

The grouping is useful for policy and UI, but it is not a law of nature. A tool can have more than one kind of effect; what matters is that those effects are described and handled explicitly.

The task loop: model-driven, engine-controlled

TaskLoop is the centre of the system. It starts when a user message arrives and ends when the task finishes, fails, is cancelled, or is waiting for someone to reply or approve an action.

It helps to separate two concepts that are often flattened into one state machine:

Execution phase — what the loop is doing right now:

• INITIALIZING
• THINKING
• EXECUTING_TOOL
• FINALIZING

Task status — what the task is waiting for or how it ended:

• RUNNING
• WAITING_FOR_REPLY
• WAITING_FOR_APPROVAL
• DONE
• FAILED
• CANCELLED

Exact names can differ. The important part is not to label a user cancellation as a system failure just because both stop the loop.

One pass through the loop looks like this.

1. Run pre-flight checks. Check for masterAbort or an interjection, rebuild the visible tool list if the context has changed, and make sure the task is still allowed to run.

2. Call the model. Read the current SessionContext, apply redaction and length checks, and send the provider-specific request. Retry transient transport failures with bounded backoff. Do not treat every bad response as retryable.

3. Normalize the response. Convert provider-specific output into a common LLMTurnResult. In this design, the model is expected to express user-facing output through a tool such as message. If a turn contains neither a usable tool call nor an acceptable final response, record a SystemReminder and retry a bounded number of times instead of looping forever.

4. Persist the assistant turn. Store the model response in SessionContext as a ToolCall or AiMessage, together with fields such as thinking when the provider supplies them.

5. Apply execution policy. Run loop detection, working-directory checks, dangerous-operation approval, and any plugin hooks. A rejected call should produce a recorded result or status, not disappear into a log file that nobody can connect to the original request.

6. Execute the tool. A sensible default is one tool call at a time. Serial execution keeps ordering, error handling, and provider bookkeeping easy to reason about. Safe, independent reads can be run in parallel if the tool definitions declare that they are safe to parallelize and the result-to-call mapping remains unambiguous. “Always execute only the first call” is a policy choice, not a universal truth.

7. Persist the result. Write the returned content as a ToolResult, process attachments, waitForReply, finalResult, and similar fields, and publish progress to the frontend.

8. Maintain context. Re-inject deferred calls when the policy requires it, compact history when it crosses the configured threshold, and summarize large reads when appropriate. Then start the next pass.

The loop can finish in a few ways:

• the model calls message with a result that declares the task complete;
• a handler or provider fails in a way the system cannot recover from;
• the user stops or cancels the task;
• the task reaches its iteration limit;
• the task enters a waiting state and resumes after a reply or approval.

Interruption is not cancellation by magic

When a user sends a new message while the model is responding or a tool is running, the gateway can call taskLoop.interject(). The loop sets an interrupt flag, aborts the current model request where possible, and feeds the new message into the next turn.

Wrapping a handler in Promise.race can make the loop stop waiting immediately, but it does not necessarily stop the underlying operation. A shell process, browser request, or file write may still be running unless the handler supports cooperative cancellation through an AbortSignal or a process-specific termination mechanism.

For tools with side effects, the design should also answer three less glamorous questions:

• What cleanup happens if cancellation arrives halfway through?
• How does a restart tell “completed but response not recorded” from “never started”?
• Can a retry safely run the operation twice?

Idempotency keys, execution records, and explicit cancellation support are often more valuable here than another layer of prompt instructions.

Reliability features that earn their keep

Interjection and resume checkpoints

An unfinished task should survive a server restart without guessing what happened. When an ask tool waits for a user reply, the assistant record can store metadata.ask.answered = false. A dangerous-operation request can set metadata.approvalPending = true.

On startup, the system can scan for tasks in waiting_for_reply or waiting_for_approval, rebuild their wait parsers, and broadcast the corresponding UI state. The exact database fields are implementation details; the invariant is that the task's waiting reason is durable and recoverable.

Loop detection

Models sometimes keep reading the same missing file or issuing the same failed request. A loop detector can retain a bounded window of recent calls, hash the tool name plus normalized arguments, and flag repeated combinations above a threshold.

The first response does not have to be a hard failure. A recorded SystemReminder such as “this call has repeated; try a different approach” gives the model a chance to recover. After a stricter limit, the engine should stop the task and explain why. The threshold belongs in configuration, not in the article of faith section of the prompt.

Context: one event, several views

Context management is where a lot of otherwise convincing agents become difficult to debug. The useful mental model is not “the same string is copied everywhere.” It is “one recorded event produces several traceable projections.”

SessionContext and TaskContext

SessionContext lives for the whole session. It owns the LLM history (llmContext), the WebSocket emitter, and a reference to the current TaskContext. It typically provides methods such as append, update, getLLMContext, and loadFromDB.

TaskContext lives for one task. It holds values such as taskId, userId, modelId, the currently available tools and their filtering context, completion and failure flags, stop state, token usage, loop-detector state, and callbacks for waiting on replies or approvals.

If several tasks share one session, their history may be continuous by design. That is useful for conversational continuity, but it also creates a context-leak risk. Task boundaries, user scope, and compaction rules need to make that continuity intentional rather than accidental.

The append rule

SessionContext.append should be the normal path for adding a context event. A single append can then:

1. persist the event to the task_context table;
2. publish the event to the frontend;
3. add the event to the in-memory history used for the next model call.

These three consumers do not need byte-for-byte identical representations. The database may keep the canonical event, the frontend may render an attachment or a progress card, and the LLM view may apply redaction, compaction, provider conversion, or truncation. What they do need is a shared event identity and enough metadata to reconstruct what the model actually received.

This is a stronger and more useful guarantee than “the same content appears in all three places.” It makes the database, UI, and model views consistent without pretending that they serve the same purpose.

Useful ContextType values include:

• UserMessage: the user's original message;
• AiMessage: a user-facing response sent through message;
• ToolCall: the model's requested tool and arguments;
• ToolResult: the handler's returned result;
• SystemReminder: a state- or policy-driven reminder;
• StatusNotice: engine status such as “retrying”;
• NoToolCallRetry: a discarded turn that failed response validation;
• WorkflowStep: the current step in a larger workflow.

The frontend can use these types for rendering. The model generally sees a provider-specific role and content, not the internal classification itself.

Building the model context

Before each model call, getLLMContext can:

1. rebuild the system prompt with current configuration;
2. collect non-system entries from llmContext;
3. compact intermediate turns when the token budget is tight;
4. apply a hard context-window guard if compaction is not enough;
5. return the final LLMMessage[] for the provider adapter.

Keeping the system prompt outside llmContext makes prompt rebuilding and caching easier, but it also means the system needs a way to record the prompt version or reconstruct the exact prompt used for a given call.

Compaction: keep turns intact, keep facts that matter

Long-running tasks can produce a surprising amount of text. One 5 KB file read may consume roughly 1,500 tokens once wrapped in tool metadata and conversation structure. Repeat that a few dozen times and the context window starts charging rent.

The unit of compaction should usually be a Turn, not an arbitrary message. A turn may contain an assistant tool_call, its tool_result blocks, and an immediately following system reminder. Splitting those pieces can produce invalid provider messages: for example, a tool_use without its matching result.

A practical compaction pass looks like this:

1. group messages into turns;
2. reserve a tail budget for the newest turns, such as a configured percentage of the maximum;
3. keep the newest complete turns until the tail budget is full;
4. replace older turns with a compacted_history block;
5. apply a hard guard only if the result is still too large.

The summary for an old turn should preserve more than the tool name and arguments when the action has consequences. At minimum, it should capture:

• what the tool did;
• the important result or conclusion;
• side effects such as files written, commits created, or external requests made;
• decisions, constraints, and approvals established in that turn;
• a file path, URL, record ID, or other handle for retrieving the original when needed.

For a repeatable file read, the name and path may be enough. For a shell command that changed the environment, they are not. A summary that says only “ran shell” is technically short and practically useless.

Aggressive compaction can still be a good choice when the original data is cheap to re-read and the summary is not trusted as a source of truth. The trade-off is extra reads. That is often preferable to silently carrying an incorrect model-generated summary, but it should be measured for the workload rather than assumed.

The context-window guard is the last safety fuse. If the prompt still exceeds the limit, remove the oldest complete turns until it fits. If a single tool result is larger than the available budget, truncation is not a real fix; the tool needs pagination, a file-backed result, or a different output format.

System reminders and the turn grouper

Static rules belong in the system prompt. State-dependent reminders should be generated close to the model call. “You are at step three” or “this call has repeated” is not a permanent property of the agent; it is a fact about the current state.

When reminders affect model behaviour, store them as real SystemReminder entries with role: user or the provider-equivalent representation. That keeps the event log honest and makes replays possible.

The Turn Grouper is the component that turns a flat message list into compaction units. Its rules need to cover user messages, assistant tool calls with their matching results, system reminders, plain assistant replies, and interrupted or malformed turns. Its tests are worth treating as core infrastructure. If the grouper gets the boundaries wrong, the rest of the context system will fail in much less obvious ways.

Tools: small surface, explicit contracts

The tool layer is where an agent stops talking and starts changing things. Clear contracts matter more here than clever descriptions.

Definition, handler, and policy

ToolDefinition is optimized for the model: concise descriptions, valid parameters, and enough examples to prevent common miscalls.

ToolHandler is optimized for execution and testing. It should not reach into the agent's database or WebSocket emitter behind the engine's back. That does not make a shell or file handler a pure function; those operations obviously have side effects. The more useful rule is to keep agent bookkeeping in the engine, make external effects explicit, and return a structured, auditable result.

ToolPolicy describes who may use the tool and what approval or danger level it requires.

Registration and results

A side-effect import can be a reasonable registration mechanism for a small TypeScript service. Each tool module registers itself when imported, tools/index.ts imports the modules, and task-loop.ts only needs to import the index once. If the registry grows large, explicit registration may be easier to inspect; the important thing is that startup tests can prove which tools are actually available.

ToolHandlerResult can use a declarative shape such as:

• required content for the model;
• optional metadata for logs and UI;
• finalResult to mark completion;
• shouldBreak to exit the loop;
• summary to provide a compaction-specific description;
• attachments for files or other artifacts.

The execution path then becomes: parse and validate arguments, apply policy, call the handler, persist the result, and process the special fields. The handler does not need to know how the frontend renders a progress card.

Argument coercion

LLMs sometimes return "3" where a schema says 3, or "false" where a boolean is expected. A centralized coerceToolArgs step can correct safe type mismatches recursively before the handler runs.

Coercion should be conservative. It is fine to turn a numeric string into a number when the schema permits it. It is not fine to silently “fix” an ambiguous path, command, or enum value and hope for the best. Invalid or risky input should fail validation with a useful message.

Dynamic tool lists

The model does not need to see every tool on every turn. The visible list can be recomputed when relevant state changes:

• user role can hide administrative tools;
• a custom blacklist can disable selected tools;
• browser tools can be hidden until a browser session exists;
• workflow state can change whether plan is useful;
• task policy can remove tools outside the current working directory.

Dynamic filtering saves prompt tokens and reduces calls that cannot succeed. A filterKey or equivalent cache key can avoid rebuilding the list when nothing relevant changed.

Danger levels

A simple policy might classify tools as:

• safe: reads and queries that do not change state;
• moderate: local modifications such as writing a file, with detailed logging;
• dangerous: irreversible or broadly scoped actions such as rm -rf or git push --force, requiring explicit approval.

The labels are only useful if the policy behind them is specific. “Moderate” should not mean “we forgot to decide.” Approval records should identify the exact operation, arguments, and scope the user approved.

Fewer tools, but not one giant tool

Using a small set of versatile tools can reduce prompt size and the model's choice overload: one shell tool may cover installation, Git, and local commands; one file tool may expose a handful of clearly named actions; one search tool may accept a small set of search types.

There is a cost. A very broad shell or file tool is harder to sandbox, audit, and describe accurately. “Few but versatile” is a useful starting point, not a reason to hide every capability behind a single parameter called action.

Tools can also provide a compactTemplate so a large call can be represented by a short, structured summary during compaction.

The provider adaptation layer

Different providers disagree about message shapes, tool calls, system prompts, multimodal content, and thinking fields. The rest of the agent should not have to know all of those differences.

One implementation can expose three calling modes:

Mode	How it works	Trade-off
openai_native	Uses the provider's OpenAI-style tool fields	Mature and widely supported, but proxies can translate fields incorrectly
anthropic_native	Uses Anthropic's message and content-block format	Supports provider-specific features such as prompt caching and thinking fields
anthropic_xml	Describes tools in the prompt and parses XML calls from text	Useful when an intermediary drops native tool fields, but costs tokens and needs careful parsing

The XML mode is a fallback, not a free compatibility layer. Its parser needs to handle malformed XML, escaped content, partial streaming output, duplicate calls, and text that happens to resemble a tool block.

invoke.ts can resolve a logical model to one or more provider configurations, order them by priority, and implement failover. protocol-convert.ts handles differences in system-message placement, multimodal content, and tool_use/tool_result blocks. response-parse.ts maps provider responses into one LLMTurnResult containing fields such as content, toolCalls, toolCallSource, thinking, usage, and redacted log data. normalize.ts hides provider-specific quirks such as different locations for reasoning content.

Secrets, caching, and provider health

Literal replacement of known secrets before a model call can reduce accidental exposure, but it is not a complete secret-management boundary. Secrets can be encoded, split across messages, echoed by a tool, or reintroduced through logs. A stronger design keeps secret injection out of the model's reasoning path: the model refers to a scoped placeholder, and a trusted execution layer resolves it only when the selected tool needs it. Logs, results, and frontend output should have their own redaction rules.

Prompt caching works best when most of the system prompt is stable and dynamic data is kept in a clearly separated suffix. A zero-width marker can divide those regions if the provider requires it. The savings depend on the provider, cache lifetime, prompt shape, and hit rate; the architecture should measure them instead of promising a fixed percentage.

A circuit breaker can track recent failures per provider. When a provider crosses a configured failure threshold, the adapter skips it for a cooling period, then sends a limited half-open probe before restoring normal traffic. This avoids making every user wait through the same timeout, while still allowing a recovered provider back into rotation.

Search and web access

Search is a good example of a design choice that depends on product goals. If the system needs inspectable queries, caching, site filters, time limits, or provider portability, a separate search tool gives the engine more control. Native model search may still be the better choice when freshness, built-in citations, permissions, or lower integration work matter more.

A separate search tool can classify intent with types such as:

• info for a general lookup;
• news for recent reporting;
• research for papers and technical work;
• api for documentation;
• data for datasets;
• image for image search;
• tool for finding software or services.

The type can guide query construction and result ranking, but it should not become a hidden way of changing the user's words without a trace. Store the final query and the original intent so a search can be debugged later.

Tavily is one possible implementation when structured, model-friendly results are useful. Bocha may be a better fit for some Chinese-language sites. Brave Search, Exa, and Serper are other options. The right choice depends on region, freshness, licensing, latency, cost, and the sources the product actually needs.

A two-stage strategy is often practical: search returns candidate URLs and short summaries, then a browser or reader tool fetches the pages worth inspecting. This keeps the search response small and lets the browser handle the full document. It also adds a request and some latency, so it is an optimization for inspectability and context size, not automatically the fastest path.

A browser toolset might include visit, view for an accessibility tree, click, input, scroll, screenshot, and fetch_url for converting a page into Markdown. A reader API can provide cleaner text, with a headless browser as a fallback when the page requires JavaScript or interaction.

Rules worth protecting

The following constraints are more useful than a second page of slogans. They are checks to apply during implementation and review.

1. Do not silently rewrite the model's context. Append new events or metadata. If an existing LLM-facing message must be transformed for a provider, keep the transformation explicit and traceable.
2. Use one event path for persistence and projections. Do not write directly to the database, frontend, or in-memory history in three unrelated code paths.
3. Keep agent bookkeeping out of handlers. A handler may create an external side effect, but it should return a result the engine can persist, audit, and display.
4. Preserve tool-call pairing. Compact and replay complete turns, and keep every result tied to the correct call ID.
5. Serialize by default; parallelize by declaration. Only run calls together when their dependencies, side effects, cancellation behaviour, and result mapping are understood.
6. Make model-facing injections visible or reconstructible. A reminder that changed behaviour should appear in the event history or be recoverable from the recorded prompt version.
7. Summarize outcomes, not just actions. Compaction should preserve decisions, side effects, and important results.
8. Separate retry, cancellation, and failure. They have different recovery semantics, especially for tools that change state.

Common traps and a right-sized first version

The easiest way to overbuild an agent is to design for a future deployment that does not exist yet. A local, single-user agent may not need a multi-tenant sandbox, a provider portal, a billing system, or a fleet of background workers. Direct execution on the host can be a reasonable starting point if the working directory, permissions, approvals, and audit trail are explicit.

SQLite is often enough for a local prototype. A small schema might include tasks, task_context, tool_call_logs, sessions, and llm_logs. Moving to PostgreSQL or MySQL is driven by concurrency, backups, replication, permissions, operational ownership, and deployment constraints—not by a universal file-size cutoff.

A useful first version can include:

• one working task loop;
• in-memory context plus durable task records;
• one provider adapter;
• message and echo tools;
• structured tool results;
• basic logs and a dangerous-operation gate;
• tests for turn grouping, tool/result pairing, and resume behaviour.

Provider failover, browser automation, streaming, compaction, and a larger tool catalogue can follow once the basic loop is observable. The point is not to avoid production concerns; it is to avoid building five abstractions before the first tool call has completed successfully.

A build order with acceptance criteria

The following stages are milestones, not calendar promises. The time required depends heavily on the provider count, UI scope, security model, and how much “production-ready” is expected to mean.

Stage 0: environment and scaffolding

Choose the stack—perhaps TypeScript, Node 22+, pnpm, Vite, Vitest, Zod, Better-SQLite3, Drizzle ORM, and Hono or Fastify—and make the smallest service start reliably.

Acceptance criteria: the service starts, a test runs, configuration loads, and the project can record a task without calling a model.

Stage 1: the minimal loop

Implement user message → model response → tool call → tool result with in-memory context. Support only OpenAI-style calls and two tools: message and echo.

Acceptance criteria: a recorded task can be replayed from its events, and a malformed tool call fails with a visible error.

Stage 2: provider adaptation

Add openai_native and anthropic_native through a common LLMTurnResult. Keep provider conversion and response parsing out of TaskLoop.

Acceptance criteria: the loop behaves the same way against both adapters, and provider-specific fields do not leak into core task logic.

Stage 3: persistence and compaction

Add the database, SessionContext, TaskContext, the Turn Grouper, compaction, and context-window limits. Test interrupted turns and tool/result pairing before optimizing summaries.

Acceptance criteria: a long task stays within its context budget, can resume after restart, and preserves important side effects in its compacted history.

Stage 4: tools and policy

Add file, shell, search, or browser tools as needed. Introduce working-directory boundaries, danger levels, approval records, argument validation, loop detection, and structured execution logs.

Acceptance criteria: every tool call has an owner, an audit record, a clear failure path, and a defined cancellation or retry policy.

Stage 5: frontend, streaming, and recovery

Add streaming output, interjection, waiting UIs, attachments, provider failover, and resume after power loss or process restart.

Acceptance criteria: the user can tell what the agent is doing, stop it without guessing what happened, approve a dangerous action with the exact scope visible, and continue an interrupted task safely.

Closing thought

An agent architecture does not have to be clever in every layer. Let the model choose the next useful action. Let the engine enforce invariants, permissions, persistence, and recovery. Let tools perform concrete work and report their effects.

If you can replay a task and explain why each action was available, allowed, executed, and recorded, the architecture is probably in decent shape. If you cannot, more prompt text is unlikely to save it. Add observability first; the extra abstraction can wait.