AI Agent API Guide: Agent Architecture, the Tool-Calling Schema, and Loop Cost Management

An AI agent is a software pattern that puts a language model in a loop with tool-calling, memory, and planning layers; on the API side, every step is a separate chat/completions request. The real production test is not architecture but cost: because history is re-sent on every step, token consumption compounds. This guide walks through the architecture layer by layer and per-step cost control with real data.

Most guides define the agent concept and list frameworks; the first wall a team hits when shipping an agent to production is neither — it is the invoice and the latency. An innocent six-step task can burn more than ten times the tokens of a single-question chat, and you usually find out at the end of the month.

In this article you will find the four layers of agent architecture and the tool-declaration schema, plus data you will not find elsewhere for this query: latency measurements from real production traffic, per-step cost math with Turkish lira equivalents, and a working budget-capped Python agent loop.

AI Agent Architecture: Planner, Tools, Memory, and the Loop

An AI agent consists of four layers: a planner that breaks the goal into steps, tools that talk to the outside world, memory that carries context between steps, and the loop (orchestration) layer that runs it all. That is also what separates it from a chatbot: a bot produces one answer to one question; an agent takes a goal and keeps calling tools and evaluating results until the goal is reached.

• Planner: The model call that decomposes the task and decides "what now?" on every turn. Since this layer largely determines agent quality, it usually runs on a flagship-class model.
• Tools: Functions such as web search, database queries, or code execution. The critical point: your code runs the tool, not the model — the model only says which tool to call with which parameters.
• Memory: Short-term memory you already have: the messages array. The common pattern for long-term memory is embeddings + a vector database; let us be honest here: Onysoft does not expose a separate /v1/embeddings endpoint. Two architectures work instead: summary memory, where an economical model condenses past steps into a single message (again via chat/completions), or generating embeddings with an external or local service and using the gateway only for the generation layer.
• Loop: Interprets the model output, runs the tool, appends the result to history, and returns to the model. Stop conditions — a finish decision, a step cap, a budget ceiling — are this layer's responsibility, and in production it is the most neglected part.

Because every one of the 739+ models in the catalog is called with the same schema, switching models between layers is a one-parameter change — which is also the foundation of the cost strategy we are about to see.

The Tool-Calling Schema: How Do You Declare Tools to the Model?

Tools are declared to the model with the OpenAI function schema, which has become the de facto standard: each tool is a name, a description of what it does, and a JSON Schema block defining its parameters. When the model sees fit, it returns a structured response meaning "call this tool with these arguments" instead of plain text.

{
  "type": "function",
  "function": {
    "name": "order_status",
    "description": "Returns the shipping status for an order number.",
    "parameters": {
      "type": "object",
      "properties": {
        "order_no": {"type": "string", "description": "Order number starting with ORD-"}
      },
      "required": ["order_no"]
    }
  }
}

The flow has three steps: (1) you send the tools list with the request, (2) if the response contains a tool_calls field you run the tool in your own code, (3) you append the result to the conversation and return to the model. The description fields in the schema matter as much as the prompt: the model picks tools by reading these texts, and a vague description is the number one cause of wrong tool selection.

An important caveat: tool-calling support varies by model and provider — not every model supports the tools parameter with the same maturity. That is why we built the example loop below with a protocol approach where the model states its tool decision as plain JSON output, which works with every chat model; on models that support tools, the same skeleton maps one-to-one onto tool_calls.

Why Does Cost Explode in an Agent Loop? Per-Step Math and Tiered Model Selection

Because every loop step re-sends the entire accumulated history as input: in a 6-step task, the initial prompt is billed 6 times and the second step's tool output 5 times. Let us make it concrete: say the system prompt + task is 1,500 tokens, and each step appends roughly 800 tokens of tool output and 400 tokens of model response to history. After six steps you pay for a total of 27,000 input + 2,400 output tokens. The same task billed on different models:

Model	Input ($/1M)	Output ($/1M)	1 task (6 steps)	1,000 tasks/day
anthropic/claude-opus-5	$7.50	$37.50	$0.2925 (₺13.91)	₺13,910
anthropic/claude-sonnet-5	$3.00	$15.00	$0.1170 (₺5.56)	₺5,564
google/gemini-3.6-flash	$2.25	$11.25	$0.0878 (₺4.17)	₺4,173
openai/gpt-5.6-terra	$1.50	$9.00	$0.0621 (₺2.95)	₺2,953
deepseek/deepseek-v4-flash	$0.21	$0.42	$0.0067 (₺0.32)	₺318
openai/gpt-5.6-luna	$0.15	$0.90	$0.0062 (₺0.30)	₺295

Measured: August 5, 2026 — api.onysoft.com live catalog. TRY equivalents calculated at the August 5, 2026 TCMB rate (1 USD = 47.555 TRY). Scenario: 6 steps, 27,000 input + 2,400 output tokens in total.

The real win is tiered selection: give planning and final synthesis to anthropic/claude-sonnet-5 (steps 1 and 6) and the intermediate steps to openai/gpt-5.6-luna, and the task cost drops to $0.0431 (₺2.05) — 63% cheaper than running every step on Sonnet and 85% cheaper than Opus, with no compromise on planning quality. At 1,000 tasks per day the difference is ₺2,051 instead of ₺13,910. The second lever is history hygiene: trimming or summarizing long tool outputs inside the loop, instead of carrying them verbatim, directly shrinks input tokens.

On generic endpoints where you do not know the step type, you can delegate selection to the gateway with onysoft/auto (OnyRouter); inside an agent loop, however, you already know whether a step is planning or tool interpretation, so pinning the model per step type is usually the sharper call. For bulk estimates, use the cost calculator.

How It Works on Onysoft: a Budget-Capped Agent Loop with the cost Field (Python)

Because Onysoft AI Gateway follows the OpenAI schema exactly, the agent loop is built with the official openai package; the only difference is base_url. The skeleton below includes three production safeguards: a step cap, a task budget, and real per-step cost tracking via the cost field returned in every response envelope:

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.onysoft.com/v1",
    api_key="sk-ony-YOUR-AGENT-KEY",  # a dedicated, limited key for the agent
)

def web_search(query): ...   # your own search function
def calculate(expr): ...     # your own calculator function
TOOLS = {"web_search": web_search, "calculate": calculate}

SYSTEM = ("You are a task agent. At every step, answer ONLY with this JSON: "
          "{\"thought\": \"...\", \"action\": \"web_search | calculate | finish\", \"input\": \"...\"}")

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "Task: find the prices of 3 rival products, compute the TRY average."},
]

TASK_BUDGET_USD = 0.05
spent = 0.0

for step in range(8):  # infinite-loop fuse
    resp = client.chat.completions.create(
        model="openai/gpt-5.6-luna",  # intermediate steps: speed class
        response_format={"type": "json_object"},
        messages=messages,
    )
    spent += (resp.model_extra or {}).get("cost", 0.0)  # real cost (USD) from the response envelope
    if spent > TASK_BUDGET_USD:
        raise RuntimeError(f"Task budget exceeded: ${spent:.4f}")

    decision = json.loads(resp.choices[0].message.content)
    if decision["action"] == "finish":
        print(decision["input"])
        break

    result = TOOLS[decision["action"]](decision["input"])
    messages.append({"role": "assistant", "content": resp.choices[0].message.content})
    messages.append({"role": "user", "content": f"Tool result: {result}"})

The cost field in the loop is not an estimate — it is the actual USD cost the gateway charged for that request; you can log per-step spend and cut the task budget in code without deploying a separate metering tool. Pointing the model parameter at a flagship for the planning step is a one-line change, and you can stream the final user-facing answer token by token over SSE (stream=True).

Even if your budget cutter has a bug, there is a second safety net: before dispatching any request, Onysoft compares 1.2 times the estimated cost against your balance; if it is not covered, the request never reaches the provider and returns HTTP 402 — a runaway loop cannot push your balance negative. The third layer is key-based: generate a separate sk-ony key for the agent in the dashboard and give it its own spending limit, and worst-case damage stays capped at that key's limit, leaving your main application key untouched. Schema details are in the API documentation.

The Step Budget: Planning Agent Duration with Real Latency Data

An agent's total duration is roughly step count × per-step latency, and it should be planned from measurements, not guesses. Numbers from our own production traffic: across 6,959 requests through the gateway in the last 14 days, average end-to-end response time was 3.8 seconds, with the fastest at 0.3 seconds. On the average profile, a 6-step agent takes about 23 seconds — fine for a background job, unacceptable in front of a live user.

In practice the time budget is built with three decisions:

• Shrink the step count: The biggest win lives here; a good planner can fold three tool calls into one turn, and independent tool steps can run in parallel.
• Give intermediate steps to the speed class: The 0.3-second floor was measured with lightweight models; luna/flash-class intermediate steps can pull a loop turn under a second.
• Stream the final answer: With SSE on the last step, the first token hits the screen within a few seconds; perceived waiting drops far below the total duration.

A five-item checklist before going live: a step cap, a task budget via the cost field, a dedicated agent key with its own limit, a latency target, and per-step logging. For the broader picture see the AI API guide, and for the full set of cost mechanisms the cost control article; open a free account and try your first agent prompts in the Playground.

Last updated: August 5, 2026 · Data: api.onysoft.com live catalog