What Is RAG and How Do You Set It Up? Step-by-Step RAG for Turkish Data with Real Cost Math

RAG (Retrieval-Augmented Generation) is the architecture that feeds a language model's answer with passages retrieved from your own documents. The setup has three layers: an embedding layer that chunks documents and turns them into vectors, a vector database that runs similarity search, and the LLM that generates the answer. You build the first two with open-source tools and connect the generation layer to Onysoft AI Gateway's OpenAI-compatible endpoint.

Most answers to "what is RAG" stay at the concept level: a diagram is drawn, components are listed, and two critical questions are left open. First, architectural honesty: which layer should be built with which tool, and what should you — and should you not — expect from your LLM provider? Second, money: every token you put into the context lands on your invoice — so how many tokens should you put there?

This guide answers both directly: a layer-by-layer setup plan, embedding and chunking specifics for Turkish data, a 4K vs 32K context cost comparison calculated with live catalog prices, working production code, and real latency measurements from production traffic.

What Is RAG? How Retrieval-Augmented Generation Works

RAG is the architecture in which a search over your own knowledge sources runs before the language model generates, and the answer is grounded in the retrieved passages. The model then speaks not only from the general knowledge in its training data but from information that is current and specific to you at the moment the question is asked: hallucination drops, and every answer can cite its sources.

The system works in three stages:

• 1. Indexing (once, then on every update): Sources such as PDFs, help articles, contracts, and product docs are read; split into chunks that preserve meaning; each chunk is converted into a numeric vector by an embedding model and written to a vector database.
• 2. Retrieval (per question): The user's question is embedded with the same model, and the most similar chunks (top-k) are found in the database.
• 3. Generation (per question): The retrieved chunks plus the question are sent to the LLM with an instruction; the model produces the answer from this context only.

The key difference from fine-tuning is updatability: when the knowledge changes you do not retrain the model, you only update the index. That is why RAG is the default architecture for internal knowledge assistants, document Q&A, and support bots — scenarios where the knowledge changes often.

RAG Architecture: Why Build the Three Layers Separately?

In a healthy RAG setup, the embedding layer, the vector database, and the generation layer are three separate components — not one block glued to a single provider. Let us be direct here: Onysoft AI Gateway does not offer a /v1/embeddings endpoint. You build the embedding layer with an open-source local model or an external embedding service; the layer Onysoft takes on is generation, through its OpenAI-compatible chat/completions endpoint.

This separation is not a workaround for a gap — it is simply correct engineering practice, because the two layers have opposite lifecycles:

• The embedding model must stay fixed. Every record in your vector database lives in the space that model produced; changing the model means re-indexing the entire corpus. You want to make that decision deliberately, rarely, and on a plan.
• The generation model must be freely swappable. When a better or cheaper model ships, migrating should be a one-line model field change that never touches your index. That is exactly where connecting the generation layer to a gateway pays off: you can switch among the 739+ models in the catalog with zero re-indexing cost.

On the vector-database side, the common options: if you already run PostgreSQL, the pgvector extension is the first stop for most teams; Qdrant, Milvus, and Weaviate for dedicated engines; FAISS as a serverless library for prototypes and small corpora. Below roughly a million chunks, this choice matters far less than embedding quality — start with the simple option.

RAG on Turkish Data: Embedding Choice and Chunking Strategy

In Turkish RAG the first quality-defining decision is the embedding model and the second is the chunking strategy — the generation model can only answer as well as the context those two find. Turkish is an agglutinative language: a single word like "arabalarımızdakiler" carries the information of a five-word English phrase. Classic keyword search is therefore weak in Turkish, and semantic vector search (dense retrieval) is a major advantage — but only if the embedding model saw a serious amount of Turkish during training.

When choosing a model, look at three concrete criteria: the presence of Turkish in the multilingual training data and the model's score on a retrieval benchmark that includes Turkish; the vector dimension (larger dimensions mean higher storage and search cost); and the license and hosting mode (a local open-source model gives data sovereignty, an external service removes operational load). Most importantly: base the decision on your own data, not on general leaderboards. Build a golden set of 50-100 real questions from your corpus mapped to their correct passages, and measure candidate models against it. That half-day investment closes weeks of "why is it retrieving irrelevant chunks" debates in production.

On chunking, a common and sensible starting point: chunks of 300-800 tokens, split at paragraph or sentence boundaries, with 10-15 percent overlap; attach the document name and section heading to each chunk as metadata. Also watch two Turkish-specific traps: the dotted/dotless İ/ı distinction breaks in standard lowercase functions (İSTANBUL becomes i̇stanbul-style artifacts instead of istanbul) — make normalization locale-aware; and OCR-sourced text with corrupted ğ/g and ş/s silently degrades embedding quality — clean it before indexing.

How Many Tokens Belong in the Context? The 4K vs 32K Cost Math

The practical formula is: context budget ≈ top_k × average chunk size — and that number is the main driver of per-query cost in RAG, because every chunk you place in the context is billed as input tokens on every question. For example, top_k=5 × 800 tokens ≈ a 4,000-token context is a solid starting point for most document Q&A scenarios; going to 32,000 tokens with top_k=40 is the "stuff in whatever you found" approach.

Let's price the difference with live sale prices. Scenario: 300 tokens of system instruction + 100-token question + context (4,000 or 32,000) as input; 500 output tokens; 1,000 queries per day (30,000 per month). Generation-layer prices:

Model	Input ($/1M)	Output ($/1M)	Input (TRY/1M)	Output (TRY/1M)
anthropic/claude-opus-5	$7.50	$37.50	₺356.66	₺1,783.31
anthropic/claude-sonnet-5	$3.00	$15.00	₺142.67	₺713.33
openai/gpt-5.6-terra	$1.50	$9.00	₺71.33	₺428.00
openai/gpt-5.6-luna	$0.15	$0.90	₺7.13	₺42.80
google/gemini-3.6-flash	$2.25	$11.25	₺107.00	₺534.99
deepseek/deepseek-v4-flash	$0.21	$0.42	₺9.99	₺19.97

The same scenario per query and per month:

Setup	Input tokens	Per query (USD)	Per query (TRY)	Monthly, 30,000 queries (TRY)
Sonnet 5 + 4K context	4,400	$0.0207	₺0.98	≈ ₺29,532
Sonnet 5 + 32K context	32,400	$0.1047	₺4.98	≈ ₺149,370
Luna + 4K context	4,400	$0.0011	₺0.05	≈ ₺1,584
Luna + 32K context	32,400	$0.0053	₺0.25	≈ ₺7,576

Measured: August 5, 2026 — api.onysoft.com live catalog. TRY equivalents calculated at the August 5, 2026 TCMB rate (1 USD = 47.555 TRY).

The conclusion is stark: growing the context from 4K to 32K multiplies per-query cost by roughly 5x on the same model — with Sonnet 5, the monthly gap exceeds ₺119,000. And a wide context does not automatically raise quality; it is a known model behavior that information buried in the middle of a long context can be overlooked. The right order of operations: improve retrieval quality (embedding + chunking) first and hit high precision with a narrow context, then widen the context only against a measured need. Run the math with your own traffic profile in the cost calculator.

How It Works on Onysoft: the Generation-Layer Code

The generation layer connects to a single endpoint that follows the OpenAI schema exactly: https://api.onysoft.com/v1. Take your sk-ony- prefixed key from the dashboard and use the official openai Python package as-is — the chunks coming from the retrieval layer go into the context, and the model answers from that context only:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.onysoft.com/v1",
    api_key="sk-ony-YOUR_KEY",
)

# Retrieval layer: most relevant chunks from the vector DB for the query vector
# (embedding: local open-source model or external service; search: pgvector/Qdrant etc.)
chunks = vector_db.search(query_vector, top_k=5)  # 5 x ~800 ≈ 4,000-token context
context = "\n\n---\n\n".join(f"[{i+1}] {c.text}" for i, c in enumerate(chunks))

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",  # any model in the catalog
    max_tokens=600,
    messages=[
        {"role": "system", "content": "Answer only from the provided context. "
            "If it is not in the context, do not invent — say \"this is not in the sources\". "
            "Cite the chunk number you used in square brackets."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
    ],
)
print(response.choices[0].message.content)

The raw JSON of the response carries the real cost of that query in the cost field — no separate metering stack is needed for per-query cost tracking in RAG:

{
  "model": "anthropic/claude-sonnet-5",
  "usage": {
    "prompt_tokens": 4412,
    "completion_tokens": 486,
    "total_tokens": 4898
  },
  "cost": 0.0205
}

Building a chat interface? Streaming (SSE) is supported on the same endpoint: with stream=True you start rendering the moment the first token arrives — the single setting that most improves RAG's perceived speed. Schema details are in the API documentation; migrating from an existing OpenAI integration, the SDK migration guide walks through it step by step.

Going to Production: Latency Budget, Cost Ceiling, and Key Separation

Plan three things up front when taking RAG to production: the latency budget, the cost ceiling, and key separation. On latency the picture is clear: vector search finishes in the millisecond class, and total time is set by the generation layer. A measurement from our own production traffic: across 6,959 successful requests through the gateway in the last 14 days, average end-to-end response time was 3.8 seconds and the fastest was 0.3 seconds. Long-generation flagship requests pull the average up; with lightweight models and streaming, the RAG experience drops below a second.

On cost, the riskiest RAG failure is the runaway loop: a broken retry or agent loop firing 32K-context requests dozens of times a minute grows the bill fast. On Onysoft the ceiling for that scenario is structural: before every request the gateway estimates the cost, applies a 1.2x buffer, and compares it against your balance; if the balance does not cover it, the request returns HTTP 402 without ever reaching the provider. Combined with the prepaid balance, the absolute ceiling on spend is the amount you loaded — the full mechanism is in the cost control guide.

Key separation is part of the same discipline: give the RAG service its own sk-ony- key and define a per-key token limit — heavy trial traffic during indexing, or a leaked key, cannot sweep the production budget. Log the cost field together with a query ID and "how many lira does a question burn" stops being a guess; you also update your context-budget decisions with that data. To start, open a free account and try your own context + question pattern across several models side by side in the Playground; for the broader picture, the AI API guide is a good next stop.

Last updated: August 5, 2026 · Data: api.onysoft.com live catalog