Onysoft AI Gateway Node: Run AI Models on Your Own Computer and Use Them Through the API

Starting today, models on Onysoft AI Gateway don't have to run in the cloud. They can also run on your own computer. Onysoft AI Gateway Node is a desktop app for Windows 10/11 (64-bit) and Apple Silicon Macs. It runs open-source language models on your computer's GPU and lets you use them through the same API endpoint as every other model.

This post covers what the app does, how it works, which models fit on which hardware, and the real numbers we measured today on our own Mac. We also spell out what is available today and what is not.

What Is Onysoft AI Gateway Node?

Onysoft AI Gateway Node is a local desktop app that runs open-source language models on your own computer and connects them to your Onysoft account. The model runs on your hardware; you reach it from the dashboard chat, or from external tools with your personal API key.

The highlights:

• Platforms: Windows 10 and Windows 11 (64-bit), macOS 13.3 or later (Apple Silicon, M1 and newer).
• Engine: models run on the open-source llama.cpp engine: Metal on the Mac, the graphics card on Windows.
• Catalog: 31 open-source models including Gemma 4, Qwen3, Ministral 3, Phi-4 and gpt-oss-20b, all licensed under Apache 2.0 or MIT.
• Hardware check and recommendations: the app measures your hardware and uses AI-assisted recommendations to show the models that suit it.
• Sharing level: Low, Balanced or High. You decide how much of your machine goes to the models.
• Up to 3 models at once (1 in CPU mode).
• Free to use: requests to the model on your own device are not charged to your balance; token counts and duration are still logged.
• No open ports: your computer never accepts inbound connections. The app opens the connection to our server.

Why Run a Model on Your Own Computer?

Local AI, meaning a model running on your own hardware, is not the right choice for every job. For some jobs, though, it has clear advantages:

Privacy and control

The model runs on your computer, so your request isn't processed in a model provider's cloud. You know exactly which model, which version and which file is running. The model file sits on your disk and doesn't change unless you delete it. That matters for tests, internal tools and experiments where you need the same model to answer tomorrow.

Cost

Requests to the model on your own device are free. You use a GPU or a Mac you already own; your only extra costs are electricity and bandwidth. For lots of short, repetitive jobs (summarizing, classification, drafting, code-completion experiments) the savings add up quickly.

Same API, same tools

You don't need to learn a new tool to use a local model. The endpoint you already use for cloud models on Onysoft is the one you use for your own device. Only the model name changes.

Honest limits

A local model is only as capable as your hardware. A 4B to 12B parameter model on a 16 GB machine can't match the largest cloud models on every task. For long, complex reasoning, cloud models remain a strong option. On Onysoft both live in the same account and behind the same endpoint, so you can pick per task: fast, frequent jobs on your device, heavy jobs on the cloud models in our catalog.

How It Works: No Open Ports, the App Connects Out

The most important design decision in Onysoft AI Gateway Node is this: your computer never opens a port. No router settings, no port forwarding, no firewall rules. The app on your computer always opens the connection to the Onysoft server and asks for work over that connection.

The path a request takes:

1. Your tool (n8n, Cursor, your own code) sends the request to https://api.onysoft.com/v1/chat/completions as usual. The model name starts with node/.
2. The server checks that the key is your personal key and that the requested model is ready on one of your online devices.
3. The request is queued so that only your devices can pick it up.
4. The Node app on your computer picks up the job over its own outbound connection and runs the model locally (127.0.0.1).
5. The result goes back to the server and returns to your tool in the same format as any other model.
6. Token counts and duration are written to your usage log; the charge is 0.

Your tool ──HTTPS──▶ api.onysoft.com ◀──outbound connection── Node app
                       (queue)                            (your computer)
                                                                │
                                                        llama.cpp + model
                                                           127.0.0.1

Signing in to the app works the same way: the app shows a device code and you approve it in your browser with your Onysoft account. The app also checks for updates and tells you when a new version is out.

Hardware Requirements

The app measures your hardware and only recommends models that fit. The basic requirements:

Platform	Operating system	Hardware	Runs on
Windows (GPU)	Windows 10 / 11, 64-bit	NVIDIA (CUDA) recommended; discrete AMD and Intel GPUs are experimental via Vulkan. At least 6 GB of video memory and an up-to-date driver.	Graphics card
Windows (CPU mode)	Windows 10 / 11, 64-bit	If the GPU isn't enough: a 64-bit CPU with AVX2 and at least 8 GB of RAM.	Processor (CPU); slow, but it works
macOS	macOS 13.3 or later	Apple Silicon (M1 and newer), at least 8 GB of unified memory; 16 GB or more recommended for larger models. Intel Macs are not supported.	GPU (Metal)

Each model needs roughly 2 to 20 GB of free disk space; the app won't recommend a model that doesn't fit on your disk.

Which model fits on which computer?

The memory set aside for models (the budget) is calculated as follows:

• Mac: unified memory × 66% (75% above 36 GB) × sharing level. On a Mac the GPU and the system share the same memory, so not all of it goes to models.
• Windows: (video memory − 0.75 GB) × sharing level.
• Sharing level: Low 60%, Balanced 80%, High 95%.

A model needs more than its file size: the context memory (KV cache) and a small buffer are added. At the Balanced level, for a single model, roughly:

Hardware	Approx. budget	Models that fit (of 31)	Largest examples
Mac, 8 GB	~4.2 GB	6	Qwen3 4B, Gemma 4 E2B
Mac, 16 GB	~8.4 GB	15	Gemma 4 12B
Mac, 24 GB	~12.7 GB	21	gpt-oss-20b
Mac, 32 GB	~16.9 GB	24	Mistral Small 3.2 24B, Gemma 4 26B-A4B
Mac, 48 GB or more	~28.8 GB or more	31	The whole catalog
Windows, 8 GB GPU	~5.8 GB	8	Gemma 4 E4B, Qwen2.5 Coder 7B
Windows, 12 GB GPU	~9.0 GB	15	Gemma 4 12B
Windows, 16 GB GPU	~12.2 GB	21	gpt-oss-20b
Windows, 24 GB GPU	~18.6 GB	27	Qwen3 Coder 30B-A3B

Calculated with the app's budget formula and the memory requirements listed in the catalog. Multiple models share the same budget; the app does the exact calculation on your hardware.

CPU mode

On Windows, if your GPU isn't enough, the app can switch to CPU mode. It needs a CPU with AVX2 and at least 8 GB of RAM. The budget is calculated after reserving 4 GB for the operating system; only 1 model runs at a time, and only model files under 5.5 GB can be chosen. Responses are noticeably slower than on a GPU, so in CPU mode the default max_tokens is shorter if you don't set one (256 without streaming, 512 with streaming). Still, it lets you run a local model on a computer with no suitable graphics card for short jobs and experiments.

The Model Catalog: 31 Open-Source Models

Every model in the catalog is licensed under Apache 2.0 or MIT. The models are GGUF builds of the open weights published by their makers, quantized to roughly 4 bits (Q4 or MXFP4). After downloading, the app verifies every model file against its SHA256 hash; a file that doesn't match is discarded.

Model families and makers:

• Gemma 4 (Google): E2B, E4B, 12B, 26B-A4B and 31B.
• Qwen (Alibaba / Qwen): Qwen3 4B, 8B, 14B, 30B-A3B, 32B; Qwen3.5 4B and 9B; Qwen3.6 35B-A3B; Qwen3.8 27B; Qwen2.5 Coder 7B and 14B; Qwen3 Coder 30B-A3B.
• Mistral: Ministral 3 (3B, 8B, 14B), Mistral Small 3.2 24B and Devstral Small 2 24B.
• Phi-4 (Microsoft): Phi-4 14B, Phi-4 mini 3.8B and Phi-4 Reasoning Plus 14B.
• gpt-oss-20b (OpenAI).
• GLM (Zhipu): GLM-4 9B and GLM-4.7 Flash.
• Granite 4.2 (IBM): 3B and 8B.
• DeepSeek R1 0528 Qwen3 8B (DeepSeek).

Models marked "MoE" use a mixture-of-experts architecture: the total parameter count is large, but only part of it runs for each token. In Qwen3 30B-A3B, for example, about 3 billion of the 30 billion parameters are active, so it needs a lot of memory but runs closer to the speed of a small model.

The full list (purpose is the catalog label; file size is the model file you download):

Model	Catalog ID	Purpose	File	License
Ministral 3 3B	ministral-3-3b-instruct-2512-q4_k_m	Fast / light	2.1 GB	Apache 2.0
Granite 4.2 3B	granite-4.2-3b-q4_k_m	Fast / light	2.2 GB	Apache 2.0
Phi-4 mini 3.8B	phi-4-mini-instruct-3.8b-q4_k_m	Fast / light	2.5 GB	MIT
Qwen3.5 4B	qwen3.5-4b-q4_k_m	Fast / light	2.7 GB	Apache 2.0
Gemma 4 E2B	gemma-4-e2b-it-qat-q4_0	Fast / light	3.3 GB	Apache 2.0
Qwen3 30B-A3B (MoE)	qwen3-30b-a3b-q4_k_m	Fast / light	18.6 GB	Apache 2.0
Qwen3 4B	qwen3-4b-q4_k_m	General	2.5 GB	Apache 2.0
Qwen3 8B	qwen3-8b-q4_k_m	General	5.0 GB	Apache 2.0
Gemma 4 E4B	gemma-4-e4b-it-qat-q4_0	General	5.2 GB	Apache 2.0
Ministral 3 8B	ministral-3-8b-instruct-2512-q4_k_m	General	5.2 GB	Apache 2.0
Granite 4.2 8B	granite-4.2-8b-q4_k_m	General	5.3 GB	Apache 2.0
Qwen3.5 9B	qwen3.5-9b-q4_k_m	General	5.7 GB	Apache 2.0
GLM-4 9B 0414	glm-4-9b-0414-q4_k_m	General	6.2 GB	MIT
Gemma 4 12B	gemma-4-12b-it-qat-q4_0	General	7.0 GB	Apache 2.0
Ministral 3 14B	ministral-3-14b-instruct-2512-q4_k_m	General	8.2 GB	Apache 2.0
Qwen3 14B	qwen3-14b-q4_k_m	General	9.0 GB	Apache 2.0
Gemma 4 26B-A4B (MoE)	gemma-4-26b-a4b-it-qat-q4_0	General	14.4 GB	Apache 2.0
GLM-4.7 Flash (MoE)	glm-4.7-flash-q4_k_m	General	18.3 GB	MIT
Qwen3.8 27B	qwen3.8-27b-q4_k_m	General	19.0 GB	Apache 2.0
Qwen3 32B	qwen3-32b-q4_k_m	General	19.8 GB	Apache 2.0
Qwen3.6 35B-A3B (MoE)	qwen3.6-35b-a3b-q4_k_m	General	20.4 GB	Apache 2.0
Qwen2.5 Coder 7B	qwen2.5-coder-7b-q4_k_m	Code	4.7 GB	Apache 2.0
Qwen2.5 Coder 14B	qwen2.5-coder-14b-q4_k_m	Code	9.0 GB	Apache 2.0
Devstral Small 2 24B	devstral-small-2-24b-instruct-2512-q4_k_m	Code	14.3 GB	Apache 2.0
Qwen3 Coder 30B-A3B (MoE)	qwen3-coder-30b-a3b-q4_k_m	Code	18.6 GB	Apache 2.0
DeepSeek R1 0528 Qwen3 8B	deepseek-r1-0528-qwen3-8b-q4_k_m	Reasoning	5.0 GB	MIT
Phi-4 14B	phi-4-14b-q4_k_m	Reasoning	9.1 GB	MIT
Phi-4 Reasoning Plus 14B	phi-4-reasoning-plus-14b-q4_k_m	Reasoning	9.1 GB	MIT
gpt-oss 20B (MoE)	gpt-oss-20b-mxfp4	Reasoning	12.1 GB	Apache 2.0
Gemma 4 31B	gemma-4-31b-it-qat-q4_0	Reasoning	17.7 GB	Apache 2.0
Mistral Small 3.2 24B	mistral-small-3.2-24b-instruct-2506-q4_k_m	Turkish	14.3 GB	Apache 2.0

The name you use in the API is the catalog ID with a node/ prefix, for example node/gemma-4-e2b-it-qat-q4_0 or node/gpt-oss-20b-mxfp4.

Using Your Own Device Through the API

There are two ways to use the model on your own device:

• Dashboard chat: it appears in the model list as "My device · <model>". Pick it and start chatting.
• API: with the personal API key you create in the dashboard, from external tools such as n8n, Cursor, LangChain, the OpenAI libraries or your own code.

The endpoint and the key are the same as for every other model in the catalog. Only the model name is different:

curl https://api.onysoft.com/v1/chat/completions \
  -H "Authorization: Bearer sk-ony-YOUR-KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "node/qwen3-4b-q4_k_m",
    "messages": [
      {"role": "user", "content": "Hi! Could you briefly introduce yourself?"}
    ],
    "max_tokens": 300
  }'

The same request with the OpenAI Python library:

from openai import OpenAI

client = OpenAI(base_url="https://api.onysoft.com/v1", api_key="sk-ony-YOUR-KEY")

resp = client.chat.completions.create(
    model="node/qwen3-4b-q4_k_m",
    messages=[{"role": "user", "content": "Summarize this sentence: ..."}],
)
print(resp.choices[0].message.content)

Model naming

Model name	Meaning
node/qwen3-4b-q4_k_m	Any of your online devices where this model is ready.
node/qwen3-4b-q4_k_m@12	Only your device number 12. If it is offline, the request doesn't go to another device and returns 503 node_unavailable.

To see which models are ready right now, call GET https://api.onysoft.com/v1/node/models. For each model, the response includes the name to use (id), the device-specific name (device_model_id) and whether it is ready (ready).

Limits to know

• Only personal API keys work; project and partner keys get 403 node_owner_only.
• Up to 4 requests per user are processed at the same time.
• If you don't set max_tokens, the default is 2048; the maximum is 8192.
• tools, tool_choice and response_format are passed to the device; we confirmed tool calling in today's test.
• stream:true is supported, but because the device sends the answer once it has finished generating, the chunks arrive together when the answer is complete.

All parameters, error codes and time limits are in the Models on Your Own Device documentation.

Real Measurements: Apple M5, 16 GB

The results below were measured on September 22, 2026 on an Apple M5 Mac with 16 GB of unified memory. Requests went through the Onysoft API, the same way a customer would send them.

Measurement	Result
Qwen3 4B generation speed	~26.5 tokens/s
Gemma 4 E2B generation speed	~22 tokens/s
Short answer, non-streaming (through the API, end to end)	0.8–1.5 s
Short answer, streaming (through the API, end to end)	~1 s
Tool calling (tools)	Worked

These numbers come from a single computer and short answers. Speed depends on the model, your GPU or memory, the length of the answer and how many models are running at once. Larger models are slower, and CPU mode is noticeably slower than a GPU.

Security and Privacy

• Requests only go to your devices. A request whose model starts with node/ runs only on the devices of the key's owner. Another user's request can't reach your device this way.
• No open ports. Your computer accepts no inbound connections; the model only listens on the local address (127.0.0.1) and the app opens the connection.
• Model files are verified. Every file is checked against its SHA256 hash; a mismatched file is not used.
• Sign-in with a device code. The app shows a code and you approve it in your browser with your account.
• Transparent logging. The request passes through the Onysoft server so the result can reach you, and, as with every other model, it is written to your own usage log with token counts, duration and content. That way you can follow your own-device usage in the dashboard too.

The Onysoft GPU Sharing Program

Onysoft AI Gateway Node is distributed as part of the Onysoft GPU Sharing Program, which is open to both individuals and organizations.

It takes three steps:

1. Apply: submit an individual or organizational application on the GPU Sharing Program page. As part of the application you read and accept the program's agreements and policies.
2. Approval: your application is reviewed, and you are notified once it is approved.
3. Download and connect: approved users download the Windows or macOS app from the dashboard (My GPU Sharing), link it to their account with a device code and choose their models.

What is available today, and what is not

Feature	Status today
Using the model on your own device from the dashboard chat	Available
Using the model on your own device through the API (node/)	Available, free
Windows and macOS (Apple Silicon) app	Available, from the dashboard after approval
CPU mode on Windows	Available
Routing other customers' requests to your device	Off
Revenue sharing or payouts	Not available

In other words, today Onysoft AI Gateway Node lets you use your own hardware for your own work. We'll announce new options separately as the program grows.

Wrapping Up

With Onysoft AI Gateway Node, anyone with a capable graphics card or an Apple Silicon Mac can run open-source models such as Gemma 4, Qwen3, Phi-4 and gpt-oss-20b on their own computer, and use them for free through the API they already know. In short:

• 31 open-source models, all Apache 2.0 or MIT, downloaded and verified with SHA256.
• Windows 10/11 and Apple Silicon Macs; CPU mode on Windows when the GPU isn't enough.
• Same endpoint: POST /v1/chat/completions, model name node/<catalog_id>.
• No open ports; requests only go to your own devices.
• About 26.5 tokens/s with Qwen3 4B on an Apple M5 with 16 GB; short answers in about 1 second through the API.

To apply and learn more, visit the GPU Sharing Program page. Technical details are in the documentation, and all cloud models are on the models page.

Related reading: setting base_url in the OpenAI SDK, using AI code assistants with one key, KVKK-compliant AI usage and what we learned running an open-source model on our own server.