AP
Agent Provider Depot
Opinionated picks for agents
OpenAPI

Agent classes / Local and offline agent / Hosted open-weight model inference

Hosted open-weight model inference for a local and offline agent

Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-09-14 (stale)

The class: Runs without calling a hosted model API, using local weights, a local index, or an air-gapped serve path.

  • No network call on the answer path
  • Say when the local corpus has no support

Depot Pick: Fireworks AI

When not to use it: Need Hub-native custom model endpoints (HF) or ultra-low latency on narrow SKUs (Groq).

Runner-up: Together AI.

Cheapest matching sourced SKU (usd_per_million_input_tokens): Chutes Qwen3.8-27B (Qwen/Qwen3.8-27B-TEE) $0.039 — source, asOf 2026-10-27. All sourced prices

Failure modes this class and job share (6)

  • Context overflow: If the input exceeds the window, say so and retrieve a narrower slice. Do not answer from a truncated head as if it were the whole source.
  • Lost in the middle: Ask the agent to name the passage it used. Spot-check questions whose only support sits in the middle of the context, not only at the edges.
  • Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
  • Schema drift: Validate every tool call and every loaded table against the current schema. Quarantine mismatches instead of coercing them.
  • Silent truncation: Tools must return whether the payload was truncated and how long the original was. Fail the task if the answer depends on the missing span.
  • Stale retrieval: Store the source URL and the time it was fetched. If a newer canonical page exists, or two stored sources disagree, do not answer from the older one.

Free options this class uses (21)

  • Qwen3-0.6B-Base (open_weights, oss) — source, asOf 2026-10-25
  • DeepSeek-V3.2 weights (open_weights, oss) — source, asOf 2026-10-25
  • Gemma models under the Gemma Terms of Use (open_weights, free) — source, asOf 2026-10-26
  • Ollama (local_runtime, oss) — source, asOf 2026-10-25
  • llama.cpp (local_runtime, oss) — source, asOf 2026-10-25
  • vLLM (local_runtime, oss) — source, asOf 2026-10-25
  • Qwen2.5-Coder-7B-Instruct (open_weights, oss) — source, asOf 2026-10-25
  • Qwen3-8B (open_weights, oss) — source, asOf 2026-10-25
  • Phi-4 (open_weights, oss) — source, asOf 2026-10-25
  • gpt-oss-20b (open_weights, oss) — source, asOf 2026-10-25
  • DeepSeek-R1 weights (open_weights, oss) — source, asOf 2026-10-25
  • SmolLM2-1.7B-Instruct (open_weights, oss) — source, asOf 2026-10-25
  • OLMo-2-1124-7B-Instruct (open_weights, oss) — source, asOf 2026-10-25
  • Mistral-7B-Instruct-v0.3 (open_weights, oss) — source, asOf 2026-10-25
  • Qwen2.5-VL-7B-Instruct (open_weights, oss) — source, asOf 2026-10-25
  • GLM-4-9B-Chat (open_weights, free) — source, asOf 2026-10-25
  • Granite-3.1-8B-Instruct (open_weights, oss) — source, asOf 2026-10-25
  • MLX (local_runtime, oss) — source, asOf 2026-10-25
  • SGLang (local_runtime, oss) — source, asOf 2026-10-25
  • llamafile (local_runtime, oss) — source, asOf 2026-10-25
  • exo (local_runtime, oss) — source, asOf 2026-10-25