AI Infrastructure & Model APIs

Quick Decision

AI infrastructure tools sit underneath the apps people see. They route model calls, host open models, run GPU workloads, store embeddings, power RAG, transcribe audio, generate media, and help teams compare cost, latency, quality, and control without rebuilding the stack every month.

This category is for developer and platform buyers. If the user is choosing a chatbot, start with AI Chatbots. If the team is shipping an AI product, agent, retrieval layer, or model-backed workflow, this is the better lane.

The late-May infrastructure update is agent control. CoreWeave’s training-to-inference loop pushes traces, evals, RL, inference, and W&B tooling into one reliability story. OpenAI’s Rosalind Biodefense trusted-access expansion shows that specialist frontier models may ship as gated capability programs. Sysdig’s LLM-agent intrusion report makes runtime telemetry and least-privilege design part of infrastructure buying, not only security cleanup.

The June 3 update widens that control story. Microsoft Build put Work IQ and Foundry around enterprise agents; GitHub made the Copilot SDK generally available while AI Credits became the agent-usage meter; NVIDIA pushed enterprise agents, Cosmos 3, open physical-AI agent skills, Alpamayo 2 Super, RTX Spark, and DGX Station for Windows; Postman launched AI Engineer for API work; RelationalAI moved agentic decision intelligence deeper into Snowflake; 7AI kept security agents in the proactive-hunting lane; and the White House AI cybersecurity order put advanced AI cyber capability into public-sector and critical-infrastructure policy. Infrastructure buyers should evaluate agent stacks by context access, runtime isolation, traces, evals, spend controls, simulation/data pipelines, local-vs-cloud compute, and write-action approvals.

The June 24 update keeps model availability as a first-class infrastructure risk. Claude Fable/Mythos access is still not a normal public self-serve buyer route, GPT-5.2 is retired from ChatGPT, and OpenAI faces reported state-AG scrutiny. Direct frontier API buyers should now document the exact model route in production, the fallback route if a model is suspended, account-specific, preview-only, or retired, the retention policy for that model class, the staff/client access exposure for restricted routes, and the legal/privacy review path for sensitive users. The AI Model Availability & Churn Tracker is now the canonical AiPedia surface for these app/API/router distinctions.

The June 16 infrastructure update is governed data agents. Google Cloud’s data-agent rollout puts Conversational Analytics, Data Engineering Agent, Looker agents, Gemini Enterprise data access, Data Agent Kit, Managed MCP Servers for Databases are GA, while many of the more ambitious analytics, Looker, Gemini Enterprise, and commerce routes remain preview. Infrastructure teams should evaluate these by IAM scope, roles/mcp.toolUser, service permissions, separate production identities, SQL verification, BigQuery spend limits, job labels, audit logging, Model Armor payload logging, and GA-versus-preview fallback plans.

Use OpenRouter when you need one API across many model providers. The current pricing page lists pay-as-you-go access to 400+ models and 60+ providers, with budget controls, activity logs, prompt caching, preferred vendor selections, and model-priced token billing. Its May 27 funding signal makes the category clearer: routing, fallback, governance, and spend visibility are becoming production infrastructure, not just developer convenience.

Use direct vendor APIs when native features matter. OpenAI API is the default direct route for broad multimodal app work. Claude API is the direct route for long reasoning, writing, code, and document workflows. Gemini API inputs, or Veo video generation are part of the product. The June 22 Gemini recheck keeps Gemini 3.5 Flash pricing mode-specific: standard, batch/flex, priority, grounding, tools, and media rows need separate cost modeling.

Use Mistral AI or Groq when price/performance, open-model strategy, European infrastructure, or low-latency inference matters. The June 24 Groq check adds Qwen 3.6 27B, Kimi buyer math. The June 24 Mistral check keeps the timeline and cost model honest: Mistral 3 officially launched on December 2, 2025, while Medium 3.5’s model-card date is April 28, 2026. Current Mistral pricing lists Large 3 at $0.50/M input and $1.50/M output, Medium 3.5 at $1.50/M and $7.50/M, and Small 4 at $0.10/M and $0.30/M, but the Small 4 model card still lists $0.15/M and $0.60/M, and the pricing FAQ still uses a generic Mistral Large $2/$6 example. Benchmark real prompts, confirm the live Studio quote, and pin exact model IDs before switching because model quality, output length, retries, aliases, and source drift change the bill.

Use Replicate or fal.ai when the job is hosted image, video, audio, 3D, or custom-model inference. The June 9 Replicate check keeps it strongest as a broad model catalog and custom-model deployment layer: public models may bill by hardware time or by input/output, while most private deployments bill setup, idle, and active time unless they are labeled fast-booting fine-tunes. fal is stronger when successful-output billing and fast media APIs are the buyer problem; the June 2 check keeps prepaid credits, queue behavior, failed-output billing, and the 50% batch discount as the key pricing details to model.

Use Fireworks AI when the workload is production inference over open or commercial models., cached-token discounts, batch jobs, dedicated GPU deployments, fine-tuning, and B200/B300 capacity are the actual purchase.

Use Browserbase when the infrastructure problem is web interaction.. It belongs here when agents need reliable browser sessions, Fetch/Extract, replay, and model routing rather than just another LLM.

Use Firecrawl when the infrastructure problem is web data for agents. The June 28 check adds it as the cleaned-content counterpart to Browserbase: search, scrape, crawl, structured extraction, screenshots, Interact, and LLM-ready markdown in one credit-based API. Model the exact endpoint mix before buying because crawls, screenshots, browser minutes, and retries can change the bill quickly.

Use Composio when agents need app actions and user-scoped auth. The June 28 check keeps it in the tool-calling infrastructure lane: 1000+ app toolkits, managed authentication, native session tools, hosted MCP URLs, and usage-based tool-call pricing. It is a developer tool layer, not a no-code workflow canvas.

Use LiteLLM when LLM traffic needs an open-source gateway. The June 28 check adds LiteLLM/proxy lane for one OpenAI-compatible interface across 100+ providers, with routing, virtual keys, spend tracking, budgets, guardrails, and Enterprise controls such as SSO, logs, fallback quality, and enterprise-directory licensing still need review.

Use LangSmith when agent observability, evals, and deployment control need a LangChain-native home. The June 28 check keeps Developer at $0/seat/month, Plus at $39/seat/month, and Enterprise custom, but the buyer risk is broader than seats: traces, extended retention, deployment runs, deployment uptime, Fleet, Engine, sandboxes, and outside model/API spend all need usage limits.

Use Braintrust when the infrastructure problem is eval discipline. Braintrust belongs here when datasets, experiments, prompt comparisons, traces, scores, monitoring, and human review need to decide whether a model, prompt, retrieval, or agent change is safe to ship. Starter is useful for prototypes; Pro at $249/month is the first serious shared-team tier, but usage meters still matter.

Use Arize Phoenix when the infrastructure problem is OpenTelemetry-native AI observability. Phoenix belongs here for teams that need traces, evals, prompt iteration, datasets, and experiments tied to LLM app behavior. Self-hosted Phoenix is the open-source path; AX Pro at $50/month is the hosted small-team route, but span volume and Elastic License 2.0 limits need review.

Use OpenLIT when OpenTelemetry alignment is the first requirement. The June 28 check adds OpenLIT traces, metrics, logs, token and cost tracking, prompt workflows, evals, dashboards, vector database monitoring, and GPU monitoring. The self-hosted product is listed at $0 forever, while OpenLIT Cloud is coming soon with no public price verified.

Use Opik when agent traces and evals need an OSS-to-hosted path. Opik belongs here when teams want trace debugging, Test Suites, assertions, LLM-as-judge metrics, production monitoring, annotation queues, and Comet-hosted retention. Start with OSS or Free Cloud, then model Pro Cloud at $19/month plus span and retention meters before rollout.

Use Inspect AI when evals need to be code-defined and sandboxed. The June 28 check adds Inspect AI as the MIT-licensed UK AI Security Institute and Meridian Labs framework for coding, agentic, reasoning, knowledge, behavior, and multimodal evals. It supports datasets, agents, tools, scorers, Inspect View, a VS Code extension, external agents, MCP or custom tools, and sandbox routes. It is free software, but model calls, compute, sandboxes, storage, and review time still need budgets.

Use Guardrails AI when LLM outputs need runtime validation before downstream systems trust them. Guardrails adds reusable validators, Guards, input/output checks, Pydantic-style structured data, on-fail policies, and Guardrails Hub installs around app code. The Apache-2.0 framework is the validated route; public hosted or remote-validator pricing was not verified in the June 28 check.

Use LM Evaluation Harness when benchmark comparability matters. EleutherAI’s MIT framework is the standard-benchmark lane for 60+ academic benchmarks, hundreds of task variants, custom prompts and metrics, and backend routes across Hugging Face, vLLM, API models, OpenAI-compatible local servers, SGLang, OpenVINO, NeMo, Megatron-LM, and more. Pair it with private product evals before making release decisions.

Use OpenAI Evals only as a migration bridge. OpenAI’s deprecations page says Evals platform deprecation was announced June 3, 2026, existing evals become read-only on October 31, 2026, and the dashboard/API are scheduled to shut down on November 30, 2026. Existing users should inventory, export, or migrate eval coverage; new teams should choose a maintained eval stack instead.

Use LangWatch when the team wants open-source LLMOps with traces and evals. LangWatch sits between local eval frameworks and managed observability products: traces, evaluations, datasets, AI gateway workflows, DSPy optimization, self-hosting, and event-sourcing operations. Paid plans start from EUR59/month, but event and retention meters decide real cost.

Use Respan when LLMOps and gateway routing should share one traffic history. The June 28 check adds Respan as the active Keywords AI successor surface: traces, metrics, evals, prompt management, monitors, spend limits, and an OpenAI-compatible gateway sit in one product lane. Free is the first instrumentation route, Team is the shared paid route, and Enterprise is custom. Confirm the live Team base price, retention, data-handling terms, gateway latency, and provider bills before standardizing.

Use Patronus AI when AI reliability needs managed evals plus frontier simulation research. Patronus covers evaluators, experiments, datasets, comparisons, traces, prompts, annotations, guardrails, and Percival-assisted eval creation, while its current company positioning also highlights Digital World Models for simulating agent actions in digital workflows. Confirm whether procurement is buying eval operations, Digital World Model access, or services.

Use DeepEval when LLM evaluation should live in code. DeepEval evals, agent evals, chatbot tests, safety checks, tracing, and CI gates. Confident AI is the hosted platform route when teams need shared evaluation, observability, red teaming, governance, and security controls.

Use Traceloop when OpenTelemetry alignment is the observability requirement. Traceloop and OpenLLMetry instrument LLM apps with OpenTelemetry traces, quality checks, alerts, prompt and model change testing, and IDE reruns. The ServiceNow acquisition path can help enterprise buyers, but roadmap and procurement details need live confirmation.

Use Ragas when RAG and LLM evaluation should live in code. Ragas is the open-source framework lane for metrics, synthetic test data, experiments, and cost-aware evaluation loops. It is free as a framework, but evaluator model calls, embeddings, generated test data, and human review still need a budget.

Use OpenPipe when fine-tuning can lower cost or latency. OpenPipe belongs here when production logs can become datasets, fine-tuned models, DPO runs, evaluations, and hosted inference. Start only after the team has enough clean examples and baseline evals to prove a specialized model beats prompt-only changes.

Use Portkey when LLM traffic needs a gateway control plane. Portkey, governance, caching, and analytics. The Developer plan is explicitly not the production path; Production and Enterprise should be modeled around recorded logs, retention, overages, private hosting, compliance, and provider spend.

Use Zep when the infrastructure problem is production agent memory. Zep is the context-engineering lane for temporal context graphs, persistent user/session memory, and managed deployment options. It belongs beside Mem0 and vector databases, but buyers should decide whether they need memory extraction and graph context or only document retrieval.

Use promptfoo when AI security testing is part of the infrastructure gate. promptfoo is now positioned around open-source evals, red teaming, guardrails, model security, MCP proxy checks, code scanning, and enterprise/on-premise testing workflows. OpenAI announced a March 2026 acquisition agreement, and promptfoo says it is now part of OpenAI, so procurement should verify contract and data-handling paths.

Use Mem0 when the infrastructure problem is agent memory. Mem0 belongs beside vector databases and agent runtimes when an app needs persistent, scoped memories across users, sessions, and agents. The managed Platform is the faster route; the Apache-2.0 open-source route shifts vector database, model, embedder, hosting, deletion, and privacy controls back to the team.

Use Tavily when agents need real-time web search and extraction as an API. Tavily is closer to a search, extract, crawl, map, and research primitive than a browser infrastructure layer. Model basic versus advanced search, extract batches, mapping, crawl size, research mode, retries, and agent-loop behavior before production.

Use LlamaIndex when agents need structured context over private data. The June 28 check adds it as the framework lane for RAG, context augmentation, data connectors, indexes, retrieval, workflows, and agents over domain-specific data. The framework is MIT-licensed, but LlamaParse/LlamaCloud credits, model calls, embeddings, vector storage, ingestion jobs, and evals still need separate budgets.

Use Haystack when RAG and agent apps should be built as open-source pipelines. Haystack belongs here for Apache-2.0 components, pipelines, document stores, tools, agents, multimodal search, and reusable LLM app architecture. deepset AI Platform is the managed route when deployment, governance, testing, support, security, or dedicated resources matter.

Use DSPy when prompt behavior needs measurable optimization. DSPy is a coding-first framework, but it belongs in infrastructure shortlists when an LLM system needs signatures, modules, metrics, optimizers, and examples instead of hand-tuned prompt sprawl. Budget for model calls and data work before assuming optimization will lower costs.

Use Agno when agents need an owned AgentOS-style runtime. Agno is primarily an automation and agent-platform framework, but it belongs in infrastructure shortlists when teams need agents, teams, workflows, memory, knowledge, storage, traces, audit logs, interfaces, and control-plane management in their own stack. Free open source is the first route; Pro is $150/month for a live AgentOS control plane; Enterprise is custom.

Use Instructor when structured LLM outputs are the infrastructure boundary. Instructor is the MIT-licensed library lane for validated JSON, Pydantic-style schemas, retries, and provider adapters. It is useful when extraction, classification, enrichment, or tool arguments need typed outputs, but it still needs evals, retry budgets, and monitoring.

Use Chainlit when a Python LLM workflow needs a quick conversational interface. Chainlit is an Apache-2.0 framework for prototypes, internal chat tools, and demos around RAG or agent workflows. Treat it as a UI framework, not a managed chatbot platform, and verify maintainer/support route, auth, persistence, hosting, and observability before production use.

Use Deepgram when speech is infrastructure. Deepgram is a better fit for product teams adding STT, TTS, audio intelligence, or voice agents than for creators who only need a one-off transcript.

Use Hugging Face when model discovery, model cards, datasets, Spaces, and managed endpoints need to live in one open-AI collaboration surface. The June 23 pricing check keeps Pro at $9/month, Team at $20/user/month, Enterprise from $50/user/month, storage at $12/TB public and $18/TB private before volume discounts, ZeroGPU on RTX Pro 6000 Blackwell, and Inference Endpoints starting around low hourly CPU pricing.

Buyer Paths

How to Choose

• Model routing: Pick OpenRouter when you need one OpenAI-compatible API across many providers.
• Direct LLM APIs: Pick OpenAI, Claude, or Gemini when native features, procurement, and provider-specific controls matter.
• Cost and latency: Pick Mistral AI, OpenRouter, or Groq when you can benchmark quality against real prompts and need tighter unit economics.
• Open-model infrastructure: Pick Together AI when you need hosted inference, fine-tuning, dedicated endpoints, code sandboxes, and GPU capacity for open-model products. The June 25 check separates serverless model-token pricing from dedicated inference and GPU clusters, so benchmark your actual traffic before assuming a single “open model” unit cost.
• Model catalog and experiments: Pick Hugging Face for discovery, datasets, model cards, demos, Spaces, ZeroGPU, and endpoints.
• Media and community models: Pick Replicate when the job is running image, video, audio, or custom models by API. The June 9 check confirms buyers should model public output-priced examples separately from hardware-time runs and private deployments that can bill while idle.
• Fast media APIs: Pick fal.ai when successful-output billing, image/video/audio/3D endpoints, and fast experimentation matter.
• Production inference: Pick Fireworks AI when hosted model APIs, batch inference, dedicated GPU deployments, and fine-tuning are more important than a polished chatbot UI.
• Browser automation: Pick Browserbase when an AI agent, scraper, QA runner, or workflow needs managed browsers, Search/Fetch, Functions runtime, identity, observability, Model Gateway, and Stagehand-style automation.
• Web data extraction: Pick Firecrawl when the agent needs pages transformed into markdown or structured data more than it needs a long-running browser session.
• Agent tool access: Pick Composio when the hard problem is app actions, user authentication, session tools, MCP access, and connector maintenance.
• LLM gateways: Pick LiteLLM when the team wants a self-hosted OpenAI-compatible gateway, routing, virtual keys, budgets, guardrails, and provider fallback before model calls reach apps.
• Managed LLMOps and gateway: Pick Respan when traces, evals, prompts, monitors, spend limits, and gateway routing should share the same production traffic history.
• OpenTelemetry-first observability: Pick OpenLIT when the team wants Apache-2.0 self-hosted LLM telemetry and is ready to own collectors, storage, retention, and dashboards.
• Agent eval operations: Pick Opik when agent traces, Test Suites, LLM-as-judge metrics, and hosted span retention are the buying reason.
• Custom eval frameworks: Pick Inspect AI when coding, agent, safety, reasoning, knowledge, behavior, or multimodal evals need to live in Python tasks with tools, scorers, and sandboxes.
• Runtime guardrails: Pick Guardrails AI when validated structured outputs, reusable validators, and on-fail policies need to run before downstream actions trust model output.
• Model benchmarks: Pick LM Evaluation Harness when standard academic or leaderboard-style benchmark comparability is the job.
• OpenAI eval migration: Treat OpenAI Evals as a migration item because hosted evals become read-only on October 31, 2026 and shut down on November 30, 2026.
• Agent observability: Pick LangSmith when LangChain or LangGraph production work needs traces, evals, prompt workflows, deployment, and usage controls in one managed lane.
• AI evals: Pick Braintrust when release quality depends on datasets, experiments, prompt comparisons, scored outputs, and review workflows.
• OpenTelemetry AI observability: Pick Arize Phoenix when traces, prompt iteration, evals, datasets, and experiments should sit close to engineering instrumentation.
• Enterprise AI reliability: Pick Patronus AI when managed evals, traces, datasets, guardrails, and frontier agent-simulation research are part of the buying question.
• OpenTelemetry LLM observability: Pick Traceloop when OpenLLMetry instrumentation, quality checks, alerts, and ServiceNow procurement alignment matter.
• Open-source LLMOps: Pick LangWatch when the team wants traces, evals, datasets, DSPy optimization, and self-hosting in one open-source product surface.
• Code-first LLM evals: Pick DeepEval for broad LLM, RAG, agent, safety, tracing, and CI evals in code, and Ragas for RAG-specific metrics, synthetic test data, and experiments in Python.
• Fine-tuning: Pick OpenPipe when logs can become datasets and specialized models can reduce cost or latency.
• Typed LLM calls: Pick BAML when structured outputs, generated clients, robust parsing, and tests should be part of the app code contract.
• Gateway governance: Pick Portkey when live AI traffic needs routing, keys, prompts, guardrails, caching, logs, budgets, and retention policy in one control layer.
• Agent memory: Pick Zep or Mem0 when the product needs explicit, persistent memory rather than only retrieved documents.
• Red-team testing: Pick promptfoo when the team needs local evals, jailbreak proxy review, and security evidence before launch.
• Agent search API: Pick Tavily when the app needs web search, extraction, crawl, mapping, and research endpoints rather than a browser session.
• RAG frameworks: Pick LlamaIndex when context augmentation and agents over private data are the job; pick Haystack when open-source components, pipelines, document stores, and reusable LLM app architecture are the job.
• LLM program optimization: Pick DSPy when a repeated LLM task has examples and metrics that can guide optimization.
• Agent platform runtime: Pick Agno when developers want an open-source AgentOS-style stack for agents, teams, workflows, memory, knowledge, traces, audit logs, and interfaces.
• Structured outputs: Pick Instructor when the problem is validated JSON or typed LLM output in application code.
• Conversational AI interface: Pick Chainlit when a Python RAG or agent workflow needs a quick chat UI for prototypes or internal tools.
• Speech APIs: Pick Deepgram when speech-to-text, voice agents, or audio intelligence are infrastructure, not just creator utilities.
• Serverless GPU apps: Pick Modal when you want Python jobs, endpoints, queues, sandboxes, and GPU workloads without Kubernetes. The June 25 check keeps Starter at $0 with $30/month credits, Team at $250/month plus compute with $100/month credits, B200 at $0.001736/sec, H200 at $0.001261/sec, H100 at $0.001097/sec, region multipliers at 1.5x to 1.75x, non-preemptible execution at 3x, and B200+ as a compatibility route that can run on B200 or B300 while billing as B200.
• Open-weight model family: Pick Llama when infrastructure needs self-hostable or provider-hosted open weights card at $0.11/M input and $0.34/M output, and Together Maverick at $0.27/M input and $0.85/M output. Treat provider-specific availability, context, and pricing as live checks.
• Local model runtime: Pick LM Studio when developers need a desktop GUI plus native v1 REST API, OpenAI-compatible and Anthropic-compatible endpoints, MCP support, SDKs, CLI server control, and LM Link for Llama, Qwen, Mistral, and other open weights. LM Studio has been free for ordinary home and work use since its July 2025 terms change, while Enterprise is the sales-led route for SSO, model/MCP gating, and private collaboration.
• Managed vector search: Pick Pinecone, backups, imports, and reranking before treating the database price as the whole retrieval bill.
• Open vector databases and agent memory: Pick Weaviate or Qdrant when self-hosting optionality and control matter. The June 10 Weaviate check keeps Free, Flex from $45/month, Plus from $280/month, Premium from $400/month, Weaviate Embeddings at $0.025-$0.065 per 1M tokens, Query Agent at a free 1,000-request/month trial path or $30/org/month with 4,000 included requests, and Engram generally available as a managed memory/context service for agents. The June 25 Qdrant check keeps the Free Cloud testing tier at 1GB RAM and 4GB disk; Standard as usage-based production cloud; Premium as the enterprise-support tier; Hybrid/Private Cloud as the control-first path; and v1.18.2 as the latest release checked, with security fixes included in the release notes.

Money Pages To Keep Current

• Best pay-as-you-go AI tools and APIs was refreshed June 27, 2026 to separate true metered API usage from flat subscriptions and keep OpenAI, Claude, Gemini, OpenRouter, Mistral, Groq, Replicate, fal, Deepgram, ElevenLabs, and Fish Audio pricing risk in one buyer path.
• Best open source AI tools was refreshed June 27, 2026 for Ollama, LM Studio, Open WebUI, Llama, Mistral, DeepSeek, FLUX, Stable Diffusion, Whisper, and Hugging Face because open-model buyers often compare local control against hosted pay-as-you-go APIs.
• Best AI tools for developers is the June 6 verified developer guide for separating Cursor, GitHub Copilot AI Credits, Claude Code shared limits/API credits, Codex token credits, Replit Agent, and Aider BYOK API costs.
• A new OpenRouter vs direct APIs comparison would capture buyers choosing between a model router and direct OpenAI/Anthropic/Google contracts.
• A new Replicate vs fal.ai comparison would capture image/video/API buyers choosing between broad model catalog and fast media-generation infrastructure.

Watchouts

Infrastructure tools are powerful because they hide messy systems. That can also hide cost and governance risk. Before standardizing, test real workloads, pin model routes where quality matters, model retry costs, and document what data can pass through each provider.

Do not publish infrastructure pages with old flat monthly subscription framing. The buyer question is usually total workload cost: input tokens, output tokens, cached tokens, web/search tools, video seconds, generated images, GPU runtime, voice minutes, channels, retries, and failed generations.