Multi-provider Model Access
Enables calling over 100 LLMs using a consistent OpenAI input/output format with automatic translation to each provider's specific endpoints.
Spend Tracking and Budgets
Monitors costs per project or user and allows setting budget limits across different LLM deployments.
Load Balancing and Fallback Logic
Supports retry and fallback mechanisms across multiple deployments through a Router feature to ensure reliability.
Centralized API Gateway
Provides authentication, authorization, and multi-tenant cost tracking with virtual keys for secure access control.
Consistent Output Format
Ensures text responses are always available at the same location in the response structure regardless of the LLM provider used.
Rate Limiting and Guardrails
Allows control over request rates and implementation of safety guardrails on a per-project basis.